Grafana
This page is about the analytics and visualization application Grafana, one of our recommended components for Cinchy v5 on Kubernetes.
Grafana is an open source analytics and interactive visualization web application. When connected to your Cinchy platform, it provides charts, graphs, and alerting capabilities (Image 1).
Grafana, and its paired application Prometheus (which consumes metrics from the running components in your environment) is the recommended visualization application for Cinchy v5 on Kubernetes.
Image 1: An example visualization from Grafana
Grafana has a robust library of documentation of tutorials designed to help you learn the fundamentals of the application. We have listed a few notable ones below:
When using the default configuration pairing of Grafana and Prometheus, Prometheus is already set up as a data source in your metrics dashboard.
There are some saved dashboards that come out of the box with your Cinchy deployment. These dashboards will provide a great jumping off point for your metrics monitoring, and you can always customize, manage, and add further dashboards at your leisure.
- 1.Navigate to the left navigation pane, select the Dashboards icon > Manage (Image 2).
Image 2: Accessing your Grafana Dashboards
2. You will see a list of all of the Dashboards available to you (Image 3). Clicking on any of them will take you to a full metrics view (Image 4).
Image 3: Step 2, Viewing all dashboards
Image 4: Step 3, an example visualization
3. You can favourite any of your commonly used or most important dashboards by clicking on the star (Image 5).
Image 5: Step 3, favouriting a dashboard
4. Once you have favourited a dashboard, you can easily find it by navigating to the left navigation pane, select the Dashboards icon > Home. This will open the Dashboards Home. You can see both your favourite and your recent dashboards in this view (Image 6)
Image 6: Step 4, favourite and recent dashboards
Your Cinchy v5 deployment comes with some out-of-the-box dashboards premade for you. You are able to customize these to suit your specifications. The following are a few notable ones:
Purpose: This dashboard provides a general overview of your entire cluster including all of your environments and pods (Image 7).
Image 7: Kubernetes/Compute Resources/Cluster dashboard
Metrics:
The following are some example metrics that you could expect to see from this dashboard:
- CPU Usage
- CPU Quota
- Memory Usage
- Memory Requests
- Current Network Usage
- Bandwidth (Transmitted and Received)
- Average Container Bandwidth by Namespace
- Rate of Packets
- Rate of Packets Dropped
- Storage IO & Distribution
Purpose: This dashboard is useful for looking at environment specific details (Image 8). You can use the namespace drop down menu to select which environment you want to visualize (Image 9). This can be particularly helpful during load testing. You are also able to drill down to a specific workload by clicking on its name.
Image 8: The Kubernetes/Compute Resources/Namespace (Workloads) dashboard
Image 9: Use the namespace drop down menu to select which environment you want to view details for
Metrics:
The following are some example metrics that you could expect to see from this dashboard:
- CPU Usage
- CPU Quota
- Memory Usage
- Memory Quota
- Current Network Usage
- Bandwidth (Transmitted and Received)
- Average Container Bandwidth by Workload
- Rate of Packets
- Rate of Packets Dropped
Grafana allows you to set up push alerts against your dashboards and queries. Once you have created your dashboard, you can follow the steps below to set up your alert.
Grafana does not have the capability to run alerts against queries with template variables.
To send emails out from Grafana, you need to configure your SMTP. This would have been done in the automations script run during your initial Cinchy v5 deployment. If you did not input this information at that time, you must do so before setting up your email alerts.
Your notifications channel refers to who will be receiving your alert. To set one up:
- 1.Click on the Alert icon on the left navigation tab (Image 10), and locate "Notifications Channel"
Image 10: The Alert icon
2. Click the "Add a Channel" button
3. Add in the following parameters, including any optional checkboxes you wish to use (Image 11):
Name: The name of this channel
Type: You have several options here, but email is the most common
Addresses: Input all the email addresses you want to be notified of this alert, separated by a comma
Image 11: Setting up your Notification Channel
4. Click Test to send out a test email, if desired.
5. Save your Notification Channel
The following details how to set up alerts on your dashboards. You can also set up alerts upon creation of your dashboard from the same window.
- 1.Navigate to the dashboard and dashboard panel that you want to set up an alert for. In this example, we are setting up an alert for CPU usage on our cluster.
- 2.Click on the dashboard name > Edit
- 3.Click on the Alert tab (Image 12).
Image 12: The alert tab
4. Input the following parameters to set up your alert (Image 13):
- Alert Name: A title for your alert
- Alert Timing: Choose how often to evaluate and for how long. In this example it is evaluated every minute for five minutes.
- Conditions: Here you can set your threshold conditions for when an alert will be sent out. In this example, it is sent when the average of query A is above 75.
- Set what happens if there's no data, or an error in your data
- Add in your notification channel (i.e., who will be sent this notification)
- Add a message to accompany the alert.
- Click Apply > Save to finalize your alert.
Click on an image to enlarge it.
Image 13: Your alert parameters
Below are a few alerts we recommend setting up on your Grafana.
Set up this alert to notify you when the CPU Usage on your nodes exceeds a specified limit.
You can use the following example queries to set up a dashboard that will capture CPU Usage by Node (Image 14).
avg by (node_name) (100 - ((avg by (cpu,node_name) (irate(node_cpu_seconds_total{mode="idle"}[1m]))) * 100))
100 - ((avg by (cpu,node_name) (irate(node_cpu_seconds_total{mode="idle"}[1m]))) * 100)
Image 14: CPU Usage Dashboard Query
Set up your alert. This example uses a threshold limit of 75 (Image 15).
Image 15: CPU Threshold Alert
Set up this alert to notify you when the Memory Usage on your nodes exceeds a specified limit.
You can use the following example queries to set up a dashboard that will capture CPU Usage by Node (Image 16)
((node_memory_MemTotal_bytes-node_memory_MemAvailable_bytes) / (node_memory_MemTotal_bytes))*100
Image 16: Memory Usage Query
Set up your alert. This example uses a threshold limit of 85 (Image 17).
Image 17: Memory Usage Alert
Set up this alert to notify you when the Disk Usage on your nodes exceeds a specified limit.
You can use the following example queries to set up a dashboard that will capture Disk Usage by Node (Image 18)
(sum((node_filesystem_size_bytes))by(node_name) - sum((node_filesystem_free_bytes))by(node_name)) *100/(sum((node_filesystem_avail_bytes))by(node_name)+(sum((node_filesystem_size_bytes))by(node_name) - sum((node_filesystem_free_bytes))by(node_name)))
Image 18: Disk Usage Query
Set up your alert. This example uses a threshold limit of 80 (Image 17).
Image 17: Disk Usage Alert
Set up this alert to check the amount of iowait from the CPU. A high value usually indicates a slow/overloaded HDD or Network.
You can use the following example queries to set up a dashboard that will capture the CPU Iowait (Image 19).
(sum(irate(node_cpu_seconds_total{mode="iowait"}[1m]))by(node_name) * 100 / 4)
Image 18: Iowait Query
Set up your alert. This example uses a threshold limit of 60 (Image 19).
Image 19: Iowait Alert
This capability was added in Cinchy v5.4.
Your Grafana password can be updated in your deployment.json file (you may have renamed this during your original deployment).
- 1.Navigate to "cluster_component_config" > "grafana".
- 2.The default password is set to "prom-operator"; update this with your preferred new password, written in clear text.
- 3.Run the below command in the root directory of your devops.automations repo to update your configurations. If you have changed the name of your deployment.json file, make sure to update the command accordingly.
dotnet Cinchy.DevOps.Automations.dll "deployment.json"
4. Commit and push your changes.
5. If your environment is not set-up to automatically apply upon configuration,navigate to the ArgoCD portal and refresh your component(s). If that does not work, re-sync.
Last modified 1mo ago