Setup Prometheus and Grafana monitoring on Kubernetes cluster using Helm
In this article, we are going to discuss Prometheus and Grafana and how we can set the monitoring for any Kubernetes clusters using Helm charts. We are also going to learn how we can connect Prometheus and Grafana together and set up a basic dashboard on Grafana to monitor resources on the Kubernetes cluster.
The article contains the following sections:
- Introduction
- Pre-requisites
- Prometheus Architecture
- Benefits of helm deployment over manual deployment
- Installing Helm
- Installing Prometheus and Grafana
- Add prometheus data source into Grafana dashboard
- Conclusion
- References
Introduction
Prometheus and Grafana work together to render monitoring. Prometheus takes care of data fetching as a data source and feeds that data into Grafana, which can be used to visualize data with attractive dashboards.
About Prometheus
- Prometheus is an Open-source systems monitoring and alerting toolkit.
- Prometheus collects and stores the metrics as time series data.
- It provides out-of-box monitoring capabilities for container orchestration platforms such as Kubernetes.
About Grafana
- Grafana is a multi-platform Open-source analytics and interactive visualization web application.
- It provides charts, graphs and alerts for the web when connected to supported data services.
- Grafana allows us to query, visualize, alert on and understand our metrics, no matter where they are stored. Some supported data sources in addition to Prometheus are AWS CloudWatch, AzureMonitor, PostgreSQL, Elasticsearch and many more.
- We can create our own dashboards or use the existing ones provided by Grafana. We can personalize the dashboards as per our requirements.
About Helm
- Helm is the package manager for Kubernetes.
- It allows us to streamline installation and management of Kubernetes applications.
- Helms use a packaging format called as Charts which is basically a collection of yaml manifest files.
Prerequisites
- Users must have basic knowledge of Kubernetes.
- A Kubernetes cluster should be available where we can set up. Prometheus and Grafana. In this article, we have used minikube to create a new Kubernetes cluster. Minikube can be used to set up a local Kubernetes cluster on Windows or macOS or Linux.
Prometheus Architecture
This diagram illustrates the architecture of Prometheus and its ecosystem components:
Prometheus Server: The main server which stores and scrapes time series data.
TSDB(Time Series Database): Metrics are a critical aspect of any system to understand its health and operational state. Design of any system requires collection, storage and reporting of metrics to provide a pulse of the system. Data is stored over a series of time intervals and needs an efficient database to store and retrieve this data.OpenTSDB Time Series Database is one such time series database that can serve that need.
PromQL: Prometheus defines a rich query language in the form of PromQL to query data from the time series database.
Pushgateway: Available to support short lived jobs.
Exporters: They are used to promoting metrics data to the prometheus server.
Alertmanager: Used to send notifications to various communication channels like slack, email to notify users.
Benefits of Helm Charts over manual Kubernetes deployment
- Without Helm for Kubernetes, we rely on Kubernetes YAML files to configure Kubernetes workloads. These YAML files specify everything needed for deploying containers. Everything, from the way each Pod needs to be configured to how load balancing is done by the Kubernetes cluster, has to be mentioned in those YAML files. Thus, to set up a new Kubernetes workload, we need to create a YAML file for that workload. To do it manually means writing multiple YAML files — one for each workload we create. Workloads can be deployments, service, configMaps, secrets and configuration files.
- With Helm for Kubernetes, we can download existing Helm charts which have manifests files already available instead of having to write separate YAML files for each application manually. Below points highlights the merits of Helm over manual deployments.
- Helm charts contain templates for various Kubernetes resources that combine to form an application.
- A Helm chart can be templatized when deploying it on different Kubernetes clusters. Helm Charts can be created in such a way that environment or deployment-specific configurations can be extracted out to a separate file so that these values can be specified when the Helm Chart is deployed. For example, we need not have separate charts for deploying an application in development, staging, and production environments.
- On top of it, Helm provides repository support where Charts are available for popular tools like Prometheus and Grafana which can be directly used to set up pods.
Installing Helm
The Helm project provides binaries and scripts to fetch and install Helm. These are the official methods to get Helm releases. In addition to that, the Helm community also provides methods to install Helm through different package managers. Installation through those methods can be found on the official website of helm(https://helm.sh/docs/intro/install/). We have covered binary installation in the steps mentioned below.
- Download the desired version from Helm’s GitHub repository based on the OS. In this example, we have used the Windows operating system.
- Unzip the downloaded file (helm-v3.9.0-rc.1-windows-amd64.zip) and copy the helm binary in any folder and then add the folder path in the environment path variable. Once done, we are good to use Helm.
Installing Prometheus and Grafana
- Go to the following repo, https://artifacthub.io/ .
- Look for official charts for Prometheus and Grafana and the repositories into the helm repo using the following commands:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
- Next, we need to install Prometheus using the following command:
helm install prometheus prometheus-community/prometheus
- It will install all the required components of the Prometheus system with a single command. Without Helm Charts we would have to write the manifest file ourselves.
- We can expose the prometheus-server service to the internet using nodeport but the GUI provided by prometheus is not as good as the one provided by Grafana. We can use the following command for the same:
kubectl expose service prometheus-server — type=NodePort — target-port=9090 — name=prometheus-server-ext
minikube service prometheus-server-ext
- We can install Grafana using the following command:
helm install grafana grafana/grafana
- It will create the following components on the cluster for Grafana:
- We can expose Grafana service to the external world using the following commands:
kubectl expose service grafana — type=NodePort — target-port=3000 — name=grafana-ext
minikube service grafana-ext
- We can find the username and password required to log in into Grafana using the following commands. It will show the values in encrypted format, which we can decode using OpenSSL and base 64 formats.
kubectl get secret — namespace default grafana -o yaml
echo “password_value” | openssl base64 -d ; echo
echo “username_value” | openssl base64 -d ; echo
Add prometheus data source into Grafana dashboard
- Login into the Grafana dashboard using the admin user and password generated in the previous step.
- In the add data source section, provide the service URL generated for prometheus-server-ext after installation and save it.
- Next, we need to create a new dashboard. We can either create a dashboard from scratch or import the dashboards available for Prometheus on grafana.com. For the simplicity of this article, we are using a dashboard available on grafana.com numbered “3662”. Add the dashboard number and hit load.
- On the next screen add the data source for this dashboard which is Prometheus in our case and hit import.
- We will have a beautiful dashboard ready for us which we can modify according to our requirements.
Conclusion
We can see that through the use of helm charts, it is effortless to set up Prometheus and Grafana monitoring on the Kubernetes cluster as compared to the manual deployments done through Kubernetes yaml files.
References:
