Monitoring & Alerting with Prometheus + Grafana + Alertmanager
In this article, we are going to discuss Prometheus and Grafana and how we can set the monitoring for any Kubernetes clusters using Helm charts. We are also going to learn how we can connect Prometheus and Grafana together and set up a basic dashboard on Grafana to monitor resources on the Kubernetes cluster.
Introduction
Running complex applications on actual servers is complicated and things can go haywire for several reasons such as :
- Disk Full : No new data can be store
- Software Bug : Request Errors
- High Temperature : Hardware Failure
- Network Outage : Services cannot communicate
- Low Memory Utilization : Money wasted
Being able to monitor the performance of your system is essential. If system resources become too low it can cause a lot of problems. The ability to know what is happening can help determine whether system upgrades are needed, or if some services need to be moved to another machine.
When a performance issue arises, there are 4 main areas to consider: CPU, Memory, Disk I/O, and Network. The ability to determine where the bottleneck is can save you a lot of time.
There are different ways in which you could do the monitoring but in our case , we’ll be using Prometheus.
About Prometheus
- Prometheus is an Open-source systems monitoring and alerting toolkit.
- Prometheus collects and stores the metrics as time series data.
- It provides out-of-box monitoring capabilities for container orchestration platforms such as Kubernetes.
About Grafana
- Grafana is a multi-platform Open-source analytics and interactive visualization web application.
- It provides charts, graphs and alerts for the web when connected to supported data services.
- Grafana allows us to query, visualize, alert on and understand our metrics, no matter where they are stored. Some supported data sources in addition to Prometheus are AWS CloudWatch, AzureMonitor, PostgreSQL, Elasticsearch and many more.
- We can create our own dashboards or use the existing ones provided by Grafana. We can personalize the dashboards as per our requirements.
About Helm
- Helm is the package manager for Kubernetes.
- It allows us to streamline installation and management of Kubernetes applications.
- Helms use a packaging format called as Charts which is basically a collection of yaml manifest files.
some of the important terms :
- Prometheus Server : The main server that scrapes and stores the scraped metrics in a time series DB.
- Scrape : Prometheus server uses a pulling method to retrieve metrics.
- Target : The Prometheus servers clients that it retrieves info from.
- Exporter : Target libraries that convert and export existing metrics into Prometheus format.
- Alert Manager : Component responsible for handling alerts.
Prometheus Architecture
Prometheus, a system and service monitoring system, collects metrics from pre-defined targets via a pull model. Targets can be found by Service Discovery or manually configured to pull data from endpoints of your application like an API server, redis server or SQL server.
Installation through Helm Charts
Benefits of Helm Charts over manual Kubernetes deployment
- Without Helm for Kubernetes, we rely on Kubernetes YAML files to configure Kubernetes workloads.
- With Helm for Kubernetes, we can download existing Helm charts which have manifests files already available instead of having to write separate YAML files for each application manually.
- Helm charts contain templates for various Kubernetes resources that combine to form an application.
- A Helm chart can be templatized when deploying it on different Kubernetes clusters.
- On top of it, Helm provides repository support where Charts are available for popular tools like Prometheus and Grafana which can be directly used to set up pods.
Installing Prometheus and Grafana
prometheus
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/prometheus- It will install all the required components of the Prometheus system with a single command. Without Helm Charts we would have to write the manifest file ourselves.
“As you can see in the screenshot, the Prometheus-server and Prometheus-alertmanager pods are in the ‘containerCreating’ state. This is because we haven’t added the ConfigMap to them yet. Let’s quickly do this.
Firstly, let’s discuss what the ConfigMap for the Prometheus server and Alertmanager consists of:
- Prometheus.yml: This configuration file contains settings for the Prometheus server itself. It includes information about the targets to scrape (e.g., which services to monitor), the retention policies for time series data, and other server-specific settings.
- AlertingRules.yml: The AlertingRules configuration file defines the alerting rules that Prometheus should use to generate alerts based on the collected metrics. These rules specify conditions, thresholds, and actions to take when certain conditions are met, helping you monitor your services effectively.
- Alertmanager.yml: Including this in your Alertmanager ConfigMap allows you to tailor the alerting process to your organization's needs, ensuring that important alerts are properly routed and notifications are sent to the appropriate channels.
After creating the 3 files you can use below command to apply those as cm in cluster
kubectl create configmap prometheus-server --from-file=prometheus.yml --from-file=alerting_rules.yml
kubectl scale deploy prometheus-server --replicas=0
kubectl scale deploy prometheus-server --replicas=1
kubectl create configmap prometheus-alertmanager --from-file=alertmanager.yml
kubectl scale deployment prometheus-alertmanager --replicas=0
kubectl scale deployment prometheus-alertmanager --replicas=1 
Get the Prometheus server URL by running these commands in the shell:
export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace default port-forward $POD_NAME 9090
Get the Prometheus alertmanager URL by running these commands in the shell:
export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace default port-forward $POD_NAME 9093
Grafana
helm repo add bitnami-azure https://marketplace.azurecr.io/helm/v1/repo
helm repo update
helm install grafana bitnami-azure/grafana 
Get the Grafana URL by running these commands in the shell:
kubectl expose service grafana --type=NodePort --target-port=3000 --name=grafana-server
export POD_NAME=$(kubectl get pods | grep grafana | awk '{print $1}')
kubectl --namespace default port-forward $POD_NAME 3000
Set up credentials to log into Grafana using kubectl. The commands appeared in the installation’s output; here are the commands in use:
echo "User: admin"
echo "Password: $(kubectl get secret grafana-admin --namespace default -o jsonpath="{.data.GF_SECURITY_ADMIN_PASSWORD}" | base64 --decode)"
Log in with your new credentials, and you will see the Grafana dashboard.
Congratulations! You now have a working Grafana installation in your Minikube cluster with the ability to log in. The next step is to configure Grafana to work with Prometheus to gather data and show your steady state.
Configure Grafana with Prometheus
Now that you can log in to your Grafana instance, you need to set up the data collection and dashboard. Since this is an entirely web-based configuration, I will go through the setup using screenshots. Start by adding your Prometheus data collection. Click the gear icon on the left-hand side of the display to open the Configuration settings, then select Data Source.
Select Prometheus.
Because you configured your Prometheus instance to be exposed on port 80, use the service name prometheus-server and the server port 80.
By scrolling to the bottom of the screen and clicking Save and Test. You should see a green banner that says Data source is working
“Keep in mind”: copy the DATA_SOURCE_UID which is on the top of the url, which then used to import the monitoring dashboard
Import monitoring dashboards
To import a dashboard, first click the plus (+) sign on the left-hand side to create a dashboard, then click Import in the dropdown list
After importing your own customize Dashoboard.json file, click the load button to load your Dashboard
Congratulations! You have a set up basic data collection from Prometheus about your cluster.
