Monitoring a Kubernetes Cluster using Prometheus and Grafana

5 min readSep 3, 2024

Kubernetes has emerged as the leading platform for managing cloud-native applications. It allows developers to deploy, scale, and manage applications seamlessly. However, as your Kubernetes cluster grows in complexity, ensuring everything operates smoothly becomes increasingly challenging. That’s where monitoring and observability come into play.

In this guide, we’ll explore how to set up Prometheus and Grafana to monitor your Kubernetes environment effectively using Helm. These tools will help you keep track of your system’s performance, spot issues before they escalate, and make data-driven decisions to maintain a healthy IT environment.

Why Monitoring is Crucial in Kubernetes

Kubernetes orchestrates multiple components, from containers to services, across a distributed system. While this makes it powerful, it also introduces potential points of failure. Without proper monitoring, it’s difficult to know if something is wrong until it’s too late. Monitoring allows you to track the health, performance, and availability of your services, ensuring you can react quickly to any problems.

Prometheus vs. Grafana: Understanding Their Roles

Prometheus: The Monitoring Engine

Prometheus is an open-source monitoring system specifically designed for reliability in dynamic cloud environments like Kubernetes. It works by collecting real-time metrics from your applications and infrastructure. Think of it as a watchful eye, constantly gathering data on how your systems are performing.

Prometheus is particularly well-suited for Kubernetes because it can discover services dynamically and scale as your infrastructure grows. It uses a powerful query language called PromQL, which allows you to create complex queries to analyze your data.

Grafana: The Visualization Tool

While Prometheus is great at collecting and storing data, Grafana is used for visualizing that data. Grafana is a flexible and powerful visualization tool that can create dashboards, graphs, and alerts based on the metrics collected by Prometheus.

Imagine trying to understand how your systems are performing by reading lines of raw data, it’s tedious and inefficient. Grafana turns that data into clear, interactive dashboards, making it easier to spot trends, anomalies, and insights at a glance.

Setting Up Prometheus and Grafana on Kubernetes Cluster using Helm

I set up a single-node Kubernetes cluster on bare metal for a demo using Minikube.

minikube start --vm-driver=none

Step 1: Install Prometheus

1. Create a Namespace for Monitoring: It’s a good practice to keep your monitoring tools in their own namespace.

kubectl create namespace monitoring

2. Deploy Prometheus Using Helm: Helm is a package manager for Kubernetes that simplifies the installation of applications.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/prometheus --namespace monitoring

You can view the pods by running the command below.

kubectl get pods -n monitoring

3. Access the Prometheus UI: Once Prometheus is running, you can access its web interface to start exploring your metrics.

kubectl --namespace monitoring port-forward deploy/prometheus-server 9090

Open your browser and go to http://localhost:9090.

Step 2: Install Grafana

1. Deploy Grafana Using Helm: Similarly, you can install Grafana with Helm.

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install grafana grafana/grafana --namespace monitoring

2. Access the Grafana UI: Grafana also provides a web-based interface.

kubectl --namespace monitoring port-forward deploy/grafana 3000

Navigate to http://localhost:3000 in your browser. The default login is admin/admin.

3. Add Prometheus as a Data Source:

Enter into the Grafana pod:

kubectl exec -it <grafana-pod-name> --namespace monitoring -- /bin/bash

Once you are inside the pod, run the following command to reset the admin password. After that you will be able to login.

grafana-cli admin reset-admin-password admin

Then inside the Grafana portal, click Add data source, choose Prometheus from the list, and set the URL to http://prometheus-server.monitoring.svc.cluster.local:9090.

Click Save & Test to ensure the connection works.

Step 3: Create a Dashboard in Grafana

1. Create a New Dashboard:

Click on the plus sign (+) in the sidebar and choose Dashboard.
Click Add new panel.

2. Query Your Data:

Select Prometheus as the data source.

You can add the prebuild dashboard in JSON files you can get it from grafana.com/dashboard website.

Customize the graph settings to fit your needs.

3. Save Your Dashboard: Once you’re satisfied with your dashboard, save it so you can monitor your systems in real-time.

Conclusion

Prometheus and Grafana are essential tools for monitoring and observability in a Kubernetes environment. Prometheus excels at collecting and storing metrics, while Grafana provides the powerful visualizations that make it easier to understand what’s happening in your systems.

By following the steps outlined in this guide, you can set up a robust monitoring system that helps you ensure your Kubernetes clusters are running smoothly. Remember, proactive monitoring not only helps prevent outages but also enables you to optimize performance and plan for future growth.

If you have any questions, feel free to leave a comment below.