Azure Kubernetes Service Monitoring with Grafana and Prometheus

Kubernetes is an open-source container orchestration system for automating software deployment, scaling, and management.

However, it’s distributed nature means monitoring everything that is happening within the cluster can be a challenge.

Prometheus and Grafana make our experience better.

Prometheus is a free software application used for event monitoring and alerting. It records metrics in a time series database built using an HTTP pull model, with flexible queries and real-time alerting.

Grafana is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources.

Here by we will set up a Kubernetes cluster using Azure Kubernetes Service (AKS) and deploy Prometheus and Grafana to gather monitoring data and visualize them.

Step 1 - Login to az account and create resource group

az login
az group create --name aks-prometheus --location centralindia

Step 2 —Create AKS Cluster and connect to it and verify 3 nodes are created as expected.

az aks create --resource-group aks-prometheus --name aks1 --node-count 3 --node-vm-size Standard_B2s --generate-ssh-keys
az aks get-credentials --resource-group aks-prometheus --name aks1
kubectl get nodes

Step 3— Install Prometheus and Grafana using helm. Verify required Pods are up and running.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace
kubectl --namespace monitoring get pods -l "release=prometheus"

Step 4 — Create a port forward to access the Prometheus query interface.

kubectl port-forward --namespace monitoring service/prometheus-kube-prometheus-prometheus 9090

Hit http://localhost:9090 and confirm Prometheus is Up and running.

Step 5— Create a port forward to access Grafana UI too.

kubectl port-forward --namespace monitoring service/prometheus-grafana 8080:80

Hit http://localhost:8080 and confirm Grafana UI is Up and running.

Step 6— Lets Create Azure Container Repository to save a sample app docker image.

Step 7 — Build a container image using below docker file.

Docker image contains a python web app and python test file as shown below.

FROM python:3.7-slim
RUN pip install flask
WORKDIR /app
COPY firstapp.py /app/firstapp.py
ENTRYPOINT ["python"]
CMD ["/app/firstapp.py"]

from flask import Flask
app = Flask('first-app')
@app.route('/')
def hello():
  return "Hello World! Lets sail together - First App.\n"
if __name__ == '__main__':
  app.run(host = '0.0.0.0', port = 8080)

import unittest
from firstapp import hello
class TestFirstApp(unittest.TestCase):
    def test_hello(self):
        self.assertEqual(hello(), "Hello World! Lets sail together - First App.\n")
if __name__ == '__main__':
  unittest.main()

docker build --tag first-app:v2 .

Step 8— Push the docker image to Azure Container Registry.

docker push shaaksacr.azurecr.io/first-app:v2

Step 9 — Lets deploy the build app using below file

kubectl apply -f firstapp.yaml
kubectl get pods

apiVersion: apps/v1
kind: Deployment
metadata:
  name: first-app
spec:
  replicas: 1
  selector:
    matchLabels:
      run: first-app
  template:
    metadata:
      labels:
        run: first-app
    spec:
      containers:
      - name: first-app
        image: shaaksacr.azurecr.io/first-app:v2
        ports:
            - name: http
              containerPort: 80

Step 10 —Then expose as a service and validate service on AKS.

apiVersion: v1
kind: Service
metadata:
  name: "first-app-service"
spec:
  selector:
    run: "first-app"
  ports:
    - protocol: "TCP"
      port: 80
      targetPort: 8080
  type: LoadBalancer

kubectl apply -f firstappservice.yaml
kubectl get svc

Step 11 — Validate Grafana is able to check the Pod and Namespace.

CPU Usage tracked in Grafana.

Memory Usage tracked in Grafana.

Network Usage tracked in Grafana.

That's it for now…..will come up with more post on Grafana and Prometheus soon…..keep learning and Lets Sail Together….!!