Monitoring Kubernetes Clusters with Grafana

Cluster Dashboard by Grafana App for Kubernetes

With the support of Grafana App for Kubernetes, which integrates the data collected from Kubelet, Kube-State Metrics, and Node Exporter with data available via the Kubernetes API, it’s advisable to use the app to build a monitoring system on Kubernetes Cluster with Grafana.

In this article, I’ll use Google Kubernetes Engine (GKE) as the cluster example, combining with a local-running Grafana server, to give a step-by-step instruction to setup the monitoring system.

Let’s start from scratch with a new cluster. Go to the add cluster page: https://console.cloud.google.com/kubernetes/add?template=your-first-cluster.

At the cluster detail page you can find show credentials to get username/password and CA certificate. We’ll use them to connect to the cluster in Grafana.

After the cluster is created, fetching credentials for kubectl or use cloud shell to connect to your cluster. Then follow the steps to create each components.

• create the monitoring namespace:

$ kubectl apply -f namespace.yaml

• create the service account and cluster role with the permission of getting cluster info for prometheus.

$ kubectl apply -f prometheus-rbac.yaml

Note: If you were failed to create cluster role, create a cluster role binding to yourself first. (Email is case-sensitive)

$ kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=${YOUR_EMAIL_ADDRESS}

Role-Based Access Control | Kubernetes Engine | Google Cloud

• create the config for prometheus to collect data from Kubernetes

$ kubectl apply -f prometheus-config.yaml

• create the deployment of prometheus

$ kubectl apply -f prometheus-deploy.yaml

• create the prometheus service

$ kubectl -f apply prometheus-svc.yaml

• create the grafana deployment

$ kubectl apply -f grafana.yaml

• expose grafana deployment with kubectl expose

$ kubectl expose deployment grafana --type=LoadBalancer --namespace=monitoring

kubectl expose will create the service of LoadBalancer type for you, and if you run on GKE, Amazon EKS or other cloud providers, they will assign a public IP for this service. Get the IP via $ kubectl get svc -n monitoring

NAME         TYPE           CLUSTER-IP    EXTERNAL-IP  
grafana   LoadBalancer   10.7.245.47      {IP}

Now you can access Grafana via http://{IP}:3000/

• create node-exporter daemon set to export node information

kubectl apply -f node-exporter.yaml

• create kube-state-metrics deployment to collect metrics about the cluster

kubectl apply -f state-metrics-deploy.yaml

• create the service account and cluster role with the permission of getting cluster info for state-metrics.

kubectl apply -f state-metrics-rbac.yaml

Now, configure your cluster setting on Grafana.

• Enable the app at http://{IP}:3000/plugins/grafana-kubernetes-app/edit
• Add a data source of prometheus type: http://{IP}:3000/datasources/new

• Add a cluster at http://{IP}:3000/plugins/grafana-kubernetes-app/page/cluster-config with username/password and CA certificate.

Yeah! Now you finally get the three built-in dashboards working. :-)

Cluster Dashboard

Nodes Dashboard

Container Dashboard