Kubernetes Monitoring using InfluxDB & Grafana in Google Cloud Platform

In this post i would like to share with you all on the proof of concept we did for kubernetes monitoring. We wanted to monitor our kubernetes environment for various custom metric parameters like cpu,memory,disk I/O etc at a node & pod level in a beautiful dashboard. We opted for the open source analytics platform Grafana which has inbuilt plugin to various data sources like influxDB , cloudwatch, stackdriver, datadog, appdynamics etc. Even though you can still do the kubernetes monitoring using stackdriver monitoring, grafana boasts a better UI visualization capability.

So the idea is simple , Heapster aggregates the metrics from the kubernetes nodes and pods , which will sink into InfluxDB’s database , which can be later used by grafana to visualize the metrics. Lets see how this was done in GCP.

Helm package manager was used to deploy the InfluxDB & grafana charts into this kubernetes environment. We had taken a sample app from kubernetes-engine-samples from google cloud platform github link, The link is provided below

git clone https://github.com/GoogleCloudPlatform/kubernetes-engine-samples
cd kubernetes-engine-samples/hello-app
export PROJECT_ID="$(gcloud config get-value project -q)"
docker build -t gcr.io/${PROJECT_ID}/hello-app:v1 .
docker images
gcloud auth configure-docker
docker push gcr.io/${PROJECT_ID}/hello-app:v1

Created a kubernetes cluster with 3 nodes and deployed the sampleapp with the hello-app container image, we had 3 pod replicas for testing purpose.

Exposed the sampleapp with a loadbalancer service,

Tested the connectivity

Helm

Helm is a package manager used for deploying applications in Kubernetes , all the related services, rolebindings, serviceaccounts , configmaps, secrets etc will be taken care when we deploy through helm, More information is available in the below link

wget https://storage.googleapis.com/kubernetes-helm/helm-v2.9.1-linux-amd64.tar.gz
tar zxfv helm-v2.9.1-linux-amd64.tar.gz
gcloud container clusters get-credentials sample --zone us-central1-a --project projectname
kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud config get-value account)
kubectl create serviceaccount tiller --namespace kube-system
kubectl create clusterrolebinding tiller-admin-binding --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
./helm init --service-account=tiller
./helm update
./helm version

If you want to initialize the helm with RBAC , you need to provide certificate and key details,

helm init \
— override ‘spec.template.spec.containers[0].command’=’{/tiller, — storage=secret}’ \
— tiller-tls \
— tiller-tls-verify \
— tiller-tls-cert=cert.pem \
— tiller-tls-key=key.pem \
— tls-ca-cert=ca.pem \
— service-account=accountname
Persistent Volumes

Before we went ahead with the installation of the influxdb and grafana, we created the GCE disks for persistence,

gcloud compute disks create influxdbdisk grafanadisk --zone us-cenrtral1-a --size=10gi

The yaml file for persistent volume & persistent volume claim creation is given in the below link,

kubectl create -f pv-pvc.yaml
kubectl get pv
kubectl get pvc

Under Kubernetes storage , you can view these new volumes.

InfluxDB

Next we proceeded with the installation of InfluxDB, using helm package manager. We had edited the default values.yaml file to given the persistent volume claim

persistence:
enabled: true
useExisting: true
name: influxdbclaim
storageClass: ""
accessMode: ReadWriteMany
size: 2Gi
./helm install --name sampledb -f influxdbvalues.yaml stable/influxdb

All the related service , configmap and workload will be deployed automatically.

InfluxDB can be accessed via port 8086 on the following DNS name from within your cluster:

- http://sampledb-influxdb.default:8086

You can easily connect to the remote instance with your local influx cli. To forward the API port to localhost:8086 run the following:

kubectl port-forward — namespace default $(kubectl get pods — namespace default -l app=sampledb-influxdb -o jsonpath=’{ .items[0].metadata.name }’) 8086:8086

You can also connect to the influx cli from inside the container. To open a shell session in the InfluxDB pod run the following:

kubectl exec -i -t — namespace default $(kubectl get pods — namespace default -l app=sampledb-influxdb -o jsonpath=’{.items[0].metadata.name}’) /bin/sh

To tail the logs for the InfluxDB pod run the following:

kubectl logs -f — namespace default $(kubectl get pods — namespace default -l app=sampledb-influxdb -o jsonpath=’{ .items[0].metadata.name }’)
Grafana

Next we installed grafana using helm package manager, run the below command to install grafana, the default values.yaml was edited to enable the persistence volume claim and to set the userid and password.

persistence:
enabled: true
storageClassName: ""
accessModes:
- ReadWriteMany
size: 2Gi
annotations: {}
subPath: ""
existingClaim: grafanaclaim
adminUser: admin
adminPassword: password

In the newer grafana versions >5.1 , the Docker container are owned by id/gid 472 (the grafana user/group), the container crashes when persistence is enabled , so we have modified the security context , to run the process as a root user.

securityContext:
runAsUser: 0

grafanavalues.yaml file is available in the below

./helm install --name samplegrafana -f grafanavalues.yaml stable/grafana

You can get your ‘admin’ user password by running:

kubectl get secret — namespace default samplegrafana -o jsonpath=”{.data.admin-password}” | base64 — decode ; echo

The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:

samplegrafana.default.svc.cluster.local

you can get the Grafana URL to visit by running these commands in the same shell:

export POD_NAME=$(kubectl get pods — namespace default -l “app=grafana,release=samplegrafana” -o jsonpath=”{.items[0].metadata.name}”)
kubectl — namespace default port-forward $POD_NAME 3000
Heapster

Both the grafana and influxdb deployments should be up and running before making the changes to the heapster, the heapster workload will be running under the kube-system namespace, export the heapster deployment’s yaml file and edit it to sink the details to the influxdb database.

kubectl get deployments -n kube-system
kubectl get deployment heapster-v1.5.3 -n kube-system -o yaml > heapster.yaml

Right after the spec , make the below changes

spec:
containers:
- command:
- /heapster
- --source=kubernetes.summary_api:""
- --sink=influxdb:http://sampledb-influxdb.default:8086
name: heapster

apply the changes to the heapster deployment

kubectl apply -f heapster.yaml

Verified the data in influxDB by connecting to the container, a new database called k8s will be created in influxdb

Used port-forwarding to access the grafana console.

Logged in with the admin credentials, ( you might have to use a strong password )

Add the influxDB data source,

configure the data source and test it

Add new Dashboard and add a graph, under the metrics section select the query on metrics name , container name & pod name. Give the title name under the general section and finally save the dashboard.

So now to check , whether the data is persistent on the grafana and influxdb , delete the pods , you can see that the data on the influxdb and grafana are still available on the new pod that was created . This concludes the setup , Thanks for reading through this post, Hope you find this useful.

Thanks