Going open-source in monitoring, part III: 10 most useful Grafana dashboards to monitor Kubernetes and services

Sergey Nuzhdin
Jul 21, 2017 · 4 min read

There are dozens of ready dashboards available on grafana.net/dashboards and Github. Many of them just work, but many do not. In most cases, you just need to fix template variables, but some require deeper involvement. For example dashboard for PostgreSQL monitoring. After import, it welcomes you with the error message “Datasource named ${DS_PROMETHEUS} was not found”. Setting the correct datasource name in dashboard settings does not help because it has an error in __inputs declaration. The easiest way to fix it is to edit JSON before import.

It takes time to find useful dashboards, make them work for you and then choose one. For example, if you’ll search for Kubernetes dashboards with datasource Prometheus you’ll get, among others, 5 results with the same name.

In this post, I want to cover some of the most useful dashboards available to help you monitor your Kubernetes cluster and services deployed on it.


This post is one of a series of posts about monitoring of infrastructure and services. Other posts in the series:

0. Intro

1.Deploying Prometheus and Grafana to Kubernetes (this article)

2. Creating the first dashboard in Grafana

3. 10 most useful Grafana dashboards to monitor Kubernetes and services

4. Configuring alerts in Prometheus and Grafana

5. Collecting errors from production using Sentry

6. Making sense of logs with ELK stack

7. Replacing commercial APM monitoring

8. SLA, SLO, SLI and other useful abstractions


Cluster view

This dashboard usually the first one you search for after deploying Kubernetes + Prometheus. Among the 5 shown in the screenshot above only 2 is really good. Actually, it is the same dashboard saved by two people.

Node view

This one is great when you need to understand what is wrong with the particular node. It has much more stats compared to the cluster overview.

Deployment metrics monitoring

Next level of monitoring after the node — deployment. This dashboard will show you everything about selected deployment.

Pod metrics monitoring

Pod level monitoring. Shows mostly the same stats as the deployment dashboard but on Pod level. Pod info row shows some random values in Pod container and Pod IP Address places, but other values seem fine.

Application metrics monitoring

This one is good as an example of how to monitor your application deployed to Kubernetes. It has metrics like request rate, error rate and response times for your application alongside with different resource usage stats.

PostgreSQL

After you fix the broken variable declaration — it just works :) I saved fixed one here

ElasticSearch

The best one so far is 266. It supports most of the metrics needed and required least amount of tweaks to make it work. There are several other options — 2322, 2347, 718. But they either lack graphs or did not work at all for me.

Redis

Memcached

Etcd

Monitoring of Etcd is covered on CoreOS website here. Dashboard suggested in this manual has hardcoded datasource, so you have to edit it before import. So I made datasource configurable and saved it in the repo here. By default, Etcd is not monitored by Prometheus at all. So, you need to tell it where your Etcd lives. If you have self-hosted Etcd annotations to the Etcd service will work, otherwise, you need to update prometheus config with the location on your Etcd nodes.

- job_name: etcd
  static_configs:
  - targets:   ['10.240.0.32:2379','10.240.0.33:2379','10.240.0.34:2379']

Wrap Up

Sources for PostgreSQL and Etcd dashboards, with changes I did, are available on GitHub.

Stay tuned.

Like this article?


Originally published at blog.lwolf.org on July 21, 2017.

Sergey Nuzhdin

Written by

Software/DevOps engineer