Build A Monitoring Dashboard by Prometheus + Grafana

Prometheus + Grafana is a common combination of tools to build up a monitoring system. In this article, I’ll share how to get a dashboard by a tiny but complete example, which monitors how many requests are handled by a (dummy) API server.

Before or after your application is launched, you start wondering how many requests are handled in a period of time (QPS), the average response time, and meanwhile, how many resources (CPU, memory, I/O) are being used.

Photo by Stephen Dawson on Unsplash

Prometheus: Metrics Collector

Architrcture Overview from Prometheus Github

Prometheus, a system and service monitoring system, collects metrics from pre-defined targets via a pull model. Targets can be found by Service Discovery or manually configured to pull data from endpoints of your application like an API server, redis server or SQL server.

Grafana: Data Visualizer

Grafana supports lots of data sources such as CloudWatch, Stackdriver, Elasticsearch, and sure, Prometheus. After connecting Grafana with the data sources, you could create dashboard and panel manually, or import various dashboards shared on offical website.

A demo dashboard at play.grafana.org

To create a full example of a monitoring dashboard with Prometheus and Grafana, we need three components:

1. Metrics Exporter: usually server which handle requests.
2. Prometheus Server: periodly pull metrics from exporters/jobs
3. Grafana: query prometheus server to generate visual dashboards

Metrics Exporter: A Dummy API Server

First, prometheus server needs your metrics to be exported so that it can collect metrics and generate basic graphs for you.

ejhsu/go-prometheus-exporter

Here, I created a dummy API server in GO as a metrics exporter and implement a Counter for total number of API calls and a CounterVec for number of API calls on each endpoint. The CounterVec type (and all other Vec types) allows you to define/attach labels on metrics so that you could do flexible query later. See the offical document to find more metric types.

After running the dummy API server, the metrics can be access at http://localhost:9453/metrics and many pre-defined prometheus metrics are already exported like go_info:

# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.10.7"} 1.0

or http_requests_total

# HELP http_requests_total Total number of HTTP requests made.
# TYPE http_requests_total counter
http_requests_total{code="200",handler="prometheus",method="get"} 16.0

Also, the counter for API calls is exported:

# HELP myexporter_counter_metric This is a counter for number of total api calls
# TYPE myexporter_counter_metric counter
myexporter_counter_metric 0.0

Then we access the dummy API endpoints several times, the custom metrics become:

# HELP myexporter_counter_metric This is a counter for number of total api calls
# TYPE myexporter_counter_metric counter
myexporter_counter_metric 24.0
# HELP myexporter_counter_vec_metric This is a counter vec for number of all api calls
# TYPE myexporter_counter_vec_metric counter
myexporter_counter_vec_metric{endpoint="/api/apple"} 9.0
myexporter_counter_vec_metric{endpoint="/api/banana"} 15.0

So far, we have made it ready for Prometheus server to pull metrics.

Prometheus Server

Next, let’s use docker to start a prometheus server. The api-server job is added in advance to the following config file to scrape metrics from our dummy api server.

In Prometheus terms, an endpoint you can scrape is called an instance, usually corresponding to a single process. A collection of instances with the same purpose, a process replicated for scalability or reliability for example, is called a job.

Say, you downloaded the config to ~/workdir/config/prometheus.yml, then you’re ready to run this commend in ~/workdir/

$ docker run --name prometheus -d -p 127.0.0.1:9090:9090 --volume="$PWD/config":/etc/config prom/prometheus --config.file=/etc/config/prometheus.yml

That’s it! Now Prometheus server is accessible at http://localhost:9090/ and there are two targets in http://localhost:9090/targets. Further, before you use PromQL(Prometheus Query Language) to get some graphs, see some offical examples.

A graph shows how many requests are handled by our dummy API server

Grafana Server

Finally, after prometheus server can pull metrics from our API server, let’s generate a auto refreshable dashboard using Grafana.

Again, use docker to run a Grafana within a single commend.

$ docker run -d --name=grafana -p 3000:3000 grafana/grafana

Browse localhost:3000 and use admin/admin as username/password to access Grafana.

• Add a data source of Prometheus type, fill the URL field with host.docker.internal:9090 and save.

• Import the dashboard example to Grafana at http://localhost:3000/dashboard/import

Woohoo! The auto refreshable and beautiful dashboard is built! Try access your API server and see the dashboard is reflecting the number of requests automatically.

Summary

In this article, a tiny dashboard that monitors #requests is demonstrated with a exporter collecting simple metrics. It’s worth mentioning that there are many well-known exporters like go-grpc-prometheus, which can be integrated with grpc server easily and collect lots of useful metrics.

References

prometheus/prometheus

Grafana - The open platform for analytics and monitoring