Docker daemon metrics in Prometheus

A couple of days ago, I sent a tweet regarding metrics exposition of the Docker daemon. As it got retweeted several times, I though this deserved additional details on how those metrics can be activated and how they can be sent to Prometheus, a great monitoring tool.

TL;DR

Metrics exposition is currently an experimental feature of the Docker daemon. Those metrics have a Prometheus like format and can thus be imported in this tool very easily.

Enable metrics in Docker for Mac

To activate this option in Docker for Mac, we just need to modify the configuration defined in the advanced tab of the daemon settings. The process is pretty much the same in Docker for Windows. Setting the metric-addr option to 0.0.0.0:9999 allows to have access to the /metrics endpoint on port 9999 on any interface of the host.

Note: for Linux users, the --metrics-addr option needs to be provided in the list of the Docker daemon’s startup options. This can be done in the /etc/docker/daemon.json file or in the service configuration file somewhere in the /etc/systemd/system/docker.service.d folder for systemd systems.

The configuration is persisted in the /etc/docker/daemon.json file on the host as the following output confirms.

Note: the first command is used to run a shell on the Docker host (the boot2docker VM that runs in xhyve). It basically runs a container in the pid namespace of the host.

Check instantaneous raw metrics

After a restart of the Docker daemon, the metrics are available from http://localhost:9999/metrics as shown below.

Note that those metrics are available on the macOS host through port 9999 but they are exposed by the underlying Docker daemon.

The metrics are updated when the page is reloaded, so in order to keep the history we will send them to Prometheus.

Setup Prometheus to ingest the metrics

Prometheus is a monitoring solution. It collects metrics in its embedded time series database, it allows to query and graph them, it also provides an alert mechanism and a lot of other cool things.

As we will run Prometheus in a container, we will target the metrics exposed by the Docker daemon through the Docker0 bridge which IP is 172.17.0.1 as the following output shows.

We will use the following configuration for Prometheus. It defines a single job named Docker4Mac that will collect the metrics exposed by the Docker daemon.

We then run Prometheus in a container as follow, providing the configuration file defined above.

$ docker container run -d -p 9090:9090 -v /tmp/prom.yml:/etc/prom.yml quay.io/prometheus/prometheus -config=/etc/prom.yml

Through Prometheus web interface (exposed on port 9090), we can see the /metrics endpoint is accessible through the bridge0 interface and we can also list the available metrics.

Selecting a metric, we can graph its evolution. Below is an example with the total number of HTTP requests.

Next steps

Once the metrics are in Prometheus, they can easily be used to build great dashboards with Grafana. The cleanest and easiest way would be to create a docker-compose.yml file with Prometheus and Grafana. A future post might illustrate this.

Summary

The exposition of the Docker daemon metrics is still an experimental feature in Docker 17.03, and then still subject to change but it’s definitely moving the monitoring of a daemon into the right direction.