Get Real-Time Docker Performance Metrics

How to monitor the health of your containers using free, open-source tools

Nassos Michas
Feb 26 · 7 min read
Photo by Simon Connellan on Unsplash.

Whether you have a handful of containers running in your development machine or hundreds of containers running in production, knowing what happens behind the scenes can be invaluable. Do you see excessive disk activity and you don’t know which container is causing it? What about high memory or network usage that may negatively affect the remaining containers?

In this piece, we’ll explore three of the most well-known, widely used Docker performance monitoring tools. I deliberately have chosen only free, open-source tools. There are many more commercial ones — some of them very good and popular — providing great analysis capabilities and nice user interfaces.


Docker Stats

The default output of docker stats (image by author).

The default columns contain information about the CPU, memory, network, disk, and the number of threads created by the container. You can quickly filter the displayed containers by passing their id or name and select the columns being displayed with the --format option. For example:

docker stats — format “{{.Name}}\t{{.CPUPerc}}” nifi-dev mysql-dev
Filtering docker stats (image by author).

You may call docker stats the poor man’s metrics. However, if nothing else is available, it provides a wealth of details to get you started investigating a problem.


cAdvisor

Running cAdvisor Docker Image.

Once running, cAdvisor can be accessed either via its REST-based API or via a graphical user interface available at http://localhost:8080 (as per the
-p option you have specfied when instantiating the container). Here’s a glimpse of the user interface provided by cAdvisor:

Monitoring a Docker container with cAdvisor (image by author).

cAdvisor’s graphical user interface is pleasant and rich with information presented in a variety of formats, such as numbers, gauges, and charts.

As mentioned above, cAdvisor is not only for Docker monitoring. Therefore, when you open its main page, you’re greeted with a hierarchy of resources or subcontainers — courtesy of Google’s lmctfy, one of them being your Docker Engine. To dig into the statistics of your Docker containers, choose the /docker link or the Docker Containers link on top of the page. You can then choose the exact container you’re interested in to obtain additional information and metrics.


Prometheus

This is where Prometheus comes in:

Prometheus architecture (image by Prometheus Authors).

Here’s an official overview:

Prometheus is an open-source monitoring and alerting toolkit, built back in 2012. Prometheus works well for recording any purely numeric time series. It fits both machine-centric monitoring as well as monitoring of highly dynamic service-oriented architectures. In a world of microservices, its support for multi-dimensional data collection and querying is a particular strength.

Prometheus can collect metrics in a variety of ways, including data push gateways, exporters, or via your own application using one of the supported client libraries. The list of the supported extractor is admittedly impressive.

Exporting your Docker Engine’s real-time metrics to Prometheus is straightforward, as Docker daemon itself is capable of exporting metrics in the exact format Prometheus expects them in. Alternatively, you can take advantage of the cAdvisor container we deployed before, having Prometheus fetch the metrics from cAdvisor instead. Each approach produces a unique set of metrics (with a slight overlap), so I suggest you experiment with both to see which one suits your needs.

Let’s explore both options next.

Exporting metrics via Docker daemon

{
"metrics-addr" : "0.0.0.0:9323",
"experimental" : true
}

Note: Be aware that according to when you read this piece, you may be able to skip “experimental”: true. Also, take into account that 0.0.0.0 denotes that the metrics will be accessible by any host. You may also try with 127.0.0.1 if you want to allow only your own host.

Exporting metrics via cAdvisor

Running and connecting Prometheus

Connecting Prometheus to Docker metrics (image by author).

Before Prometheus is started, let’s take a quick look at the figure above. As you see, Prometheus is pulling data from the Docker Engine. It’s not the Docker Engine pushing the metrics. Therefore, Prometheus needs to be configured appropriately so that it knows where to fetch the data from. Such configuration takes place via the prometheus.yml file that we define next:

prometheus.yml configuration file

Note: When Prometheus is configured with the above YAML configuration, it will try to contact the host running the Docker Engine to communicate with the Docker daemon serving the metrics. The hostname that you should use depends on your operating system. On the sample file used here, I’m using a hostname working in Mac, which is docker.for.mac.host.internal. If you’re on Windows or Linux, you can replace it with the IP address of your host.

So, time to run Prometheus in Docker. For this quick demonstration, we will use the same underlying Docker Engine exporting metrics via cAdvisor and Docker daemon in parallel. Create a prometheus.yml file using the above content, modified for your operating system, and execute:

Creating a Prometheus Docker instance.

After a few seconds, you should be able to access the Prometheus web interface at http://localhost:9090. The default page expects you to define the query to execute so that you can see your live metrics. Here’s an example:

Viewing the metrics in Prometheus (image by author).

Prometheus’s web interface gives quite a few options for search customization, such as:

  1. Define your query to filter the data by. Prometheus supports a rich query language, allowing you to see exactly the data you want.
  2. All collected metrics categories are available to be quickly selected from this dropdown.
  3. You can choose to view the results either on a graph or in a tabular format.
  4. Define the time period for which metrics are displayed.

You now have Prometheus configured and fetching data from two different data sources (although from the same Docker Engine). By expanding the configuration file used, you can define all your additional data sources and monitor your complete fleet of Docker Engines. If you want to create more advanced graphs than what Prometheus provides out of the box, Grafana integrates nicely with Prometheus and provides a plethora of additional visualization options.


Conclusion

To get a quick glimpse into your Docker Engine, the built-in docker stats serves as a starting point. cAdvisor can be easily integrated into your environment, providing a web interface with several different charts and gauges. Finally, if you want to monitor multiple Docker Engines, Prometheus provides a powerful infrastructure to collect and visualize metrics for a large number of data sources.

Thank you for reading this piece. I hope to see you in the next one.

Better Programming

Advice for programmers.

Thanks to Zack Shapiro

Nassos Michas

Written by

Software engineer | Cert. Scrum master | Cert. Professional for Requirements Engineering | CTO at European Dynamics

Better Programming

Advice for programmers.

More From Medium

More from Better Programming

More from Better Programming

More from Better Programming

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade