Monitoring containers with Prometheus

Clique aqui para a versão em Português.

I believe that today, more than ever, there are several developers who are adept at using a service-oriented architecture. Some of these developers study ways to migrate monolithic architectures to this new structure and others study how is the best way to start a new project based on it. That’s why a few weeks ago I published a text about a possible “recipe” for microservices, a way to begin a service-oriented architecture.

In my opinion, it is a really an interesting way for developing projects. It can have tons of benefits, like:

• services can be reusable;
• services share a formal contract;
• low coupling;
• the services are self-employed;
• services are able to be discovered;
• facilitates error detection and correction thereof etc.

Your eyes are shining with excitement, right? I know!

However, my dear reader, do not forget about our lovely friend Edward Murphy. Yes, that law so dreaded is also present in our service-oriented architecture. Well, as he said, “if anything can go wrong, it will”. And then you wonder, how can something go wrong in such architecture? Look, believe in Edward, this guy has a point. If there is 1% of chance that something goes wrong, it’s better to be safe than sorry!

One way to prevent this avalanche of bad things is monitoring your services. To this end that I wrote this post :) I want to present the Prometheus!

Prometheus is an open-source software for monitoring and alerting. It was originally developed by SoundCloud (yes, the song company).
Basically, Prometheus depends on exporters that get information (eg on the host, docker containers etc.) and leave it exposed in an API. Then Prometheus offers you an interface where you can search, filter the data and even create alerts according to the criteria you want. These alerts can be sent via email, page, Slack etc.

If you want further information, you can take look at its documentation.

Let’s develop a hello world to understand it better?

Let’s begin!

In this tutorial we will use the Prometheus on a Docker container.
In order to centralize the information, we will start all containers in a docker-compose.yml.
Would be as follows:

I am using two exporters: Node-Exporter and CAdvisor.

I believe that most people want to get specific host metrics such as CPU, memory and RSS free disk space. For this end, we will use the Node-Exporter. CAdivisor will be used to monitor Docker and its containers. Some time ago Prometheus staff had a “container-exporter”, but they now recommend the use of CAdvisor.

There is a huge list of exporters > HERE.

Well, after that we need to create two files that are mentioned in our Prometheus container volume.
The first file is the prometheus.yml, which will be as follows:

We have some global variables:

• scrape_interval: how frequently to scrape targets by default.(default 1m)
• scrape_timeout: how long until a scrape request times out. (default 10s).
• evaluation_interval: how frequently to evaluate rules. (default 1m)
• external_labels: external systems (federation, remote storage, Alertmanager).
• rule_files: rule files specifies a list of files from which rules are read.

Finally we have the scrappe_configs where we will name our exporters and how Prometheus will access them. The cool thing is that we can have exporters of several different hosts by simply adding a target [‘IP-DO-HOST: port’].

Now, the only file that is missing is our alert.rules. I have created a simple one:

There are two sample alerts. The first is a well desired one, which is an alert that fires in case of a particular container gets stopped. In this case, if a container is absent for 5 seconds, an alert is issued.

The second one is a cool function that Prometheus has, the predict_linear. You can use it to make linear forecasts. This warning is a forecast that evaluates if disc will be filled in 4 hours.

Alerts are very flexible … You can create them according to your need.

Now we are ready to start our stack!

Execute docker-compose up on your project’s directory and see the magic happening on http://localhost:9090/ ;)

Initial screen

On this initial screen, you can run multiple commands to filter the information offered by your exporters. You can also see this information in different periods as for hours, minutes, seconds, until such date etc.

By clicking at Alerts, we will have a view of the status of the alerts we have created.

Alerts’ Screen

This is how it will appear when an alert is triggered … And if you configure, you can receive alerts that are FIRING via Slack, for example.

That’s all, folks!

In my opinion, it is worthy giving a chance to Prometheus and use it to monitor your stack. Among the other open-sources options I’ve read, it excelled a lot.

So, what are your considerations about Prometheus? ;)