A quick guide on how to get up and running locally using Docker-compose.
If you’ve made it here you probably already have an app that you want to add some metrics to and have displayed. This is a good idea, mainly because:
- Having to kubectl into the pod all the time to ascertain how your apps performing can be a pain.
- It’ll impress the project manager with some fancy graphs.
- You can quickly see what's going on in your app when sysadmin won’t give you access to the DB (probably for the best).
Note: Prometheus is for logging metrics, or in other words, numbers. The four types of metrics available; Counter, Gauge, Histogram and Summary only take numbers as parameters, try putting in anything else and it’ll throw an error. This may seem obvious but if you want to get the most popular searched item for example, then I’d suggest using the ELK stack instead.
The first issue I had was understanding how all the different tools and frameworks fit together. I had wrongly assumed that using the Prometheus python client library along with pulling the latest Grafana Docker image would be enough to see some sweet pie charts. I was wrong. The missing piece of the puzzle was of course, getting the Prometheus server running in the middle. Our end goal is to have an architecture like this:
- We create the metrics in our app.
- Prometheus polls our app for metrics at an endpoint and stores them.
- Grafana polls Prometheus for metrics and displays them.
Configuring the Django App
In this example we’ll be using a python Django app as the source of the metrics. These apps can be pretty big, so I won’t include the whole source but just the extra changes you need to make.
Rather confusingly there are two client libraries we can use:
The first is a Django add-on that makes it a lot easier to gather metrics about the model used in your app, like CRUD operations used on the model, how many times an endpoint has been hit and the time the requests took. It's great at capturing the basic stuff you’ll want to do meaning you don’t have to keep count yourself. However, it won’t capture any business logic you may be doing inside your API, like how much S3 resource you may need to allocate for example, for this we need the second client library. Be sure to add both to your requirements.txt
A lot of this is covered in the official docs, but in summary:
Set up the django_prometheus Django add-on:
- In settings.py: add django_prometheus to the installed apps and to the top and bottom on middleware:
- Add the extension to any classes you have in models.py
- Add a path to the urls.py so Prometheus can poll the app, the framework automatically sets it to /metrics, with no trailing slash.
Set up prometheus_client for custom metrics:
- Add a custom metric for any business logic you may want to log. For this I’ve classed out the creation of metric types.
so I can reuse them in an endpoint where I want to log something
You can test this by pinging a GET on your /metrics endpoint. You should see something like this:
Configuring the Prometheus server
This is done via a .yml file that you’ll need to create. Our Docker-compose file will load it into the Prometheus server when it starts.
The targets (line 27) and alias (line 29) are references to our Django app which will be running under the Docker network. Line 24 tells Prometheus what to poll to get the metrics.
Once your app has been configured and the .yml file created, we can spin up our Django app, Prometheus and Grafana containers from the Docker-compose file.
Start the services by running:
If everything started ok then you should be able to navigate to:
http://localhost:9090 for Prometheus
and http://localhost:3000 for Grafana
Note: Both Prometheus and Grafana will only display custom metrics once they’ve been hit. So searching for my_counter in the Prometheus server won’t return anything unless something has happened in the app to generate it.
A quick note on Error handling:
Make sure you get the correct indentation and be careful of tabs in the .yml files, otherwise you may see:
err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: line 19: did not find expected key"
If you see the following error message, it means Prometheus can’t find the .yml file and the path is wrong in the docker-compose file:
unknown: Are you trying to mount a directory onto a file (or vice-versa)?
At this point we’ve got a running app with the Prometheus server configured to poll our metrics endpoint. The next step is to get Grafana to poll the Prometheus server.
There should be an option to add a Prometheus data source.
Add the following configuration to your Grafana instance:
Hitting save should test the connection and return successful.
And that's it… all that's left is to play around in Grafana to get the metrics looking how you want them.
So there, hopefully now you’ll be able to create a set of bedazzling charts fit to adorn any project management slide deck. Let me know how you get on.