A Guide to using Prometheus and Grafana for logging API metrics in Django

Alex
Alex
Jun 29 · 5 min read

A quick guide on how to get up and running locally using Docker-compose.

Prometheus, deity of fire, wonders if it was all worth it
Prometheus, deity of fire, wonders if it was all worth it
Credit: public domain

If you’ve made it here you probably already have an app that you want to add some metrics to and have displayed. This is a good idea, mainly because:

  • Having to kubectl into the pod all the time to ascertain how your apps performing can be a pain.
  • It’ll impress the project manager with some fancy graphs.
  • You can quickly see what's going on in your app when sysadmin won’t give you access to the DB (probably for the best).

Note: Prometheus is for logging metrics, or in other words, numbers. The four types of metrics available; Counter, Gauge, Histogram and Summary only take numbers as parameters, try putting in anything else and it’ll throw an error. This may seem obvious but if you want to get the most popular searched item for example, then I’d suggest using the ELK stack instead.

The first issue I had was understanding how all the different tools and frameworks fit together. I had wrongly assumed that using the Prometheus python client library along with pulling the latest Grafana Docker image would be enough to see some sweet pie charts. I was wrong. The missing piece of the puzzle was of course, getting the Prometheus server running in the middle. Our end goal is to have an architecture like this:

Image for post
Image for post
  • We create the metrics in our app.
  • Prometheus polls our app for metrics at an endpoint and stores them.
  • Grafana polls Prometheus for metrics and displays them.

Configuring the Django App

In this example we’ll be using a python Django app as the source of the metrics. These apps can be pretty big, so I won’t include the whole source but just the extra changes you need to make.

Rather confusingly there are two client libraries we can use:

pip install django-prometheus

and

pip install prometheus_client

The first is a Django add-on that makes it a lot easier to gather metrics about the model used in your app, like CRUD operations used on the model, how many times an endpoint has been hit and the time the requests took. It's great at capturing the basic stuff you’ll want to do meaning you don’t have to keep count yourself. However, it won’t capture any business logic you may be doing inside your API, like how much S3 resource you may need to allocate for example, for this we need the second client library. Be sure to add both to your requirements.txt

A lot of this is covered in the official docs, but in summary:

Set up the django_prometheus Django add-on:

  • In settings.py: add django_prometheus to the installed apps and to the top and bottom on middleware:
  • Add the extension to any classes you have in models.py
  • Add a path to the urls.py so Prometheus can poll the app, the framework automatically sets it to /metrics, with no trailing slash.

Set up prometheus_client for custom metrics:

  • Add a custom metric for any business logic you may want to log. For this I’ve classed out the creation of metric types.

so I can reuse them in an endpoint where I want to log something

You can test this by pinging a GET on your /metrics endpoint. You should see something like this:

Configuring the Prometheus server

This is done via a .yml file that you’ll need to create. Our Docker-compose file will load it into the Prometheus server when it starts.

The targets (line 27) and alias (line 29) are references to our Django app which will be running under the Docker network. Line 24 tells Prometheus what to poll to get the metrics.

Configuring Docker-Compose

Once your app has been configured and the .yml file created, we can spin up our Django app, Prometheus and Grafana containers from the Docker-compose file.

Start the services by running:

docker-compose up

If everything started ok then you should be able to navigate to:

http://localhost:9090 for Prometheus

and http://localhost:3000 for Grafana

Note: Both Prometheus and Grafana will only display custom metrics once they’ve been hit. So searching for my_counter in the Prometheus server won’t return anything unless something has happened in the app to generate it.

A quick note on Error handling:

Make sure you get the correct indentation and be careful of tabs in the .yml files, otherwise you may see:

err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: line 19: did not find expected key"

If you see the following error message, it means Prometheus can’t find the .yml file and the path is wrong in the docker-compose file:

unknown: Are you trying to mount a directory onto a file (or vice-versa)?

Configuring Grafana

At this point we’ve got a running app with the Prometheus server configured to poll our metrics endpoint. The next step is to get Grafana to poll the Prometheus server.

There should be an option to add a Prometheus data source.

Image for post
Image for post

Add the following configuration to your Grafana instance:

Image for post
Image for post

Hitting save should test the connection and return successful.

And that's it… all that's left is to play around in Grafana to get the metrics looking how you want them.

Conclusion

So there, hopefully now you’ll be able to create a set of bedazzling charts fit to adorn any project management slide deck. Let me know how you get on.

The Startup

Medium's largest active publication, followed by +732K people. Follow to join our community.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store