Centralised logging for Django, Gunicorn and Celery using ELK Stack

Sanchit Sokhey
Apr 22, 2017 · 3 min read

Last week I was trying to find a solution to implement centralised logging for the project I’m working on and I came across ELK (Elasticsearch, Logstash and Kibana) stack among the most popular open source solutions. Despite being one of the most popular solution, I couldn’t find a single go-to guide for full fledged integration with Django, Gunicorn and Celery.

Edit: Works only for Python 2.7.x

So this is how I did it. Hope it helps.

Libraries Used

Steps

Setup ELK Stack (version ≥ 5.3)

Ubuntu:
- Install ElasticSearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html
- Install Kibana:
https://www.elastic.co/guide/en/kibana/current/setup.html
- Install Logstash: https://www.elastic.co/guide/en/logstash/current/installing-logstash.html

Mac (using Homebrew):
- Install ElasticSearch: brew install elasticsearch
- Install Kibana:
brew install kibana
- Install Logstash
brew install logstash

Once ELK Stack is up and running you can point your browser at localhost:5601 (default kibana port) to see something like this:

Kibana’s default view

Sending Django Logs to Logstash

These logs are important, along with gunicorn logs, for logging errors as they contain the WSGI request and traceback info.
- Install the
Python Logstash (python logging handler for logstash) using: pip install python-logstash
- Add the following in django’s settings:

Note: I’m using Level “ERROR” because I will be using Gunicorn to transfer the other request logs. (You can use any level depending on your requirements)

At this point, django will start sending logs to localhost:5959 but logstash is still not configured to read the input and transfer the logs to elasticsearch.
So we need to add the following configuration file for Logstash:

Note: Make sure the logstash port (5959) and elastic hosts (localhost:9200) are according to your settings.

- Now run logstash using ./logstash -f path/to/logstash.conf
- Open kibana dashboard
locahost:5601 and create the index logstash-*
- Start Django server and browse a page that raises some error (because my logging settings are only for level-ERROR)
Voila! All the django logs can be seen on kibana dashboard.

Sending Gunicorn Logs to Logstash

- Install Python JSON Logger using pip install python-json-logger
- Create/modify the gunicorn config file to the following:

Note: Modify line 34 to the logstash’s hostname and port number

- Run gunicorn server with --log-config path/to/gunicorn.conf argument.
Eg:
gunicorn project.wsgi:application --bind 0.0.0.0:8000 --log-config path/to/gunicorn.conf
- Enjoy your gunicorn logs on Kibana dashboard. You can also create separate dashboards for gunicorn and django logs.

Sending Celery Logs to Logstash

The easiest way to implement this is to add a TCPLogstashHandler as a handler to the celery’s default logger.
Add the following lines to your django’s signal_handlers.py:

Note: Modify line 3 to the logstash’s hostname and port number

That’s it. Now it’s time for the DevOps team to get to work.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store