Centralised logging for Django, Gunicorn and Celery using ELK Stack

Last week I was trying to find a solution to implement centralised logging for the project I’m working on and I came across ELK (Elasticsearch, Logstash and Kibana) stack among the most popular open source solutions. Despite being one of the most popular solution, I couldn’t find a single go-to guide for full fledged integration with Django, Gunicorn and Celery.

Edit: Works only for Python 2.7.x

So this is how I did it. Hope it helps.

Libraries Used

Steps

Ubuntu:
- Install ElasticSearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html
- Install Kibana:
https://www.elastic.co/guide/en/kibana/current/setup.html
- Install Logstash: https://www.elastic.co/guide/en/logstash/current/installing-logstash.html

Mac (using Homebrew):
- Install ElasticSearch: brew install elasticsearch
- Install Kibana:
brew install kibana
- Install Logstash
brew install logstash

Once ELK Stack is up and running you can point your browser at localhost:5601 (default kibana port) to see something like this:

Kibana’s default view

These logs are important, along with gunicorn logs, for logging errors as they contain the WSGI request and traceback info.
- Install the
Python Logstash (python logging handler for logstash) using: pip install python-logstash
- Add the following in django’s settings:

Note: I’m using Level “ERROR” because I will be using Gunicorn to transfer the other request logs. (You can use any level depending on your requirements)

At this point, django will start sending logs to localhost:5959 but logstash is still not configured to read the input and transfer the logs to elasticsearch.
So we need to add the following configuration file for Logstash:

Note: Make sure the logstash port (5959) and elastic hosts (localhost:9200) are according to your settings.

- Now run logstash using ./logstash -f path/to/logstash.conf
- Open kibana dashboard
locahost:5601 and create the index logstash-*
- Start Django server and browse a page that raises some error (because my logging settings are only for level-ERROR)
Voila! All the django logs can be seen on kibana dashboard.

- Install Python JSON Logger using pip install python-json-logger
- Create/modify the gunicorn config file to the following:

Note: Modify line 34 to the logstash’s hostname and port number

- Run gunicorn server with --log-config path/to/gunicorn.conf argument.
Eg:
gunicorn project.wsgi:application --bind 0.0.0.0:8000 --log-config path/to/gunicorn.conf
- Enjoy your gunicorn logs on Kibana dashboard. You can also create separate dashboards for gunicorn and django logs.

The easiest way to implement this is to add a TCPLogstashHandler as a handler to the celery’s default logger.
Add the following lines to your django’s signal_handlers.py:

Note: Modify line 3 to the logstash’s hostname and port number

That’s it. Now it’s time for the DevOps team to get to work.

Security Engineer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store