Asynchronous Tasks with Celery + Redis in Django

Originally published at melvinkoh.me

Suppose your task takes about 20 seconds to finish before responding your client. Imagine you are the user of your application. It happens that a lot that processing can be actually be spared until later time whenever user makes a request to your site. AJAX is one way of doing tasks asynchronously, but we want to run our Python code asynchronously. Therefore, we resort to using Task Queue and Message Broker.

The typical process flow of a request to Django app: 1. Receive request 2. Server relay to a specific View 3. Process request in View 4. Respond to request

Say we want to fetch some data externally and process them whenever there is an update. Sounds familiar right? Yes, you can think of it as a Webhook on your CI server.

In this post, I won’t be discussing about why I choose Redis over other message broker like RabbitMQ, ActiveMQ or Kafka. The scope of this post is about applying task queue to facilitate execution of asynchronous task in Django. If you new to task queue, have no idea how to implement async tasks, or looking for solution to integrate Celery with Django, keep reading!

Installation of Celery and Redis

First, make sure you installed Celery and Redis interface, you can do so by downloading from PyPi.

pip install celery redis