Background Asynchronous Process using Django+Celery+Redis

Creating Background Asynchronous Process in Web application development is inevitable and Celery makes the whole process simple and easier. The main focus of this blog is to create a background process using Django and Celery, with Redis and as the message broker.

1.Download and Install Redis

You can download and install Redis from, https://redis.io/download. Make sure Redis is up and the Redis server is running.

Note: The Redis installed in your system should have version greater than 3.0

2. Basic Installations

As a first step, create a virtual environment and install the following:

$ virtualenv scheduler-env
$ cd scheduler-env
$ source bin/activate
$ pip install Django==3.1.3
$ pip install celery==5.0.0
$ pip install django-celery-beat==2.1.0
$ pip install django-celery-results==2.0.0
$ pip install redis==3.5.3

3. Creating a Django Project and App and adding basic configurations

$ django-admin startproject testproj
$ cd testproj
$ django-admin startapp testtasks

Add, the following in testproj/testproj/settings.py

#testproj/testproj/settings.py
INSTALLED_APPS += [
    'django_celery_beat',
    'django_celery_results',
    'testtasks'
]
CELERY_RESULT_BACKEND = "django-db"
CELERY_IMPORTS = ('testtasks.tasks')

4. Run Migrations

python manage.py makemigrations
python manage.py migrate

5. Add celery.py and update project configurations.

Create a file celery.py in the path testproj/testproj and add the following to the file.

Note: Replace ‘testproj ’in the file with the name of your project

#testproj/testproj/celery.py
import os
from celery import Celery
#set the default django settings to celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'testproj.settings')
app = Celery('testproj')
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
#message broker configurations
BASE_REDIS_URL = os.environ.get('REDIS_URL', 'redis://localhost:6379')
app.conf.broker_url = BASE_REDIS_URL
#creating a celery beat scheduler to start the tasks
app.conf.beat_scheduler = 'django_celery_beat.schedulers.DatabaseScheduler'

Update the celery project configurations in testproj/testproj/__init__.py

#testproj/testproj/__init__.py
from .celery import app 

6. Creating Tasks

Create a task file ‘tasks.py’ in the path testproj/testtasks and add the following. If you have noticed, this tasks.py is imported in the settings.py as a celery import.

#testproj/testtasks/tasks.py
import random
from celery import Celery
from testproj.celery import app
@app.task(name='tasks.add')
def add(x, y):
    return x + y

7. Verifying the Tasks

Open a terminal and start the worker process,

$ celery --app=testproj worker -l INFO

And in another terminal, open shell

$ python manage.py shell

Execute the following in shell,

>>> from testtasks.tasks import *
>>> add.delay(10,20)
<AsyncResult: 2f7057cb-7c2f-49b8-9f9b-640d54a62b6e>

When the task is invoked, you can see the worker process executing the task and giving the result in the terminal in which the worker process is running.

8. Creating Periodic Tasks

You can also create periodic processes that execute the tasks in a specific interval. Let's create a periodic task, which runs for every 3 seconds.

Add the following in celery.py

#testproj/testproj/celery.py
app.conf.beat_schedule = {
    'add-every-3-seconds': {
        'task': 'tasks.add',
        'schedule': 3.0,
        'args': (16, 16)
    },
}
app.conf.timezone = 'UTC'

The Above snippet of code runs the task ‘add’ for every 3 seconds. And you can check the following by running worker and beat in separate terminals. You can see the Celery beat starts the tasks and the worker process executes them.

Note: Change ‘testproj’ to your project name

Terminal 1- Worker process

$ celery --app=testproj worker -l INFO

Terminal 2- Celery Beat

$ celery -A testproj beat -l INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler

References:

1. https://docs.celeryproject.org/en/latest/django/first-steps-with-django.html
2. https://docs.celeryproject.org/en/latest/userguide/tasks.html
3. https://docs.celeryproject.org/en/stable/userguide/periodic-tasks.html