Celery: Working with multiple queues

Photo by Adrien Delforge on Unsplash

If you are using a single queue for every task you are executing, sometimes it might take more time to execute the last task. This arises when your celery queue is already fed up with many tasks in line.

So we are going to create multiple queues and decide which task has to be routed to which queue. Here i’ll create 2 queues which will have their own responsibilities

1. mailer
2. uploader

But here, i’m putting a time delay for the both tasks to replicate the mailing and uploading functionalities

Before we move forward, we need to

Create a virtual environment and activate it

Install dependencies

pip install django celery redis

Create a django project

django-admin startproject src

Create an app

django-admin startapp app

And add the content below for the respective files

Open settings.py file and register your app in the INSTALLED_APPS. and also add the below snippet at the bottom of the file

CELERY_BROKER_URL = “redis://localhost:6379”
CELERY_TASK_ROUTES = { ‘app.tasks.mailer’: {‘queue’: ‘mail-queue’}, ‘app.tasks.uploader’: {‘queue’: ‘upload-queue’},}

Create celery.py inside project root folder and add the snippet

Create a tasks.py file inside the app and create 2 tasks named mailer and uploader

Create 2 views to trigger them named mail_queue_view and upload_queue_view

Create 2 URLs in project root urls.py file and link them with above views

1. /send-mails
2. /upload-files

Finally, we have to run 2 celery commands to manage 2 queues

celery -A src worker -n srcnode -Q mail-queue -l info
celery -A src worker -n srcnode -Q upload-queue -l info

Run the project

./manage.py runserver

Open your browser and hit the below URLs in 2 different tabs

When you hit the below URL, you can see the tasks will be executed by mail-queue

http://localhost:8000/send-mails

The below URL will triggers the tasks executed by upload-queue

http://localhost:8000/upload-files

celery multi-queue

If you want to isolate apps to specific queue. Modify CELERY_TASK_ROUTES variable like below

CELERY_TASK_ROUTES = {
    ‘app1.tasks.*’: {‘queue’: ‘queue-1’},
    ‘app2.tasks.*’: {‘queue’: ‘queue-2’},
}

All app1 tasks will be routed to queue-1 and all app2 tasks will be routed to queue-2

Find the complete project in github

GitHub - teja-official/django-celery-multiple-queues