Getting Started with Celery & RabbitMQ

Getting Started with Celery

In this article, I’m going to cover the basic concept of Celery.

• What is a task queue?
• Introduction to Celery
• What is a Broker?
• Choosing a Broker
• Installing Celery
• Working with Celery
• Conclusion

Let's get started….

1. What is a task queue?

• The Task is essentially a python function; a task that can be completed given a set of parameters. A Task Queue is a set of tasks to be run.
• There can be multiple named queues defined in an application. Task queues let applications perform work, called tasks, asynchronously outside of a user request.
• If an app needs to execute work in the background, it adds tasks to task queues. The tasks are executed later, by worker services. The Task Queue service is designed for asynchronous work.

2. Introduction to Celery

• Celery is an open-source asynchronous task queue or job queue which is based on distributed message passing for Python web applications used to asynchronously execute work outside the HTTP request-response cycle.
• Celery can run on a single machine, on multiple machines, or even across data centers.
• Celery communicates via messages, usually using a broker to mediate between clients and workers. To initiate a task the client adds a message to the queue, the broker then delivers that message to a worker.
• Celery is Simple, Fast, Flexible, and Highly Available.
• Celery is easy to integrate with web frameworks like Django, Flask, etc.

3. What is a Broker?

• The broker is the third-person facilitator between a buyer and a seller.
• Celery requires a solution to send and receive messages; usually, this comes in the form of a separate service called a message broker.
• In celery, the broker is Redis, RabbitMQ, etc who conveying the message between a client and celery.

4. Choosing a Broker

• In this case, I’m using RabbitMQ.
• RabbitMQ is feature-complete, stable, durable, and easy to install. It’s an excellent choice for a production environment.
• You can install & setup the RabbitMQ by using the following commands:

sudo apt-get install rabbitmq-server

5. Installing Celery

Celery is on the Python Package Index (PyPI), so it can be installed with standard Python tools like pip or easy_install

pip install celery

6. Working with Celery

• Create the file tasks.py

from celery import Celery

app = Celery('tasks', broker='pyamqp://guest@localhost//')

@app.task
def add(x, y):
    return x + y

• You can now run the worker by executing our program with the worker argument:

celery -A tasks worker --loglevel=INFO

• For help you can you the following commands:

celery worker --help
celery help

• Let's create a file test.py to call our task

from tasks import add
result = add.delay(4, 4)
#delay() is used to call a task
print(result.ready())
#ready() returns whether the task has finished processing or not.
print(result.get(timeout=1))
#get() is used for getting results
result.get(propagate=False)
#In case the task raised an exception, get() will re-raise the exception, but you can override this by specifying the propagate argument

7. Conclusion

• Celery is a good option to do asynchronous tasks in python. It is distributed, easy to use.
• Celery helps us take off some load from the application server, and which helps us serve your requests fast.

Thanks for reading. If you found the article useful don’t forget to clap and do share it with your friends and colleagues. :) If you have any questions, feel free to reach out to me.
Connect with me on 👉 LinkedIn, Github :)