Get to know Celery

What is celery

Celery is a distributed task queue which means it lets you run tasks from your python script asynchronously. An asynchronous task is defined by a computation that runs on a background thread and whose result is published on the UI thread. Sometimes we would have to wait till a process is complete to move or execute the following function. We can avoid this situation using CELERY. Instead of waiting for the process to complete you can send it over to celery and continue with the rest of the task.

Advantages:

• Celery can run on a single machine, multiple machines, or even across data centers.
• Celery is Simple, Fast, Flexible, and Highly Available.
• Celery is easy to integrate with web frameworks like Django, flask, etc.

Installation ( Ubuntu ):

Install celery using the command:

pip install celery

In addition to Celery, you will also need a task queue and a broker. Task queues are used as a mechanism to distribute work across threads or machines.

The broker is the third-person facilitator between a buyer and a seller. Celery requires a solution to send and receive messages; usually, this comes in the form of a separate service called a message broker. Some of the most commonly used brokers are RabbitMQ, Redis, Amazon SQS, etc.

I’m using RabbitMQ. You should install erlang to use RabbitMQ. Command to install erlang and rabbitmq

sudo apt-get install erlang
sudo apt-get install rabbitmq
sudo systemctl enable rabbitmq-server
sudo systemctl start rabbitmq-server
sudo systemctl status rabbitmq-server
sudo rabbitmq-plugins enable rabbitmq_managemen

Now you will be able to log in to RabbitMQ. You can log in by going to the localhost with port number 15762( localhost:15672/ ). It has default login credentials.

USERNAME: guest
PASSWORD: guest

You can also create your own credentials using the commands given below,

create a user       :   sudo rabbitmqctl add_user admin admin
setting usertags    :   sudo rabbitmqctl set_user_tags admin administrator
setting permissions :   sudo rabbitmqctl set_permissions -p / admin "."".""."

Now our task queue is ready. you can connect your program to the task queue using AMQP URL. AMQP stands for Advanced Message Queuing Protocol.

AMQP URL: pyamqp://guest@localhost//

Working with Celery

Create a file “main.py”

from celery import Celery

app = Celery('tasks', broker='pyamqp://guest@localhost//')

@app.task
def twice(x):
    return x*2

You can now run the worker by executing our program with the ‘worker’ argument:

celery -A <filename> worker --loglevel=info

Now open a new tab in the terminal, go to the driver you saved your file in and type ‘python’ for script mode. Import the file and run the required function.

syntax : 
  from <file_name> import <function_name>

In this case:
  >>>from main import twice
  >>>twice.delay()

Delay is used to push the task to celery. After running this if you check the celery terminal you will find the output there once the task is done. In an asynchronous task, the process that takes time could be sent to celery and the system could execute the rest without waiting for the rest.

Conclusion

Celery is a good option to do asynchronous tasks in python. It is distributed, and easy to use. Celery helps us take off some load from the application server, which helps us serve your requests fast.