Asynchronous tasks in Python with Celery
Celery is an asynchronous task queue/job queue based on distributed message passing. Understand what is synchronous/asynchronous programming it’s fundamental to understand celery. The section below explains it, if you already know it, you can jump to When to use it? section or go straight to Celery section.
Synchronous & asynchronous code.
In this question in stackoverflow, the user themightysapien have done a great analogy to explain synchronous and asynchronous code:
Synchronous Execution
My boss is a busy man. He tells me to write the code. I tell him: Fine. I get started and he’s watching me like a vulture, standing behind me, off my shoulder. I’m like “Dude, WTF: why don’t you go and do something while I finish this?”
he’s like: “No, I’m waiting right here until you finish.” This is synchronous.
Asynchronous Execution
The boss tells me to do it, and rather than waiting right there for my work, the boss goes off and does other tasks. When I finish my job I simply report to my boss and say: “I’m DONE!” This is Asynchronous Execution.
An asynchronous code is a piece of code that runs independent and separated from the execution flow of a program. We use this when we don’t want to block the main execution flow of a program.
You might be thinking: — Ok, I got it! but why the hell I want or need to do this???
A good reason would be if this piece of code could take to much time to get finished and you don’t want to wait.
When to use it?
Example 01: The sign up
Lets pretend that we have an website and we’re developing a feature to signup. In signup process, we have to:
validate user's credentials;
store user's info in a database;
send a confirmation e-mail;
The SMTP server, responsable to send the email, sometimes takes 3, 4 or even 6, 7 seconds to do it. The user doesn’t have to be confirmed to start using the platform. We don't want to make them wait so long, watching a boring load screen. During this time, the user could give up the signup. In this case, we need to do everything synchronous except the email part. We will just put the mail in box to be delivered later. Check the snippet below.
Only the function start_to_send_confirmation_email should be asynchronous. In execution time, the interpreter will jump from line 3 to line 4 before the email be sent. The piece of code from function start_to_send_confirmation_email will be executed separated from this flow. The user will be returned before the email be sent.
Example 02: The financial report
A Financial Analyst clicks in a button to see the monthly results. This will take a while and the analyst doesn’t want to wait a long and boring loading screen to see the results, he wants to continue to use the platform and check the results later when they are ready.
As a developer, you don’t want a request pending for minutes, or even hours. Actually this request has a great chance to reach timeout limits by your web server or DNS provider (damn it cloudflare).
— Ok, now I know what is async code and when I should use it. But, how can I do it in python?
To do this we can use python thread module and do it by hand or use a task job like Celery.
Celery
The principle is very simple: one (or many) programs add tasks in a queue to be executed. This queue is also known as Message Broker. Celery isn't just a library, it runs in a different service than the application, like databases, SMTP servers, etc. This service watch the queue and every time a new task is added celery executes it.
A celery task in many cases is a complex code, that needs a powerful machine to execute it. Like i said before, celery runs in a service split from your application, what means that you can run it in a separated machine and could have as many queues as you wish. This helps a lot to scale an application.
Our First Celery App
We are going to create our first celery app. This have to be simple and focused in understanding celery concepts.
Pré-requisites:
We will use Redis as message broker. Basically, a message broker organizes the queue, it’s an abstraction. Celery doesn’t have the intent to organize queues, it just executes tasks.
Create an empty directory and name it as you wish and do the steps below inside it.
- Create a business logic file called first_app.py;
- Create a folder called celery_stuff;
- Inside celery_stuff folder, create an __init__.py file;
- Inside celery_stuff folder, create an tasks.py file;
Your project tree should looks like this:
├── celery_stuff
│ ├── __init__.py
│ └── tasks.py
└── first_app.py
Let’s create a function to executes a celery task asynchronous in first_app.py. Note that start_serve_a_beer it’s a normal sync function created by us. The serve_a_beer.delay() will put the task in the queue (message broker) to be execute by celery service. The serve_a_beer task doesn't exists yet and will be created in the next steps.
Now, in celery_stuff/tasks.py, create a celery app instance, the task serve_a_beer and register it to the celery app.
Running
Before we run our app, we need to run celery service because when we run the first_app.py, the code will add a task to the queue (line 11) and if celery service wasn’t running the task won’t be executed.
Running Celery
$ celery -A celery_stuff.tasks worker -l debug
Running First App
$ python first_app.py
If everything is good, you should see this in celery logs:
Working with multiple tasks
Let’s get back to celery_stuff/tasks.py and add a new task called serve_a_coffee.
In first_app.py file, let’s import a new task called serve_a_coffee and start them. The file now should looks like this.
Run celery and first_app again.
$ celery -A celery_stuff.tasks worker -l debug
$ python first_app.py
Both tasks should be executed.
Working with multiple queues
Imagine this code in production, executing thousand of tasks per minute. Of course this tasks is very similar, but pretend that the task serve_a_beer is slowest and more complex than serve_a_coffee. The process to make a coffee is really simple, you will need water, coffee and a filter! But, to do a beer, you will need a lot of ingredients and wait for at least a month! When you have different features, is a wise choice to split it in different queues. This way, you can choose different machines to run this different tasks.
Defining the queues
We didn’t defined a queue. So, tasks will be sent to the default celery queue. Beers and coffees were queued together. This is bad because every time a lot of beers are being made, the coffees will have to wait a lot.
Let’s change the code to split our tasks into specific queues. In celery_stuff/tasks.py file, configure the routes in the app. In the task_routes configuration we can specify the queue for each task.
Running celery with different queues!
In the previous section, we run 2 services:
- App service
- Celery service
Now we will run 3 services:
- App service
- A Celery service to serve_a_beer
- Another Celery service to serve_a_coffee
Run the commands below to set up all of them;
$ python first_app.py
$ celery -A celery_stuff.tasks worker -l debug -Q beer
$ celery -A celery_stuff.tasks worker -l debug -Q coffee
We used the -Q argument to specify the queue. If we had 10 different queues, we could run a celery service for each one of them!
You can, of course, use the same celery service for many queues as you wish. This is very common in development mode.
$ celery -A celery_stuff.tasks worker -l debug -Q beer,coffee
Are you feeling lazy to do the step by step and install everything? Just clone the repository on github and run it with docker.
First, change line 4 on celery_stuff/first_app.py to:
Now, just run the containers.
$ docker-compose run --build
You should see a beer and a coffee in the logs!
Conclusion
Celery is a good option to do asynchronous tasks in python. It is distributed, easy to use and to scale. In this article, we have learned what is a celery app, how to create tasks, how to route tasks to different queues and how to run celery services to specific queues. Celery could be a little bit frustrating in first steps, specially for some gaps in the documentation, but it’s very easy to use and abstracts a lot of complexities. I hope I have filled some of these gaps, specially for the first steps!
