Anyone who knows a bit of programming knows how to call a function. A function is a small self contained routine that does some task or other. Sometimes it takes input and usually it produces some kind of effect, often returning some kind of value. In most programming languages you just put the name function, open some parentheses, maybe add in a parameter or two (or more), and then close the parentheses. Python is no exception
So far, so simple. However, what if what you want to do is not that simple? Maybe the task will take a very long time and you want to return some kind of result before it is finished Maybe the task needs to happen on another computer. This is where celery comes in. Celery’s website describes itself thus:
Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.
The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).
Yeah, I’ve used celery for years and I’m not 100% sure what all that means. The documentation is extremely comprehensive, but is not particularly easy for a beginner to get hold of.
At its most basic, celery is a framework for running tasks. A task can be thought of as a function with a little magic thrown in. The magic is added by a python decorator
is a function
Is a celery task (assuming app is a Celery instance set up correctly)
When using Celery, there will be a worker process running somewhere. A Celery worker process knows what tasks can be run (the app,task magic adds the task to a registry of tasks), and the worker connects to something called a queue to find out what tasks to run. A queue is simply a list of things that is processed first in, first out. One queue can be connected to multiple workers — think of it like the Post Office, where there will be one queue for all the cashiers, and when a cashier has finished his current tasks, he will press a button, and the announcement “Cashier Number 5 please” will call the next person waiting in the queue. Similarly, one worker can listen to multiple queues. Maybe in our post office there is a special counter for large parcels. People with large parcels join that queue, and the worker on that counter deals with them in order, but when there are no parcel customers, he can help a customer who is just wanting a book of stamps in the main queue.
As for the actual details of how a queue works, that is something that Celery generally doesn’t care about — it can connect to various message queueing systems or “brokers”. The All-Singing All-Dancing message queueing system is RabbitMQ, but for a simple system you can use Redis
So a worker pulls tasks out a queue. What puts them in? It can be called from any bit of python code that has the celery library loaded. Simply configure celery (what queue and what message broker you want) and then simply call
If the sending app and worker app are running on the same codebase, it is even easier
Instead of calling the function in the normal way, call
So what would all this actually be used for? Let’s work through a simple example
You are going to put together the next YouTube, people can upload videos, you then save them somewhere, and then transcode them into different resolutions so they can be played over the web.
You begin with a single server that does everything and all in a single thread. The user hits submit on their file upload and the page continues loading until it is ready and then display the message “Your Video is now ready”
This works fine, except some people — seeing a page loading for a minute or two — think it is broken, hit back and attempt the upload again. You are wasting a lot of space on duplicate videos and the users aren’t happy about the experience
The first thing you can do is where you begin encoding the video, use .delay() and a local queue using redis to make it run asynchronously, The page then returns to the user almost immediately with a message that now says “Your video will be ready shortly”. Everyone’s happy and people start to use your site a lot. You need to add more servers to manage the load. But you notice that sometimes you can upload a video and it is ready in moments, other times it takes ages. You look into it, and it depends on if you hit a server that is already running a transcode you just join the back of the queue, and sometimes a few can stack up, or you can hit a machine with an empty queue and it happens immediately
The solution to this, is to share the queues between all the servers, and machine can put a transcode task in the queue, and whenever a server is not transcoding a video it requests one from the queue to run. This spreads the load well, but if someone uploads a 4K video, it uses up a lot of memory on that particular server. It gets through it, but the Web requests that that server responds to are slow and laggy while it is doing the transcoding. At this point you need to split your server. One pool will just be web servers, the others will be the transcoders. Then extra load on the transcoders will not affect the web servers at all. While you could run the same code on both that is currently running just without the web server running on one set and without the worker process running on the other, it would make more sense to split the code and remove all the transcoding from the webservers and web code from the transcoders. Since the webserver no longer knows about the transcode task, you will need to use send_task to send the task by name rather than just importing the function and calling it with delay()
You then need to start running hourly reports. You set up a cron job on one of your servers and it all goes well, but because you are using auto-scaling it gets rotated out and the reports stop working. You could make one instance “special” so that it doesn’t ever get removed, but this is a little ugly. You try putting the cron job on all the server, but then you get half a dozen duplicate reports each time. Celery can help here to.
A special celery app can be run called Celery Beat. This adds tasks to a celery queue on a regular schedule. Then if all server listen to this queue, then when a task is added to the queue, one will remove the task and run it.
I hope this has given a very high level overview of what Celery is for and what it can do.