Scaling Django Applications with Celery
Scale is a common problem for any growing technology company. Early in a company’s lifecycle, development speed is often the top priority with the focus on creating a proof of concept and demonstrating the value of the company rather than on the scaling issues that are encountered by a larger company. When you’re trying to get one user, the last thing on your mind is scaling to accommodate hundreds of thousands of users. However, as a company gathers marketshare, the scale (pun-intended) tips.
For example, imagine building an application that takes in a request from a pizzeria and sends a message to all of the customers of that pizzeria reminding them to order their favorite pizza. The first thing you need to do is handle a single request that gathers context for all of the customers of a pizzeria and sends a reminder message to each customer. You build a Django application to do just that.
So what’s the problem?
Let’s say that it takes 1 second to gather all of the context for each customer and send the message, and that each pizzeria has 60 customers. Therefore it takes 1 minute (1 second/customer x 60 customers) for each request to complete. Now let’s imagine each Django application can only handle 1 request at a time. With only a single pizzeria, your application never receives more than 1 request per minute, and everything works just fine. However, as your company adds more pizzerias, imagine your company starts to receive 10 requests per minute, and requests start timing out because your application cannot process the requests fast enough. Therefore, you spin up 10 Django applications to handle the 10 requests per minute, put them behind a load balancer, and watch your company’s server budget evaporate because you are scaling the entire application to accommodate the resource needs of a single bottleneck: sending the message.
Celery to the rescue!
“Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system.” — Celery Documentation
Thankfully, Celery can help you scale the bottlenecks of your application asynchronously and independently. Celery is a task processing system that allows units of work to be executed asynchronously using task queues. Therefore, rather than replicate the entire application, we can use Celery to perform asynchronous workloads without impacting the performance of the rest of the application. In our example, we can use Celery to process the context gathering and message sending for a single customer. Now, instead of the Django application processing every message for each customer, the application just has to iterate through the customers, calling the Celery task for each customer. This will reduce the amount of time it takes for each request to complete, freeing up the Django application to process incoming requests instead of spending processing time gathering contexts and sending messages.
Let’s talk specifics
Now that we can complete our requests much faster, the throughput of our requests increases, and the need to replicate the entire Django application decreases. For example, imagine that now it only takes 1 second for our Django application to send all 60 messages out to Celery for processing. Previously, each Django application could only handle 1 request per minute. However, using Celery, each Django application can now handle 60 requests per minute, increasing the throughput of request processing by a factor of 60. Where it previously took 10 Django applications to handle the 10 requests per minute, we can handle 6 times that amount of requests with only 1 Django application.
With Celery, instead of replicating our entire Django application to increase the throughput of our application, we only need to replicate our Django application to increase the throughput of Django’s request processing. Even though we still need to scale our Celery workers to increase the throughput of message processing, we can do so without the additional overhead of replicating an entire Django application, saving our company’s server budget. Furthermore, we can now take advantage of Celery’s retry processing to improve the reliability of our Django application, but more about that in my next post, Improving Reliability of Django Applications with Celery.
