A complete guide to production-ready Celery configuration
Note: For non-members, this article is also available at https://progressstory.com/tech/python/production-ready-celery-configuration/
Whenever we work on some heavy data-intensive application or some long-running tasks, it generally slows down the performance of the application, and users have to wait until the task is completed. It was okay in the legacy systems where we used to wait for long to just load some single-page applications.
Modern users expect pages to load instantaneously, to solve this we consider many solutions like multiprocessing, multithreading, asynchronous functions using async/await or the Message Queues.
At a high level, message queuing is pretty simple. A process, called the Producer, publishes Messages to a Queue, where they are stored until a Consumer process is ready to consume them.
Celery decreases performance load by running part of the functionality as postponed tasks either on the same server as other tasks or on a different server. These workers can then make changes in the database, update the UI via webhooks or callbacks, add items to the cache, process files, send…
