Both Celery and Python RQ supports the following
- Bulk jobs queueing
- Task dependent on other tasks (up streams and down streams)
- Task success/failure callbacks
- Job timeout in queue
- Failure timeout to kick jobs out of the queue
- Bypassing workers (run tasks in eager mode without detaching to background process)
- Flower monitoring, RQ Dashboard
- Celery beat, RQ scheduler
Celery has tons of other options but let’s forget as we may not need them at all
Some helpful articles on Celery
Celery at Scale using AWS ASG
https://medium.com/23andme-engineering/celery-74062f510d72
Celery at Scale using Kubernetes HPA
https://learnk8s.io/scaling-celery-rabbitmq-kubernetes
Scale up Messaging Queue with Celery (Processes vs Threads) — Part 1
Handling I/O Bound Tasks with Python Celery using Processes vs Threads Pool
