How to create a dead letter queue in Celery + RabbitMQ
Recently, I started learning how to run async tasks in Celery, which has been used in our company product. One interesting requirement is how to create a dead letter queue in Celery?
RabbitMQ has already supported arguments,
x-dead-letter-routing-key, when creating a new queue.
I have made a simple example to create a dead letter queue:
You need a rabbitmq as a broker and redis as a backend. Run
docker-compose up -d to start containers.
celery -A tasks worker --loglevel=info
# open ipython
from tasks import add, div
div.delay(2,0) # throws ZeroDivisionError
Reject has to be used with
acks_late together, because by default the worker will acknowledge to remove the message from the queue. If
acks_late is set, an acknowledgement is sent after the result of task is returned. When
acks_late is enabled, the worker can reject the task that will be redelivered to a dead letter queue.
- Originally, I had the same that if we don’t include
app.conf.task_queuesand start workers at first time,
deadletterqueue will not be created.
- According to this question, K.P. proposed a solution to create
deadletterqueue which works very well. So I modified my example as well.