Guaranteed message delivery with RabbitMQ
RabbitMQ is a mature and stable messaging server that uses the AMQP protocol to interchange messages between producers and consumers.
It can serve several uses-cases, including: transient channels, fan-out messages, federation and others.
Here I will explain some (maybe all…) configurations and parameters for my most common use-case: Guaranteed message delivery for worker job queues.
This means that if a message is accepted by RabbitMQ, it will be consumed and only removed from the queue once acknowledge.
This will cover exchange, queue, consumer and producer setup. And will not cover RabbitMQ server reliability and clustering. I also do not care for ordering as I use multiple consumers most of the time.
Quick RabbitMQ recap
To use rabbitmq you open a connection to the server. On this connection you can open one or more channels, those channels are what you use to publish and consume messages.
Recommendation is that you publish in one channel and consume from another. Also in most implementations channels are not thread-safe, so you need one channel per thread or make sure to synchronize its access.
To “put a message on rabbitmq” you publish it to an exchange. An exchange receives a message and route it to a queue.
To get messages from rabbitmq you consume from a queue. To remove messages you ack or reject them. You can also nack and requeue, to keep the message on the queue.
Declare is the act of creating an exchange and a queue. Bind is what connect an exchange to a queue.
Not losing your exchanges and queues
First we need to make sure that our exchanges and queues survive restarts and disconnects, for this we make our exchanges and queue “durable”.
For exchange this means that `durable=true`, `auto-delete=false` and `internal=false` need to bet set as arguments when you declare it.
For queue this means that `durable=true`, `auto-delete=false` and `exclusive=false` need to be set as arguments when you declare it. But there are more arguments for the queue in a few paragraphs below.
This means that the exchange and queue will continue to exist even if no more client exist and in case of server restart. The declared exchange and queue are durable.
You will also need to bind the exchange to a queue, to make the messages accessible.
Not losing rejected messages
Now, when consuming messages fails they are rejected. If you want to catch those failed messages, you set up a dead-letter-queue.
A dead-letter-queue , accompanied by a dead-letter-exchange, is a way for you to capture rejected messages from the queue.
You set up them up as regular exchange, normally suffixed with “.dlx” and queue ending in “.dlq” and bind them. Also remember to make both durable.
When declaring you queue you setup `x-dead-letter-exchange=exchange-name.dlx` and `x-dead-letter-routing-key=` as arguments.
Summary of exchange and queues declaration
You declare, in order:
- A durable exchange
- A durable exchange for failed messages (dlx)
- A durable queue for failed messages (dlq)
- Bind dlx to dlq
- A durable queue, with dead-letter arguments
- Bind the first exchange to the last queue
Of course this is in case of a single exchange to a single queue, if you want to route to many queues you need to repeat the dlq and queue steps.
Consuming messages safely
There are actually more than on way to consume messages in AMPQ, but the common and recommended one is using the consume method.
You should set `auto-ack` to false, and be sure to only ack messages when they are good to be removed.
If you fail to process a message you should reject it so they go to the DLQ.
If you fail to ack or reject the messages, it will go back to the queue eventually.
This will ensure messages only leave the queue when you are done with them, and that they go to the dlq in case of failure.
Once you consume and ack a message in the DLQ it will be gone. You can also shove the messages from the DLQ back into the original queue to reprocess.
Publishing safely
Having a durable queue does not mean a durable message. Messages will still be kept in memory and be lost on restart or in case of non-routable messages.
To make sure a message will be routed to a queue you set it as `mandatory=true`. This means publish will only return success if the message end up in a queue, and you receive error if the message is not routable (like no binding on the exchange, or a bad routing key rule).
To make sure the message goes to disk you set `persistent=true`. This will make sure rabbitmq write it to disk on the queue.
And for last. you should wait for confirmation that the message was written.
There are other methods such as transactions, but the faster and recommended way is to use waiting for confirmation.
You call the confirm or wait_for_confirms on an open channel to wait for rabbitmq to confirm that the messages sent on this channels were written to where they should.
You can wait on each message or publish a batch and wait once. Waiting is per channel.
It can also be useful to use `lazy` messages to control memory usage to make rabbitmq write to disk as soon as possible. Unless you use quorum queue, but that is for another post.
Conclusion
These are all the measure that I know to make sure your rabbitmq messages are safe. Apart from cluster, but that is another story.
Important to note that each persistence, each guarantee we add will impact performance. Direct transient messages are probably the faster you can go. That said, this is still a very fast and efficient messaging server.
If you keep a close eye on memory consumption, disk space and number of open connections to your server, RabbitMQ is very likely to not fail you, even without all above measures.
So, did I miss something? Got any other tips?
