Tweaking RabbitMQ to suit your needs is a topic that deserves more than just casual attention.
Recently I had a discussion with a coworker on how to configure AMQP consumers, and in particular, what prefetch count to use. The observation was that a queue emptied much quicker when the consumer used a high prefetch count, or no prefetch count at all. Well yes, that is to be expected, but it is not the whole story.
To clarify, the consumer prefetch count (in the following simply referred to as prefetch count) is a value that tells RabbitMQ how many messages to send to a consumer before starting to listen for acknowledgements. If it is set to 1, the consumer will need to acknowlegde the received message before it will receive the next message. If it is unset, a consumer will receive all queued messages, and only then RabbitMQ will be expecting acknowledgements. If the prefetch count is set to any other number, RabbitMQ will not send more messages before at least one of the sent messages has been acknowledged.
This all applies to the situation where explicit acknowledgements are required. It is also possible to use automatic acknowledgement, and in that case the prefetch count is not applied. Similarly, when using `get` rather than consuming a queue, messages are retrieved one at a time anyway.
If the question is what prefetch count to use, the answer, of course, is that it depends. The general goal is to keep all consumers of a queue maximally busy, but there are a number of factors to consider:
- Messages that are delivered to a consumer are not available to other consumers. This can be exaggerated to the case where no prefetch count is set, and the first consumer that becomes available will receive all pending messages, leaving other consumers idle.
- The consumer will have to keep all received messages in memory while processing. Depending on the implementation and the environment of the consumer, this needs to be taken into account. High or unset prefetch counts may even simply crash the consumer before any processing is done.
- Messages that have been sent to a consumer are also kept in memory by RabbitMQ until they are acknowledged. Depending on the size and number of messages a high prefetch count may cause problems for the RabbitMQ server.
- Like any network protocol, there is a certain amount of communication overhead with AMQP in the form of sending and acknowledging messages. If the actual processing of a message is short in comparison to the overhead, setting a low prefetch count will hurt performance. Note that when using TLS to secure data transfer, the network overhead will be even greater.
- When a consumer fails and disconnects from RabbitMQ, any unacknowledged messages are normally redelivered to the top of the queue they came from. A higher prefetch count may therefore result in performance loss when the consumption of messages is a fragile process, for example because consumers need to interface with other services that may be unavailable. If this is a structural problem, it is probably better to configure deadlettering to set up a secondary flow to handle failures without interrupting the message flow.
- Consuming messages is rarely a continuous process. Often, the number of messages supplied to a system is highly variable, and part of the reason RabbitMQ is used, is to function as a buffer to absorb the peaks. These peaks may result from message generation itself not being continuous, but also from outages resulting in a message backlog building up in a queue. The consequence is that the consumers will need to be able to deal with periods of sustained delivery of high volumes of messages. Therefore consumers should be configured to handle the maximum of messages they can handle, and theprefetch count plays a significant role in this.
Taking these factors into consideration, we can formulate some guidelines for choosing a reasonable prefetch count.
- Regardless of the message volume, always set a prefetch count. It will ensure that whatever the situation, both RabbitMQ and the consumers will not get into a situation where enough messages reside in memory to cause problems.
- For proper round-robin message distribution with more than one consumer, set the prefetch count to 1. At the price of more overhead in message transfer, messages are evenly distributed over all consumers, requeueing is almost instantly after a consumer fails, and the memory load for both the consumers and RabbitMQ is as small as possible. This is a good approach for message based microservice architectures. Scaling to higher message volumes while preserving quality of service can then be done by adding more consumers.
- When optimizing the prefetch count for systems which need to handle high volumes of messages (either continuously or periodically), benchmarking is the way to go. Test the system with a range of values and determine at what value the networking overhead becomes insignificant compared to the time processing the messages. Also take into consideration the possibility of the consumer failing when processing a message. Higher prefetch counts in combination with frequent consumer failures will result in performance loss due to requeueing, so in this case it is prudent to use a lower value than optimal for throughput.
- Even for systems that process low numbers of messages where the consumers are idling for much of the time, determine the prefetch count for a message volume that would fully load the system. This ensures that unexpected backlogs will be handled cleanly, while not noticeably affecting the throughput for regular message volumes.
Although there is much more to tuning a message based system using RabbitMQ, choosing a suitable prefetch count for RabbitMQ consumers is instrumental. Hopefully you now have a good insight into how to go about this.