RabbitMQ Performance Optimization

The following are some advise on what you should try to do, in order to optimize the performance of your RabbitMQ broker:

1. Use Quorum Queues

Historically, classic queues supported replication via classic mirrored queues, but this should be avoided. Using a variant of the Raft protocol, a distributed consensus algorithm, quorum queues are both safer and achieve higher throughput than mirrored queues.

2. Keep Queues Short

Short queues are the fastest. A message published to an empty queue will go straight to the consumer, as soon as the queue receives the message. However, keep in mind that a persistent message in a durable queue will go to the disk. The recommendation is to keep fewer than 10,000 messages in one queue.

Queues should be kept short because many messages in a queue can put a heavy load on RAM usage. In order to free up RAM, RabbitMQ starts flushing (page out) messages to disk. The process usually takes time and blocks the queue from processing messages when there are many messages to page out. A large number of messages might have a negative impact on the broker since the process deteriorates queuing speed.

It is also time-consuming to restart a cluster with many messages because the index must be rebuilt and it takes time to sync messages between nodes in the cluster.

3. Enable Lazy Queues for Long Queues

Lazy queues gives you a more predictable performance and we recommend using them when you know that you will have large queues from time to time. Situations where, many messages are sent at once (e.g. when processing batch jobs) or where there is a risk that the consumers will not be able to keep up with the speed and consistency of the publishers, is best handled using lazy queues.

Disable lazy queues if high performance is required, if queues are always short, or if a max-length policy exists.

4. Limit Queue Size With Max-length or TTL

Another recommendation for applications that often get hit by spikes of messages, and where throughput is more important than anything else, is to set a max-length on the queue. This keeps the queue short by discarding messages from the head of the queue so that it never becomes larger than the max-length setting.

You can also limit queue size with TTL (Time-To-Live). Declaring a queue with the x-message-ttl property means that messages will be discarded from the queue if they haven’t been consumed within the specified time.

5. Consider the Number of Queues

Queues are single-threaded in RabbitMQ. Achieve better throughput on a multi-core system using multiple queues and multiple consumers. Optimal throughput is achieved with as many queues as cores on the underlying node(s).

The RabbitMQ management interface collects and calculates metrics for every queue in the cluster. This might slow down the server if there are thousands upon thousands of active queues and consumers. The CPU and RAM usage may also be negatively affected by too many queues.

7. Auto-delete Unused Queues

Client connections can fail and potentially leave unused queues behind, which might affect the performance of the system. There are three ways to have queues deleted automatically:

• The first option is to set a TTL policy on the queue. For example, a TTL policy of 28 days will delete queues with no messages consumed from them in the last 28 days.
• Another option is an auto-delete queue, that gets deleted when its last consumer has canceled or when the channel/connection is closed, alternatively, when it has lost the TCP (Transmission Control Protocol) connection with the server.
• Finally, an exclusive queue is only used (consumed from, purged, deleted, etc.) by its declaring connection. Exclusive queues are deleted when their declaring connection is closed or gone due to underlying TCP connection loss or other circumstances.

8. Set Limited Use on Priority Queues

RabbitMQ supports adding “priorities” to classic queues. Classic queues with the “priority” feature turned on are commonly referred to as “priority queues”. Priorities between 1 and 255 are supported, however, values between 1 and 5 are highly recommended. It is important to know that higher priority values require more CPU and memory resources, since RabbitMQ needs to internally maintain a sub-queue for each priority from 1, up to the maximum value configured for a given queue.

9. Don’t Use Too Many Connections or Channels

Try to keep connection/channel count low. Each connection uses about 100 KB of RAM, even more if TLS (Transport Layer Security) is used. Thousands of connections can become a heavy burden on a RabbitMQ server. In a worst-case scenario, the server can crash due to running out of memory.

Avoid connection and channel leaks. Connection leaks can cause RabbitMQ to run out of memory. Make sure your clients are not leaking connections. If you have more than 10 connections from the same host, you may have a connection leak.

Also, a large number of connections and channels can have an negative effect on the performance of the RabbitMQ management interface and make it slow to work with. Metrics must be collected, analyzed, and displayed for every connection and channel that consumes server resources.

10. Don’t Open and Close Connections or Channels Repeatedly

Repeatedly opening, and closing channels will create a higher latency because more TCP packets have to be sent and received. The handshake process for an AMQP connection is actually quite involved and requires at least seven TCP packets, even more if TLS is used. It is recommended that each process only create one TCP connection with multiple channels, in that connection, for different threads. Connections should be long-lived so channels can be opened and closed more frequently if required. Even channels should be long-lived if possible.

Do not open a channel every time a message is published. Best practice includes reusing connections and multiplexing a connection between threads with channels, when possible.

11. Separate Connections for Publisher and Consumer

For the highest throughput, you should separate the connections for publisher and consumer. Unless the connections are separated between publisher and consumer, messages may not be consumed. This is especially true if the connection is in flow control, constricting the message flow even more.

Another thing to remember is that RabbitMQ may cause back pressure on the TCP connection when the publisher sends too many messages to the server. When consuming on the same TCP connection, the server might not receive the message acknowledgments from the client, affecting the performance of message consumption and the overall server speed.

12. Use Persistent Messages and Durable Queues To Ensure You Don’t Loose Any Messages

To prepare for broker restarts, broker hardware failure, or broker crashes, declare queues as durable and send messages with delivery mode persistent. The latter is to ensure messages are on disk. Messages, exchanges, and queues that are not durable and persistent are lost during a broker restart. Therefore, if you cannot afford to lose any messages, make sure that your queue is declared as “durable” and your messages are sent with delivery mode “persistent” (delivery_mode=2).

13. Use Transient Messages and Non-durable Queues for High Performance

Persistent messages are heavier as they have to be written to disk. Similarly, lazy queues have the same effect on performance even though they are transient messages. For high performance, use transient messages and for high throughput, use temporary/non-durable queues.

14. Consider Prefetch Size

The prefetch value is used to specify how many messages are consumed at the same time. It is used to get as much out of the consumers as possible.

The RabbitMQ default prefetch setting gives clients an unlimited buffer, meaning that RabbitMQ, by default, sends as many messages as it can to any consumer that looks ready to accept them. Messages sent are cached by the RabbitMQ client library in the consumer until processed. A typical mistake is to have an unlimited prefetch, where one client receives all messages and runs out of memory and crashes, and then all messages are redelivered.

A prefetch count that is too small may hurt performance, as RabbitMQ is typically waiting to get permission to send more messages.

A prefetch count that is too large could instead deliver many messages to one single consumer, and keep other consumers in an idling state.

15. Use the Latest Stable Versions

Stay up-to-date with the latest stable versions of RabbitMQ and Erlang.