I have worked with RabbitMQ a long time. Its easy to make various mistakes which prevent you from getting the best out of it. Each application has its own performance requirements, like some of them require really high throughput while others process batch jobs where its fine if there are some delays. We will start with general do’s and don’ts in part 1 and dive into specifics later.
- RabbitMQ queues are single threaded: Since RabbitMQ queues are single threaded, one queue can handle max 50k messages/sec (also depends on available RAM and swap size).
- Don’t let your queues grow too long: Since RabbitMQ stores all unconsumed messages in RAM, if your queues start to grow too much it will lead to memory and performance issues on broker. RabbitMQ will start throttling your publishers using Flow Control. If your memory consumption breaches a preset threshold, broker will stop accepting any new messages altogether.
- Use lazy queues for batch publishing jobs: Lazy queues automatically store all the messages to the disk. They are loaded into the memory only when requested by the consumer. This helps with the memory consumption issues in broker. You can use this option, if your consumers are fine with somewhat higher latencies.
- Set TTL/max-length for all the queues to limit size: This will help you ensure that your queues don’t grow beyond a certain size and cause memory issues on the broker.
- Limit Number of queues: Since RabbitMQ management interface tracks and calculates several metrics for all the queues on the cluster, having too many queues might cause performance issues.
- Always have separate connections for Publishers and Consumers: RabbitMQ applies flow control on publishers if consumers can’t keep up with them, which limits the bandwidth of connection. If your consumers and publishers share the same connection, then your consumers will also be throttled by RabbitMQ, which might lead to a cascading effect bringing down the broker.
- Persistent messages and Durable queues: If you want your queues and messages to survive broker restarts and crashes, then declare your queues as durable and set delivery mode to persistent for messages. However, performance would be impacted by persistent messages as they have to be always stored in the disk.
- Prefetch count: Prefetch count determines how many messages will be delivered to the consumer at any point of time before receiving acknowledgement. Having too small of a prefetch count would hurt performance, as broker will be waiting for the acknowledgement most of the time. If you have a very high prefetch count, it might lead to all messages being delivered to a single consumer, resulting in other consumers sitting idle. You need to figure out the correct prefetch value for your use case based on number of consumers, network latency, consumer latency.
For my other articles on RabbitMQ :