RabbitMQ is a very good solution to many problems. If you don't use it correctly, It will be a monster. Yes, it is a monster.
1. Don’t use RabbitMQ or any queues unless it is necessary
What? Yes. Most people use message queues to create a Picasso (A Famous Painter) Like Architecture. Most of the cases, you don’t need it. If you add more and more components to your architecture means that you have more and more dependencies. So, it is better to avoid dependence. Always Keep it Simple, Stupid (KISS).
An alternative to Queue — Most of the time database is good enough to achieve your goals. Most of the time Queue increases the complexity. Advice — Evaluate before you implement
2. Don’t implement without doing proper research (learn theories)
There are some theories that you need to learn. Don’t just play around it and use it. You should know some theories, features like How to design queues, exchanges, exchange_type, channel vs connection.
Then you will be able to apply correct theories and implement proper patterns to improve the performance and utilization.
3. Don’t use misuse connections
Anyone knows that we should not create many connections. Each connection uses at least 100 kb of RAM. If the application opens more and more connections without closing them, the application will not be able to create connections after some point because out of memory. That means the RabbitMQ will reject the new connection. Worst case the rabbitMQ will be crashed.
First of all, make sure that you close the connection. To monitor that you can use Management UI. You will be able to see the following UI.
The best practice is to reuse connections and multiplex a connection between threads with channels.
4. Don’t add large messages in queues
If you try to pass a large message in queue, it may be not a good way to do it. For example, at the beginning of a project, we send a large payload via RabbitMQ. It worked as expected without any performance issues. After a few months, we got a lot of messages with a large payload, then the rabbitMQ was crashed. Then we move the payload to S3, then we put S3 URL to rabbitMQ.
large payload means
- RabbitMQ keeps in the RAM. If it has large, then put some of them to disk (page out). Then the rabbitMQ can be slow.
- Page out may block the queue from processing
- It may take more time for indexing with large files. It also takes more time to restart the rabbitMQ.
5. Auto-delete queues you are not using
Having an unused queue means that rabbitMQ has to allocate resources for unused queue. That could affect performance. You can add the following setting to auto-delete queue
- TTL (Time to Live ) policy https://www.rabbitmq.com/ttl.html
5. Don't forget to use Prefetch
The prefetch means — Number of messages that sends to the consumer at the same time. API should be utilized as much as possible. If we send 1 as prefetch, it may not affect the API because the CPU or RAM will not utilized.
The prefetch value is used to specify how many messages are being sent to the consumer at the same time. It is used to get as much out of your consumers as possible.
A too-small prefetch count means the RabbitMQ is waiting to get permission to send more messages. more about prefetch — https://www.rabbitmq.com/blog/2012/05/11/some-queuing-theory-throughput-latency-and-bandwidth/