RabbitMQ is an open-source message broker that originally implemented AMQP(Advanced Message Queueing Protocol). Message brokers adheres to protocols for communicating between softwares using well-defined messages.
WHY use RabbitMQ?
Message queueing enables software applications to connect and scale. Applications can connect and communicate to each other. Messaging is asynchronous, decouple applications by separating sending and receiving data. Message queueing is embodiment of various software practices publish/subscribe, asynchronous processing or work queues. It also acts as temporary message storage until received. RabbitMQ is a message-broker and comes with its pros and cons.
- Reliability: RabbitMQ can be configured to trade-off performance for reliability (Persistence, Acknowledgements, Publisher Confirms and HA). This point can be taken in for both pro and con as there is trade-off between throughput and reliability.
- Flexible Routing: Any message before reaching queue goes through exchange. Exchanges are responsible for handling routing of messages and comes in several types (Fanout, Direct, Topic and Custom Implementation)
- Clustering: RabbitMQ servers on local network can be clustered together, forming a single logical broker. And many more . . .
RabbiMQ comes with its advantages and disadvantages. RabbitMQ provides multiple fine tuning options for best results. While working on it recently, I came across a few which I want share here and are as follows —
RabbitMQ connections requires at least 7 TCP packets for handshake. Channels can be opened and closed more frequently if needed. Best practice is to reuse connections and multiplex a connection between threads with channels.
- AMQP connections: 7 TCP packages
- AMQP channel: 2 TCP packages
- AMQP publish: 1 TCP package (more for larger messages)
- AMQP close channel: 2 TCP packages
- AMQP close connection: 2 TCP packages
- Total 14–19 packages (+ Acks)
Heartbeats defend against certain network equipment which may terminate ‘idle’ TCP connection when there is no activity on them for a certain period of time. The heartbeat_timeout value defines after what period of time the peer TCP connection should be considered unreachable(down) by RabbitMQ and client libraries. Default value for heartbeat_timeout is 60 seconds. The value is negotiated between the client and RabbitMQ server at the time of connection. When both values are non-zero, lower of the requested values is used.
Clients send heartbeat frame every (heartbeat_timeout / 2) seconds. If two consecutive frames are missed, the peer is considered to be unreachable and the TCP connection is closed. Clients will have to re-connect afterwards. A good practice is to keep heartbeat_timeout between 5 to 20 seconds. Setting low value can lead to false positives due to transient network congestion, short-lived server flow control etc.
Blocked Connection Timeout
Blocked connection notifications are sent when broker is running low on resources(memory or disk). A connection.blocked notification is sent to publishing connections the first time RabbitMQ is low on resources.
blocked_connection_timeout value defines after what period of time the peer TCP connection is interrupted and dropped.
A blocked connection may last for an indefinite period of time, stalling the connection and possibly resulting in hang until the connection is unblocked. blocked_connection_timeout is intended to interrupt a connection that has been blocked longer than the given timeout value. Best practice is to keep the value between 150 to 300 seconds.
The persistence layer has two components: Queue Index and Message Store. The queue index is responsible for maintaining knowledge about where a given message is in a queue, along with whether it has been delivered and acknowledged. Therefore there is one queue index per queue. The message store is key-value store for messages, shared among all queues in server. Messages can either be stored directly in the queue index, or written to message store.
Under memory pressure, the persistence layer tries to write as much out to disk as possible, and remove as much from memory. Following must remain memory:
- Each queue maintains metadata for each unacknowledged message. The message itself can be removed from memory if its destination is the message store.
- The message store needs an index. The default message store index uses a small amount of memory for every message in the store.
If messages are written to queue indices:
- Messages can be written to disk in one operation rather than two; for small messages this can be substantial gains. The queue index keeps blocks of fixed number of records in memory; for not-so-small messages, memory consumption can be substantial. Generally, messages with size ~4096 bytes(inclusive of headers) are optimum to store in queue indices. RabbitMQ configuration variable to define the size of messages to be written in queue indices is queue_index_embed_msgs_below.
- Messages stored in queue indices do not require entry in the message store, thus do not have memory cost on that front. Although, unacknowledged messages with destination queue index, are always kept in memory.
- If a message is routed to multiple queues through an exchange, the message will be written to each queue indices. If such a message is written to the message store, only one copy needs to be written.
If there are any more ways to fine tune RabbitMQ with better performance and reliability, please reach out.