RabbitMQ: Concepts and Best Practices

Introduction

In most modern applications, we are moving towards a microservices architecture. As the microservices architecture consists of numerous independent services, the connections between these services become colossally large. Consequently, the system becomes convoluted.

We can’t rely on request/response-based systems in this situation since applications usually wait for another service’s response. A client triggering an inherently time-consuming service, such as one that requires significant computing resources, can exacerbate this situation. This can impact the user experience if the client is a front-end application as the UI becomes unresponsive.

All these factors create a need for asynchronous communication between the microservices, and Message Based Systems can help. In such a system, a new component called an intermediary is introduced to facilitate communication between two applications/services instead of directly establishing communication between them. This middle component is known as Message Broker.

Message Brokers aid in decoupling applications as they are not directly interacting with each other. Each application only needs to adhere to the data format of this Message Broker, and it does not need to handle the nuances of interacting with different services. Therefore, it simplifies incorporating heterogeneous applications in the system.

Message Brokers provide the ability to que messages, allowing web servers to respond to requests quickly instead of being forced to perform resource-heavy procedures on the spot that may delay response time. Message queueing is also good when you want to distribute a message to multiple consumers or to balance loads between workers.

RabbitMQ

RabbitMQ is an extremely popular open-source Message Broker used for building message-based systems. Although RabbitMQ supports multiple protocols, the most commonly used is AMQP.

AMQP 0–9–1 (Advanced Message Queuing Protocol) is a messaging protocol that enables conforming client applications to communicate with conforming messaging middleware brokers. It’s an application layer protocol that transmits data in binary format. In this application, data is sent as frames.

Key Concepts

● Producer: Application that sends the messages.

● Consumer: Application that receives the messages.

● Queue: Stores messages that are consumed by applications

● Connection: A TCP connection between your application and the RabbitMQ broker.

● Channel: Lightweight connections that share a single TCP connection. Publishing or or consuming messages from a queue is done over a channel.

● Exchange: Receives messages from producers and pushes them to queues depending on rules defined by the exchange type. A queue must be bound to at least one exchange to receive messages.

● Binding: Bindings are rules that exchanges use (among other things) to route messages to queues.

● Routing key: A key that the exchange uses to decide how to route the message to queues. Think of the routing key as an address for the message.

● Users: It is possible to connect to RabbitMQ with a given username and password. Users can be assigned permissions such as rights to read, write, and configure privileges within the instance. Users can also be assigned permissions for specific virtual hosts.

● Vhost, virtual host: Virtual hosts provide logical grouping and separation of resources. Users can have different permissions to different vhost(s), and queues and exchanges can be created so they only exist in one vhost.

Message Flow in RabbitMQ

1. Producer publishes messages to exchange via a channel established between them at the time of application startup.
2. Exchange receives the message and finds appropriate bindings based on message attributes and exchange types.
3. Selected binding is then used to route messages to intended queues.
4. The message stays in the queue until handled by the consumer.
5. Consumers receive the messages using channels established usually at application startup.

Exchange Types

Exchanges are entities that receive messages. Exchanges take a message and route it into zero or more queues. The routing algorithm used depends on the exchange type and rules called bindings.

There are four types of exchanges.

● Direct (Default): The message is routed to the queues whose binding key matches the message’s routing key. It is primarily used for unicast routing of messages.

● Fanout: It routes messages to all the queues bound to it, and the routing key is ignored.

● Topic: The topic exchange does a wildcard match between the routing key and the routing pattern specified in the binding.

● Header: The header exchanges use the message header attributes for routing.

Consumer Acknowledgements and Publisher Confirms

Since messages sent over the network are not guaranteed to reach the destination, RabbitMQ provides a delivery and processing confirmation mechanism. Delivery processing confirmation from the consumer to the broker is known as consumer acknowledgment. Broker confirmation to publish that message is received is known as publisher confirms.

Both can be positive and negative. Positive acknowledgment from consumers means a message is successfully processed, and negative means failures in processing. The same thing applies in case the publisher confirms.

RabbitMQ supports the acknowledgment of multiple messages at once.

Prefetching Messages

Since message delivery and acknowledgment are both done asynchronously, there can be more than one message “in flight” on a channel at any moment. This means we have a sliding window of deliveries that are unacknowledged.

For most consumers, it makes sense to limit the size of this window to avoid the unbounded buffer (heap) growth problem on the consumer end. This is done by setting a “prefetch count” value. The value defines the maximum number of unacknowledged deliveries permitted on a channel. When the number reaches the configured count, RabbitMQ will stop delivering more messages on the channel until at least one of the outstanding ones is acknowledged. This can be used as a simple load-balancing technique or to improve throughput if messages tend to be published in batches. In general, increasing prefetch will improve the message delivery rate to consumers.

Best Practices

· Make sure your queue size remains small. Having a large queue puts a heavy load on RAM usage. To free up RAM, RabbitMQ flushes messages to disk, thereby adversely impacting the queuing speed and performance of the broker.

● Limit the queue size by either setting TTL for the message or the max length of the queue. It should be used in cases where consumers can bear message loss.

● Make sure to do message processing at the consumer end asynchronously if possible and immediately acknowledge back. This impacts the queue depth. If your application’s message processing time is high, you must go for asynchronous processing. In message-critical applications like the payments system, you should acknowledge only after message processing so unprocessed messages don’t go missing. The faster you acknowledge back, the higher the throughput.

● Reuse TCP connections and use channels to multiplex connection between threads. Ideally, each process should open a single connection and use multiple channels in that connection for different threads in your application.

● Limit the number of connections. Many connections will generate a lot of metrics which increases CPU consumption. If connections can’t be reduced, increase the statistics collection interval.

● Don’t open and close connections frequently. This may result in higher latency as multiple TCP packets will be sent and received. Connections should be long-lived.

● If your application can’t afford message loss, your queue should be declared durable, and messages should be sent with persistent delivery mode.

● If the producer and consumer lie within a single application, we should use separate connections for both to remain isolated from each other’s flow control.

● Prefetch limits should be set with caution. Too low a value can hurt queue performance, and too high a value will keep some consumers idle most of the time. A simple calculation to get an estimation would be to divide the total round trip time by the processing time for each message.

Conclusion

Developing a message-based application using RabbitMQ facilitates asynchronous communication between multiple confirming parties. It has a wide set of configuration controls that can be fine-tuned based on your requirements. Some common use cases for RabbitMQ are multiplayer games, group chats, and background processing of tasks. To provide monitoring capabilities, RabbitMQ comes with a Management UI plugin that helps track useful metrics and provides useful insights into the system.

Note that this blog only focuses on the basic concepts of RabbitMQ and the best practices you should use while developing an application. You can always refer to official documentation for further details.

About the Author

Anubhav Jain is a Senior Software Development Engineer at Clearwater Analytics, with over 11 years experience in Software Engineering . He has extensive experience in working on large scale distributed applications and helping teams migrate to cloud.
In his free time , he loves to play cricket and learn new technologies.