The Stuff That Every Developer Should Know About Message Queues
Whether you are a novice or a seasoned IT professional, you’ll get to work with a message queue someday. This article is an attempt to collect everything you need to know about message queues for designing and building asynchronous, scalable, and reliable solutions with message queues.
A visual version of this article is available as a YouTube video as well. Check that out here if you are interested.
A queue is a data structure that mimics a real word queue. For example, you can think of a group of people lined up to receive something or vehicles queued at a toll gate on a highway. A queue has two ends, the head, and the tail, where two operations are performed on them. Items are added from the tail of the queue and removed from the head. A queue maintains a FIFO order so that the first item added to the queue will be the first one to be removed.
A message queue
A message queue is a queue full of messages, designed based on the principles explained above.
Messages are stored on the queue until they are processed and deleted. The parties who put messages into the queue are called producers while the parties who remove messages from the queue are called consumers. As far as the queue is concerned, a message can be an arbitrary sequence of bytes exchanged between producers and consumers. From a business standpoint, a message can be an e-Commerce order, user registration, or anything that represents it.
A typical enterprise messaging infrastructure consists of message producers, consumers, and a message queue communicating over a network using well-defined message formats. The message queue sits between producers and consumers, making their communication and existence decoupled. When a producer wants to communicate with a consumer, it constructs a message and puts that into the queue. The consumer keeps polling the queue for new messages and eventually receives when there are new messages. The below diagram illustrates this ecosystem in detail.
There are various messaging standards and protocols used by producers and consumers to communicate with message queues. These messaging protocols vary with the type of message queue implementation and use case you want to build with that. For example, JMS and AMQP are widely used for enterprise messaging needs. Lightweight protocols like MQTT and STOMP are widely popular for IoT workloads.
The inner workings of a message queue
A message queue is a distributed system, made of multiple servers called brokers. These brokers get together and form a cluster, thus making it highly available, scalable, and reliable.
Once a message is received, the message queue redundantly stores the message across multiple brokers for durability. This means there can be multiple copies of the same message, scattered across multiple brokers. In the event of a failure of a broker, its messages can be recovered from the remaining brokers in the cluster.
Lifecycle of a message in a message queue
A journey of a message starts from the producer and ends at the consumer. The following scenario describes the lifecycle of a message in a queue, from creation to deletion.
- Producer sends a message to the queue.
The producer constructs a message and sends that to the message queue. The consumer may or may not available to consume the message at this time. So the queue keeps the message until a consumer is available
2. Consumer retrieves the message
When a consumer is ready to process the message, it consumes the message from the queue. But the queue does not delete the message immediately. It stays there until the consumer completes the processing. In the meantime, the queue temporarily locks the message to prevent the same message from being read by another consumer.
3. Consumer processes the message and deletes it from the queue
After processing, the consumer deletes the message from the queue to prevent the message from being consumed again by another consumer.
An important point to highlight here is that any number of producers can send messages to the same queue. But the queue guarantees that each message is processed by a single consumer.
Message exchange patterns
Producers and consumers use the following patterns to exchange messages with each other through a message queue.
1. One-way messaging
This is also known as the point-to-point messaging style. The producer simply sends a message to the queue with the expectation that a consumer will retrieve it and process it at some point.
Consumer polls the queue for new messages and ultimately receives. Here, the producer is not aware of the existence of the consumer or how the message is being processed. Also, it doesn’t wait for a response from the consumer.
2. Request/response messaging
This is also known as the RPC (Remote Procedure Call) messaging style. The producer sends a message to a queue and expects a response from the consumer. If the response is not delivered within a reasonable interval, the producer does either of the following.
- Send the message again.
- Handle the situation as a timeout or failure.
This pattern usually requires a separate communications channel in the form of a dedicated message queue to which the consumer can send its response messages. The producer listens for a response on this queue.
3. Broadcast messaging
This is also known as pub/sub messaging or fanout messaging style. The producer sends a message to a queue, and multiple consumers can read the same copy of the message (consumers do not compete for messages in this scenario). This can be used to notify consumers that an event has occurred of which they should all be aware, and may be used to implement a publisher/subscriber model.
In the above diagram, a topic acts like a queue where producers send messages with metadata in the form of attributes. Each consumer can create a subscription for the topic, specifying a filter that examines the values of message attributes. Any messages sent to the topic with attribute values that match the filter are automatically forwarded to that subscription. A consumer retrieves messages from a subscription in a similar way to a queue.
Features common to message queues
Many message queue implementations today share a common set of features.
1. At-least-once processing
Once you put a message to a message queue, its infrastructure makes sure that the message is not lost and delivered to its consumers under any circumstance.
2. Exactly-once processing
A producer may send the same message twice. For example, a producer might crash after sending a message but before completing any other work it was performing. Another producer will replace the original producer and it could repeat the message. So the queue will have duplicate messages.
Most message queues have built-in support for duplicate message detection and removal. This is based on the message Ids and is known as de-duping. The message queue internally maintains records of the ids of the messages that have been delivered. When a new message arrives, the queue checks its message id against the records and drops immediately, if it had been delivered already.
This makes sure each message is delivered to its consumer exactly once (and only once). Ideally, the message processing logic should be idempotent so that, if the work performed is repeated, this repetition does not change the state of the system.
3. Message ordering
Some solutions may require that messages are processed in a specific order. For example, financial services and e-commerce customers, and those who use messages to update database tables.
Message queues provide best-effort ordering which ensures that messages are delivered in the same order as they’re sent. Most message queues offer two flavours for message ordering.
The order of messages may not be guaranteed. But they offer high throughput.
Messages are delivered in the order in which they are sent. Offers a limited throughput.
Choose standard vs FIFO queues based on your need. A scenario like a user avatar image resizing or user signup information processing doesn’t need the order of incoming messages. But having a higher message throughput is critical. Conversely, message order will be critical when processing inventory adjustments or ledger transactions.
Benefits of having a message queue
In any enterprise architecture, a message queue plays a critical role and adds several benefits to the overall application architecture. Let’s discuss them in detail.
A message queue enables reliable messaging
A message queue enables reliably exchanging messages among application components. Even though consumers fail, the queue will deliver the message to another consumer.
A queue makes sure that a message is consumed by only one consumer. When a message is being consumed by a consumer, the queue puts a lock on the message so that other consumers can’t consume it. This lock has a time limit. If the consumer fails to delete the message during the locked period, the lock will timeout and the queue lets other consumers consume it.
If a consumer fails while processing a message, the queue will release its lock on the message so that another consumer can retrieve it and processes it. Hence, a message queue ensures that a message is processed at-least-once.
A message queue decouples your workloads
A message queue provides temporal decoupling so that the producer and consumer don’t have to run concurrently. A producer can send a message to the message queue regardless of the availability of the consumer. Conversely, the consumer isn’t restricted by the producer’s availability.
Moreover, this decoupled nature introduces an asynchrony to the architecture. After sending a message, the producer doesn’t have to wait until the consumer completes it. This makes the producer responsive. Also, producers and consumers can go offline for maintenance tasks, they can be upgraded independently.
Decoupled nature promotes an evolving architecture as well. Adding new producers and consumers to a message queue is straightforward and makes less impact on the existing application architecture.
A message queue enables scalable message processing
A message queue helps to scale your application by distributing the load across multiple consumers (load balancing). Also, it will act as a buffer to smooth any spikes in the traffic (load leveling).
A message queue distributes the processing across consumers and improves the overall throughput.
For example, producers may send a large number of messages to a queue that is serviced by many consumers. Consumers can be added dynamically to scale out the system if the queue length grows, and they can be removed when the queue has drained. Consumers can run on different servers to spread the load.
A message queue can protect consumers from sudden bursts of messages sent by multiple producers.
If the above example is considered, a message queue acts as a buffer and allows consumers to gradually drain messages at their own pace without stressing the system. This eliminates the need of adding more consumers to handle the work, which sometimes time-consuming and expensive.
A message queue enables cross-platform integration
Message queues enable integrating components running on different platforms, and that is built by using different programming languages and technologies.
These heterogeneous components can be configured to produce and consume messages from a message queue to accomplish a particular business use case. For example, on Nodejs order API can put orders in a queue, hoping that they’ll be processed by an order processor written in Java.
Some advanced scenarios
Apart from the basic characteristics discussed above, there are some advanced use cases associated with message queue implementations which can be important when designing messaging solutions.
Dead Letter Queue (DLQ)
The DLQ is a special queue built into the message queue which holds the messages that couldn’t be delivered or processed.
The message queue itself or any consumer can put messages to the DLQ. For example, a consumer can put a message into DLQ if it fails to process the message after several attempts. The DLQ keeps the messages until they are retrieved from the queue, which is a manual operation most of the time.
Message expiration (Time-bound message processing)
A message might have a limited lifetime, and if it is not processed within this period it might no longer be relevant and should be discarded.
For example, a message must be processed within 2 hours. Otherwise, it should be expired and discarded. Message queues usually put expired messages into the DLQ.
A message will be visible on the message queue only after a specific date and time. The message should not be available to a consumer until this time.
For example, the message will be invisible for a period set by the producer. So consumers can’t see the message. When that period elapses, the message will be ready for consumption.
A poison message is a message that cannot be handled because it’s malformed or contains unexpected information.
For example, a consumer handling a message could throw an exception and fail. This will cause the message to be returned to the queue again to be handled by another consumer. The new consumer will repeat the same logic and returns the message to the queue. This will make an indefinite loop, wasting CPU cycles and hindering the handling of other valid messages in the queue.
Hence, it is important to detect and remove poison messages as early as possible. Usually, poison messages are put into the dead letter queue.
A message can be assigned a priority to determine where in the queue the message is added, to make sure that higher priority messages get bumped to the front of the line and processed first.
In this example, the producer assigns the priority level 1 to the message so that it’ll end up in the front end of the queue by circumventing the messages with low priority.
As a developer or an architect, you’ll have to deal with a message queue at least once in the lifetime. Hence, it is important to understand the fundamentals of message queues, what they are capable of, and when to use them.
Let’s recap our discussion so far.
- Choose a message queue if you are building an application that needs a guaranteed end-to-end delivery of messages.
- If your application architecture evolves over time, introduce a message queue to promote indirection, loose coupling, and asynchrony.
- Choose between throughput vs message ordering on a case by case basis. Sometimes, you don’t need FIFO.
- There are many message queue implementations in the market to choose from. Choose the one that best fits your needs. Don’t go after the hype.