The Stuff That Every Developer Should Know About Message Queues

Photo by Joanna Kosinska on Unsplash

Whether you are a novice or a seasoned IT professional, you’ll get to work with a message queue someday. This article is an attempt to collect everything you need to know about message queues for designing and building asynchronous, scalable, and reliable solutions with message queues.

A visual version of this article is available as a YouTube video as well. Check that out here if you are interested.

The basics

A queue

A queue is a data structure that mimics a real word queue. For example, you can think of a group of people lined up to receive something or vehicles queued at a toll gate on a highway. A queue has two ends, the head, and the tail, where two operations are performed on them. Items are added from the tail of the queue and removed from the head. A queue maintains a FIFO order so that the first item added to the queue will be the first one to be removed.

Items are added from the tail end. Items are removed from the head.

A message queue

A message queue is a queue full of messages, designed based on the principles explained above.

Messages are stored on the queue until they are processed and deleted. The parties who put messages into the queue are called producers while the parties who remove messages from the queue are called consumers. As far as the queue is concerned, a message can be an arbitrary sequence of bytes exchanged between producers and consumers. From a business standpoint, a message can be an e-Commerce order, user registration, or anything that represents it.

A typical enterprise messaging infrastructure consists of message producers, consumers, and a message queue communicating over a network using well-defined message formats. The message queue sits between producers and consumers, making their communication and existence decoupled. When a producer wants to communicate with a consumer, it constructs a message and puts that into the queue. The consumer keeps polling the queue for new messages and eventually receives when there are new messages. The below diagram illustrates this ecosystem in detail.

An ecosystem of a typical messaging infrastructure

There are various messaging standards and protocols used by producers and consumers to communicate with message queues. These messaging protocols vary with the type of message queue implementation and use case you want to build with that. For example, JMS and AMQP are widely used for enterprise messaging needs. Lightweight protocols like MQTT and STOMP are widely popular for IoT workloads.

The inner workings of a message queue

A message queue is a distributed system, made of multiple servers called brokers. These brokers get together and form a cluster, thus making it highly available, scalable, and reliable.

Once a message is received, the message queue redundantly stores the message across multiple brokers for durability. This means there can be multiple copies of the same message, scattered across multiple brokers. In the event of a failure of a broker, its messages can be recovered from the remaining brokers in the cluster.

A message queue redundantly stores messages across multiple brokers. A,B and C are messages.

Lifecycle of a message in a message queue

A journey of a message starts from the producer and ends at the consumer. The following scenario describes the lifecycle of a message in a queue, from creation to deletion.

  1. Producer sends a message to the queue.

The producer constructs a message and sends that to the message queue. The consumer may or may not available to consume the message at this time. So the queue keeps the message until a consumer is available

2. Consumer retrieves the message

The message is temporary hidden from other consumers while it is being consumed.

When a consumer is ready to process the message, it consumes the message from the queue. But the queue does not delete the message immediately. It stays there until the consumer completes the processing. In the meantime, the queue temporarily locks the message to prevent the same message from being read by another consumer.

3. Consumer processes the message and deletes it from the queue

Consumer deletes the message from the queue after processing.

After processing, the consumer deletes the message from the queue to prevent the message from being consumed again by another consumer.

An important point to highlight here is that any number of producers can send messages to the same queue. But the queue guarantees that each message is processed by a single consumer.

Message exchange patterns

Producers and consumers use the following patterns to exchange messages with each other through a message queue.

1. One-way messaging

This is also known as the point-to-point messaging style. The producer simply sends a message to the queue with the expectation that a consumer will retrieve it and process it at some point.

Consumer polls the queue for new messages and ultimately receives. Here, the producer is not aware of the existence of the consumer or how the message is being processed. Also, it doesn’t wait for a response from the consumer.

2. Request/response messaging

Dedicated queues are required for the communication.

This is also known as the RPC (Remote Procedure Call) messaging style. The producer sends a message to a queue and expects a response from the consumer. If the response is not delivered within a reasonable interval, the producer does either of the following.

  1. Send the message again.
  2. Handle the situation as a timeout or failure.

This pattern usually requires a separate communications channel in the form of a dedicated message queue to which the consumer can send its response messages. The producer listens for a response on this queue.

3. Broadcast messaging

This is also known as pub/sub messaging or fanout messaging style. The producer sends a message to a queue, and multiple consumers can read the same copy of the message (consumers do not compete for messages in this scenario). This can be used to notify consumers that an event has occurred of which they should all be aware, and may be used to implement a publisher/subscriber model.

In the above diagram, a topic acts like a queue where producers send messages with metadata in the form of attributes. Each consumer can create a subscription for the topic, specifying a filter that examines the values of message attributes. Any messages sent to the topic with attribute values that match the filter are automatically forwarded to that subscription. A consumer retrieves messages from a subscription in a similar way to a queue.

Features common to message queues

Many message queue implementations today share a common set of features.

1. At-least-once processing

Once you put a message to a message queue, its infrastructure makes sure that the message is not lost and delivered to its consumers under any circumstance.

2. Exactly-once processing

A producer may send the same message twice. For example, a producer might crash after sending a message but before completing any other work it was performing. Another producer will replace the original producer and it could repeat the message. So the queue will have duplicate messages.

Most message queues have built-in support for duplicate message detection and removal. This is based on the message Ids and is known as de-duping. The message queue internally maintains records of the ids of the messages that have been delivered. When a new message arrives, the queue checks its message id against the records and drops immediately, if it had been delivered already.

This makes sure each message is delivered to its consumer exactly once (and only once). Ideally, the message processing logic should be idempotent so that, if the work performed is repeated, this repetition does not change the state of the system.

3. Message ordering

Some solutions may require that messages are processed in a specific order. For example, financial services and e-commerce customers, and those who use messages to update database tables.

Message queues provide best-effort ordering which ensures that messages are delivered in the same order as they’re sent. Most message queues offer two flavours for message ordering.

Standard queues

The order of messages may not be guaranteed. But they offer high throughput.

FIFO queues

Messages are delivered in the order in which they are sent. Offers a limited throughput.

Choose standard vs FIFO queues based on your need. A scenario like a user avatar image resizing or user signup information processing doesn’t need the order of incoming messages. But having a higher message throughput is critical. Conversely, message order will be critical when processing inventory adjustments or ledger transactions.

Benefits of having a message queue

In any enterprise architecture, a message queue plays a critical role and adds several benefits to the overall application architecture. Let’s discuss them in detail.

A message queue enables reliable messaging

A message queue enables reliably exchanging messages among application components. Even though consumers fail, the queue will deliver the message to another consumer.

A queue makes sure that a message is consumed by only one consumer. When a message is being consumed by a consumer, the queue puts a lock on the message so that other consumers can’t consume it. This lock has a time limit. If the consumer fails to delete the message during the locked period, the lock will timeout and the queue lets other consumers consume it.

If a consumer fails while processing a message, the queue will release its lock on the message so that another consumer can retrieve it and processes it. Hence, a message queue ensures that a message is processed at-least-once.

A message queue decouples your workloads

A message queue provides temporal decoupling so that the producer and consumer don’t have to run concurrently. A producer can send a message to the message queue regardless of the availability of the consumer. Conversely, the consumer isn’t restricted by the producer’s availability.

Moreover, this decoupled nature introduces an asynchrony to the architecture. After sending a message, the producer doesn’t have to wait until the consumer completes it. This makes the producer responsive. Also, producers and consumers can go offline for maintenance tasks, they can be upgraded independently.

Decoupled nature promotes an evolving architecture as well. Adding new producers and consumers to a message queue is straightforward and makes less impact on the existing application architecture.

A message queue enables scalable message processing

A message queue helps to scale your application by distributing the load across multiple consumers (load balancing). Also, it will act as a buffer to smooth any spikes in the traffic (load leveling).

Load balancing

A message queue distributes the processing across consumers and improves the overall throughput.

Pool of consumers competing for messages. This will drain the queue faster.

For example, producers may send a large number of messages to a queue that is serviced by many consumers. Consumers can be added dynamically to scale out the system if the queue length grows, and they can be removed when the queue has drained. Consumers can run on different servers to spread the load.

Load leveling

A message queue can protect consumers from sudden bursts of messages sent by multiple producers.

The message queue acts as a buffer and smoothen the spikes in the incoming traffic.

If the above example is considered, a message queue acts as a buffer and allows consumers to gradually drain messages at their own pace without stressing the system. This eliminates the need of adding more consumers to handle the work, which sometimes time-consuming and expensive.

A message queue enables cross-platform integration

Message queues enable integrating components running on different platforms, and that is built by using different programming languages and technologies.

A message queue as an integration hub.

These heterogeneous components can be configured to produce and consume messages from a message queue to accomplish a particular business use case. For example, on Nodejs order API can put orders in a queue, hoping that they’ll be processed by an order processor written in Java.

Some advanced scenarios

Apart from the basic characteristics discussed above, there are some advanced use cases associated with message queue implementations which can be important when designing messaging solutions.

Dead Letter Queue (DLQ)

The DLQ is a special queue built into the message queue which holds the messages that couldn’t be delivered or processed.

The message queue itself or any consumer can put messages to the DLQ. For example, a consumer can put a message into DLQ if it fails to process the message after several attempts. The DLQ keeps the messages until they are retrieved from the queue, which is a manual operation most of the time.

Message expiration (Time-bound message processing)

A message might have a limited lifetime, and if it is not processed within this period it might no longer be relevant and should be discarded.

For example, a message must be processed within 2 hours. Otherwise, it should be expired and discarded. Message queues usually put expired messages into the DLQ.

Message scheduling

A message will be visible on the message queue only after a specific date and time. The message should not be available to a consumer until this time.

For example, the message will be invisible for a period set by the producer. So consumers can’t see the message. When that period elapses, the message will be ready for consumption.

Poison messages

A poison message is a message that cannot be handled because it’s malformed or contains unexpected information.

For example, a consumer handling a message could throw an exception and fail. This will cause the message to be returned to the queue again to be handled by another consumer. The new consumer will repeat the same logic and returns the message to the queue. This will make an indefinite loop, wasting CPU cycles and hindering the handling of other valid messages in the queue.

Hence, it is important to detect and remove poison messages as early as possible. Usually, poison messages are put into the dead letter queue.

Priority messaging

A message can be assigned a priority to determine where in the queue the message is added, to make sure that higher priority messages get bumped to the front of the line and processed first.

Messages are ordered according to the priory set by the producer.

In this example, the producer assigns the priority level 1 to the message so that it’ll end up in the front end of the queue by circumventing the messages with low priority.

Summary

As a developer or an architect, you’ll have to deal with a message queue at least once in the lifetime. Hence, it is important to understand the fundamentals of message queues, what they are capable of, and when to use them.

Let’s recap our discussion so far.

  • Choose a message queue if you are building an application that needs a guaranteed end-to-end delivery of messages.
  • If your application architecture evolves over time, introduce a message queue to promote indirection, loose coupling, and asynchrony.
  • Choose between throughput vs message ordering on a case by case basis. Sometimes, you don’t need FIFO.
  • There are many message queue implementations in the market to choose from. Choose the one that best fits your needs. Don’t go after the hype.

--

--

--

EdU is a place where you can find quality content on event streaming, real-time analytics, and modern data architectures

Recommended from Medium

FacileThings Goals for 2018

Day 12 Phase 1 complete

Modern C++ in Advent of Code: Day4

Introduction to Agile and Lean Practices for Data-Driven Product Development

Upgrading OutSystems Platform to version 11

Deploy Flask App to the Apache2 Web Server

Switch from sqlite3 to PostgreSQL

Amazon EC2

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Dunith Dhanushka

Dunith Dhanushka

Editor of Event-driven Utopia(eventdrivenutopia.com). Technologist, Writer, Developer Advocate at StarTree. Event-driven Architecture, DataInMotion

More from Medium

Ruling the Event-Driven Architecture with RabbitMQ

Microservices Cross-Cutting Concerns Design Patterns

Microservices: The Saga Pattern

DESIGN PATTERN FOR MICROSERVICES — AGGREGATOR PATTERN