Exploring Message Streaming Architectures: A Guide to Pub/Sub, Queues

In today’s interconnected digital landscape, real-time data processing is crucial for powering various applications, from e-commerce platforms to IoT devices. Message streaming architectures play a pivotal role in enabling real-time communication and data processing. I’ll delve into two fundamental messaging patterns: Publish/Subscribe (Pub/Sub) and Message Queuing, exploring their architectures, and use cases.

Pub/Sub

Definition: The Pub/Sub (Publisher/Subscriber) model is a messaging pattern used in software architecture to facilitate asynchronous communication between different components or systems. In this model, publishers produce messages that are then consumed by subscribers, mediated by a message broker.

Pub/Sub

Pub/Sub messaging allows to creation of decoupled applications easily with a reliable communication method and enables users to create Event-driven architectures. Event-driven architecture (EDA) is a software design pattern that enables a system to detect events (such as a transaction or site visit) and act on them in real-time or near real-time. This pattern replaces the traditional Request/Response architecture where services would have to wait for a reply before they could move on to the next task.

In publish-subscribe, the sender of the message doesn’t know anything about receivers. The message is being sent to the topic. After that, it’s distributed among all endpoints subscribed to that topic. It can be useful e.g. for implementing a notifications mechanism or distributing independent tasks.

Components of Pub/Sub Architecture

• Publisher: The Publisher is responsible for creating and sending messages to the Pub/Sub system. Publishers categorize messages into topics or channels based on their content. They do not need to know the identity of the subscribers.
• Subscriber: The Subscriber is a recipient of messages in the Pub/Sub system. Subscribers express interest in receiving messages from specific topics. They do not need to know the identity of the publishers. Subscribers receive messages from topics to which they are subscribed.
• Topic: A Topic is a named channel or category to which messages are published. Publishers send messages to specific topics, and subscribers can subscribe to one or more topics to receive messages of interest. Topics help categorize messages and enable targeted message delivery to interested subscribers.
• Message Broker: The Message Broker is an intermediary component that manages the routing of messages between publishers and subscribers. It receives messages from publishers and forwards them to subscribers based on their subscriptions. The Message Broker ensures that messages are delivered to the correct subscribers and can provide additional features such as message persistence, scalability, and reliability.
• Message: A Message is the unit of data exchanged between publishers and subscribers in the Pub/Sub system. Messages can contain any type of data, such as text, JSON, or binary data. Publishers create messages and send them to the Pub/Sub system, and subscribers receive and process these messages.
• Subscription: A Subscription represents a connection between a subscriber and a topic. Subscriptions define which messages a subscriber will receive based on the topics to which it is subscribed. Subscriptions can have different configurations, such as message delivery guarantees (e.g., at-most-once, at-least-once) and acknowledgment mechanisms.

Key characteristics of Pub/Sub

• Decoupling of components: Publishers and subscribers are loosely coupled, allowing for scalable and resilient systems.
• Asynchronous communication: Subscribers receive messages as they become available, enabling real-time processing.
• Scalability: Pub/Sub systems are designed to handle large volumes of messages and scale horizontally to accommodate growing workloads.
• Event-Driven Architecture: Pub/Sub is well-suited for event-driven architectures. Publishers can emit events and subscribers can react to these events without tight coupling between them.
• Dynamic Subscriptions: Pub/Sub allows for dynamic subscriptions. Subscribers can subscribe to different topics or classes of messages at runtime, which adds flexibility to the system.

Real-World Example of Pub/Sub Architecture (Tweets)

• Publishers: Users post tweets, which are published to the Twitter platform.
• Subscribers: Followers of a user subscribe to their tweets to receive updates.
• Topics: Each user’s tweets can be considered a topic, and subscribers receive updates for the topics they are interested in.
• Message Broker: Twitter’s backend infrastructure acts as the message broker, routing tweets from publishers to subscribers.
• Message: Each tweet is a message that is published by the user and received by their followers.

Popular Pub/Sub systems

1. Apache Kafka
2. Amazon Kinesis
3. Google Cloud Pub/Sub
4. Azure Event Hubs
5. Many more….

Message Queuing

Definition: A message queues is a form of service-to-service communication that facilitates asynchronous communication. It asynchronously receives messages from producers and sends them to consumers.

A Message Queue is a form of communication and data transfer mechanism used in computer science and system design. It functions as a temporary storage and routing system for messages exchanged between different components, applications, or systems within a larger software architecture.

With a message queue, we can efficiently support a significantly higher volume of concurrent messages, messages are pushed in real-time instead of periodically polled, messages are automatically cleaned up after being received and we don’t need to worry about any deadlocks or race conditions.

Message Queuing

Components of a Message Queues Architecture

• Message Producer: The message producer is responsible for creating and sending messages to the message queue. This can be any application or component within a system that generates data to be shared.
• Message Queue: The message queue is a data structure or service that stores and manages the messages until they are consumed by the message consumers. It acts as a buffer or intermediary between producers and consumers.
• Message Consumer: The message consumer is responsible for retrieving and processing messages from the message queue. Multiple consumers can read messages concurrently from the queue.
• Message Broker (Optional): In some message queue systems, a message broker acts as an intermediary between producers and consumers, providing additional functionality like message routing, filtering, and message transformation.

Key characteristics of Message Queues

• Asynchronous Communication: Message queues allow applications to send and receive messages without having to wait for a response. This is essential for building scalable and reliable systems.
• Decoupling: Message queues decouple applications from each other, allowing them to be developed independently. This makes systems more flexible and easier to maintain.
• Scalability: Message queues can be scaled to handle large volumes of messages by adding more servers. This makes them ideal for high-traffic applications.
• Reliability: Message queues can be designed to be highly reliable, with features such as message persistence, retries, and dead letter queues. This ensures that messages are not lost even in the event of failures.
• Workflow Management: Message queues can be used to implement complex workflows, such as order processing and payment processing. This can help improve the efficiency and accuracy of these processes.

Real-World Example of Message Queues Architecture (Email)

A simple example of a message queue is an email inbox. When you send an email, it is placed in the recipient’s inbox. The recipient can then read the email at their convenience. This email inbox acts as a buffer between the sender and the recipient, decoupling them from each other.

Popular Message Queues systems

1. RabbitMQ
2. Amazon Simple Queue Service (SQS)
3. Azure Service Bus
4. Many more….

Message streaming architectures are essential building blocks for enabling real-time communication and data processing in modern applications. By understanding the characteristics, use cases, and implementation considerations of Pub/Sub and Message Queuing architectures, developers can design resilient, scalable, and future-proof messaging systems to meet the demands of today’s interconnected world.

That is all for this post, I’m excited to see you in the next couple of posts with the following topics.

1. Distributed Systems
2. Kafka
3. RebitMQ
4. Many more….

If you found this blog post useful then clap, comment, and follow.

🤝 Let’s connect on LinkedIn: Raheel Butt