Unleashing the Power of Event-Driven Messaging Architecture: A Deep Dive into Pub-Sub Model

7 min readJun 29, 2023

In today’s fast-paced digital landscape, businesses strive to deliver real-time responsiveness and scalability in their systems. This is where event-driven messaging architecture comes into play. By leveraging event-driven principles, businesses can achieve seamless communication, decoupling of components, and efficient processing of events.

In this article, we will delve into the concept of event-driven messaging architecture, provide real-world examples, explore its benefits for businesses, and demonstrate how to implement it using AWS SNS, SQS, Lambda, and DLQ. Additionally, we will explore the pub-sub model, a key component of event-driven architectures.

Pub-Sub Model

The publish-subscribe (pub-sub) model is a messaging pattern that enables the decoupling of components in an event-driven architecture.

In this model, publishers are responsible for producing events or messages, while subscribers register their interest in specific topics or event types. When a publisher sends an event to a topic, the event is distributed to all subscribed subscribers, allowing for asynchronous and scalable communication.

Case Study : LinkedIn Notifications

In platforms like LinkedIn, the pub-sub model is commonly used to enable users to receive real-time notifications based on their interests. Let’s try to understand each steps :

Publisher — LinkedIn Platform

LinkedIn serves as the publisher, responsible for generating events when specific actions occur, such as profile views, connection requests, job opportunities, or updates from connections.

Subscribers — LinkedIn Users

Users on LinkedIn are the subscribers who express their interest in receiving notifications related to activities happening within their professional network.

Topics — User Activities

These can be user activities like : connection requests, profile views, job opportunities, skill endorsements, etc.

Workflow

When a relevant activity occurs, such as someone sending a connection request, LinkedIn generates an event associated with that activity.
The event is published to the corresponding topic related to the specific type of activity.
Users who have subscribed to those topics receive notifications about the events through various channels, including the LinkedIn website, mobile app push notifications, or email notifications.
The platform delivers the notifications to the subscribers in real-time.
Subscribers can interact with the notifications, such as accepting connection requests, exploring job opportunities, or engaging with updates from connections, directly from the notification.

This pattern ensures a seamless and engaging user experience, allowing users to interact with content as soon as it is published by their subscribed accounts.

Understanding Event-Driven Messaging Architecture using AWS

Event-driven messaging architecture revolves around the exchange of events among different components of a system. Events represent significant occurrences or changes in the system, such as user interactions, system events, or data updates.

Rather than relying on tightly coupled systems, event-driven architectures enable loose coupling by ensuring that components are notified of events and can react accordingly.

Although there are many other technologies available for building event-driven messaging infrastructure, we have chosen to leverage several AWS services for this task. AWS provides a comprehensive suite of services that offer numerous benefits for building scalable and robust event-driven architectures. Let’s explore the key components involved:

SNS (Simple Notification Service):

A fully managed pub-sub messaging service.
Provides message publishing and delivery to subscribers.
Supports multiple communication protocols and delivery mechanisms, such as HTTP, email, SMS, and mobile push notifications.
Allows topic-based event routing and fanout, where a single event can be sent to multiple subscribers.

2. SQS (Simple Queue Service):

A fully managed message queuing service.
Enables decoupling of components and asynchronous communication between services.
Provides reliable message storage and delivery.
Offers two types of queues: standard queues for high-throughput messaging and FIFO (First-In-First-Out) queues for ordered messaging.

3. AWS Lambda:

A serverless compute service.
Allows running code without provisioning or managing servers.
Supports event-driven architecture by enabling functions to respond to events from various sources.
Automatically scales based on the incoming request volume.
Supports a wide range of programming languages.

4. DLQ (Dead Letter Queue):

A mechanism to handle and store failed or unprocessed messages.
Captures messages that couldn’t be successfully processed by the consumer.
Provides an opportunity for error analysis and debugging.
Allows for implementing retry mechanisms or manual intervention for problematic messages.

Event flow :

An event occurs in a service (such as a change in a db table).
The service publishes a message to an SNS topic.
The SNS topic fans-out the message to the subscribed SQS queues.
The SQS queues store the message until an consumer retrieves it for processing.
The consumer then polls out the message from queue, if the message is successfully processed, it deletes it from the queue.
If the consumer fails to process the message, pollers deletes the message from queue and the message is sent to DLQ.
The DLQ stores the messages until a consumer retrieves it for analysis or reprocessing.

Fanout architecture

Fanout is a technique often used in pub-sub architectures to achieve message distribution to multiple subscribers efficiently. In a fanout architecture, a message is delivered to multiple recipients without the publisher being aware of the subscribers or their specific identities. It involves replicating the message and sending it to each subscriber independently.

Customer order processing

Let’s take an example of order processing done by an e-commerce business. We will understand how businesses can handle a large number of customer orders and process them efficiently.

Customer places an order through the e-commerce website or application.
Order details are sent as an event to SNS.
SNS publishes the “Order Placed Event” to a topic.
The topic is configured to send the event to SQS queues.
Each SQS queue represents a different component or service for processing specific aspects of the order.
Lambda functions are triggered by their respective SQS queues.
Multiple instances of the Lambda function can run in parallel.
Lambda function retrieves order details and performs business logic.
- One Lambda function may handle inventory management and update the stock levels.
- Another Lambda function may handle datalake ingestion.
- Yet another Lambda function may handle shipping logistics and generate shipping labels.
Lambda function generates a response, such as order confirmation or error message.
Successful order processing removes the event from the SQS queue.

Parallel Processing and Scalability:

With the fanout architecture, each Lambda function subscribed to an SQS queue independently processes the order event.
The fanout pattern allows for parallel processing of the order across multiple services, enhancing scalability and performance.
As the load increases, AWS automatically scales the number of Lambda instances to handle the incoming order events efficiently.

Conclusion

In conclusion, event-driven messaging architecture, coupled with the power of AWS services such as SNS, SQS, and Lambda, empowers businesses to achieve seamless communication, scalability, and efficient processing of events. The pub-sub model allows for decoupling of components and asynchronous communication, ensuring that components are notified of events and can react accordingly. By leveraging AWS services, businesses can benefit from the following advantages:

Fully managed services: AWS provides fully managed services like SNS, SQS, and Lambda, which reduce the operational burden of setting up and managing infrastructure.
Scalability: AWS services automatically scale based on the incoming request volume, allowing businesses to handle a large number of events efficiently.
Reliability: SQS provides reliable message storage and delivery, ensuring that messages are not lost in transit.
Flexibility: Lambda supports a wide range of programming languages, giving businesses the freedom to write functions in their preferred language.
Error handling: DLQ (Dead Letter Queue) provides a mechanism to capture and store failed or unprocessed messages, enabling error analysis, debugging, and implementing retry mechanisms.