Choosing the Right Messaging Approach: Event Streaming or Message Queues?

Choosing the right messaging technology is essential for building a scalable and responsive architecture. In this article, you will uncover insights into the key features of both event streaming and message queues. This exploration aims to empower you to make informed decisions, assisting in identifying the most suitable fit for your requirements.

Summary

The intention behind our messages becomes crucial when selecting the most suitable messaging technology. In the realm of asynchronous architectures (e.g. event-driven architecture), we commonly encounter two message types: commands and events. Commands find their sweet spot in message queues, while events nicely integrate with event-streaming.

In message queues, a single consumer/service can action your command and the message can be promptly deleted after it’s successfully processed. More than one consumer can be added if there is a need to process the queue faster. Additionally, you can ensure the order of the commands using FIFO(First-In-First-Out) queues.

In event streaming, multiple consumers/services can subscribe to your topic/channel, staying up-to-date with real-time events. Unlike commands, events linger even after being consumed by one participant, allowing other consumers to replay the historical event data. You can think about it like watching a YouTube video and wanting to replay the first 10 minutes (and also your friends can watch the same stream regardless of what you have just consumed).

Introduction

The need for async messaging occurs when you start to distribute your services and we end up having producer and consumer services using a message broker in the middle. Achieving asynchronous messaging can be facilitated through the adoption of message queues or event-streaming solutions. In this article, we will explore the distinctive features of those two prominent messaging approaches and understand how we can make the right choice when it comes to selecting one.

Popular Message Queues

• Amazon SQS(simple queue service)
• Azure Service Bus Queue or Azure Storage Queue.
• RabbitMQ
• Apache Pulsar

Popular Event Streaming Tools/Platforms

• Apache Kafka
• Azure Event Grid
• Amazon Kinesis
• Redis Streams
• Apache Pulsar

What is the intention of your message?

In traditional request-response architecture, HTTP requests can be categorized as queries and commands. However, in asynchronous architectures, the paradigm shifts from requests to messages. These messages serve a dual purpose — they can function as notifications, providing updates about recent occurrences, or as commands, initiating specific actions. Thus, we discern two fundamental message types which are events and commands.

Events

An event is a message that signifies that a specific thing has happened or a state has changed. For instance, it could indicate that an order has been placed, a payment has been received, and so forth. Here is an event example in JSON format:

// OrderPlaced
{
  "metadata":{ // event metadata
    "id": "xbz19827759", // event id
    "source": "example-payment-service", // event source
    "domain": "payment",
    "datacontenttype": "application/json"
  },
  "data": {  // actual payload
    "orderId": "oxp3745",
    "productId": "prt1782",
    "quantity": 2,
    "customerId": "cus186493",
    "timestamp": "2023-01-03T15:20:30"
  }
}

When it comes to structuring the event data there are a couple of event types that can be used; Notification, Event-carried State Transfer (ECST), Delta Events, and Domain Events. If you decide to use events in your system I would highly recommend learning about the event types and also embracing the Event-First thinking.

Commands

A command is a message that requests the execution of a specific action or operation. For instance; it might involve tasks such as validating an email address or saving details for a new user. Here is a command example in JSON format:

// PlaceOrder
{
  "customerId": "cus591827",
  "productId": "456",
  "quantity": 2,
  "timestamp": "2023-01-01T12:34:56"
}

Events and commands can be similar in terms of the payload but as mentioned earlier they do exist for different purposes. Commands invoke a behaviour whereas events notify us that something has just happened.

Which tool is the best fit for your message?

In my experience, I typically adhere to the criteria outlined below when selecting the most suitable technology. While this approach has proven effective for my projects, it’s essential to note that the dynamics of each project vary, accompanied by unique requirements and limitations;

• When dealing with events, it is recommended to use an event streaming technology such as Apache Kafka, Azure Event Grid, or a similar solution for both production and consumption.
• For commands, the optimal choice is to use a message queue tool, such as Amazon SQS, Azure Service Bus Queues, or a similar solution, for both production and consumption.

I hope that the distinction between message types and the corresponding technologies has been clear to you. Now let’s wrap up the article with a list that highlights some of the differences between the two messaging approaches;

Event-Streaming vs Message Queues

Event-streaming:

• Consumers can use a pointer/offset to go back and forth between messages.
• Different consumers can subscribe to the same channel and get notified about the same event.
• Message ordering is not a key feature but it can be established on a partition/shard level.
• It’s also a good fit for the systems that rely on immediate action on data (e.g. fraud detection).

Message queues:

• Consumers read a message from the queue and it gets deleted after they process the message.
• High-load tasks can be distributed via horizontally scaled consumers so the consumers can share the load.
• You can enforce message ordering with FIFO queues and process the tasks in order.
• Data doesn’t have to get processed in real-time the goal is to process the messages in the desired time interval.

That’s all I have for this article, in the next one I’ll pick two messaging technologies and share an overview to explore what kind of features are added to those tools to solve our messaging needs.

Closing Thoughts

The adoption of request-response architecture has a long history, resulting in mature tooling and established best practices. However, in the dynamic realm of asynchronous architectures, there remains a need for more resources to enhance our practices. Sharing knowledge becomes invaluable in fostering improvements within this evolving landscape.

Thank you for reading to the end of this article! If this topic resonated with you and you’re eager for more, consider subscribing for future articles. Stay tuned for further exploration and insights!