Azure Messaging: When to use What and Why? Post 1

Published in

Slalom Technology

7 min readJul 18, 2020

This article will assist you in choosing a cloud messaging service for your next event-driven project in Azure. We will explore Azure Service Bus, Azure Event Hubs, Event Grid, and Queue Storage — examining when to use which service and why. We will build a reference decision tree that can facilitate choosing a cloud messaging service.

The article is broken down into three posts:

Post 1: Event-Driven Architecture: Review Event-Driven Architecture (EDA) and Message Brokers concepts and benefits in a distributed environment.

Post 2: Azure Messaging Solutions: Review of each service’s architecture concepts and where it fits in the Azure cloud. We explore the types of problems each service is designed to solve.

Post 3: Choosing a Messaging Solution — Decision Tree: Analysis of 5 considerations that could impact your decision when choosing a cloud messaging service including variations, examples, and recommended services.

Post 1: Event-Driven Architecture

The future of enterprise applications is in the cloud. Native and hybrid cloud applications are distributed by nature. A cloud platform will likely have a mix of Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS) components that must interact to accomplish a desired common business value.

Distributed cloud systems bring a lot of compelling benefits such as consumption-based billing models, on-demand independent scaling, easy deployments, built-in monitoring, and abundant ready-to-use third-party offerings. In a distributed architecture, applications exchange information via messages and events to allow the decoupling of components. Services that generate events are called producers while services that consume events are called consumers.

What is the difference between a message and an event anyways?

Messages and events are usually used interchangeably but they are functionally different. A message is a command or an instruction from one system to another, such as order placement. When an order is placed, the service that creates it (producer) sends a message containing the order details to the payment service (consumer) for processing. The producer of the message is decoupled from the consumer, but the consumer knows what type of message to expect. A message can be sent synchronously or asynchronously.

An event, on the other hand, is a lightweight notification about the change of state in the system: “something happened”. The producer fires off the event but does not have any expectation about how the information carried in the event will be handled. There could be one or many consumers interested in an event. Each consumer might have a specific function such as monitoring, anomaly detection, or simply logging. There are two types of events: discrete and series (also known as a stream). A discrete event is a one-time notification such as the creation of blob in a blob container. A series of events are produced in a sequence over time. Telemetry is a common use case; for example, monitoring the health and load of a system. Another use case is event streaming from IoT devices.

Event-Driven Architecture

In an Event-Driven Architecture (EDA), the application is a threefold implementation: there are event producers, event consumers, and messaging infrastructure modules that are used to receive, validate, retain, and route events to appropriate consumers. These tools are known as Message Brokers. RabbitMQ and Kafka are two popular open-source message brokers.

The EDA is accomplished in two models:

Pub/Sub: Consumers or subscribers are known by the message brokers and messages are released only once and can never be replayed. Pub/Sub is illustrated in Fig. 2.
Event Streaming: Consumers do not subscribe to the message broker, instead, they read events from any position in the stream. Subscribers are responsible to advance their reading positions or offsets. Event Streaming is illustrated in Fig. 3.

Why should you use EDA?

As mentioned above, EDA presents many compelling benefits especially in distributed environments:

Decoupling: Coupling event producers with event consumers will cause a bottleneck in your infrastructure if events are not consumed at the same rate as they are produced. Coupled services are also more difficult to maintain and scale.
Reactive interactions: Seamless real-time communication between events producers and consumers enables automation across your infrastructure.
Load balancing: Modern applications generate massive amounts of data that cannot be consumed in real-time. By acting as message load balancers, message brokers enable distributing events across multiple consumers when the volume goes up. This improves throughput and reduces latency in the system.
Load Leveling: By acting as a message load-leveling tool, instead of scaling up consumers, a message broker system acts as a buffer for messages which allow steady consumption instead of stressing the system. As illustrated in Fig. 4, it is necessary to have a triage system in between to coordinate this imbalance between producers and consumers.

Message Brokers Concepts and Features

Message brokers are infrastructure resources that are used to receive, validate, retain, and route events from producers to appropriate consumers. There are many message brokers offerings: on-prem vs cloud implementations (managed), open-source vs paid.

Most cloud vendors have developed their own message brokers. In this post, we will explore four message brokers from Microsoft Azure: Azure Queue Storage, Azure Service Bus, Azure Event Hubs, and Azure Event Grid. RabbitMQ, ActiveMQ, and Kafka are the three most popular open-source message brokers among software engineers.

Choosing a message broker depends on the requirements at hand, but these features will always influence the final choice:

Durability: Durability is a message property that enables recovery if the broker fails and restarts. To achieve this, message brokers have to store messages and their payloads on disk or other storage mechanisms such as storage accounts in Azure.
Persistence: Persistence is similar to durability but applies to message entities such as queues, topics, subscriptions, and exchanges. Once messages are recovered from a failure, a message broker should recover their containing entities in order to function properly.
Dead-Lettering: Ability to persist and deliver failed messages in a special queue called Dead-Letter Queue. Failure reasons may include network failures, a deleted queue, a full queue, authentication failure, or a failure to deliver on time. A powerful message broker should natively offer dead lettering.
Poison Message Queue: If the receiving application fails to process a message, it is placed in a poison-queue for further investigations. Some message brokers combine the function of dead-lettering and poison messages. Azure Service Bus is an example.
Protocols: Protocols mandate interoperability behavior of message brokers and receiving clients between vendors. Advanced Messaging Queuing Protocol (AMQP), HTTP/HTTPS, and Kafka are commonly implemented in most of the message brokers.
Duplicate Detection: Ability to prevent duplicating messages. In many scenarios, producers can send the same messages. Without the ability to prevent this fault, consumers will translate the same message more than once. An ideal message broker should natively prevent sending the same message more than once.
Delivery Guarantee: There are three receiving modes. The first mode is At-Least-Once (ALO); when a message is produced by the source, an ideal message broker should deliver it to consumers at least once even in case of a failure. The second mode is At-Most-Once (AMO); once a message is requested by a consumer, the message broker should make sure it is only processed once even in case of failure. The third mode is Effectively Once; by combining ALO, AMO, and duplicate detection, a message broker can guarantee that each message is processed exactly once.
Message Ordering: Some scenarios require message ordering with first-in-first-out (FIFO). Advanced message brokers such as Azure Service Bus offer this feature natively. Other message brokers can achieve this feature with additional custom logic.
Sessions: Sessions characteristic enables consuming clients to process messages based on a certain logical grouping such as time or any other predefined Id. This requirement is very important in event streaming scenarios such as telemetry and web analytics (browsing sessions).
Batching: Batching enables message brokers to receive messages in batches.
Pub/Sub: With pub/sub, all subscribers are known by the message brokers. It can be implemented in two patterns: push and pull. With a push pattern, message brokers push messages to consumers as soon as they arrive in the queue. With a pull scenario, subscribers constantly check and pull messages from queues.

Competing vs Partitioned Consumers: In a pull scenario, if multiple subscribers are polling from the same queue, we say they are competing, hence the term Competing Pattern. If messages are divided into multiple routes, we say they are partitioned.

Conclusion

In this first post, we looked at the Event-Driven Architecture: the concepts and benefits. We also defined message brokers and explored their features. In the next post, we will look at four popular message brokers in Azure: Azure Queue Storage, Azure Service Bus, Azure Event Hubs, and Azure Event Grid.

Continue to read the series: