Azure Messaging Services -How to Choose the Right Messaging Technology in Azure
A quick overview of messaging services offered by Microsoft Azure and guide to select the best fit for an application
Why Messaging?
A highly reliable messaging system enables distributed microservices to communicate securely and asynchronously across networks. A messaging system allows loose coupling which means both Sender and Receiver need not be online at the same time. Also, both sender and receiver can produce and consume at their own velocity and scale-up horizontally as the workload increases.
Azure Messaging Services
Azur offers various services for messaging. Each of these services is designed to serve different use cases and do not compete against each other. Let’s understand each of these services:
1. Azure Storage Queues
Azure Storage Queues is part of the Azure storage account. It is a simple, reliable, scalable, and persistent message queuing service. It follows REST-based HTTP verbs like GET and PUT to read and insert messages into the queue. An application can also look at the message on the head of the queue using the PEEK command.
A queue can store millions of messages up to the total capacity of a storage account where each message can be up to 64 KB and a maximum TTL of 7 days.
1.1 Azure Storage Queues Use Cases
Azure Storage queue is a simple queue service useful to store a large number of small size messages. It offers a secure way to access messages from anywhere in the world. It can be used to create workloads or tasks to be processed asynchronously and message order guarantee is not a must.
2. Azure Service Bus
Azure Service Bus is a secure, highly available & reliable Platform-as-a-service (PaaS) enterprise integration message broker for asynchronous transfer of data between applications. It supports Advanced Message Queuing Protocol (AMQP) which is a TCP based efficient and reliable open protocol and makes a solution vendor-neutral. A developer can use different client libraries to connect to the message broker.
A sender sends messages to the queue. Queue sends back the ACK receipt after successfully replicating the message. On the other side, A receiver connect to the queue to retrieve a message and after completing the processing, it sends back an ACK for message deletion. A message can be moved into a Dead-letter queue (DLQ) to discard or later consumption, in case of an error.
Azure Service Bus supports various advanced queuing operations and some of the key differentiating capabilities are:
- Message sessions
- Auto forwarding
- Scheduled delivery
- Message deferral
- Batching
- Transactions
- Filtering and actions
- Autodelete on idle
- Duplicate detection
2.1 Azure Service Bus Queue
Azure Service Bus Queue is an ordered, time-stamped message storage. it can be used for point-to-point communication which means a single producer of the message works with a single receiver that reads from the queue. The messages are delivered in a pull mode (a.k.a competing consumer pattern) and a consumer is responsible to deQueue the message after it is processed to free up the storage.
2.2 Azure Service Bus Topic
Azure Service Bus Topic is also a queue internally but it is designed for point to multi-point communication. This means we have a message publisher writing messages to the topic and each topic subscription receives a copy of the message. Messages are received from a subscription identically to the way they are received from a queue.
2.3 Azure Service Bus Use Cases
Service bus advanced queueing operations enable it for enterprise messaging. It is useful when an application wants to send a directed message(Command) to other applications and expect them to perform a specific action.
For example, an e-Commerce application after receiving an order sends a directed message to the Procurement-service to procure a product, a Warehouse system to pack a product, and a CRM system to notify the customer.
3. Azure Event Hub
Azure Event Hub is a fully managed PaaS Big data event streaming platform & event ingestion service capable of receiving and processing millions of messages per second with low latency. It is designed to handle the volume, variety & velocity of events. It also supports Advanced Message Queuing Protocol (AMQP). Each consuming application gets its own event stream.
The event hub internally divides messages into buffers called as partitions to maintain ordered event sequence. Also, the event hub can capture events and store them into Azure Blob storage. All event publishers are issued a token to securely allow external devices and mobile apps to inject events.
Azure Event hub can expose Apache Kafka endpoint to allow an application earlier using Kafka as a message broker, migrate to the event hub without making any code changes.
3.1 Azure Event Hub Use Cases
Event hubs are best suited for high-speed data ingestion scenarios like telemetry and distributed data streaming and we need to aggregate or analyze the event stream.
It is useful when applications want to publish facts like change in state and do not expect any specific action from consumers. For use-cases like publishing time series Click-stream from a website to track user journey, A sensor equipment sending temperature and pressure data to track the major change in these properties, etc. can be easily done with an event hub.
4. Azure Event Grid
Azure Event Grid is a fully-managed event routing service. Event Grid distributes events from different sources to different handlers, such as Azure Functions or logic apps. Messages are pushed to event handlers, this means handlers should be capable of handling the load or implement a throttling mechanism.
A publisher emits events to a topic without knowing the consumers and on the other hand, event subscribers can create a subscription on that topic. Consumers can configure certain filter criteria such as the event type, event subject or based on data in the payload.
The event sources send events to the Event Grid and the Event Grid forwards relevant events to the subscribers.
4.1 Azure Event Grid Use Cases
Azure event grid is useful in scenarios where an application needs to react to the changes that happened in another service. for example, To process a file when a new file is uploaded in blob storage, Tag a virtual machine as it spun up, Run a workflow when a new user is added to a resource group, etc. This avoids the need for a long polling or hammer polling and saves on network resources.
It is also useful to filter and fan out an event to multiple destinations. It ensures a reliable delivery to a destination by retrying an event up to 24 Hrs using exponential backoff.
5. Comparison & Decision Tree
Here is a quick comparison of each messaging technology on the basis of most common characteristics:
We can use the following decision tree as a starting point to quickly determine a messaging service based on use case:
Most of these services are offered in various tiers. Service features and cost varies on the tier. Therefore, the tier should be considered while choosing a messaging service.
6. Summary
In this article, we have explored four different Azure messaging services which enable us to develop reliable and resilient distributed applications. Choosing between them is a matter of deciding the type of data that needs to be passed between components (commands or facts), and then what features you need to deliver and process the data.
Reference
- Azure Service Bus: https://azure.microsoft.com/en-in/services/service-bus/
- Azure Event Hub: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-about
- Azure Storage Queue: https://docs.microsoft.com/en-us/azure/storage/queues/storage-queues-introduction
- Azure Event Grid: https://docs.microsoft.com/en-us/azure/event-grid/overview
