Azure Service Bus
Artifacts — Queues, Topics, Subscriptions.
[Tags] — Azure Service Bus, Azure Service Bus Namespaces, Azure Service Bus Artifacts, Messaging, Publish-Subscribe, Routing & Correlation, Poison Messages, Dead Lettering & Error Handling, Security.
Background: There are several capabilities that are desired to have in the Messaging Service we are choosing for our services like-
- Communication — How can applications in different environments communicate effectively and reliably with one another?
- Security — How can the confidentiality and integrity of messages be maintained?
- Reliable Delivery — How can the sending application be sure that the receiving application will receive all transmitted messages?
- Low-Latency — How can the transmission and processing time of messages be kept as low as possible?
- Availability — What level of up-time will a messaging system provide.
- Scalability — How easily can a messaging system be upgraded to handle an increase in the message processing load?
Microsoft Azure Service Bus Technologies —
- Relay Service — A way to do service remoting i.e., it enables you to expose a service hosted on a private cloud to external clients. Clients interact with the Relayed service in the same manner that they would if they were on the local network, except that they access it via the Service Bus Relay endpoint.
- Brokered Messaging — Provides a durable messaging platform with components such as Queues, Topics and Subscriptions. Has features like publish-subscribe, temporal decoupling, reliably storage of messages, Load-leveling.
- Event Hubs
- Notification Hubs
Maximum number of namespaces per subscription — 100
Maximum number of queues and topics per namespace — 10,000
Maximum message size — 256 KB or 1 MB
Maximum message header size — 64 KB
Maximum storage size for a Queue or Topic — 5 GB
Maximum number of topics in a namespace — 10,000
Maximum number of subscriptions on a topic — 2000
Point-to-point messaging channel —
Benefit is “Load leveling” — which enables producers and consumers to send and receive messages at different rates.
“Pull-based load balancing”- Computing Consumer pattern — each message is processed by only one of the worker processes.
Using queues to intermediate between message producers and consumers provides an inherent “loose coupling” between the components.
Topics and subscriptions
Publish/subscribe messaging —
In contrast to queues, in which each message is processed by a single consumer, topics and subscriptions provide a one-to-many form of communication, in a publish/subscribe pattern.
Useful for scaling to very large numbers of recipients, each published message is made available to each subscription registered with the topic. Messages are sent to a topic (and delivered to one or more associated subscriptions) and received from subscriptions. Filter rules can also be set on a per-subscription basis.
A brokered message is made up of two components —
- Header (context)
- Message body (content)
Header contains information that is used by the application as well as the Service Bus itself — like Message Id, Session Id, Label etc and properties in a key-value property bag. CorelationId is used for routing and SessionId is used for corelation.
Some properties are filled in the Service Bus itself like SequenceNumber, DeliveryCount, EnqueuedTimeUtc, ExpiresAtUtc, LockedUntilUtc, LockToken, Size, State etc.
Once a message is created, message body cannot be changed — immutable, whereas header can be modified.
Features in Topics (publish subscribe messaging)
Topics (Publish Subscribe Messaging) offers more flexibility than Queues (point-to-point messaging).
Routing with the help of subscription rules. Each subscription can have a filter rule to govern what message comes in it.
Note: Message body cannot be used to route messages. Fields you would like to use for message routing must be added to message header property bag as well.
Subscribers vs Subscriptions
Subscribers and Subscriptions are two different things. A Subscription can have multiple subscribers which will compete for messages on that subscription. If you want independent applications to each receive a copy of the message sent to a topic, they will each have to create their own subscription to the topic.
CorrelationId — Used for more efficient ‘routing’ between topics and subscriptions.
SessionId — Used to ‘correlate messages’ in receiving applications.
Messages can have a correlation identifier in the message header so that a group of related messages can be identified using a common correlation identifier at the subscriber.
We can use message correlation in an asynchronous request-response scenario where we can correlate reply with request.
Possible Errors while sending a message on Azure Service Bus:
- Network Error, Throttling exception, Internal service error in service bus, Security exception, Endpoint not found exception.
Retry policies available and customizeable.
Each subscription will have it’s own dead-letter queue. There are certain factors that make the message dead-lettered implicitly, like Max Delivery Count exceeded (max repeatedly failed processing of message), Message Expired (when message expiration is enabled on queue or subscription) and Routing Failure exception (when enabled on queue or subscription). Receiving end can also explicitly dead-letter a message.
We can configure it by setting the TimeToLive property in the message itself before sending it or DefaultMessageTimeToLive property in topic/queue/subscription.
SAS — Shared Access Signatures
The Standard and Premium Tiers provide the support for Topics. You can get much more better and predictable performance when using Premium Tier, using more message units.
Event Driven Architecture (Link)
- Prior to Azure SDK 2.0, clients had to poll for the messages constantly (like having an infinite loop to poll queue/subscriptions). With the advent of event driven messaging in 2.0 release (called message pump, provides ‘OnMessage’ callback), clients no more have to keep polling the queue.
- Remember to mark message as complete once processed to remove it from the queue/subscription. The default behavior when you dequeue a message is called “Peek-Lock” it will lock the message so no one else can get it while your processing it and will remove it when you commit. It will unlock if you fail to commit, so it could be visible on the queue or subscription again and will picked up again on next call to OnMessage on that queue/subscription.
- - Topic size can only be specified while creating the topic. Once topic is created, topic size cannot be modified.
- - If no subscriptions are there, messages sent to the topic will be lost. There won’t be any exception on publisher side when this happens.
- If the publisher needs to know if there are active subscriptions before sending the message, use the EnableFilteringMessagesBeforePublishing property while creating the Topic, and catch the NoMatchingSubscriptionException while sending message to Topic. Please note that this feature is only for use in development and testing, and it not meant to be used in production. See stackoverflow.
- - Topic size is max 5 GB in standard namespace and 80 GB in premium namespace. Throws Exception in sending side. Also, topic as such has no size, it’s the size of the subscriptions. So if you create a topic with 80 GB and if the topic has N subscriptions, each subscription has size = 80/N GB.
- o For premium Topic, by default partitioning will be enabled. For standard, you can set it from code by setting EnablePartitioning to true.
- - What does the AccessedAt property in the TopicDescription, SubscriptionDescription and QueueDescription represents? Is it a way I can find when the message to the Topic was last posted (likewise for queue, subscription)? Answer seems no, because the AccessedAt value is updated even when there is no one actively sending to topic. May be it takes into account any connection to topic to update AccessedAt.
- - To clean up the Topics/Subscriptions not in use, you can use the AutoDeleteOnIdle property while creating the Topic/Subscription.
- - To clean up the dead letter queue which is a subqueue in your subscription, read the messages from the queue and call complete() on it to remove it from the dead-letter queue. In the Microsoft Azure Dashboard, in the right column there is a field called “Queue Length”. This shows the number of messages in the queue, active and dead lettered. To access the dead-letter queue of your subscription from code, see this.
- - See this link. Active messages on a subscription includes the deferred state messages as well as normal messages. The messages that are deferred won’t be automatically delivered to OnMessage. You need to know their sequence number to process them in a specific way.
- EnableDeadLetteringOnMessageExpiration in SubscriptionDefinition. Expired message will be moved to dead-letter queue (DLQ) in a subscription. If above is set to true. Default is false. DLQ holds msgs that couldn’t be delivered to receiver or mgs that couldn’t be processed. Msgs remain in DLQ until you explicitly retrieve them from DLQ and call Complete() on the dead-letter message. You can specify TTL on a message or on a subscription.