Decouple Long-running Tasks from HTTP Request Processing — Using Azure Service Bus

Part 3: Discuss how to use Azure Service Bus as a message broker instead of an in-memory message broker and how topics and queues are auto-configured.

System Diagram by Author

Background

In the last couple of articles, we designed a smart coffee machine system that would decouple long-running tasks, i.e. making coffee, from the regular HTTP request processing, so that the user would get a response right away; we also took a very first stab at implementing the smart coffee machine system using .NET 5 and built a system with the following features:

1. A Web API project that hosts our REST API endpoints, one of which is to accept a make coffee request.
2. An in-memory message broker that manages message queues, publishers and subscribers, allowing the messages to be handled asynchronously.
3. An in-process consumer that subscribes to the messages for the make coffee requests, and perform the tasks to actually make coffee.

This is probably the simplest implementation that will work as a proof-of-concept. As mentioned in the last article, in order to improve the system’s scalability and make it a more production-ready system, we’d like to use an actual message broker like Azure Service Bus or RabbitMQ hosted on a separate server. This will allow our messages to persist and survive any Web API application restarts. In addition, it will let these message brokers do what they are good at, such as message forwarding, retrying, dead-lettering, etc..

Goal

The goal of this article is to improve upon the v1 implementation from the last article, and introduce Azure Service Bus (ABS) as a message broker to our system.

By the end of this article, we should have the following items implemented:

1. Azure resources created, including Azure Service Bus namespace, Shared Access Policy.
2. Web API application updated to communicate with Azure Service Bus.
3. Topics and Queues automatically configured in Azure Service Bus by our Web API application on startup.
4. End-to-end workflow for a message (technically an event) to flow from the Web API application to Azure Service Bus topic, and to our consumer to process it.

Getting Started

In order to start using Azure Service Bus as our message broker, we are going to take 2 steps.

1. Create Azure resources from the Azure Portal
2. Use Azure Service Bus in Web API application

Step 1 — Create Azure resources from the Azure Portal

You should be able to follow along by looking at the screenshots. However, if you would like to try it out on your own, you would need an Azure account.

The first thing we want to create is a Service Bus namespace, which is like the concept of a VS solution that contains all the topics, queues, access tokens, logging, etc. your project has. Standard tier is required as a minimal.

Next we want to create a Shared Access Policy so that we could generate a connection string for our application to use, in order to communicate with the Service Bus namespace. We need “Manage” access checked so that our application can create topics and queues in Service Bus.

Once the SAS Policy is created, copy the connection string value, and we will need it in our Web API application.

That’s it! We now have our Azure Service Bus resource ready to go. Note the topics and queues are not created yet, as we will use our Web API application to create them.

If you are not familiar with Azure Service Bus Topics, they are the resources used for pub-sub pattern. Events are published to a topic, and subscribers subscribe to the topic in order to receive a copy of the events. Events in a topic could also be forwarded to a queue as a message. Once a message in a queue is consumed, it is removed from the queue. In odd cases when a message could not be consumed by a consumer and acknowledged, it may be sent to a dead-letter queue for error handling.

Step 2 — Use Azure Service Bus in Web API application

In order to use use the Azure Service Bus namespace created in the above step, only a few lines of code is needed! If you recall from our last article, the in-memory message broker was registered as a service dependency in the Startup.cs class. That’s exactly where we will go to make our quick updates.

A few quick notes on the updated Startup.cs

1. We commented out the in-memory message broker at line 25 and added Azure Service Bus at line 31.
2. Line 33 retrieves the Azure Service Bus connection string from the appsettings.json file, where the key should be named “AzureServiceBusConnectionString”, and the value should be the SAS connection string you copied from the above step.
3. Line 37 makes sure the Topics and Queues are created in Azure Service Bus using the default naming convention, when the Web API application starts up. This is how we automatically create relevant topics and queues through our application.

You can also refer to this pull request to see all the code changes needed to swap out in-memory message broker and replace it with Azure Service Bus.

Network Topology

Even though the code change is small above, the new code does create quite a few resources in Azure and make the workflow slightly more complex but resilient. A system network topology is probably the best way to understand it. We will lay out the network topology the same structure as our system diagram, so that we can cross reference the components.

As you can see, we have a Web API, a message broker using Azure Service Bus, and a consumer / worker that does the work. Sweet!

It is worth-noting the workflow is as follows at the high-level:

• A topic is created for the message contract “IOrderAccepted” under the Service Bus Namespace.

• A subscription to the above topic is created for the message consumer “MakeCoffeeConsumer”, and the subscription is named as “MakeCoffee” by default naming convention.

• The subscription created above will forward the events to a message queue called “MakeCoffee”, which is also created by the application following the default naming convention. Note this “MakeCoffee” is a message queue, even though it has the same name as the “MakeCoffee” subscription.

Let’s see it in action!

It is not just smoke and mirrors… If we run the Web API application, we should see something like this in the console with logging information on the Topic, Queue, and Subscription.

If we want to see the peek at the message that is flown among our Azure resources, we can set a breakpoint within the “MakeCoffeeConsumer” and call the SmartCoffee endpoint. Note we are requesting an expresso and we should get a response with an order id right away.

Once the breakpoint hits, as below

we should be able to use the Service Bus Explorer to Peek at our messages in the Queue. Note if you do not set a breakpoint, the message would be processed and be removed from the Queue, so you may not see any messages in your queue.

This is a sample message peeked. Note the order Id is the same as the order Id in the Web API response above.

Screenshot by Author

Conclusion

We have successfully added Azure Service Bus (ASB) as a message broker into our system and tested out the end-to-end workflow. A regular HTTP request triggers an event to be published to an ASB Topic, and the Topic has a Subscription that forwards the event to a ASB Queue as a message. The consumer then picks up the message from the Queue, completes the task, and remove the message from the Queue. Beauty!

Hope you have enjoyed the baby steps we are taking to implement our distributed messaging system in order to decouple long-running tasks. Thank you for reading! Cheers.

All articles for this project:

1. Covering system design: REST API Best Practices — Decouple Long-running Tasks from HTTP Request Processing
2. Covering minimal viable product: Decouple Long-running Tasks from HTTP Request Processing — Using In-Memory Message Broker
3. Covering Azure Bus Service as a message broker: Decouple Long-running Tasks from HTTP Request Processing — Using Azure Service Bus
4. Covering scaling out consumers: Decouple Long-running Tasks from HTTP Request Processing — Scalable Consumers

All of the code is hosted in a GitHub repo here. For a snapshot of code reflecting this article, please refer to branch v2-use-azure-service-bus-mesage-broker. You could also refer to this pull request for the code changes needed for what we’ve covered above. The “main” branch will be the most up-to-date branch and has any future updates. Feel free to use the whole project or part of it to kick start your next exciting adventure!