Decouple Long-running Tasks from HTTP Request Processing — Scalable Consumers

Part 4 : Discuss how to use a separate Console Application as our consumer instead of an in-process consumer so it can easily be scaled out to multiple instances.

Background

In the last several articles, we have designed a smart coffee machine system that would decouple long-running tasks, i.e. making coffee, from the regular HTTP request processing. We have also taken our iterative approach to build a solution that implements the design. As you can see from the system diagram above, by far we have implemented our REST API using .NET 5, created and configured our Azure Service Bus resources including Topics and Queues, we have also created a worker / consumer that actually performs the long-running tasks. Below is just a quick recap on the network topology with detailed information on the inner working of the Azure Service Bus.

However, so far the worker has been running in the same process as the Web API application and is heavily tied to the API application lifecycle. That means we can not scale out the worker / consumer instances independently without interfering with the API application. This is something we want to address in this article and improve our implementation.

Goal

We want to decouple the consumer / worker from the Web API application and move it into a separate application. We have the options of Console Application, Azure Functions, etc.. For our demonstration, we will use Console Application so that we can manually scale it out into multiple instances and see how multiple instances of the consumers will all be consuming messages from the message broker, i.e., Azure Service Bus.

Getting Started

We are going to take the following steps to break our consumer into its separate application.

1. Create a new Console Application and configure it to retrieve messages from Azure Service bus
2. Remove the current consumer from our Web API application
3. Test running multiple instances of our console application and see how they can all be working on different messages.

New Console Application

A new Console Application is created in the same VS solution using .NET 5. The project is called “Services” as it shall run as a stand alone application, like a microservice.

As I mentioned in the previous article, we purposely keep the consumer classes and the message contracts in class libraries so that new projects like this console application could refer to them without referring to the whole Web API application.

The new console application only has two files other than the project files. The bread and butter part is in the Program.cs file.

A few key takeaway points:

1. Mass Transit framework is again used as a high-level abstraction on how message brokers should be configured.
2. The same Azure Service Bus namespace connection string used in the Web API (see previous article on how to retrieve it from Azure Portal) will be used here in order to establish a connection with Azure Service Bus. We store it in appsettings.json file and it will be built into our IConfiguration.
3. Line 26 registers our consumer, MakeCoffeeConsumer, so that whenever there is any messages in the “MakeCoffee” Queue, one instance of this console application will grab a lock / lease on the message and work on it. The lock / lease makes sure no other instances of the console application will work on the same message. It is worth-noting that Azure Service Bus will handle routing and load balancing, so that one message can only be consumed by one instance. If a message could not be delivered, it will be forwarded to a dead-letter queue after a number of retries for manual error handling. We will see this in the next section when we do our test runs.
4. Line 32 to 35 allows the console application to communicate with Azure Service Bus topics and queues.

Web API Application Update

Now that the consumer is in its own application, we should remove the consumer registration in our Web API application. All we need is to comment out or remove the lines where “AddConsumer” is called in the Startup.cs in the Web API project. That is!

Test Runs

Moment of truth! This is my favorite part!

First, let’s run one instance of our Web API application using “dotnet run” in Windows PowerShell.

Second, let’s run multiple instances of our new Console Application by using “dotnet run” with the no build flag. We need to flag no-build, because we can not build the application and overwrite the executable file while the first instance is already running the executable file. Otherwise, you might see error message like “Could not copy “…\apphost.exe” to “bin\Debug\net5.0\SmartCoffeeMachine.Services.exe”. Exceeded retry count of 10. Failed.

dotnet run --no-build

We should see from the console that both instances of our new Console Application are running and are configured to listen on the “MakeCoffee” queue endpoint.

Lastly, let’s send multiple requests to our Smart Coffee endpoint with coffee type “a”, “b”, “c”, “d”. If we look at our console log again for the Console Application instances, we can see the left instance consumed the messages and made coffee “a” and “c”, and the right instance made coffee “b” and “d. This is pretty cool, as Azure Service Bus is essentially handling messages, but also acting like a load-balancer making sure messages are distributed across the consumer instances! In addition, each message can only be consumed once because once a consumer instance is working on a message, it has signed a lease on the message and locked the message from other instances! Beauty!

Conclusion

There we go! We now have worker / consumer instances running outside of our Web API process. In fact, they most likely will run on different servers from the Web API application. We now could update our system diagram to reflect the multiple instances.

Updated Network Topology Indicating Multiple Consumer Instances

All articles for this project:

1. Covering system design: REST API Best Practices — Decouple Long-running Tasks from HTTP Request Processing
2. Covering minimal viable product: Decouple Long-running Tasks from HTTP Request Processing — Using In-Memory Message Broker
3. Covering Azure Bus Service as a message broker: Decouple Long-running Tasks from HTTP Request Processing — Using Azure Service Bus
4. Covering scaling out consumers: Decouple Long-running Tasks from HTTP Request Processing — Scalable Consumers

All of the code is hosted in a GitHub repo here. For a snapshot of code reflecting this article, please refer to branch v3-console-applciation-consumer. You could also refer to this pull request for the code changes needed for what we’ve covered above. The “main” branch will be the most up-to-date branch and has any future updates. Feel free to use the whole project or part of it to kick start your next exciting adventure!

Many thanks for reading. Cheers!