Using Azure Functions Premium to process sales orders
Got 300,000 sales transactions you need to process in 1 hour?
Say you have an e-commerce app (such as Chai tea ☕️ delivery) and you are about to launch a killer sales campaign. Your app must be ready to process large number of orders per hour while keeping the infrastructure cost down. Here are the requirements:
- Each sales order consists of 5 JSON messages and need to be processed in an FIFO pattern (First In, First Out).
- The sales processors must be able to scale out/in based on demand (such as number of unprocessed orders).
- Need to monitor/troubleshoot the real time performance of sales processors.
That sounds simple, right? Not really…
- How to ensure the JSON messages for each transaction is processed in the right order when we have an surge of 100k orders processed by scaled instances of my apps?
- Which message broker should we use for decoupling the back end?
- How to dynamically scale based on load/events? What are the controls I have?
- How to protect downstream services (such as relational DB) from being overloaded?
- How to prevent and control runaway executions?
Function App Premium + Service Bus to the rescue
Sales orders are ingested into the Service Bus queues at a very fast pace (as fast as the client CPU can handle). Our sales processor functions will be triggered by messages in queues. We have 1 pre-warmed instance and the function app will scale between minimum of 1 and maximum of 20 instances.
The functions will store the messages in Redis and we will use redis-cli to examine whether the messages have arrived in the correct order.
Once the sales processors are triggered by messages, we will use the live metrics and logs search in App Insights to observe the real-time performance of Function Apps.
Let’s deploy the services
- Service Bus Namespace & Queues (enable Sessions for FIFO guarantee)
- Azure Function Premium Plan (with .NET Core runtime stack)
- Function App triggered by Service Bus Queue (either use the Portal or VS Code)
- Application Insights (for monitoring the Function App)
- Azure Cache for Redis (install redis-cli on your laptop)
- Service Bus Explorer (run from your laptop)
If you have deployed everything in one resource group…
Deploy the C# Function App
Before we generate the orders in next section, let’s review the function app code:
- Clone from my GitHub repo (Forked from @Jeff Hollan’s original repo)
- Launch VS Code from the
functions-csharp-servicebus-sessionsfolder. - Update your
local.settings.json(e.g. connection strings, etc.) and hit F5 to debug locally.
In SessionTrigger.cs, here’s the Function entry point with additional logging:
The host.json file. Even though I have included both messageHandlerOptions & SessionHandlerOptions in JSON, only the latter is honored since we enabled “Sessions” for the Service Bus Queues:
Set the MaxConcurrentSessions to 2000. According to Azure SDK documentation:
The maximum number of concurrent calls that the library can make to the user’s message handler. Once this limit has been reached, more messages will not be received until at least one of the calls to the user’s message handler has completed.
You can experiment here by changing the value to 1 and observe how SLOW 🐌 it will be when processing 1 session at a time.
Now, let’s publish the Function App from VS Code or Function Core Tools and verify the artifacts in Azure portal:
Let’s generate 100,000 sales orders!!
Launch VS Code from the session-queue-message-generator folder to generate the 50k sales orders with 5 messages per order (a total of 250k messages). Simply update the connection string and the queue name in launch.json. Simply Ctrl+F5 to send messages to the Service Bus queue.
Since we have a rather large number of messages, we have 2 queues with 1 dedicated Function per queue. Repeat the same to pump messages into the 2nd queue. You can change the number of sessions and in Program.cs. With 2 queues, we have a total of 100k sales order (500k messages!).
The 2 functions should be triggered by messages in those 2 queues and start processing/storing the messages into Redis. See Redis Cache Writes metrics below.
The Cache Write Average above can give us an idea on how fast the data is being written to a back end data store. Remember the MaxConcurrentSessions in previous section? Make sure you adjust that number to avoid overwhelming the downstream services.
Let’s also confirm the messages have been stored in the right order in Redis as well: “Yes, FIFO.”
Function App’s scaling behavior
Now, let’s review the metrics in App Insights. How did our Function App perform?
Our Function app processed 500k messages in about 20 minutes 🚀.
Here’s the average response time (178 ms) for the functions:
When I have 2 instances, you will see both instances are hitting high CPU load (90–95%):
When scaled out to 10 instances, the CPU load is lower now on each instance (20–22%):
Let’s use the log search in App Insights to review the end to end transaction details. On this particular Function host instance, it processed Session ID “Order-4” with 5 messages in FIFO order:
Conclusion
Azure Functions App Premium Plan allows us to scale up & down, out & in. Together with Application Insights, we can optimize the cost/performance ratio for your apps. When more details about the performance are revealed in metrics, we can plan ahead and prevision the Pre-Warmed instances when needed. At the same time, we can control the maximum number of burstable instances to protect ourselves.
To learn more, read How Scaling works and Scalability Best Practices.
Consider the Azure Functions Premium plan in the following situations:
- Your function apps run continuously, or nearly continuously.
- You have a high number of small executions and have a high execution bill but low GB second bill in the Consumption plan.
- You need more CPU or memory options than what is provided by the Consumption plan.
- Your code needs to run longer than the maximum execution time allowed on the Consumption plan.
- You require features that are only available on a Premium plan, such as virtual network connectivity.
Have fun with Azure ☁️ and Go Serverless!! 💪
