Azure Cosmos DB + Functions Cookbook — monitoring Trigger pending work
This time we are revisiting the Azure Cosmos DB Trigger and exposing a very interesting feature, the Remaining Work Estimator.
Scenario
You are currently using the Azure Cosmos DB Trigger, your Functions are triggering on changes, but you would like to understand if your Functions are “lagging” behind processing changes.
If the rate at which the data is ingested in the database is consistently higher than the rate at which you process them (because your Function does too many things or it’s slow in what it does), then it will start to lag behind. But how can you tell? Read on!
Ingredients
This is based on the fact that you are already using the Cosmos DB Trigger. The Trigger, as we talked on previous posts, uses the Cosmos DB Change Feed Processor Library underneath. But one cool thing not many are aware of, is that the Library has another tool, called Remaining Work Estimator, that can return the difference between the current state of the Leases versus the last known state of the Change Feed.
Since the Leases represent the current point in time your Function is processing the Change Feed at, this tool effectively can estimate how far you are lagging behind.
Using the Remaining Work Estimator is not straightforward, but we’ll use a couple of tricks:
- We’ll use the Cosmos DB Input Binding to obtain the DocumentClient instances for each collection (the one we are listening to, and the one that stores the leases). Since Azure Functions V2 uses Dependency Injection, we are actually pulling in the same DocumentClient instances our Trigger uses, effectively maintaining the connections, and not raising them nor increasing memory consumption.
- We’ll leverage the Change Feed Processor Library methods, since that package is already included in the Cosmos DB Functions Extension.
We can then leverage the Estimator inside a TimerTrigger and send all the information to Azure Application Insights!
Recipe
Let’s assume we have a normal Cosmos DB Trigger, and we are reading changes happening in a particular collection, which names we are reading from the configuration:
To monitor its progress, we’ll use a TimerTrigger, and pull both DocumentClients in:
The TimeTrigger will read the value from the Estimator and log it. The GetRemainingWorkEstimator implementation is Lazy:
And for reference, we are using this helper to parse the Cosmos DB connection string:
This will basically send, every 1 second, the status of the Remaining Work, to your logging solution, which can be Azure Application Insights as a custom metric.
If you see a constantly raising trend with no dips, it means your Function is constantly lagging behind and the difference is increasing, time to optimize!
The complete code for this article is available on GitHub.
Stay tuned for more recipes!
Other posts in this series:
- Azure Cosmos DB + Functions Cookbook — static client
- Azure Cosmos DB + Functions Cookbook — HTTP querying
- Azure Cosmos DB + Functions Cookbook — output collector
- Azure Cosmos DB + Functions Cookbook — live migration
- Azure Cosmos DB + Functions Cookbook — search indexing
- Azure Cosmos DB + Functions Cookbook — secure client
- Azure Cosmos DB + Functions Cookbook — Connection modes
- Azure Cosmos DB + Functions Cookbook — multi trigger
- Azure Cosmos DB + Functions Cookbook — Multi master & preferred region
- Azure Cosmos DB + Functions Cookbook — Shared throughput and new health logs
