Azure Cosmos DB + Functions Cookbook — live migration

Published in

Microsoft Azure

5 min readFeb 7, 2018

Continuing with this series of short but sweet recipes you can use with Azure Cosmos DB and Azure Functions, I’ll focus this time on real-time data transfer scenarios based on the use of the Azure Cosmos DB Change Feed.

Scenario

You need to do a live data migration between two Azure Cosmos DB accounts in real-time or you want to keep two Azure Cosmos DB collections in sync to offload data analysis and post-processing.

Ingredients

This time around, we’ll be using the Azure Cosmos DB Trigger to, well, trigger the Function and the Azure Cosmos DB Output Binding (which we already used in another recipe, in case you want to understand its internal workings). The Trigger is an interesting ingredient, internally, it leverages the Change Feed Processor Library in such a transparent way that it hides all the complexity required to hook to the Change Feed and process the data in a scalable way allowing you to trigger a Function based on changes (inserts or updates, no deletes yet) that happen in a given collection. Changes are sent to the Function in the form of a batch or list of Documents that can be iterated and processed.

But how does it really work in Azure Functions? What if I scale my Function to several instances or I use it in a Consumption Plan?

The answer is quite simple. The Change Feed Processor Library maintains an auxiliary collection where it store Leases, one Lease per Partition Key Range in your collection (read more about Partitioning here). When a Function instance is created using the Cosmos DB Trigger, it will claim all the Leases, and start processing the changes in each Partition Key Range in parallel (this is important as the Change Feed guarantees order of changes within the Partition). If more instances appear, the load will be evenly distributed (this actually means the Leases will be distributed among all the instances) and each instance will process the changes in the Partition Ranges it owns, this way, we can distribute the compute load automatically. I’ll be writing a dedicated post shortly with more detailed info if you are interested in the Trigger internals, but the takeaway is that it will automatically balance compute load among all existing Function instances.

IMPORTANT NOTE: The Cosmos DB Output Binding uses the SQL API to persist information, please do not use it with Mongo API enabled accounts.

Recipe

This recipe requires the existence of three different collections:

The collection where the changes will occur, we’ll call it “monitored” collection.
The collection where we want the changes to be saved, either because we are doing a migration of data or because we want to maintain both in sync, we’ll call it “target” collection.
The collection that will store the Leases used by the Cosmos DB Trigger. It will be automatically created in case it does not exist, we’ll call it “lease” collection.

It is important to understand the RU relationship between #1 and #2. If the provisioned RU for #2 is much lower than #1, the changes might start to lag behind, since the changes will be written in #1 at a higher cadence than what #2 can receive simply because the Function will try to write at the same speed that changes are happening and might get throttled.

Let’s start with our function.json file, where we specify the bindings. In the cosmosDBTrigger we define the Database, Collection and Connection String to the “monitored” collection and the name of the “lease” collection which will be created. In the documentDB binding, we define the Database, Collection and Connection String to the “target” collection.

In C#, the entire live migration scenario takes less than 5 lines of code. The Function receives the batch of changes in the form of a IReadOnlyList<Document> and you can iterate over it to access the changes (inserts or updates).

You can alter or change the document before saving it or just keep it as-is. Notice that we are using the IAsyncCollector we discovered in the previous recipe to save each document.

In NodeJS, the code is quite similar, the input is a simply Array of objects/Documents:

As you can see, the entire process is extremely short and simple! The result is that on each change or set of changes that happen in the “monitored” collection, the Function will trigger, and send those changes to the “target” collection. It is leveraging a lot of complex things for you, like connection handling, static instances, scaling and load distribution, everything so you can focus on creating a Function that does just what you need.

Afterword on error handling

There will be a more detailed post about the Trigger coming but there is a small (but not less important) topic I wanted to write about, and that is error handling.

At the time of this post’s writing, there is no automatic dead-lettering or retry logic if your Function’s code fails during Runtime using the Cosmos DB Trigger, the batch of changes received as input will be lost, the next Function execution will happen with the next set of changes.

Our current recommendation is that you add some try/catch logic in the case that you are processing the documents against another service or through some complex logic that can fail at some point. You can catch the failed documents and send them to another storage or queue for post-mortem analysis or later processing (Azure Storage Queues, Service Bus or Event Hubs are good candidates). This try/catch logic is specially important if your code is doing something after reading the documents, like sending an email, to avoid the entire execution failing for an unrelated task (that is also why you might want to consider having small and specialist Functions instead of big generalist ones).

Updated content!

For an updated content that has an optimized take on how to persist the documents on the destination, see https://github.com/Azure-Samples/cosmos-dotnet-functions-migration

Stay tuned for more recipes!