Veyo Technology: Using an Azure Lease Blob as a Distributed Mutex

It is common to run more than one instance of an application at the same time. At Veyo, we have a tool that imports information about patients into a service every night. The information comes from different sources such as CSV files, EDI binary files, and relational databases. The patients belong to different health provider accounts, so we run multiple instances of the import tool - one for each account. The tools run on different VMs, and they run at a specific time but only one instance is allowed to run at a time since multiple instances using the same account will cause data corruption. We decided to use Microsoft Azure Lease Blob to ensure that only one instance of this tool is running at a time.

This post will go over an implementation of the Lease Blob pattern and also dive into a few of the gotchas you might want to look out for when using this pattern.

Definitions:

Mutex — a variable that is used to ensure that some resource is accessed by at most one entity at a time

Semaphore — a generalized mutex that allows access to some resource by at most N entities at a time

See Wikipedia for more information.

Lease Blob is simply a locking operation that can be invoked via a remote API call to Microsoft Azure. It is the key behind the mutex implementation.

How It Works

The following outlines how the process works:

1. When an app runs, this instance tries to acquire a lock.
2. If the lock was acquired successfully, the instance continues; otherwise, it stops gracefully indicating a conflict.
3. The instance releases the lock after it finishes.

How to Release the Lock When the Instance Terminates Suddenly

There are cases when an application terminates unexpectedly and you might think the lock will never be released. Luckily, Lease Blob has a timeout feature which means the lock has to be renewed before the timeout window elapses. This also means when the application terminates for any reason, the lock timeout window elapses and the lock is eventually released, so that another instance can run.

Mutex Implementation

The following C# class represents a distributed mutex using Azure Lease Blob

NOTE: The code above requires the WindowsAzure.Storage NuGet package.

The class DistributedMutex is instantiated with three parameters:

1. connectionString: This is the connection which looks like DefaultEndpointsProtocol=https;AccountName=myaccount01;AccountKey=7coPZqzh+jE0358oiU3Clwn9ym9L8C+LWb6NK1+4QzjzQK2gmuQ==;EndpointSuffix=core.windows.net
2. storageContainerName: Name of the container to use. This could be the application name (eg cleanup-tool). If this container does not exist, it will be automatically created. There are restrictions on container names, See Azure Documentation for more information. You can also share the same container name across different tools and applications.
3. key: Unique value that determines the tool running. This will be the blob name. See below.

For the key part, in order to only allow one instance of a tool called ImportEligibility, a key value of “ImportEligibility” can be used. However, there are cases when the same tool runs based on different parameters, for example, suppose the cleanup application runs against two different databases A and B, where the parameter databaseName is passed to the application. In this case, it is ok for one instance of A and one instance of B to run at the same time. The key values can be ImportEligibility-A and ImportEligibility-B for A and B, respectively.

There are also restrictions on the blob name. See Azure Documentation for more information.

Example Using DistributedMutex

The following example shows how to use the DistributedMutex class:

Summary

We used Microsoft Azure Lease Blob to solve our problem of controlling the number of instances being run at the same time. This solution can be enhanced by creating a semaphore instead of a mutex which allows an arbitrary number of instances instead of only 1.

For another use case please check out the Lease Blob plays a key role in implementing Azure WebJobs Singleton.

At Veyo, we are building a company of driven people that are empowered by technology to revolutionize healthcare logistics. Want to work at Veyo? Check out our jobs page.