A Simple Active/Passive Cluster Implementation With Redis

We, as developers, want our applications to be up all the time. This is actually a common mission where redundancy and horizontal scalability is in place; so we have just decided to distribute the same application to different environments. So far so good.

However, distributing the same process in different environments may be a problem when it comes to some tasks which should not be executed concurrently on more than one instance of the application. In this post, I propose a simple approach to overcome this issue. My example will be built around a cron job scheduler job which will be problematic in case of two(or more) occurences of the same jobs are executed concurrently. In a cluster, same problem may come up in different shapes and there are various alternative solutions. To keep our example simple and lean, there will be a single job in our environment.

Specifications Of The Cron Scheduler

1. Every first minute of an hour, a batch data transfer task is executed.
2. Same scheduler will be installed to two different environments to preserve redundancy.
3. It is not reasonable when two concurrent tasks of the job are executed.

As scheduler is installed to different environments, following problems need to be addressed:

1. Which host will handle the responsibility of executing the job? The host, which is in charge, is the master.
2. What happens if the master dies or becomes unresponsive? How other pending host will be activated as master (which is called slave in the rest of this post)

Choosing a Master

My approach here will be, storing a mutual resource among all cluster instances. Whoever succeeds acquiring the mutex, will perform as master host.

Why Redis? Redis is a well-known key value store and widely used in system design for several years. It support high availability and has a configurable persistence system. However, same mission can be achieved by several other tools like Consul, Zookeeper or simply a database. I felt more comfortable with Redis and that’s the motivation behind my choice. If you wish to use another tool, I wouldn’t expect applied principles to change.

Here are our options with Redis:

1. Redlock algorithm described here in detail https://redis.io/topics/distlock
2. SETNX command which is an optimistic concurreny supported key declaration.
3. SET command with some LUA scripts.

Here, my choice will be using Redlock algorithm for following reasons:

1. It supports high availability (Although there are criticisms)
2. There is an awesome library for C# (RedLock.Net) and it has out-of-box resiliency features

Let’s Start Building Our Application

Initially we have to initialize a RedLockFactory instance. Our strategy will be to try to acquire red lock reference initially.

However, if the application instance fails to acquire the lock, it will periodically check if it can get the lock to claim a master role. Just not to code this functionality in every job separately, a RedLock wrapper (called LockReference) can be introduced.

Our sample job is responsible for tracking availability status. If the application instance doesn’t hold a valid lock, it will try to claim lock via LockReference wrapper.

Full source code can be found here: https://github.com/kutluarasli/cron-job-cluster-sample

In this sample application, I tried to keep every aspect as simple as possible; However, a real world example may be more sophisticated than example code. Anyway, I would not expect basic principles to change. These are:

1. RedLock ensures only one instance of the application holds a valid lock at the same time.
2. When the master instance dies, lock is released once the eviction period is over.
3. As mutual resource is set to free, a running instance of the application would acquire the lock and promote itself as master.

I created a library on GitHub to perform these tasks. It is not tested on production yet but you might want to follow it for further developments.

https://github.com/kutluarasli/minilock