Kubernetes: Perform a task only in one instance of a multi-instance microservice deployment

During application development, it often happens that a routine of a program has to be executed only once, even in a multi-instance environment. Just a few examples for such functions are:

1. Initialization routines
2. Database Migrations
3. Scheduling cronjobs (tasks which are rerun according to a defined schedule, but should only be run in a single instance of the deployment)
4. Establishing a persistent connection to an external service and listen for events

Implementing this behavior in a single instance deployment is dead simple. The regular program flow will execute the necessary steps of initialization, database migrations, establishing connections and scheduling cron jobs.

The task at hand is much harder to solve, thinking about a multi-instance deployment environment like Kubernetes. Deploying multiple replicas of the same software results in all instances running the initialization routines, scheduling the same cronjobs and establishing a connection to an external service. The downsides of this are dependent on the executed routine itself, but are ranging from just a negative performance impact because of the unnecessary work which is done, to database inconsistencies or hitting rate limits of third-party integrations. Using multiple replicas, those functions now need to be executed conditionally, if, and only if, no other instance did not already run them. The execution needs to be synchronized across all replicas.

Therefore, the answer to the question: “How can I perform a task only in a single instance of a multi-instance microservice deployment?” involves developing and deploying at least another service, which keeps track of synchronization and is in charge of invoking the mentioned routines only in a single instance of the deployment.

If the application is running inside a Kubernetes Environment, this behavior can be achieved without the need of deploying another software to the infrastructure. Kubernetes itself is capable to handle the synchronization of the replicas.

Inside this article, I will present two different approaches how to utilize Kubernetes to synchronize replicas.

StatefulSets

The StatefulSet approach is the simpler and preferable approach for most applications, which require simple synchronization management. Using Kubernetes Stateful sets, each instance will get an ordinal index which persists over application restarts [1]. For example, defining a StatefulSet with the application name web the instances will be named web-0 ,web-1 ,web-2 , … The name of the instance can be injected inside the pod as an environment variable and the index can be extracted.

The application can use the POD_NAME environment variable, to determine, if it is the leader.

Advantages

1. Simple to set up
2. The execution flow is easy to follow

Disadvantages

1. Slower deployment process, since Kubernetes will wait for the previous replicate to be running before starting the next one.
2. There can only be a single leader, even if there are numerous tasks, which should be executed once but in parallel (Leader Election might not be the appropriate pattern here anyway. Message-Broker should be preferred for those tasks.)
3. Error handling, if the leader fails, has to be implemented manually or concepts like Health checks have to be utilized.
4. A headless service needs to be created
5. StatefulSets do not provide any guarantees on the termination of pods when a StatefulSet is deleted. To achieve ordered and graceful termination of the pods in the StatefulSet, it is possible to scale the StatefulSet down to 0 prior to deletion.

See more limitations inside the Kubernetes Documentation about StatefulSets.

Leader Election

Leader election describes the process of selecting a single instance of an application as the leader inside a multi-instance deployment environment.

Coordinate the actions performed by a collection of collaborating instances in a distributed application by electing one instance as the leader that assumes responsibility for managing the others. This can help to ensure that instances don’t conflict with each other, cause contention for shared resources, or inadvertently interfere with the work that other instances are performing.

- Microsoft Azure Documentation (https://docs.microsoft.com/en-us/azure/architecture/patterns/leader-election)

The Kubernetes API provides a simple leader election based upon a distributed lock resource. All replicas are candidates inside a race to become the leader, by attempting to mark a distributed resource with their identifier. Kubernetes itself guarantees, that only a single instance will win this race and becomes the leader. Once the election finished, the leader periodically pings the Kubernetes API to renew its leader position. All other candidates are periodically endeavoring to lock the resource themselves. This ensures, that, if the leader fails, a new leader is elected quickly [2].

Advantages

1. The programming pattern is not limited to Kubernetes, but can be used in all environments. Nevertheless, the provided sample implementation uses the Kubernetes API and does not easily translate to other environments.
2. The approach is more error resistant since a failover is built in.
3. Multiple locks can be defined. There can be multiple leader instances which are responsible for a subset of tasks.

Disadvantages

1. The used state variable has to be defined using reactive programming patterns like Event-Emitters, which ensures, that the program reacts to any changes. This increases the complexity of the program.
2. Increases the workload on the Kubernetes controller.

Implementation in Golang

The implementation of a leader election in GolangI with Kubernetes can be done using the k8s.io packages. It consists of the LeaderElectionstruct (Event-Emitter) and multiple listeners. This section includes example source code, showing how to implement a leader election using the Kubernetes API.

A new leader election object gets created using the variables:

• podName : provide the name of the pod or any random string. It is just necessary, that the podName variable differs between all deployed instances.
• lockName : The name of the lock, the program is attempting to acquire leadership over.
• namespace : The namespace the pod is running inside.

The listener is a wrapper struct around a single function, which should only get executed by the leader. A simple Listener implementation might look like this. Its purpose is, to encapsulate the isRunningstate. This simplifies the process of defining idempotent functions.

Finally, the usage of the defined structs and methods is as simple as writing a single function, which listens to the cancel channel and executes the program logic.

The last step is the deployment of the application. Since the application uses Kubernetes resources to handle the leader election process, the pod requires a new set of permissions. Kubernetes does use Role-Based-Access-Control, therefore, a new role and a new service account need to be created, which gives access to the leasesresource.

Conclusion

Kubernetes does provide StatefulSets, which makes it easy to implement tasks, that should only be executed inside a single instance of a multi-instance deployment. This approach is sufficient for a big range of applications and use cases. If a more error resistant approach is needed and a faster deployment, the leader election pattern might be more suitable. Some Kubernetes knowledge is required, to create and set up the different synchronization approaches.

If you enjoyed this article and would like to learn more about us, visit: l3montree.com

Resources

[1] https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/

[2] https://kubernetes.io/blog/2016/01/simple-leader-election-with-kubernetes/