Why we built our own k8s Redis operator — Part 1

At PayU, we used Redis within our production environment as an in-memory message queue for our internal microservices usage. Until 2018, before we migrated our entire production environment from Mesos-DC/OS to Kubernetes, we deployed Redis as a single node instance that was able to scale only vertically. When we migrated to Kubernetes, and as our external API traffic increased, we needed to make sure that our production system remained fully horizontally scalable without any exceptions. At that point, we realized that we needed to switch from a single Redis node deployment to a full Redis Cluster that will run on top of Kubernetes.

The challenges

Since Mesos-DC/OS, we are running all the infrastructure on top of AWS. Our production environment is running in the eu-central-1 (Frankfurt) region and it’s spread across 3 availability zones. We decided to use Amazon EKS so that Amazon would run the control plane for us, while we would only manage the worker nodes (i.e : data plane). During our design meetings for the Kubernetes migration, and as part of the lessons we learned from past experience, we decided that we needed to be prepared to lose the full availability zone in production and still make sure our system is fully functioning without having any data loss. This assumption alone has greatly affected the way we need to deploy our applications inside Kubernetes. For stateless applications deployments, the solution is pretty easy using k8s features like podAntiAffinity & topologyKey. For stateful applications (like Redis) things get more complex. Redis was the last stateful application to be migrated to k8s. By the time we started to work on moving Redis to k8s, our SRE team already had a lot of experience of running and managing stateful applications inside k8s. Our first obvious choice for managing the Redis cluster was to use the Kubernetes Stateful controller. After struggling with Redis cluster for a while, we understood that there are 2 key questions that could not be answered when using the stateful controller:

• How can we make sure that the Redis follower node is deployed on a different AZ than its corresponding leader node?
• When the Redis leader fails and his follower takes the lead, how can we automate the process of deploying a new Redis node that will join the current cluster and trigger CLUSTER MEET on all the current nodes?

In order to solve those questions by a human, this person would need to have deep knowledge of how the Redis system works and behaves, how to deploy it, and how to react if there are problems. But if we want all those problems to be solved automatically and not by humans, it means we need a specific k8s controller. and that means Kubernetes Operator

Searching for the solution

After understanding that we need an operator in order to manage and deploy the Redis cluster on Kubernetes, the first and obvious way was to use an existing and well-maintained open-source solution. By the time we started to look for an open-source solution, there were already some very good projects out there. We focused on 3 key features that are mandatory for us:

• Full AZ/Rack aware deployment. Leader & Follower can never be on the same AZ/Rack
• Auto recovery from all the scenarios humans can handle without data loss, like losing less than or equal to 33% of all the leaders or losing all the followers
• Horizontally scalable

After an in-depth search, we weren’t able to find any project that had all our mandatory features. Specifically, the AZ aware feature was missing from all the projects we found and the auto-recovery feature was not fully covered by most of them. At that point, we decided we would implement our own operator to have all those features combined.

The way to production

When we started writing our Redis operator we had to decide about the tools and the tech stack we were going to use:

1. The chosen programming language was Golang. It was the obvious choice since the entire Kubernetes ecosystem is written in Golang and it’s the official API client language
2. In order to facilitate the development process, we needed to deploy a local k8s environment on our local machines and develop the operator on top of it. The chosen tool for that was Kind — a cli tool for running local Kubernetes clusters using a Docker container.
3. For quick iterations on a code running in live containers of the Kind cluster, we first chose Telepresence but later on, we changed to Tilt. Tilt gives you smart rebuilds and live updates, so all the changes on your code are deployed quickly to Kind.

We decided that for the first stable version (1.0.0), the following features would be implemented:

• HA Setup: multiple nodes, each with multiple replicas
• AZ/Rack aware: HA node distribution able to use multiple availability zones
• Auto-recovery: Automatic recovery from multiple failure scenarios
• Rolling updates: Enables to update Redis pod settings gracefully

Next up

In part number 2 of this blog post, we will describe in detail how we implemented all the features and share some of the challenges we had along the way.

P.S
Contributions are more than welcome! You can check our wiki pages in the project for wanted/missing features. Don’t forget to star :)

https://github.com/PayU/redis-operator