GitOps Deployment and Kubernetes

Riskified developers deploy code to production multiple times a day, which requires us to have a robust deployment process. When we started migrating to K8s, the deployment process was a significant concern for us. We had to replace a trusted process built over time: a script that looped through the servers, pulled and started the new image, and waited for healthcheck. It was a simple process, but moving to K8s, we could no longer perform a loop. In K8s, you must update the deployment object with the new image tag.

We started designing a new deployment process by setting four goals we wanted to accomplish:

1. Easy to understand. The less magic, the better.
2. Declarative. We want to declare the new state of our service.
3. GitOps. Git is our single source of truth, and every process should start from a change in Git. For our deployment process, that means we declare our state in Git, and an automated process applies it to the cluster.
4. Observability. It should be easy to understand how each component of the application had been deployed, and monitor the status of the components at all times.

After evaluating a few tools and processes, we decided to combine Helm and ArgoCD. In this article, I’ll share with you how we’ve used these tools in our GitOps deployment, and list the benefits of the new process.

Deployment phases

Deployments can be split into two phases. The first starts with a commit/merge and ends with a new image in the registry. The image has passed all the tests and can be used in production. This phase is pure GitOps as it is triggered by a Git action.

The second phase starts with a new image in the registry and ends when all the servers are running the new version, and passing health checks.

GitOps: the second phase

The second phase was the focus of our efforts. Our first idea was to save the K8s objects in a Git repo and update them at the end of the CI, then trigger another action to apply the new update to the cluster. This will create a great GitOps declarative flow, but K8s objects are complex and we want our developers to manage them.

Helm

Helm is a package manager; it provides the ability to template K8s objects so that they could be reused. Using key-value files, users of the template can provide different values. With Helm, we created a set of generic templates that developers can use to quickly deploy their images to K8s, hiding the complicated parts of K8s and decreasing the learning curve for developers. You can read more about this in my previous blog post. Using this technique, our deploy process begins when the image tag value is changed, or when any other value changes.

Observability: ArgoCD

Helm introduces magic to the process. From a small set of values, it generates a deployment, a service, and a lot more objects. Magic makes it harder for developers to understand the production environment. Our fourth parameter should cope with this problem. Good observability can detangle some of the magic and make the process more understandable.

ArgoCD is a GitOps continuous delivery tool for K8s and had recently joined the Cloud Native Computing Foundation (CNCF). ArgoCD allows to continuously sync helm charts to the cluster. Among other features, ArgoCD has a rich UI that increases observability, displaying all the application components and their status in a single place.

In the above picture, you can see our rails-demo application. You can see the K8s components and their health status, and you can see that it is synced to the Git repo. The application was composed using our generic Helm chart. The developer enjoys a simple Helm chart while also getting a view of the result in production.

ArgoCD in practice

ArgoCD is kind of a reconciling loop from Git to K8s:

ArgoCD watches the Git repo and runs the Helm template; these templated files are then compared to the desired state in the cluster — this is the sync status in the application view. If there are differences, ArgoCD can apply the templated files and change the K8s desired state — this can be manual or automatic. ArgoCD also watches the live K8s objects and compares them to the K8s desired state — this is the health status in the application view.

No `$helm install`

ArgoCD doesn’t use `$helm install`. It is not a Helm rapper, but rather a declarative GitOps deploy tool. The way you construct the templated K8s objects in Git doesn’t need to be related to the way you deploy them to the cluster. We use Helm for templating, and we use ArgoCD for deploying and managing the cluster. Each tool is the best for its purpose. `$kubectl apply` is also the easiest way to change the desired state in the cluster. Using it helps us reduce the magic around the deployment process.

The complete deployment process

After publishing a new docker image, we trigger a script that commits the new tag to the application Helm chart and another script that forces sync in ArgoCD and waits for the ArgoCD application to return to a healthy status.

Summary

Migration to K8s comes with significant risk. The deployment process should reduce the risk and generate more confidence in the process. Using ArgoCD and Helm as described gave us the confidence we needed and reduced our concerns. Reducing magic and increasing observability can help you do better deploys.