How to Build a CI/CD Process That Deploys on Kubernetes and Focuses on Developer Independence

Published in

Riskified Tech

7 min readDec 2, 2020

Every Site Reliability Engineer or DevOps Engineer is familiar with one of the most painful areas in a CI/CD process: being the bottleneck for developers who want to build and deploy a new service.

Each CI/CD tool that comes out holds many options and details, and K8s is no less complicated. It’s irrational to expect the developer, who should focus on writing code, to follow it all.

So how can you build a proper CI/CD process that focuses on developer independence?

Or in other words:

How can you, as a DevOps Engineer, give the proper tools to the developers in your company, so they can build and operate their CI/CD processes on their own?

The CI/CD Process

Overview

Choosing proper CI/CD solutions raises lots of questions, whether the CI/CD process focuses on developer independence or not:

What platform can provide the best HA infrastructure?
What solutions provide the easiest and fastest ways to deploy new microservices, update existing microservices, or revert in case of failures?

These questions should be taken into account when choosing our CI/CD solutions.

Our CI/CD process includes the following steps (this is a general guideline that will be explained in detail below):

Developer pushes code to a GitHub repository
CircleCI (CI Platform) is triggered and a CircleCI Workflow starts running a CI process Following various workflow tests, it:
a. builds a Docker image and pushes it to our Docker container registry;
b. updates the service Helm Chart (in GitHub) — updates its value file to include the new Docker image tag (that we built in paragraph a);
c. syncs ArgoCD (CD Platform) with the new Helm Chart values to deploy changes in the k8s environment.
ArgoCD starts to apply new configurations in the k8s cluster

Let’s dive into the process, and show how we can create a CI/CD process that keeps the questions above in mind, while providing developer independence.

Creating a CI Process that engages developers

When you choose the right tool to make a CI process that focuses on developer independence, you should ask yourself:

What will make our developers engaged in the CI process?

How do I make our developers write, operate and manage their own CI process every time a new service is created, or a current service changes?

Among the various tools the industry offers for CI tools, you can find CircleCI, Jenkins, GoCD, Buddy, and more. Let’s take a deeper look…

CircleCI

Each CI tool offers its own advantages and disadvantages, and considering the different aspects of each tool, we’ll focus on CircleCI.

CircleCI is great for focusing on developer autonomy for various reasons.

CircleCI has a great developer-driven configuration: each service repository has its own config.yml file that includes the CI workflow details, which developers can easily edit and tweak.

As opposed to a big competitor of CircleCI — Jenkins — there are no centralized plugins that a DevOps Engineer needs to manage, which means less bottlenecks.

After configuring the CircleCI server to your needs, you can configure a trigger (i.e. GitHub action) that triggers the CircleCI workflow.

The CircleCI workflow is configured using a config.yml file that holds all the CI steps.

You can create a general template file, with general steps and guidelines that a developer will use as a reference when they write their own config.yml template file.

The config.yml template file resides in the service repository, so developers can have easy access to it.

So what does the developer need to do?

The developers can use the general template mentioned before, that has the general steps you decided on.

They’ll tweak the template file to match their service, and put it in their service repository.

The following is a snippet from a config.yml file that:

a. checks out code and builds&pushes a docker image to a docker registry;

b. triggers ArgoCD in order to deploy on k8s (explained below). In order to trigger ArgoCD with custom options, we created a custom image with a script that does it.

image: &image
  steps:
    - checkout
    - attach_workspace:
        at: .
    - setup_remote_docker
    - run:
        name: Build & push docker image
        command: |
          docker login
          docker build -t $REPO_NAME:$TAG -f $DOCKERFILE_PATH .
          docker push $REPO_NAME:$TAGdeploy: &deploy
  working_directory: .
  docker:
  - image: <k8s-argocd-sync custom image>:<tag>
  steps:
  - add_ssh_keys:
      fingerprints:
      - <>
  - run:
      name: Deploy Job (argoCD)
      command: |
        ./<deployment_script> -OPTIONS

In order to learn more about CircleCI and view full examples, you can visit CircleCI website.

Finally, all the developers need to do is to put a config.yml file in their repo that matches their service. That’s it — developers can now independently operate their own CI.

Make developers operate their CD Process

When you create a CD process, in addition to making the process secure, reliable, and stable, there are several more aspects regarding developer independence that should be considered. Some of them are:

How to make it easy to understand, so that the developer can build and apply k8s configurations without a full understanding of the architecture and the different resources in it
How to make fast rollouts, that developers can easily do on their own

Of course there are many more, but we will focus on answering these.

K8s include many options and commands that developers aren’t familiar with — and aren’t supposed to be.

So how do you make all your k8s templates and configurations as condensed as possible for developers, while maintaining all your environment default configurations?

The answer is simple — Helm Charts.

Helm Charts

Whether your deployment tool is ArgoCD, Jenkins, or something else — it is highly recommended to use Helm Charts with it.

With Helm Charts, you can create a collection of templates that include all of the resources and configurations for a new microservice — according to your standards and security aspects.
Then, you can easily change the values of the resources within the chart — via the values.yaml file.

Additionally, deploying the chart in different environments (production, staging, etc.) is much easier with different values.yaml files for each environment, and a global values.yaml file that will hold the common values for all the envs.

If you are not familiar with Helm Charts, I recommend reading more about it on the official Helm website.

At Riskified, we created our own Parent Chart that holds all of the resources that a microservice might need, and the default values. Every microservice has a Child Chart that is dependent on the Parent Chart we manage. The Child Chart requires the Parent Chart via the requirements.yaml file.

Of course any developer can add other values to the values.yaml file, in order to override the default values (like port field) and control what resources will be created.

The only values that our Chart requires, is the name of the microservice and its image tag. So in the end, what the developer and the CI/CD process have to fill out, is just the following:

The Parent Chart will create resources, such as deployment, service, ingress, serviceMonitor, and more, with our default values.

To summarize — how does the Parent Chart boost developer autonomy?

Developers do not need to familiarize themselves with all of the k8s resources behind their microservices
Microservices are deployed with default values that can be overridden, if needed
All microservices are deployed according to the company standards, which we control via the Parent Chart

We’ve also created a Parent Helm Chart that holds Istio resources, in case a service needs to be shared externally, which the microservice Child Chart can require in addition to the Parent Chart mentioned above.

We manage our Charts in the GitHub repository with their own Changelog. When Riskified developers have a requirement that those charts do not fulfill — we add it to the Helm Charts we manage, and release a new version.

ArgoCD

So how can developers actually apply their configurations easily?

ArgoCD is a great option for this. Why?

Mainly because of its great UI — with representation for every k8s object the service uses and its state, easy way to check logs, ability to see differences between applied and soon-to-be applied configurations, and apply changes to k8s objects (or in ArgoCD terms — Sync).

ArgoCD enhances the developer experience with k8s and allows developers to deploy easily, without having to be familiar with various k8s commands.

Wrapping up

So how does it all come together?

At the end of the CI process, the CI changes the corresponding value.yml file in GitHub with the new docker image tag that has been built (as part of the process).

CircleCI triggers ArgoCD, and asks it to sync the new configuration in the GitHub repository (new image tag) and apply the change in the k8s cluster.

And that’s it, we are done!

Now a developer has everything they need to operate their own CI/CD process easily -

they can edit their own CI process, and tweak it to match their service
they can deploy easily by applying k8s configurations via ArgoCD

The developer can proceed with writing code, without bottlenecks — and you have more time to think about the next cool tool you want to use! 😌