Image for post
Image for post
How to build a CI/CD process that deploys on Kubernetes and focuses on developer independence

How to Build a CI/CD Process That Deploys on Kubernetes and Focuses on Developer Independence

Liron Cohen
Dec 2, 2020 · 7 min read

Every Site Reliability Engineer or DevOps Engineer is familiar with one of the most painful areas in a CI/CD process: being the bottleneck for developers who want to build and deploy a new service.

Each CI/CD tool that comes out holds many options and details, and K8s is no less complicated. It’s irrational to expect the developer, who should focus on writing code, to follow it all.

So how can you build a proper CI/CD process that focuses on developer independence?

Or in other words:

How can you, as a DevOps Engineer, give the proper tools to the developers in your company, so they can build and operate their CI/CD processes on their own?

The CI/CD Process

Choosing proper CI/CD solutions raises lots of questions, whether the CI/CD process focuses on developer independence or not:

What platform can provide the best HA infrastructure?
What solutions provide the easiest and fastest ways to deploy new microservices, update existing microservices, or revert in case of failures?

These questions should be taken into account when choosing our CI/CD solutions.

Image for post
Image for post
Riskified’s CI/CD Process

Our CI/CD process includes the following steps (this is a general guideline that will be explained in detail below):

  1. Developer pushes code to a GitHub repository
  2. CircleCI (CI Platform) is triggered and a CircleCI Workflow starts running a CI process Following various workflow tests, it:
    a. builds a Docker image and pushes it to our Docker container registry;
    b. updates the service Helm Chart (in GitHub) — updates its value file to include the new Docker image tag (that we built in paragraph a);
    c. syncs ArgoCD (CD Platform) with the new Helm Chart values to deploy changes in the k8s environment.
  3. ArgoCD starts to apply new configurations in the k8s cluster

Let’s dive into the process, and show how we can create a CI/CD process that keeps the questions above in mind, while providing developer independence.

When you choose the right tool to make a CI process that focuses on developer independence, you should ask yourself:

What will make our developers engaged in the CI process?

How do I make our developers write, operate and manage their own CI process every time a new service is created, or a current service changes?

Among the various tools the industry offers for CI tools, you can find CircleCI, Jenkins, GoCD, Buddy, and more. Let’s take a deeper look…

Each CI tool offers its own advantages and disadvantages, and considering the different aspects of each tool, we’ll focus on CircleCI.

CircleCI is great for focusing on developer autonomy for various reasons.

CircleCI has a great developer-driven configuration: each service repository has its own config.yml file that includes the CI workflow details, which developers can easily edit and tweak.

As opposed to a big competitor of CircleCI — Jenkins — there are no centralized plugins that a DevOps Engineer needs to manage, which means less bottlenecks.

After configuring the CircleCI server to your needs, you can configure a trigger (i.e. GitHub action) that triggers the CircleCI workflow.

The CircleCI workflow is configured using a config.yml file that holds all the CI steps.

You can create a general template file, with general steps and guidelines that a developer will use as a reference when they write their own config.yml template file.

The config.yml template file resides in the service repository, so developers can have easy access to it.

So what does the developer need to do?

The developers can use the general template mentioned before, that has the general steps you decided on.

They’ll tweak the template file to match their service, and put it in their service repository.

The following is a snippet from a config.yml file that:

a. checks out code and builds&pushes a docker image to a docker registry;

b. triggers ArgoCD in order to deploy on k8s (explained below). In order to trigger ArgoCD with custom options, we created a custom image with a script that does it.

image: &image
steps:
- checkout
- attach_workspace:
at: .
- setup_remote_docker
- run:
name: Build & push docker image
command: |
docker login
docker build -t $REPO_NAME:$TAG -f $DOCKERFILE_PATH .
docker push $REPO_NAME:$TAG
deploy: &deploy
working_directory: .
docker:
- image: <k8s-argocd-sync custom image>:<tag>
steps:
- add_ssh_keys:
fingerprints:
- <>
- run:
name: Deploy Job (argoCD)
command: |
./<deployment_script> -OPTIONS

In order to learn more about CircleCI and view full examples, you can visit CircleCI website.

Finally, all the developers need to do is to put a config.yml file in their repo that matches their service. That’s it — developers can now independently operate their own CI.

When you create a CD process, in addition to making the process secure, reliable, and stable, there are several more aspects regarding developer independence that should be considered. Some of them are:

  • How to make it easy to understand, so that the developer can build and apply k8s configurations without a full understanding of the architecture and the different resources in it
  • How to make fast rollouts, that developers can easily do on their own

Of course there are many more, but we will focus on answering these.

K8s include many options and commands that developers aren’t familiar with — and aren’t supposed to be.

So how do you make all your k8s templates and configurations as condensed as possible for developers, while maintaining all your environment default configurations?

The answer is simple — Helm Charts.

Whether your deployment tool is ArgoCD, Jenkins, or something else — it is highly recommended to use Helm Charts with it.

With Helm Charts, you can create a collection of templates that include all of the resources and configurations for a new microservice — according to your standards and security aspects.
Then, you can easily change the values of the resources within the chart — via the values.yaml file.

Additionally, deploying the chart in different environments (production, staging, etc.) is much easier with different values.yaml files for each environment, and a global values.yaml file that will hold the common values for all the envs.

If you are not familiar with Helm Charts, I recommend reading more about it on the official Helm website.

At Riskified, we created our own Parent Chart that holds all of the resources that a microservice might need, and the default values. Every microservice has a Child Chart that is dependent on the Parent Chart we manage. The Child Chart requires the Parent Chart via the requirements.yaml file.

Of course any developer can add other values to the values.yaml file, in order to override the default values (like port field) and control what resources will be created.

The only values that our Chart requires, is the name of the microservice and its image tag. So in the end, what the developer and the CI/CD process have to fill out, is just the following:

Image for post
Image for post
Helm chart mandatory values

The Parent Chart will create resources, such as deployment, service, ingress, serviceMonitor, and more, with our default values.

To summarize — how does the Parent Chart boost developer autonomy?

  • Developers do not need to familiarize themselves with all of the k8s resources behind their microservices
  • Microservices are deployed with default values that can be overridden, if needed
  • All microservices are deployed according to the company standards, which we control via the Parent Chart

We’ve also created a Parent Helm Chart that holds Istio resources, in case a service needs to be shared externally, which the microservice Child Chart can require in addition to the Parent Chart mentioned above.

We manage our Charts in the GitHub repository with their own Changelog. When Riskified developers have a requirement that those charts do not fulfill — we add it to the Helm Charts we manage, and release a new version.

So how can developers actually apply their configurations easily?

ArgoCD is a great option for this. Why?

Mainly because of its great UI — with representation for every k8s object the service uses and its state, easy way to check logs, ability to see differences between applied and soon-to-be applied configurations, and apply changes to k8s objects (or in ArgoCD terms — Sync).

Image for post
Image for post
(picture taken from ArgoCD website)

ArgoCD enhances the developer experience with k8s and allows developers to deploy easily, without having to be familiar with various k8s commands.

Wrapping up

So how does it all come together?

At the end of the CI process, the CI changes the corresponding value.yml file in GitHub with the new docker image tag that has been built (as part of the process).

CircleCI triggers ArgoCD, and asks it to sync the new configuration in the GitHub repository (new image tag) and apply the change in the k8s cluster.

And that’s it, we are done!

Now a developer has everything they need to operate their own CI/CD process easily -

  • they can edit their own CI process, and tweak it to match their service
  • they can deploy easily by applying k8s configurations via ArgoCD

The developer can proceed with writing code, without bottlenecks — and you have more time to think about the next cool tool you want to use! 😌

Riskified Technology

Engineering, Data Science, Architecture, Scaling and more

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store