CI/CD with Kubernetes and Helm

In this blog, I will be discussing about implementation of CI/CD pipeline for microservices which are running as containers and being managed by kubernetes and helm charts

Note: Basic understanding of docker, kubernetes, helm and Jenkins is required. I will discuss about the approach but will not go deep into its implementation. Please refer original documentation for deeper understanding about these technologies.

Before we start designing the pipeline for kubernetes and helm based architecture, few things we need to consider are

1. Helm chart code placement
2. Versioning strategy
3. Number of environments
4. Branching model

Helm Chart Code Placement

Helm chart code can be placed along with code or in a separate repo but we ended up in keeping it along with source code. Points which made us inclined to this decision were

a. If developer add some variables which needs to be updated in Kubernetes config map or secrets, this knowledge needs to be shared with the owner of helm chart repo. This will add make things complex and buggy

b. CI scripts also needs to be updated so it can interact with different repo either by lambda functions or some other ways

c. Microservices tends to grow into very large number and if we keep helm chart in separate repo we would be requiring 2 times the original repos

Versioning Strategy

Whenever new build is deployed in preview or review version has to be updated so it can reflect what new things it have. Apart from many options we have like using third party service or custom scripts for this, we can also use simple yet powerful feature of git, that is describe command

Git describe is useful command, more can be read from

https://git-scm.com/docs/git-describe

This command finds the most recent tag and return it with adding suffix on it with number of commits done after the tag and abbreviated object name of the most recent commit

Example: If tag is 1.0.1 and we have 10 commits after it, git describe will return us 1.0.1–10-g1234

This number can be used as version

Environments

We need multiple environments which can be used for development, testing, staging, production etc, for shipping our code to production. While we can create as many environments we may need but they does make the development life cycle bit complex. Generally, three types of environments works in most of the cases so unless there is very specific need we should stick with them.

a. Preview environment, where developers can quickly deploy and test their changes before raising pull request for review

b. Staging environment, this is kind of pre-production environment where reviewer host the changes for final review with different stakeholders

c. Production environment, as the name suggests, this is where our running build lives

In kubernetes, there is concept of namespaces which can give us isolated environments within same cluster, similar to what VPC does in AWS. So instead of creating different cluster for each environments we can same cluster and separate them by namespaces ( for production we can have separate one)

Branching Model

Vincent git flow model has been widely used in industry, we would be listing out few details in considering this model .

Below is the diagram for flow which has been explained in detail

Stage 1: Developer make changes in code and test it locally and then push the changes to git which will trigger the Jenkins pipeline. Pipeline will build the new docker image and tags it with output of git describe command and pushes it to docker registry. Same tag is updated in the helm chart code and new chart is pushed as well into chart museum. In the end deployment is done by running the helm update command against new version of chart by Jenkins in development namespace. Finally, developer raises the pull request

Stage 2: Reviewer reviews the code and accept it once satisfied and merge into release branch. Version number also gets updated by removing all the suffix by incrementing it based on its major/minor/path release, new docker image is created with new tag and helm chart also get updated with same tag. Jenkins then run the helm update in staging namespace so it can be verified by others

Stage 3: If staging looks good, last step would be to push the changes into production. Jenkins pipeline wont create new image or chart in this case, it only uses the last helm chart version and update the chart in the production namespace

Overall, CI/CD in general is complex exercise if goal is to make it fully automated, though this articles doesn’t cover lot of details but information shared can be useful while designing your pipeline. If you are doing this over AWS, spot instances can be used for preview environment to stay cost effective