Deploy using Helm Charts on GKE: Continuous Delivery Pipeline using Google Cloud Build & Google Cloud Deploy

This blog discusses the steps required to create a continuous integration and continuous delivery (CI/CD) pipeline for multiple Google Kubernetes Engine (GKE) clusters for multiple environments using Helm, Cloud Build, and Cloud Deploy. The resulting delivery pipeline will be as shown in the diagram below.

CICD pipeline

This pipeline will accomplish the following:

1. A pipeline that will automatically build and deploy a new version of the Helm chart to GKE whenever a change is made to the source code.
2. A pipeline which can deploy to multiple environments based on natural progression of the feature.
3. A pipeline that will automatically roll back to the previous version of the Helm chart if the new version fails to deploy successfully.

Services & Tools

Let’s look at the services and tools used to build these pipelines

Helm is a package manager for Kubernetes. It provides a way to install and manage Kubernetes applications and services. Helm charts & templates can be used to deploy applications on multiple clusters with associated configurations, bringing uniformity and consistency in deployment across environments.

Cloud Build is a continuous integration, delivery, and deployment platform that lets you run consistent automated builds on Google Cloud. It can import source code from a variety of repositories and then build and deploy your code to a variety of targets.

Cloud deploy can help us create a pipeline to define release processes through environments such as dev, stage and production. It provides one step promotion or rollback of releases. Cloud Deploy uses Skaffold to automate the deployment of Kubernetes applications. It uses Skaffold to create a pipeline that builds the application, tests it, and then deploys it to Kubernetes.

Adding Configurations to Code

Let us start by examining the code structure and manifests that are essential to create and run the pipeline.

You can also access the full code repository here.

Code Structure

skaffold.yaml

apiVersion: skaffold/v2beta16
kind: Config
build:
  tagPolicy:
    customTemplate:
     template: "latest"
  artifacts:
  - image: helm-cicd-image
    context: ./
profiles:
- name: dev
  deploy:
    helm:
      releases:
      - name: devrelease
        chartPath: helm/dev
        valuesFiles:
         - helm/dev/values.yaml
- name: staging
  deploy:
    helm:
      releases:
      - name: stagerelease
        chartPath: helm/stage
        valuesFiles:
         - helm/stage/values.yaml
- name: prod
  deploy:
    helm:
      releases:
      - name: prodrelease
        chartPath: helm/prod
        valuesFiles:
         - helm/prod/values.yaml

This file contains build and deploy sections. The build section uses a Dockerfile to build and upload images to the Artifact Registry. The profile section has different deploy configurations for each environment which are referenced while a release is applied to a GKE cluster. Skaffold uses the templates in the helm/templates folder to generate Kubernetes manifests and then replaces the placeholder variables with values fromvalues.yaml for each environment.

Helm Charts & values.yaml

Helm Files

The templates folder contains templates for Kubernetes constructs, such as deployments and services. Variables in these templates are replaced by the values provided in values.yaml.

Cloud Deploy files

Cloud Deploy files

pipeline.yaml

apiVersion: deploy.cloud.google.com/v1
kind: DeliveryPipeline
metadata:
  name: helm-cicd-pipeline
  labels:
    app: helm-cicd
description: helm-cicd delivery pipeline
serialPipeline:
  stages:
  - targetId: dev
    profiles:
    - dev
  - targetId: staging
    profiles:
    - staging
  - targetId: prod
    profiles:
    - prod

This YAML files defines the various stages of deployment using a serial pipeline.

prod.yaml

apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
  name: prod
  annotations: {}
  labels: {}
description: prod
requireApproval: true
gke:
  cluster: projects/<project-id>/locations/<zone>/clusters/helm-cd-testing-prd

The requiredApproval flag in the prod.yaml file enables a manual approval check for deployments to the production environment. The development release on the development cluster will be automatically performed once the pipeline has been created as part of the initial build.

cloudbuild.yaml

substitutions:
  _REGION:<zone>
steps:
- name: 'gcr.io/k8s-skaffold/skaffold'
  entrypoint: 'sh'
  args:
  - -xe
  - -c
  - |
    # Build and push images
    skaffold build --file-output=/workspace/artifacts.json \
                   --default-repo=${_REGION}-docker.pkg.dev/$PROJECT_ID/helm-cicd-repo \
                   --push=true
- name: 'google/cloud-sdk:latest'
  entrypoint: 'sh'
  args:
  - -xe
  - -c
  - |
    gcloud config set deploy/region ${_REGION}
    sed -i s/PROJECT_ID/$PROJECT_ID/g deploy/*
    gcloud deploy apply --file deploy/pipeline.yaml
    gcloud deploy apply --file deploy/dev.yaml
    gcloud deploy apply --file deploy/staging.yaml
    gcloud deploy apply --file deploy/prod.yaml
    gcloud deploy releases create rel-${SHORT_SHA} \
                        --delivery-pipeline helm-cicd-pipeline \
                        --description "$(git log -1  --pretty='%s')" \
                        --build-artifacts /workspace/artifacts.json \
                        --annotations "commit_ui=https://source.cloud.google.com/$PROJECT_ID/helm-cicd-demo/+/$COMMIT_SHA"
artifacts:
  objects:
    location: 'gs://$PROJECT_ID-gceme-artifacts/'
    paths:
    - '/workspace/artifacts.json'
options:
  machineType: E2_HIGHCPU_8
timeout: 3600s

This file integrates the function of all other files and is used to create required resource on Google Cloud. There are two steps defined in this file.

• The first step reads the skaffold.yaml files and builds the Docker image from the source code, then uploads it to the Artifact Registry.
• The second step reads the Cloud Deploy manifests and creates a code deployment pipeline and release for initial development environment.

Create Cloud Build Trigger

We will create a trigger in Cloud Build that will run whenever any code is pushed to the repository. This trigger will pull the code from source repository and read cloudbuild.yaml to run build steps. The build will first create a Docker image, upload it to Artifact Registry, and then create a Cloud Deploy pipeline. It will also create a release which will deploy code on development cluster.

Cloud build trigger

You may now either execute the build manually by console or push a code change to source repository which will trigger it.

Cloud Deploy Pipeline

Upon successful completion of cloud build trigger, you can proceed to Cloud Deploy to operate and manage releases for different environments.

Cloud Deploy pipeline

Now your automated deployment pipeline with Helm Charts is ready. You can either promote a change to next environment or roll back the change using controls available on the console. You can also look at Deployment failure rate to determine your change failure rate.

Apart from serial pipeline you can create different strategies like standard or canary which enables you to automate the verification of feature before it is released to masses.

Please refer to following links to learn more about these services

• Using Skaffold with Cloud Deploy
• Using Helm & Kustomize with Skaffold & Cloud deploy
• Code repository for this blog
• Quickstart to Cloud Deploy