CI/CD pipeline for Google Kubernetes Engine - full GCP

9 min readJun 18, 2022

disclaimer : all my writings, opinions and thoughts are mine.

Introduction

In this article I will guide you through the steps to deploy a very simple CI CD pipeline for Google Cloud Kubernetes Engine with a production environment only. I won’t use Terraform for this tutorial.

Why this article : GCP Certifications

As of writing this, I just joined Google Cloud !!!! All content in this article is my own.

I am not yet GCP certified and I wanted to share with you my path and labs to get certified and what better way than practicing and getting our hands dirty ?
This article begins with a very simple architecture that I will improve and share with you on medium over the time until we eventually get a final complex architecture.

Architecture Overview

As you already know, It is highly recommended to have at least a development environment instead of releasing directly into production environment.
I wanted to keep it simple with a single environment in order to let you assimilate the stuff and then get it yourself to the next level by customising it for your use case. Plus, I won’t cover container image promotion strategy since we deploy directly into production environment.

At the end of this guide, you will be able to continuously release new version of your app to your k8s prod cluster.

The following pipeline relies only on GCP services, up to you to modify accordingly if you use Gitlab or whatever :

Steps involved :

You commit/push code change on your source code repo
It triggers the pipeline automatically
Cloud build package and builds your docker image
It stores the new built image in Artifact Registry
Code Deploy pull the image and deploys the release to your GKE cluster

Lets’s breakdown each components to understand what’s going on :

GitHub is our source code repository , you know it already.
When you update your code on Github then it will automatically trigger Cloud Build.
I’ll show you how to integrate it with Google Cloud Build in order to trigger our pipeline.

Google Cloud Build is the managed build and deploy service, meaning you won’t have to manage and patch servers or any headaches Sysops stuff.
You just give Cloud Build a YAML file instructing to run some commands.
Of course, under the hood, Cloud Build uses a container image to run your command that is why you need to choose an appropriate container image.

Google Cloud Artifact Registry is a private container registry for storing our docker image.

Google Cloud Deploy will deploy our new container onto our Kubernetes cluster. Behind the scenes it uses Cloud Build and also Skaffold to set Kubernetes Deployments and expose Services on the cluster.

Google Cloud Kubernetes Engine is the managed Kubernetes service meaning you won’t have to manage VMs and focus and what’s more important to you. Of course it uses the underlying Google Compute Engine service.

Why CI/CD

I hope you are already familiar with CI CD and that your are not manually uploading your code directly on your production servers (lol)

However, integrating our code changes in a continuous and secure way (CI) and deploying automatically to our environment (CD) is today a very well known best practice.

If you are not familiar with CI CD then I suggest you go check at this very interesting article

Why Kubernetes

Kubernetes was initially developed by Google for internal use and then Google released it to let it be open source back in 2014.

Kubernetes manage and run your containers in a scalable and consistent way and ideal for building microservices.

Containers are lightweight compared to VM and let you bundle and pack your app in order to help you scale easily your workload. Plus, you can run your container on any machine or environment bringing high level of portability and for your app. hopefully you know it already.

Google Kubernetes Engine (GKE) is the Google managed Kubernetes service which comes into flavours :

Standard where you need to care about nodes and custom configuration and managed scalability - you pay for the underlying compute resources
Autopilot fully managed where control and data plane are GCP managed meaning your benefits from Google’s SREs best practices - you pay only for the pods you use

If you want to learn more about differences between GKE standard and autopilot you can read this article and go to Google Cloud documentation here.

Why Google Cloud

Google Cloud is one of the best cloud providers with multiple regions worldwide
Sustainability : Google Cloud is the cleanest cloud provider
Google Cloud is no-lock-in cloud provider because it mainly relies on open source technologies like Kubernetes
Google Cloud philosophy is to let you control your workload and helps you go multi-cloud if necessary by not locking you with proprietaries technologies
Kubernetes among many other tech was initially developed by Google before becoming open-source
Security with encryption at rest by default
Reliability is Google Cloud first concern with multiple redundant zones in each GCP regions and many other security practices
Google Cloud network IP backbone is highly available, with many PoPs (Point Of Presences), with low latency redundant routes
You can choose to use GCP “premium tier” network meaning that you stay on GCP’s Autonomous System — with a very reasonable extra price compared to “standard tier”
GCP console is the most easy to use and very well integrated with the GCP CLI (gcloud)
Cost transparency and predictability : unlike many other cloud providers, when you select workloads to launch then GCP shows you what it will cost you
Consistency and consumer centric : if you build app for consumers you can integrate consistently with Google “legacy” services like Google Search or Ads

Our hands-on code

Github repo

https://github.com/samy-fadel/gcp-ci-cd-gke

Base image and app source code

We will use a very simple sample code for our Webapp to be deployed on a Kubernetes cluster. It will be a very simple static web page served by an Apache webserver (httpd docker image).

The code

deployment.yaml specifies the states you want for your workload. For example, number of replicas, your container image and your deployment name.
service.yaml exposes your workload on a specific port (HTTP 80 in our case)
cloudbuild.yaml describes the command you want Google Cloud Build to execute. You could execute the commands from your laptop but if you want to replicate and automate this then Cloud Build is the one to choose instead of using a script and running it from a VM you would have to manage patch etc…
clouddeploy.yaml describes your delivery pipeline and the target cluster on which to deploy your container image
Dockerfile describes how to build your custom container image from apache webserver and index.html
skaffold.yaml describing where are located our manifest files (deployment.yaml & service.yaml). Here don’t use Skaffold’s build deploy capabilities locally but Google Cloud Build and Cloud Deploy
index.html your custom app code, in our case a sample HTML code

Hands-on !

In the GCP console create a new GCP project and copy to clipboard your project ID
Enable services APIs (by default services API are disabled on GCP when you create a new project)

gcloud services enable cloudbuild.googleapis.com clouddeploy.googleapis.com artifactregistry.googleapis.com container.googleapis.com

Open Cloud Shell terminal and set an environment variable in order to save you some time instead of copying pasting your own project ID in each of the below commands :

PROJECT_ID=paste here your project ID

Verify your environment variable has correctly been set by running to display your project ID :

echo ${PROJECT_ID}

Add “clouddeploy.jobRunner” permission on your compute service account in cloud shell terminal with :

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
    --member=serviceAccount:$(gcloud projects describe ${PROJECT_ID} \
    --format="value(projectNumber)")-compute@developer.gserviceaccount.com \
    --role="roles/clouddeploy.jobRunner"

Add “container.developer” permission on your compute service account in cloud shell terminal with :

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
    --member=serviceAccount:$(gcloud projects describe ${PROJECT_ID} \
    --format="value(projectNumber)")-compute@developer.gserviceaccount.com \
    --role="roles/container.developer"

Add “artifactregistry.admin” permission on your Cloud Build service account in cloud shell terminal with :

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
    --member=serviceAccount:$(gcloud projects describe ${PROJECT_ID} \
    --format="value(projectNumber)")@cloudbuild.gserviceaccount.com \
    --role="roles/artifactregistry.writer"

Add “artifactregistry.reader” permission on your compute service account in cloud shell terminal with :

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
    --member=serviceAccount:$(gcloud projects describe ${PROJECT_ID} \
    --format="value(projectNumber)")-compute@developer.gserviceaccount.com \
    --role="roles/artifactregistry.reader"

Add “clouddeploy.releaser” permission on your Cloud Build service account in cloud shell terminal with :

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
    --member=serviceAccount:$(gcloud projects describe ${PROJECT_ID} \
    --format="value(projectNumber)")@cloudbuild.gserviceaccount.com \
    --role="roles/clouddeploy.releaser"

Add “clouddeploy.releaser” permission on your compute service account in cloud shell terminal with :

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
    --member=serviceAccount:$(gcloud projects describe ${PROJECT_ID} \
    --format="value(projectNumber)")-compute@developer.gserviceaccount.com \
    --role="roles/clouddeploy.releaser"

Add “clouddeploy.serviceAgent” permission on your Cloud Build service account in cloud shell terminal with :

gcloud iam service-accounts add-iam-policy-binding  $(gcloud projects describe ${PROJECT_ID} \
    --format="value(projectNumber)")-compute@developer.gserviceaccount.com \
    --member=serviceAccount:$(gcloud projects describe ${PROJECT_ID} \
    --format="value(projectNumber)")@cloudbuild.gserviceaccount.com \
    --role="roles/clouddeploy.serviceAgent"

Create default VPC only if it does not yet exist:

gcloud compute networks create default --project=${PROJECT_ID} --subnet-mode=auto --mtu=1460 --bgp-routing-mode=regional

Create an autopilot GKE cluster using gcloud shell (IDE vs code possible…)

gcloud container clusters create-auto cluster-1 --project=${PROJECT_ID} --region=us-central1

Create an Artifact Registery Repo in Cloudshell terminal

gcloud artifacts repositories \
     create my-docker-repo \
     --repository-format=docker \
     --location=us-central1 \
     --description="Docker repository"

Sync Trigger with your GitHub repo in Cloud Build, hit create trigger in trigger tab :

Fill the trigger name and go to source and hit “connect new repository”

Select GitHub

Hit continue then you will be asked to install the Cloud Build App in your own GitHub repo and you just have to follow the steps displayed on your screen. It’s pretty straightforward.
Then select your repo from the dropdown list and select “main” branch and hit “create”

Clone the content of this Github repo on your laptop

git clone https://github.com/samy-fadel/gcp-ci-cd-gke.git

Copy the content into your own local laptop repo and make sure to write the value of your own project id in codedeploy.yaml
Install gcloud on your laptop by following these instructions
cd into your repo and then deploy pipeline with

gcloud deploy apply --file=clouddeploy.yaml \
    --region us-central1 \
    --project=${PROJECT_ID}

Then cd into your local repo and push the copied code to your main remote GitHub branch that is connected to the Cloud Build Trigger previously created :

git add .
git commit -m "first commit"
git push

This will trigger the pipeline.
Wait a few minutes
Go to GKE console in “Services & Ingress” tab and click on the endpoint IP to check your newly deployed website
Now you can work on your code app and improve it !

Conclusion

Wow it’s been a long journey full of gcloud commands, hopefully you got familiar with gcloud commands structure now and GCP CI CD services.

Of course in real life enterprise environment you will implement and automate all this with Terraform.

I will continue to post such articles adding more and more advanced features on this architecture (Devops, high availability, scalability, RPO/RTO, security, database, caching…)