GitOps-style continuous delivery with Cloud Build and GitHub

In this article, use CloudBuild and GitHub to implement the CI/CD strategies.

Image for post
Image for post
  1. Push application changes to the App repository
  2. Run the Cloud Build
    ・Push the docker image to Container Registry
    ・Automatically update the manifest files and push them to the Manifest repository
  3. Run the Cloud Build
    ・Apply the latest manifest files to Google Kubernetes Engine
    ・Commit the application to a specific branch of the Manifest repository

Prerequisites

  • The repositories of the application and the manifest is divided

Environment construction

App: istsh/go-grpc-health-probe-sample
Manifest: istsh/go-grpc-health-probe-sample-manifests

1. Enable the API

$ gcloud services enable \
container.googleapis.com \
cloudbuild.googleapis.com \
containeranalysis.googleapis.com

2. Grant the Cloud Build access to the Google Kubernetes Engine

Image for post
Image for post

3. Register your the SSH key in the Secret Manager

  1. Create the workingdir directory and create a new the SSH key in it (The name of the directory is optional)
  2. Enter id_github as the file name
    * The passphrase is blank
  3. In the Google Cloud Console, go to the Security > Secret Manager
  4. Click +CREATE SECRET
    ・The name is id_github
    ・Select the SSH key that was created
    ・Click Create Secret
$ mkdir -p ~/workingdir
$ cd ~/workingdir
$ ssh-keygen -t rsa -b 4096 -C ${GITHUB_EMAIL}
Image for post
Image for post

4. Add the public SSH key to the GitHub

  1. Input the title and paste the public SSH key that was created
  2. Select the checkbox of Allow write access and click Add key
    * This key will be used to push to the Manifest repository, so be sure to select it
  3. Delete the local SSH key since it is not needed
Image for post
Image for post

5. Grant a service account of the Cloud Build access to the Secret Manager

  1. From the table, find the email address that ends in @cloudbuild.gserviceaccount.com and click the edit icon
  2. Add the Secret Accessor role in Secret Manager and click Save

6. Preparation in each repositories

+-------------+-----------------+----------------------+
| environment | app branch name | manifest branch name |
+-------------+-----------------+----------------------+
| dev | develop | candidate-dev |
| staging | staging | candidate-staging |
| production | production | candidate-production |
+-------------+-----------------+----------------------+

This section describes the settings for the dev environment.

Preparation of the App repository

cloudbuild.yaml

timeout: 780s
steps:
- id: build and push grpc-gateway image
name: gcr.io/kaniko-project/executor:latest
args:
- --destination=gcr.io/$PROJECT_ID/grpc-gateway:${SHORT_SHA}
- --cache=true
- --cache-ttl=6h
- --dockerfile=Dockerfile.proxy
- --build-arg=PROJECT_ID=${PROJECT_ID}

- id: build and push grpc-server image
name: gcr.io/kaniko-project/executor:latest
args:
- --destination=gcr.io/$PROJECT_ID/grpc-server:${SHORT_SHA}
- --cache=true
- --cache-ttl=6h
- --dockerfile=Dockerfile.server
- --build-arg=PROJECT_ID=${PROJECT_ID}

- id: access the id_github file from secret manager
name: gcr.io/cloud-builders/gcloud
entrypoint: 'bash'
args:
- '-c'
- |
gcloud secrets versions access latest --secret=id_github > /root/.ssh/id_github
volumes:
- name: 'ssh'
path: /root/.ssh

- id: set up git with key and domain
name: 'gcr.io/cloud-builders/git'
entrypoint: 'bash'
args:
- '-c'
- |
chmod 600 /root/.ssh/id_github
cat <<EOF >/root/.ssh/config
Hostname github.com
IdentityFile /root/.ssh/id_github
EOF
ssh-keyscan -t rsa github.com > /root/.ssh/known_hosts
volumes:
- name: 'ssh'
path: /root/.ssh

- id: connect to the repository
name: 'gcr.io/cloud-builders/git'
args:
- clone
- --recurse-submodules
- git@github.com:istsh/go-grpc-health-probe-sample-manifests.git
volumes:
- name: 'ssh'
path: /root/.ssh

- id: switch to candidate-dev branch
name: 'gcr.io/cloud-builders/gcloud'
dir: go-grpc-health-probe-sample-manifests
entrypoint: /bin/sh
args:
- '-c'
- |
set -x && \
git config --global user.email $(git log --format='%an <%ae>' -n 1 HEAD | sed 's/.*\<\([^>]*\)\>.*/\1/g') && \
git fetch origin candidate-dev && git switch candidate-dev && \
git fetch origin master && git merge origin/master
volumes:
- name: 'ssh'
path: /root/.ssh

- id: generate manifest for grpc-gateway
name: 'gcr.io/cloud-builders/gcloud'
dir: go-grpc-health-probe-sample-manifests/k8s/base/grpc-gateway
entrypoint: /bin/sh
args:
- '-c'
- |
set -x && \
sed "s/GOOGLE_CLOUD_PROJECT/${PROJECT_ID}/g" deployment.yaml.tpl | \
sed "s/COMMIT_SHA/${SHORT_SHA}/g" > deployment.yaml

- id: generate manifest for grpc-server
name: 'gcr.io/cloud-builders/gcloud'
dir: go-grpc-health-probe-sample-manifests/k8s/base/grpc-server
entrypoint: /bin/sh
args:
- '-c'
- |
set -x && \
sed "s/GOOGLE_CLOUD_PROJECT/${PROJECT_ID}/g" deployment.yaml.tpl | \
sed "s/COMMIT_SHA/${SHORT_SHA}/g" > deployment.yaml

- id: push generated manifests to candidate-dev branch
name: 'gcr.io/cloud-builders/gcloud'
dir: go-grpc-health-probe-sample-manifests
entrypoint: /bin/sh
args:
- '-c'
- |
set -x && \
git add k8s/base/grpc-gateway/deployment.yaml k8s/base/grpc-server/deployment.yaml && \
git commit \
--author="Cloud Build Service Account <$(gcloud auth list --filter=status:ACTIVE --format='value(account)')>" \
-m "Deploying images
- gcr.io/istsh-sample/grpc-gateway:${SHORT_SHA}
- gcr.io/istsh-sample/grpc-server:${SHORT_SHA}
Built from commit ${COMMIT_SHA} of repository go-grpc-health-probe-sample-manifest" && \
git push origin candidate-dev
volumes:
- name: 'ssh'
path: /root/.ssh

The steps are performed in the following order.

  1. Create an image of the grpc-gateway and push it to the Container Registry
    The tag is :${SHORT_SHA}
  2. Create an image of the grpc-server and push it to the Container Registry
    The tag is :${SHORT_SHA}
  3. Get the GitHub SSH key from the Secret Manager and place it in /root/.ssh/
  4. Configuration to connect to the GitHub
  5. Clone the Manifest repository
  6. Switch to the candidate-dev branch and merge the master branch
    If you’ve merged the manifest changes into the master, they’re merged into the candidate-dev branch in this step
  7. Change a part of the content of deployment.yaml.tpl in grpc-gateway and create or change the deployment.yaml
    GOOGLE_CLOUD_PROJECT -> ${PROJECT_ID}
    COMMIT_SHA -> ${SHORT_SHA}
  8. Change a part of the content of deployment.yaml.tpl in grpc-server and create or change the deployment.yaml
    GOOGLE_CLOUD_PROJECT -> ${PROJECT_ID}
    COMMIT_SHA -> ${SHORT_SHA}
  9. Commit deployment.yaml and push it to the Manifest repository

Preparation of the Manifest repository

Create two branches.

+---------------+--------------------------------------------------+
| branch name | description |
+---------------+--------------------------------------------------+
| candidate-dev | The branch that commits the most recent manifest |
| | files |
| dev | The branch that triggers the deployment |
+---------------+--------------------------------------------------+

Preparation template of manifest file

For builds that are run by pushing the App repository, you use the yaml template to automatically generate a manifest file.
So you will have a template of manifest file that needs to be generated or changed automatically.
In this case, you need to automatically generate or change deployment.yaml, so you will create a deployment.yaml.tpl file.
I will only show you one file here.

deployment.yaml.tpl

apiVersion: apps/v1
kind: Deployment
metadata:
name: grpc-server
spec:
replicas: 2
revisionHistoryLimit: 3
selector:
matchLabels:
app: grpc-server
template:
metadata:
labels:
app: grpc-server
spec:
volumes:
- name: envoy
configMap:
name: envoy-grpc-server-config
containers:
- name: grpc-server
image: gcr.io/istsh-sample/grpc-server:COMMIT_SHA
imagePullPolicy: Always
ports:
- containerPort: 8080
command: ["/server"]
readinessProbe:
exec:
command: [ "/bin/grpc_health_probe", "-addr=:8080" ]
initialDelaySeconds: 1
livenessProbe:
exec:
command: [ "/bin/grpc_health_probe", "-addr=:8080" ]
initialDelaySeconds: 1
- name: envoy-proxy
image: envoyproxy/envoy:v1.14-latest
imagePullPolicy: Always
command:
- "/usr/local/bin/envoy"
args:
- "--config-path /etc/envoy/envoy.yaml"
ports:
- name: app
containerPort: 10000
- name: envoy-admin
containerPort: 8001
volumeMounts:
- name: envoy
mountPath: /etc/envoy

It’s worth noting that the :COMMIT_SHA.
Each time the CloudBuild pushes images to the Container Registry, it replaces the :COMMIT_SHA with the ID of the commit that triggered it.

cloudbuild.yaml

steps:
- id: deploy
name: 'gcr.io/cloud-builders/kubectl'
args:
- 'apply'
- '-k'
- 'k8s/overlays/dev/'
env:
- 'CLOUDSDK_COMPUTE_ZONE=${_CLOUDSDK_COMPUTE_ZONE}'
- 'CLOUDSDK_CONTAINER_CLUSTER=${_CLOUDSDK_CONTAINER_CLUSTER}'

- name: gcr.io/cloud-builders/gcloud
entrypoint: 'bash'
args: [ '-c', 'gcloud secrets versions access latest --secret=id_github > /root/.ssh/id_github' ]
volumes:
- name: 'ssh'
path: /root/.ssh

- name: 'gcr.io/cloud-builders/git'
entrypoint: 'bash'
args:
- '-c'
- |
chmod 600 /root/.ssh/id_github
cat <<EOF >/root/.ssh/config
Hostname github.com
IdentityFile /root/.ssh/id_github
EOF
ssh-keyscan -t rsa github.com > /root/.ssh/known_hosts
volumes:
- name: 'ssh'
path: /root/.ssh

- name: 'gcr.io/cloud-builders/git'
args:
- clone
- --recurse-submodules
- git@github.com:istsh/go-grpc-health-probe-sample-manifests.git
volumes:
- name: 'ssh'
path: /root/.ssh

- name: 'gcr.io/cloud-builders/gcloud'
id: Copy to dev branch
dir: go-grpc-health-probe-sample-manifests
entrypoint: /bin/sh
args:
- '-c'
- |
set -x && \
git config --global user.email $(git log --format='%an <%ae>' -n 1 HEAD | sed 's/.*\<\([^>]*\)\>.*/\1/g') && \
git fetch origin dev && git switch dev && \
git checkout $COMMIT_SHA k8s/base/grpc-gateway/deployment.yaml && \
git checkout $COMMIT_SHA k8s/base/grpc-server/deployment.yaml && \
git commit \
--author="Cloud Build Service Account <$(gcloud auth list --filter=status:ACTIVE --format='value(account)')>" \
-m "Manifest from commit $COMMIT_SHA

$(git log --format=%B -n 1 $COMMIT_SHA)" && \
git push origin dev
volumes:
- name: 'ssh'
path: /root/.ssh

substitutions:
_CLOUDSDK_COMPUTE_ZONE: us-central1-b
_CLOUDSDK_CONTAINER_CLUSTER: go-grpc-health-probe-sample

The steps are performed in the following order.

  1. Apply the latest manifest files to the Google Kubernetes Engine
  2. Get the GitHub SSH key from the Secret Manager and put it in /root/.ssh/
  3. Configuration to connect to the GitHub
  4. Clone the Manifest repository
  5. Switch to the dev branch
  6. Reflect the deployment.yaml committed in the candidate-dev branch, commit and push

7. Add triggers of the Cloud Build

The trigger for App Repository

  1. From Connect Repository, select the GitHub and click Continue
  2. Select the target App repository and click Connect Repository
  3. Click Create push trigger
  4. Select to push to branch in Events
  5. Input ^develop$ in branch in Source
  6. Select Build Configuration > Cloud Build configuration file and enter the path to cloudbuild.yaml

The trigger for Manifest Repository

  1. From Connect Repository, select the GitHub and click Continue
  2. Select the target Manifest repository and click Connect Repository
  3. Click Create push trigger
  4. Select to push to branch in Events
  5. Input ^candidate-dev$ in branch in Source
  6. Select Build Configuration > Cloud Build configuration file and enter the path to cloudbuild.yaml

8. The commitments

Author: Cloud Build Service Account <xxx@cloudbuild.gserviceaccount.com>
Date: Sun Sep 6 14:25:37 2020 +0000

Deploying images
- gcr.io/istsh-sample/grpc-gateway:12792a4
- gcr.io/istsh-sample/grpc-server:12792a4

Built from commit 12792a48c682312075ff4988367eb0b7393d28cd of repository go-grpc-health-probe-sample-manifest

And if the CloudBuild of the Manifest repository triggered by this push is successful, it will be pushed to the Manifest repository with a commit comment like this.

Author: Cloud Build Service Account <xxx@cloudbuild.gserviceaccount.com>
Date: Sat Sep 5 16:16:40 2020 +0000

Manifest from commit beaa97bc46cfb624f1f325da1702d99fec0c9af0

Deploying images
- gcr.io/istsh-sample/grpc-gateway:12792a4
- gcr.io/istsh-sample/grpc-server:12792a4

Built from commit 12792a48c682312075ff4988367eb0b7393d28cd of repository go-grpc-health-probe-sample-manifest

Conclusion

Written by

I’ll be writing articles about the golang in order to become a golang expert. I’ll also aim to improve my English language skills.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store