Cloud Build + Cloud Deploy: Best Siblings

We are aware of the importance of CI/CD for automating builds, tests, and deployments can accelerate your development workflow, improve code quality, and get features to users faster. With one of the important clients at my company, I had a chance to design a fully functional CI/CD flow for Cloud Run deployments.

This article explores a complete CI/CD pipeline using Cloud Build and Cloud Deploy for deploying an application to Cloud Run. Cloud Build acts as your Continuous Integration (CI) engine, automating the building, tagging, and publishing of Docker images to the GCP Artifact Registry. Cloud Deploy then takes over as the Continuous Deployment (CD) tool, seamlessly deploying your application to Cloud Run based on the image published by Cloud Build, once stable and verified the same image gets promoted to the production environment from the same CD pipeline.

Scenario

This scenario outlines a deployment pipeline for a Marvel(hypothetical) application. The pipeline involves:

1. Continuous Integration (CI): Code is built and tested automatically.
2. Continuous Delivery (CD) with Environment Promotion:

• Stable and tested images are deployed first to Cloud Run in a development (DEV) environment GCP project via Cloud Build and Cloud Deploy.
• Promotion to Production (PROD): After manual review and approval, the stable DEV image can be promoted to the production environment GCP project via Cloud Deploy.

While this scenario is for this article, real-world deployments often involve an additional staging (STAGE) or QA environment between DEV and PROD.

Architecture Flow Diagram for Cloud Build and Cloud Deploy

GitHub Code Link

For our scenario, we have one cloud build trigger and one cloud deploy pipeline that deploys to dev and promotes the same to prod further.

The cloud build trigger and cloud deploy pipeline(with two targets) exist in the same dev project only. We are promoting the deployment to prod from dev project.

Cloud Build

As mentioned Cloud Build is our CI engine here, the first step is to write cloudbuild.yaml that contains our docker image building, tagging, publishing and triggering subsequent cloud deploy pipeline logic.

Reference to GCP Secret Manager has also been written in the below code just to show the usage. Multiple references can be done easily from secretEnv list in the particular build step. Add an availableSecrets field to specify the secret version and environment variables to use for your secret and specify the secret using the environment variable prefixed with $$ in the args field.

steps:
- name: 'gcr.io/cloud-builders/docker'
  id: Build docker image
  secretEnv: ['REPO_URL']
  entrypoint: 'sh'
  args:
  - -xe
  - -c
  - |
    docker build \
    --tag=${_REGION}-docker.pkg.dev/$PROJECT_ID/${_AR_REGISTRY_NAME}/${_IMAGE}:$SHORT_SHA \
    --tag=${_REGION}-docker.pkg.dev/$PROJECT_ID/${_AR_REGISTRY_NAME}/${_IMAGE}:latest \
    --build-arg REPO_URL=$$REPO_URL \
    -f ${_DOCKERFILE_PATH} \
    .

- name: 'gcr.io/cloud-builders/docker'
  id: Push docker image with SHA tag
  args: ['push', '${_REGION}-docker.pkg.dev/$PROJECT_ID/${_AR_REGISTRY_NAME}/${_IMAGE}:$SHORT_SHA']

- name: 'gcr.io/cloud-builders/docker'
  id: Push docker image with latest tag
  args: ['push', '${_REGION}-docker.pkg.dev/$PROJECT_ID/${_AR_REGISTRY_NAME}/${_IMAGE}:latest']

- name: 'google/cloud-sdk:latest'
  id: Trigger Cloud Deploy
  entrypoint: 'sh'
  args:
  - -xe
  - -c
  - |
    gcloud config set deploy/region ${_REGION}
    gcloud deploy apply --file cloud-deploy/${_IMAGE}/${_ENV}/pipeline.yaml
    gcloud deploy apply --file cloud-deploy/${_IMAGE}/${_ENV}/target.yaml
    gcloud deploy releases create ${_IMAGE}-$SHORT_SHA \
                        --delivery-pipeline=${_IMAGE} \
                        --skaffold-file=${_IMAGE}.yaml
availableSecrets:
  secretManager:
  - versionName: projects/$PROJECT_NUMBER/secrets/REPO_URL/versions/latest
    env: 'REPO_URL'

The file here is templatized for catering to multiple application deployment needs. We pass the value of substitution variables directly from the cloud build trigger.

If this would be your first trigger, Host Connection and linking a repository is a prerequisite else proceed with a trigger.

From GCP Console ➜ Cloud Build ➜ Repositories ➜ 2nd Gen ➜ Create Host Connection. Choose your versioning control host and connect it. In the case of GitLab and Bitbucket connection, you need to supply tokens which would be stored in Secret Manager.

Note: Cloud Build Connection Admin, Cloud Build Editor and Secret Manager Admin are required to make the connection. I tried with more narrower permissions of Secret Manager but it didn’t work for me.

Once the connection is made successfully, it is visible in the Cloud Build repositories GUI and now repository linkage needs to be done. From the Link Repository choose the connection and repositories.

Cloud Build Connection

The next step would be to create a cloud trigger that would use our cloudbuild.yaml file.

Cloud Build Trigger

This trigger gets triggered every time there is a commit to the iron-man folder in the devbranch in the marvel GitLab repo whereas any changes in README.md are ignored. Multiple branches can be added in the cloud build trigger by specifying regex expression in an acceptable regular expression syntax, for instance, ^(dev|development|(fix|dev-fix).*)$ expression would trigger the pipeline for dev, development and branches prefixed with fix and dev-fix branches.

CI is a wrap! Moving on to CD!

Cloud Build

Cloud Build Schema

Use secrets from Secret Manager in Cloud Build

Cloud Deploy

Cloud Deploy takes care of our application deployment on Cloud Run in an automated fashion and further promotes the deployment to the live environment from a single GUI.

Let’s understand the code first then deployment. For Cloud Deploy, we need skaffold.yaml , pipeline.yaml , manifest YAML & target.yaml . Let’s understand each file briefly.

## skaffold.yaml (can be renamed, marvel.yaml in our case)

apiVersion: skaffold/v4beta7
kind: Config
metadata: 
  name: marvel
profiles:
- name: dev
  manifests:
    rawYaml:
    - cloud-deploy/marvel/dev/dev.yaml
- name: prod
  manifests:
    rawYaml:
    - cloud-deploy/marvel/prod/prod.yaml
deploy:
  cloudrun: {}

This file defines a Skaffold configuration for deploying different applications(marvel) to various environments(devand prod) by specifying the desired profiles(devand prod). The manifests (dev.yaml) for each deployment are referenced within the configuration itself.

## pipeline.yaml

apiVersion: deploy.cloud.google.com/v1
kind: DeliveryPipeline
metadata:
  name: marvel
  labels: 
    app: marvel
description: Cloud Deploy Pipeline for Marvel
serialPipeline:
  stages:
    - targetId: marvel-dev
      profiles:
        - dev
    - targetId: marvel-prod
      profiles:
        - prod

This file defines a Cloud Deploy delivery pipeline configuration for an application named “marvel”. serialPipelinesection defines the pipeline structure which executes stages one after another. The marvel-prodwould run after the dev is done.

## target.yaml

apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
  name: marvel-dev
  labels: {}
requireApproval: false
run:
  location: projects/{DEV_GCP_PROJECT_ID}/locations/us-central1

---
apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
  name: marvel-prod
  labels: {}
requireApproval: true
run:
  location: projects/{PROD_GCP_PROJECT_ID}/locations/us-central1

This file defines two separate configurations for Cloud Deploy targets for different environments — devand prod. requireApproval is set to true for production, indicating that deployments to this target require manual approval before they are executed. This adds an extra layer of control. rundefines where the application will be deployed.

## manifest.yaml (dev and prod)

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: marvel-dev
  annotations:
    run.googleapis.com/description: marvel-dev
spec:
  template:
    metadata:
      annotations:
        run.googleapis.com/cloudsql-instances: ${DEV_CLOUDSQL_CONNECTION_STRING} # just for illustration
    spec:
      containers:
        - name: marvel-dev
          image: us-central1-docker.pkg.dev/{DEV_GCP_PROJECT_ID}/marvel-registry/marvel:latest
          ports:
            - name: http1
              containerPort: 80
          resources:
            limits:
              cpu: 1000m
              memory: 1Gi

---
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: marvel-prod
  annotations:
    run.googleapis.com/description: marvel-prod
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/minScale: "2"
        run.googleapis.com/startup-cpu-boost: "true"
        run.googleapis.com/cloudsql-instances: ${PROD_CLOUDSQL_CONNECTION_STRING} # just for illustration
    spec:
      containers:
        - name: marvel-dev
          image: us-central1-docker.pkg.dev/{DEV_GCP_PROJECT_ID}/marvel-registry/marvel:latest
          ports:
            - name: http1
              containerPort: 80
          resources:
            limits:
              cpu: 4000m
              memory: 2Gi
          startupProbe:
            timeoutSeconds: 600
            periodSeconds: 600
            failureThreshold: 1
            tcpSocket:
              port: 80

The manifest file for prodhas a similar structure with some differences in values like cloudsql-instances or resources. Since the strategy here is standard promotion the value of an image in the case of prod.yaml remains the same as dev.yaml

Cloud Run YAML Reference

Important Note: To deploy resources across different GCP projects, specific IAM permissions are required. Grant following permissions: —

1. Cloud Run Developer & Service Account User — DEV project default service account ({DEV_PROJECT_NUMBER}-compute@developer.gserviceaccount.com) in PRODProject
2. Artifact Registry Reader — PROD Cloud Run Service Agent (service-{PROD_PROJECT_NUMBER}@serverless-robot-prod.iam.gserviceaccount.com) in DEVproject

Referencing back the cloudbuild.yaml the last step has the following lines of code —

#1 gcloud config set deploy/region ${_REGION}
#2 gcloud deploy apply --file cloud-deploy/${_IMAGE}/${_ENV}/pipeline.yaml
#3 gcloud deploy apply --file cloud-deploy/${_IMAGE}/${_ENV}/target.yaml
#4 gcloud deploy releases create ${_IMAGE}-$SHORT_SHA \
                        --delivery-pipeline=${_IMAGE} \
                        --skaffold-file=${_IMAGE}.yaml

1. Sets the default region for Cloud Deploy deployments.
2. Deploys the delivery pipeline configuration defined in the pipeline.yaml file.
3. Deploys the deployment target configuration defined in the target.yaml file.
4. Creates a new release of your application using Cloud Deploy using the specified pipeline and potentially leveraging the skaffold.yaml file.

Important Note (mentioning again to avoid confusion): The cloud build trigger and cloud deploy pipeline(with two targets) exist in the same dev project only. We are promoting the deployment to prod from dev project.

Time to get this code live…

The very first time the last step of the cloudbuild.yaml creates the cloud deploy pipeline first based on pipeline.yaml with target and then create a release.

When the pipeline runs, build information is available in Cloud Build History and logs can be visible upon clicking on a specific build ID.

The cloud build triggers 4 additional builds for —

1. Zipping the code and uploading the code to the GCS bucket created by GCP for Cloud deploy,
2. Unzipping the code and copying the content for deployment
3. Deploying the Cloud Run the service to the dev
4. Deploying the Cloud Run the service to the dev after the promotion

Cloud Deploy GUI

Releases information is available along with logs. Deployment logs are visible in the cloud build triggers logs #3.

Cloud Deploy Release

Once it is successful, you can see the Cloud Run service in the dev project as it doesn’t require any manual approvals.

Cloud Run Deployment

After the cloud run service is running in dev, you may see the Promote option enabled in the marvel-dev target pipeline pointing to the marvel-prod .

Cloud Deploy Pipeline

Let’s promote it to prod. Promotion to prod is manual however you can automate this too using the Automations functionality of Cloud Deploy.

Deploy to Production

Once you hit Redeploy, approval from the approver awaits. Once approved by the approver Cloud Deploy deploys the application to the production cloud run service.

In case of rollback, easy rollback is possible from the same cloud deploy GUI. One needs to select the version of rollback and hit Rollback.

Cloud Deploy Easy Rollback Option

Based on your pipeline you may see GCP Recommended Alerts and create your own.

Cloud Deploy is fairly a new CD offering by GCP and I hope to see more functionalities in the foreseeable future.

Anddd that’s it. That’s a way to do CI/CD using GCP native tools.

Cloud Deploy

Cloud Deploy for GKE

Cloud Deploy for Anthos

Cloud Deploy Alerts

Skaffold Docs

Read my other Cloud/DevOps/Infra blogs:-

Rohan Singh - Medium

Read industry professional interviews:-

Rohan Singh

Silly Sit-Downs(SSD) with Rohan

View list

7 stories

Clap if you find this tutorial informative and useful.

Follow and subscribe to my medium space to stay tuned for interviews and tech blogs. Till then have a good day and Sayonara!!!