Create a CI/CD Pipeline using GitHub Actions and Google Cloud

Dazbo (Darren Lester)
Google Cloud - Community
17 min read2 days ago

At-a-Glance

In this guide, I’m going to show you how to create a CI/CD pipeline that automates the test and build of our application code, and then deploys to Google Cloud services.

I’ll be adding this CI/CD capability to the Image Text Extractor and Translator Application that I previously documented here. A quick reminder of this application’s very basic architecture:

Image-Text-Translator Application Architecture

So our CI/CD pipeline will only need to deploy to Google Cloud Run and Google Cloud Functions. (By the way, Cloud Functions has now been rebranded… Cloud Run Functions.)

CI/CD Basics

CI/CD is an abbreviation for Continuous Integration and Continuous Delivery. It’s overall goals:

  • To improve the speed and quality of software delivery.
  • To allow a much higher frequency of software changes (and optionally — production releases of these changes).
  • To allow for shorter development cycles with associated faster feedback loops.
  • All whilst reducing manual toil and human error.

Automation is at the heart of CI/CD.

Continuous Integration

We can describe this as:

The ability to make frequent code changes which are validated with automated testing, and then merged back into the main code branch regularly. We integrate code frequently.

So typically: a developer checks-in code to a feature branch, which is part of a shared repo. This triggers automated building and testing of the code. If the tests pass, then the developer can raise a pull request (PR) to merge their changes back into the main branch.

Continuous Delivery

We can describe this as:

The practice of ensuring that code is always deployable to production. Continuous Delivery relies on workflows that are triggered when changes are pushed into the main branch. The deployment system typically deploys the application to a staging environment. Here, automated tests, including integration tests, can run. If the tests pass, the build is tagged as a release candidate.

A common source of confusion: Continuous Deployment is different to Continuous Delivery. With Continuous Deployment, every release candidate is automatically deployed to production. But with Continuous Delivery, the deployment artefact can be deployed to production, if we choose to do so.

CI/CD Tooling in the Google Cloud Environment

Git Repo Hosting

There are various source repo hosting services we can use in our Google CI/CD pipeline. We can use Google’s own Cloud Source Repositories. But we could also use GitHub, GitLab, or Bitbucket.

In this demo, I’m going to use GitHub.

CI/CD

  • Google Cloud Build — this is Google’s fully-managed, serverless CI/CD tool. It integrates with all the source repo services I mentioned above. It can run build workflows, including: testing, building, vulnerability scanning, and deployment. It can push build artefacts — e.g. container images or Java archives — to Google Cloud Artifact Registry. And it can deploy to various Google Cloud services, such as GKE, Cloud Run, and Cloud Functions.
  • GitHub Actions — a CI/CD tool built directly into GitHub, and therefore agnostic of our target cloud environment. Again, we can use it to run tests, build deployment artefacts, push to Google Artifact Registry, and deploy to Google services like Cloud Run and Cloud Functions.

In this demo, I’m going to use GitHub Actions.

Binary Artefact Management

Common tools for storing artefacts — such as container images and languages packages — include: Sonatype Nexus, JFrog Artifactory, Docker Hub, and Google Artifact Registry.

The latter is Google’s fully-managed artifact management service. It can store a wide variety of artefacts, and includes built-in vulnerability scanning. (It has replaced an older service called Google Container Registry.)

In this demo, I’m going to use Google Artifact Registry.

Overall Pipeline

This illustration shows the overall CI/CD pipeline, with various tooling options. The top line shows the tooling we’ll use in this demo.

CI/CD pipeline and tooling

Note that I’m going to build the pipeline so that it only deploys to Google Cloud when our branch is merged with the main branch. So, a typical workflow would be:

  1. Engineer creates a code branch, to develop a feature or fix a bug.
  2. Engineer commits and pushes their changes. The pipeline runs, which builds and tests.
  3. If the tests pass, the engineer raises a pull request (PR), to merge their changes into the main branch.
  4. The PR is approved and the changes are merged into the main branch.
  5. The update of the main branch causes the pipeline to run, which builds and deploys to Google Cloud.

Pre-Reqs If You Want to Follow Along

My blog here describes how to manually build and deploy the Image Text Translator and Extractor application. (I really need a shorter name for it… From now on, it will be referred to as: ImgTranslactor!) It provides the source, and step-by-step instructions for locally development, testing, and deployment to Google Cloud.

In this blog, we’re going to build on this by adding CI/CD capability, including Unit Testing. Before you continue, you might want to:

  • Get familar with the ImgTranslactor application and associated repo.
  • Perform the one-time Google Project Setup steps.
  • Ensure you’ve followed the steps (in the previous blog) to give your service account appropriate permissions. We’ll be using this service account to run the CI/CD pipeline from within GitHub.
  • Perform the local development environment setup.
  • Clone the repo, and checkout the v1.0 tag.
# Clone the repo
git clone https://github.com/derailed-dash/image-text-translator.git

# Check out the v1.0 tag - this is the "pre-CI/CD" tag
git checkout v1.0

cd image-text-translator

# Create a Python virtual env. For example...
python3 -m venv .venv
# And now ACTIVATE it. E.g.
source .venv/bin/activate

# Install the Python dependencies now
python3 -m pip install -r requirements.txt

Make sure you’re set to the correct project. Run these commands from your environment with Google gcloud installed:

# Check we have the correct project selected
gcloud auth login
gcloud config list project
export PROJECT_ID=<enter correct project ID>

gcloud auth application-default login

# If you're set to the wrong project
gcloud config set project $PROJECT_ID
gcloud auth application-default set-quota-project $PROJECT_ID

# Setup environment variables, including ADC
source scripts/setup.sh

Check you can run the application locally. E.g.

cd app/ui_cr/
source ../../scripts/setup.sh # Initialise vars if we're in a new terminal

# Run the Flask App
python app.py

And check you can deploy successfully to Google Cloud, via Artifact Registry.

Google Artifact Registry
Deployed services

Create GitHub Actions Workflow

Overview

A GitHub Actions workflow fires in response to an event, such as code being pushed. The workflow is composed of one or more jobs which can run sequentially or in parallel. Each job is composed of one or more sequential steps, which can be scripts or reusable workflows. Note that each job runs in its own runner, i.e. a machine that runs the steps of the job.

GitHub Actions workflow

Each workflow is defined using a yaml file, and is stored in the .github/workflows directory of your repository.

So, to recap:

GitHub Actions organisation

GitHub Secrets

Our application makes use of Google APIs, and requires a Google service account with appopriate permissions. I showed you how to create this service account in the previous blog.

In order for our GitHub workflow to be able to access Google Cloud APIs, it needs to be able to supply credentials to Google Cloud. We do this by allowing the GitHub workflow to be run by our service account. There are a couple of ways we can do this:

  1. The easier way — we upload the service account JSON credentials as a GitHub secret.
  2. The better way — with Workload Identity Federation. This allows GitHub to authenticate to Google Cloud services without the need for a service account key. Instead, GitHub uses OpenID Connect (OIDC) to request a short-lived access token from Google Cloud. It relies on establishing a trust relationship between GitHub and Google Cloud.

For the purposes of this demo, I’m going to stick with the easy way. So the first thing we need to do is upload our service account key to GitHub as a secret.

In GitHub, navigate to Settings, then Secrets and variables, and select Actions.

Storing secrets in GitHub

We’ll create the following secrets:

  • GCP_PROJECT_ID
  • GCP_SVC_ACCOUNT — Not the full email; just the account name. (We’ll concatenate this in our workflow, to create the full email.)
  • GCP_SVC_ACCOUNT_CREDS

For the service account credentials I recommend storing a base64-encoded version of the JSON file. This avoids issues with the our workflow reading JSON that might have embedded special characters and newlines. Let’s make the base64-encoded version of the JSON file like this:

base64 $GOOGLE_APPLICATION_CREDENTIALS > service_account_encoded.txt

Copy the contents of the file created, and store that in our GitHub secret.

When we’re done, our repo secrets should look like this:

GitHub repo secrets

Store Some Variables

While we’re here, we can store some variables. These are the variables that are not sensitive. I’m going to pull some variables from the setup.sh script that already exists in my repo.

Storing repo variables in GitHub

Create the Initial CI/CD Workflow

From GitHub, select Actions. I’ve created a new workflow called build-and-test.yml. It looks like this:

# This workflow builds and tests our Python application.
name: Build and Test
run-name: ${{ github.actor }} launched the Build-and-Test workflow 🚀
on:
push:
paths:
- 'app/**' # Only run if our push contains a file in the app folder

pull_request:
branches:
- 'master' # This ensures the workflow also runs when merging
paths:
- 'app/**' # Only run if our push contains a file in the app folder

workflow_dispatch: # Allows manual triggering of the workflow.

permissions:
contents: read

env:
PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
SVC_ACCOUNT: ${{ secrets.GCP_SVC_ACCOUNT }}
GOOGLE_APPLICATION_CREDENTIALS: ${{ github.workspace }}/${{ secrets.GCP_SVC_ACCOUNT }}.json
SVC_ACCOUNT_EMAIL: ${{ secrets.GCP_SVC_ACCOUNT }}@${{ secrets.GCP_PROJECT_ID }}.iam.gserviceaccount.com
REGION: ${{ vars.GCP_REGION }}
FUNCTIONS_PORT: ${{ vars.GCP_DEV_FUNCTIONS_PORT}}
FLASK_RUN_PORT: ${{ vars.FLASK_RUN_PORT }}
BACKEND_GCF: https://${{ vars.GCP_REGION }}-${{ secrets.GCP_PROJECT_ID }}.${{ vars.GCP_FUNCTION_URI_SUFFIX }}

jobs:
build_and_test:
runs-on: ubuntu-latest

# Use the matrix strategy to run build and test steps for both backend_gcf and ui_cr
# This allows us to re-use this job and run it for each application folder
strategy:
matrix:
app-folder:
- app/backend_gcf
- app/ui_cr

steps:
- uses: actions/checkout@v4

- name: Check env vars
run: |
echo "Environment variables configured:"
echo PROJECT_ID="$PROJECT_ID"
echo REGION="$REGION"
echo SVC_ACCOUNT="$SVC_ACCOUNT"
echo SVC_ACCOUNT_EMAIL="$SVC_ACCOUNT_EMAIL"
echo GOOGLE_APPLICATION_CREDENTIALS="$GOOGLE_APPLICATION_CREDENTIALS"
echo BACKEND_GCF="$BACKEND_GCF"
echo FUNCTIONS_PORT="$FUNCTIONS_PORT"
echo FLASK_RUN_PORT="$FLASK_RUN_PORT"

# Creates a credentials file that will be used for ADC later
- name: Create credentials file
run: |
echo "${{ secrets.GCP_SVC_ACCOUNT_CREDS }}" | base64 --decode > $GOOGLE_APPLICATION_CREDENTIALS
echo "Checking the credentials:"
head -n 3 $GOOGLE_APPLICATION_CREDENTIALS
shell: bash

- name: Set up Python 3.10
uses: actions/setup-python@v5
with:
python-version: "3.10"
cache: 'pip' # caching pip dependencies

- name: Install dependencies
run: |
echo "Installing Python dependencies"
python -m pip install --upgrade pip
pip install flake8 pytest
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi

- name: Lint with flake8
working-directory: ${{ matrix.app-folder }}
run: |
# stop the build if there are Python syntax errors or undefined names
echo "Running first flake8:"
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
echo "Flake8 exit code: $?"
echo "Running second flake8:"
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
echo "Flake8 exit code: $?"

- name: Test with pytest
working-directory: ${{ matrix.app-folder }}
run: |
pytest || if [ $? -eq 5 ]; then echo "No tests were found, but continuing..."; else exit 1; fi

A few things to note:

  • The workflow is triggered when an engineer pushes a change to any branch other than master. (It can also be triggered manually, in the GitHub UI.)
  • I’m using the paths filter to only run the workflow when we commit files under the app folder.
  • We define a number of environment variables, by reading GitHub variables and secrets. I’ve defined these at the workflow level, meaning that these environment variables are accessible to any jobs in the workflow.
  • We can concatenate multiple secrets or variables, to create new variables.
  • I’m using the pre-canned actions/setup-python@v5 to setup the Python environment, and cache any pip dependencies. (This speeds up future runs.)
  • Although I’ve only defined one job called build-and-test, we’re using the matrix strategy to run it twice; i.e. we run the entire job for both folders sitting under the app folder.
  • When I read in the service account credentials secret, I need to decode it from base64, which I do with base64 --decode.

Save the workflow and commit the change into the repo. Now this workflow will be triggered whenever we push a change to our app!

I can test it by invoking manually. And a few seconds later, this is the result:

Running the GitHub Actions workflow

It works!!

We can drill into either of the jobs and take a look at the logs. Note that GitHub automatically redacts any of our secrets from the logs.

Viewing the workflow logs

I still need to add the ability to deploy our tested applications to Google Cloud. I’ll come back to that later.

Adding Unit Tests

This is a good time to introduce some unit tests. We can use these to validate that our Python applications work, and to check that any code changes we make don’t break the tests. One of the advantages of a CI/CD pipeline is that we can run these unit tests automatically, whenever we push our code changes to our repo.

Since this is a CI/CD article, not a unit testing article, I’m not going to explain the unit testing in any detail.

Create a Unit Test for Cloud Function

I start by creating a unit test for the Cloud Function, called test_main.py:

import pytest
from unittest.mock import patch, MagicMock # To mock external API calls
from flask import Flask, request # To simulate HTTP requests
from io import BytesIO

# Import the function from main.py
from backend_gcf.main import extract_and_translate

@pytest.fixture
def app():
""" Fixture for Flask app context, for creating HTTP requests """
app = Flask(__name__)
return app

# Use patch to replace the Google clients with mock objects
@patch('backend_gcf.main.vision_client')
@patch('backend_gcf.main.translate_client')
def test_extract_and_translate_with_posted_image(mock_translate_client, mock_vision_client, app: Flask):
with app.test_request_context(
method='POST',
data={
'uploaded': (BytesIO(b'sample image data'), 'test_image.jpg'),
'to_lang': 'en'
},
content_type='multipart/form-data'
):
# Mock Vision API response
mock_vision_client.text_detection.return_value = MagicMock(
text_annotations=[MagicMock(description="Put a glass of rice and three glasses of water in a saucepan")]
)

# Mock Translate API response
mock_translate_client.detect_language.return_value = {"language": "uk"}
mock_translate_client.translate.return_value = {"translatedText": "Put a glass of rice and three glasses of water in a saucepan"}

# Call the function
response = extract_and_translate(request)

# Check the result
assert response == "Put a glass of rice and three glasses of water in a saucepan"

This works by using the unittest.mock module to replace the vision_client and translate_client API calls with mock objects that simulate the responses from the external APIs. It also creates a Flask application context for simulating HTTP requests to the Cloud Function.

Note that we also need __init__.py files in both the backend_gcf folder and the tests subfolder. This is required so that pytest can understand our package structure.

Commit and Push in a Branch

Here we will:

  • Create a new feature branch for our unit test.
  • Commit and push our feature branch. This should cause our GitHub Actions workflow to trigger.
  • Create a pull request (PR) to merge the feature branch into our main branch.
  • Approve the PR and merge.
  • Delete the feature branch.

Here are the steps in detail:

# Create the feature branch for our unit tests
git checkout -b feature/add-unit-tests
git add .
git commit -m "Add unit tests for extract_and_translate function"
git push -u origin feature/add-unit-tests

The workflow triggers and runs successfully!

And we can look at the logs to check our unit test has executed:

Running our unit test with Pytest

Now we raise the Pull Request in GitHub:

Create the PR
The new branch can be merged

We can now merge the PR:

Action the merge request

And the merge was successful:

Merge successful

Now we can delete the feature branch in GitHub, switch back to the main branch locally, update locally, and remove any orphaned local branches:

git checkout main
git pull # update local, since the remote now has the merged changes
git branch -d feature/add-unit-tests # delete the local branch
git fetch --prune # optionally prune references to any deleted remote branches

Create a Unit Test for the Flask UI Application

We’ll follow the same process as before.

I start by creating app/ui_cr/tests/test_app.py. If you’re interested in the unit test code, then you can view it in the repo. Then I commit in a new branch, and the workflow executes:

The workflow runs with the new unit test

As before, we now need to create the PR, approve the PR and merge, switch back to the main branch, and delete/prune the feature branch.

Add a Workflow to Deploy to Google Cloud

Previously we created a workflow to build and test the application. Here we’ll create a separate workflow that deploys our application to Google Cloud Functions and Google Cloud Run.

We’ll create a new workflow, called deploy.yml:

name: Deploy to GCF and Cloud Run
run-name: ${{ github.actor }} launched the Deploy workflow 🚀
on:
workflow_run:
workflows: ["Build and Test"] # Run the deploy after successful Build and Test
types:
- completed

workflow_dispatch: # Allows manual triggering of the workflow.

permissions:
contents: read

env:
PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
SVC_ACCOUNT: ${{ secrets.GCP_SVC_ACCOUNT }}
GOOGLE_APPLICATION_CREDENTIALS: ${{ github.workspace }}/${{ secrets.GCP_SVC_ACCOUNT }}.json
SVC_ACCOUNT_EMAIL: ${{ secrets.GCP_SVC_ACCOUNT }}@${{ secrets.GCP_PROJECT_ID }}.iam.gserviceaccount.com
REGION: ${{ vars.GCP_REGION }}
BACKEND_GCF: https://${{ vars.GCP_REGION }}-${{ secrets.GCP_PROJECT_ID }}.${{ vars.GCP_FUNCTION_URI_SUFFIX }}
CR_UI_IMAGE_NAME: ${{ vars.GCP_REGION }}-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/image-text-translator-artifacts/image-text-translator-ui:${{ github.sha }}
BUILD_LOGS_BUCKET: gs://${{ secrets.GCP_PROJECT_ID }}-gcloud-logs

jobs:
deploy_backend_gcf: # Only deploy if build_and_test was successful on master
if: ${{ (github.event_name == 'workflow_run' && github.event.workflow_run.conclusion == 'success') || github.event_name == 'workflow_dispatch' }} && github.ref == 'refs/heads/master'
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4

- name: Check env vars
run: |
echo "Environment variables configured:"
echo PROJECT_ID="$PROJECT_ID"
echo REGION="$REGION"
echo SVC_ACCOUNT="$SVC_ACCOUNT"
echo SVC_ACCOUNT_EMAIL="$SVC_ACCOUNT_EMAIL"
echo BACKEND_GCF="$BACKEND_GCF"

# Authenticate with Google Cloud
- name: Authenticate to Google Cloud
run: |
echo "${{ secrets.GCP_SVC_ACCOUNT_CREDS }}" | base64 --decode > $GOOGLE_APPLICATION_CREDENTIALS
gcloud auth activate-service-account $SVC_ACCOUNT_EMAIL --key-file=$GOOGLE_APPLICATION_CREDENTIALS
gcloud config set project $PROJECT_ID
gcloud config set functions/region $REGION

# Deploy backend_gcf to Google Cloud Functions
- name: Deploy to Google Cloud Functions
working-directory: app/backend_gcf
run: |
gcloud functions deploy extract-and-translate \
--gen2 \
--max-instances 1 \
--region $REGION \
--runtime=python312 \
--source=. \
--trigger-http \
--entry-point=extract_and_translate \
--no-allow-unauthenticated \
--service-account=$SVC_ACCOUNT_EMAIL

# Allow this function to be called by the service account
gcloud functions add-invoker-policy-binding extract-and-translate \
--region=$REGION \
--member="serviceAccount:$SVC_ACCOUNT_EMAIL"

deploy_ui_cr:
if: ${{ (github.event_name == 'workflow_run' && github.event.workflow_run.conclusion == 'success') || github.event_name == 'workflow_dispatch' }} && github.ref == 'refs/heads/master'
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4

# Authenticate with Google Cloud
- name: Authenticate to Google Cloud
run: |
echo "${{ secrets.GCP_SVC_ACCOUNT_CREDS }}" | base64 --decode > $GOOGLE_APPLICATION_CREDENTIALS
gcloud auth activate-service-account $SVC_ACCOUNT_EMAIL --key-file=$GOOGLE_APPLICATION_CREDENTIALS
gcloud config set project $PROJECT_ID
gcloud config set run/region $REGION

# Build and deploy the ui_cr container to Cloud Run
- name: Build Docker image
working-directory: app/ui_cr
run: |
gcloud auth configure-docker $REGION-docker.pkg.dev
gcloud builds submit --gcs-log-dir $BUILD_LOGS_BUCKET --tag $CR_UI_IMAGE_NAME

- name: Deploy to Cloud Run
run: |
export RANDOM_SECRET_KEY=$(openssl rand -base64 32)
gcloud run deploy image-text-translator-ui \
--image=$CR_UI_IMAGE_NAME \
--region=$REGION \
--platform=managed \
--allow-unauthenticated \
--max-instances=1 \
--service-account=$SVC_ACCOUNT_EMAIL \
--set-env-vars BACKEND_GCF=$BACKEND_GCF,FLASK_SECRET_KEY=$RANDOM_SECRET_KEY

A few notes about this:

  • We want this workflow to run after the Build and Test workflow executes, but only if the workflow was successful, and only if it was run against the main branch.
  • We define an image name for our Cloud Run image, and we tag it with the git commit SHA. This gives us an unique image tag.
  • We have two jobs: one to deploy to Google Cloud Functions, and one to deploy to Cloud Run. There is no dependency between them, so they can be executed in parallel.
  • In both jobs, we first run gcloud auth to issue commands as the service account.
  • Then, we pretty much just run the same commands that we used when we were deploying manually, as described in this guide.
  • Our gcloud builds submit command specifies a GCS logging directory. I’ll cover this next…

Create a GCS Bucket for Cloud Build Logs

When we run the gcloud builds submit command to create our application container image, Google Cloud Build attempts to write logs to a GCS bucket. The service account will not have access to the default bucket, so we should explicitly create a bucket for it.

To create the bucket and give our service account access to it, run the following commands using an identity with sufficient privileges, e.g. an org admin or project editor:

# Create the bucket
export BUILD_LOGS_BUCKET=gs://${PROJECT_ID}-gcloud-logs
gsutil mb $BUILD_LOGS_BUCKET
# Give our service account permission to write to it
gsutil iam ch serviceAccount:$SVC_ACCOUNT_EMAIL:roles/storage.admin $BUILD_LOGS_BUCKET

# If we want to test from a terminal...
gcloud auth activate-service-account $SVC_ACCOUNT_EMAIL --key-file=$GOOGLE_APPLICATION_CREDENTIALS
gsutil ls $BUILD_LOGS_BUCKET

Test the Workflow

We can run it manually from GitHub:

Running our deployment workflow

And …

We can check the new image is in Google Artifact Registry, tagged with the SHA commit:

Viewing container images in Google Artifact Registry

And we can check that the new version has been deployed to Cloud Run:

Deployed!

Wrap-Up

And that’s it! It takes surprisingly little time and effort to introduce a working CI/CD pipeline using GitHub Actions. From now on I can make changes to this application, knowing that my changes will be automatically tested, and deployed to Google Cloud whenever I merge a tested change back into the main branch.

Some Observations

In this demo, I’ve used the same service account to run the Google Cloud services as I’ve used to deploy them. This is not great practice and means that our service account is over-provisioned. It would be better to split this out into two separate accounts. So that’s an enhancement I’ll add later.

Useful Links

ImgTranslactor!

Google Cloud

GitHub and Actions

Unit Testing

Before You Go

  • Please share this with anyone that you think will be interested. It might help them, and it really helps me!
  • Please give me claps! You know you can clap more than once, right?
  • Feel free to leave a comment 💬.
  • Follow and subscribe, so you don’t miss my content. Go to my Profile Page, and click on these icons:
Follow and Subscribe

--

--

Dazbo (Darren Lester)
Google Cloud - Community

Cloud Architect and moderate geek. Google Cloud evangelist. I love learning new things, but my brain is tiny. So when something goes in, something falls out!