Securing Software Supply Chain on Google Cloud

Published in

Google Cloud - Community

8 min readOct 5, 2022

This was originally written by Balesh Kumar who since (unfortunately) moved on from my team before we got the time to open source the associated code.

Introduction

The purpose of this document is to provide a step by step guide and related artifacts to set up a secure CI/CD pipeline for a containerized workload. While the overall ecosystem security involves multiple layers, from securing underline physical infrastructure to actual code, this document focuses on the Application/Container security that can be automated in the CI/CD pipeline.

System Overview

In this demo, we are using a simple java application (with Gradle build tool) that is containerized using Dockerfile and CI/CD pipeline is configured using cloudbuild.yaml. Other technologies/services used to implement this CI/CD pipelines are:

Cloud Build Steps:

Standard CI/CD Pipeline

A standard CI/CD pipeline generally consists of 4–5 stages. For example, Build, Test, Static code Analysis and QA stage. These stages are more developer focused and It only makes sure code builds successfully, it follows general coding guidelines and it fulfills functional requirements.

Artifacts used in these stages are the following:

Application code
Dockerfile
Container Image
Kubernetes Manifest files

All of these artifacts are vulnerable to security lapses. Following is the list of vulnerabilities they can introduce.

Application Code

SQL Injection
Cross site scripting
Code Injection
Sensitive Cookies without “HttpOnly” flag
and many more

Scanning tool: SonarQube

Dockerfile

Unnecessary library and packages that increase attack surface
Unofficial base image that can be vulnerable
Image without tag make application inconsistent
and many more

Scanning tool: Hadolint, Confest

Container Image

Buffer Overflow that can lead to Denial of service
Integer overflow that can impact confidentiality
Command Injection
and many more

Scanning tool: Container Analysis

Kubernetes manifest file

Least privilege principle violation
Mutable file system increase attack surface area
No limit on CPU & memory can lead to DOS via resource exhaustion
Unnecessary capability given to a container can lead to increase syscall attack surface

Scanning tool: Kubesec, Confest

Secured CI/CD Pipeline

With a little more effort, we can handle all the vulnerabilities discussed in the previous section.

Demo scope & output

CI/CD pipeline for this demo consists of 11 steps and out of them 7 steps are created to make this supply chain more secure.

Build steps can be broken down into categories like Static code analysis, Build+Push, Vulnerability Scanning + Attestation and Generate Kubernetes Manifest + Deploy.

Demo setup

First create a project, or set your existing project as default:

gcloud projects create $PROJECT_ID –set-as-default

Change the following variables below in bold to match your project name, region and zone:

PROJECT_ID=sec-soft-chain
REGION=australia-southeast1
ZONE=australia-southeast1-b

Then run this script, comments in the script explain functionality

# Enable the Required API’s
gcloud services enable container.googleapis.com
gcloud services enable artifactregistry.googleapis.com
gcloud services enable ondemandscanning.googleapis.com
gcloud services enable cloudkms.googleapis.com
gcloud services enable binaryauthorization.googleapis.com# Create a cluster for this demo, note these settings should not be used for a production cluster 
gcloud beta container - project $PROJECT_ID clusters create "software-secure-supply" - zone $ZONE - no-enable-basic-auth - cluster-version "1.22.8-gke.201" - release-channel "regular" - machine-type "e2-standard-4" - image-type "COS_CONTAINERD" - disk-type "pd-standard" - disk-size "100" - metadata disable-legacy-endpoints=true - scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" - max-pods-per-node "110" - num-nodes "2" - logging=SYSTEM,WORKLOAD - monitoring=SYSTEM - enable-ip-alias - network "projects/$PROJECT_ID/global/networks/default" - subnetwork "projects/$PROJECT_ID/regions/$REGION/subnetworks/default" - no-enable-intra-node-visibility - default-max-pods-per-node "110" - no-enable-master-authorized-networks - addons HorizontalPodAutoscaling,HttpLoadBalancing,GcePersistentDiskCsiDriver - enable-autoupgrade - enable-autorepair - max-surge-upgrade 1 - max-unavailable-upgrade 1 - enable-shielded-nodes - node-locations $ZONE - enable-binauthz# Create an Artifact Repository
gcloud artifacts repositories create "${PROJECT_ID}-repo" - location=$REGION - repository-format=docker# Allow the Cloud Build Service Account to run scans
gcloud projects add-iam-policy-binding $PROJECT_ID - member=serviceAccount:$(gcloud projects describe $PROJECT_ID - format="value(projectNumber)")@cloudbuild.gserviceaccount.com - role=roles/ondemandscanning.admin# Allow the Cloud Build Service Account to deploy to GKE
gcloud projects add-iam-policy-binding $PROJECT_ID - member=serviceAccount:$(gcloud projects describe $PROJECT_ID - format="value(projectNumber)")@cloudbuild.gserviceaccount.com - role=roles/container.developer# Allow Cloud Build Service Account the permission to attest
gcloud projects add-iam-policy-binding $PROJECT_ID - member=serviceAccount:$(gcloud projects describe $PROJECT_ID - format="value(projectNumber)")@cloudbuild.gserviceaccount.com - role=roles/containeranalysis.notes.attacher

Setup Cloud Source Repository

Clone the demo repository to your local machine and copy it to your own Cloud Source repository

git clone https://github.com/GoogleCloudPlatform/cloud-build-software-delivery

2. Push the cloned repo to your own Cloud Source repository:

# Create a repo called sec-soft-chain
gcloud source repos create sec-soft-chain# Configure credentials
gcloud init && git config — global credential.https://source.developers.google.com.helper gcloud.sh# Add your new repo as a remote repo called google
git remote add google https://source.developers.google.com/p/$PROJECT_ID/r/sec-soft-chain# Push code to the new repo called google
git push — all google

Create the Cloud Build trigger

In the Cloud console, go to Cloud Build Triggers
Enable the API (if not already enabled)
Click Create Trigger
In the Trigger Settings window, enter the following details:

In the Name field, enter build-vulnz-deploy
For Event choose Push to a branch
In the Repository field, choose your repo from the menu
In the Branch field, enter ^main$
For Configuration, select Cloud Build configuration file (yaml or json)
In the Location, select Repository enter the default value /cloudbuild.yaml

5. Add the following Substitution Variable pairs:

_IMAGE_NAME with the image from Cloud Source Repository
_COMPUTE_REGION with the value australia-southeast1 (or the region you chose in the beginning).
_KMS_KEYRING with the value binauthz
_KMS_LOCATION with the value australia-southeast1 (or the region you chose in the beginning).
_VULNZ_ATTESTOR with the value vulnz-attestor
_VULNZ_KMS_KEY with the value vulnz-signer
_VULNZ_KMS_KEY_VERSION with the value 1
_SONAR_LOGIN can be blank for now
_SONAR_PROJECT can be blank for now
_SONAR_ORG can be blank for now

6. Click Create.

Congratulations!!!

At this point you have successfully configured most of the CI/CD pipeline, excluding the two stages highlighted below, and can pause here to test the work by commenting out the final 2 stages from your cloudbuild.yaml (lines 17–24 and 62–78).

Secure CI/CD Pipeline with 2 steps highlighted

Next we will configure:

Sonarqube for static code analysis
Binary Authorization

Configure Sonarqube

Modify cloudbuild.yaml

The Cloudbuild.yaml file in the repo needs a few changes these can be done manually by replacing each instance of the following variables:

<ZONE>
<REGION>
<REPO>
<IMAGE>
<PROJECT_ID>

Or the below scripts should set these as per the variables configured previously:giot

sed -i “s/<ZONE>/$ZONE/g” cloudbuild.yaml
sed -i “s/<REGION>/$REGION/g” cloudbuild.yaml
sed -i “s/<PROJECT_ID>/$PROJECT_ID/g” cloudbuild.yaml
sed -i “s/<REPO>/$PROJECT_ID-repo/g” cloudbuild.yaml
sed -i “s/<IMAGE>/image/g” cloudbuild.yaml
<PROJECT_ID>/<REPO>/<IMAGE>
“${PROJECT_ID}-repo”
$REGION-docker.pkg.dev/$PROJECT_ID/$PROJECT_ID-repo/

Setup Sonarqube

Clone Sonarqube Cloud Builders repository
Create custom builder by running following commands:

git clone https://github.com/GoogleCloudPlatform/cloud-builders-community
cd sonarqube
gcloud builds submit . — config=cloudbuild.yaml

3. Configure sonarqube (we will configure sonarqube online, but you can set up your own sonarqube server and configure that too).

Create a token by navigating to Account page then click on security tab
Next we need to use the “Analyze New Project” option to set up the project in sonarcloud. Note: Use setup manually option
Note down the token you created, project key and the organization name
Update the Cloud Build Trigger Variables for:
_SONAR_ORG
_SONAR_PROJECT
_SONAR_LOGIN

Binary Authorization

Note: Given below is the list of stripped down actions borrowed from this link

Build and register the custom build step with Cloud Build

Clone the Google Cloud build community repo:

git clone https://github.com/GoogleCloudPlatform/gke-binary-auth-tools ~/binauthz-tools

2. Configure the Binary Authorization signer for Cloud Build:
Before use, the code for the custom build step must be built into a container and pushed to Cloud Build. To do this, run the following commands:

gcloud builds submit — project $PROJECT_ID — tag “gcr.io/$PROJECT_ID/cloudbuild-attestor” ~/binauthz-tools

3. The custom build step was pushed to your current project’s Google Container Registry and is now ready for use

Create Cloud KMS asymmetric key for signing attestations.

In Cloud Shell, create a Cloud KMS key ring named binauthz:

gcloud kms keyrings create “binauthz” \
 — project “${PROJECT_ID}” \
 — location “${REGION}”

Create an asymmetric Cloud KMS key named vulnz-signer which will be used to sign and verify vulnerability scan attestations:

gcloud kms keys create “vulnz-signer” \
 — project “${PROJECT_ID}” \
 — location “${REGION}” \
 — keyring “binauthz” \
 — purpose “asymmetric-signing” \
 — default-algorithm “rsa-sign-pkcs1–4096-sha512”

Create a Container Analysis note named vulnz-note

curl “https://containeranalysis.googleapis.com/v1/projects/${PROJECT_ID}/notes/?noteId=vulnz-note" \
 — request “POST” \
 — header “Content-Type: application/json” \
 — header “Authorization: Bearer $(gcloud auth print-access-token)” \
 — header “X-Goog-User-Project: ${PROJECT_ID}” \
 — data-binary @- <<EOF
{
“name”: “projects/${PROJECT_ID}/notes/vulnz-note”,
“attestation”: {
“hint”: {
“human_readable_name”: “Vulnerability scan note”
}
}
}
EOF

Give CloudBuild Service account name permission on note

curl “https://containeranalysis.googleapis.com/v1/projects/${PROJECT_ID}/notes/vulnz-note:setIamPolicy" \
 — request POST \
 — header “Content-Type: application/json” \
 — header “Authorization: Bearer $(gcloud auth print-access-token)” \
 — header “X-Goog-User-Project: ${PROJECT_ID}” \
 — data-binary @- <<EOF
{
“resource”: “projects/${PROJECT_ID}/notes/vulnz-note”,
“policy”: {
“bindings”: [
{
“role”: “roles/containeranalysis.notes.occurrences.viewer”,
“members”: [
“serviceAccount:${CLOUD_BUILD_SA_EMAIL}”
]
},
{
“role”: “roles/containeranalysis.notes.attacher”,
“members”: [
“serviceAccount:${CLOUD_BUILD_SA_EMAIL}”
]
}
]
}
}
EOF

Create the vulnerability scan attestor:

gcloud container binauthz attestors create “vulnz-attestor” \
 — project “${PROJECT_ID}” \
 — attestation-authority-note-project “${PROJECT_ID}” \
 — attestation-authority-note “vulnz-note” \
 — description “Vulnerability scan attestor”

Add the public key for the attestor’s signing key:

gcloud beta container binauthz attestors public-keys add \
 — project “${PROJECT_ID}” \
 — attestor “vulnz-attestor” \
 — keyversion “1” \
 — keyversion-key “vulnz-signer” \
 — keyversion-keyring “binauthz” \
 — keyversion-location “${REGION}” \
 — keyversion-project “${PROJECT_ID}”

Grant the Cloud Build service account permission to verify attestations made by vulnz-attestor:

gcloud container binauthz attestors add-iam-policy-binding “vulnz-attestor” \
 — project “${PROJECT_ID}” \
 — member=serviceAccount:$(gcloud projects describe $PROJECT_ID — format=”value(projectNumber)”)@cloudbuild.gserviceaccount.com \
 — role “roles/binaryauthorization.attestorsViewer”

Grant the Cloud Build service account permission to sign objects using the vulnz-signer key:

gcloud kms keys add-iam-policy-binding “vulnz-signer” \
 — project “${PROJECT_ID}” \
 — location “${REGION}” \
 — keyring “binauthz” \
 — member serviceAccount:$(gcloud projects describe $PROJECT_ID — format=”value(projectNumber)”)@cloudbuild.gserviceaccount.com \
 — role ‘roles/cloudkms.signerVerifier’

Configure Binary Authorization default policy like below

Further considerations

Purpose built OS images
Reducing Attack surface (limiting/disabling Port/user/services/packages)
Kernel Hardening with AppArmor & Seccomp
Behavioural Analytics/Falco
Managing Secrets
Regular Upgrade
Restrict API Server Access
Least privilege access using RBAC
Exercise caution with Service accounts
CIS Benchmarking
Secure Dashboard
Secure Ingress to Cluster
Network Policies
Auditing

Note, this was also posted on my blog and instructions in the github readme