Optimizing CI in Google Cloud Build

In part 1 of this article below, we will explore multiple methods to performance tune the continuous integration (CI) process using Google Cloud Build, so that your overall build and deployment time will decrease, regardless of your software stack or complexity.

In this article, we’ll cover the following topics

• Principles related to Container image size
• Small Container Images — Use multi-stage and distroless with Golang
• Spring Layertools multi-stage Dockerfile
• About distroless container images from Google
• Google Cloud buildpacks
• Using Google Cloud’s Buildpacks from Cloud Build
• Multi Architecture Container Images
• Select the VM Shape for faster builds within Cloud Build
• Container Image Caching
• Migrate remote files which you own, to your registry
• Artifact Registry is the evolution of Google Container Registry (GCR)
• What’s not been covered
• Summary

In part 2, we’ll cover more advanced topics which include

• What is Bazel and how it works
• Using Bazel for deterministic builds
• Example of how faaassst that Bazel actually is
• Setting up a GCS bucket with bazel for remote caching
• Using Bazel from Cloud Build
• Bazel: Exploring remote persistent workers
• Use parallel build steps
• Kaniko or Docker caching
• Summary

Getting Started

Google Cloud’s DevOps stack is a comprehensive suite of tools and practices designed to streamline software development, deployment, and management and built on the Google Cloud Platform (GCP).

In this article, we’re going to zoom in on the use of Cloud Build, specifically around identifying areas in which to optimize continuous integration.

Principles related to Container image size

• Container images are immutable — they contain all dependencies and run the same anywhere
• It’s important not to re-create an OS within a container image, you want to include your application and the runtime only
• As a general principle, don’t store data or large blobs in your application container, however you could always use GCS fuse to mount a volume containing a large blob accessible by your application
• Create as few layers as possible in your container image
  Since Docker 1.10 the COPY, ADD and RUN statements add a new layer to your image

Small Container Images — Use multi-stage with Golang

Use multi-stage build as the Golang runtime is based upon Debian and will result in large container images.

FROM golang:1.22 as build
WORKDIR /go/src/app
COPY . .
RUN go mod download
RUN go vet -v
RUN go test -v
RUN CGO_ENABLED=0 go build -o /go/bin/app

FROM gcr.io/distroless/static-debian12
EXPOSE 8080
COPY - from=build /go/bin/app /
CMD ["/app"]

Spring Layertools

• Leverage Spring Layertools [link] to generate efficient container images (SpringBoot > 2.3)
• Spring Layertools is a feature of Spring Boot that allows you to generate more efficient container images
• Incremental layer rebuilding of changed layers which speeds up build times
• Layer your JAR file by change frequency
• dependencies
• snapshot-dependencies
• resources
• application

Spring Layertools multi-stage Dockerfile

If you re-compile your code more often than you upgrade the version of Spring Boot you use, it’s often better to separate things a bit. The builder stage extracts the directories that are needed later. Each of the COPY commands relates to the layers extracted by the jarmode. Read further here.

FROM eclipse-temurin:22-jre as builder
WORKDIR application
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} application.jar
RUN java -Djarmode=layertools -jar application.jar extract

FROM eclipse-temurin:22-jre
WORKDIR application
COPY - from=builder application/dependencies/ ./
COPY - from=builder application/spring-boot-loader/ ./
COPY - from=builder application/snapshot-dependencies/ ./
COPY - from=builder application/application/ ./
ENTRYPOINT ["java", "org.springframework.boot.loader.JarLauncher"]

The Eclipse Temurin distribution of OpenJDK, was developed by the Eclipse Adoptium community and has rapidly gained popularity among Java developers since its initial Java SE release in August 2021. It is now the most widely used OpenJDK build in production environments. Temurin binaries are open-source licensed, cross-platform and TCK-certified for Java SE.

Small Container Images — Use distroless from Google

“Distroless” images contain only your application and its runtime dependencies. They do not contain package managers, shells or any other programs you would expect to find in a standard Linux distribution.

FROM node:8 as build
WORKDIR /app
COPY package.json index.js ./
RUN npm install

FROM gcr.io/distroless/nodejs
COPY - from=build /app /
EXPOSE 3000
CMD ["index.js"]

From the shell, examine the size of your container image.

$ docker images | grep node-distroless
node-distroless 3b44b3b7f1a5 87.4MB

About distroless container images from Google

If you want to manually write your Dockerfiles, distroless simply provides the base image.

• Supports architectures — amd64, arm64, arm, s390x, ppc64le
• Your app is a best practice employed by Google
• Images for nodejs18, nodejs20, java11, java17, python3, base-debian12 and more
• Example Dockerfiles here
• Pro-tip: Distroless images are minimal and lack shell access. If you run into issues during the development it can be useful to use the :debug image that provides a shell so you can work out what is going wrong.

Google Cloud buildpacks

Takes your app code, writes your Dockerfile, compiles code, creates and pushes a container image.

https://cloud.google.com/docs/buildpacks/overview

GCP’s Buildpacks — Secure by Default

• Base OS and packages taken from Ubuntu LTS distributions (18, 22, etc.)
• Critical components backed by Canonical CVE protection over Main + Universe
• GCP Buildpacks provide all the necessary ingredients to get customer source code up and running
• Base images patched weekly against security vulnerabilities.

Using Google Cloud’s Buildpacks from Cloud Build

To use Build Packs from Cloud Build, here’s some sample cloudbuild.yaml code

steps:
- name: "gcr.io/k8s-skaffold/pack"
id: "BUILD and PUSH with cloudpacks"
script: pack build - builder=gcr.io/buildpacks/builder - publish" gcr.io/<project>/<path-to-image>:$COMMIT_SHA"
- name: "gcr.io/cloud-builders/gcloud"
id: "DEPLOY to cloud run"
script: gcloud run deploy "<name of service>" - image" gcr.io/<project>/<path-to-image>:$COMMIT_SHA" - region" <region>"
options:
automapSubstitutions: true

Multi Architecture Container Images

If you’ve completed migrating to arm64, then there’s no need to build images for amd64.

1. Building for both architectures will increase your container image size with no benefit
2. And also slow down your build

$ docker buildx build . -t PATH_TO_REGISTRY - platform linux/amd64,linux/arm64 - push
# see size
$ docker images | grep myapp
$ docker buildx build . -t PATH_TO_REGISTRY - platform linux/arm64 - push
# Compare size
$ docker images | grep myapp

Cloud Build — Select the VM Shape for faster builds

The free tier provides the e2-standard-2 shape, however you will want more CPU power.

Here’s an example of the performance improvements that you could see, when you throw more compute

# With machineType e2-standard-2 (default)
Step #2 - "build_java": INFO: Elapsed time: 56.452s, Critical Path: 4.71s

Step #2 - "build_java": INFO: Elapsed time: 19.044s, Critical Path: 1.77s
Select the machinetype that you desire, in your cloudbuild.yaml
options:
 machineType: 'E2_HIGHCPU_32'

https://cloud.google.com/build/docs/optimize-builds/increase-vcpu-for-builds
https://cloud.google.com/build/pricing

Container Image Caching

Each Docker image consists of layers stacked on top of each other. When you use the — cache-from flag, the build process will rebuild all layers starting from the modified layer until the end. This means that if you modify a layer at the beginning of your Docker build, using — cache-from won’t offer much advantage.

In your build configuration file, you can add the — cache-from argument to indicate the image you want to use as a cache source for your Docker build.

It is recommended to use a cached docker image to speed up your container image builds.

options:
 - name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t',
'us-central1-docker.pkg.dev/${PROJECT_ID}/${_ARTIFACT_REGISTRY_REPO}/myimage:latest', '--cache-from', 
'us-central1-docker.pkg.dev/${PROJECT_ID}/${_ARTIFACT_REGISTRY_REPO}/myimage:latest', '.']

Why not to use an on-prem based container registry

• Avoid using an on-prem based container registry, a design pattern with high latency
• Many network hops means an increased latency over using a registry within Google Cloud
• Pushing and pulling container images will take longer
• Your extending your trust perimeter — on-prem and cloud security teams is a risk
• Internet egress costs

Migrate remote files which you own, to your registry

Don’t add dependencies in your Dockerfile which are located on a separate Container registry
if you own these files, as it’ll create additional latency as these images are pulled down from a separate registry.

Ensure that these dependencies are stored on the same container registry.

The common cause for this, is when the on-prem container registry has not been fully migrated to the cloud.

Don’t do this in your Dockerfile.

ADD https://2ndregistry.mycorp.com/vault/${VERSION}/vault_${VERSION}_linux_amd64.zip /tmp/

Use a managed container registry in the Cloud

Any on-prem container registry will have multiple hops to and from your container runtime in the cloud resulting in increased latency in your CI due to publishing and pulling container images.
Base your Google Cloud Artifact Registry in the region closest to your container runtimes.

Artifact Registry is the evolution of GCR

Google Cloud’s Artifact Registry is the new and updated container registry system, which has replaced Google Container Registry (GCR).

It is scalable, reliable, and secures build artifact management across containers and packages.

What’s not been covered

Continuous Testing is a part of the Continuous Integration cycle however I’ve purposefully not covered this because of the complexities involved with different types of testing and tooling provided by 3rd parties. I suggest that you explore how much of your customers CI cycle is spent within testing as part of your investigation.

Container Runtime Optimisation (we’re focusing on Build Optimization) …
zStandard compression of container images allows for 3x faster container image layer decompression over standard gzip, but it is incompatible with GKE image streaming.

Summary

We’ve covered a myriad of techniques and configuration options which will help you to achieve a faster build and deployment time. To wrap up, here are a few final tips.

When you begin this work, time how long that your CI process is taking and then measure once you’ve implemented some of these techniques.

Do experiment with the different techniques that I’ve outlined above because that’s the best method to determine how large of a performance gain that you can achieve.