Using Cloud Build with a multi-stage Docker build file and a private repo

Published in

Google Cloud - Community

5 min readJan 31, 2021

Using a multi-stage Docker file is a great way to use a container to build a Go app or service and then package the resulting binary in the most minimal container image possible. There’re a lot of great examples out there, but here’s the one that intro’d me to the concept from back in the day, Building Docker Images for Static Go Binaries | by Kelsey Hightower.

Cloud Build is a great managed build pipeline process with a lot of flexibility … so let’s combine the two! Oh, and let’s also throw in a twist, which is really the point of this note: the multi-stage build uses a module located in a private git repo.

To do this, I put together few different concepts to achieve the following:

Use Cloud Build to create a Go service, where …
the Go service relies on a module in a private git repo via …
a multistage Docker build file

When it comes down to it, I need a secure way to provide the multi-stage container build process access to a private key.

Cloud Build’s docs …

Cloud Build’s documentation has an article about how to access private GitHub repositories using a key stored in Secret Manager with all the required setup and a useful straightforward build configuration example.

To recap that article, one sets up a key pair and applies the public key to a particular Github repository’s deployment settings (a key for a single repository, a deploy key) then uploads the private key to Secret Manager. This key will be accessed during a Cloud Build step for the ephemeral container-based build process and points out the proper role to provide the Cloud Build service account to access secrets, Secret Manager Secret Accessor (roles/secretmanager.secretAccessor). The key retrieved from Secret Manager is not stored in any layer of the final image being built. That article concludes with a Cloud Build config that clones a private repo’s contents to the local build workspace, building the code within the Cloud Build pipeline. Nifty, but we’ve got a multi-stage Dockerfile that will be used to do the building.

Git via SSH with Secret Manager … in a Dockerfile

With Go modules, building a Go app will first download the dependent modules specified. This is where the tricky bits happen — Go needs to know how to access a private git repo, via SSH, and in a container.

Go typically uses HTTPS to access git repos, but to use an SSH key in a multistage Dockerfile, one has to do a few things:

Have a key available on the host to provide to the multistage container
Make the key available within the container via a build-time ARG variable
Configure Go to use SSH rather than HTTPS for the target git repo

Lastly, pass in that built-time ARG variable (the ssh private key) via the docker build’s — build-arg parameter.

The first part is easy enough, it’s documented above: use Secret Manager to obtain and write the key to a file. In this case, the host is the build process and the place to store this key temporarily is a Volume mount.

Second, I modified the Dockerfile to bring in special environment variable passed as a build-arg to docker build ARG and then a few commands to make sure the build container registered the SSH key and also makes the global switch from HTTPS (default) to SSH for accessing the git repo, in this case GitHub. One could switch to SSH access with other git repos like gitlab, bitbucket, etc. too. Since I’m using GitHub here, I didn’t bother adding any of the other repos.

The next part of configuring Go to use private repositories is adding in the GOPRIVATE env variable to contain prefixes of repos that are private.

Here’s the Dockerfile in full:

# Start by building the application.
FROM golang:1.15-buster as buildARG SSH_PRIVATE_KEYWORKDIR /root/src/app
ADD . /root/src/app# import the private key
# note: intermediary images are deleted and not present in final image layers
RUN mkdir -p ~/.ssh && umask 0077 && echo "${SSH_PRIVATE_KEY}" > ~/.ssh/id_rsa \
&& git config --global url."git@github.com:".insteadOf https://github.com/ \
&& ssh-keyscan github.com >> ~/.ssh/known_hosts# https://golang.org/cmd/go/#hdr-Module_configuration_for_non_public_modules
ENV GOPRIVATE github.com/ghchinoy/robotreadmeRUN go get -d -v ./...
RUN go build -o /root/bin/app# Now copy it into our base image.
FROM gcr.io/distroless/base-debian10
COPY --from=build /root/bin/app /
CMD ["/app"]

Things I’ve learned along the way

Cloud Build’s escape hatch: Docker entrypoint

Using a base builder along with the entrypoint parameter allows for a greater flexibility in the arguments to be used — you’re not restricted to the parameters that the builder’s entrypoint can take. I struggled with this a bit since the default parameters didn’t properly do shell substitution (where I wanted to cat in the private key). Once I did this, I had a minor revelation that all cloud builders can be treated just like any other container.
The docker image used in a container build step does mount volumes. Typically, if you use the default cloud builder you see examples like the following which, even if you were to add in a volume mount, wouldn’t indicate how to access the volume in the args param. With the entrypoint one can reference the mount volume where, in this case, the ssh key will be.

Typical build step using the docker cloud builder, nothing special here

- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'gcr.io/$PROJECT_ID/$_SERVICE/v1, '.']

This gets replaced with a two-step, the first to retrieve the key from Secret Manager (where the secret is named my-github-deploy-key) and add it to the builder’s workspace as a volume and a second that builds the image, using an entrypoint and that volume containing the key.

# Access the id_github file from Secret Manager
- name: gcr.io/cloud-builders/gcloud
  entrypoint: 'bash'
  args: [ '-c', 'gcloud secrets versions access latest --secret=my-github-deploy-key > /root/.ssh/id_github' ]
  volumes:
  - name: 'ssh'
    path: /root/.ssh# Build the container image
- name: 'docker'
  entrypoint: 'sh'
  args: [ '-c', 'docker build --build-arg SSH_PRIVATE_KEY="$(cat /root/.ssh/id_github)" -t gcr.io/$PROJECT_ID/$_SERVICE .' ]
  volumes:
  - name: 'ssh'
    path: /root/.ssh

Google Cloud Secret Manager and Cloud KMS

The alternative for managing secrets is to use Cloud KMS. Lots of examples around using Cloud KMS which made me think that not too many people have used Cloud Secret Manager for this particular purpose. This article has a good rundown of the differences between Secret Manager and Cloud KMS: Secret Manager conceptual overview | Secret Manager Documentation

Cloud Build’s Workspace — it’s the shared temp space for a Cloud Build pipeline between steps and can be used for state between steps. While I didn’t use this in the final example, it sure helped while I was creating it.

Using Cloud Build with a multi-stage Docker build file and a private repo

Things I’ve learned along the way

Written by G. Hussain Chinoy