Building Multi-arch Docker Images

Note: Most of this post is written with Gitlab-CI in mind, but I’ve included a sample commit with a similar config for Google Cloud Build near the bottom.

A few weeks ago I rebuilt my Kubernetes cluster, it now has a combination of amd64 and arm nodes (adding a box with an i5 processor to my collection of Raspberry Pis). When I initially launched all my Docker images I noticed the ones that got scheduled on the amd64 box crashed with fatal errors. As a quick fix, I set nodeAffinity so everything would prefer the arm nodes, but it’s not a long-term solution.

spec:
  selector:
    matchLabels:
      app: appname
  replicas: 1
  template:
    metadata:
      labels:
        app: appname
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - arm

Docker has multi-arch support, which lets you build separate images for each architecture/environment and glue them together using a manifest file. Here’s how I migrated my simple Go app to a multi-arch setup.

Update the Dockerfile

Previously, my Dockerfile was set up to always build for arm processors using:

RUN GOARCH=arm GOARM=6 go build -o /go/bin/import

To support multiple architectures, I moved GOARM and GOARCH to an argument (opt) defining environment variables which can be injected and changed as needed:

ARG opts
RUN env ${opts} go build -o /go/bin/import

Updating the CI script (.gitlab-ci.yml)

I’m using gitlab for my continuous builds. Building one image was simple, I just needed commands like:

docker build --pull -t $IMAGE .
docker push $IMAGE

Multi-arch requires a few more steps:

1. Building images for each architecture.
2. Pushing images for each architecture.
3. Creating a manifest file that references each architecture.
4. Annotating which tag is the special arm build.
5. Pushing that manifest file.

To create separate images for each architecture, we can duplicate the commands and set different build-args to pass in the opts we set up in the Dockerfile. Also, add a tag for each build with the architecture information so we can tell which image is which. Since I run my binaries in the scratch [empty] container, I also need to disable cgo for the amd64 build — this is done by default when cross-compiling (like for arm).

docker build --build-arg opts="CGO_ENABLED=0 GOARCH=amd64" --pull -t $IMAGE:amd64 .
docker push $IMAGE:amd64
docker build --build-arg opts="GOARCH=arm GOARM=6" --pull -t $IMAGE:arm32v6 .
docker push $IMAGE:arm32v6

The docker manifest command is still in experimental, so before we can use it we need to turn it on with an environment variable, DOCKER_CLI_EXPERIMENTAL: enabled. Without that, you’ll get errors like “docker manifest create is only supported on a Docker cli with experimental cli features enabled”. Note: The Docker cli is different from the Docker daemon, enabling experimental mode for the Docker daemon won’t help.

To create the manifest with the two images, tag it with our architecture information, and push it, use commands like:

docker manifest create $IMAGE $IMAGE:amd64 $IMAGE:arm32v6
docker manifest annotate $IMAGE $IMAGE:arm32v6 --os linux --arch arm
docker manifest push $IMAGE

Fin

Google Cloud Build is much more strict at escaping commands and flags configured in the cloudbuild.yaml file and requires a bit more syntax to break out each docker command into a separate step. This commit shows the changes I made with sample cloudbuild.yaml and Dockerfile files below.

Gitlab, as I showed above, is a bit simpler. Here are my final .gitlab-ci.yml and Docker files respectively.