Building Multi-arch Docker Images
Note: Most of this post is written with Gitlab-CI in mind, but I’ve included a sample commit with a similar config for Google Cloud Build near the bottom.
A few weeks ago I rebuilt my Kubernetes cluster, it now has a combination of amd64
and arm
nodes (adding a box with an i5 processor to my collection of Raspberry Pis). When I initially launched all my Docker images I noticed the ones that got scheduled on the amd64
box crashed with fatal errors. As a quick fix, I set nodeAffinity
so everything would prefer the arm
nodes, but it’s not a long-term solution.
spec:
selector:
matchLabels:
app: appname
replicas: 1
template:
metadata:
labels:
app: appname
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- arm
Docker has multi-arch support, which lets you build separate images for each architecture/environment and glue them together using a manifest file. Here’s how I migrated my simple Go app to a multi-arch setup.
Update the Dockerfile
Previously, my Dockerfile was set up to always build for arm processors using:
RUN GOARCH=arm GOARM=6 go build -o /go/bin/import
To support multiple architectures, I moved GOARM
and GOARCH
to an argument (opt
) defining environment variables which can be injected and changed as needed:
ARG opts
RUN env ${opts} go build -o /go/bin/import
Updating the CI script (.gitlab-ci.yml)
I’m using gitlab for my continuous builds. Building one image was simple, I just needed commands like:
docker build --pull -t $IMAGE .
docker push $IMAGE
Multi-arch requires a few more steps:
- Building images for each architecture.
- Pushing images for each architecture.
- Creating a manifest file that references each architecture.
- Annotating which tag is the special
arm
build. - Pushing that manifest file.
To create separate images for each architecture, we can duplicate the commands and set different build-args to pass in the opts
we set up in the Dockerfile. Also, add a tag for each build with the architecture information so we can tell which image is which. Since I run my binaries in the scratch [empty] container, I also need to disable cgo for the amd64 build — this is done by default when cross-compiling (like for arm).
docker build --build-arg opts="CGO_ENABLED=0 GOARCH=amd64" --pull -t $IMAGE:amd64 .
docker push $IMAGE:amd64docker build --build-arg opts="GOARCH=arm GOARM=6" --pull -t $IMAGE:arm32v6 .
docker push $IMAGE:arm32v6
The docker manifest command is still in experimental, so before we can use it we need to turn it on with an environment variable, DOCKER_CLI_EXPERIMENTAL: enabled
. Without that, you’ll get errors like “docker manifest create is only supported on a Docker cli with experimental cli features enabled”. Note: The Docker cli is different from the Docker daemon, enabling experimental mode for the Docker daemon won’t help.
To create the manifest with the two images, tag it with our architecture information, and push it, use commands like:
docker manifest create $IMAGE $IMAGE:amd64 $IMAGE:arm32v6
docker manifest annotate $IMAGE $IMAGE:arm32v6 --os linux --arch arm
docker manifest push $IMAGE
Fin
Google Cloud Build is much more strict at escaping commands and flags configured in the cloudbuild.yaml
file and requires a bit more syntax to break out each docker
command into a separate step. This commit shows the changes I made with sample cloudbuild.yaml
and Dockerfile
files below.
Gitlab, as I showed above, is a bit simpler. Here are my final .gitlab-ci.yml
and Docker
files respectively.