Building and Testing Multi-Arch Images with Docker Buildx and QEMU

Building Docker images for multiple architectures has become increasingly important as ARM64 machines, and other processor architectures and operating systems, have grown in popularity in recent years. In this guide, we will walk through how to build a debian-based image for both linux/amd64 and linux/arm64 using Docker’s buildx tool and use QEMU to emulate an ARM environment so that we can build for multiple platforms. We will also go over how to leverage arguments automatically populated by Docker such as TARGETOS and TARGETARCH.

For this tutorial, we will be installing helm, a package manager for Kubernetes (this is just an example, any other application or CLI tool can also be used), in the Dockerfile for the particular operating systems and machine architectures that we specify in the build arguments. We will then pull the Docker images and verify that the correct helm binaries have been installed by running the helm version command.

First, you will need to install buildx by following these instructions on the Docker site. Note that to use buildx, you must currently have Docker version 19.03 or above. Verify that the installation succeeded by running docker buildx version.

Next, we will need to set up QEMU, an open source machine emulator, to build the image for both ARM and AMD machines. Run the following commands to install QEMU packages and execute the registering steps, as documented here:

sudo apt-get install qemu binfmt-support qemu-user-static # Install the qemu packages
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes # This step will execute the registering scripts 
docker run --rm -t arm64v8/ubuntu uname -m # Testing the emulation environment
aarch64

(Note: if you run into issues building the images for other platforms later in the tutorial, you can try following these steps to register the QEMU binaries instead).

To start using buildx to build multi-arch images, you will first need to run the buildx create command and set the platforms to be supported by the builder instance with the --platform flag. Though we will be building for linux/amd64and linux/arm64, other platforms and operating systems can also be specified. For instance, builds for windows on AMD64 would set the --platform as windows/amd64. More information on target platforms for builds can be found on the Docker docs site.

Note that running buildx create is only needed the first time you run buildx. In following runs, you could reuse this same builder, or append new nodes to existing builders.

docker buildx create --platform linux/amd64,linux/arm64 --use

When you run docker buildx ls, you should see the builder instance you just created with the platforms you specified, as follows:

NAME/NODE         DRIVER/ENDPOINT           STATUS         PLATFORMS
hungry_merkle     docker-container
  hungry_merkle0  unix:///var/run/docker.sock inactive linux/amd64*, linux/arm64*

Next, set up the following Dockerfile, which has the base image as a configurable build argument:

To build this image for both linux/amd64 and linux/arm64, we need to run the buildx build command and specify the target platforms with the --platform flag, similar to what we did with buildx create.

When the --platform is specified, Docker will automatically set the TARGETARCH, TARGETOS, TARGETPLATFORM, and TARGETVARIANTwith the respective target architecture, operation system, platform, and variant component of the platform, based on the build result. So, for example, if the target platform is linux/amd64, then TARGETOS would be set to linux, TARGETARCH would be set to amd64, and TARGETPLATFORM would be set to linux/amd64.

In addition to these args, Docker also automatically sets BUILDPLATFORM, BUILDOS, BUILDARCH, and BUILDVARIANT, which are populated based on the platform of the machine performing the build, as opposed to the build result or target. More details on docker buildx args can be found on the Docker docs site.

For the base image, we will use the debian-base image from the Google Container Registry (GCR): k8s.gcr.io/build-image/debian-base:bullseye-v1.4.1.

First, make sure that you have logged into your Docker account:

DOCKER_USER_ID=<docker_user_id>
docker login -u $DOCKER_USER_ID

Then, run the following buildx command:

docker buildx build --build-arg base_image=k8s.gcr.io/build-image/debian-base:bullseye-v1.4.1 --output="type=docker" -t $DOCKER_USER_ID/multi-arch-helm --platform linux/amd64,linux/arm64 .

After running this command, however, you will receive the following error:

error: docker exporter does not currently support exporting manifest list

This error occurs because buildx currently does not support loading multi platform build results to docker images. Instead, we will need to push the multi-arch build result to a registry by using type=registry for the --output instead of type=docker, as follows:

docker buildx build --build-arg base_image=k8s.gcr.io/build-image/debian-base:bullseye-v1.4.1 --output="type=registry" -t $DOCKER_USER_ID/multi-arch-helm --platform linux/amd64,linux/arm64 .

The build should succeed, and you should also be able to see the output of the two echo commands in the Dockerfile, and both the AMD64 and ARM64 helm binaries being installed:

=> [linux/amd64 2/4] RUN echo "I'm building for amd64 on linux"
=> [linux/arm64 2/4] RUN echo "I'm building for arm64 on linux"
=> [linux/amd64 4/4] RUN curl https://get.helm.sh/helm-v3.9.3-linux-amd64.tar.gz -o helm.tar.gz
=> [linux/arm64 4/4] RUN curl https://get.helm.sh/helm-v3.9.3-linux-arm64.tar.gz -o helm.tar.gz

On Docker Hub, you should be able to see the image with both the AMD64 and ARM64 digests.

Note that although this appears to be a single Docker image with one tag, it is actually a set of two images — one for linux/amd64, the other for linux/arm64 — organized by a manifest list, as explained by this guide on Google Cloud docs.

You can also inspect the manifest in your terminal by running docker manifest inspect — note that you will need Docker experimental cli features enabled to do this:

export DOCKER_CLI_EXPERIMENTAL=enabled
docker manifest inspect $DOCKER_USER_ID/multi-arch-helm

The output should look something like this:

Now, let us test both the AMD64 and ARM64 images, which we are able to do thanks to QEMU.

# Use the digest for the AMD64 image
docker run $DOCKER_USER_ID/multi-arch-helm@sha256:207c196db5b881d7096696d41cc47a2be477607c7815860aa27c78edd208d9e4 helm version

The output should show that the image has been successfully pulled, and the helm version command should print the version (v3.9.3) that was specified in the Dockerfile:

version.BuildInfo{Version:"v3.9.3", GitCommit:"414ff28d4029ae8c8b05d62aa06c7fe3dee2bc58", GitTreeState:"clean", GoVersion:"go1.17.13"}

You can also run uname -m to verify that the architecture is correct:

docker run $DOCKER_USER_ID/multi-arch-helm@sha256:207c196db5b881d7096696d41cc47a2be477607c7815860aa27c78edd208d9e4 uname -m 
x86_64

Now, let us do the same for the ARM64 image:

# Use the digest for the ARM64 image
docker run $DOCKER_USER_ID/multi-arch-helm@sha256:1c3ceee2d8de1f4a21271db1b453ee754d0209a600ebd72f2057f417ef85d277 helm version

You should receive the same output:

version.BuildInfo{Version:"v3.9.3", GitCommit:"414ff28d4029ae8c8b05d62aa06c7fe3dee2bc58", GitTreeState:"clean", GoVersion:"go1.17.13"}

Run uname -m to verify the architecture:

docker run $DOCKER_USER_ID/multi-arch-helm@sha256:1c3ceee2d8de1f4a21271db1b453ee754d0209a600ebd72f2057f417ef85d277 uname -m 
aarch64

If these commands succeed, then congrats — you’ve successfully built and tested a multi-arch image with Docker buildx and QEMU!