Preparation toward running Docker on ARM Mac: Building multi-arch images with Docker BuildX

Today, Apple announced that they will ditch Intel and switch to ARM-based chips. This means that Docker for Mac will only be able to run ARM images natively.

So, Docker will no longer be useful when you want to run the same image on Mac and on x86_64 cloud instances? Nope. Docker will remain useful, as long as the image is built with support for multi-architectures.

If you are still building images only for x86_64, you should switch your images to multi-arch as soon as possible, ahead of the actual release of ARM Mac.

Note: ARM Mac will be probably able to run x86_64 images via an emulator, but the performance penalty will be significant.

Building multi-arch images with Docker BuildX

To build multi-architecture images, you need to use Docker BuildX plugin (docker buildx build ) instead of the well-known docker build command.

If you have already heard about the BuildKit mode ( export DOCKER_BUILDKIT=1 ) of the built-in docker build command, you might be confused now, and you might be wondering BuildKit was deprecated. No worry, BuildKit is still alive. Actually Docker BuildX is built on BuildKit technology as well, but significantly enhanced compared to the BuildKit mode of the built-in docker build. Aside from multi-arch build, Docker BuildX also comes with a lot of innovative features such as distributed build on Kubernetes clusters. See my previous post for the further information.

Docker BuildX is installed by default if you are using Docker for Mac on macOS. If you are using Docker for Linux on your own Linux VM (or baremetal Linux), you might need to install Docker BuildX separately. See https://github.com/docker/buildx#installing for the installation steps on Linux.

The three options for multi-arch build

Docker BuildX provides the three options for building multi-arch images:

1. QEMU mode (easy, slow)
2. Cross-compilation mode (difficult, fast)
3. Remote mode (easy, fast, but needs an extra machine)

The easiest option is to use QEMU mode (Option 1), but it incurs significant performance penalty. So, using cross compilation (Option 2) is recommended if you don’t mind adapting Dockerfiles for cross compilation toolchains. If you already have an ARM machine (doesn’t need to be Mac of course), the third option is probably the best choice for you.

Option 1: QEMU mode (easy, slow)

This mode uses QEMU User Mode Emulation for running ARM toolchains on a x86_64 Linux environment. Or the quite opposite after the actual release of ARM Mac: x86_64 toolchains on ARM.

The QEMU integration is already enabled by default on Docker for Mac. If you are using Docker for Linux, you need to enable the QEMU integration using the linuxkit/binfmt tool:

$ uname -sm
Linux x86_64
$ docker run --rm --privileged linuxkit/binfmt:v0.8
$ ls -1 /proc/sys/fs/binfmt_misc/qemu-*
/proc/sys/fs/binfmt_misc/qemu-aarch64
/proc/sys/fs/binfmt_misc/qemu-arm
/proc/sys/fs/binfmt_misc/qemu-ppc64le
/proc/sys/fs/binfmt_misc/qemu-riscv64
/proc/sys/fs/binfmt_misc/qemu-s390x

Then initialize Docker BuildX as follows. This step is required on Docker for Mac as well as on Docker for Linux.

$ docker buildx create --use --name=qemu
$ docker buildx inspect --bootstrap
...
Nodes:
Name:      qemu0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

Make sure the command above shows both linux/amd64 andlinux/arm64 as the supported platforms.

As an example, let’s try dockerizing a “Hello, world” application (GNU Hello) for these architectures. Here is the Dockerfile:

FROM debian:10 AS build
RUN apt-get update && \
  apt-get install -y curl gcc make
RUN curl https://ftp.gnu.org/gnu/hello/hello-2.10.tar.gz | tar xz
WORKDIR hello-2.10
RUN LDFLAGS=-static \
  ./configure && \
  make && \
  mkdir -p /out && \
  mv hello /out
FROM scratch
COPY --from=build /out/hello /hello
ENTRYPOINT ["/hello"]

This Dockerfile doesn’t need to have anything specific to ARM, because QEMU can cover the architecture differences between x86_64 and ARM.

The image can be built and pushed to a registry using the following command. It took 498 seconds on my MacBookPro 2016.

$ docker buildx build \
  --push -t example.com/hello:latest \
  --platform=linux/amd64,linux/arm64 .
[+] Building 498.0s (17/17) FINISHED

The images contains binaries for both x86_64 and ARM. The image can be executed on any machine with these architectures, without enabling QEMU.

$ docker run --rm example.com/hello:latest
Hello, world!
$ ssh me@my-arm-instance
[me@my-arm-instance]$ uname -sm
Linux aarch64
[me@my-arm-instance]$ docker run --rm example.com/hello:latest
Hello, world!

Option 2: Cross-compilation mode (difficult, fast)

The QEMU mode incurs significant performance penalty for interpreting ARM instructions on x86_64 (or x86_64 on ARM after the actual release of ARM Mac). If the QEMU mode is too slow for your application, consider using the cross-compilation mode instead.

To cross-compile the GNU Hello example, you need to modify the Dockerfile significantly:

FROM --platform=$BUILDPLATFORM debian:10 AS build
RUN apt-get update && \
  apt-get install -y curl gcc make
RUN curl -o /cross.sh https://raw.githubusercontent.com/tonistiigi/binfmt/18c3d40ae2e3485e4de5b453e8460d6872b24d6b/binfmt/scripts/cross.sh && chmod +x /cross.sh
RUN curl https://ftp.gnu.org/gnu/hello/hello-2.10.tar.gz | tar xz
WORKDIR hello-2.10
ARG TARGETPLATFORM
RUN /cross.sh install gcc | sh
RUN LDFLAGS=-static \
  ./configure --host $(/cross.sh cross-prefix) && \
  make && \
  mkdir -p /out && \
  mv hello /out
FROM scratch
COPY --from=build /out/hello /hello
ENTRYPOINT ["/hello"]

This Dockerfile contains two essential variables: BUILDPLATFORM and TARGETPLATFORM .

BUILDPLATFORM is pinned to the host platform ("linux/amd64" ). TARGETPLATFORM is conditionally set to the target platforms ( "linux/amd64" and "linux/arm64" ) and is used for setting up cross compilation tool chains like aarch64-linux-gnu-gcc via a helper script cross.sh . This kind of helper script can be a mess depending on the application’s build scripts, especially in C applications.

For comparison, cross-compiling Go programs is relatively straightforward:

FROM --platform=$BUILDPLATFORM golang:1.14 AS build
ARG TARGETARCH
ENV GOARCH=$TARGETARCH
RUN go get github.com/golang/example/hello && \
  go build -o /out/hello github.com/golang/example/hello
FROM scratch
COPY --from=build /out/hello /hello
ENTRYPOINT ["/hello"]

The image can be built and pushed as follows without enabling QEMU. Cross-compiling GNU hello took only 95.3 seconds.

$ docker buildx create --use --name=cross
$ docker buildx inspect --bootstrap cross
Nodes:
Name:      cross0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/amd64, linux/386
$ docker buildx build \
  --push -t example.com/hello:latest \
  --platform=linux/amd64,linux/arm64 .
[+] Building 95.3s (16/16) FINISHED

Option 3: Remote mode (easy, fast, but needs an extra machine)

The third option is to use a real ARM machine (e.g. an Amazon EC2 A1 instance) for compiling ARM binaries. This option is as easy as Option 1 and yet as fast as Option 2.

To use this option, you need to have an ARM machine accessible via docker CLI. The easiest way is to use ssh://<USERNAME>@<HOST> URL as follows:

$ docker -H ssh://me@my-arm-instance info
...
 Architecture: aarch64
...

Also, depending on the network configuration, you might want to add ControlPersist settings in ~/.ssh/config for faster and stabler SSH connection.

ControlMaster auto
ControlPath ~/.ssh/control-%C
ControlPersist yes

This remote ARM instance can be registered to Docker BuildX using the following commands:

$ docker buildx create --name remote --use
$ docker buildx create --name remote \
  --append ssh://me@my-arm-instance
$ docker buildx build \
  --push -t example.com/hello:latest \
  --platform=linux/amd64,linux/arm64 .
[+] Building 113.4s (22/22) FINISHED

The Dockerfile is same as Option 1.

FROM debian:10 AS build
RUN apt-get update && \
  apt-get install -y curl gcc make
RUN curl https://ftp.gnu.org/gnu/hello/hello-2.10.tar.gz | tar xz
WORKDIR hello-2.10
RUN LDFLAGS=-static \
  ./configure && \
  make && \
  mkdir -p /out && \
  mv hello /out
FROM scratch
COPY --from=build /out/hello /hello
ENTRYPOINT ["/hello"]

In this example, an Intel Mac is used as the local and an ARM machine is used as the remote. But after the release of actual ARM Mac, you will do the opposite: use an ARM Mac as the local and an x86_64 machine as the remote.

Performance comparison

1. QEMU mode: 498.0s
2. Cross-compilation mode: 95.3s
3. Remote mode: 113.4s

The QEMU mode is the easiest, but taking 498 seconds seems too slow for compiling “hello world”. The cross-compilation mode is the fastest but modifying Dockerfile can be a mess. I suggest using the remote mode whenever possible.

We’re hiring!

NTT is looking for engineers who work in Open Source communities like Kubernetes & Docker projects. If you wish to work on such projects please do visit our recruitment page.

To know more about NTT contribution towards open source projects please visit our Software Innovation Center page. We have a lot of maintainers and contributors in several open source projects.

Our offices are located in the downtown area of Tokyo (Tamachi, Shinagawa) and Musashino.