Building x86_64 Docker Containers on Apple Silicon

Chris Torris Olsen
Dec 4, 2020 · 10 min read

I recently bought a new M1 Macbook Pro. My old one, a mid-2012 Retina, recently took its final breath, so there was no time to wait. For developers, the M1 is definitely not ready, but I decided to live with a few months of difficult transition rather than buy something that would soon be outdated.

One of the major components that at the time of writing don’t work is Docker, although it’s apparently close. For many of us, having Docker work is critical to our workflow, and I wanted to see if I could make that happen before Docker is officially ready. Also, just because Docker is released for the M1 doesn’t mean we can use containers meant to run on x86_64 seamlessly. The last part of this article is likely to be relevant for a good while, and understanding how emulation will support my work in the future is useful.

Image for post
Image for post
The promised land.

I’ll also note that my approach is unapologetically purist. We didn’t buy a laptop with a fancy new architecture to have it polluted by binaries from a world we left behind. Aside from what’s running inside your containers, and the Docker client, everything is running on the M1 without Rosetta 2.

Part 1: The Docker client

Part 2: Running a virtual machine

It’s about to get a little gnarly.

Wrapping Virtualization.framework

First, we want a folder to keep all our stuff in. Make sure you have the XCode command line tools available, and then download and compile vftool .

$ mkdir ubuntu-docker-m1
$ cd ubuntu-docker-m1
$ xcode-select --install
<Follow the on-screen instructions>
$ git clone https://github.com/evansm7/vftool.git
Cloning into 'vftool'...
<...>
$ clang -framework Foundation -framework Virtualization vftool/vftool/main.m -o vftool/vftool.bin
$ file vftool/vftool.bin
vftool/vftool.bin: Mach-O 64-bit executable arm64

Almost there, but not quite. We have vftool installed, but if you run it with a kernel at this point, you’ll get an error:

Configuration vaildation failure! Error Domain=VZErrorDomain Code=2 “Virtualization requires the “com.apple.security.virtualization” entitlement” UserInfo={NSDebugDescription=Virtualization requires the “com.apple.security.virtualization” entitlement}

To give it the entitlement, we need to sign the compiled binary. For that, you need a self-signed certificate. Open Keychain Access and use the Certificate Assistant to create one:

Image for post
Image for post

Clicking that will open a dialog. Set the certificate type to Code Signing, copy whatever is in the Name field, and create it.

$ codesign --entitlements vftool/vftool/vftool.entitlements -s "<NAME ON CERTIFICATE>" vftool/vftool.bin

Congratulations, you now have a working virtualisation wrapper.

Downloading and setting up a virtual machine running Ubuntu Focal

$ mkdir vm
$ curl -o - https://cloud-images.ubuntu.com/releases/focal/release/unpacked/ubuntu-20.04-server-cloudimg-arm64-vmlinuz-generic | gunzip > vm/vmlinuz
$ curl -o vm/initrd https://cloud-images.ubuntu.com/releases/focal/release/unpacked/ubuntu-20.04-server-cloudimg-arm64-initrd-generic
$ curl -o - https://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-arm64.tar.gz | tar zxvC vm

We also want to resize the disk image, otherwise we’ll run out of space. This would be easy to do with qemu-img , but it doesn’t compile natively on M1 with Homebrew. As I wrote earlier, this is a purist approach, so we’ll just use dd . It’s hacky, but it works. Let’s give it, say, roughly 20 gigs. (We’ll resize the actual partition later.)

$ ls -lh vm/focal-server-cloudimg-arm64.img
-rw-r--r-- 1 chris staff 1.3G 4 Dec 17:02 vm/focal-server-cloudimg-arm64.img
$ dd if=/dev/zero of=vm/focal-server-cloudimg-arm64.img seek=20000000 obs=1024 count=0
0+0 records in
0+0 records out
0 bytes transferred in 0.000011 secs (0 bytes/sec)
$ ls -lh vm/focal-server-cloudimg-arm64.img
-rw-r--r-- 1 chris staff 19G 4 Dec 17:03 vm/focal-server-cloudimg-arm64.img

Perfect. Now, let’s run this thing, but without specifying the root file system for now, so we can make some changes.

$ vftool/vftool.bin -k vm/vmlinuz -i vm/initrd -d vm/focal-server-cloudimg-arm64.img -m 1024 -a "console=hvc0"

Run that, and note the TTY it’s connected to, which in this case for me was /dev/ttys009 . Open a second terminal window and connect to it.

$ screen /dev/ttys009
<LOTS OF OUTPUT>
(initramfs)

Copy and paste the following (again, with thanks to droidix). This will change the root password to root , and set up SSH and networking.

mkdir /mnt
mount /dev/vda /mnt
chroot /mnt

touch /etc/cloud/cloud-init.disabled

echo 'root:root' | chpasswd

ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa
ssh-keygen -f /etc/ssh/ssh_host_dsa_key -N '' -t dsa
ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N '' -t ed25519

cat <<EOF > /etc/netplan/01-dhcp.yaml
network:
renderer: networkd
ethernets:
enp0s1:
dhcp4: true
version: 2
EOF

Optionally, while you’re at it, add your SSH key for ease of access later.

mkdir /root/.ssh
cat <<EOF > /root/.ssh/authorized_keys
ssh-rsa <...>
EOF

We’re done here, so let’s make that known to the VM.

exit
umount /dev/vda

Go back to the first terminal window and hit Ctrl+C to kill the VM.

Starting the virtual machine properly

$ vftool/vftool.bin -k vm/vmlinuz -i vm/initrd -d vm/focal-server-cloudimg-arm64.img -m 2048 -a "console=hvc0 root=/dev/vda"

Again, note the device, and use screen to connect to it like above. You should be met with an Ubuntu login prompt, for which both the username and password is root . If you set up your SSH key in the previous step, you can type hostname -I to get the IP address of your virtual machine, and ssh into it with the root user. This allows you to detach screen (hit Ctrl+A then D) if you find that annoying to deal with — the VM will happily live on without it attached. Either way, we now have Ubuntu for arm running in a virtual machine on M1.

# uname -a
Linux ubuntu 5.4.0-56-generic #62-Ubuntu SMP Mon Nov 23 19:17:58 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

Remember I mentioned we need to resize the filesystem? Let’s get that done now.

# resize2fs /dev/vda
resize2fs 1.45.5 (07-Jan-2020)
Filesystem at /dev/vda is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 3
The filesystem on /dev/vda is now 5000000 (4k) blocks long.

That’s it!

Part 3: The Docker daemon

# apt-get update
# apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common
# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
# add-apt-repository "deb [arch=arm64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
# apt-get update
# apt-get install docker-ce docker-ce-cli containerd.io

Note: At this point, my VM and my VPN (Mullvad) were not happy about each other’s presence, and the VM could not reach the internet unless I disconnected the VPN. Let me know if you find a solution.

At this point, Docker is up and running. The installation suggests you run the hello-world image, which you can do now if you want to.

# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
<...>

Now, let’s set up the Docker daemon to accept connections from the outside world. The instructions are here, but again, you can follow my instructions if you’re lazy.

# mkdir -p /etc/systemd/system/docker.service.d/
# cat <<EOF > /etc/systemd/system/docker.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd
EOF
# cat <<EOF > /etc/docker/daemon.json
{
"hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:2375"]
}
EOF
# systemctl daemon-reload
# systemctl restart docker.service

Let’s also make sure it’s listening.

# lsof -i | grep dockerd
dockerd 5794 root 7u IPv6 38573 0t0 TCP *:2375 (LISTEN)

The Docker daemon is set up and running. Make sure you get the IP address of the VM with hostname -I, and let’s move on to the client.

Part 4: Building x86_64 Docker containers

$ export DOCKER_HOST=192.168.64.8

You can verify the connection by checking that the hello-world image used above pops up.

$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
hello-world latest a29f45ccde2a 11 months ago 9.14kB

You can stop here if you just want to build images for ARM. Congratulations, you’ve done it. If Docker for Mac is updated for M1 in two days, you’ll probably feel like you just wasted a lot of time. If you want more, let’s venture on. It should be useful even after Docker is updated.

Setting up Docker Buildx

$ sed -i '' '/experimental/ s/disabled/enabled/' ~/.docker/config.json

Creating a small app to containerise

$ mkdir container
$ cd container
$ cat <<EOF > Dockerfile
FROM python:3.9
WORKDIR /app
COPY app.py .
RUN pip install fastapi uvicorn
EXPOSE 8000
CMD uvicorn --host 0.0.0.0 app:app
EOF
$ cat <<EOF > app.py
from fastapi import FastAPI
app = FastAPI()@app.get("/")
def root():
return {"message": "Hello World"}
EOF

Let’s build and run that for arm to see it running. You can open it in the browser on port 8000 with the IP address you got above.

$ docker buildx build --platform=linux/arm64 -t testimage:arm . 
$ docker run -p 8000:8000 testimage:arm
INFO: Started server process [6]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Note: The container did not quit gracefully sometimes. If that happens to you, go to the VM terminal window you have open, and type docker kill $(docker ps -q) , which kills all running containers.

If you try to build it for x86_64, things don’t go as well:

$ docker buildx build --platform=linux/amd64 -t testimage:x86_64 .
<...>
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to load LLB: runtime execution on platform linux/amd64 not supported

The magic ingredient: emulation

# apt-get install -y qemu-system-x86 qemu-user qemu-user-static

We also need to recreate the builder so buildx understands that it has new platforms to target, and then use that one as our default:

$ docker buildx create --name builder
$ docker buildx inspect builder --bootstrap
<...>
Platforms: linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386
$ docker buildx use builder

That looks pretty good! Let’s try it out. You’ll probably notice that this is a bit slower than the first run, due to emulation. The --load option below makes sure the image doesn’t just get discarded.

$ docker buildx build --platform=linux/amd64 --load -t testimage:x86_64 .

Let’s run the images we built:

$ docker run testimage:x86_64 uname -m
x86_64
$ docker run testimage:arm uname -m
aarch64

And we’re done! This only works because the base image is available for multiple architectures, but many popular Docker images are. You’ve now got two images that built for x86_64 and arm respectively, and both run in your virtual machine — albeit one with emulation. If you run the newest image normally and visit the app in the browser, it works just fine.

End notes

There are still going to be some quirks in this setup. qemu does not work perfectly, and most of its use in Docker seems to be from people on x86_64 architectures building for arm, so any bugs the other way are likely to just be popping up now that M1s are being distributed. But if you’re building images that use interpreted code, with few or no compiled dependencies, this should work just fine! Adrian Mouat has a blog post on the Docker Blog that explains further, and has some helpful alternatives to explore, including cross-compilation.

PS: A much easier solution to this is to run an x86-based virtual machine on the free tier of a cloud service of your choice. To do that, install Docker, configure your daemon to receive connections from the outside, and set DOCKER_HOST in your environment appropriately. No other options are required, as the machine you’re building on doesn’t know any better.

The Startup

Medium's largest active publication, followed by +771K people. Follow to join our community.

Chris Torris Olsen

Written by

Tech lead and International Relations student. https://twitter.com/ctolsen

The Startup

Medium's largest active publication, followed by +771K people. Follow to join our community.

Chris Torris Olsen

Written by

Tech lead and International Relations student. https://twitter.com/ctolsen

The Startup

Medium's largest active publication, followed by +771K people. Follow to join our community.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store