Docker on macOS at native speed using Ubuntu Virtual Machine (both Intel and Apple Silicon CPU)

Jan Mikeš
Mar 24 · 9 min read

Spoiler: This is IMHO the best possible Docker performance setup on mac and it completely blew up my mind! Docker on mac can be fun again!

More and more developers are asking me “How come, your Docker on mac is so fast and what is your setup?” - so I decided to write series of posts, not only about the Docker setup, but including other tools, tips, tricks and hacks I use.

Over the years, I have adopted Docker in my development workflow and basically anything I do, I use Docker for that. I tried many tricks to improve performance and address issues, some examples — docker sync, tmpfs volumes, docker machine, cached volumes,.. but every solution brings its own problems (well, including the one I will introduce today 😅) and none of them was really a silver bullet. Im not going deeper into describing the root cause of the performance issues, it is no secret, that Docker’s performance on mac is huge pain and many articles were already written about mac's Docker performance.

Docker Desktop for Mac

If you ever tried Docker on mac, you have probably used “Docker Desktop for Mac” for it.

The traditional and most common approach is that Docker runs directly on your mac.

To demonstrate the problem, here is open api documentation, that developers in Carvago loads every day. Many, many times!

Benchmarks were made on my late 2016 Macbook pro, in the highest possible configuration + as a bonus comparison, with my new M1 Macbook.

Really! This is not fake! API doc page takes 160s to load, without warmed up cache.
6.2s with warmed up cache. Better, but still very painful.

Docker in Ubuntu Virtual Machine — the new silver bullet?

With this approach, I have been testing for last several months, you run virtual machine with some linux os (I prefer Ubuntu) and Docker inside linux.

Challenge is syncing files as you keep source of truth on mac. As far as I know, there is no way that linux (and therefore Docker) is aware of files stored in your mac as linux has its own disk image. Syncing files will be covered in separate post later.

3.7s without warmed up cache, much better!
Wow, 300ms with warmed up cache!
85ms with same approach on M1 ❤️

Ok, let's get our hards dirty!

I would love to give credits to author of this post as it was huge inspiration for me, but I faced several problems and spent a lot of time figuring out what is wrong after following the original post step by step and it just did not work for me, so here I go with detailed step by step guide what works for me on both my machines.

I like to keep things clean, so I created ubuntu-vm directory, which will be home for everything related to the VM.

mkdir ~/ubuntu-vm
cd ~/ubuntu-vm

Download all files needed for Ubuntu 20.04 VM — Choose depending on your M1 or Intel based CPU

M1 (ARM64):

curl -o initrd https://cloud-images.ubuntu.com/focal/current/unpacked/focal-server-cloudimg-arm64-initrd-genericcurl -o kernel.gz https://cloud-images.ubuntu.com/focal/current/unpacked/focal-server-cloudimg-arm64-vmlinuz-genericgunzip kernel.gzcurl -o disk.tar.gz https://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-arm64.tar.gz

Intel based (AMD64):

curl -o initrd https://cloud-images.ubuntu.com/focal/current/unpacked/focal-server-cloudimg-amd64-initrd-genericcurl -o kernel https://cloud-images.ubuntu.com/focal/current/unpacked/focal-server-cloudimg-amd64-vmlinuz-genericcurl -o disk.tar.gz https://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-amd64.tar.gz

Unzip files + clean up

tar xvfz disk.tar.gz
rm README disk.tar.gz
mv focal-server-* disk.img

Download and build vftool

git clone git@github.com:evansm7/vftool.git
cd vftool
make
cd ..

Run VM for the first time

vftool/build/vftool -k kernel -i initrd -d disk.img -m 4096 -a "console=hvc0"

You should see output like this:

2021-03-24 00:28:24.053 vftool[51014:324940] vftool (v0.3 10/12/2020) starting2021-03-24 00:28:24.053 vftool[51014:324940] +++ kernel at kernel, initrd at initrd, cmdline 'console=hvc0', 1 cpus, 4096MB memory2021-03-24 00:28:24.054 vftool[51014:324940] +++ fd 3 connected to /dev/ttys0002021-03-24 00:28:24.054 vftool[51014:324940] +++ Waiting for connection to:  /dev/ttys003

Open new terminal window and run this command to connect to VM for the first time (please note, there will be random number and it might not always be 003 in /dev/ttys003:

screen /dev/ttys003

Now, it is extremely important to find out, what static IP address shall your VM have. Big thanks to my great colleague, linux expert, Jakub Janata who saved me hours by sharing this trick.
On your mac run ifconfig bridge100 | grep 'inet ' and you will see output like inet 192.168.64.1 ... - change last digit from 1 to anything else, (I used 2) and note this IP somewhere, you will use it in next step and later too.

This will set root password to root, VM static IP address to 192.168.64.2, DNS server to google (without setting DNS, internet connection was not working on my intel mac's VM), generate SSH keys and some other for me very magic stuff I have no idea about:

mkdir /mnt
mount /dev/vda /mnt
chroot /mnt

touch /etc/cloud/cloud-init.disabled

echo 'root:root' | chpasswd

ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa
ssh-keygen -f /etc/ssh/ssh_host_dsa_key -N '' -t dsa
ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N '' -t ed25519

cat <<EOF > /etc/netplan/01-dhcp.yaml
network:
version: 2
ethernets:
enp0s1:
dhcp4: true
addresses: [192.168.64.2/24]
nameservers:
addresses: [8.8.8.8, 8.8.4.4]
EOF

exit
umount /dev/vda

Kill the VM with CTRL+C in the terminal window in which you originally ran vftool.

Resize disk to 100GB

Note that space will be fully allocated and blocked, even if you use only some of it.

Original disk is very small, let's resize it to 100GB (change it to any other desired value). This operation might take few minutes to finish:

dd if=/dev/zero bs=1m count=100000 >> ./disk.img

If you encounter dd: invalid number: '1m' change 1m to 1M

Now start VM again:

vftool/build/vftool -k kernel -i initrd -d disk.img -m 4096 -a "root=/dev/vda console=hvc0"

Again, log in using screen and adjust disk size in VM itself (use root login and password):

screen /dev/ttys003resize2fs /dev/vda

Set up ssh key

Run cat ~/.ssh/id_rsa.pub in your mac's terminal and copy the output.

Now change into VM's terminal and run mkdir ~/.ssh to create directory, nano ~/.ssh/authorized_keys, paste the output there and save.
Then run chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys

Your VM is all set up now and you should never have to use the screen command again to enter it! You can kill it with ctrl+c again.

Important notes about VM

Because vftool is command line script, without GUI, to keep your VM running, you need to keep it “hanging out” in terminal window, for example, like this:

To enter the VM via ssh, always use different terminal window/tab.

Never force kill VM anymore!

Kiling your VM using ctrl+c or cmd+q (or other ungraceful way) can cause troubles and will soon or later lead into unrecoverable state! Trust me, it happened to me 😅. If you want to stop your VM use halt command!

RIP localhost

It might be obvious, but better safe than sorry. Previously you accessed your applications on something like http://localhost:8080. This is no longer true and you will be accessing them now on the new address of the VM http://192.168.64.2:8080 or http://ubuntu:8080 if you give your VM a hostname.

Let's get more comfortable now

Because you will be starting your VM often, there are several useful (optional) steps I totally recommend to do!

Give your VM a name

This will give your VM name “ubuntu” so you do not have to remember the IP address:

echo '192.168.64.2 ubuntu' | sudo tee -a /etc/hosts

Global command alias for starting the VM

This will create command start-ubuntu that will start your VM.
I use ZSH, you might need to replace .zshrc with .bashrc.
You might want to tweak 4096 to some different amount of memory as well.

echo 'alias start-ubuntu="~/ubuntu-vm/vftool/build/vftool -k ~/ubuntu-vm/kernel -i ~/ubuntu-vm/initrd -d ~/ubuntu-vm/disk.img -m 4096 -a \"root=/dev/vda console=hvc0\" -t 0"' >> ~/.zshrc

Note the command contains -t 0 which permits the console to use stdin/stdout (it is mentioned in vftool docs).

Install Docker on VM

Follow official documentation: https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository

Verify that you can run in VM docker run hello-world

Do not forget to install docker-compose: https://docs.docker.com/compose/install/

Add non-root user (optional)

I created janmikes user for Docker (change to your name, obviously 😊). I recommend having same user name as you do on your mac.

adduser janmikes
usermod -aG sudo janmikes
usermod -aG docker janmikes

Disable need of password when running commands with sudo as user, run visudo and add this to the end of file:

janmikes ALL=(ALL) NOPASSWD: ALL

Copy ssh key that we previously added so you can have passwordless ssh to non-root user as well:

su janmikes
mkdir ~/.ssh
sudo cat /root/.ssh/authorized_keys > ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

To verify that you can SSH to VM as regular user and run Docker commands without sudo run this from your mac terminal:

ssh ubuntu 'docker run hello-world'

Install Docker binares on Mac (optional)

Because we will be running Docker fully in VM, Docker on mac is not required at all, but sometimes it is handy to have docker command there - all we need are binaries.

Follow official Docker guilde at:
https://docs.docker.com/engine/install/binaries/#install-client-binaries-on-macos

You might as well want to install docker-compose via homebrew:
brew install docker-compose

Now change DOCKER_HOST variable (add to bash profile) so all Docker commands run inside of VM even when running on mac (change username):

export DOCKER_HOST=ssh://janmikes@ubuntu

Oh my ZSH + iTerm2 (optional)

Very recommended step by me is install iTerm2 and ZSH + https://ohmyz.sh on both your mac and VM. I use this great guide: https://gist.github.com/kevin-smets/8568070

Why? Not only it is sexy, but it allows me to instantly recognise if I am using VM terminal or mac terminal, based on icons (you can of course adapt it to your needs):

BACKUP!!! (optional)

Do not forget to backup your disk! Now, when you have Docker, non-root user, ZSH, it is the best time to create a backup!

Happened to me, that after some failure (after ctrl+c VM kill 😒) I had to start from scratch, which is very unfortunate. Using tar command on 100GB disk took on my M1 mac like 5 minutes to finish with result of 1.15GB backup file:

tar -zcvf disk-backup.img.tar.gz disk.img

Recap

From the beginning, it might be confusing that to run most of your commands you need to ssh into VM and run it there, but sooner or later, you will get used to it.

It is important to understand, that VM is completely separated and isolated environment from your mac, just like you would control remote ubuntu server.

If you followed every step, including optional steps, this is final state:

  • VM in ~/ubuntu-vm
  • Docker installed on VM
  • VM hostname is ubuntu
  • You can ssh root@ubuntu without password
  • You can ssh ubuntu without password
  • Global start-ubuntu command on mac
  • Backup at ~/ubuntu-vm/disk-backup.img.tar.gz
  • docker and docker-compose binaries are installed on Mac with DOCKER_HOST set to VM
  • OhmyZSH on both mac + VM