Docker on macOS at native speed using Ubuntu Virtual Machine (both Intel and Apple Silicon CPU)
Spoiler: This is IMHO the best possible Docker performance setup on mac and it completely blew up my mind! Docker on mac can be fun again!
More and more developers are asking me “How come, your Docker on mac is so fast and what is your setup?” - so I decided to write series of posts, not only about the Docker setup, but including other tools, tips, tricks and hacks I use.
Over the years, I have adopted Docker in my development workflow and basically anything I do, I use Docker for that. I tried many tricks to improve performance and address issues, some examples — docker sync, tmpfs volumes, docker machine, cached volumes,.. but every solution brings its own problems (well, including the one I will introduce today 😅) and none of them was really a silver bullet. Im not going deeper into describing the root cause of the performance issues, it is no secret, that Docker’s performance on mac is huge pain and many articles were already written about mac's Docker performance.
Docker Desktop for Mac
If you ever tried Docker on mac, you have probably used “Docker Desktop for Mac” for it.
To demonstrate the problem, here is open api documentation, that developers in Carvago loads every day. Many, many times!
Benchmarks were made on my late 2016 Macbook pro, in the highest possible configuration + as a bonus comparison, with my new M1 Macbook.
Docker in Ubuntu Virtual Machine — the new silver bullet?
With this approach, I have been testing for last several months, you run virtual machine with some linux os (I prefer Ubuntu) and Docker inside linux.
Challenge is syncing files as you keep source of truth on mac. As far as I know, there is no way that linux (and therefore Docker) is aware of files stored in your mac as linux has its own disk image. Syncing files will be covered in separate post later.
Ok, let's get our hards dirty!
I would love to give credits to author of this post as it was huge inspiration for me, but I faced several problems and spent a lot of time figuring out what is wrong after following the original post step by step and it just did not work for me, so here I go with detailed step by step guide what works for me on both my machines.
I like to keep things clean, so I created
ubuntu-vm directory, which will be home for everything related to the VM.
Download all files needed for Ubuntu 20.04 VM — Choose depending on your M1 or Intel based CPU
curl -o initrd https://cloud-images.ubuntu.com/focal/current/unpacked/focal-server-cloudimg-arm64-initrd-genericcurl -o kernel.gz https://cloud-images.ubuntu.com/focal/current/unpacked/focal-server-cloudimg-arm64-vmlinuz-genericgunzip kernel.gzcurl -o disk.tar.gz https://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-arm64.tar.gz
Intel based (AMD64):
curl -o initrd https://cloud-images.ubuntu.com/focal/current/unpacked/focal-server-cloudimg-amd64-initrd-genericcurl -o kernel https://cloud-images.ubuntu.com/focal/current/unpacked/focal-server-cloudimg-amd64-vmlinuz-genericcurl -o disk.tar.gz https://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-amd64.tar.gz
Unzip files + clean up
tar xvfz disk.tar.gz
rm README disk.tar.gz
mv focal-server-* disk.img
Download and build vftool
git clone email@example.com:evansm7/vftool.git
Run VM for the first time
vftool/build/vftool -k kernel -i initrd -d disk.img -m 4096 -a "console=hvc0"
You should see output like this:
2021-03-24 00:28:24.053 vftool[51014:324940] vftool (v0.3 10/12/2020) starting2021-03-24 00:28:24.053 vftool[51014:324940] +++ kernel at kernel, initrd at initrd, cmdline 'console=hvc0', 1 cpus, 4096MB memory2021-03-24 00:28:24.054 vftool[51014:324940] +++ fd 3 connected to /dev/ttys0002021-03-24 00:28:24.054 vftool[51014:324940] +++ Waiting for connection to: /dev/ttys003
Open new terminal window and run this command to connect to VM for the first time (please note, there will be random number and it might not always be 003 in
Now, it is extremely important to find out, what static IP address shall your VM have. Big thanks to my great colleague, linux expert, Jakub Janata who saved me hours by sharing this trick.
On your mac run
ifconfig bridge100 | grep 'inet ' and you will see output like
inet 192.168.64.1 ... - change last digit from
1 to anything else, (I used
2) and note this IP somewhere, you will use it in next step and later too.
This will set root password to
root, VM static IP address to
192.168.64.2, DNS server to google (without setting DNS, internet connection was not working on my intel mac's VM), generate SSH keys and some other for me very magic stuff I have no idea about:
mount /dev/vda /mnt
echo 'root:root' | chpasswd
ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa
ssh-keygen -f /etc/ssh/ssh_host_dsa_key -N '' -t dsa
ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N '' -t ed25519
cat <<EOF > /etc/netplan/01-dhcp.yaml
addresses: [18.104.22.168, 22.214.171.124]
Kill the VM with
CTRL+C in the terminal window in which you originally ran
Resize disk to 100GB
Note that space will be fully allocated and blocked, even if you use only some of it.
Original disk is very small, let's resize it to 100GB (change it to any other desired value). This operation might take few minutes to finish:
dd if=/dev/zero bs=1m count=100000 >> ./disk.img
If you encounter
dd: invalid number: '1m' change
Now start VM again:
vftool/build/vftool -k kernel -i initrd -d disk.img -m 4096 -a "root=/dev/vda console=hvc0"
Again, log in using screen and adjust disk size in VM itself (use
root login and password):
screen /dev/ttys003resize2fs /dev/vda
Set up ssh key
cat ~/.ssh/id_rsa.pub in your mac's terminal and copy the output.
Now change into VM's terminal and run
mkdir ~/.ssh to create directory,
nano ~/.ssh/authorized_keys, paste the output there and save.
chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys
Your VM is all set up now and you should never have to use the
screen command again to enter it! You can kill it with
Important notes about VM
Because vftool is command line script, without GUI, to keep your VM running, you need to keep it “hanging out” in terminal window, for example, like this:
To enter the VM via ssh, always use different terminal window/tab.
Never force kill VM anymore!
Kiling your VM using
cmd+q (or other ungraceful way) can cause troubles and will soon or later lead into unrecoverable state! Trust me, it happened to me 😅. If you want to stop your VM use
It might be obvious, but better safe than sorry. Previously you accessed your applications on something like
http://localhost:8080. This is no longer true and you will be accessing them now on the new address of the VM
http://ubuntu:8080 if you give your VM a hostname.
Let's get more comfortable now
Because you will be starting your VM often, there are several useful (optional) steps I totally recommend to do!
Give your VM a name
This will give your VM name “ubuntu” so you do not have to remember the IP address:
echo '192.168.64.2 ubuntu' | sudo tee -a /etc/hosts
Global command alias for starting the VM
This will create command
start-ubuntu that will start your VM.
I use ZSH, you might need to replace
You might want to tweak
4096 to some different amount of memory as well.
echo 'alias start-ubuntu="~/ubuntu-vm/vftool/build/vftool -k ~/ubuntu-vm/kernel -i ~/ubuntu-vm/initrd -d ~/ubuntu-vm/disk.img -m 4096 -a \"root=/dev/vda console=hvc0\" -t 0"' >> ~/.zshrc
Note the command contains
-t 0 which permits the console to use stdin/stdout (it is mentioned in vftool docs).
Install Docker on VM
Follow official documentation: https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository
Verify that you can run in VM
docker run hello-world
Do not forget to install
Add non-root user (optional)
janmikes user for Docker (change to your name, obviously 😊). I recommend having same user name as you do on your mac.
usermod -aG sudo janmikes
usermod -aG docker janmikes
Disable need of password when running commands with
sudo as user, run
visudo and add this to the end of file:
janmikes ALL=(ALL) NOPASSWD: ALL
Copy ssh key that we previously added so you can have passwordless ssh to non-root user as well:
sudo cat /root/.ssh/authorized_keys > ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
To verify that you can SSH to VM as regular user and run Docker commands without sudo run this from your mac terminal:
ssh ubuntu 'docker run hello-world'
Install Docker binares on Mac (optional)
Because we will be running Docker fully in VM, Docker on mac is not required at all, but sometimes it is handy to have
docker command there - all we need are binaries.
Follow official Docker guilde at:
You might as well want to install docker-compose via homebrew:
brew install docker-compose
DOCKER_HOST variable (add to bash profile) so all Docker commands run inside of VM even when running on mac (change username):
Oh my ZSH + iTerm2 (optional)
Why? Not only it is sexy, but it allows me to instantly recognise if I am using VM terminal or mac terminal, based on icons (you can of course adapt it to your needs):
Do not forget to backup your disk! Now, when you have Docker, non-root user, ZSH, it is the best time to create a backup!
Happened to me, that after some failure (after
ctrl+c VM kill 😒) I had to start from scratch, which is very unfortunate. Using
tar command on 100GB disk took on my M1 mac like 5 minutes to finish with result of 1.15GB backup file:
tar -zcvf disk-backup.img.tar.gz disk.img
From the beginning, it might be confusing that to run most of your commands you need to ssh into VM and run it there, but sooner or later, you will get used to it.
It is important to understand, that VM is completely separated and isolated environment from your mac, just like you would control remote ubuntu server.
If you followed every step, including optional steps, this is final state:
- VM in
- Docker installed on VM
- VM hostname is
- You can
ssh root@ubuntuwithout password
- You can
ssh ubuntuwithout password
start-ubuntucommand on mac
- Backup at
docker-composebinaries are installed on Mac with
DOCKER_HOSTset to VM
- OhmyZSH on both mac + VM