Docker Kata 004

>>> Disk Space Party!

Docker for Mac / Windows is great, but there’s a good chance that if you’re a power user, you’re probably going to run out of disk space soon.

Why?

As you may have experienced, Docker for Mac / Windows is a fantastic, well integrated tool. Running Alpine Linux inside of a tech stack powered by the latest Unikernel technology, you’ve probably experienced a bit of joy at the native integration between filesystem and networking. That said, there’s a few issues that the Docker Inc. team is hard at work to fix.

Today, we’ll explore one of the more popular Github issues: the space used by the ‘~/Library/Containers/com.docker.docker/‘ directory. As you can see from the file path, we’ll be considering Docker for Mac (DfM) from here onwards. I’d recommend reading through a few Docker forum posts if you’d like a sense of the history of the problem. The easiest way to get up to speed would be on the latest Github issue: #371, and here previously.

There’s a great comment by yankcrime in issue 371 that we’ll explore more here: shrinking down a virtual disk image (VDI).

Restore to stock. This WILL delete all of your images/containers!

Spoiler Alert: if you want to do this the “easy” way, you can simply reset your Docker for Mac installation to stock and your qcow file will regenerate. This <<< WILL delete all of your images and containers however, so proceed with caution. Repeat WARNING! this will delete all of your images and containers.

Don’t want to take the easy route out? Well, read on below.

History

If you do some reading about Virtualbox and the way that it handles VMs, you’ll come to learn about sparse formats. If you’re a DfM user such as myself, you’re mostly likely familiar with the creation of sparse vmdk disks, which allow you to create virtual hard disk drives (HDDs) that grow as your VM’s footprint expands. “Sparse” refers to a settings where you have files with large sections with no valid data, which then doesn’t take up any space on the host computer.

When creating a fixed-size image, an image file will be created on the host which will roughly have the same size as the virtual disk. Notably, the creation of a fixed-size image can be ponderous, especially when creating large images.

A peek at storage managers past.

For more flexible storage management, we can use dynamically allocated images. This will initially be very small and not occupy any space for unused virtual disk sectors, but will grow every time a disk sector is written to for the first time, until the drive reaches the maximum capacity chosen when the drive was created.

While this format takes less space initially, it does consume some additional compute resources. Generally with DfM, growth rate slows over time and the average penalty for write operations will be negligible.

If you’ve ever used the commands below, you’re probably familiar with experimenting between Standard (dynamic) and Fixed disks. You also needed to “zero out the free space” when performing these conversions, which allows the maximum image compaction for your Virtual Disk Image (VDI) or VMDK. This is generally done by using the dd command to write zeroes to a file, and then to delete that file. We’ll use that technique when fixing the Docker qcow issue.

$ VBoxManage clonehd [old-VDI] [new-VDI] — variant Standard
$ VBoxManage clonehd [old-VDI] [new-VDI] — variant Fixed

Another example of sparse file creation is seen in this command for OSX. This command will create a 10GB sparse disk image:

$ hdiutil create /path/to/image.dmg -type SPARSE -fs “Case-sensitive Journaled HFS+” -size 10g -volname “Name of Volume”

But, that’s enough of a history lesson for now. Let’s save some HDD space!

The Solution

Lets check the current size of our qcow2 file. You can copy this command directly, your path should be the same.

$ du -sh ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2

This will return the size of your qcow file.

48G --snipped--/com.docker.driver.amd64-linux/Docker.qcow2

Ouch! Let’s get that 48GB down to something manageable, in order to make space for our GIF collection.

Pre-requisites

In order to get started, you’re going to need to install qemu via Homebrew in order to compress our disk image. We’ll be qemu-ing the VDI in order to re-compress the qcow2 disk image.

Make sure that you have Brew installed. Here are instructions. Once you have brew, install qemu via Brew. Your installation should look similar to the following:

$ brew install qemu
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> Updated Formulae
hopenpgp-tools
==> Installing dependencies for qemu: libpng, libtasn1, gmp, gnutls, gettext, pcre, glib, pixman
…
==> Installing qemu
==> Downloading https://homebrew.bintray.com/bottles/qemu-2.7.0.yosemite.bottle.tar.gz
######################################################################## 100.0%
==> Pouring qemu-2.7.0.yosemite.bottle.tar.gz
/usr/local/Cellar/qemu/2.7.0: 126 files, 140.5M
$

Connect to the Alpine Virtual Machine

First thing we’ll need to do is take advantage of the DfM configuration with Alpine. Specifically, we’ll need to use the tty available in the configuration, paired with screen, to log into the box. Connecting to the tty is not supported by tmux . Believe me, I checked!

Run the following command (you’ll need to hit enter again due the mystical nature of terminal to see the login prompt). This command takes advantage of unix sockets to connect to the socket of your Docker Alpine image.

$ screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

Here is the command in context.

$ screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
…
Welcome to Moby
Kernel 4.4.20-moby on an x86_64 (/dev/ttyS0)
                       ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           \______ o           __/
             \    \         __/
              \____\_______/
moby login: root

Login with ‘root’ (no password) and you should be in!

Let’s also take a peek at the filesystem before we fix it up.

moby:~# df -h
Filesystem Size Used Available Use% Mounted on
tmpfs 999.6M 164.1M 835.4M 16% /
tmpfs 199.9M 128.0K 199.8M 0% /run
cgroup_root 10.0M 0 10.0M 0% /sys/fs/cgroup
dev 10.0M 0 10.0M 0% /dev
shm 999.6M 0 999.6M 0% /dev/shm
/dev/vda1 47.0G 15.4G 29.2G 34% /var
/dev/vda1 47.0G 15.4G 29.2G 34% /var
osxfs 464.8G 397.7G 66.8G 86% /Users
osxfs 464.8G 397.7G 66.8G 86% /Volumes
osxfs 464.8G 397.7G 66.8G 86% /tmp
osxfs 464.8G 397.7G 66.8G 86% /private
osxfs 464.8G 397.7G 66.8G 86% /host_docker_app
/dev/vda1 47.0G 15.4G 29.2G 34% /run/log
osxfs 464.8G 397.7G 66.8G 86% /var/log

Pretty big — let’s fill the disk up with zeroes so we can shrink it back down using the properties of sparse disks.

We should stop the Docker service to avoid any transient errors in the meantime.

moby:~# service docker stop
* Stopping docker
moby:~#

Now, we can run our dd command to fill up the disk. This make take a few minutes depending on how new your laptop is.

moby:~# dd if=/dev/zero of=/var/tempfile

This command will take a little while to error out, probably a few minutes, so we’ll wait on that to complete before moving onto the next step.

moby:~# dd if=/dev/zero of=/var/tempfile
…
dd: writing ‘/var/tempfile’: No space left on device
67248137+0 records in
67248136+0 records out

Once that process errors out, you can remove the temporary file.

moby:~# rm /var/tempfile

Once you’ve done that, logout of the Alpine VM (with screen, it’s ctrl-a, k) and quit the Docker for Mac client via the menu bar icon in OSX. Don’t worry about restarting the Docker client on the Alpine VM, that will come back once you start up the Docker client again.

QEMU to the Rescue

Now, let’s use the qemu utility to re-compress our VDI. You should be able to cut and paste these files as well, since the path is common to DfM installations.

Step 1:

$ mv ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2 \ 
~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2.original
$

Step 2:

$ du -sh ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2.original
 48G /Users/jesse/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2.original
$ qemu-img convert -O qcow2 \
~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2.original \
~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2

Step 3:

$ rm ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2.original
$ du -sh ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
885M

Once you complete this series of steps, you’ll see that your qcow2 file back down to normal, freeing up all the extra space on your Macbook.

Be sure to start up Docker for Mac again, and then you should be all set. Be sure to weigh in on the Github issue if you want to have your say on how and when the solution is developed.

In the meantime you might have to run through this process again until the underlying bug is finished, but I’ll publish a small gist with a command line utility that should do all this for you (if I find some time!).

Come get some love!

Thanks for reading, and as ever — click the heart below if this helped!