Reclaim disk space by removing stale and unused Docker data

including stopped containers, orphan images, unused volumes, Docker builder cache, and more

11 min readSep 17, 2021

Recently I faced an issue with Docker Desktop for Mac. I was writing a Dockerfile in an attempt to containerize existing NodeJS apps. It was an endless loop — tweak a Dockerfile a bit, build a new version of an image, run the container, figure out something is wrong, tweak a Dockerfile once again, build an image once again, and so on and so on. Suddenly, at some point, the Docker engine refused to build a new image once again, complaining it has run out of disk space.

write /var/lib/docker/overlay2/r6i5qjvzgo679wk9lyvu0qgzi/diff/yarn.lock: no space left on device

Such an issue is not something you’re likely to face on an everyday basis. So in this post, I’m going to shed some light on the cause and share the recipe for how to remedy this problem.

But first, let’s explore the difference between how the Docker engine works on macOS and on Linux.

Docker client-server architecture

Let’s quickly recall the high-level client-server Docker architecture.

Docker Engine client-server architecture

On a Linux, both Docker client and daemon are running on the same machine, and talk to each other via REST API called Docker engine API through the Unix domain socket located at /var/run/docker.sock

We can even connect to this socket using the netcat utility and manually feed it the raw HTTP request text. Append two trailing newlines to send the request (as HTTP protocol prescribes). For example, let's query the Docker engine for all images (similar to what docker imagesCLI command does for you under the hood). Here is the API documentation for the “List images” endpoint.

$ nc -U /var/run/docker.sock
GET /images/json HTTP/1.1
Host: docker.comHTTP/1.1 200 OK
Api-Version: 1.40
Content-Type: application/json
Date: Thu, 16 Sep 2021 19:02:21 GMT
Docker-Experimental: false
Ostype: linux
Server: Docker/19.03.13 (linux)
Transfer-Encoding: chunked...the remaining response body

Here is the screenshot of the full JSON response from the Docker engine.

“/images/json” response from the Docker Engine API

On macOS, Docker daemon runs inside a Virtual Machine

On macOS, things are a bit complicated, since docker daemon and underlying container runtime require a Linux-based environment. Thus, the Docker engine does not run directly on your host machine but runs inside a dedicated minimal Linux OS virtual machine specially built for running containers. For example, with “MacOS 10.15” and “Docker Desktop For Mac 2.4.0.0” being in use, the virtual machine has the Linux docker-desktop 4.19.76-linuxkit distro running. LinuxKit is a lightweight, container-optimized Linux distribution.

Docker client (Docker CLI) on the other hand still runs on your host machine as a regular binary and talks to the Docker daemon running inside a Linux virtual machine through the Unix domain socket in the same way, as I explained before.

# Docker client is a part of "Docker Desktop for Mac" app
$ readlink `which docker`
/Applications/Docker.app/Contents/Resources/bin/docker

Out of space disk error and “Docker.raw” file

Getting back to the “out of disk space” error.

write /var/lib/docker/overlay2/r6i5qjvzgo679wk9lyvu0qgzi/diff/yarn.lock: no space left on device

/var/lib/docker/... directory is where the Docker daemon persists all its data: images, containers, volumes. On macOS, it exists inside an aforementioned Linux virtual machine, so you cannot directly explore its contents from the host machine.

All you can find on the host machine is the single huge Docker.raw file which is reported to be 60GB (allocated size of the file, tells the maximum potential disk size which can be used), but actually taking 22GB in terms of how many bytes are actually consumed by the file on the file system right now.

$ cd ~/Library/Containers/com.docker.docker/Data/vms/0/data$ ls -lh Docker.raw
-rw-r--r-- 1 oleksii.samoshkin 1846670294 60G 2021-09-16 21:48 Docker.raw$ ls -skh Docker.raw
22G Docker.raw

Similar information can be found on the Dashboard of the “Docker Desktop for Mac” app.

Sooner or later you will run out of space allocated for the Docker Engine to store all its data (is 60GB in my case). It could happen even sooner than you expect, for example when creating a new Dockerfile that assumes a lot of image build/rebuild cycles.

Getting into the shell of the Docker VM on Mac

The simplest approach as of 2021 is to use nsenter program running in a privileged container. It will get you into the shell of the Docker VM:

$ docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh

Whoa, what’s going on here.

We are running a container using the Debian image, which already has nsenter binary installed. pid=host means you get access to the process space of the VM running Linux docker-desktop 4.19.76-linuxkit . Basically it removes the process isolation that containers normally promote. Then nsenter says “whatever is PID 1, use that as context, and enter all the namespaces of that, and run a shell there”.

Alternatively, instead of using a full-blown Debian image (124 MB), you can use the justincormack/nsenter1 image, which does not use any base image at all (FROM scratch directive), but builds nsenter binary from sources using C compiler. This is an ultra-lightweight image, that takes only 101Kb.

$ docker run -it --rm --privileged --pid=host justincormack/nsenter1

So it will bring you into the /bin/sh shell inside the Docker VM, where you can find the famous /var/lib/docker directory. You can explore how the docker daemon manages its data, or see the disk usage on a per-directory basis.

docker run -it --rm --privileged --pid=host justincormack/nsenter1/ # cd /var/lib/docker# Explore the space distribution
/var/lib/docker # du -csh *
20.0K   builder
5.5M    buildkit
404.0K  containerd
396.0K  containers
89.1M   image
88.0K   network
16.4G   overlay2
20.0K   plugins
4.0K    runtimes
4.0K    swarm
4.0K    tmp
4.0K    trust
1.4G    volumes
17.9G   total/var/lib/docker # du -csh volumes/*
31.9M   volumes/0a7a66280fbb815de.....f63776ac
12.0K   volumes/564162944bf44c50f.....d0d34ce7
48.2M   volumes/8faf86bb916d717e7.....07954193
16.0K   volumes/app_certs
1.2G    volumes/app_node_modules
8.0K    volumes/c6043fa4e5dc31603w....w803fa08a
48.2M   volumes/cd58df7a53c7df1340....782281915
48.2M   volumes/ffbd465f433b7eebb3....1195c5d4c
48.0K   volumes/metadata.db
1.4G    total

Or you can drill down inside one of your volumes and check its contents. For example, I have an “app_node_modules” volume with many NPM packages installed.

/var/lib/docker # ls -1 volumes/app_node_modules/_data | head -n 10
@babel
@bcoe
@cnakazawa
@danmarshall
@deck.gl
@grpc
@hapi
@istanbuljs
@jest
@loaders.gl

Still, the approach of jumping inside the Docker VM is limited, because it uses a read-only file system, and it’s not intended to be tinkered with. For example, it’s not possible to install vimto edit some files inside the volume.

/var/lib/docker # which apk
/sbin/apk/var/lib/docker # apk add --no-cache vim
ERROR: Unable to lock database: Read-only file system
ERROR: Failed to open apk database: Read-only file system

Anyway, at least it demystifies how the Docker Engine works on the macOS, and now you know where the /var/lib/docker directory comes from and what is its purpose.

How to clean Docker data and reclaim space

The Docker objects that stay around may be of various types: containers, images, volumes, networks.

First, you might be interested in the high-level Docker disk usage information.

$ docker system df
TYPE                SIZE                RECLAIMABLE
Images              10.17GB             7.155GB (70%)
Containers          354B                354B (100%)
Local Volumes       1.105GB             151.1MB (13%)
Build Cache         3.409GB             3.409GB

Next, remove all stopped containers.

$ docker ps --filter “status=exited” -q | xargs -r docker rm --force# or through a simpler built-in command
$ docker container prune --force

Or you might want to remove all the containers, no matter whether they’re running or not. If I were cleaning up stuff on my local machine, I’d rather just drop all of them.

$ docker ps -a -q | xargs -r docker rm — force

Remove dangling images. Docker images consist of multiple layers. Dangling images are layers that have no relationship to any tagged images. They are no longer used and just consume disk space.

$ docker images -f “dangling=true” -q | xargs -r docker rmi -f# or through a simpler built-in command
$ docker image prune

Also, you might want to remove all images, that have no running containers associated with them. Beware that if all containers were removed at the previous step, this command will just remove all the images. This might be not something you would like to do.

# NOTE: you cannot query all unused images through "docker images" command
$ docker image prune --all --force

Remove dangling volumes. The “dangling volume” semantic is a bit different from what the “dangling image” means. A dangling volume is one that exists and is no longer connected to any containers — just an unused volume.

You might end up with quite many Docker volumes over time, especially when new anonymous volumes are created at each container spin up. It’s a good practice to check and clean your volumes regularly.

$ docker volume ls -qf dangling=true | xargs -r docker volume rm# or through a simpler built-in command
$ docker volume prune --force

Clean the Docker builder cache. Normally, Docker caches the results of the first build of a Dockerfile, allowing subsequent builds to be fast.

DOCKER_BUILDKIT=1 docker builder prune --all --force

Remove networks, which are not used by at least one container. Networks are usually created and removed by tools like Docker Compose, so probably, you don’t need to remove them manually. But anyway I will put this command here for the sake of completeness.

# will remove all networks not used by at least one container
docker network prune --force

Use the “docker system prune” shortcut command

Instead of removing all those objects individually one by one, Docker provides you with a single “kill-em-all” command — docker system prune . This command takes care of all those object types:

all stopped containers
unused networks
dangling images and/or unused images
dangling volumes
builder cache.

$ docker system prune --force --volumes

Shrink the “Docker.raw” file on macOS

According to the Docker docs, once you removed those objects from inside the Docker VM, it might take a few minutes to reclaim space on the host machine depending on the format of the disk image file.

if the file is named Docker.raw space on the host should be reclaimed within a few seconds.
if the file is named Docker.qcow2 space will be freed by a background process after a few minutes.

However, space is only freed when images are deleted. Space is not freed automatically when files are deleted inside running containers.

You might want to trigger a space reclamation at any point manually by running the official docker/desktop-reclaim-space image.

$ cd ~/Library/Containers/com.docker.docker/Data/vms/0/data/# Given that block-size = 1KB. 
# Space used = 22135MB
$ ls -sk Docker.raw
22666720 Docker.raw# Discard the unused blocks on the file system
$ docker run --privileged --pid=host docker/desktop-reclaim-space# Savings are (22666720 - 22110948)/1024 = ~542MB
$ ls -sk Docker.raw
22110948 Docker.raw

docker/desktop-reclaim-space image uses the aforementioned nsenter1 binary to get access to the shell of Docker VM, and then it discards the unused blocks on the file system using the fstrim program.

fstrim is used on a mounted filesystem to discard (or “trim”) blocks which are not in use by the filesystem. This is useful for solid-state drives (SSDs) and thinly-provisioned storage.

$ docker inspect docker/desktop-reclaim-space...
“Entrypoint”: [
 “/usr/bin/nsenter1”,
 “/sbin/fstrim”,
 “/var/lib/docker”
 ],
...

PITFALL: Apple M1 users will not be able to run docker/desktop-reclaim-space image, since it was built only for “linux/amd64" architecture.

See this issue for how the error looks like, so you’re not get confused.

$ docker run --privileged --pid=host docker/desktop-reclaim-space
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
setns:mnt: Invalid argument

Restart the Docker engine

Finally, if you have some strange issue, or want to make sure you have the pristine Docker state ever, restart the Docker engine. If you’re on Mac, it’s easier to restart through the “Docker Desktop” app manually. However, it’s also possible to do using uselaunchd and launchctl if you need to automate things.

# see if Docker service is running
local docker_service=$(launchctl list | grep "com.docker.docker" | awk '$0 != "-" { print $3 }')# stop it
if [ -n "$docker_service" ]; then
  launchctl stop "$docker_service" || true;
fi# and then start it
# not that we're starting it indirectly through the "com.docker.helper" Launch agent
launchctl start com.docker.helper

Things are much easier with systemd on Linux.

sudo systemctl stop docker.service || true
sudo systemctl start docker.service# or just
sudo systemctl restart docker.service

How much disk space I reclaimed?

Finally, let’s check out the results of the clean-up. Run the Docker disk usage report once again. As you see, ~5GB space is still occupied, because I didn’t remove all the images.

$ docker system dfTYPE                SIZE                RECLAIMABLE
Images              4.924GB             4.924GB (100%)
Containers          0B                  0B
Local Volumes       0B                  0B
Build Cache         0B                  0B

# 9203256KB/2^20 = ~8.77GB is used
$ ls -ks  Docker.raw
9203256 Docker.raw# WARN: the reported file size is still 60GB
# don't be confused
$ ls -lh  Docker.raw
-rw-r--r-- 1 oleksii.samoshkin 1846670294 60G 2021-09-17 00:27 Docker.raw

Roughly, I reclaimed 12.8GB = (21.6GB − 8.8GB). It’s the garbage that accumulated during the period of approx. 1 week working with Docker on my machine

Before that, it ate all the 60GB of space available after a month of usage or so. Now I’m running the clean-up script twice per month or so, and I’m fine.

Automate “clean up” activities through a Bash script

I’ve prepared a single script, that contains all the commands and tricks I described in this blog post. Check it out at https://github.com/samoshkin/docker-reclaim-disk-space

The usage is simple. You event don’t need to clone the repo. Just use wget or curl

$ bash -c "$(curl -fsSL https://raw.githubusercontent.com/samoshkin/docker-reclaim-disk-space/master/script.sh)"

Or with wget:

$ bash -c "$(wget -qO - https://raw.githubusercontent.com/samoshkin/docker-reclaim-disk-space/master/script.sh)"

Basically, the script does the following:

prints the Docker disk usage information
interactively prompts you for confirmation
removes stopped containers
removes orphan (dangling) images layers
removes unused volumes
removes Docker build cache
shrinks the Docker.raw file, if you’re on the macOS
restarts the Docker engine (through launchctl on macOS or systemctl on Linux). Waits until the Docker is up and running after the restart.
prints the Docker disk usage once again

If you want to suppress interactive prompts, pass -y flag. If you don’t want to restart the Docker engine, pass the --no-restart flag.

Here is the short screencast of this script in action.

Resources

Disk utilization in Docker for Mac | Docker Documentation https://docs.docker.com/desktop/mac/space/

Docker prune explained — usage and examples https://takacsmark.com/docker-prune/

justincormack/nsenter1: simple nsenter to namespaces of pid 1 https://github.com/justincormack/nsenter1

Where is /var/lib/docker on Mac/OS X — https://stackoverflow.com/a/65645462

Getting a Shell in the Docker Desktop Mac VM https://gist.github.com/BretFisher/5e1a0c7bcca4c735e716abf62afad389

Docker overview | Docker Documentation https://docs.docker.com/get-started/overview/#docker-architecture