QuickTip : Why is my disk usage steadily growing when using docker?
In the past few years Docker and K8S have turned into the de-facto industry standard in containerization and easily scaling any infrastructure ( large or small ).
These two systems have had such an incredible adoption that many of us have neglected the flaws that might be apparent in them. This quicktip will teach you about one such flaw and an easy solution to something that most people will not even realize is right within reach.
Why is my disk usage steadily growing over time?
A couple days ago I was horrified when I logged into one of our staging servers and saw the following graph:
When looking at an (almost) exact replica of this setup on one of our production servers, the graph looked like this:
Both servers were workers in a larger kubernetes cluster, and after further investigation it seemed that none of our production servers were affected, but all of our staging workers exhibited the same strange symptoms of steady disk usage growth over a longer period of time. So I asked the question “Why is my disk usage steadily growing over time?”.
As my mind was racing to unearth the truth behind this dilemma I started listing the differences between our production and staging setup and dove head first into Docker and K8S documentation. After a short but tedious search I found the reason ( and as it turned out an easy solution, to an otherwise interesting challenge ).
Docker builds have a slight pitfall
To show you what I mean with the title of this section, let me show you a simple example. Let’s say that we have a simple docker image extending from Alpine Linux that has bash installed :
After building this image, its size will be approx 35.8MB:
Further investigation with the docker history command shows you the size of each layer in this image:
As you can see, Alpine takes up about 5.5MB, the bash installation takes up an approximate 5.4MB, and an additional 25MB for vim. Let’s say that after a while we realize that all of this can actually be done in a oneliner and make the adjustments to the Dockerfile:
After building the image, it now takes up an approximate 34.4MB (thanks to — no-cache):
However, Docker stores untagged builds separately and does not remove old latest builds by design. And does not remove the cached layers, only the intermediate containers.
Running a quick docker images proves this:
So after 3 builds instead of the original 34.4MB, we are actually using up an approximate 106MB of disk space. The older image is still lying around. So in the end all of our latest builds start piling up these images and they’re chipping away disk space without being used.
We had found the culprit since our staging cluster always uses latest builds. The whole reason why our production cluster was not seeing this particular issue is because it uses tagged images and the amount of deployments is a lot lower than the “several a day” deploys on our staging cluster.
Enter the Docker System namespace
Since Docker 1.13 a set of commands belonging to the docker system namespace have been added. Below you will find a short description of two commands in the docker system namespace that will help you battle this particular issue:
docker system df
Provides you with the amount of disk space docker is using on your system, this also gives you a nice overview in the reclaimable section as to the amount of disk space that could potentially be reclaimed when running a prune:
docker system prune
Allows you to reclaim unused disk space by removing dangling/unused images.
In most cases this removal is safe to do, however do note that if you are running docker <= 17.06.0 this will also remove volumes that are unused.
docker system prune handles cleanup of unused networks, containers and images.
In my particular case on my local machine this managed to reclaim a total of 4.224GB (volumes were not deleted):
And on our staging server it managed to take our disk usage down to a reasonable percentage again:
When using latest builds docker keeps previously built images on disk. Use the docker system df and docker system prune commands to take care of cleaning up these pesky unused images/networks/containers.