QuickTip : Why is my disk usage steadily growing when using docker?

Valkyrie Studios
Apr 3, 2019 · 5 min read

In the past few years Docker and K8S have turned into the de-facto industry standard in containerization and easily scaling any infrastructure ( large or small ).

These two systems have had such an incredible adoption that many of us have neglected the flaws that might be apparent in them. This quicktip will teach you about one such flaw and an easy solution to something that most people will not even realize is right within reach.

Why is my disk usage steadily growing over time?

A couple days ago I was horrified when I logged into one of our staging servers and saw the following graph:

Image for post
Image for post

When looking at an (almost) exact replica of this setup on one of our production servers, the graph looked like this:

Image for post
Image for post

Both servers were workers in a larger kubernetes cluster, and after further investigation it seemed that none of our production servers were affected, but all of our staging workers exhibited the same strange symptoms of steady disk usage growth over a longer period of time. So I asked the question “Why is my disk usage steadily growing over time?”.

As my mind was racing to unearth the truth behind this dilemma I started listing the differences between our production and staging setup and dove head first into Docker and K8S documentation. After a short but tedious search I found the reason ( and as it turned out an easy solution, to an otherwise interesting challenge ).

Docker builds have a slight pitfall

To show you what I mean with the title of this section, let me show you a simple example. Let’s say that we have a simple docker image extending from Alpine Linux that has bash installed :

Image for post
Image for post

After building this image, its size will be approx 35.8MB:

Image for post
Image for post

Further investigation with the docker history command shows you the size of each layer in this image:

Image for post
Image for post

As you can see, Alpine takes up about 5.5MB, the bash installation takes up an approximate 5.4MB, and an additional 25MB for vim. Let’s say that after a while we realize that all of this can actually be done in a oneliner and make the adjustments to the Dockerfile:

Image for post
Image for post

After building the image, it now takes up an approximate 34.4MB (thanks to — no-cache):

Image for post
Image for post

However, Docker stores untagged builds separately and does not remove old latest builds by design. And does not remove the cached layers, only the intermediate containers.

Running a quick docker images proves this:

Image for post
Image for post

So after 3 builds instead of the original 34.4MB, we are actually using up an approximate 106MB of disk space. The older image is still lying around. So in the end all of our latest builds start piling up these images and they’re chipping away disk space without being used.

We had found the culprit since our staging cluster always uses latest builds. The whole reason why our production cluster was not seeing this particular issue is because it uses tagged images and the amount of deployments is a lot lower than the “several a day” deploys on our staging cluster.

Image for post
Image for post

Enter the Docker System namespace

Since Docker 1.13 a set of commands belonging to the docker system namespace have been added. Below you will find a short description of two commands in the docker system namespace that will help you battle this particular issue:

docker system df

Provides you with the amount of disk space docker is using on your system, this also gives you a nice overview in the reclaimable section as to the amount of disk space that could potentially be reclaimed when running a prune:

Image for post
Image for post

docker system prune

Allows you to reclaim unused disk space by removing dangling/unused images.

In most cases this removal is safe to do, however do note that if you are running docker <= 17.06.0 this will also remove volumes that are unused.

docker system prune handles cleanup of unused networks, containers and images.

Image for post
Image for post

In my particular case on my local machine this managed to reclaim a total of 4.224GB (volumes were not deleted):

Image for post
Image for post

And on our staging server it managed to take our disk usage down to a reasonable percentage again:

Image for post
Image for post

tldr;

When using latest builds docker keeps previously built images on disk. Use the docker system df and docker system prune commands to take care of cleaning up these pesky unused images/networks/containers.

Resources

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store