From Virtual Machine Sprawl to Docker Sprawl
In the old days as a system administrator, I ran a lot of physical servers running virtual machines. I was managing hundreds of virtual machines and anytime a developer needed a testing setup, I could build and deploy an environment identical to production for development or testing. Now we thought this was great at the time, although three years later it seems incredible that we went through the process given how much containers technology has progressed.
By “in the old days”, I mean three years ago, as the world had drastically changed due to the arrival of Docker. Docker is not new technology, but it is a container technology that is a better version of what we had with virtual machines. Instead of each machine operating in its own separate virtual machine, now each application can run in its own virtual environment inside a container. Docker uses container technology which has been around for years, but was not easy to use. Three years ago, docker came along and placed a friendly face on what is a difficult to use technology.
The great thing about containers is how lightweight they are, and the not so great thing about containers is how many images may be created for those containers. Each container is stored stored as an image and each of those images takes up space on your hard drive. Docker creates images by building them from a series of layers. As Docker is building the images, intermediary images may be created from the layers used to build the image. By default, Docker does not delete intermediary images automatically, they are left in case they are needed again.
Docker has many advantages over the old ways of deploying services but docker does not care about disk space and that can be a problem.
In your development environments, container images can take up a lot of space and managing those issues becomes a task for the sys admin or DevOp. For example, when you use Jenkins to build your production images, you will start to have high disk usage from all the untagged images created during a new image build that builds up over time. Jenkins server will fill up with old and outdated images and those need to be intelligently purged.
You need to run the same image for testing, staging, and deployment. You do not want use a Dockefile to create a container image for testing, then re-create that image at a different time for deployment with the same Dockerfile. Differences , however subtle, can creep into your container images from building at different times. It is essential to avoid this in a continuous integration environment by using the same image for development, testing, and deployment.
In avoiding image sprawl, we can get overzealous in deleting container images. We can run commands that will delete all nonrunning containers with out regard if they are needed or not This is fine to prevent container build up on your laptop development machine since the deleted image will just be re-built, but it is not something you want to do in your production environment.
The key to cleaning up images is intelligently purging old , unused and dangling images, and not just deleting them en mass. There are a few tools for removing unused images. There are some command line tools for removing unused images on the web that I will not repeat here. I would not recommend those scripts in a production environment. This is like trying to shave with a butcher knife, it may work, but if you mess up it can be very bloody. Beyond those, I would recommend two scripts for keeping your images in check.
docker-clean The first is Docker Clean which is an open source tool that has a lot of options. Docker clean can be found at
docker-gc docker-gc is another tool that was released open source by Spotify as a Docker garbage collection application.
Docker 1.9 Docker 1.9 has new volume management tool that can Removing dangling images and intermediary images that are created while building other containers.
docker rmi $(docker images -f “dangling=true” -q)
Docker containers and images are powerful for DevOps, but they do have their weak points. Intelligently managing your images will pay dividends in your CI and application deployment.
If you are serious about your images, you should consider using Docker registry. Docker Registry is an open source server application that stores and lets you distribute Docker images, just like Docker Hub.
Was this post interesting? If so, please click the ❤ below to let me know , it really does help. Greg Coleman is a freelance consultant based in Annapolis, Maryland. You can usually find me traveling to places with awesome food or awesome people. Often in a cafe coding away on my laptop. If you are looking for a contractor, you can contact me here https://gregorycoleman.github.io
