Rotating Docker Logs — Keeping your overlay folder small
TL;DR
Learn from my mistakes. If you are using the default json-file driver, configure max-file
and max-size
options to have Docker automatically rotate and remove old logs.
The Problem
4:30 pm. Time to go home, figure something out for dinner, relax and — — THE BUILDS ARE FAILING. Classic.
Can’t push or pull to our self-hosted registry. The container is running, so why would it not be able to read/write to the disk unless…the disk is full?
After some quick digging, turns out the Docker overlay folder is big. Like almost 200 GB big. But a quick glance shows that we only have 4 images and a few small volumes, so why would that overlay folder be so big?
The answer: logs. a LOT of logs.
The Fix
By default, Docker uses the json-file log driver and saves those logs in the overlay folder. Docker doesn’t limit the size of the files or how many log files there can be for a single container. If you continue to run that container, the log file will grow as long as it has drive space to do so. Thankfully, it’s pretty easy to set these limits.
If you are running from the CLI, your option looks like:
docker run --log-opt max-size=10m --log-opt max-file=5 my-app:latest
This sets a maximum of five log files with a max size of 10 Mb each. So at most 50 Mb of logs…