Docker Repository Cleanup for Sonatype Nexus
Sonatype Nexus Repository Manager is a great tool for storing/proxying artifacts. It supports many types of artifacts (Maven, Docker, APT, etc.) and multiple types of repositories: hosted, proxy, and group.
Problem
Recently, I experienced some storage problems with my hosted Docker repository. The blob store containing the hosted Docker repository, which contains my Docker images, got really big, and I couldn’t reduce its size.
Nexus provides some tooling for maintaining your repositories. Intuitively, cleanup policies seemed to be the solution to the problem, but they weren’t.
Most of my Docker images contain only the latest, dev, stable, and version tags, which I don’t want to delete in any condition, so I couldn’t set any cleanup policy.
In fact, the cleanup policy is designed to remove ephemeral or aging (and old) artifacts like snapshot JARs. You shouldn’t delete any of your versioned artifacts unless you have no other choice.
My problem was not the old artifacts. Although I didn’t have many images stored inside the repository, the disk usage was increasing and wouldn’t decrease.
Solution
First, even if I set up some cleanup policies or manually deleted all of my Docker images or any other artifacts, the disk usage wouldn’t change because Nexus would soft delete them.
Nexus has the Tasks feature, which I wasn’t aware of. This feature runs scheduled maintenance tasks, such as cleanup tasks, controlled by a Cleanup Service task.
Nexus docs say that all deletion operations are soft deletes; in order to hard delete them, we need to set a Compact blob store task for each blob store.
However, this wasn’t the solution again, but I was close.
When I read further down the docs, I came to a section called Docker Cleanup Strategies.
Docker images are made of layers, which are mapped to blobs in the Nexus blob store. When my CI pipeline creates images, some of the layers change with every new image. When pushed to the Docker registry, they are saved as new blobs.
My problem was that the old, orphaned blobs wouldn’t be deleted from the blob store, and I didn’t have any tasks to clean them. Even if there were only one tag in my repository, every time a new image was pushed, orphaned blobs would stay in the repository.
This was the cause of my issue with the storage.
To prevent this situation, Nexus has a “Delete unused manifests and images” task, which can be scheduled to remove these orphaned/unused layers. Of course, this removal action is a soft delete; in order to persist, you need to run the “Compact blob store” task afterward.
There is also the “Delete incomplete uploads” task, which soft deletes incomplete uploads if they exist in your repository.
Conclusion
After all tasks were completed, blob disk usage was low, and It was a very happy moment.
So, if you are using Nexus Repository Manager, make sure that you set up maintenance tasks correctly according to your repository type.
Except for the “Cleanup Service” task, which doesn’t hard delete anything, the other three tasks are essential but not set up by default. I think Nexus Web UI should include some kind of setup wizard, a message about the necessity of these tasks, or a shortcut to add them when creating a repository or blob store.