The Shortcomings of Multi-Stage Dockerfiles
Since its introduction in 2017, the Docker multi-stage build feature has been promoted as a practical way of obtaining streamlined images. However, I have always viewed it as a bodge that channels users towards bad habits. Instead of explaining why I think one should never use this feature ever, I would rather like to layout the shortcomings that must be overcome in order for the feature to become a better practice.
The Selling Point
Multi-stage builds were introduced into Docker as a way of using multiple containers to produce a final image. This facilitated the implementation of what was dubbed the builder pattern in which a separate build image was created before the binary image. The build image would contain all of the necessary build tools while the binary image would only contain the application binary and the necessary runtime platform. For example, if the source code is written in Java, the build image would contain a JDK, but the binary image would only need a JRE.
This is in contrast with a more naive way of building the binary image which would be to start from an image containing all of the build tools and the runtime platform. This naive way produces an image that is much larger and that contains programs and data that are unnecessary for running the application. Not only is this deadweight going to impact the bandwidth needed for downloading the image to production, but it leaves potential attack vectors laying around.
The True Purpose
I would like to argue that the true motivation behind multi-stage builds was to fix a problem with using Docker Hub automated builds.
In keeping with the philosophy of making container technology as accessible as possible, Docker Hub has long facilitated image production through automated builds. All you need is to have the Dockerfile hosted on a GitHub or Bitbucket code repository and to link the image repository to it. Docker Hub will then build the image on every push.
This involves cloning the code repository and then running docker build. This ruled out the builder pattern. A user had to choose between living with unnecessarily large image sizes or using an alternative to Docker Hub automated builds. Docker probably saw a drop in the use of this feature, and understandably wanted to bring users back. Features, like this one, that are accessible to a wide variety of users have a high potential for attracting business.
The Competition
When I first heard about multi-stage builds, I didn’t see much use for it personally. I wasn’t using Docker Hub automated builds and had no plans to in the immediate future. When I experimented with the feature, I started to draw comparisons with competing solutions.
The first class of competitors that I identified was the class of continuous integration tools. With these tools, one also defines a build process as a pipeline of stages. For example, the stages could be compiling and packaging. Often there could also be stages for various types of testing, code quality evaluation or any number of other static or dynamic analyses.
When compared to the wide range of CI tools — and there are a lot — multi-stage builds seem to hold up fairly well. They have the huge advantage of being able to run on a local machine. This makes the feedback loop much shorter, especially when working on the definitions of the stages themselves. Tweaking a CI script can take longer because the pipeline can only be initiated by the CI server which can only happen after changes have been pushed to the central code repository.
However multi-stage builds are not quite as feature-rich. This isn’t such a problem for me because I don’t find many of the extra features useful.
One of the main reasons that I don’t rely on many of the features CI tools provide is that I prefer to delegate to a build automation tool. I would rather the pipeline just have one stage consisting of calling the build automation tool with a specific set of arguments. I find that this is a better assignment of responsibilities. The CI tool is responsible for scheduling builds, whilst the build automation tool is responsible for knowing what to build and in which order.
This method has the added benefit of allowing for the build actions to be run easily on any machine. It also allows for more versatility in running build actions. For example, I might want to skip certain stages or specify a development environment. There are only ways two ways to vary a Docker multi-stage build: using the --target option to stop at a specific stage or using build arguments. With most build automation tools, there are many more useful ways to vary the build. My favourite example is Maven profiles mainly because they shorten the amount of typing on the command line.
If I want some or all of the build actions to be run within a container, this is possible without using a multi-stage Docker build. Some build automation tools facilitate this to various degrees. In my opinion, Haskell Stack has the easiest option for enabling all build actions to be run in a container. Bazel always sandboxes build actions and provides options for using Docker containers as part of that process.
The Challenge
In my opinion, Docker multi-stage builds don’t currently compare favourably to the competition from build automation tools such as Maven or Bazel. I think that it remains useful for the narrow case of quickly getting an image built through Docker Hub automated builds. For the feature to start becoming competitive beyond that, I would recommend expanding the feature to provide more ways of varying the build steps that are run. I would also recommend providing ways of shortening the command that must be typed for certain variations, perhaps in the same vein as Maven profiles.
