Parallelize your docker-compose build

Peter Kilczuk
Jun 4, 2019 · 2 min read

docker-compose 1.23.0, released on the 30th of October, 2018, brought in a new nice feature — ability to build your containers in parallel.

From the release notes:

Added [the] — parallel flag to the docker-compose build command, allowing
Compose to build up to 5 images simultaneously.

Parallel pulls by default have been around for a while (since 1.21.0 released in April 2018), so now you might want to take advantage of parallel builds as well, in order to reduce your build times. Your mileage will vary, though, depending on your circumstances.

Is it faster?

TL;DR: Yes.

I am currently on a project consisting of different 15 docker images, most of them built from the base Python image with various layers applied on top of that. Here’s how it compared:

  • 1m45s sequentially vs. 2m10s in parallel (~20% faster) if only the context needs to be rebuilt
  • 19m vs. 6m30s (~300% faster) if the images need to be rebuilt from scratch

Whilst your savings will usually sit somewhere in between, depending on the number of dependencies that need to be retrieved from the internet and layers that need to be rebuilt as a consequence, it’s still a saving of time.

Is it stable?

Yes. We’ve had parallel builds turned on for a couple of months, and haven’t had an issue with them once. We run parallel builds both on our dev laptops as well as a part of a Jenkins build on AWS EC2.

Does it always build five images at a time?

That number mentioned in the release notes seemed a bit weird. Why five and not any other number such as, for instance, the number of cores available? So, I was curious to find out, whether that was really the case. And looking at the , yes it was! It appears that five is (it’s not even a multiple of two), but one of the Github users rightly pointed out, that and can potentially cause the resources available to stay idle when they could be used better during the build.

However, on my 8 core CPU, the benefits of increasing the parallelism to 8 were marginal.

Summarizing, I don’t see reasons not to use parallel builds with docker compose. Even though it won’t make your builds instant, it can still you a few precious minutes every day.

To write this article, I’ve used the following sources:

devconnected — DevOps, Sysadmins & Engineering

Tutorials & Guides for DevOps, sysadmins and software engineers.

Peter Kilczuk

Written by

devconnected — DevOps, Sysadmins & Engineering

Tutorials & Guides for DevOps, sysadmins and software engineers.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade