Building containers without waiting for pull completion of base images on BuildKit

This post introduces one of the recent optimization features in BuildKit aiming to speed up building images — lazy pulling of base images.

In general, every build starts from the base image preparation (i.e. FROM instruction). An artifact of that build is created by adding changes to that base image through running commands on it (i.e. RUN ) and copying files to/from outside (i.e. COPY ), etc. Though the base image preparation is a common step in building images, it can be time-consuming when that needs to be pulled from the registry. This can occur for example in the following situations,

• The first time of the bulid using that base image.
• Building an image in a temporary instance on the cloud without importing cache.
• Just after base images are removed from the build cache.

Building containers with lazily pulling the base images on BuildKit

Lazy pulling of the base image during the build

BuildKit is a container image builder developed in moby project, which is also integrated to Docker since 18.09. It has features for efficient building of images, including concurrent multi-stage build.

Recently, BuildKit added experimental support for "lazy pulling" of base images, aiming to speed up the build. Lazy pulling in this post means BuildKit runs build stages without waiting for the pull completion of the base images but fetches necessary files/chunks contained in these images on demand. This lazy pulling feature is implemented leveraging stargz/eStargz images (see the later section "Stargz/eStargz image formats" for more details).

The following diagram shows the time to take for three simple builds on a GitHub Actions runner (eastus2) with GitHub Container Registry.

Time to take for the builds (lower is better)

• In gcc-10.2.0, an image printing “hello” is built using gcc:10.2.0 as the base image.
• In golang-1.15.3-buster, an image printing “hello” is built using golang:1.15.3-buster as the base image.
• In python-3.9-buster, an HTTP server image returning “hello” is built using python:3.9.0-buster as the base image.

legacy builds use standard images as their base images. stargz and estargz use stargz- and eStargz-formatted base images respectively, which are lazily pulled. The result shows that stargz/eStargz shorten the time to take for the build from several tens of seconds to a few seconds in the best case.

NOTE: In all cases in the above, the build artifacts are stored in the local cache but not output to elsewhere. When the build artifact is output (e.g. pushed to the registry), the final stage’s base image needs to be fully downloaded for sending it to the destination, which can cost extra time. For avoiding this, consider outputting the artifact to the same registry as its base image for hopefully reusing the image blobs existing in the registry instead of letting BuildKit copy it.

Quick Start

You can try Stargz/eStargz-based lazy pulling feature with the standalone version of BuildKit (buildkitd) through Docker Buildx which is a Docker CLI plugin supporting full features of BuildKit. Note that this lazy pulling feature is currently unsupported by docker build.

Once enabled Buildx on your Docker, create and run a new lazy-pull-enabled builder using the BuildKit release image moby/buildkit:master. Stargz/eStargz-based lazy pulling can be enabled by specifying the runtime flag --oci-worker-snapshotter=stargz to buildkitd.

$ docker buildx create --use --name lazy-builder \
         --driver docker-container \
         --driver-opt image=moby/buildkit:master \
         --buildkitd-flags '--oci-worker-snapshotter=stargz'
$ docker buildx inspect --bootstrap lazy-builder

In this example, we build the following hello-world program written in Go.

Hello-world program written in Go

In the Dockerfile, we use eStargz-formatted golang image (ghcr.io/stargz-containers/golang:1.15.3-buster-esgz) as the base image. This Dockerfile can be built without waiting for the completion of pulling this golang image. Instead, BuildKit lazily fetches necessary files/chunks of that image from the registry on-demand.

Dockerfile lazily pulling the base image

Then this Dockerfile can be built using docker buildx build. The build should take shorter than the non-eStargz base images.

$ docker buildx build --load -t hello /tmp/hello
$ docker run --rm hello
Hello, world!

Stargz/eStargz image formats

This lazy pulling feature is available for stargz and eStargz images. They are OCI/Docker-compatible image formats lazily pullable from standard registries (e.g. Docker Hub, Github Container Registry, etc.). Because of this compatibility, they can be pulled and run by standard runtimes including Docker and containerd.

Some pre-built stargz/eStargz images are available under the ghcr.io/stargz-containersrepository including ghcr.io/stargz-containers/golang:1.15.3-buster-esgz eStargz image used in the above example. You can also create any stargz/eStargz images by converting the original OCI/Docker image into stargz/eStargz using the converter commands (e.g. ctr-remoteand stargify).

Stargz and eStargz image formats

Stargz image format

Stargz is an image format proposed by Google CRFS project, which stands for Seekable tar.gz. This enables BuildKit to seek the image layer and selectively fetch the necessary files/chunks of that image from the registry on-demand.

eStargz image format

eStargz is our extended format of stargz, developed in stargz snapshotter project. This comes with additional features including chunk verification and prefetch for avoiding overheads of on-demand fetching.

For lazily pulling stargz/eStargz images, BuildKit uses Stargz Snapshotter plugin by containerd project. For more details about stargz/eStargz images and stargz snapshotter, please refer to the previous post.

containerd/stargz-snapshotter

Feedbacks are always welcome!

Currently, stargz/eStargz-based lazy pulling is an experimental feature of BuildKit. Helps for more bug fixes and design/performance improvements are needed for stabilizing this feature. We are very welcome for any kinds of feedbacks and patches to BuildKit and Stargz Snapshotter repositories!

We’re hiring!

NTT is looking for engineers who work in Open Source communities like Kubernetes & Docker projects. If you wish to work on such projects please do visit our recruitment page. To know more about NTT contribution towards open source projects please visit our Software Innovation Center page. We have a lot of maintainers and contributors in several open source projects. Our offices are located in the downtown area of Tokyo (Tamachi, Shinagawa) and Musashino.