Speeding Up Pulling Container Images on a Variety of Tools with eStargz
Over the past year, eStargz-based lazy pulling of containers has been available on a variety of runtimes and builders in the community. I introduce some of them with examples in this post.
TL;DR
Lazy pulling of eStargz images is available on container-related tools including Kubernetes, containerd, nerdctl, CRI-O, Podman, BuildKit, Kaniko, etc. Note that some of them are highly experimental. We’re continuously working on maturing the feature.
In 2021, lazy pulling will be a more and more common image distribution technique. Aiming at the interoperability of lazy pulling, we proposed eStargz to OCI Image Specification. Please feel free to join the discussion!
What’s lazy pulling and eStargz?
Throughout the container lifecycle, pulling image is known as one of the time-consuming steps. According to Harter, et al.,
pulling packages accounts for 76% of container start time, but only 6.4% of that data is read
This has been affecting the variety of use-cases that require pull, including large batch jobs and building images, etc. One of the root causes is the current OCI Image Specification because it doesn’t allow container runtimes to run containers before the entire image contents being available on the node.
In the previous post, I introduced OCI-alternative but 100% OCI compatible image format eStargz as a solution to this problem.
This is an image format for enabling an image distribution technique called lazy pulling. This allows container runtimes to startup containers without waiting for the entire image contents. Instead, the necessary chunks of contents (e.g. individual files) are fetched on-demand. This shortens the container startup latency from tens of seconds into a few seconds at the best.
You can create an eStarge image using eStargz-aware image builders (e.g. Kaniko) or image converters (e.g. ctr-remote and nerdctl). You can push it to the standard container registry and lazily pull it using eStargz-aware runtimes. Importantly, even eStargz-agnostic runtimes can pull and run eStargz in just the same way as OCI/Docker images, because of eStargz’s compatibility with OCI.
eStargz is developed in Stargz Snapshotter project, a non-core subproject of containerd. This is based on stargz format proposed by Google CRFS project but comes with practical features including content verification and performance optimization.
Lazy pulling of eStargz image on tools in the community
In this section, I introduce some of the tools supporting eStargz-based lazy pulling.
Kubernetes
You can perform lazy pulling of eStargz on Kubernetes without patches.
Containerd or CRI-O is required as CRI runtime. They also need to be configured to enable lazy pulling. We’ll discuss the detailed configuration later, in each of the sections of containerd and CRI-O.
For example, CERN has speeded up their analysis pipeline 13x by eStargz-based lazy pulling on Kubernetes.
Managed Kubernetes services don’t officially support lazy pulling of eStargz as of now. So you need to configure the nodes manually. There is an example of customizing the nodes on GKE by Ricardo Rocha, CERN.
On KinD, you can try eStargz using our customized node image. The following example performs lazy pulling of an eStargz image (ghcr.io/stargz-containers/node:13.13.0-esgz). We are working on upstreaming this configuration to KinD.
$ docker build -t estargz-kind-node https://github.com/containerd/stargz-snapshotter.git
$ kind create cluster --name estargz-demo --image estargz-kind-node
$ cat <<EOF | kubectl --context kind-estargz-demo apply -f -
apiVersion: v1
kind: Pod
metadata:
name: nodejs
spec:
containers:
- name: nodejs-estargz
image: ghcr.io/stargz-containers/node:13.13.0-esgz
command: ["node"]
args:
- -e
- var http = require('http');
http.createServer(function(req, res) {
res.writeHead(200);
res.end('Hello World!\n');
}).listen(80);
ports:
- containerPort: 80
EOFPlease note that additional kubeconfig is required on the node for performing lazy pulling with private registries as of now.
containerd and nerdctl
Containerd supports lazy pulling since version 1.4. Stargz Snapshotter is the plugin that enables containerd to handle eStargz.
On Kubernetes, the following configuration (/etc/containerd/config.toml) enables this feature.
Stargz Snapshotter plugin (containerd-stargz-grpc binary) also needs to run as a separate process. There is an example systemd unit file in our repo.
If you want to try lazy pulling on desktop, you can use nerdctl — a Docker-compatible CLI of containerd. Its --snapshotter=stargz option enables lazy pulling with Stargz Snapshotter. The following example performs lazy pulling of an eStargz image (ghcr.io/stargz-containers/fedora:30-esgz). For the detailed configuration, please refer to the document.
$ nerdctl --snapshotter=stargz run -it --rm ghcr.io/stargz-containers/fedora:30-esgzNote that Docker doesn’t support lazy pulling as of now (it’s on proposal).
CRI-O and Podman
CRI-O experimentally supports lazy pulling. The plugin that enables this is called Additional Layer Store. We provide an implementation of this plugin Stargz Store for enabling CRI-O to handle eStargz.
On Kubernetes, the following configuration (/etc/containers/storage.conf) enables lazy pulling.
Stargz Store (stargz-store binary) needs to run as a separate process as well. There is an example systemd unit file in our repo.
Podman can perform lazy pulling using Stargz Store as well, with the same configuration as CRI-O. The following command lazily pulls an eStargz image (ghcr.io/stargz-containers/fedora:30-esgz).
$ podman run -it --rm ghcr.io/stargz-containers/fedora:30-esgzThis is an experimental feature so please note that there are limitations. As of now, chunk verification of eStargz is not possible on Podman and CRI-O and Podman doesn’t support exporting (podman push and podman save) lazily pulled eStargz images. We are working on eliminating the limitations.
BuildKit
BuildKit is an image builder developed in moby project and is integrated into Docker since 18.09. BuildKit has been experimentally supporting lazy pulling of eStargz-formatted base images since v0.8.0.
But how lazy pulling affects building images?
In general, every build starts from preparing base images (i.e. FROM instruction of Dockerfile). When you build an image using large base images that aren’t cached on the node, they need to be pulled there and it can take a long time. This lazy pulling feature is trying to solve it.
You can try this feature using the standalone version of BuildKit (buildkitd) with Docker Buildx CLI supporting full features of BuildKit. docker build doesn’t support lazy pulling as of now.
The following command enables BuildKit’s lazy pulling feature.
$ docker buildx create --use --name lazy-builder \
--buildkitd-flags '--oci-worker-snapshotter=stargz'
$ docker buildx inspect --bootstrap lazy-builderIn the following Dockerfile, BuildKit performs lazy pulling of the eStargz base image (ghcr.io/stargz-containers/golang:1.15.3-buster-esgz).
You can build this image using buildx, performing lazy pulling.
$ docker buildx build --load -t hello /tmp/hello
$ docker run --rm hello
Hello, world!For more details, please refer to the document or the previous post.
Creating eStargz images with tools in the community
You can create eStargz using eStargz-aware builders or converters. This section introduces some of them.
ctr-remote
ctr-remote is a CLI of containerd developed in Stargz Snapshotter project. This supports converting an OCI/Docker image into eStargz. For example, the following converts ubuntu:21.04 into eStargz.
$ ctr-remote image pull docker.io/library/ubuntu:21.04
$ ctr-remote image optimize --oci \
docker.io/library/ubuntu:21.04 ghcr.io/ktock/ubuntu:21.04-esgz
$ ctr-remote image push ghcr.io/ktock/ubuntu:21.04-esgzctr-remote also supports optimizing eStargz images. Based on the runtime information (e.g. ENTRYPOINT, ENV, etc.) baked into the source image or provided through CLI options, it predicts which files are likely accessed during runtime and marks them in the result eStargz. They will be prefetched before startup the container to avoid the NW-related overhead of accessing them.
The following example converts ubuntu:21.04, optimizing it for bash and ls commands.
$ ctr-remote image optimize --oci \
--entrypoint='[ "/bin/bash", "-c" ]' --args='[ "ls" ]' \
docker.io/library/ubuntu:21.04 ghcr.io/ktock/ubuntu:21.04-lsnerdctl
nerdctl supports creating eStargz images. The following commands build an image ghcr.io/ktock/foo:1 and convert it into eStargz. Here, --oci flag is recommended for enabling the content verification feature of eStargz.
$ nerdctl build -t ghcr.io/ktock/foo:1 .
$ nerdctl image convert --estargz --oci \
ghcr.io/ktock/foo:1 ghcr.io/ktock/foo:1-esgz
$ nerdctl push ghcr.io/ktock/foo:1-esgzYou can optimize eStargz image by manually specifying the list of files to prefetch, through the CLI option. The following example converts ubuntu:21.04, optimizing it for bash and ls commands.
$ cat <<EOF > /tmp/record.json
{ "path" : "/usr/bin/bash" }
{ "path" : "/usr/bin/ls" }
EOF
$ nerdctl image convert --estargz --oci \
--estargz-record-in=/tmp/record.json \
ubuntu:21.04 ubuntu:21.04-lsKaniko
Kaniko is an image builder runnable in containers and Kubernetes. Since v1.5.0, it experimentally supports building eStargz. GGCR_EXPERIMENT_ESTARGZ=1 environment variable enables this, as shown in the following.
$ docker run --rm -e GGCR_EXPERIMENT_ESTARGZ=1 \
-v /tmp/context:/workspace \
-v ~/.docker/config.json:/kaniko/.docker/config.json:ro \
gcr.io/kaniko-project/executor:v1.6.0 \
--destination "ghcr.io/ktock/sample:esgz"Please note that the base images need to be pre-converted to eStargz as of now because Kaniko doesn’t convert them and that optimizing eStargz image is currently not supported.
eStargz and Lazy pulling in 2021
In this post, I introduced some of the tools that support eStargz. For the complete list and the latest status, please see also our tracker thread.
In 2021, lazy pulling will be a more and more common image distribution technique in the community. For moving this trend further forward with keeping the interoperability, we’ve proposed eStargz to OCI Image Spec. Please feel free to join the discussion!
We are continuously improving the implementation of lazy pulling (i.e. Stargz Snapshotter and Stargz Store). And as mentioned in this post, some integrations are experimental and have limitations. We’ll also work on maturing them.
Any feedbacks are always welcome! If you are new to lazy pulling, you can try the quick start with nerdctl or Kubernetes (KinD).
We’re hiring!
We NTT is looking for engineers who work in Open Source communities like containerd, Docker/Moby and Kubernetes. Visit https://www.rd.ntt/e/sic/recruit/ to see how to join us.
