P2P Container Image Distribution on IPFS With Containerd

nerdctl, Docker-compatible CLI of containerd, added an experimental support of P2P container image distribution without registries, using IPFS.

You can handle images on IPFS through ipfs://CID image reference. For details about prerequisites, please see the later sections or the docs in nerdctl.

> ubuntu=bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze
> nerdctl run -it ipfs://$ubuntu echo hello

The image distributed on IPFS is 100% compatible to OCI Image Spec so existing image distribution techniques like image encryption and lazy pulling can still be used.

What’s nerdctl and containerd?

containerd is an open source container runtime project. It has been used by projects including Docker/moby and Kubernetes services including EKS, AKS and GKE.

nerdctl is a Docker-compatible CLI of containerd, developed as a subproject of containerd. This has a similar UI/UX as Docker and also provides cutting-edge features including lazy pulling (eStargz) and image encryption (ocicrypt). This has been adopted by lima and Rancher Desktop.

nerdctl: Docker-compatible CLI for contaiNERD

IPFS-based P2P image distribution with nerdctl

nerdctl v0.14 added an experimental support of P2P image distribution on IPFS. This provides a new simple way of sharing images among container users without depending on registries.

IPFS is a peer-to-peer and content-addressable data sharing protocol.
A node on an IPFS network can share data to other participants in a P2P manner without hosting a central server.

registry-based image distribution and ipfs-based image distribution

When distributing an image on IPFS, nerdctl adds it to your local IPFS storage. Other nodes look up the image on the IPFS network by its CID (hash of the image) then fetch it from your node or peers caching the chunks.

The image distributed on IPFS is 100% compatible to OCI Image Spec. So existing image distribution techniques including image encryption and lazy pulling can be applied to IPFS-based distribution as described later.

Getting started with IPFS-based P2P image distribution

You can quickly get started with IPFS-based P2P image distribution by launching ipfs daemon which connects your node to IPFS.

ipfs daemon

Use --offline flag if you don’t want the node to connect to the public network. Please see the docs in nerdctl for details about prerequisites.

nerdctl supports the image name prefix ipfs:// to handle images on IPFS.
You can push an arbitrary image to IPFS using this prefix.

> nerdctl push ipfs://ubuntu:20.04
INFO[0000] pushing image "ubuntu:20.04" to IPFS         
INFO[0000] ensuring image contents                      
bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze

The last line prints the IPFS CID of the pushed image. You can use this CID to share this image with other nodes.

The following command runs that image with pulling it from IPFS.

> ubuntu=bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze
> nerdctl run -it ipfs://$ubuntu echo hello
hello

The above image may become unavailable on IPFS (because we don’t pin it) before you try nerdctl run so ensure to push the image to IPFS in advance.

You can build a new image based on images on IPFS. In Dockerfile, instead of ipfs:// prefix, nerdctl supports the image reference formatted as localhost:5050/ipfs/CID to specify an image on IPFS.

ARG BASE=bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze
FROM localhost:5050/ipfs/$BASE
RUN echo hello > /hello

In the future version of nerdctl, ipfs:// prefix should be supported in Dockerfile.

IPFS-enabled image is OCI compatible so it can also be pushed to container registries. This is useful to ship the image to hosts that don’t support IPFS.

> nerdctl tag ipfs://$ubuntu ghcr.io/ktock/ubuntu:20.04-ipfs
> nerdctl push ghcr.io/ktock/ubuntu:20.04-ipfs

Image encryption and lazy pulling

Because of the OCI-compatibility of the image pushed to IPFS, you can use existing image distribution techniques. This section describes image encryption (ocicrypt) and lazy pulling (eStargz).

Note that image encryption and lazy pulling are exclusive i.e. they can’t be enabled together.

If you want to keep your image being secret on IPFS, you can encrypt the image using ocicrypt.

> nerdctl image encrypt --recipient=jwe:mypubkey.pem \
    ubuntu:20.04 ubuntu:20.04-encrypted
> nerdctl push ipfs://ubuntu:20.04-encrypted

To maximize the efficiency of image distribution, you can enable lazy pulling based on eStargz. This allows nerdctl to startup a container without waiting for the entire image contents becoming locally available. This can improve the cold-start performance for large images.

The following command pushes an image with formatting it as eStargz.

nerdctl push --estargz ipfs://ubuntu:20.04

nerdctl starts the container immediately after fetching necessary chunks of the image (e.g. entrypoint binary) without waiting for the entire contents.

nerdctl run --snapshotter=stargz -it ipfs://$ubuntu_esgz echo hello

Please visit our repo for more information about nerdctl and IPFS-based image distribution.

GitHub - containerd/nerdctl: contaiNERD CTL - Docker-compatible CLI for containerd, with support…

We’re hiring!

We NTT is looking for engineers who work in Open Source communities like containerd, Docker/Moby and Kubernetes. Visit https://www.rd.ntt/e/sic/recruit/ to see how to join us.