Open in app

Sign in

Write

Sign in

Caching docker images for local Kind clusters

Charles-Edouard Brétéché
5 min readJan 31, 2022

In previous stories I wrote about creating local Kubernetes clusters with Kind, and how to run various core components in the local cluster (Cilum, MetalLB, ingress-nginx).

Kind cluster with Cilium and no kube-proxy

Kind overview

medium.com

Kind, Cilium, MetalLB, and still no kube-proxy

In a previous story I explained how to run a Kubernetes cluster locally with Kind, Cilium, and without kube-proxy.

medium.com

While this works well, it can take some significant time to start because Kind does not cache docker images and has to download them every time a cluster is started.

In this article i will show you how to add image caching and configure Kind (containerd actually) to take advantage of it.

This will greatly speed up cluster start once images are in the cache.

Running a registry proxy

Docker provides a handy image for running a local registry https://hub.docker.com/_/registry.

This image can be configured to run as a remote registry proxy and has caching capabilities.

It can mirror only one registry though, as we probably want to cache images from multiple registries (quay.io, gcr.io, k8s.gcr.io, docker.io, …) we will have to run multiple instances, one for each image prefix we wish to support.

Enabling registry proxy mode is done by setting REGISTRY_PROXY_REMOTEURL environment variable to the remote registry URL.

For example, running a local registry proxy for dockerhub can be done with:

docker run -d --name proxy-docker-hub --restart=always \
  --net=kind \
  -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \
  registry:2

Note that the registry proxy runs in the kind docker network. If it doesn’t exist yet you can create it with docker network create kind.

Looking at the container logs with docker logs proxy-docker-hub will show that it started a server listening on port 5000:

time="2022-01-31T10:51:09.671881879Z" level=info msg="listening on [::]:5000" go.version=go1.11.2 instance.id=83581904-704d-46cb-9d19-1b0278f7b962 service=registry version=v2.7.1

Configuring Kind cluster registry mirrors

Kind uses the containerd container runtime and allows customizing the containerd configuration through the cluster config spec.

We can use the containerdConfigPatches stanza to add registry mirrors to containerd. Registry mirrors are registered per image name prefix.

Given the registry proxy we previously created, we need to add the following configuration to our cluster spec to start taking advantage of it:

kind create cluster --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
    endpoint = ["http://proxy-docker-hub:5000"]
nodes:
- role: control-plane
- role: control-plane
- role: control-plane
- role: worker
- role: worker
- role: worker
EOF

With this configuration, all dockerhub images will be retrieved by containerd via our registry proxy that will take care of transparently caching them.

Running multiple registry proxies

In order to cache images coming from different registries we need to run multiple registry proxy instances.

Caching images coming from docker.io, gcr.io, k8s.gcr.io and quay.io should cover our needs to run clusters with Cilium, MetalLB and ingress-nginx.

For that, we need to run four instances of the registry container, each one configured for a specific remote registry:

# dockerhub
docker run -d --name proxy-docker-hub --restart=always \
  --net=kind \
  -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \
  registry:2# quay.io
docker run -d --name proxy-quay --restart=always \
  --net=kind \
  -e REGISTRY_PROXY_REMOTEURL=https://quay.io \
  registry:2# gcr.io
docker run -d --name proxy-gcr --restart=always \
  --net=kind \
  -e REGISTRY_PROXY_REMOTEURL=https://gcr.io \
  registry:2# k8s.gcr.io
docker run -d --name proxy-k8s-gcr --restart=always \
  --net=kind \
  -e REGISTRY_PROXY_REMOTEURL=https://k8s.gcr.io \
  registry:2

At this point we have four registry proxies running, we need to tell containerd to use them, depending on the image prefix to be pulled:

kind create cluster --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
    endpoint = ["http://proxy-docker-hub:5000"]
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."quay.io"]
    endpoint = ["http://proxy-quay:5000"]
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"]
    endpoint = ["http://proxy-k8s-gcr:5000"]
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."gcr.io"]
    endpoint = ["http://proxy-gcr:5000"]
nodes:
- role: control-plane
- role: control-plane
- role: control-plane
- role: worker
- role: worker
- role: worker
EOF

Wrapping it up

To conclude this article, I’ll run the script below two times.

The first time caches will be empty, the second time caches should contain all necessary images (fetched from the first run).

We’ll then compare the timings between first and second run and validate that caching brings significant speed improvements.

# create kind network if needed
docker network create kind || true# start registry proxies
docker run -d --name proxy-docker-hub --restart=always --net=kind -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io registry:2 || true
docker run -d --name proxy-quay --restart=always --net=kind -e REGISTRY_PROXY_REMOTEURL=https://quay.io registry:2 || true
docker run -d --name proxy-gcr --restart=always --net=kind -e REGISTRY_PROXY_REMOTEURL=https://gcr.io registry:2 || true
docker run -d --name proxy-k8s-gcr --restart=always --net=kind -e REGISTRY_PROXY_REMOTEURL=https://k8s.gcr.io registry:2 || true# delete kind cluster
kind delete cluster || true# starting timer
SECONDS=0# create kind cluster
kind create cluster --image kindest/node:v1.23.1 --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true
  kubeProxyMode: none
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
    endpoint = ["http://proxy-docker-hub:5000"]
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."quay.io"]
    endpoint = ["http://proxy-quay:5000"]
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"]
    endpoint = ["http://proxy-k8s-gcr:5000"]
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."gcr.io"]
    endpoint = ["http://proxy-gcr:5000"]
nodes:
- role: control-plane
- role: control-plane
- role: control-plane
- role: worker
- role: worker
- role: worker
EOF# install cilium
helm upgrade --install --namespace kube-system --repo https://helm.cilium.io cilium cilium --values - <<EOF
kubeProxyReplacement: strict
k8sServiceHost: kind-external-load-balancer
k8sServicePort: 6443
hostServices:
  enabled: true
externalIPs:
  enabled: true
nodePort:
  enabled: true
hostPort:
  enabled: true
image:
  pullPolicy: IfNotPresent
ipam:
  mode: kubernetes
hubble:
  enabled: true
  relay:
    enabled: true
EOF# install metallb
KIND_NET_CIDR=$(docker network inspect kind -f '{{(index .IPAM.Config 0).Subnet}}')
METALLB_IP_START=$(echo ${KIND_NET_CIDR} | sed "s@0.0/16@255.200@")
METALLB_IP_END=$(echo ${KIND_NET_CIDR} | sed "s@0.0/16@255.250@")
METALLB_IP_RANGE="${METALLB_IP_START}-${METALLB_IP_END}"
helm upgrade --install --namespace metallb-system --create-namespace --repo https://metallb.github.io/metallb metallb metallb --values - <<EOF
configInline:
  address-pools:
  - name: default
    protocol: layer2
    addresses:
    - ${METALLB_IP_RANGE}
EOF# wait for pods to be ready
kubectl wait -A --for=condition=ready pod --field-selector=status.phase!=Succeeded --timeout=20m# install ingress-nginx
helm upgrade --install --namespace ingress-nginx --create-namespace --repo https://kubernetes.github.io/ingress-nginx ingress-nginx ingress-nginx --values - <<EOF
defaultBackend:
  enabled: true
EOF# wait for pods to be ready
kubectl wait -A --for=condition=ready pod --field-selector=status.phase!=Succeeded --timeout=15m# show duration
duration=$SECONDS
echo "$(($duration / 60)) minutes and $(($duration % 60)) seconds elapsed."

Results will depend on the performance of the host system.

On my local computer I observed the following measurements:

# first run (without cache)
19 minutes and 33 seconds elapsed.# second run (with cache)
3 minutes and 32 seconds elapsed.

Finally, caching images makes starting Kind clusters approximately 6 times faster, this is a very nice improvement, especially if you’re used to delete and recreate clusters from scratch often 🎉

Credits

Most of this experiment was strongly inspired from https://maelvls.dev/docker-proxy-registry-kind 🙏

Kind
Optimization
Networking
Caching

--

--

Charles-Edouard Brétéché

Written by Charles-Edouard Brétéché

178 Followers

Platform enginner, Kubernetes enthusiast

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams