Creating Air-gapped K3s System Images with Private Registry
When setting up a lightweight Kubernetes cluster with K3s, there are certain images used by the system. In air-gapped environments, you have various methods to create these images on your own machine.
The first method involves creating your own private registry on your private network and setting up mirroring to redirect the traffic from attempting to pull K3s system images from docker.io to your private registry.
The second method System image are shared with their names in a Txt file during each K3s release. K3s keeps these images in the Dockerhub Public Registry. In order to load these Images in Air-gapped environments, They also share a Tar file which contains System Images that are created by Docker CLI Save command.
In this blogpost, I will focus on the second method and try to explain how we can Load the Images shared by K3s for v1.23.5 with our own Private Registry Prefix.
As you can see below, K3s creates the Image TAR with a bash script.
In this script, the images in the TXT file below are pulled one by one from docker.io with docker CLI and tar is created with docker save.
When we download this TAR and examine its content, there is a file named manifest.json.
This file contains information about System Images and their Layers.
[
{
"Config": "746788bcc27e25d1bcc6aaef10b62f83a71aa9f3cdc91906f7dfe0cd15c9ec5c.json",
"RepoTags": [
"rancher/klipper-lb:v0.3.4"
],
"Layers": [
"e6b8cc5e282829d1e71d697875a3f1087067327dcf87b676d23da5d9159b2ec2/layer.tar",
"c3ae600a19dfd3e5832b77a601cc4e655618a97996ca8d77ddf79de60df4ab03/layer.tar",
"82adaad27b66d0cc25ddaf9986ffd782ab23a1461ebb3f91585e325e309b4cdd/layer.tar"
]
},
{
"Config": "fb9b574e03c344e1619ced3ef0700acb2ab8ef1d39973cabd90b8371a46148be.json",
"RepoTags": [
"rancher/local-path-provisioner:v0.0.21"
],
"Layers": [
"1e33021cb9e4fc8e989445d68584dd2386b15e902fc4e95fba64a5a917759466/layer.tar",
"717a12541d300e9d3e497e8836b185a9ad290e39eacdb471cf88e8a8dfb9b845/layer.tar"
]
},What’s important to us is the RepoTags value here. This value contains the name of the saved Image.
According to the K3s documentation, when we put this Tar in the /var/lib/rancher/k3s/agent/images/ directory on our machine and start the K3s server systemd service, we expect the Images in this TAR to be automatically uploaded to the system by Containerd.
First, let’s set up the K3s Cluster. For installation, we can use the following Installation script offered by K3s.
You need to set INSTALL_K3S_SKIP_DOWNLOAD environment variable as true for not downloading System Images and loading them from the directory.
export INSTALL_K3S_EXEC="server -no-deploy traefik"
export INSTALL_K3S_SKIP_DOWNLOAD=true
bash install.shBelow, we see the tags of the Images uploaded to the system in the Cluster we established with K3s, with docker.io prefix by default.
We can also verify this transaction from Containerd Logs.
How can we override these images with Private Registry Prefix and tell K3s to use these images?
As a solution, we need to update the RepoTags value of each Image JSON object in Manifest.json with our own Private Registry.
Here the TAR creation method can be done from scratch. Download the images from Docker.io, retag them and create the TAR from scratch with the script below. Thus, RepoTags values in Manifest.json will be updated according to the Tag value. We will use muhammed.io as Private Registry Prefix.
#!/bin/bash
set -e
SYSTEM_IMAGES=(
rancher/klipper-helm:v0.7.0-build20220315
rancher/klipper-lb:v0.3.4
rancher/local-path-provisioner:v0.0.21
rancher/mirrored-coredns-coredns:1.9.1
rancher/mirrored-library-busybox:1.34.1
rancher/mirrored-library-traefik:2.6.1
rancher/mirrored-metrics-server:v0.5.2
rancher/mirrored-pause:3.6
)
PRIVATE_REGISTRY_PREFIX="muhammed.io"
images=""
for image_name in "${SYSTEM_IMAGES[@]}"; do
docker pull --platform=linux/amd64 "$image_name"
docker tag "$image_name" "$PRIVATE_REGISTRY_PREFIX/$image_name"
images=" $images $PRIVATE_REGISTRY_PREFIX/$image_name "
done
docker save $images -o k3s-airgap-images-amd64-v1.23.5.tar
Or we can manually update the RepoTags values in the manifest.json in the TAR we downloaded from the Release page. After that compress the files as Tar file.
After this process, the prefix belonging to our Private registry will now be included in the images loaded by the K3s server when it starts up.
Another question is how do we tell K3s Server to use these Images and not send requests to docker.io when running System Containers?
In the current situation, our images will be loaded with muhammed.io, but the K3s server will try to download the images from docker.io. We can see an example below.
Here, we can mirror the requests going to docker.io to muhammed.io, as in the first method. But we don’t want this. We have an air-gap environment and we want to use the images in TAR.
With the system-default-registry CLI flag that we can give to K3S Server, K3s prefixes the image names of the containers it runs. This value can also be configured with the K3S_SYSTEM_DEFAULT_REGISTRY environment variable.
In our example here, we set the value of muhammed.io as an environment variable before starting the installation.
export K3S_SYSTEM_DEFAULT_REGISTRY="muhammed.io"When we install K3s Server again, this time it will use the loaded Images. As we can see below, we were able to install the system without any network traffic to muhammed.io or docker.io.
We are able to verify this by performing tcpdump operation during installation.
Conclusion
If we want to set up our Cluster using K3s, a Lightweight Kubernetes solution, in an Air-gapped environment and use our Private Registry as a Prefix in our System Images, we can create the Released TAR with our own RepoTags values and Default Registry Configuration.
Thanks for reading. If you have questions or comments regarding this article, please feel free to leave a comment below.
Would like to get in touch? Reach me out on LinkedIn:
