Deploy and Update Zarf Packages in an Air Gap

This tutorial gives an example of using Zarf to deploy a Podinfo package into an air gapped Kubernetes cluster, and then upgrading that Podinfo package to a newer version.

Prerequisites:

• Internet connected computer
• Air gapped computer (see options below)
• Zarf CLI installed
• Familiar with Zarf CLI
• Familiar with kubectl

Introduction

Have you ever needed to deploy or upgrade an application on an air gapped Kubernetes cluster? Zarf is an open source tool that helps you package up parts of the internet and deploy those parts into an air gapped environment. If you’re not familiar with Zarf, checkout my previous tutorials here, here, and here. This tutorial will assume you have some Zarf knowledge and are familiar with basic Linux and kubectl commands.

Goal

Use Zarf to deploy an air gapped Kubernetes cluster and Podinfo application

The goal of this tutorial is to first use Zarf to deploy a k3s cluster. Zarf has the ability to deploy a cluster for you, if you are on a Linux machine. If you are on any other type of machine, you will first need to provision your own Kubernetes cluster using something like MiniKube, k3d, or some other Kubernetes tool of your choice.

First, we will deploy the Zarf init package (learn about it here) and this is where you will be asked (on a Linux machine) if you want to deploy a k3s cluster. If you have already deployed your own Kubernetes cluster and are on a non-Linux machine, you will still need to run zarf init to get some other components deployed (zarf-injector, zarf-seed-registry, and more).

Next, we will use this Podinfo example repository to create a Zarf package of Podinfo. Zarf requires the creation of a ZarfPackageConfig object via a customized zarf.yaml. In the above-mentioned repository, the zarf.yaml has already been created for you. We will use zarf package create to create the Zarf Podinfo package, then we will take the Zarf init package and the Zarf Podinfo package, place them on a USB, connect that USB to an air gapped Linux machine, and deploy both the Zarf init package and the Zarf Podinfo package on the air gapped machine.

Air Gap Setup

There are several different options for setting up an air gapped environment. My development computer is a MacBook Pro, but I have a home lab that consists of this Intel Nuc 13 Pro. I love the Nuc because it doesn’t take up much space and it allows me to have access to a Linux machine, which I test on consistently. Usually I SSH over wifi into the Nuc when I want to use it. For this project, I was able to quickly air gap the Nuc by first connecting its own monitor (I used a spare travel monitor) and keyboard (also a spare I had lying around). Then I followed these commands to turn off the wifi. Note that the method of turning off your internet connection will vary depending on the specific brand and model of equipment you have. After turning off the wifi, I confirmed no internet connectivity by running ping 8.8.8.8 which failed, as expected.

If you don’t have a home lab lying around, you could deploy a Kubernetes cluster to your computer, download the Zarf CLI and Zarf init packages for your computer’s architecture, create the Zarf Podinfo package and have it locally, then turn off your wifi and confirm your computer is not connected to the internet via a cable somewhere.

You could also follow this tutorial to spin up an air gapped VM. The key here is you need to have the Zarf CLI installed on both your internet connected machine and your machine that will be air gapped. You also need to have the Zarf init package and the Zarf Podinfo package downloaded to a USB or some other medium device before you air gap your machine.

Internet Connected Environment

Create Zarf Podinfo Package

This section requires that the Zarf CLI is installed on your machine. If you’re not sure, run zarf version to see if it is installed. If you get back a version tag, you’re good to go. If the zarf command is not recognized, follow these steps to install the Zarf CLI.

Clone the Zarf repo to your local machine, then navigate to the /examples/podinfo-flux directory. Note that in this directory the zarf.yaml is already configured for you, so you can simply run zarf package create to create the Zarf Podinfo package.

git clone git@github.com:defenseunicorns/zarf.git
cd zarf/examples/podinfo-flux/
zarf package create

The Zarf Podinfo package is simply a tarball file with all of the components in it required to deploy Podinfo onto your cluster. Note that the default file architecture appears to be amd64. If you will be working on an arm64 machine, you can add architecture: arm64 to the metadata of the zarf.yaml or use the -a flag with the zarf package create command to specify the architecture (ex. zarf package create -a amd64).

Get Zarf Init Package

Our plan is to use the Zarf init package to deploy our Kubernetes cluster for us on the air gapped Nuc, so we need to obtain the appropriate Zarf init package for our architecture. You can find the Zarf init packages the same place you find Zarf releases. My air gapped machine already has Zarf v0.32.1 installed, so I am going to grab the corresponding Zarf-init package.

By clicking the link, you can download the package to your Downloads folder. I now want to move both the Zarf Podinfo package and the Zarf init package to my USB drive. On a Mac, USBs are mounted in /Volumes.

Confirm the Zarf Podinfo package and the Zarf init package are now on your USB drive.

Safely eject your USB drive.

Air Gapped Environment

Follow one of the steps discussed previously to air gap your machine. I will be using my air gapped NUC, a Linux machine for the rest of the tutorial. Plug your USB into your Linux server and mount the device. You mount the USB device by first finding the name of it with sudo fdisk -l | grep sd .

Use lsblk to see if the USB is mounted or not.

The USB is not currently mounted, so first let’s make a directory where we want to mount the device. Then use the sudo mount command to mount the USB to the directory you made.

sudo mkdir /mnt/usb
sudo mount /dev/sda1 /mnt/usb/

Confirm the mount with lsblk | grep sda1 .

Deploy the Zarf Init Package​

To initialize your cluster, you need to run the command zarf init preferably from the directory where your Zarf init package is located. This command will search for a file with the specific naming convention: zarf-init-{ARCHITECTURE}-{VERSION}.tar.zst. The architecture must match that of the cluster you are deploying to. If you are deploying to a cluster with a different architecture, you will need to specify the name of the architecture you are deploying on with the -a flag. Both my Zarf init package and Zarf Podinfo package are amd64, the appropriate architecture for my Linux machine, so I do not need to use the -a flag.

Using Zarf init to deploy a k3s cluster requires root access (not just sudo). To deploy the Zarf init package, use the following commands:

sudo su
zarf init

When prompted “Deploy the k3s component?” type y and then hit Enter (if you are not on a Linux machine, you will not see this prompt).

Once the Zarf init package has finished deploying, you can see all of your pods with kubectl get pods -A .

Deploy the Zarf Podinfo Package

In the directory where your Zarf Podinfo package lives, run the following command:

zarf package deploy <zarf_podinfo_package_name>

Once the deployment is complete, run kubectl get pods -A again to see that your Podinfo pods deployed correctly.

That’s it! You’ve just deployed the Zarf init package and Zarf Podinfo package in your air gapped cluster.

Bonus: Use Zarf to Upgrade an Application on an Air Gapped Kubernetes Cluster

Let’s say you have Podinfo running on your air gapped cluster but you need to change the version. How can you do that?

The process is very similar to what we’ve already done. You would need to go back to an internet connected device, upgrade your Zarf yaml files with the different Podinfo version, then build a new Zarf Podinfo package. You would place that new Zarf Podinfo package on your USB device, then carry that USB to your air gapped machine and deploy the new package just like before. When deploying the new Zarf Podinfo package, Zarf will perform a rolling deployment, where it takes down a few pods at a time, updates them, then restarts new pods with the updates, repeating this process until all of the pods have been updated.

To get started, check the version of Podinfo that is currently deployed with:

kubectl get pods -n podinfo -o jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}' | sort

The current version of Podinfo deployed on my air gapped machine is v6.5.4. At the time of this writing, there is a newer version of 6.6.0, so we will upgrade to that. The new Zarf Podinfo package must be built in an internet connected environment, so head back over to your internet connected machine for the next few steps.

In the zarf.yaml file, change the image used to ghcr.io/stefanprodan/podinfo:6.6.0 .

In the podinfo-source.yaml file, change the tag to 6.6.0.

Now use zarf package create to create a new Zarf Podinfo package with the updated version. Remove the old zarf-package-podinfo-flux-amd64.tar.zst file from your USB since the new Zarf package will have the same file name. Move the new Zarf Podinfo package to your USB and insert the USB to your air gapped machine.

Back on your air gapped machine, navigate to the directory where your new Zarf package is.

Because zarf init has already been run on the air gapped machine, you do not have to run the command again. Use zarf package deploy to deploy the new package.

Once the package has deployed, you can check that your new pods are running with kubectl get pods -A . Note that the age of the Podinfo pods are measured in seconds, so those are our new pods.

Run the following command to confirm that we’re using the newer version of Podinfo:

kubectl get pods -n podinfo -o jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}' | sort

Successfully upgrade to Podinfo v6.6.0!

Conclusion

Hopefully this tutorial helped you better understand how to use Zarf to deploy and upgrade packages in an air gapped environment. To clean up everything we’ve done here, simply run the following command:

zarf destroy --confirm --remove-components

You can also delete the Zarf init package and Zarf Podinfo package from your machines if desired. To learn more about Zarf and its capabilities, visit the official Zarf documentation. As always, I appreciate you taking the time to read this, and look forward to us all learning together!