Manual Air Gap Software Delivery
Deploying software to an air-gapped environment is the most challenging way to install software. What usually doesn’t help is that most software assumes internet connectivity to install it and for its day two operations. This post will walk through the manual air gap installation of Only Office onto a Fedora VM. This will be a longer-than-average blog post and includes all of the pain and skinned knees that went into getting this software installed, and that’s the point. Manual Air gap software delivery is difficult. Proceed with caution!
Feel free to skip the painful details and scroll down to the Tell Me There’s a Better Way section.
The Setup
An explanation of the topology is first needed to understand how the “air gap” is being set up.
The layout is super simple, but there’s an essential item to highlight. The air gap VM attaching to a local bridge interface in the laptop means it uses the laptop’s network interface to reach the outside world under normal operating conditions. Cool, no surprise there. This VM will have its network connectivity disabled but in a way that doesn’t shut down the VM’s virtual NIC. Instead, its network will be downed by injecting a default route to a dummy interface, making anything internet-hosted unreachable. However, the VM still has a connected route to the bridge interface on the host, which means that network connectivity to the host is still working. This posture means that data can be transferred from the host/laptop to the VM via SCP, even with the default route on the VM black holing all traffic that isn’t destined for the local bridge subnet. This type of transfer is analogous to carrying data across the air gap and will be used throughout this post.
The Installation
The software that will be installed on the VM again is Only Office. The actual software is less important than the process, and Only Office wasn’t selected to pick on them or highlight any shortcomings on their part. Most software is hard to install in an air gap, and something had to be chosen.
The first step is to offline the network in the VM, using the below script to do so:
The IP address of the VM needs to be obtained to be able to SCP data to it. On Fedora, the main interface is enp0s5. To display its IP information, run ip addr show enp0s5:
The relevant output above appears after inet on line 4, and the IP address for the VM is, which can be used to transfer data from the host to the VM.
Based on the Only Office documentation, the docs community edition can be installed as a Docker container or as an RPM package. To get the install script onto the VM, pull down the installation script on the host and SCP it over the “air gap” to the VM. The IP address of the VM needs to be obtained to be able to SCP data to it.
Note on the laptop where this was tested; the VM was assigned an IP address of 10.211.55.5 for its interface on the bridge.
From the VM now, let’s make a directory to mess around with Only Office and move the installation script there. The script isn’t executable after downloading it, so let’s update that as well. The install script requires root privileges, so let’s get a root shell. Finally, the recommended installation is via Docker, which we will select.
As expected, the installation fails because Docker isn’t installed. The following section will work through getting Docker installed.
Docker Installation
We’ll need to work through the manual installation method to install Docker on Fedora. Digging into the Fedora repositories, they have both ARM RPMs and the AMD64 RPMs. From the host machine, download the latest version of all the packages from the appropriate link corresponding to your workstation CPU architecture. The laptop used to run this installation is an M1 Mac, so the aarch64 RPMs will be downloaded.
Create a dedicated folder on the VM for the Docker RPMs:
Then transfer all of the RPMs over the air gap via SCP to the VM:
Now install the RPMs on the VM:
Ugh… Lines 14–15 show a dependency, libcgroup, that needs to be installed before installing the Docker packages. Found the appropriate package after some Googling. Download it on the host and then SCP it to the VM.
Now install it on the VM:
Now the Docker RPMs should install cleanly:
Sweet! Now let’s start Docker and make sure it’s running:
For those following along at home, this next step is optional but helpful; let’s add the current user to the docker group on the VM.
Back to Only Office
Now the Only Office container needs to be ported over to the VM. Starting with the docs-install.sh script from the Only Office site, this chunk looks for another script to download:
Line 5 above shows the other script. Let’s download it and take a look.
The install.sh script is a 1000+ line bash script that performs some checks on the environment to ensure enough resources to run the application. The information we want from this script is in line 8 above, the container image. This refers to the name and location of the image installed by this script, which can be found on Docker Hub here. Download this image to the host (note if you do not have Docker installed on your laptop/host machine, follow the installation instruction here first and then return to the course guide).
Now that the image has been pulled, the docker save command can save the container as a tarball.
Now the docker load command can be used to make Docker on the VM aware of the image:
The container is finally ready to run. The docs-install.sh script supports looking for other installation scripts locally via a flag. Let’s try using that and see what happens:
Ultimately, no container was started. The script logic cannot detect that the container image already exists locally and is trying to pull it. Let’s take a look at the help available (note unclear why, but this command took minutes to run during testing, the joys of the air gap!):
The -di flag will accept a path to a tar file to specify a container image. That should be what is needed here.
So it’s expecting an image tag even though a path to the image was provided. Let’s pass an image tag, then. Referencing the help output above, the -dv flag to the script is needed.
The CTRL-C was the air gap equivalent of rage quitting, but it already confirmed it couldn’t find the latest image. Let’s confirm we’re not crazy and see what images the VM has:
Ok, not crazy. Let’s try to run it manually, then. Digging back into the installation script, there are some parameters that need to be passed to the container. Here’s the relevant block:
Starting with the last command, it looks like a network gets referenced to run the container. Then a bunch of arguments get passed to docker run in the form of “${args[@]}”, which are a bunch of volume mounts as defined above the docker run command. We need to dig through this script and find all of the variable definitions that are in the arguments to reconstruct the command that can be passed to the VM:
Let’s export these in the VM so that they’re available.
The arguments that get passed into the container mount several volumes. Let’s create those on the VM:
Cool. Next, let’s create the network that gets referenced in the docker run command:
Let’s try starting the container:
Success! Now, how to test it? We first need to grab the IP address of the container. Let’s try docker ps:
There is no IP address info in there, but the output shows that the container is listening on 80 and 443. Note the container's name as well (nifty_bassi in this case). We’ll need that information shortly. Since each Docker network runs its own IPAM, the network itself should contain this information:
Note the onlyoffice network ID is at the beginning of line 6 and used in the second command. Using jq to parse through the output, we see that the IP of the nifty_bassi container is 172.19.0.2. Let’s try connecting to that endpoint from a browser on the VM:
Finally! Even though the screenshot above shows the UI, it took a while to display. The likely reason why is also the main point of this section: the vast majority of software assumes internet access. And to be fair to Only Office, this is usually a reasonable assumption. This is part of the reason that air gap software delivery is so difficult.
Tell Me There’s a Better Way…
As you were warned, it was painful to get to this point. The software was installed, but it took some hoops to jump through. Now imagine that instead of transferring data via SCP between a host and a VM, you must enter a Sensitive Compartmented Information Facility (SCIF, pronounced “skiff”) or drive for miles to transport the bits across the air gap. This same installation could have taken hours, if not days or weeks. And again, this post isn’t meant to pick on Only Office. Most software is not designed to be installed in an air-gapped environment.
This is why Defense Unicorns developed Zarf, an open-source project to make deploying software to air-gapped environments easy. To get this same software installed on the same system using Zarf, the following needs to be done:
- Install Zarf on a system outside of the air gap (laptop/host system in this case)
- Create a Zarf package for Only Office
- Transport the Zarf binary, Zarf init package, and Zarf Only Office package across the air gap
- Run two commands on the air gap system to install software
- Profit
Install Zarf
The process to install Zarf on MacOS is delightfully simple, as per the docs:
$ brew tap defenseunicorns/tap && brew install zarfRefer to the docs for similarly easy installation methods on other platforms.
Only Office Zarf Package
An existing package to install games was modified a bit for Only Office. Here’s the package structure on the filesystem:
The zarf.yaml file defines the package:
Notice that there are references to two manifests in this file on lines 13 and 14, which correspond to the other two files on the filesystem. Here is the content of those files:
And finally, service.yaml:
To create the Zarf package, run zarf package create in the directory where the zarf.yaml is located:
The output of this command is to create a .zst package on disk on line four below:
This is one of the files that must be transported across the air gap.
Transport Bits Across the Air Gap
As listed in a previous section, these are the files that need to be transported across the air gap:
- The Zarf binary
- The Only Office Zarf package that was just created
- The Zarf init package
The init package can be downloaded directly from Github. This walkthrough was done with Zarf version v0.25.1, so the init package was downloaded from this page. Transfer all the bits across from the host to the VM!
Installation on the Air Gap System
All subsequent steps happen on the air gap system. The first of which is to run a zarf init. This command produces a lot of output and requires some user interaction (which could be alleviated by providing additional command line flags). It must be run as root to install the k3s component, and it’s done executing in a minute or so. The full output can be viewed in this gist. It’s been removed for readability in the post. Enjoy a gif of Zarf floating before moving on to the package installation.
Next, deploy the Only Office Zarf package:
As line 55 suggests, the zarf connect only-office will open a tunnel to the application:
Now, connect via a browser to the URL in line 6 above:
Profit.
Conclusion
Zarf takes a declarative approach to solve the software delivery problem across the air gap. It took much longer to write the section of this post to install Only Office via Zarf than it did to install Only Office via Zarf. The Only Office Zarf package could easily be tested on a system to ensure it’s working correctly before transporting it across the air gap. This way, the transfer only has to happen once.
Feel free to reach out to us to see a demo at hello@defenseunicorns.com.
