Installing Containerized Applications on Internet-Restricted Environments

Sometimes you can find yourself in a situation where you have to install and run software that has a manifest file (a helm chart, a docker-compose file, a playbook etc.) which points to public repositories for container images, but the environment/infrastructure is internet restricted so the installation script cannot download required images.

Some folk refer this kind of environment as air-gapped. But I think, in a totally air-gapped infrastructure, servers shouldn’t even have network interfaces. The environment I refer to in this article is internet-restricted, but has a one-way connection to a private container registry server that can connect to the internet. Hence, maybe it can be called a bridged air gap since there is a DMZ involved:

A basic overview of the environment I’m trying to picture in this article

In this article, I will share some insights from my adventure of installing Spinnaker (https://spinnaker.io) on a K8S cluster, that has no internet connection.

There are pods with ErrImagePull and ImagePullBackOff statuses

Some repository software like Harbor (https://goharbor.io) offer proxy-cache solutions which you can utilize to get container images from popular registries like Docker Hub, via some other box on the network, that has access to the internet, for example: a server on the DMZ zone, like on the diagram above. But this functionality can be considered as limited, because not all software developers put their container images on Docker Hub, or k8s.gcr.io where you can easily configure a proxy-cache access with the help of tools like Harbor.

Harbor as a pull-through/proxy-cache repository.

Spinnaker for example, has its own installation tool called Halyard (hal for short), and by default it assumes the installation target would be able to gather container images from us-docker.pkg.dev. But since we don’t have internet connection on the K8S side, we see ErrImagePull and ImagePullBackOff statuses on spin-* pods. First, I attempted to setup a proxy-cache project on Harbor to be able to get images from us-docker.pkg.dev, but after some trial and error, I gave up and started looking for an alternative. This is when I stumbled upon this tool called Sinker (https://github.com/plexsystems/sinker), a tool written in Go, devised to gather container images from a source and push it to a target.

A screenshot from Sinker’s GitHub page.

Another handy feature Sinker has is that it can get standard input to prepare an image manifest, so you can simply redirect a kubectl command output to sinker, and boom!

kubectl -n spinnaker get pods -o jsonpath=”{.items[*].spec.containers[*].image}” | sinker create — --target repo

Now we have a file named .images.yaml on the working directory and this is the content;

target:
  repository: repo
sources:
- repository: spinnaker-community/docker/clouddriver
  host: us-docker.pkg.dev
  tag: 7.3.5–20210626040020
- repository: spinnaker-community/docker/deck
  host: us-docker.pkg.dev
  tag: 3.6.2–20210613221539
- repository: spinnaker-community/docker/echo
  host: us-docker.pkg.dev
  tag: 2.16.0–20210208200018
- repository: spinnaker-community/docker/front50
  host: us-docker.pkg.dev
  tag: 0.26.2–20210216140019
- repository: spinnaker-community/docker/gate
  host: us-docker.pkg.dev
  tag: 1.21.0–20210215200018
- repository: spinnaker-community/docker/orca
  host: us-docker.pkg.dev
  tag: 2.19.1–20210526040021
- repository: spinnaker-community/redis/redis-cluster
  host: us-docker.pkg.dev
  tag: v2
- repository: spinnaker-community/docker/rosco
  host: us-docker.pkg.dev
  tag: 0.24.1–20210526040021

We can now easily define your private repo here by modifying the target part of this manifest file and carry on. For example:

target:
  host: privaterepo.yourdomain.com
  repository: spinnaker

I’ve just renamed this manifest file as spinnaker-images.yaml after adding my private repo information as target. Now that we’re ready to pull those public images:

sinker pull --manifest spinnaker-images.yaml
INFO[0000] Finding images that need to be pulled ...
...
...
...
INFO[0171] All images have been pulled!

And then push them to our private repo:

sinker push --manifest spinnaker-images.yaml
INFO[0000] Finding images that need to be pushed ...
...
...
...
INFO[0124] All images have been pushed!

Normally, once we got the container images on our private registry, just modifying the registry path on a values.yaml for a helm chart, or image paths on a docker-compose file would do the trick. But Spinnaker’s installer Halyard, requires a little bit more steps for that same effect. It gets these kind of settings from the BOM (Bill of Materials). So we should create a custom BOM file that points to our private registry. (You can read more about it here: https://spinnaker.io/docs/guides/operator/custom-boms/)

For example, here is the BOM for Spinnaker 1.25.7 (you can get full versions list by running hal version list)

version: 1.25.7
timestamp: ‘2021–06–26 08:01:26’
services:
  echo:
    version: 2.16.0–20210208200018
    commit: 6b1f6d66071df5389e72d5fa2e8b0c65e78f5251
  clouddriver:
    version: 7.3.5–20210626040020
    commit: c7ee9cdf340fe5ce9953609eeb1405106daa68b8
  deck:
    version: 3.6.2–20210613221539
    commit: 803c512d9c000d1b5a1bd144c268dc4ecac6ef15
  fiat:
    version: 1.15.0–20210211170021
    commit: e8a17cbbb57951aabb483a5d35088bc10ba24590
  front50:
    version: 0.26.2–20210216140019
    commit: 9fd2fc75da57fd8c0f06ebdabf009a6c0d17d931
  gate:
    version: 1.21.0–20210215200018
    commit: 99362e3478f9c44643360a2b9b285fcdb63a3fc6
  igor:
    version: 1.15.0–20210208200018
    commit: 718892962443bd2d61b0b9c66d30ce0b50df150a
  kayenta:
    version: 0.20.0–20210203140017
    commit: 7fd1ded37aadf280b28215191723afcf9a1f3db8
  orca:
    version: 2.19.1–20210526040021
    commit: 1ca5f33999d911be9aaade10bdfeca0bb1c99643
  rosco:
    version: 0.24.1–20210526040021
    commit: 7be036290838d361f1bc64129a69a2f0a6099fd5 defaultArtifact: {}
  monitoring-third-party:
    version: 0.19.2–20210217024142
    commit: 3af17444d06eb41b8fbf38f4e94cf42d9f14ec18
  monitoring-daemon:
    version: 0.19.2–20210217024142
    commit: 3af17444d06eb41b8fbf38f4e94cf42d9f14ec18
dependencies:
  redis:
    version: 2:2.8.4–2
  consul:
    version: 0.7.5
  vault:
    version: 0.7.0
artifactSources:
  debianRepository: https://dl.bintray.com/spinnaker-releases/debians
  dockerRegistry: us-docker.pkg.dev/spinnaker-community/docker
  googleImageProject: marketplace-spinnaker-release
  gitPrefix: https://github.com/spinnaker

You should create your custom BOM file under /home/spinnaker/.hal/.boms/bom (this is the path for hal container) like 1.25.7.yml for example, and run the following command after editing the dockerRegistry path according to your private registry:

hal config version edit --version local:1.25.7

More info about why we add local before the version number can be found here: https://spinnaker.io/docs/guides/operator/custom-boms/#boms-and-configuration-on-your-filesystem

For Spinnaker 1.25.7, this method still tries to get redis-cluster container from the internet. A workaround for that issue can be found here (I’ve tested, it works): https://github.com/spinnaker/spinnaker/issues/3967#issuecomment-522306893

There are surely other steps that has to be followed before running hal deploy apply to ultimately install Spinnaker. I just tried to cover the areas that are related to internet restrictions. A nice installation guide for regular environments that are able to connect to the internet can be found at: https://github.com/justmeandopensource/kubernetes/blob/master/docs/setup-spinnaker.md

So this was one of the workarounds I’ve found when dealing with deployments on internet-restricted environments. I hope you find it useful. Thanks for reading!