Mini HowTo — Run `oc-mirror` inside a container for importing k8s operators into internal networks

Credits first!

Before you read this article I highly recommend reading my dear friend and colleague Hillay Amir two incredibly well-written articles about the new Red Hat utility tool oc-mirror. They are referenced in the next section.

My added-value

oc-mirror and skopeo are binaries made by Red Hat to work with imported images; The most common use cases are the ones Hillay Amir also wrote about:

1. Import operators into internal networks & Push them to the internal registry; Operator is usually made of multiple containers generated by different images, specific for the Operator’s version.

Mirroring specific versions of K8S Operators

2. Mirror Openshift version for upgrade requires pulling and pushing *TONS* of images relevant to this version.

OC-Mirror surprised us again with new abilities

There are a few commonalities when working with all of them:

1. They should run on at least Red Hat Enterprise Linux (RHEL) 8
2. They all use ~/.docker/config.json credentials file to login remote registries for pulling/pushing images

When working with customers I came across several scenarios where the customer did not have a RHEL 8 admin’s bastion server for working with these tools; I figured that the easiest & flexible solution will be to run these tools in a RHEL8-based container so that I can use these tools even in non-supported environments instead of running tons of podman’s pull & push command manually.

Also because I did some debugging I will add notes to the article.

HowTo

Part 1 —External Network

1. Working with the oc-mirror binary

Note! In the external network I assume you’ll have an updated version of Fedora (a VM on the external server is good enough or either the external server itself can run Fedora)

On your browser go to https://console.redhat.com/openshift/install/pull-secret and download the pull-secret.

The pull-secret holds credentials for accessing Red Hat’s public registries for pulling the Openshift Operators & Openshift mirrors, oc-mirror use this pull-secret for pulling the relevant images.

mkdir -p ~/.docker
mv ~/Downloads/pull-secret.txt ~/.docker/config.json

Download oc-mirror

wget https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable/oc-mirror.tar.gz
tar xvzf oc-mirror.tar.gz -C /usr/bin
chmod +x /usr/bin/oc-mirror
rm -rf oc-mirror.tar.gz

Use the tool based on Hillay’s article referenced above; Here is an example:

# List of available catalogs; 
oc-mirror list operators --catalogs --version=4.10
# ----OUTPUT----
# registry.redhat.io/redhat/redhat-operator-index:v4.10
# registry.redhat.io/redhat/certified-operator-index:v4.10
# registry.redhat.io/redhat/community-operator-index:v4.10
# registry.redhat.io/redhat/redhat-marketplace-index:v4.10

oc-mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.10
# ----OUTPUT----
# NAME                                          DISPLAY NAME                                                                 DEFAULT CHANNEL
# 3scale-operator                               Red Hat Integration - 3scale - Managed Application Services                  threescale-mas
# advanced-cluster-management                   Advanced Cluster Management for Kubernetes                                   release-2.6
# amq-broker-rhel8                              Red Hat Integration - AMQ Broker for RHEL 8 (Multiarch)                      7.10.x
# ...
# ...
# nfd
# ...
# ...

oc-mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.10 --package=nfd
# ----OUTPUT----
# PACKAGE  CHANNEL  HEAD
# nfd      stable   nfd.4.10.0-202211280957

oc-mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.10 --package=nfd --channel=stable
# ----OUTPUT----
# VERSIONS
# 4.10.0-202211280957

# Generate ImageSetConfiguration based on the data gathered above
# In this one I'm telling oc-mirror to download "nfd" operator in a specific version (4.10.0-202211280957) for Openshift version 4.10 
cat > ImageSetConfiguration.yaml << EOF
apiVersion: mirror.openshift.io/v1alpha2
kind: ImageSetConfiguration
archiveSize: 50
mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.10
    packages:
    - name: nfd
      channels:
      - name: stable
        minVersion: '4.10.0-202211280957'
        maxVersion: '4.10.0-202211280957'
EOF

oc mirror --config=ImageSetConfiguration.yaml file://nfd-operator-4.10.0-202211280957-openshift-v4.10

ls nfd-operator-4.10.0-202211280957-openshift-v4.10/
# --- OUTPUT ---
mirror_seq1_000000.tar  # Our Archive
oc-mirror-workspace # Extra metadata, worth keeping

Note! oc-mirror uses /tmp for the archiving process so you need to make sure it has enough space in this dir too.

Note! If somewhere along the way of the importing process you are also using Windows — make sure you are zipping your tar file so the Hash won’t change because otherwise the pushing process won’t work in the internal network!

zip operator.zip nfd-operator-4.10.0-202211280957-openshift-v4.10/

sha256sum operator.zip # ==> Save the output and test it again inside the internal network

2. Build the image with oc-mirror inside

cat > Containerfile << EOF
FROM registry.access.redhat.com/ubi8/ubi:latest 

COPY oc-mirror /usr/bin/oc-mirror

RUN yum install -y podman skopeo curl jq wget &&\
    yum clean all &&\
    rm -rf /var/cache/yum &&\
    wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-client-linux.tar.gz &&\
    tar xzvf openshift-client-linux.tar.gz -C /usr/bin/ &&\
    chmod +x /usr/bin/oc &&\
    rm -fr openshift-client-linux.tar.gz &&\
    mkdir -p /root/.docker/ &&\
    chmod +x /usr/bin/oc-mirror

CMD ["/usr/bin/bash"]
EOF

Build the image and save it to tar.gz file

podman build -f Containerfile -t container-tools-with-oc-mirror:latest
podman save container-tools-with-oc-mirror:latest > container-tools-with-oc-mirror.tar.gz

If you zipped the operator archive in the last section, also zip the saved image as well and only then import it

zip container-tools-with-oc-mirror.zip container-tools-with-oc-mirror.tar.gz

Part 2 — Internal network!!!

1. Run oc-mirror in container

sha256sum operator.zip # Verify the archive is well-imported

unzip operator.zip

mv nfd-operator-4.10.0-202211280957-openshift-v4.10 /SOME/PATH

unzip container-tools-with-oc-mirror.zip

podman load < container-tools-with-oc-mirror.tar.gz

podman run \               # Podman run new container
--name oc-mirror \         # Name the container
--rm \                     # Remove the container by the end of the operation
--privileged \             # Run as privileged
--net=host -it \           # Enable the container access to the network for it to be able to push the images to the network's registry
                           # "-it" for interactive shell
-v ~/.docker/config.json:/root/.docker/config.json \ # Give the container itself the config.json file
-v /SOME/PATH:/tmp \       # Map the path of the imported operator tar file into the container
container-tools-with-oc-mirror:latest \ # Use our image
/bin/bash                  # bash shell into the container

2. Push the operator from the oc-mirror container to the internal network registry

[inside the container]$ cd /tmp
[inside the container]$ oc-mirror --from /tmp docker://<registry_fqdn>:<port>/<operator_name> --dest-skip-tls
[inside the container]$ exit

Note! I pointed oc-mirror to the entire /tmp directory because there are scenarios where I have multiple tar files (small archiveSize in ImageSetConfiguration.yaml makes oc-mirror to split the operator’s archive to multiple archive-files)

3. Edit the auto-generated “CatalogSource” and “ImageContentSourcePolicy” files and save them in your Git + apply in Openshift

cd /SOME/PATH
cd oc-mirror-workspace
vi catalogSource.yaml # Edit the name of the object to the operator's name
vi imageContentSourcePolicy.yaml # Edit the name of the object to the operator's name
oc apply -f catalogSource.yaml
oc apply -f imageContentSourcePolicy.yaml