Upgrading OpenShift Container Platform in disconnected environment for IBM Storage Fusion HCI

6 min readMay 29, 2023

IBM Storage Fusion is a container-native hybrid cloud data platform that offers simplified deployment and data management for Kubernetes applications on Red Hat® OpenShift® Container Platform. IBM Storage Fusion is designed to meet the storage requirements of modern, stateful Kubernetes applications and to make it easy to deploy and manage container-native applications and their data on Red Hat OpenShift Container Platform.

IBM delivers multiple newer versions of IBM Storage Fusion HCI periodically in a year. Each IBM Storage Fusion HCI version can be installed afresh or a previous version can be upgraded to next released version.

Each IBM Storage Fusion HCI version gets installed with a predetermined Red Hat® OpenShift® Container Platform version in a fresh installation, however, it can be upgraded to required Red Hat® OpenShift® Container Platform version following Red Hat® OpenShift® Container Platform upgrade documentation provided target Red Hat® OpenShift® Container Platform version is supported by IBM Storage Fusion HCI version being used since not every Red Hat® OpenShift® Container Platform version is certified/supported by IBM Storage Fusion HCI versions. Having said that, Red Hat® OpenShift® Container Platform Z release upgrade is always supported. For example IBM Storage Fusion HCI v 2.4.0 is installed on Red Hat® OpenShift® Container Platform v4.10.21 initially but Red Hat® OpenShift® Container Platform can be upgraded to different Z releases (4.10.ZZ).

Red Hat® OpenShift® Container Platform upgrade can be completed by following Red Hat® documentation. IBM Storage Fusion HCI is supported in disconnected environments too and in such environment, to upgrade Red Hat® OpenShift® Container Platform, Red Hat® documentation for disconnected setup can be followed. This article includes steps to perform disconnected Red Hat® OpenShift® Container Platform for IBM Storage Fusion HCI with troubleshooting.

Pre-requisites

Before you begin to upgrade Red Hat® OpenShift® Container Platform in disconnected environment, there are some pre-requisites that must be met:

You have a IBM Storage Fusion HCI 2.4 or higher installed with OCP 4.10.xx version.

2. All cluster operators are available and healthy.

3. There are no machine configuration roll outs in progress.

4. IBM Storage Fusion HCI platform, storage and services are healthy. Please refer to Knowledge Centre for comprehensive list of checks.

5. Mirror images for target Red Hat® OpenShift® Container Platform 4.10.ZZ version to your enterprise registry. Refer to IBM Storage Fusion HCI mirroring steps or Red Hat® OpenShift® Container Platform documentation for mirroring images to your registry. Replace version with target Red Hat® OpenShift® Container Platform version you wish to upgrade to while mirroring images. Make note of ImageContentSourcePolicy that gets printed after mirroring command is completed. This will be used in later steps during upgrade. A sample policy where release image is being mirrored to myregistry.com/hci/oc41058


  repositoryDigestMirrors:
  - mirrors:
    - myregistry.com/hci/oc41058
    source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
  - mirrors:
    - myregistry.com/hci/oc41058
    source: quay.io/openshift-release-dev/ocp-release

6. Add image registry credentials to IBM Storage Fusion HCI platform global pull-secret in namespace openshift-config. This step is only needed if you have mirrored images to a new registry. If you used same registry from which you had installed IBM Storage Fusion HCI platform, then registry credentials are already present in your cluster.

7. Above operation start machine configuration roll out across machine configuration pools. Wait for this roll out to complete before going to next step.

8. Update two if existing ImageContentSourcePolicy in IBM Storage Fusion HCI cluster

8.1 Add new mirror for source: quay.io/openshift-release-dev/ocp-release from imagecontentsourcepolicy printed in step 5 to existing list of mirrors and save imagecontentsourcepolicy. A sample below:

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  creationTimestamp: "2023-05-04T07:43:31Z"
  generation: 2
  name: image-policy-0
  resourceVersion: "91682431"
  uid: 52eee5ea-8313-4832-bd15-f1f7e5b893e3
spec:
  repositoryDigestMirrors:
  - mirrors:
    - myregistry.com/hci/oc41058
    - myoldregistry.com/hci/oc41021
    source: quay.io/openshift-release-dev/ocp-release

8.2 This operation start machine configuration roll out across machine configuration pools. Wait for this roll out to complete before going to next step.

8.3 Add new mirror for source: quay.io/openshift-release-dev/ocp-v4.0-art-dev from imagecontentsourcepolicy printed in step 5 to existing list of mirrors and save imagecontentsourcepolicy. A sample below:

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  creationTimestamp: "2023-05-04T07:43:31Z"
  generation: 2
  name: image-policy-1
  resourceVersion: "91691000"
  uid: ef15f727-5f4d-4ae4-935b-3d0f705f7ec5
spec:
  repositoryDigestMirrors:
  - mirrors:
    - myregistry.com/hci/oc41058
    - myoldregistry.com/hci/oc41021
    source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

8.4 This operation start machine configuration roll out across machine configuration pools. Wait for this roll out to complete before going to next step.

9. Create signature config map in IBM Storage Fusion HCI cluster for Red Hat® OpenShift® Container Platform target version.

9.1 Create variables for creating signature for Red Hat® OpenShift® Container Platform target version. Here 4.10.58 is an example, ensure to use target version to which you want to upgrade to and for which you mirrored images to your private registry in step 5.

OCP_RELEASE_NUMBER=4.10.58
ARCHITECTURE=x86_64
DIGEST="$(oc adm release info quay.io/openshift-release-dev/ocp-release:${OCP_RELEASE_NUMBER}-${ARCHITECTURE} | sed -n 's/Pull From: .*@//p')"
DIGEST_ALGO="${DIGEST%%:*}"
DIGEST_ENCODED="${DIGEST#*:}"
SIGNATURE_BASE64=$(curl -s "https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/${DIGEST_ALGO}=${DIGEST_ENCODED}/signature-1" | base64 -w0 && echo)

9.2 Create config map definition yaml checksum-${OCP_RELEASE_NUMBER}.yaml. Replace values of variables from values derived in step 9.1.

apiVersion: v1
 kind: ConfigMap
 metadata:
   name: release-image-${OCP_RELEASE_NUMBER}
   namespace: openshift-config-managed
   labels:
     release.openshift.io/verification-signatures: ""
 binaryData:
   ${DIGEST_ALGO}-${DIGEST_ENCODED}: ${SIGNATURE_BASE64}

9.3 Create signature config map in cluster

oc create -f checksum-${OCP_RELEASE_NUMBER}.yaml

10. Check etcd db size

10.1 List etcd pods in openshift-etcd namespace

oc -n openshift-etcd get po

10.2 Connect to etcd pod

oc -n openshift-etcd rsh etcd-control-1-ru2.mycluster.mydomain.com

10.3 Get etcd db size

etcdctl endpoint status -w table --cluster

If above output lists etcd db size beyond 2 GB then before starting upgrade, consider defragmenting etcd db following these steps.

Upgrade Red Hat® OpenShift® Container Platform

Execute below command to start upgrade. Here sha256 is a sample value. Endure this to replace with release digest of Red Hat® OpenShift® Container Platform target version you are upgrading to and that you mirrored to your image registry. This release digest can be found from this link. Just replace right target version in link Or use value of DIGEST from step 9.1.

oc adm upgrade --allow-explicit-upgrade --to-image myregistry.com/hci/oc41058@sha256:7c2f759d9b8c42fa6f94ea1c0807a6418ad6d5fa904ab69ffaf05bc1c2a86c14

Monitor upgrade progress

Execute below command to monitor upgrade progress

oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.46   True        True          4h11m   Working towards 4.10.58: 205 of 775 done (26% complete)

Once upgrade completes, command will report version as 4.10.58 (target version you upgraded to).

For detailed status and history of upgrades use below commands

oc get clusterversion -o json|jq ".items[0].spec"
oc get clusterversion -o json|jq ".items[0].status.history"

Troubleshooting

If upgrade does not start then run below command

oc get clusterversion -o yaml

If it complains about a failed unpack job then, list all jobs in cluster and look for jobs that show 0/1 completion.

oc get jobs -A

For identified job, check events and pods. Also check events and logs for pod associated with job for investigating root cause of upgrade not beginning. Possible reasons can be:

Firewall does not allow access to registry to which new images are copied to.
Global pull-secret is not updated which thrown authentication issue during pulling release digest image.
Incorrect release digest specified in upgrade command.
Different release digest is mirrored to registry while a different digest is specified in upgrade command.

2. If upgrade is stuck and not progressing, check below command to get errors from upgrade.

oc adm upgrade
info: An upgrade is in progress. Working towards 4.10.58: 205 of 775 done (26% complete)

warning: Cannot display available updates:
  Reason: RemoteFailed
  Message: Unable to retrieve available updates: Get "https://api.openshift.com/api/upgrades_info/v1/graph?arch=amd64&channel=stable-4.10&id=23a7cca9-bafb-4065-b726-ab5de1f84205&version=4.10.58": dial tcp 44.199.134.143:443: connect: connection refused

Above error can be ignored for disconnected upgrade. Upgrade will eventually progress.

References

To learn more about IBM Fusion HCI visit https://ibm.github.io/storage-fusion/fusion-hci/overview

Disclaimer: The above article is personal and does not necessarily represent IBM’s positions, strategies, or opinions.