Testing Operator Upgrades via OLM on OpenShift 4.1

Published in

Kiali

8 min readSep 26, 2019

I’m going to describe the basic steps needed to test operator upgrades via OLM on OpenShift 4.1. Now that OLM has taken off in OpenShift 4, I’m sure this kind of thing is going to be needed by many people in the near future. I’m documenting this specifically for my use-case (the Kiali operator), but the general procedures are going to be the same for any operator.

The idea is simple. Suppose you have published your first version of your operator to the OpenShift Catalog — either as a community operator, a certified operator, or a productized Red Hat operator. Your customers/users can install this operator nicely via the OpenShift UI Console and the Operator Lifecycle Manager (OLM). But now you want to deliver a second version of your operator. Before publishing to the world, you want to test that your operator, through this OLM mechanism, can properly install/upgrade to the new version. This isn’t as easy as it seems, but here are the steps needed to perform this test.

In my use-case, I want to test upgrading the Kiali operator. The Kiali operator is a dependency of the Service Mesh operator, so we must take this into consideration, too. The Service Mesh operator actually has several dependencies — ElasticSearch, Jaeger, and Kiali operators. I need to be able to test my own dev build of the new Kiali operator version, keeping in mind that in order to test this I also need to install Service Mesh operator (and its other dependencies).

These steps will use Linux (bash) commands and will set up some environment variables along the way. Make sure you do these steps in sequence because environment variables set up in one step will be used in later steps.

Disable Out-of-box OperatorSources

The first thing that needs to be done is turn off the CVO so OpenShift’s Catalog will not resurrect the operators that are going to be deleted in the next step. This is not recommended for a production environment, but since we are doing this for development/testing purposes, this is fine to do on a development environment.

oc scale --replicas 0 -n openshift-cluster-version deployments/cluster-version-operator

We now want to remove the OperatorSources for the different out-of-box sources that have the currently published operators we need to test. Because the new Kiali operator needs to be delivered to OpenShift’s Catalog through its own OperatorSource, we must remove the old OperatorSources, lest our new operators conflict with the currently published ones. In this case, because Service Mesh, Kiali, Jaeger, and ElasticSearch are published in the redhat-operators quay.io app repository, we need to remove its OperatorSource.

oc delete operatorsources redhat-operators -n openshift-marketplace

If you are testing certified or community operators, you should remove them:

oc delete operatorsources certified-operators -n openshift-marketplaceoc delete operatorsources community-operators -n openshift-marketplace

It actually doesn’t hurt if you just remove all of the existing out-of-box OperatorSources — if you do this, you’ll completely empty out your Catalog:

for o in $(oc get operatorsources -n openshift-marketplace -o name); do oc delete $o -n openshift-marketplace ; done

Get Your Quay.io Username and Token

Because we will need the OpenShift Catalog to pull the new operators under test from an app registry (we will be using quay.io), we need to obtain credentials for a quay.io repository. Run this script, enter your quay.io username and password, and that quay.io account’s security token will be stored in the QUAY_TOKEN environment variable.

echo -n 'Your quay.io username: ' \
 && read QUAY_USERNAME \
 && echo -n 'Your quay.io password: ' \
 && export QUAY_TOKEN=$(curl --silent -H "Content-Type: application/json" -XPOST https://quay.io/cnr/api/v1/users/login -d '{"user":{"username":"'"${QUAY_USERNAME}"'","password":"'"$(read -s PW && echo -n $PW)"'"}}' | sed -E 's/.*\"(basic .*)\".*/\1/')

Push the Operator Manifest Bundles to Quay.io

Push the Kiali operator manifest bundle

At this point, let’s assume there exists a directory containing the new operator’s manifest bundle content for use by OLM. This includes things such as the ClusterServiceVersion (CSV) files for all the operator’s versions (including the new one) as well as the CustomResourceDefinition (CRD) files, etc. Assuming this bundle exists in the directory “kiali-ossm”, the operator’s package name is “kiali-ossm” (which must not conflict with any other operator package name — this is why we removed the original OperatorSources in a previous step), and the new operator manifest bundle version is “1.0.6” (which may or may not correspond to the operator’s version), we can upload this to quay.io via:

OPERATOR_DIR=kiali-ossm/
PACKAGE_NAME=kiali-ossm
PACKAGE_VERSION=1.0.6
curl --silent -H "Content-Type: application/json" -H "Authorization: ${QUAY_TOKEN}" -XPOST "https://quay.io/cnr/api/v1/packages/${QUAY_USERNAME}/${PACKAGE_NAME}" -d '{"release":"'"${PACKAGE_VERSION}"'","media_type":"helm","blob":"'"$(tar cz ${OPERATOR_DIR} | base64 -w 0 | iconv -t utf-8)"'"}'

Push the Other Operators’ Manifest Bundles

Because we removed the “redhat-operators” earlier (so our new Kiali operator package did not clash with the existing published package), this means we also removed the operator manifest bundles for Service Mesh, Jaeger, and ElasticSearch, too. Thus, we will need to publish our own copies of the Service Mesh, Jaeger, and ElasticSearch manifest bundles in the same way to the same place as we just published the Kiali operator manifest bundle above. We need to do this because we will need to install Service Mesh and its dependencies when we want to test the Kiali operator (which, as explained earlier, is a dependency of the Service Mesh operator, too). The script code below will do this. Note that this requires jq to be installed. If you have an operator you want to test and it is a standalone operator (i.e. not a dependency of another operator) you probably won’t need to do this step. But in the Kiali use-case, we need to do this. Notice we will pull down the current bundles for the ElasticSearch, Jaeger, and Service Mesh operators found in the “redhat-operators” quay.io app registry. Because we only need to test the current versions of those operators, we can simply upload a copy of the existing bundles into our own quay.io app registry.

export RH_PACKAGE_NAMESPACE="redhat-operators"
for op in elasticsearch-operator jaeger-product servicemeshoperator
do
  export op
  OP_RELEASE="$(curl --silent -H "Authorization: ${QUAY_TOKEN}" "https://quay.io/cnr/api/v1/packages?namespace=${RH_PACKAGE_NAMESPACE}" | jq '.[] | select(.name == $ENV.RH_PACKAGE_NAMESPACE + "/" + $ENV.op) | .default' -r)"
  OP_DIGEST="$(curl --silent -H "Authorization: ${QUAY_TOKEN}" "https://quay.io/cnr/api/v1/packages/${RH_PACKAGE_NAMESPACE}/${op}/${OP_RELEASE}" | jq '.[0].content.digest' -r)"
  OP_MANIFEST_TARBALL="/tmp/${RH_PACKAGE_NAMESPACE}-${op}-${OP_RELEASE}.tar.gz"
  echo -n "Retrieving ${OP_MANIFEST_TARBALL} ... "
  curl --silent -H "Authorization: ${QUAY_TOKEN}" "https://quay.io/cnr/api/v1/packages/${RH_PACKAGE_NAMESPACE}/${op}/blobs/sha256/${OP_DIGEST}" -o "$OP_MANIFEST_TARBALL"
  echo "Done."
  echo -n "Uploading ${OP_MANIFEST_TARBALL} ... "
  curl --silent -H "Content-Type: application/json" -H "Authorization: ${QUAY_TOKEN}" -XPOST "https://quay.io/cnr/api/v1/packages/${QUAY_USERNAME}/${op}" -d '{"release":"'"${OP_RELEASE}"'","media_type":"helm","blob":"'"$(cat ${OP_MANIFEST_TARBALL} | base64 -w 0 | iconv -t utf-8)"'"}'
done

Once these operator manifest bundles have been uploaded, your Quay.io account will have its own copies of the operator manifests in your account’s “Applications” tab. The first time you do this, the applications will have a lock icon indicating “private” access only — you will need to change them all to allow for public access through the quay.io browser interface.

Create A New OperatorSource

We now want to tell OpenShift where to get the new operator manifest bundles so it can populate the Catalog with the details of the operators you want to install and test. We do this by creating an OperatorSource:

cat <<EOF | oc apply -f -
apiVersion: operators.coreos.com/v1
kind: OperatorSource
metadata:
 name: ${QUAY_USERNAME}-operators
 namespace: openshift-marketplace
spec:
 type: appregistry
 endpoint: https://quay.io/cnr
 registryNamespace: ${QUAY_USERNAME}
 displayName: "${QUAY_USERNAME}'s Operators"
 publisher: "${QUAY_USERNAME}"
EOF

You can check to make sure this was processed correctly by looking at the results of:

oc get operatorsource ${QUAY_USERNAME}-operators -n openshift-marketplace

Shortcut — Install via the OpenShift UI Console

At this point, the operators are ready to be installed. You can do so using the OpenShift UI in your browser. However, if you would like to automate this, the following instructions show how to do this manually via ‘oc’ commands.

Create CatalogSourceConfig

Create the CatalogSourceConfig containing all the packages you want to install. In this case, we are installing all of Service Mesh via OLM, so we need all the packages listed (not just Kiali, but also Service Mesh, ElasticSearch, and Jaeger):

ALL_PACKAGES="${PACKAGE_NAME},elasticsearch-operator,jaeger-product,servicemeshoperator"
KIALI_OPERATOR_NAMESPACE="openshift-operators"
cat <<EOF | oc apply -f -
apiVersion: operators.coreos.com/v1
kind: CatalogSourceConfig
metadata:
 name: kiali
 namespace: openshift-marketplace
spec:
 targetNamespace: ${KIALI_OPERATOR_NAMESPACE}
 packages: ${ALL_PACKAGES}
EOF

The Optional OperatorGroup

You do not need to do this if your CatalogSourceConfig has a targetNamespace of “openshift-operators” which, in the above case, we are (see KIALI_OPERATOR_NAMESPACE). In this case, the operator will be considered with an “InstallMode” of “AllNamespaces.”

However, if you want your operator to watch its own namespace (“InstallMode” of “OwnNamespace”), then create an OperatorGroup in the namespace where the operator is to be installed (i.e. in the namespace where the Subscription will be) and set the targetNamespace as the same namespace where the operator is to be installed. Note that targetNamespace is the namespace that will be watched by the operator and can be different from the namespace where the OperatorGroup is (in that case, this would be “InstallMode” of “SingleNamespace”). We don’t need to do this with the current Kiali use-case (we want “InstallMode” of “AllNamespaces”), but one example of an OperatorGroup would be:

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
 name: kiali
 namespace: the-op-namespace
spec:
 targetNamespaces:
 - the-op-namespace

Create Subscription to the Operators

Now create subscriptions for the operators you want to install. The namespace where the Subscription is created is the namespace where the operator is installed.

Because we want to install Service Mesh with all of its components (along with Kiali) we can do this via:

cat <<EOF | oc apply -f -
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
 name: kiali
 namespace: ${KIALI_OPERATOR_NAMESPACE}
spec:
 channel: stable
 installPlanApproval: Automatic
 name: ${PACKAGE_NAME}
 source: kiali
 sourceNamespace: ${KIALI_OPERATOR_NAMESPACE}
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: elasticsearch-operator
  namespace: openshift-operators
spec:
  channel: preview
  installPlanApproval: Automatic
  name: elasticsearch-operator
  source: kiali
  sourceNamespace: ${KIALI_OPERATOR_NAMESPACE}
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: jaeger-product
  namespace: openshift-operators
spec:
  channel: stable
  installPlanApproval: Automatic
  name: jaeger-product
  source: kiali
  sourceNamespace: ${KIALI_OPERATOR_NAMESPACE}
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: servicemeshoperator
  namespace: openshift-operators
spec:
  channel: '1.0'
  installPlanApproval: Automatic
  name: servicemeshoperator
  source: kiali
  sourceNamespace: ${KIALI_OPERATOR_NAMESPACE}
EOF

Install the Operands

At this point, the operators are installed and running. Let’s now install the operands (i.e. the actual software we want ):

CONTROL_PLANE_NAMESPACE="istio-system"
oc create namespace ${CONTROL_PLANE_NAMESPACE}
oc create -n ${CONTROL_PLANE_NAMESPACE} -f https://raw.githubusercontent.com/Maistra/istio-operator/maistra-1.0/deploy/examples/maistra_v1_servicemeshcontrolplane_cr_minimal.yaml

That simply installs a minimal Service Mesh installation. Of course, your operands will vary and have their own CRs you should create. The Service Mesh minimal install does not create a Kiali CR and thus it doesn’t not install Kiali (Kiali is an optional component in Service Mesh). But let’s create a Kiali CR to install our new Kiali version:

cat <<EOF | oc apply -f -
apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
 name: kiali
 namespace: ${KIALI_OPERATOR_NAMESPACE}
annotations:
 ansible.operator-sdk/reconcile-period: "0s"
spec:
 deployment:
 namespace: ${CONTROL_PLANE_NAMESPACE}
 verbose_mode: "4"
EOF

At this point, we finally reached the stage where the new Kiali version has been installed.

Upgrade Operator

To upgrade to a new operator, upload a new manifest bundle (with a new higher package version and new CSV) to quay.io. Once that is done, you can wait for OLM to refresh (which happens once an hour) or you can “oc edit” or “oc patch” the OperatorSource and delete its status block which will immediate force a rescan of quay.io and pull in the latest operator details. You can remove the status block using “oc patch” like this:

oc patch operatorsource ${QUAY_USERNAME}-operators -n openshift-marketplace -p '[{"op":"replace","path":"/status","value":{}}]' --type json

Since the Subscriptions we created had an “installPlanApproval” of “Automatic”, the new operators will be installed automatically.

This is how you can test upgrading your operators directly through OLM without having to first publish the operator to the world.