IBM Spectrum Scale Container Native Storage Access

Installing Spectrum Scale for Persistent storage on Red Hat OpenShift Container Platform

This article is outdated: Created a newer article that is covering version CSNA 5.1.1.3

Me trying to draw a Scale container…

The first release of the containerized version of Spectrum Scale also called CNSA, was released in December, this very exciting and big shift in the Spectrum Scale product and how we can provide Persistent FileSystem for containerised environments.

So what is IBM Spectrum Scale Container Native Storage?

For people that don't know what Spectrum Scale is, we can start to describe what Spectrum Scale is, as mention on IBMs sites: IBM Spectrum® Scale is a clustered file system that provides concurrent access to a single file system or set of file systems from multiple nodes. The nodes can be SAN attached, network attached, a mixture of SAN attached, and network attached, or in a shared-nothing cluster configuration. This enables high performance access to this common set of data to support a scale-out solution or to provide a high availability platform.

So it’s a Cluster filesystem but it also has much more functions. Protocols, Replication, snapshot, DR, HDFS, Encryption, compression, FileAudit function, just to mention some of them.

So with the IBM Spectrum Scale Container Native Storage version, and the combination of CSI 2.1.0 allows for storage to be provisioned and accessed by container applications within Red Hat OpenShift Container Platform. What's new is that we now can containerize the Spectrum Scale cluster and run this on top of OpenShift.

Client clusters running on OCP don’t need any internal storage or internal filesystem, as it allows us to access storage/filesystem on any non-containerized Spectrum Scale storage cluster at the 5.1.0.1 level of code.

After talking to some customers that are using OCP or Kubernetes, they are still talking about persistent storage and that it can be difficult to manage, because most them have not just one cluster, but up to 2 digits, and they are also not running OCP on physical boxes but on top of a HyperVisor. So with the possibility to remote mount the filesystem, this will add flexibility in that it unties OpenShift from the storage hardware.

The separate Spectrum Scale Storage Cluster, this cluster could be Virtual, Physical or even an ESS (IBM Elastic Storage Server) As longs as the Scale cluster have a FileSystem and GUI. (5.1.0.1)

To get a picture of how an OCP environment could look like, I have drawn up an example with OCP (Worker Nodes) and Scale Cluster with the components.

By default, IBM Spectrum Scale Pods will be deployed to all OCP worker nodes. Optionally, you could select a subset of worker nodes to deploy IBM Spectrum Scale onto. Node Lables

There is also a podAntiAffinity and podAffinity to distribute the CNSA pods over different worker nodes and not placing the same pod on the same worker nodes.

• OCP Scale PODs
  - All OCP worker nodes have Scale Core pod (Can be decided with Node Selector)
  - 2 of X Worker Nodes holds Scale-PMCollector pod (Can be decided with Node Labels)
  - 1 of X Worker Nodes holds Scale-GUI pod.
  - 1 of X Worker Nodes holds Scale-core-operator pod.
• OCP Scale CSI PODs
  - All OCP Workers have the Scale CSI Pod.
  - 1 of X holds CSI Operator, Provisioner, CSI attacher.
• IBM Spectrum Scale Storage Cluster
  - Contains 3 Linux VMs (This could be 1 or 1000+, more realistic case, we have 3 to survive to able to have a quorum)
  - Scale Core.
  - Scale GUI on 2 of them. Only needed one of them.
  - Scale CES/Cluster Export Service (Not needed, only if you want Protocol support NFS, SMB, Object.)

So when we are deploying the CNSA, the CNSA will connect to the RestAPI/GUI on the storage cluster and do a handshake to allow the remote mounting of the filesystem. This API is also used when creating filesets for PVS, more on this later.

Container image listing for IBM Spectrum Scale

There is a Container image list over PODs and containers, this gives us some insight into what's included in the archive file downloaded from IBM.

Maybe in the future, we can download these images externally also.

PS: Remember to not distribute the Container Images, it is licensed software. check out Deployment Considerations for more details.

Some of CNSA pods need to download external container, example here is the IBM-Spectrum-scale-gui that needs Postgres and log* containers.

The IBM Spectrum Scale CSI is downloaded from the repository listed below.

This Part is going to be lengthy, so take your time and fill up on your favourite beverage. 🍹

Installing ReadHat OpenShift Cluster on vSphere.

“Instead of rewriting this, VirtuallyGhetto has covered it in detail here:

Using the new installation method for deploying OpenShift 4.5 on VMware Cloud on AWS

This will also work for an on-prem solution, as what I did.
What i used some time on was understanding the DNS record for API VIP and Ingress VIP. i created a new subdomain for cluster name and then api.<cluster-name>.<base-domain> and *.apps.<cluster-name>.<base-domain>.

When running the installation I used the Log-level: debug

./openshift-install create cluster — log-level debug

There are som Hardware and software requirements for Spectrum Scale.

Memory for my Worker Nodes was only 8GB, the minimum is 16GB. but I got the cluster up an running anyway. (but most likely I'm stretching it)
I'm also only running 1Gb network, 10Gb is minimum to get something that fly…

If you want to adjust the Memory and CPU, I found this article to drain and shutdown nodes.

https://www.redhat.com/en/blog/how-stop-and-start-production-openshift-cluster

Evacuate all or selected pods on one or more nodes:

oc adm drain <node1>
got error about i have local data and pods that couldent med moved.
- oc adm drain <node> --ignore-daemonsets --delete-local-data
oc adm uncordon ocp4-5hswh-worker-56pjx

From OCP menu — Nodes > worker-node(1) terminal and change to host binaries and run:
chroot /host
shutdown -h now

Before starting installing Spectrum Scale CNSA

Almost all of the steps below are in very good detail in IBM Knowledge Center. So if you want more details than what I cover in this article, I would recommend starting here:
The planning for IBM Spectrum® Scale Container Native Storage Access (CNSA)

I have written down how I did the installation on OCP 4.5 on vSphere with Spectrum scale Storage Cluster v5.1.0.1, this was done in the shortest and “easiest” way so I could start testing the product. In a production environment, I would strongly recommend reading through the knowledge center. And when a newer version has come out, check out whats new in KC link above.

In short, you will need: ✅

Storage Cluster

• A Spectrum Scale Storage Cluster. v5.1.0.1 or higher to support remote mount via the Operator.
  (Look out for an article in the near future on how to deploy Scale Storage Cluster with ansible soon.)
• Download Spectrum Scale packages for Storage Cluster
  (No CNSA Version yet.)
  Spectrum Scale Developer Edition Free Trial is available at this site: Developer Edition
• FYI: There is also an Ansible Role to install Spectrum Scale.
  https://github.com/IBM/ibm-spectrum-scale-install-infra

OCP Scale Cluster

• OCP Cluster with 4.5. 4.6. and now 4.7
• If you are on OCP Cluster with 4.6.6 or higher.
  If you try to install on 4.6.6. it will most likely fail as the worker nodes don’t have kernal-devel package..
  So to fix this, you need kmod MCO prior to bringing up CNSA on an OCP 4.6 cluster.
  (Updatet text 10.02.21)
• OpenShift 4.7 is only supported with CNSA 5.1.0.3
  CSI 2.2.0 is only compatible with CNSA 5.1.0.3.
  Check Out : https://www.ibm.com/docs/en/scalecontainernative?topic=planning-hardware-software-requirements
• Ensure that your DNS is configured properly so that the worker nodes can resolve the storage cluster nodes. If this is not possible, see the Host Aliases section in the custom resource.
• Access to the Internet for Container Repos, (if you don't have an internal one)
• All worker nodes must be able to communicate with each other via the host network.
  The Spectrum Scale pods will use the host network and not the Container Network Interface (CNI) overlay network
• IBM Spectrum Scale Container Native Storage Access tar archive file from Fix Central or Passport Advantage®.
• External container images required for IBM Spectrum Scale. I use a Linux host with Podman.
• If you don't have an Identity provider you can use HTPasswd, I installed this on a Centos Linux host, same VM as I used to install everything.

Installing CSNA

Preparation:

I'm using a Linux/Centos VM that has access to the OC Cluster and also have the OC Client installed, and I also have created myself a directory to work in when downloading and editing files.

Creating a new project

Create a new project using the oc new-project command.

Recommend using the namespace name: ibm-spectrum-scale-ns

oc new-project ibm-spectrum-scale-ns

Extracting the tar archive

Obtain the IBM Spectrum Scale Container Native Storage Access tar archive file from Fix Central or Passport Advantage.

This archive file contains:

• yaml files to assist with configuration of the Red Hat® OpenShift® Container Platform cluster
• container images
• yaml files required to deploy the operator

The .tgz filename is in the format Spectrum_Scale_Container_Native_Storage_Access-<version>-<arch>.tgz.

Obtain the correct one based on your architecture and extract it using the following command:

tar xvfz Spectrum_Scale_Container_Native_Storage_Access-*.tgz

Increase PIDS_LIMIT

Increase the pids_limit to 4096 on each of the worker nodes planned to run IBM Spectrum Scale. Without this change, the GPFS/Scale daemon will crash during I/O by running out of PID resources. These changes are managed by the Machine Config Operator (MCO).

Do the following steps:

1. Create the configuration yaml file to be applied. A sample is provided at spectrumscale/preinstall/increase_pid_mco.yaml from the .tgz file.

apiVersion: machineconfiguration.openshift.io/v1
kind: ContainerRuntimeConfig
metadata:
  name: increase-pid-limit
spec:
  machineConfigPoolSelector:
    matchLabels:
      pid-crio: config-pid
  containerRuntimeConfig:
    pidsLimit: 4096

2. Apply the YAML using the following command: oc create -f spectrumscale/preinstall/increase_pid_mco.yaml

containerruntimeconfig.machineconfiguration.openshift.io/increase-pid-limit created
oc get ContainerRuntimeConfig
NAME AGE
increase-pid-limit 30s

4. Modify the worker MachineConfigPool and apply the machineConfigPoolSelector created in the previous steps by setting the label: pid-crio: config-pid.

oc label machineconfigpool worker pid-crio=config-pid
oc label machineconfigpool worker pid-crio=config-pid
output: machineconfigpool.machineconfiguration.openshift.io/worker labeled

5. Check the status of the update using:

oc get MachineConfigPool
Output: 
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-f5396b88f9facb68c8f5b4c68bb33ff7   True      False      False      3              3                   3                     0                      4h35m
worker   rendered-worker-8fa1818e6a96d4c96fbe50c7c8de96f4   True      False      False      3              3                   3                     0                      4h35m

6. Verify the PID_LIMITS are increased on the worker nodes by issuing the following commands:

Get the worked nodes names with: oc get node | grep workers

# oc get node | grep worker
ocp4-q2qf4-worker-2nkfk   Ready    worker   4h25m   v1.18.3+616db59
ocp4-q2qf4-worker-c7j2q   Ready    worker   4h25m   v1.18.3+616db59
ocp4-q2qf4-worker-ljbfk   Ready    worker   4h26m   v1.18.3+616db59

# Replace <worker node> with the hostname of the worker, run one by one.

# oc debug node/ocp4-q2qf4-worker-2nkfk -- chroot /host crio-status config | grep pids_limit
# oc debug node/ocp4-q2qf4-worker-2nkfk -- chroot /host crio-status config | grep pids_limit
Starting pod/ocp4-q2qf4-worker-2nkfk-debug ...
To use host binaries, run `chroot /host`
    pids_limit = 4096
# oc debug node/ocp4-q2qf4-worker-c7j2q -- chroot /host crio-status config | grep pids_limit
Starting pod/ocp4-q2qf4-worker-c7j2q-debug ...
To use host binaries, run `chroot /host`
    pids_limit = 4096
# oc debug node/ocp4-q2qf4-worker-ljbfk -- chroot /host crio-status config | grep pids_limit
Starting pod/ocp4-q2qf4-worker-ljbfk-debug ...
To use host binaries, run `chroot /host`
    pids_limit = 4096

Container Images

During the deployment process, Red Hat OpenShift Container Platform needs to access the IBM Spectrum Scale container images. Since the images are not found in a public registry such as quay.io or registry.redhat.com, you will need to load the container images into a site-managed private container registry where the Red Hat OpenShift Container Platform cluster can access.

If you already have a private registry set up, you can load the images there, and skip to the Loading images topic and follow the procedure for your registry.

Configuring storage for the image registry in non-production clusters

You must configure storage for the Image Registry Operator. For non-production clusters, you can set the image registry to an empty directory. If you do so, all images are lost if you restart the registry.
For more information check out this link

1. To set the image registry storage to an empty directory:

oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch ‘{“spec”:{“storage”:{“emptyDir”:{}}}}’

output: config.imageregistry.operator.openshift.io/cluster patched

2. Ensure that your registry is set to “managed” to enable building and pushing of images.

oc edit configs.imageregistry/cluster

Then, change the line
managementState: Removed
to
managementState: Managed

3. Ensure the Red Hat OpenShift Container Platform Image Registry is available.

# oc get deployment image-registry -n openshift-image-registry
NAME READY UP-TO-DATE AVAILABLE AGE
image-registry 1/1 1 1 2m54

HTPasswd identity provider. 🔐

The following procedure creates a user and grants them permission to write to the integrated container image repository.

For more information, see Configuring an HTPasswd identity provider.

Note: Install httpd-tools on the host providing the authentication. The Red Hat OpenShift Container Platform cluster must be able to access this host.

On a host that can serve htpasswd.

1. yum install httpd-tools (Centos/RedHat)
2. Create the user/password entry into a file using the htpasswd command.
   change what the user and password if you need.

export USERNAME="user1"
export PASSWORD="P@ssword"
htpasswd -c -B -b  /etc/users.htpasswd $USERNAME $PASSWORD

2. Generate the HTPasswd Secret.
Create a secret called htpass-secret in the openshift-config namespace using the following command:

oc create secret generic htpass-secret \
> --from-file=htpasswd=/etc/users.htpasswd -n openshift-config
output: secret/htpass-secret created

3. Create a Custom Resource (CR) for the HTPasswd identity provider. A sample CR file is provided in the .tgz file at: spectrumscale/preinstall/optional_oauth_config.yaml.

Use the following command to set the name of the secret: htpass-secret into the CR file provided in the tar. The secret is created on the previous step.

#modify the secret name to: htpass-secret
sed -i 's/<secret-name>/htpass-secret/g' spectrumscale/preinstall/optional_oauth_config.yaml

Create the Custom Resource file.

# oc apply -f scale-binary/spectrumscale/preinstall/optional_oauth_config.yaml
output: Warning: oc apply should be used on resource created by either oc create --save-config or oc apply
oauth.config.openshift.io/cluster configured

Set permissions for the user

The user we created above must be granted permission to modify the image registry and to store the images.

export USERNAME="user1"
oc policy add-role-to-user registry-editor $USERNAME
oc policy add-role-to-user edit $USERNAME

Expose the image registry

To push the artifact into the internal registry, the registry must be exposed. For more information, see Exposing the Registry.

oc patch configs.imageregistry.operator.openshift.io/cluster \
--patch '{"spec":{"defaultRoute":true}}' --type=merge
output: config.imageregistry.operator.openshift.io/cluster patched

Loading Images 💾

Obtain the route to the container registry

The following command is used to obtain the route to the integrated Red Hat OpenShift Container Platform image registry. If you are using a site-managed container registry, obtain and note the route to your container registry.

The HOST variable that is referenced holds the route and is referenced in the later steps to push the container images to the destination container registry.

1. Switch to an Red Hat OpenShift Container Platform ADMIN account which has the permission to execute oc get routes:

oc login -u kubeadmin
oc login -u kubeadmin --certificate-authority=/etc/pki/ca-trust/source/anchors/a0806582.0

2. Set the HOST variable to the image registry route:

HOST=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')

3. Validate that HOST is not empty:

[root@ocp-admin ocp45-test2]#  echo $HOST
Output: default-route-openshift-image-registry.apps.ocp4.oslo.forum.com

Load container images and push to container registry ☁️

1. Log in to the user account created above that has access to the Red Hat OpenShift Container Platform Image Registry:

PS: I don't have a certificate so need to point to — certificate-authority

USERNAME="user1"
oc login -u $USERNAME --certificate-authority=/etc/pki/ca-trust/source/anchors/*

2. Import the IBM Spectrum Scale container images provided in the .tgz file using podman. If you don't have podman, just install it with yum install podman

For tag 5.1.0.1, run:

TAG="5.1.0.1"
 for file in `ls scale-binary/spectrumscale/*.tar`; do
     tarname=${file##*/}
     tagname=`echo $tarname | sed 's/.tar//g' | sed "s/-$TAG/:$TAG/g"`
     echo "-- Loading $file as $tagname"
     # If using docker, the load and tagging cannot be done in a single step
     podman load -i $file localhost/$tagname
 done

[root@ocp-admin ocp45-test2]# for file in `ls scale-binary/spectrumscale/*.tar`; do tarname=${file##*/}; tagname=`echo $tarname | sed ‘s/.tar//g’ | sed “s/-$TAG/:$TAG/g”`; echo “ — Loading $file as $tagname” podman load -i $file localhost/$tagname; done

 — Loading scale-binary/spectrumscale/scale-core-5.1.0.1.tar as scale-core:5.1.0.1 podman load -i scale-binary/spectrumscale/scale-core-5.1.0.1.tar localhost/scale-core:5.1.0.1
 — Loading scale-binary/spectrumscale/scale-core-operator-5.1.0.1.tar as scale-core-operator:5.1.0.1 podman load -i scale-binary/spectrumscale/scale-core-operator-5.1.0.1.tar localhost/scale-core-operator:5.1.0.1
 — Loading scale-binary/spectrumscale/scale-gui-5.1.0.1.tar as scale-gui:5.1.0.1 podman load -i scale-binary/spectrumscale/scale-gui-5.1.0.1.tar localhost/scale-gui:5.1.0.1
 — Loading scale-binary/spectrumscale/scale-monitor-5.1.0.1.tar as scale-monitor:5.1.0.1 podman load -i scale-binary/spectrumscale/scale-monitor-5.1.0.1.tar localhost/scale-monitor:5.1.0.1
 — Loading scale-binary/spectrumscale/scale-pmcollector-5.1.0.1.tar as scale-pmcollector:5.1.0.1 podman load -i scale-binary/spectrumscale/scale-pmcollector-5.1.0.1.tar localhost/scale-pmcollector:5.1.0.1

First time for me it didn't work, so check how you copy the command and line breaks.
So needed to run it again, and then it starting copying, but run out of space…
Tried again and now it’s copying from where it stopped.

for file in `ls scale-binary/spectrumscale/*.tar`; do
>      tarname=${file##*/}
>      tagname=`echo $tarname | sed 's/.tar//g' | sed "s/-$TAG/:$TAG/g"`
>      echo "-- Loading $file as $tagname"
>      # If using docker, the load and tagging cannot be done in a single step
>      podman load -i $file localhost/$tagname
>  done
-- Loading scale-binary/spectrumscale/scale-core-5.1.0.1.tar as scale-core:5.1.0.1
Getting image source signatures
Copying blob 1bba2f74c16a skipped: already exists
Copying blob ccf04fbd6e19 skipped: already exists
Copying blob 883b81808d71 skipped: already exists
Copying blob 1d663e76cbed skipped: already exists
Copying blob 937afad1d8c1 skipped: already exists
Copying blob 94591d9b792a skipped: already exists
Copying blob 75a5b2ad97c1 [--------------------------------------] 0.0b / 0.0b
Copying blob f3c860d7084b [--------------------------------------] 0.0b / 0.0b
Copying blob b7b591e3443f [--------------------------------------] 0.0b / 0.0b
Copying blob df71b995c887 [--------------------------------------] 0.0b / 0.0b
Copying blob fa83a7247878 [--------------------------------------] 0.0b / 0.0b
Copying config 76ea968516 done
Writing manifest to image destination
Storing signatures
Loaded image(s): localhost/scale-core:dev
-- Loading scale-binary/spectrumscale/scale-core-operator-5.1.0.1.tar as scale-core-operator:5.1.0.1
Getting image source signatures

3. Log in to the Red Hat OpenShift Container Platform integrated container registry via podman:

oc whoami -t | podman login -u $(oc whoami) --password-stdin --tls-verify=false $HOST

4. Tag and push the images into the Red Hat OpenShift Container Platform Image Registry:

TAG="5.1.0.1"
NAMESPACE="ibm-spectrum-scale-ns"
for file in `ls scale-binary/spectrumscale/*.tar`; do
    tarname=${file##*/}
    tagname=`echo $tarname | sed 's/.tar//g' | sed "s/-$TAG/:$TAG/g"`

    podman tag localhost/$tagname $HOST/$NAMESPACE/$tagname
    podman push $HOST/$NAMESPACE/$tagname --tls-verify=false
done

Output:
[root@ocp-admin]# TAG="5.1.0.1"
[root@ocp-admin]# NAMESPACE="ibm-spectrum-scale-ns"
[root@ocp-admin]# for file in `ls scale-binary/spectrumscale/*.tar`; do
>     tarname=${file##*/}
>     tagname=`echo $tarname | sed 's/.tar//g' | sed "s/-$TAG/:$TAG/g"`
>
>     podman tag localhost/$tagname $HOST/$NAMESPACE/$tagname
>     podman push $HOST/$NAMESPACE/$tagname --tls-verify=false
> done
Getting image source signatures
Copying blob 937afad1d8c1 done
Copying blob 1d663e76cbed done

Validate the container images 💾

The following steps are only applicable when using the Red Hat OpenShift Container Platform integrated container registry.

1. Log in to an administrator account:

oc login -u kubeadmin --certificate-authority=/etc/pki/ca-trust/source/anchors/a0806582.0

2. Ensure that the ImageStream contains the images:

for image in `oc get is -o custom-columns=NAME:.metadata.name --no-headers`; do
echo "---"
oc get is $image -o yaml | egrep "name:|dockerImageRepository"
done
Output: 
[root@ocp-admin]# for image in `oc get is -o custom-columns=NAME:.metadata.name --no-headers`; do
> echo "---"
> oc get is $image -o yaml | egrep "name:|dockerImageRepository"
> done
---
  name: scale-core
  dockerImageRepository: image-registry.openshift-image-registry.svc:5000/ibm-spectrum-scale-ns/scale-core
---
  name: scale-core-operator
  dockerImageRepository: image-registry.openshift-image-registry.svc:5000/ibm-spectrum-scale-ns/scale-core-operator
---
  name: scale-gui
  dockerImageRepository: image-registry.openshift-image-registry.svc:5000/ibm-spectrum-scale-ns/scale-gui
---
  name: scale-monitor
  dockerImageRepository: image-registry.openshift-image-registry.svc:5000/ibm-spectrum-scale-ns/scale-monitor
---
  name: scale-pmcollector
  dockerImageRepository: image-registry.openshift-image-registry.svc:5000/ibm-spectrum-scale-ns/scale-pmcollector

Configuring IBM Spectrum Scale Operator

Node Label — Node selectors

By default, IBM Spectrum Scale will be deployed to all worker nodes. Optionally, you could select a subset of worker nodes to deploy IBM Spectrum Scale onto.

You cannot select any master nodes. Node selectors can be added to the ScaleCluster custom resource file (spectrumscale/deploy/crds/ibm_v1_scalecluster_cr.yaml) to specify which nodes to deploy IBM Spectrum Scale. In the ScaleCluster CR, there is a node selector defined for worker nodes. In this case, the Operator deploys an IBM Spectrum Scale pod to nodes labelled as worker nodes.

Check out Node selectors

Operator deployment preparations

With the tar archive, we provide a set of .yaml files that are required for deployment of IBM Spectrum Scale.

Since both the namespace and the container image registry are defined by the users, the .yaml files are shipped with placeholders needing to be replaced with the actual values matching your environment before applying the .yaml files.

Note: Before applying the yaml files, ensure that you are in the correct namespace for IBM Spectrum Scale. It is preferred to explicitly specify the namespace -n ibm-spectrum-scale-ns in the commands.

Change to the directory where the tar archive was extracted and run the following steps:

1. Edit the spectrumscale/deploy/cluster_role_binding.yaml file to set the namespace: field to your namespace.

vi /spectrumscale/deploy/cluster_role_binding.yaml

Replace <namespace> with namespace ibm-spectrum-scale-ns:

name: ibm-spectrum-scale-operator
namespace: ibm-spectrum-scale-ns

2. Edit the spectrumscale/deploy/operator.yaml and modify the image: field to point to the container registry serving the scale-core-operator image.

For example, if your images are tagged 5.1.0.1 and the container registry route is image-registry.openshift-image-registry.svc:5000/ibm-spectrum-scale-ns the image values would look like:

image: image-registry.openshift-image-registry.svc:5000/ibm-spectrum-scale-ns/scale-core-operator:5.1.0.1

If not sure, check value with the following command:

for image in `oc get is -o custom-columns=NAME:.metadata.name --no-headers`; do
echo "---"
oc get is $image -o yaml | egrep "name:|dockerImageRepository"
done

3. Apply the following .yaml files to prepare deployment of the Operator:

oc create -f spectrumscale/deploy/service_account.yaml -n ibm-spectrum-scale-ns
 oc create -f spectrumscale/deploy/service_account_core.yaml -n ibm-spectrum-scale-ns
 oc create -f spectrumscale/deploy/role.yaml -n ibm-spectrum-scale-ns
 oc create -f spectrumscale/deploy/role_binding.yaml -n ibm-spectrum-scale-ns
 oc create -f spectrumscale/deploy/role_scale_core.yaml -n ibm-spectrum-scale-ns
 oc create -f spectrumscale/deploy/role_binding_scale_core.yaml -n ibm-spectrum-scale-ns
 oc create -f spectrumscale/deploy/scc.yaml -n ibm-spectrum-scale-ns
 oc create -f spectrumscale/deploy/cluster_role.yaml -n ibm-spectrum-scale-ns
 oc create -f spectrumscale/deploy/cluster_role_binding.yaml -n ibm-spectrum-scale-ns
 oc create -f spectrumscale/deploy/crds/ibm_v1_scalecluster_crd.yaml -n ibm-spectrum-scale-ns

Output:
[root@ocp-admin scale-binary]#  oc create -f spectrumscale/deploy/service_account_core.yaml -n ibm-spectrum-scale-ns
serviceaccount/ibm-spectrum-scale-core created
[root@ocp-admin scale-binary]# oc create -f spectrumscale/deploy/role.yaml -n ibm-spectrum-scale-ns
role.rbac.authorization.k8s.io/ibm-spectrum-scale-operator created
[root@ocp-admin scale-binary]# oc create -f spectrumscale/deploy/role_binding.yaml -n ibm-spectrum-scale-ns
rolebinding.rbac.authorization.k8s.io/ibm-spectrum-scale-operator created
[root@ocp-admin scale-binary]# oc create -f spectrumscale/deploy/role_scale_core.yaml -n ibm-spectrum-scale-ns
role.rbac.authorization.k8s.io/ibm-spectrum-scale-core created
[root@ocp-admin scale-binary]# oc create -f spectrumscale/deploy/role_binding_scale_core.yaml -n ibm-spectrum-scale-ns
rolebinding.rbac.authorization.k8s.io/ibm-spectrum-scale-core created
[root@ocp-admin scale-binary]#
[root@ocp-admin scale-binary]# oc create -f spectrumscale/deploy/scc.yaml -n ibm-spectrum-scale-ns
securitycontextconstraints.security.openshift.io/ibm-spectrum-scale-restricted created
[root@ocp-admin scale-binary]#
[root@ocp-admin scale-binary]# oc create -f spectrumscale/deploy/cluster_role.yaml -n ibm-spectrum-scale-ns
clusterrole.rbac.authorization.k8s.io/ibm-spectrum-scale-operator created
[root@ocp-admin scale-binary]#
[root@ocp-admin scale-binary]# oc create -f spectrumscale/deploy/cluster_role_binding.yaml -n ibm-spectrum-scale-ns
clusterrolebinding.rbac.authorization.k8s.io/ibm-spectrum-scale-operator created
[root@ocp-admin scale-binary]#
[root@ocp-admin scale-binary]#  oc create -f spectrumscale/deploy/crds/ibm_v1_scalecluster_crd.yaml -n ibm-spectrum-scale-ns
customresourcedefinition.apiextensions.k8s.io/scaleclusters.scale.ibm.com created

IBM Spectrum Scale configuration parameters

Default configuration parameters

I would recommend staying with the original parameters. if you want to change these check out here: IBM Spectrum Scale configuration parameters

Override default configuration parameters (optional step)

To allow changes to the default parameters specified above, a configMap file is provided in spectrumscale/deploy/scale-profile.yaml. Use this file to modify the values before creating the cluster for the first time. If this step is skipped, the Operator will create the configmap ibm-spectrum-scale-core-profile using the defaults indicated above.

1. Edit the configMap and modify the properties: vi spectrumscale/deploy/scale-profile.yaml
2. Create the configMap:
   oc create -f spectrumscale/deploy/scale-profile.yaml -n ibm-spectrum-scale-ns
oc get configmap -n ibm-spectrum-scale-ns | grep profile

Create GUI user on the storage cluster with ContainerOperator role

The steps described in this section must be performed on the storage cluster.

SSH into Spectrum Scale Storage GUI Node.

Remember that Scale version need to be 5.1.0.1. check with mmdiag:

[root@scale-dev-01 ~]# mmdiag
Current GPFS build: 5.1.0.1

Create CNSA Operator User and Group

Verify and create the GUI user group ContainerOperator and the user on the IBM Spectrum Scale storage cluster. This user will be used by the operator to configure the remote mount of the storage cluster file systems.

Issue the following commands in the shell of the GUI node of the storage cluster:

1. Check if the IBM Spectrum Scale GUI user group ContainerOperator exists by issuing the following command:
   /usr/lpp/mmfs/gui/cli/lsusergrp ContainerOperator

2. . If the GUI user group ContainerOperator does not exist, create it using the following command:/usr/lpp/mmfs/gui/cli/mkusergrp ContainerOperator --role containeroperator

3.Check if the IBM Spectrum Scale GUI user exists within the group ContainerOperator by issuing the following command:
/usr/lpp/mmfs/gui/cli/lsuser | grep ContainerOperator

4. If no user exists in the group ContainerOperator, create an IBM Spectrum Scale GUI user in the ContainerOperator group by issuing the following command:

/usr/lpp/mmfs/gui/cli/mkuser cnss_storage_gui_user -p cnss_storage_gui_password -g ContainerOperator
Output: 
EFSSG0019I The user cnss_storage_gui_user has been successfully created.
EFSSG1000I The command completed successfully.

Create secrets

Create a secret on the Red Hat OpenShift cluster to hold the username and password for the IBM Spectrum Scale Storage cluster GUI user and password.

This secret will be used by the Operator to communicate to the storage cluster when performing configuration for remote mount.

Storage cluster GUI user secret

The user name and password specified in this topic must match the GUI user that was created on the storage cluster in the steps above.

To create the storage cluster GUI user secret named cnsa-remote-mount-secret, issue the following command:

oc create secret generic cnsa-remote-mount-secret --from-literal=username='cnss_storage_gui_user' \
   --from-literal=password='cnss_storage_gui_password' -n ibm-spectrum-scale-ns

Custom Resource

Edit ScaleCluster Custom Resource (CR)

Before deploying the ScaleCluster Custom Resource (CR), it is required to make changes to the custom resource .yaml file.

The values provided in the CR are used for IBM Spectrum Scale cluster creation and must be valid inputs. Subsequent changes made to the CR after the creation of the IBM Spectrum Scale cluster are not supported.

Edit the spectrumscale/deploy/crds/ibm_v1_scalecluster_cr.yaml file and modify the following sections to match your environment.

So what I changed was not much, but I checked that the required parameters are set.

• NodeSelector Section:
  was the default. all nodes “Worker”.
• Container Images Section:
  if you have used default you should not be necessary to change this.
• Remote Mount Section:
  To configure a remote storage cluster for remote mount, fill out the filesystem and remoteCluster sections in the custom resource.

In my Storage Cluster, my filesystem is name gpfs01. you can check this with the following command: mmlsfs all

So I changed the StorageFS to gpfs01 and checked that the storageCluster the name was same under remote cluster section

filesystems:
 - name: "fs1"
   remoteMount:
     storageCluster: "storageCluster1"
     storageFs: "gpfs01"
   mountPoint: "/mnt/fs1"

• name : Provide a name to the file system that will be created upon CNSA deployment. This name must follow guidelines defined by DNS Label Names
• remoteMount:
  - storageCluster : Name to reference the remoteCluster definition in the CR file and should match the remoteCluster.name field below.
  - storageFs: Name of the file system on the storage cluster. You can get this information by issuing the mmlsfs all command on the storage cluster.
• mountPoint : This is the mount point where the remote file system is mounted upon CNSA cluster creation. This mountpoint must be under /mnt

RemoteCluster Section:
I changed the host to my Storage Cluster GUI.

remoteClusters:
— name: storageCluster1
  gui:
    host: “scale-dev-01.oslo.forum.com”
    secretName: “cnsa-remote-mount-secret”
    insecureSkipVerify: true
 #  contactNodes:
 #    — storageclusternode1
 #    — storageclusternode2

• name — This is the name used to identify the remote Storage Cluster, and is also referenced in file systems
• gui
  host : hostname for the GUI endpoint on the storage cluster.
  secretName : This is the name of the kubernetes secret created during the storage cluster configuration.
  insecureSkipVerify : true is the only option that is supported at this time.
• contactNodes (optional) — provide a list of storage nodes to be used as the contact nodes list. If not specified, the operator will use 3 nodes from the storage cluster.

Have attached an example of my scalecluster_cr.yml below, have removed the callhome information.

spectrumscale/deploy/crds/ibm_v1_scalecluster_cr.yaml

Create the IBM Spectrum Scale Container Native Storage Access (CNSA) Cluster

Once the custom resource is applied, the IBM Spectrum Scale Operator takes over and creates all the pods that make up an IBM Spectrum Scale Container Native Storage Access cluster. The Operator also sets up a remote cluster relationship with the storage cluster and mounts the file system on all worker nodes that run IBM Spectrum Scale. This operation can take some time.

1. Create the Operator
   oc create -f spectrumscale/deploy/operator.yaml -n ibm-spectrum-scale-ns

 Output: deployment.apps/ibm-spectrum-scale-operator created

2. Verify the operator pod is in the “Running” state
oc get pods -n ibm-spectrum-scale-ns

Output: ibm-spectrum-scale-operator-7c86d75f7d-rntqs 1/1 Running 0 61s

3. Create the Custom Resource
oc create -f spectrumscale/deploy/crds/ibm_v1_scalecluster_cr.yaml -n ibm-spectrum-scale-ns

Verify the IBM Spectrum Scale Container Native Storage Access (CNSA) Cluster ☑️

1. Verify that the Operator creates the ScaleCluster Custom Resource by checking pods and the Operator logs:

oc get pods -n ibm-spectrum-scale-ns
oc logs <scale-operator-pod> -n ibm-spectrum-scale-ns -f
oc get pods -n ibm-spectrum-scale-ns
ibm-spectrum-scale-core-4xlkx                  0/1     Init:0/2   0          79s
ibm-spectrum-scale-core-bv5s2                  0/1     Init:0/2   0          79s
ibm-spectrum-scale-core-jnbrz                  0/1     Init:0/2   0          79s
ibm-spectrum-scale-gui-0                       0/9     Init:0/1   0          78s
ibm-spectrum-scale-operator-7c86d75f7d-rntqs   1/1     Running    0          11m
ibm-spectrum-scale-pmcollector-0               0/2     Init:0/1   0          78s

Want to check what's happening, you can check the logs.

oc logs ibm-spectrum-scale-operator-7c86d75f7d-rntqs -n ibm-spectrum-scale-ns -f

For me, it took 20min before all pods were up an running.

oc get pods -n ibm-spectrum-scale-ns
NAME                                           READY   STATUS    RESTARTS  
ibm-spectrum-scale-core-4xlkx                  1/1     Running   0          
ibm-spectrum-scale-core-bv5s2                  1/1     Running   0          
ibm-spectrum-scale-core-jnbrz                  1/1     Running   0          
ibm-spectrum-scale-gui-0                       9/9     Running   0          
ibm-spectrum-scale-operator-7c86d75f7d-rntqs   1/1     Running   0          
ibm-spectrum-scale-pmcollector-0               2/2     Running   0          
ibm-spectrum-scale-pmcollector-1               2/2     Running   0          

Note: The resulting cluster will contain one operator pod, one GUI pod, two pmcollector pods, and one core pod per worker node that has the specified nodeSelector label.

Run with -o wide to get the nodes where the pods are running.

oc  get pods -o wide -n ibm-spectrum-scale-ns

2. Verify the Spectrum Scale cluster has been correctly created:

remember to change the last digits in ibm-spectrum-scale-core-xxxx

oc exec ibm-spectrum-scale-core-4xlkx -n ibm-spectrum-scale-ns -- mmlscluster
GPFS cluster information
========================
  GPFS cluster name:         ibm-spectrum-scale.ibm-spectrum-scale-ns.ocp4.oslo.forum.com
  GPFS cluster id:           15871122664024915663
  GPFS UID domain:           ibm-spectrum-scale.ibm-spectrum-scale-ns.ocp4.oslo.forum.com
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
  Repository type:           CCR
Node  Daemon node name         IP address   Admin node name          Designation
----------------------------------------------------------------------------------
   1   ocp4-q2qf4-worker-2nkfk  10.3.3.242  ocp4-q2qf4-worker-2nkfk  quorum-manager-perfmon
   2   ocp4-q2qf4-worker-c7j2q  10.3.3.241  ocp4-q2qf4-worker-c7j2q  quorum-manager-perfmon
   3   ocp4-q2qf4-worker-ljbfk  10.3.3.240  ocp4-q2qf4-worker-ljbfk  quorum-manager-perfmon

3. Verify that the storage cluster has been configured:

oc exec ibm-spectrum-scale-core-4xlkx -n ibm-spectrum-scale-ns -- mmremotecluster show all
Cluster name:    scale-dev.oslo.forum.com
Contact nodes:   scale-dev-01.oslo.forum.com,scale-dev-02.oslo.forum.com,scale-dev-03.oslo.forum.com
SHA digest:      05bee677a955f25b305fc013872176510f01bccd8645571f4f5e730259c40a8e
File systems:    fs1 (gpfs01)

4. Verify the storage cluster filesystem has been configured:

oc exec ibm-spectrum-scale-core-4xlkx -n ibm-spectrum-scale-ns -- mmremotefs show
Local Name  Remote Name  Cluster name       Mount Point        Mount Options    Automount  Drive  Priority
fs1         gpfs01       scale-dev.oslo.forum.com /mnt/fs1           rw               yes          -        0

5. Verify the storage cluster filesystem has been remotely mounted:
By checking the with the mmlsmount and the filesystem name

oc exec ibm-spectrum-scale-core-4xlkx -n ibm-spectrum-scale-ns -- mmlsmount fs1 -L
File system fs1 (scale-dev.oslo.forum.com:gpfs01) is mounted on 6 nodes:
  10.3.3.161     scale-dev-02.oslo.forum.com scale-dev.oslo.forum.com
  10.3.3.160     scale-dev-01.oslo.forum.com scale-dev.oslo.forum.com
  10.3.3.162     scale-dev-03.oslo.forum.com scale-dev.oslo.forum.com
  10.3.3.241     ocp4-q2qf4-worker-c7j2q   ibm-spectrum-scale.ibm-spectrum-scale-ns.ocp4.oslo.forum.com
  10.3.3.242     ocp4-q2qf4-worker-2nkfk   ibm-spectrum-scale.ibm-spectrum-scale-ns.ocp4.oslo.forum.com
  10.3.3.240     ocp4-q2qf4-worker-ljbfk   ibm-spectrum-scale.ibm-spectrum-scale-ns.ocp4.oslo.forum.com

Deploying IBM Spectrum Scale Container Storage Interface (CSI)

This step could also be done from the OpenShift GUI, but you need to do the prereqs.

Complete the following steps for storage cluster, CNSA cluster, and custom resource file for CSI:

1. Ensure the following pre-installation tasks on the storage cluster are completed.
   Note: this is done on Storage Cluster GUI node

• Create an IBM Spectrum Scale user group “CsiAdmin” using the following command:

/usr/lpp/mmfs/gui/cli/mkusergrp CsiAdmin --role csiadmin 

• Create an IBM Spectrum Scale user in the “CsiAdmin” group. This user must be used on IBM Spectrum Scale Container Storage Interface driver configuration. Issue this command on the GUI node to create the user.

/usr/lpp/mmfs/gui/cli/mkuser csi-storage-gui-user -p  csi-storage-gui-password -g CsiAdmin

• Perfileset quota on the file systems to be used by IBM Spectrum Scale Container Storage Interface driver is set to No

mmlsfs gpfs01 --perfileset-quota
flag                value                    description
------------------- ------------------------ -------------------------------
--perfileset-quota No                       Per-fileset quota enforcement

• Fileset Quota is enabled in the file systems to be used by IBM Spectrum Scale Container Storage Interface driver

mmchfs gpfs01 -Q yes

• Verify that quota is enabled.

mmlsfs gpfs01 -Q

flag                value                    description
------------------- ------------------------ -----------------------------------
-Q                 user;group;fileset       Quotas accounting enabled
                    user;group;fileset       Quotas enforced
                    none                     Default quotas enabled

• Enable quota for root user by issuing the following command:

mmchconfig enforceFilesetQuotaOnRoot=yes -i

• Ensure that the controlSetxattrImmutableSELinux parameter is set to “yes” by issuing the following command:

mmchconfig controlSetxattrImmutableSELinux=yes -i

• Enable filesetdf of the file system by using the following command:

mmchfs gpfs01 --filesetdf

2. Ensure the following pre-installation tasks on the CNSA cluster are completed.

Apply the IBM Spectrum Scale Container Storage Interface driver node label to the worker nodes where CNSA pods are running. By default, these will be the Red Hat OpenShift Container Platform worker nodes.

• For example, to label all Red Hat OpenShift Container Platform worker nodes to use node label “scale=true”, issue the following command:

oc label nodes -l node-role.kubernetes.io/worker= scale=true
output: 
node/ocp4-q2qf4-worker-2nkfk labeled
node/ocp4-q2qf4-worker-c7j2q labeled
node/ocp4-q2qf4-worker-ljbfk labeled

• Create a CNSA GUI user for CSI:

oc exec -c liberty ibm-spectrum-scale-gui-0 -- /usr/lpp/mmfs/gui/cli/mkuser csi-cnsa-gui-user -p csi-cnsa-gui-password -g CsiAdmin
EFSSG0019I The user csi-cnsa-gui-user has been successfully created.
EFSSG1000I The command completed successfully.

3. Gather information necessary for Configuring Custom Resource (CR) for IBM Spectrum Scale Container Storage Interface (CSI) driver from the CNSA cluster.

• The GUI host for CNSA follows the format of ibm-spectrum-scale-gui.<namespace>. In this case, our GUI host would be named ibm-spectrum-scale-gui.ibm-spectrum-scale-ns.
• Identify node mapping:
• You can use the following REST API call to validate the IBM Spectrum Scale node names under the mount:nodesMountedReadWrite section.
• From the infrastructure node, issue the following command against an existing IBM Spectrum Scale pod:

oc exec ibm-spectrum-scale-core-4xlkx -- curl -k https://ibm-spectrum-scale-gui.ibm-spectrum-scale-ns/scalemgmt/v2/filesystems/fs1?fields=mount -u "csi-cnsa-gui-user:csi-cnsa-gui-password"

• Retrieve OpenShift pod by issuing the oc get pods -n ibm-spectrum-scale-ns command

Output: 
name” : “fs1”,
nodesMountedReadWrite” : [ “ocp4-q2qf4-worker-2nkfk”, “ocp4-q2qf4-worker-c7j2q”, “ocp4-q2qf4-worker-ljbfk”, “scale-dev-01.oslo.forum.com”, “scale-dev-02.oslo.forum.com”, “scale-dev-03.oslo.forum.com” ],

Retrieve and note down the CNSA cluster ID:

From the infrastructure node, issue the following command against an existing IBM Spectrum Scale pod:

oc exec ibm-spectrum-scale-core-4xlkx -- curl -s -k https://ibm-spectrum-scale-gui.ibm-spectrum-scale-ns/scalemgmt/v2/cluster -u "csi-cnsa-gui-user:csi-cnsa-gui-password" | grep clusterId

• Retrieve and note down the CNSA cluster ID:

“clusterId” : 15871122664024915663,

• Retrieve and note down the storage cluster ID:
  My Storage Cluster is scale-dev-01

curl -s -k https://scale-dev-01/scalemgmt/v2/cluster -u "csi-storage-gui-user:csi-storage-gui-password" | grep clusterId
Output: clusterId" : 14324309008622167364

The storage GUI host is the same host used in the Remote Mount section of CNSA Custom Resource

remoteClusters:
  - name: storageCluster1
    gui:
      host: "example-gui.com" <----

Check your file: spectrumscale/deploy/crds/ibm_v1_scalecluster_cr.yaml

remoteClusters:
  - name: storageCluster1
    gui:
 host: “scale-dev-01.domain.com”

Install CSI driver

The CSI have the following functional support

• Static provisioning: Ability to use existing directories and filesets as persistent volumes
• Lightweight dynamic provisioning: Ability to create directory-based volumes dynamically
• Fileset-based dynamic provisioning: Ability to create fileset based volumes dynamically
• Supported volume access modes: RWX (readWriteMany) and RWO (ReadWriteOnce)

Phase 1: Deploying the Operator

1. Create the namespace for the IBM Spectrum Scale CSI driver
   oc new-project ibm-spectrum-scale-csi-driver
2. Deploy the Operator.
   oc create -f https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-csi/v2.1.0/generated/installer/ibm-spectrum-scale-csi-operator.yaml -n ibm-spectrum-scale-csi-driver

In the KC the URL had an /IBM®/ this should only be /IBM/

https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-csi/master/generated/installer/ibm-spectrum-scale-csi-operator.yaml

1. Verify that the Operator is deployed, and the Operator pod is in running state.

oc get pod,deployment -n ibm-spectrum-scale-csi-driver

NAME                                                   READY   STATUS    RESTARTS   AGE
pod/ibm-spectrum-scale-csi-operator-56955949c4-tvv6q   1/1     Running   0          28s
NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ibm-spectrum-scale-csi-operator   1/1     1            1           29s

Phase 2: Deploying IBM Spectrum Scale Container Storage Interface driver — CSI

1. Create a secret for the CNSA GUI user created in step 2 of Performing pre-installation tasks for CSI Operator deployment by issuing the following command:
   oc create secret generic secret-cnsa --from-literal=username=csi-cnsa-gui-user --from-literal=password=csi-cnsa-gui-password -n ibm-spectrum-scale-csi-driver
2. Create a secret for the GUI user on the storage cluster created in Storage cluster configuration.
   oc create secret generic secret-storage --from-literal=username=csi-storage-gui-user --from-literal=password=csi-storage-gui-password -n ibm-spectrum-scale-csi-driver

oc label secret secret-storage product=ibm-spectrum-scale-csi
3. Download the sample custom resource file on your cluster.
   curl -O https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-csi/v2.1.0/operator/deploy/crds/csiscaleoperators.csi.ibm.com_cr.yaml
4. Modify the parameters in the file to suit your environment. For more information, see Configuring Custom Resource (CR) for CSI driver. Once finished modifying the CR, return to this page and continue to step 5.

I copied the table below, so that i could fill in the details after each of the steps.

Configuring Custom Resource (CR) for CSI driver

There was som jumping between multiple pages to be sure that i have created the file correctly.

After also checking this: csi_remote_cluster_support

so after after editing: csiscaleoperators.csi.ibm.com_cr.yaml
I ended up with this file. (Shorten down)

5. Create the CR file.

oc create -f csiscaleoperators.csi.ibm.com_cr.yaml -n ibm-spectrum-scale-csi-driver

Output: csiscaleoperator.csi.ibm.com/ibm-spectrum-scale-csi created

6. Verify that IBM Spectrum Scale Container Storage Interface driver is installed, Operator and driver resources are ready, and pods are in running state.

oc get pod,daemonset,statefulset -n ibm-spectrum-scale-csi-driver

My Pod crashed, so I needed to check the logs.

oc logs ibm-spectrum-scale-csi-mbdqz ibm-spectrum-scale-cs

I have typed in the wrong cluster ID (or more correctly I copied the ID but the last digit was missing…) As the error was saying: Failed to Initialize scale CSI Driver: ID doesnt match the cluster

rest_v2.go:586] rest_v2 doHTTP. endpoint: https://scale-dev-01.oslo.forum.com:443/scalemgmt/v2/cluster, method: GET, param: <nil>
I1217 20:51:52.048073 1 http_utils.go:74] http_utils HttpExecuteUserAuth. type: GET, url: https://scale-dev-01.oslo.forum.com:443/scalemgmt/v2/cluster, user: csi-storage-gui-user
I1217 20:51:52.198898 1 http_utils.go:44] http_utils UnmarshalResponse. response: &{0x5a5c00 0xc00001e680 0x6d56a0}
I1217 20:51:52.199043 1 rest_v2.go:43] rest_v2 isStatusOK. statusCode: 200
E1217 20:51:52.199064 1 gpfs.go:202] Cluster ID 1432430900862216736 from scale config doesnt match the ID from cluster 14324309008622167364.
E1217 20:51:52.199073 1 gpfs.go:143] Error in plugin initialization: Cluster ID doesnt match the cluster
F1217 20:51:52.199083 1 main.go:65] Failed to initialize Scale CSI Driver: Cluster ID doesnt match the cluster
[root@ocp-admin scale-binary]#

• So deleted the operator and tried again.

oc delete -f csiscaleoperators.csi.ibm.com_cr.yaml
Output: csiscaleoperator.csi.ibm.com “ibm-spectrum-scale-csi” deleted

• Create the CR again.
  oc create -f csiscaleoperators.csi.ibm.com_cr.yaml -n ibm-spectrum-scale-csi-driver
• Checked again, and the pods were starting.

# oc get pod,daemonset,statefulset -n ibm-spectrum-scale-csi-driver
NAME                                                   READY   STATUS    RESTARTS   AGE
pod/ibm-spectrum-scale-csi-attacher-0                  1/1     Running   0          9m53s
pod/ibm-spectrum-scale-csi-hvg6n                       2/2     Running   0          9m35s
pod/ibm-spectrum-scale-csi-jtj2x                       2/2     Running   0          9m35s
pod/ibm-spectrum-scale-csi-operator-56955949c4-2z77d   1/1     Running   0          7h9m
pod/ibm-spectrum-scale-csi-provisioner-0               1/1     Running   0          9m44s
pod/ibm-spectrum-scale-csi-rd6fd                       2/2     Running   0          9m35s
NAME                                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/ibm-spectrum-scale-csi   3         3         3       3            3           scale=true      9m35s
NAME                                                  READY   AGE
statefulset.apps/ibm-spectrum-scale-csi-attacher      1/1     9m53s
statefulset.apps/ibm-spectrum-scale-csi-provisioner   1/1     9m44s

Are you still here? 😃 we are almost there now!

Create Storage Class and PVC for Spectrum Scale.

The CSI for CSNA have the following functional support

• Static provisioning: Ability to use existing directories and filesets as persistent volumes
• Lightweight dynamic provisioning: Ability to create directory-based volumes dynamically
• Fileset-based dynamic provisioning: Ability to create fileset based volumes dynamically
• Supported volume access modes: RWX (readWriteMany) and RWO (ReadWriteOnce)

Fileset: you are maybe asking yourself, what is fileset and what is different between independent and dependent fileset.

From the KC: GPFS utilizes a file system object called a fileset. A fileset is a subtree of a file system namespace that in many respects behaves like an independent file system. Filesets provide a means of partitioning the file system to allow administrative operations at a finer granularity than the entire file system:

• Filesets can be used to define quotas on both data blocks and inodes.
• The owning fileset is an attribute of each file and can be specified in a policy to control initial data placement, migration, and replication of the file’s data. See Policies for automating file management.
• Fileset snapshots can be created instead of creating a snapshot of an entire file system.

Independent and dependent filesets:
An independent fileset is a fileset with its own inode space. An inode space is a collection of inode number ranges reserved for an independent fileset. An inode space enables more efficient per-fileset functions, such as fileset snapshots.

Dependent fileset shares the inode space of an existing, independent fileset. Files created in a dependent fileset are assigned inodes in the same collection of inode number ranges that were reserved for the independent fileset from which it was created.

When the file system is created, only one fileset, called the root fileset, exists. The root fileset is an independent fileset that cannot be deleted. It contains the root directory as well as any system files such as quota files. As new files and directories are created, they automatically become part of the parent directory’s fileset. The fileset to which a file belongs is largely transparent for ordinary file access, but the containing fileset can be displayed along with the other attributes of each file using the mmlsattr -L command.

So if we look into this some more.

We have 2 directories in my mount /mnt/gpfs01 on the Storage Cluster

• pvc-9* is a test with independent fileset.
• spectrum-scale-csi-volume-store is also a fileset

# ls -la
pvc-9b789e4c-744b-4520–9b04–487fa6ae29ca
spectrum-scale-csi-volume-store

If I list filesets in the filesystem gpfs01.
We can see 3 filesets, same as above but also root, all filesets are linked to /mnt/gpfs01/*

[root@scale-dev-01 gpfs01]# mmlsfileset gpfs01
Filesets in file system ‘gpfs01’:
Name Status Path
root Linked /mnt/gpfs01
spectrum-scale-csi-volume-store Linked /mnt/gpfs01/spectrum-scale-csi-volume-store
pvc-9b789e4c-744b-4520–9b04–487fa6ae29ca Linked /mnt/gpfs01/pvc-9b789e4c-744b-4520–9b04–487fa6ae29ca

Note: To check the attributes for each filset, use the command: mmlsattr -L <fileset>

mmlsattr -L spectrum-scale-csi-volume-store/
file name: spectrum-scale-csi-volume-store/
metadata replication: 1 max 2
immutable: no
appendOnly: no
flags:
storage pool name: system
fileset name: spectrum-scale-csi-volume-store
snapshot name:
creation time: Thu Dec 17 21:08:00 2020
Misc attributes: DIRECTORY
Encrypted: no

If we look into lwdir directory and pvc claim, the fileset is spectrum-scale-csi-volume-store.

mmlsattr -L spectrum-scale-csi-volume-store/lwdir/pvc-984b42ce-c9c4-4764-baa8-4765d227dccb/
file name:            spectrum-scale-csi-volume-store/lwdir/pvc-984b42ce-c9c4-4764-baa8-4765d227dccb/
metadata replication: 1 max 2
immutable:            no
appendOnly:           no
flags:
storage pool name:    system
fileset name:         spectrum-scale-csi-volume-store
snapshot name:
creation time:        Thu Dec 17 22:01:16 2020
Misc attributes:      DIRECTORY
Encrypted:            no

To check what filesets have quotas, use the command mmrepquouta

mmrepquota -j gpfs01 --block-size G

From the Spectrum Scale GUI on your Storage Server, you will also get a good overview. Example: Type, Limits/quota, inode, Snapshots,

What the correct Volume for you, why chose one over the other or in combination❓

Lightweight dynamic provisioning:

• A new directory is created for every new volume provisioned dynamically using lightweight volume storageclass.
• Suitable for creating a large number of volumes (>10000).
• No Quota support. (PVC size cannot be imposed on storage due to lack of directory level quota support).
• Snapshot is done in the parent fileset, so snapshot is done over all data in the same fileset.
• Max 256 snapshots per fileset.

Independent Fileset-based dynamic provisioning:

• A new independent file set is created for every new volume provisioned dynamically using independent file set storageclass.
• Suitable when the number of volumes required is less than 1000.
  This can be fixed by adding more file systems. 254 File Systems per Spectrum Scale cluster.
• Snapshot support on each PVC.
• Quota Support on each PVC.
  PVC size can be honored on storage by means of file set quota.
• Inode management (The CSI is doing the inodeLimit, this is is calculated using this formula: volume size / block size of the file system.)
• Your own iNodeLimit parameter.
• IBM Spectrum Scale administrator can leverage management benefits of file sets on these volumes, such as snapshots and backup.

Dependent Fileset-based dynamic provisioning:

• A new dependent file set is created for every new volume provisioned dynamically using dependent file set storageclass.
• Suitable when number of volumes required is less than 10000.
• Quota support. PVC size can be honored on storage by means of file set quota.
• Snapshot is done on the parent fileset.
• No need to manage inodes.

Note: There are two ways in which volumes can be provisioned

• Dynamic provisioning: The resource is created on IBM Spectrum Scale file system dynamically during PVC creation.
• Static provisioning: The resource is pre-existing on IBM Spectrum Scale file system and can be used to define a PV.

You can choose to create all of Storage Class, but it also god ide to have an idea of which of them are the correct choice for your environment. 🤔

I will cover following StorageClass

• Lightweight dynamic provisioning
• Fileset-based dynamic provisioning with independent filesets.

Note: You can also create the storage-class from the OpenShift GUI.

Example for lightweight volume Storage Class:

1. On Storage Cluster: Check that the default fileset is created and linked,
   and then create the folder for lightweight volume, for example lwdir.
   The default fileset is spectrum-scale-csi-volume-store if not changed in the CSI installation

# mmlsfileset gpfs01 
Filesets in file system 'gpfs01':
Name                     Status    Path
root                     Linked    /mnt/gpfs01
spectrum-scale-csi-volume-store Linked /mnt/gpfs01/spectrum-scale-csi-volume-store

2. Create directory lwdir (only need to be done the first time)

# mkdir -p /mnt/gpfs01/spectrum-scale-csi-volume-store/lwdir/

3. Create the file: ibm-spectrum-scale-lt.yaml and copy in the example below.

Change the following:

• volBackendFs: your filesystem name on OCP Scale Cluster
• volDirBasePath: default: spectrum-scale-csi-volume-store/lwdir
  Remember to create the dir. example: lwdir

4. Apply the configuration to create StorageClass

oc apply -f ibm-spectrum-scale-csi-lt.yaml
Output: storageclass.storage.k8s.io/ibm-spectrum-scale-csi-lt created

Test with a PVC.

1. Create a pvc-lt.yaml file with the following content.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: scale-static-lt-pvc
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

2. Create a PVC by issuing the following command:

oc apply -f pvc.yaml

3. Check the status of PVC with oc get pcc --all-namespaces

The find pvc you created, the status should be bound and some seconds old.

oc get pvc — all-namespaces | grep static
ibm-spectrum-scale-csi-driver scale-static-pvc Bound pvc-f213c69b-1b27–44a2-aed5–3a3c10418c6c 1Gi RWX ibm-spectrum-scale-csi-lt 36s

If we check the Storage Cluster, we can see that we have the PVC in LWDIR.

cd /mnt/gpfs01/spectrum-scale-csi-volume-store/lwdir
ls -la

pvc-f213c69b-1b27-44a2-aed5-3a3c10418c6c

Note: if you want the Storage Class to be default.
From the OCP GUI and Storage Class.
Edit Annotations > KEY: storageclass.kubernetes.io/is-default-class : value true

Or add this under Metadata when creating the Storage Class.

annotations:
  storageclass.kubernetes.io/is-default-class: ‘true’

Example for Independent fileset storageClass:

1. Create the file: ibm-spectrum-scale-fileset.yaml and copy the example below.

If you have gone with the default, you only need to change the following:

• ClusterID to your StorageCluster/Owning Cluster.
  Run following command to get it:
  curl -s -k https://storage-gui.com/scalemgmt/v2/cluster -u "csi-storage-gui-user:csi-storage-gui-password" | grep clusterId
• volBackendFs: Your filesystem name on OCP Cluster
• GUI/UID: Default is Is 1000, change or remove the lines to use root.

2 .Apply the configuration to create StorageClass

oc apply -f ibm-spectrum-scale-csi-fileset.yaml

Test with a PVC claim

1. Create a pvc-fileset.yaml file:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: scale-static-fileset-pvc
spec:
  storageClassName:  ibm-spectrum-scale-csi-fileset
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

2. Create a PVC by issuing the following command:

oc apply -f pvc-fileset.yaml

3. Check the status of PVC with oc get pcc --all-namespaces

The find pvc you created, the status should be bound and it's 25 seconds old.

oc get pvc --all-namespaces | grep static-filset
ibm-spectrum-scale-csi-driver   scale-static-filset-pvc                    Bound    pvc-a93d632e-62b8-4666-aa9c-4b0855420cca   1Gi        RWX            ibm-spectrum-scale-csi-fileset   25s

• If we check the Storage Cluster, we can see that we have the PVC under /mnt/gpfs01

ls -la /mnt/gpfs01 | grep pvc-a9
pvc-a93d632e-62b8-4666-aa9c-4b0855420cca

• Check that the fileset i created.

mmlsfileset gpfs01 -L | grep pvc-a93
pvc-a93d632e-62b8-4666-aa9c-4b0855420cca 5 1572867

• To check the quota, that should be 1Gb here.

mmrepquota -j gpfs01 --block-size G
pvc-a93d632e-62b8-4666-aa9c-4b0855420cca - quota 1

Example for Dependent fileset storageClass:

Description of the fields

• volBackendFs:
  Name of the file system under which the fileset should be created. File system name is the name of the remotely mounted file system on the primary cluster.
• clusterId:
  Cluster ID of the owning cluster.
• uid:
  uid/username that will be assigned to the fileset. This is optional. The uid/gid must exist on the IBM Spectrum Scale GUI node of accessing and owning clusters. Default value is 1000
• gid:
  gid/group name that will be assigned to the fileset. This is optional.
  The gid/group name must exist on the IBM Spectrum Scale GUI node.of the accessing and owning clusters. Default value is 1000
• filesetType:
  Default is independent. This is optional.
• parentFileset:
  Parent fileset name. Valid with filesetType=dependent. Default is root.
• inodeLimit
  inode limit for fileset. This is optional. Valid with filesetType=independent. If not specified, inodeLimit is calculated using formula Volume Size / Block size of filesystem.

Note: for more information check also out:: config_csi_storagclass on KC

Ta-da! 👏

The IBM CSNA and IBM Spectrum Scale CSI should now be complete. 🎊

It was a long post so I hope this helped you with the installation, at least help you understand more of the product and installation.

This is the first release so I'm sure that we are going to see new features and improvements, There is Grafan Bridge coming, so I will try to cover this in a future post.