IBM Spectrum Scale Container Native Storage Access-v2

Installing Spectrum Scale for Persistent storage on Red Hat OpenShift Container Platform — Updated to 5.1.2.1 >

This article needed to be republished as the product and installation had major changes and simplifications since the first release: 5.1.0.1. More on that later.

The first release of the containerized version of Spectrum Scale also called CNSA, was released in December 2020, this is a very exciting and big shift in the Spectrum Scale product and how we can provide Persistent FileSystem for containerised environments.

So what is IBM Spectrum Scale Container Native Storage?

For people that don’t know what Spectrum Scale is,
We can start to describe it as mention on IBMs sites: IBM Spectrum® Scale is a clustered file system that provides concurrent access to a single file system or set of file systems from multiple nodes. The nodes can be SAN attached, network attached, a mixture of SAN attached, and network attached, or in a shared-nothing cluster configuration. This enables high performance access to this common set of data to support a scale-out solution or to provide a high availability platform.

So it’s a Cluster filesystem but it also has much more functions. Protocols, Replication, snapshot, DR, HDFS, Encryption, compression, FileAudit function, just to mention some of them.

So with the IBM Spectrum Scale Container Native Storage version, and the combination of CSI allows for storage to be provisioned and accessed by container applications within RedHat OpenShift Container Platform. What’s new is that we now can containerize the Spectrum Scale cluster and run this on top of OpenShift/K8

Client clusters running on OCP don’t need any internal storage or internal filesystem, as it allows us to access storage/filesystem on any non-containerized Spectrum Scale storage cluster at the 5.1.2.1 level of code.

After talking to some customers that are using OCP or Kubernetes, they are still talking about persistent storage and that it can be difficult to manage, because most of them have not just one cluster, but up to 2 digits, and they are also not just running OCP on physical boxes but on top of a HyperVisor. So with the possibility to remote mount the filesystem, this will add flexibility in that it unties OpenShift from the storage hardware.

The separate Spectrum Scale Storage Cluster, this cluster could be Virtual, Physical or even an ESS (IBM Elastic Storage Server). As longs as the Scale cluster have a FileSystem and GUI. (5.1.1.2)

To get a picture of how an OCP environment could look like, I have drawn up an example with OCP (Worker Nodes) and Scale Cluster with the components.

By default, IBM Spectrum Scale Pods will be deployed to all OCP worker nodes. Optionally, you could select a subset of worker nodes to deploy IBM Spectrum Scale onto. Node Labels

OCP Information

• Namespace: ibm-spectrum-scale
  - All OCP Worker’s nodes have Scale Core pod (Can be decided with Node Selector)
  - 2 of X-Worker’s Nodes holds Scale-PMCollector pod (Can’t be decided with Node Labels anymore in 5.1.1.3.designation in 5.1.1.3. Instead, a node selector can be specified in the cluster CR.)
  - 2 of X-Worker’s Nodes holds Scale-GUI pod.
• Namespace: ibm-spectrum-scale-operator
  - 1 of X-Worker’s Nodes holds scale-core-controller pod.
• Namespace: ibm-spectrum-scale-csi
  - All OCP Workers have the Scale CSI Pod.
  - 1 of X-Workers holds CSI Operator, Provisioner, CSI attacher.

External IBM Spectrum Scale Cluster

• IBM Spectrum Scale Storage Cluster
  - Contains 3 Linux VMs (This could be 1 or 1000+, more realistic case, we have 3 to survive failure and quorum)
  - Scale Core.
  - Scale GUI on 2 of them. Only needed one of them.
  - Scale CES/Cluster Export Service (Not needed, only if you want Protocol support NFS, SMB, Object.)

So when we are deploying the CNSA, the CNSA will connect to the RestAPI/GUI on the storage cluster and do a handshake to allow the remote mounting of a filesystem. This API is also used when creating filesets for PVC, more on this later.

What’s changed in CNSA and CSI since last time.

There are some major changes in the product since the first post (5.1.0.1)
Just to mention some changes that are worth mentioning.

What’s new in the IBM Spectrum Scale Container Native - v5.1.1.3

• IBM Spectrum Scale Container Native Storage Access images are now hosted on the entitled IBM Cloud Container Registry.
• GitHub will be used to install IBM Spectrum Scale Container Native Storage Access instead of the tarball install method.
  - https://github.com/IBM/ibm-spectrum-scale-container-native
• Separate operator namespace.
  - In 5.1.1.3, the operator and the cluster live in separate namespaces (aka cluster-wide operator).
• Multiple custom resources and controllers.
  - Instead of one CR ScaleCluster, it’s now one CR for Cluster, Deamon, Filesystem, RemoteCluster, and so on.
  And the operator contains multiple controllers. There is a separate controller for each CR.

oc get crds | grep scale.spectrum.ibm.com

• Core pods are managed by a daemon controller instead of DaemonSet.
  - This means that Deamon Controller knows more about Spectrum Scale and how to keep it running. (example quorum)
• Rolling pod upgrade.
• Set memory and CPU requests and limits for pods.
• Support for Spectrum Scale storage cluster filesystem encryption.
  - This at rest encryption, encrypted filesystem.
  - You will need a Key Server
• Automated deployment of Spectrum Scale CSI Driver
  - The deployment of IBM Spectrum Scale CSI Driver is now automated by the new CSI controller.
• Using the GUI
  - We can now access and use the Scale GUI on the CNSA cluster
• Automated IBM Spectrum Scale performance monitoring bridge for Grafana.
  - The Grafana bridge can be deployed by the operator to two Grafana bridge pods

For More information Check-Out: What’s New 5.1.1.3

Container image listing for IBM Spectrum Scale

There is a Container image list over PODs and containers, this gives us some insight into what’s images is need.
One of the changes is that all images required for deployment of a IBM Spectrum Scale container native cluster will be sourced from the IBM Container Repository.

Picture from IBM Documentation

As Noted on IBM Doc: The above list is purely informational, no user action is required to obtain or define this list of images if in a non-air-gapped environment. For more information about instructions to mirror the above list of images in an airgapped environment, see Airgap Setup.

For this article, we will use the IBM Container Repository.

This Part is going to be lengthy, so take your time and fill up on your favourite beverage. 🍹 ☕️

Installing RedHat OpenShift Cluster on vSphere.

Instead of rewriting this, VirtuallyGhetto has covered it in detail here:

Using the new installation method for deploying OpenShift 4.5 on VMware Cloud on AWS

This will also work for an on-prem solution, as I did.
What I used some time on was understanding the DNS record for API VIP and Ingress VIP. i created a new subdomain for cluster name and then api.<cluster-name>.<base-domain> and *.apps.<cluster-name>.<base-domain>.

There are some hardware and software requirements for Spectrum Scale.

The minimum memory for Worker Nodes is 16GB.
I’m also only running a 1Gb network, 10Gb is the minimum to get something that flies 🐝

So to change the default variable for CPU and Memory you can create the install-config file.
Create a folder called example: install-config.

./openshift-install create install-config --dir=install-config

Change the value under vSphere:

When running the installation I point to the folder with YAML file and Log-level: debug

./openshift-install create cluster --dir=install-config --log-level debug

Before starting installing Spectrum Scale CNSA

Almost all of the steps below are in very good detail in IBM Knowledge Center. So if you want more details than what I cover in this article, I would recommend starting here:
The planning for IBM Spectrum Scale Container Native Storage Access (CNSA)

I have written down how I did the installation on OCP 4.8.11 on vSphere with Spectrum scale Storage Cluster v5.1.

2.1, this was done in the shortest and “easiest” way so I could start testing the product. In a production environment, I would strongly recommend reading through the IBM Documentation. And when a newer version has come out, check out what’s new in the Documentation link above.

For installing on IBM Power. Check Out:
installing-ocp-on-ibm-power
ocp4-upi-powervm

In short, you will need: ✅

Storage Cluster

• A Spectrum Scale Storage Cluster. v5.1.1.3 or higher to support remote mount via the Operator.
• FYI: I recently wrote an article on how to deploy Scale Storage Cluster with ansible using this Ansible Role. https://github.com/IBM/ibm-spectrum-scale-install-infra

OCP Scale Cluster

• OCP Cluster with 4.7, 4.8, 4.9
  (PLEASE check IBM Documentation and FAQ for an updated list)
• For normal installation 3 masters and 3 worker nodes.
  (Check out Compact Cluster Support if you to install CNSA on smaller cluster)
• Ensure that your DNS is configured properly so that the worker nodes can resolve the storage cluster nodes. If this is not possible, see the Host Aliases section in the custom resource.
• Access to the Internet for IBM Container Repository.
  For an Airgapped environment, check out see Airgap Setup.
• All worker nodes must be able to communicate with each other via the host network.
  The Spectrum Scale pods will use the host network and not the Container Network Interface (CNI) overlay network
• Firewall Recommendation.
  Ensure that ports 1191, 443, and the ephemeral port ranges are open on the storage cluster and on any network switches between the storage and container native cluster. Otherwise, the container native cluster will not be able to remotely mount the file system from the storage cluster.
• Access to the IBM Cloud Container Registry and IBM Spectrum Scale Container
• Get the entitlement key. by login in with your IBMid that is associated with the entitled software and click ClickGet entitlement key

Installing CNSA

Preparation:

I’m using a Linux/Centos VM that has access to the OC Cluster and also have the OC Client installed, and I also have created myself a directory to work in when downloading and editing files.

Machine Configuration (MCO)

Applying the Machine Configuration provided will drive a rolling update of the OpenShift nodes and could take many minutes to complete. In order for the new configurations to take effect the nodes within the pool are required to be rebooted. Applying the supplied YAML files will perform the following tasks:

• Increase PID_LIMITS: Increase the pids_limit to 4096. Without this change, the GPFS daemon will crash during I/O by running out of PID resources.
• Kernel Devel/Header Packages: Installs the kernel related packages for IBM Spectrum Scale to successfully build its portability layer.
• Increase vmalloc kernel parameter: Modifies kernel parameters required to operator properly with RHCOD. Applies only to IBM Spectrum Scale running on Linux® on System Z.

Applying MachineConfigOperator (MCO) Settings

Apply the following set of Machine Configurations depending on your OCP version and machine’s architecture:

Link to newest version of MCO on IBM Doc

If you are running x86_64, apply the following:

For. 4.7

oc apply -f https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-container-native/v5.1.2.1/generated/mco/ocp4.7/mco_x86_64.yaml

For 4.8:

oc apply -f https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-container-native/v5.1.2.1/generated/mco/ocp4.8/mco_x86_64.yaml

For 4.9:

oc apply -f https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-container-native/v5.1.2.1/generated/mco/ocp4.9/mco_x86_64.yaml

If you are running IBM Power ppc64le, apply the following:

For 4.7

oc apply -f https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-container-native/v5.1.2.1/generated/mco/ocp4.7/mco_ppc64le.yaml

For 4.8:

oc apply -f https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-container-native/v5.1.2.1/generated/mco/ocp4.8/mco_ppc64le.yaml

For 4.9

oc apply -f https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-container-native/v5.1.2.1/generated/mco/ocp4.9/mco_ppc64le.yaml

Verifying MachineConfigOperator (MCO) Settings

Complete the following steps:

1. Check the status of the “UPDATED” by using the following command:
   oc get MachineConfigPool

This will take some time, as it needs to take each node.

2. Verify that the PID_LIMITS 👇

Note: This command will run through all the worker nodes. Use with discretion if you have a large system.

Verify that the PID_LIMITS are increased on the worker nodes by issuing the following command:

oc get nodes -lnode-role.kubernetes.io/worker= -ojsonpath="{range .items[*]}{.metadata.name}{'\n'}" | xargs -I{} oc debug node/{} -T -- chroot /host crio-status config | grep pids_limit

3. Validate that the kernel-devel package is successfully applied onto the Red Hat OpenShift Container Platform worker nodes by using the following command:

oc get nodes -lnode-role.kubernetes.io/worker= \
-ojsonpath="{range .items[*]}{.metadata.name}{'\n'}" | \
xargs -I{} oc debug node/{} -T -- chroot /host sh -c "rpm -q kernel-devel"

Adding IBM Cloud Container Registry credentials ☁️

For images to be properly pulled at the pod level, the OpenShift global pull secrets must be modified to contain credentials to access the IBM Cloud Container Registry.

Note: The following steps are for users whose OpenShift cluster is accessing the IBM Cloud Container Registry. For instructions of an airgapped environment, see Airgap setup for network restricted Red Hat OpenShift Container Platform clusters section.

1. Create a base64 encoded string of the credentials used to access the image registry.
   Note: For using IBM Cloud Container Registry, the credentials will be the fixed cp user and the generated entitlement key.

You will receive a long key that we need to base64 encode.
Please check your string here for spaces between the double-quotes.

echo -n “cp:REPLACE_WITH_GENERATED_ENTITLEMENT_KEY” | base64 -w0

Then you will have a BASE64 ENCODED String something like this:
(This is not a valid one.. 😅)

Y3A6IGVT2lKSVV6STFOaUo5LmV5SnBjM01pT2lKSlFrMGdUV0Z5YTJWMGNHeGhZMlVpTENKcFlYUWXdNRE1zSW1wMGFTSTZJbVEzWlRJpSaFltTTRaR0UzTm1NMlpUTTJNMlppTUdVeEluMC5qV2dHX1JpV1VnaDhPejhHQVh1c2ZFVzNMZVEcsdsad0tsaUlUd3lNa0lTZmJ3Cg=

2. Create an authority.json to include the base64 encoded string of your credentials, the fixed username cp (used to access cp.icr.io repository), and generated entitlement key for the IBM Cloud Container Registry.

Please check your string here for spaces between the double-quotes.

3. The following step will take the authority.json and include it as a new authority in your .dockerconfigjson, stored as a temp_config.json.

- Note: Using the IBM Cloud Container Registry as our authority, we use cp.icr.io as the input key for the contents of authority.json.

Make sure you have jq installed.

[root@ocp-admin ~]# jq — version
jq-1.6

Run the following command:

oc get secret/pull-secret -n openshift-config -ojson | jq -r '.data[".dockerconfigjson"]' | base64 -d - | jq '.[]."cp.icr.io" += input' - authority.json > temp_config.json

Note: If you get an error like this, check your authority.json is correct and that there are no hidden spaces.

To verify your authority credentials were created in the resulting file:

4. Use the contents of the temp_config.json file, and apply the updated config to the OpenShift cluster.

oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=temp_config.json

To verify your pull-secret has been updated with your new authority, issue the following command and confirm your authority is present.

oc get secret/pull-secret -n openshift-config -ojson | jq -r '.data[".dockerconfigjson"]' | base64 -d -

5. The updated config will now be rolled out to all nodes in the OpenShift cluster. Nodes will be cycled through one at a time and will not be schedulable before rebooting. (Only for OCP 4.7 and below) You may observe nodes through watch oc get nodes

Note: Red Hat OpenShift Container Platform 4.7 and above versions do not reboot the nodes. For more information about updating the global pull secret in your OpenShift cluster, see Updating the global cluster pull secret.

6. When the global pull secret has been updated, remove the temporary files that were created. rm authority.json temp_config.json

Deploy the Scale Operator

To deploy the IBM Spectrum Scale container native operator, use the following command:

oc apply -f https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-container-native/v5.1.2.1/generated/installer/ibm-spectrum-scale-operator.yaml

Validate that the operator pods is running in the following namespaces:

oc get pods -n ibm-spectrum-scale-operator

oc get pods -n ibm-spectrum-scale-csi

Labels and Annotations ✍️

I have not added any labels or annotations at this stage. However, I have added some text from the documentation to describe how this can be set.

Designation Labels

IBM Spectrum Scale container native automatically assigns certain designations to some worker nodes. Normally you do not need to explicitly designate the worker nodes. However, it is possible to do so using node labels.

The following mechanisms are supported to designate IBM Spectrum Scale container native nodes:

• Automatic (Recommended) — Allows the Operator to designate the nodes automatically.
• Manual (Optional) — Allows administrators to have more control of the placement of Spectrum Scale node designations (like the quorum designation) to pods on specific worker nodes.

Automatic

If the user does not label any nodes as quorum nodes, the Operator will automatically apply quorum annotations to a subset of the nodes in the cluster. The number of nodes to be annotated depends on the number of nodes in the cluster:

• If the number of nodes in the cluster definition is less than 4, all nodes are designated as quorum nodes.
• If the number of nodes in the cluster definition is between 4 and 9 inclusive, 3 nodes are designated as quorum nodes.
• If the number of nodes in the cluster definition is between 10 and 18 inclusive, 5 nodes are designated as quorum nodes.
• If the number of nodes in the cluster definition is greater than 18, 7 nodes are designated as quorum nodes.

Check out the documentation here: https://www.ibm.com/docs/en/scalecontainernative?topic=cluster-node-labels-annotations

Requests and Limits Annotations

For IBM Spectrum Scale container native core pods, users are able to override default memory and CPU resource requests. See Managing Resources for Containers for more information.

Memory/CPU limits are automatically set to the capacity of the node and requests default to 25% of the node’s capacity. However, requests may be specified on a per-node basis using the following node annotation.

Memory

• To set the memory request to 50G on worker1: oc annotate nodes

oc annotate nodes worker1.example.com scale.spectrum.ibm.com/memory=”50G”

CPU

• To set the CPU request to 5 vCPU/Core on worker2: oc annotate node

oc annotate node worker2.example.com scale.spectrum.ibm.com/cpu=”5"

Note: The recommendation is to set the resource requests/limits at the role level in the cluster CR atcluster.spec.deamon.roles.resources

Network

• I have not tested this one, however it looks like you can set the Spectrum Scale daemon-network. For example, if you have more than one physical/virtual nic on your worker nodes, you can set what spectrum scale should use as daemon network

oc annotate nodes worker.example.com scale.spectrum.ibm.com/daemon-network=10.33.3.100

Storage Cluster Configuration

Firewall recommendations

Ensure that ports 1191, 443, and the ephemeral port ranges are open on the storage cluster and on any network switches between the storage and container native cluster. Otherwise, the container native cluster will not be able to remotely mount the file system from the storage cluster.

For more information about how to set the port range in the container native cluster, see Ephemeral Port Range.

For more information about all Spectrum Scale services, see Securing the IBM Spectrum Scale system using firewall.

Creating GUI user credentials for IBM Spectrum Scale storage cluster

The GUI user group ContainerOperator must exist on the IBM Spectrum Scale storage cluster. You must add a user to this group that is used by the operator to configure the remote mount of the storage cluster file systems.

Creating Operator User and Group

Complete the following commands in the shell of the GUI node of the storage cluster:

1. To verify whether the IBM Spectrum Scale GUI user group ContainerOperator exists, run the following command:

/usr/lpp/mmfs/gui/cli/lsusergrp ContainerOperator

2. To create the ContainerOperator GUI user group if it does not exist, run the following command:

/usr/lpp/mmfs/gui/cli/mkusergrp ContainerOperator — role containeroperator

3. To verify whether an IBM Spectrum Scale GUI user exists within the ContainerOperator group, run the following command

/usr/lpp/mmfs/gui/cli/lsuser | grep ContainerOperator

4. To create the GUI user for the ContainerOperator group, run the following command:

/usr/lpp/mmfs/gui/cli/mkuser cnsa_storage_gui_user -p mycnsapassword! -g ContainerOperator

By default, user passwords expire after 90 days. If the security policy of your organization permits it, then run the following command to create the user with a password that never expires:

/usr/lpp/mmfs/gui/cli/mkuser cnsa_storage_gui_user -p mycnsapassword! -g ContainerOperator -e 1

Note: The -e 1 parameter is only available for IBM Spectrum Scale storage cluster 5.1.1.0 or later.

To test the user and to find the ClusterID we can use a RestAPI call using curl

curl -s -k https://lab-scale1/scalemgmt/v2/cluster -u “cnsa_storage_gui_user:mycnsapassword!” | grep clusterId

Pre-installation tasks for CSI driver deployment

Complete the following tasks on the storage cluster:

1. Create an IBM Spectrum® Scale user group “CsiAdmin” using the following command:

/usr/lpp/mmfs/gui/cli/mkusergrp CsiAdmin — role csiadmin

2. Create an IBM Spectrum Scale user in the “CsiAdmin” group. This user must be used on IBM Spectrum Scale Container Storage Interface driver configuration. Issue this command on the GUI node to create the user

/usr/lpp/mmfs/gui/cli/mkuser csi-storage-gui-user -p my_csi_password! -g CsiAdmin -e 1

3. Ensure perfileset quota on the file systems to be used by IBM Spectrum Scale Container Storage Interface driver is set to No

# /usr/lpp/mmfs/bin/mmlsfs gpfs01 --perfileset-quota

4. Quota is enabled in the file systems to be used by IBM Spectrum Scale Container Storage Interface driver

/usr/lpp/mmfs/bin/mmchfs gpfs01 -Q yes

Verify that quota is enabled.

/usr/lpp/mmfs/bin/mmlsfs gpfs01 -Q

5. Enable quota for root user by issuing the following command:

mmchconfig enforceFilesetQuotaOnRoot=yes -i

6. Ensure that the controlSetxattrImmutableSELinux parameter is set to “yes” by issuing the following command:

mmchconfig controlSetxattrImmutableSELinux=yes -i

-i option, is to do cluster changes without the need to restart GPFS daemon, not all parameters will support this.

Note: To check the current scale config run:

/usr/lpp/mmfs/bin/mmlsconfig

7. Enable filesetdf of the file system by using the following command:

/usr/lpp/mmfs/bin/mmchfs gpfs01 --filesetdf

Creating secrets for storage cluster GUI

Create a secret on the Red Hat OpenShift cluster to store the username and password for the IBM Spectrum Scale Storage cluster GUI user and password.

This secret is used by the Operator to communicate with the storage cluster while performing configuration for remote mount.

Two new secrets must be added for each storage cluster being configured for remote mount on the IBM Spectrum Scale container native cluster.

• Create a secret for the storage cluster ContainerOperator GUI user
• The username and password specified in this topic must match the GUI user that was created on the storage cluster in step 4 of Creating Operator User and Group.
• To create the storage cluster GUI user secret named cnsa-remote-mount-storage-cluster-1 in the ibm-spectrum-scale namespace, run the following command:

Note: The name of this secret should match the secretName field in the RemoteCluster CR

If the ibm-spectrum-scale namespace don’t exist yet, then wait untill you have applied the scale_v1beta1_cluster_cr.yaml file.

oc create secret generic cnsa-remote-mount-storage-cluster-1 --from-literal=username='cnsa_storage_gui_user' --from-literal=password='mycnsapassword!' -n ibm-spectrum-scale

2. Create a secret for the storage cluster CsiAdmin GUI user

The username and password specified in this topic must match the GUI user that was created on the storage cluster in step 2 of Pre-installation tasks for CSI Operator deployment.

Note: The name of this secret should match the csiSecretName field in the RemoteCluster CR

To create the storage cluster GUI user secret named csi-remote-mount-storage-cluster-1 in the ibm-spectrum-scale-csi namespace, run the following command:

oc create secret generic csi-remote-mount-storage-cluster-1 --from-literal=username=csi-storage-gui-user --from-literal=password=my_csi_password! -n ibm-spectrum-scale-csi

Then Label the Secret

oc label secret csi-remote-mount-storage-cluster-1 -n ibm-spectrum-scale-csi product=ibm-spectrum-scale-csi

Configuring the IBM Spectrum Scale container native cluster custom resources

Before deploying the cluster, it is required for you to make changes to the sample scale_v1beta1_cluster_cr.yaml file.

Note: Don’t use my cluster_cr.yaml example, it’s only used to show how you should populate the file .

Save the sample copy provided at GitHub using the following command:

curl -fs https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-container-native/v5.1.2.1/generated/scale_v1beta1_cluster_cr.yaml > scale_v1beta1_cluster_cr.yaml || echo "Failed to curl Cluster CR"

This file is used to set configuration and, once deployed, will initiate the IBM Spectrum Scale cluster creation. The following sections will help guide you through this process.

For more options than I will cover, Please check out the IBM Documentation.

• Cluster
• Callhome
• Filesystem
• Encryption

Edit Custom Resource (CR)

To edit the file, you can use VI/VIM or better an editor that supports YAML. ATOM/InteliJ/VSCode

Cluster

cluster.Spec.License

The spec.license section allows the user to accept and choose the IBM Spectrum Scale edition that will be deployed in the container native cluster. The user must:

• Review the appropriate license documentation via the URL link in the CR.
• Accept the license by specifying true in the license.accept field.
• Supply the edition being used in the license.license field.

Specifying an edition without proper entitlement will result in image pull failures during deployment.

license:
    accept: true
    license: data-access

See oc explain cluster.spec.license for more information.

Cluster.spec.deamon.clusterProfile

The cluster.spec.deamon.clusterProfile section allows you to set Cluster specific settings, there already default parameters,

Ephemeral Port Range
If the storage cluster has the ephemeral port range configured, you will need to set tscCmdPortRange on the container native cluster to match the range.

If the storage cluster has the ephemeral port range configured, you will need to set tscCmdPortRange on the container native cluster to match the range.

For example, if the storage cluster is configured to use port range, 60000–61000, set this value under the clusterProfile section in the Cluster CR

To check if your storage Cluster has the tscCmdPortRange defined.
Run this command on your storage cluster. mmlsconfig | grep tsc

FileSystems

filesystems.metadata.names

The name that you set for the Filesystem CR under metadata.name is the name of the custom resource and also becomes the name of the remote filesystem mount point at /mnt/<metadata.name>. In the sample, the name of the local filesystem is remote-sample and will be mounted at /mnt/remote-sample. You can define more than 1 Filesystem CR.

filesystems.spec.remote

Set the details under the remote section to reflect the storage cluster filesystem being mounted as fs and the name of the RemoteCluster created as cluster.
filesystem.spec.remote.clustercluster is the name of the RemoteCluster resource same as remotecluster.metadata.name
filesystem.spec.remote.fs
fs is the name of the filesystem on the RemoteCluster to mount

With this, they mean the cluster that contains the filesystem = Storage Cluster

See oc explain filesystem.spec.remote for more information.

RemoteCluster CR

The sample RemoteCluster custom resource can be found under
kind: RemoteCluster in scale_v1beta1_cluster_cr.yaml file.

remotecluster.metadata.name

The name that you set for the RemoteCluster CR under metadata.name identifies the remote storage cluster you want to create an auth too. This name is what you will reference in the Filesystem CR spec.remote.cluster to identify the remote storage cluster serving the filesystem. You can define more than 1 RemoteCluster.

remotecluster.spec.contactNodes

contactNodes: I comment out the contact nodes as if nothing is specified the operator will use 3 nodes obtained from the storage cluster.

Validate that the secret for the storage cluster has been created describe in Storage cluster GUI user secret.

remotecluster.spec.gui

1. cacert: I comment out the cacert: line and change the
2. csiSecretName:Validate that the secret for the CSI remote cluster has been created describe in Storage cluster GUI user secret.
3. host:Set the GUI details to match your remote storage GUI in the spec.gui section: My storage cluster GUI is lab-scale1.oslo.forum.com
4. insecureSkipVerify: true As I don’t have a proper certificate anyway.
5. csiSecretName:Validate that the secret for the storage cluster has been created describe in Storage cluster GUI user secret.

Just some more information about each value for RemoteCluster CA

remotecluster.metadata.name name of the CR used to identify the remote storage cluster in the filesystem CR

remotecluster.spec.contactNodesProvide a list of storage nodes names to be used as the contact nodes list. The names should be the daemon node name in the spectrum scale storage cluster. If nothing is specified, the operator will use 3 nodes obtained from the storage cluster. Note: To list the contact nodes, issue mmlscluster on the storage cluster and use the listed daemon node names.

remotecluster.spec.gui.hosthostname for the GUI endpoint on the storage cluster.

remotecluster.spec.gui.cacertname of the Kubernetes ConfigMap containing the CA certificate for the storage cluster GUI

remotecluster.spec.gui.secretNameThis is the name of the Kubernetes secret created during the storage cluster configuration.

remotecluster.spec.gui.insecureSkipVerifyControls whether a client verifies the storage cluster's GUI certificate chain and hostname. If set true, TLS is susceptible to machine-in-the-middle attacks. Default is false.

See oc explain remotecluster.spec for more information

Configure Certificate Authority (CA) certificates for storage cluster

To Configure IBM Spectrum Scale container native to use Transport Layer Security (TLS) verification to guarantee secure HTTPS communication with the storage cluster GUI, check out the following link.

https://www.ibm.com/docs/en/scalecontainernative?topic=cluster-configure-certificate-authority-ca-certificates

Have attached an example of my scalecluster_cr.yml below,
I have removed evrything above callhome.

Note: Don’t use my cluster_cr.yaml example, it’s only used to show how you should populate the file . download the newest one from git.

Creating the IBM Spectrum Scale container native cluster

Deploy the cluster by applying the custom resource that was modified in the previous Custom Resource section.

Once the customer resources file is applied, IBM Spectrum Scale Operator creates all the pods that make up an IBM Spectrum Scale container native cluster.

oc apply -f ./scale_v1beta1_cluster_cr.yaml

Immediately I had an issue with the ImagePullBackOFF.

error authenticating creds for “cp.icr.io”: Requesting bear token: invalid status code from registry 400 (Bad Request)

This was pointing to my authentication against cp.icr.io.
This was caused by a line break in the 64encode

So I went through my authentication.json and checked my BASE64_ENCODED_KEY

After I fixed the issue, the pods came up one by one.
Note: I have 4 worker nodes, and it toke me about 10min just for the pods in ibm-spectrum-scale namespace.

Lets us verify that the CNSA cluster has been correctly created:

1. Verify that the Operator creates the cluster by checking pods
   We have 3 namespaces that are used for Spectrum Scale CNSA
   Let us check that all pods are running.

oc get pods -n ibm-spectrum-scale
oc get pods -n ibm-spectrum-scale-operator
oc get pods -n ibm-spectrum-scale-csi

Note:
ibm-spectrum-scale: The resulting cluster will contain two gui pods, two pmcollector pods, and one core pod per node that matches the specified nodeSelector

ibm-spectrum-scale-csi: The resulting cluster will contain two attacher, operator,provisioner,snapshotter,resizer,and one CSI pod per node that matches the specified nodeSelector

2. Verify that the IBM Spectrum Scale cluster has been correctly created:

Run the mmlscluster command:

oc exec $(oc get pods -lapp.kubernetes.io/name=core \
 -ojsonpath="{.items[0].metadata.name}" -n ibm-spectrum-scale)  \
 -c gpfs -n ibm-spectrum-scale -- mmlscluster 

mmlscluster output

Run the mmgetstate command:

mmgetstate -a will give a status for the GPFS daemon on all nodes in the CNSA cluster

oc exec $(oc get pods -lapp.kubernetes.io/name=core \
 -ojsonpath="{.items[0].metadata.name}" -n ibm-spectrum-scale)  \
 -c gpfs -n ibm-spectrum-scale -- mmgetstate -a

Output from mmgetstate

3. Verify that the Remote Cluster auth has been successfully created.

Check the status of the RemoteCluster to ensure that the Status Ready is True.

Get a list of the remote clusters.

• oc get remoteclusters -n ibm-spectrum-scale

# oc get remoteclusters -n ibm-spectrum-scale
NAME AGE
labscale 11d

We use describe to get more information of the remote cluster to check its status.

oc describe remotecluster labscale -n ibm-spectrum-scale

If the storage cluster auth was created successfully, you should see a Status similar to: The remote cluster has been configured successfully.

4. Verify the storage cluster filesystem has been configured:

Check the status of the Filesystem to ensure that the Status Success is True.

Get a list of the filesystems:

• oc get filesystems -n ibm-spectrum-scale

Describe the sample filesystems to check status:

• oc describe filesystems fs1 -n ibm-spectrum-scale

If the filesystem was created successfully, you should see a Status similar to:
The remote filesystem has been created and mounted.

5. You can also manually verify that the filesystem is mounted using the mmlsmount command.

PS: change your filesystem name

oc exec $(oc get pods -lapp.kubernetes.io/name=core \
-ojsonpath="{.items[0].metadata.name}" -n ibm-spectrum-scale)  \
-c gpfs -n ibm-spectrum-scale -- mmlsmount fs1 -L

Example output from mmlsmount

Troubleshooting: Some tips. 👓

If you hit into problems deploying, I would start with checking the IBM Doc Troubleshooting section:

Normally I would start checking.

1. Checking the pods status. oc get pods -n ibm-spectrum-scale-x
2. Check the describe of a pod: oc describe <podname> -n ibm-spectrum-scale-x
3. check the logs of the pod: oc logs <podname> -n ibm-spectrum-scale-x

Logon to Core pods

If you need, you can log into the cores pods using :oc exec -it <pod> -n ibm-spectrum-scale -- sh

From here, we can run Spectrum Scale commands.

Cleanup / Reinstall

If you need to remove the CNSA: Follow the Cleanup Procedure.

Accessing the IBM Spectrum Scale GUI

Users created on the Openshift Container Platform (OCP) can log in to the IBM Spectrum Scale container native GUI via single-sign-on (SSO) using the OAuth implementation.

For more information about OpenShift Users, check out the doc on CNSA GUI

To access the IBM Spectrum Scale GUI:

1. In a browser, navigate to: https://ibm-spectrum-scale-gui-ibm-spectrum-scale.apps.<ocp domain>/, where <ocp domain> is the domain of your OpenShift cluster. You should see the IBM Spectrum Scale GUI login page.
   If the domain is ocp4.example.com, the URL would be: https://ibm-spectrum-scale-gui-ibm-spectrum-scale.apps.ocp4.example.com.
2. Click Sign in, which should redirect you to the Red Hat Openshift Container Platform login page.
3. Authenticate using your OCP user credentials. On success, you should be redirected back to the IBM Spectrum Scale GUI homepage.

Configuring IBM Spectrum Scale Container Storage Interface (CSI) driver

Use the following sections to help with deploying IBM Spectrum Scale CSI with IBM Spectrum Scale container native:

• Configuring storage class to use CSI driver
• Managed CSI fields
• Setting primary file set

Create Storage Class and PVC for Spectrum Scale.

Before we go into int each we should briefly cover the Kubernetes Storage Terminology:

Persistent Volume (PV) is a unit of storage that has been provisioned by an administrator or dynamically provisioned via a storage driver or plug-in.

Persistent Volume Claim (PVC) is a request for storage by a user. PVCs consume PV resources.

StorageClass(SC) provides the ability to describe the classes of storage that can be dynamically provisioned. SC will be used to map to quality-of-service, backup or other storage features.

The CSI for CNSA have the following functional support

• Static provisioning: Ability to use existing directories and filesets as persistent volumes
• Lightweight dynamic provisioning: Ability to create directory-based volumes dynamically
• Fileset-based dynamic provisioning: Ability to create filesets based volumes dynamically
• Supported volume access modes: RWX (readWriteMany) and RWO (ReadWriteOnce)

Fileset: you are maybe asking yourself, what is a fileset and what is the difference between independent and dependent fileset.

From the IBM Doc: GPFS utilizes a file system object called a fileset. A fileset is a subtree of a file system namespace that in many respects behaves like an independent file system. Filesets provide a means of partitioning the file system to allow administrative operations at a finer granularity than the entire file system:

• Filesets can be used to define quotas on both data blocks and inodes.
• The owning fileset is an attribute of each file and can be specified in a policy to control initial data placement, migration, and replication of the file’s data. See Policies for automating file management.
• Fileset snapshots can be created instead of creating a snapshot of an entire file system.

Independent and dependent filesets:
An independent fileset is a fileset with its own inode space. An inode space is a collection of inode number ranges reserved for an independent fileset. An inode space enables more efficient per-fileset functions, such as fileset snapshots.

Dependent fileset shares the inode space of an existing, independent fileset. Files created in a dependent fileset are assigned inodes in the same collection of inode number ranges that were reserved for the independent fileset from which it was created.

When the file system is created, only one fileset, called the root fileset, exists. The root fileset is an independent fileset that cannot be deleted. It contains the root directory as well as any system files such as quota files. As new files and directories are created, they automatically become part of the parent directory’s fileset. The fileset to which a file belongs is largely transparent for ordinary file access, but the containing fileset can be displayed along with the other attributes of each file using the mmlsattr -l command

Let's dig into this some more.

We have 2 directories in my mount /mnt/gpfs01 on the Storage Cluster

• pvc-9* is a test with an independent fileset.
• spectrum-scale-csi-volume-store is also a fileset (for Lightweight)
  PS: as of 5.1.x this is now: primary-fileset-fs1–197279292439329xxxx

# ls -la
pvc-9b789e4c-744b-4520–9b04–487fa6ae29ca
spectrum-scale-csi-volume-store
primary-fileset-fs1-1972792924393295911

If we list the filesets in the filesystem gpfs01. mmlsfileset filesystemname
We can see tree filesets, same as above but also root, all filesets are linked to /mnt/gpfs01/*

Note: To check the attributes for each fileset, use the command:

mmlsattr -L filesetname

If we look into lwdir directory and PVC claim, the fileset is a spectrum-scale-csi-volume-store.
On newer CSI/CNSA version this will be: primary-fileset-fs1–1972792924393295911

To check what filesets have quotas, use the command mmrepquouta

mmrepquota -j gpfs01 --block-size G

From the Spectrum Scale GUI on your Storage Server, you will also get a good overview. Example: Type, Limits/quota, inode, Snapshots,

What’s the correct Storage Class/Volume for you, why chose one over the other or in combination❓🤔

Lightweight dynamic provisioning:

• A new directory is created for every new volume provisioned dynamically using lightweight volume storage-class.
• Suitable for creating a large number of volumes (>10000).
• No Quota support. (PVC size cannot be imposed on storage due to lack of directory level quota support).
• Snapshot is done in the parent fileset, so snapshot is done over-all data in the same fileset.
• Max 256 snapshots per fileset.

Independent Fileset-based dynamic provisioning:

• A new independent file set is created for every new volume provisioned dynamically using independent file set storageclass.
• Suitable when the number of volumes required is less than 1000.
  This can be fixed by adding more file systems (254 File Systems per Spectrum Scale cluster.)
• Snapshot support on each PVC.
• Quota Support on each PVC.
  PVC size can be honoured on storage by means of file set quota.
• Inode management (The CSI is doing the inode Limit, this is is calculated using this formula: volume size/block size of the file system.)
• Your own iNodeLimit parameter.
• IBM Spectrum Scale administrator can leverage management benefits of file sets on these volumes, such as snapshots and backup.

Dependent Fileset-based dynamic provisioning:

• A new dependent file set is created for every new volume provisioned dynamically using the dependent file set storageclass.
• Suitable when the number of volumes required is less than 10000.
• Quota support. PVC size can be honoured on storage by means of fileset quota.
• Snapshot is done on the parent fileset.
• No need to manage inodes.

Note: There are two ways in which volumes can be provisioned

• Dynamic provisioning: The resource is created on the IBM Spectrum Scale file system dynamically during PVC creation.
• Static provisioning: The resource is pre-existing on the IBM Spectrum Scale file system and can be used to define a PV.

You can choose to create all of Storage Class, but it is also a good idea to know which of them are the correct choice for your environment. 🤔

Let’s configure som Storage Class: 👷

Sample Storage Class

• New with this version is that there I already a sample fileset based Storage Class created using the primary file system as the volBackendFs.
  We can use this to create other storage classes, this can be helpful since the remote cluster ID is provided.
• Use oc get storageclass ibm-spectrum-scale-sample -oyaml > storageClass_fileset.yaml to create a copy of this storage class. Then configure parameters as desired and create the configured storage class using the command below:

I will cover the following StorageClass

• Lightweight dynamic provisioning
• Fileset-based dynamic provisioning with independent filesets.

Note:

• You can create the storageclass from the OpenShift GUI.
• There is now support for Volume Expansion, Clones and SnapShot.
  Check out it out here:

Example for lightweight volume Storage Class:

1. On Storage Cluster: Check that the default fileset is created and linked,
   and then create the folder for lightweight volume, for example, lwdir.

The default fileset is primary-fileset-fs1–19727929243932xxx random number. if not changed in the CSI installation

On newer CSI/CNSA version this will be: primary-fileset-fs1–1972792924393295911

If the fileset doesn't exist, you can create it with the following command on Storage Cluster:

Create the fileset:

mmcrfileset gpfs01 spectrum-scale-csi-volume-store --inode-space new

Link the fileset to filesystem.

mmlinkfileset gpfs01 spectrum-scale-csi-volume-store -j /mnt/gpfs01/spectrum-scale-csi-volume-store

Create the directory: lwdir (only need to be done the first time)

mkdir -p /mnt/gpfs01/spectrum-scale-csi-volume-store/lwdir

3. Create the file: ibm-spectrum-scale-lt.yaml and copy in the example below.

Change the following:

• volBackendFs: your filesystem name on OCP Scale Cluster
• volDirBasePath: default: spectrum-scale-csi-volume-store/lwdir
  Remember to create the dir. example: lwdir

4. Apply the configuration to create StorageClass

oc apply -f ibm-spectrum-scale-csi-lt.yaml

Test with a Persistent Volume Claim.

1. Create a pvc-lt.yaml file with the following content.
   Note: the name of the storageClassName.

2. Create a PVC by issuing the following command:

oc apply -f pvc-lt.yaml

3. Check the status of PVC with oc get pvc --all-namespaces

The find PVC you created, the status should be bound and some seconds old.

If we check the Storage Cluster, we can see that we have the PVC in LWDIR.

cd /mnt/gpfs01/spectrum-scale-csi-volume-store/lwdir -la
pvc-f213c69b-1b27-44a2-aed5-3a3c10418c6c

Note: if you want the Storage Class to be the default.

• GUI: From the OCP GUI and Storage Class.
  Edit Annotations > KEY: storageclass.kubernetes.io/is-default-class : value true
• CLI: Or add this under Metadata when creating the Storage Class.

annotations:
  storageclass.kubernetes.io/is-default-class: ‘true’

Example for Independent fileset storageClass:

1. Create the file:ibm-spectrum-scale-fileset.yamland copy the example below.

If you have gone with the default, you only need to change the following:

• ClusterID to your StorageCluster/Owning Cluster.
  You can also get this from the auto-created CSI StorageClass.
  oc get storageclass -oyaml > storageClass_filesets.yaml
To collect it manually: run the following command to get it:
  curl -s -k https://storage-gui.com/scalemgmt/v2/cluster -u "csi-storage-gui-user:csi-storage-gui-password" | grep clusterId
• volBackendFs: Your filesystem name on OCP/CNSA Cluster
• GUI/UID: Default is 1000. add to change this
• VolumeExpansion: To enable the expansion of a volume, you must set the allowVolumeExpansion parameter to true in the StorageClass.

2. Apply the configuration to create StorageClass

oc apply -f ibm-spectrum-scale-csi-fileset.yaml

Test with a PVC claim

1. Create pvc-fileset.yaml

2. Create a PVC by issuing the following command:

oc apply -f pvc-fileset.yaml

3. Check the status of PVC with oc get pvc --all-namespaces

The find PVC you created, the status should be bound and it’s 25 seconds old.

• If we check the Storage Cluster, we can see that we have the PVC under /mnt/gpfs01

ls -la /mnt/gpfs01 | grep pvc-a9
pvc-a93d632e-62b8-4666-aa9c-4b0855420cca

• Check that the fileset is created.

mmlsfileset gpfs01 -L | grep pvc-a93pvc-a93d632e-62b8-4666-aa9c-4b0855420cca

• To check the quota, that should be 1Gb here.

mmrepquota -j gpfs01 --block-size G

Example for Dependent fileset storageClass:

Description of the fields

Please check out IBM Doc for updated function and information: config_csi_storagclass

• volBackendFs:
  Name of the file system under which the fileset should be created.
  The file system name is the name of the remotely mounted file system on the primary cluster.
• clusterId:
  Cluster-ID of the owning cluster.
• uid:
  uid/username that will be assigned to the fileset. This is optional. The uid/gid must exist on the IBM Spectrum Scale GUI node of accessing and owning clusters. The default value is 1000
• gid:
  gid/group name that will be assigned to the fileset. This is optional.
  The gid/group name must exist on the IBM Spectrum Scale GUI node. of the accessing and owning clusters. Default value is 1000
• filesetType:
  Default is independent. This is optional.
• parentFileset:
  Parent fileset name. Valid with filesetType=dependent. Default is root.
• inodeLimit
  inode limit for fileset. This is optional. Valid with filesetType=independent. If not specified, inodeLimit is calculated using formula Volume Size / Block size of filesystem.
• permissions (Support for kubernetes subPath)
  Permissions to be set on volume parent directory. This parameter is optional.
  
  Note: For subpath support, add permissions = 777 in the storageClass. Using 777 directory permission might pose security risk.
  Use 777 directory permission with caution.
  The use of incorrect permissions might lead to loss of data access.

Note: for more information check also out: config_csi_storagclass on KC

Setting the Primary FileSet

Note: This is applicable only if the primary filesystem from the same storage cluster is shared by two or more CNSA cluster

• After the CSI CR has been created by the CSI controller the primary file set needs to be set in order to avoid the naming conflict. Once this field has been added the CSI driver pods will be deleted and recreated one by one.
• Run oc edit csiscaleoperator -n ibm-spectrum-scale-csi and add the primaryFset field:

Ta-da! 👏

The IBM Spectrum Scale CNSA and IBM Spectrum Scale CSI installation should now be complete. 🎊

It was a little shorter post than last time, hopefully, this was helpful when you do the installation yourself, or at least help you understand more of the product and installation.

I will try to make some shorter articles soon, where we look into Grafana, Snapshot, several Filesystem/clusters and other topics that could be interesting.