Using IBM Spectrum Scale for storage in IBM Cloud Private

IBM Spectrum Scale

Published in

IBM Cloud

10 min readNov 27, 2017

IBM Spectrum Scale is a flexible software-defined storage that can be deployed as high performance file storage or a cost optimized large-scale content repository. IBM Spectrum Scale is deployed at the most demanding enterprises in the world for both high performance and high scale. Refer to https://www-03.ibm.com/systems/storage/spectrum/scale/ for more details. Features that make Spectrum Scale suitable for persistent volume storage include: performance, scalability, information lifecycle management, quota, encryption, compression, availability, and snapshots.

Storage for Containers

One of the main resources inside any cluster is storage. Before you can deploy most Kubernetes-based applications, you must first create storage from the underlying infrastructure. This provisioned storage allows for persistence of an application’s data. However, for an application to consume this storage, you must also create a volume. Volumes are used to make requests on behalf of an application to the provisioned storage. You can create volumes from Spectrum Scale that are used as persistent storage for applications.

This post describes how to configure IBM Spectrum Scale as a storage backend for POD volumes in IBM Cloud Private clusters. While there are multiple ways to use a persistent volume from IBM Spectrum Scale as a POD’s volume in IBM Cloud Private, this post describes how to use the Hostpath and NFS drivers.

Before you begin

In both scenarios(Hostpath and NFS), you need a IBM Spectrum Scale cluster and a persistent volume. You can also create the custom application used in this post using following steps.

Create a IBM Spectrum Scale cluster and designate several nodes as IBM Cloud Private worker nodes. In this example, dghostrhel2, dghostrhel3 and dghostrhel4 are IBM Cloud Private worker nodes

[root@dghostrhel2 ~]# mmlscluster
GPFS cluster information
========================
GPFS cluster name: dgcluster.dghost1
GPFS cluster id: 13644380956435301453
GPFS UID domain: dgcluster.dghost1
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp
Repository type: CCRNode Daemon node name IP address Admin node name Designation
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 
1 dghost1 192.168.122.96 dghost1 quorum-manager
2 dghost2 192.168.122.124 dghost2 quorum-manager
3 dghost3 192.168.122.80 dghost3 quorum-manager
4 dghost4 192.168.122.217 dghost4 quorum-manager
9 dghostrhel1 192.168.122.221 dghostrhel1
10 dghostrhel2 192.168.122.222 dghostrhel2
11 dghostrhel3 192.168.122.223 dghostrhel3
12 dghostrhel4 192.168.122.224 dghostrhel4
13 dghostrhel5 192.168.122.225 dghostrhel5

2. Create a volume on the gpfs0 filesystem and mount it on the /gpfs/gpfs0 path.

[root@dghostrhel1 ~]# mmlsfs gpfs0flag value description
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —
-f 8192 Minimum fragment size in bytes
-i 4096 Inode size in bytes
-I 16384 Indirect block size in bytes
-d nsd1;nsd2 Disks in file system
 …
 …
-A yes Automatic mount option
-o none Additional mount options
-T /gpfs/gpfs0 Default mount point
— mount-priority 0 Mount priority

3. Create the custom application. The parameter values and file names that are used in this example are also used in later steps.

3a. Create a directory named alpine4gpfs.

3b. Create the application script in the alpine4gpfs directory. The script creates a directory in the Docker container and generates random number of files in it. Name the file number_generator.sh

 #!/bin/sh
 
 #/gpfs is the mounted directory from host
 randomnum=$(( ( RANDOM % 20 ) + 1 ))
 iodir=/gpfs/${HOSTNAME}_${randomnum}
 if [[ -f /gpfs/thisisgpfs ]]
 then
 echo “Writing data into gpfs dir…”
 #Sleep for some time·
 sleep 10
 mkdir $iodir
 for i in `seq 1 $randomnum`
 do
 dd if=/dev/urandom of=/${iodir}/FILE${i} bs=512K count=10 &>/dev/null
 md5sum /${iodir}/FILE${i} > /${iodir}/FILE${i}.md5
 date > /${iodir}/FILE${i}.date
 done
 else
 echo “No GPFS Volume..Exiting”
 fi

Note : The above script checks for /gpfs/thisisgpfs file inside the container before generating data. Hence create the file ‘thisisgpfs’ inside fileset or nfs export which will be used as persistent volume for the sample application.

3c. Create the dockerfile in the alpine4gpfs directory.

 FROM alpine
 MAINTAINER Your_name
 ADD number_generator.sh /number_generator.sh
 RUN chmod +x /number_generator.sh
 ENTRYPOINT [“/number_generator.sh“]

3d. Create the custom application. From the alpine4gpfs directory, run these commands:

 docker login mycluster.icp:8500
 docker build -t alpine4gpfs .
 docker tag alpine4gpfs mycluster.icp:8500/admin/alpine4gpfs
 docker push mycluster.icp:8500/admin/alpine4gpfs

Note : The above command upload the image 'alpine4gpfs' in the 'admin' namespace. In case 'admin' namespace does not exist, The image upload fails with the error message ’unauthorized: authentication required’. One will need to create the required namespace for successfully uploading the image.
Steps to create the ‘admin’ namespace :
Login to IBM Cloud Private >>Manage>>Namespaces>>Click on “Create Namespace” then Create namespace with name ‘admin’

Using Storage from Spectrum Scale with IBM Cloud Private via the Hostpath driver

Prerequisites for this example:

IBM Spectrum Scale Cluster
IBM Cloud Private V2.1.0

You can choose to use some or all IBM Spectrum Scale client nodes as a IBM Cloud Private worker nodes. If not all IBM Cloud Private worker nodes are IBM Spectrum Scale clients, then you must use a node tag when you use volumes that you create from IBM Spectrum Scale. You can create volumes by using the directory or fileset commands from the IBM Spectrum Scale File system. Creating volumes by using fileset is beneficial because it lets you partition the file system to allow administrative operations at a finer granularity than the entire file system. For more information about fileset, see https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adv_filesets.htm.

1. Create the fileset that the persistent volume uses

1a. Create a fileset that is named ScaleCfCHostpath:

[root@dghostrhel2 ~]# mmcrfileset gpfs0 ScaleCfCHostpath

The output looks like this:

Fileset ScaleCfCHostpath created with id 2 root inode 33547.

1b. Link the fileset to the volume:

[root@dghostrhel2 ~]# mmlinkfileset gpfs0 ScaleCfCHostpath -J /gpfs/gpfs0/ScaleCfCHostpath

The output looks like this:

Fileset ScaleCfCHostpath linked at /gpfs/gpfs0/ScaleCfCHostpath

1c. Display the fileset details:

[root@dghostrhel2 ~]# mmlsfileset gpfs0

The output looks like this:

Filesets in file system ‘gpfs0’:
Name Status Path
root Linked /gpfs/gpfs0
ScaleCfCHostpath Linked /gpfs/gpfs0/ScaleCfCHostpath

The fileset ScaleCfCHostpath is mounted on the /gpfs/gpfs0/ScaleCfCHostpath path. This path is used to create the volume in IBM Cloud Private.

2. Create the Persistent volume

2a. Log in to the IBM Cloud Private management console.

2b. From the menu, click Platform > Storage.

2c. Click Create PersistentVolume.

2d. Enter the Persistent volume details. For this example, use the following values:

 Name — spectrumscalestorage
 Label — storage
 Value — spectrumscale
 Capacity — 1
 Unit — Gi
 Access Mode — ReadWriteMany
 Reclaim Policy — Retain
 Storage Type — Hostpath
 Key — Path ; Value — /gpfs/gpfs0/ScaleCfCHostpath

You can provide these values by using the fields and lists in the Create New PersistentVolume window as shown in these images:

2e. Click Create.

2f. From the PersistentVolume tab, review the status of the new persistent volume. Before you can use the persistent volume, its status must be Available.

3. Create the Persistent volume claim

3a. Click the PersistentVolumeClaim tab.

3b. Click Create PersistentVolumeClaim.

3c. Enter the volume details. For this example, use the following values:

 Name — spectrumscalevolume
 Storage Requests — 512
 Unit — Mi
 Access Mode — ReadWriteMany
 Volume Selector Label — storage
 Value — spectrumscale

3d. Click Create.

3e. From the PersistentVolumeClaim tab, review the status of the new persistent volume claim. The persistent volume claim should have a status of Bound, which means that the persistent volume claim is bound to the persistent volume that contains the label storage=spectrumscale and meets its storage request requirements.

Click PersistentVolume and review the changes to the persistent volume ‘spectrumscalestorage’.

You can see that the persistent volume claim ‘ spectrumscalevolume’ is bound to the persistent volume ‘spectrumscalestorage’. The persistent volume claim ‘spectrumscalevolume’ is available to an application or pod.

4. Deploy an application that uses the new volume claim
A container is created for each instance of the application. Each container has a storage volume, and the volume contains a directory that stores the script’s random data.

4a. From the IBM Cloud Private console, click the menu and then click Workloads.

4b. From the Workloads page, click Jobs > Batch jobs.

4c. Click Create Job.

4d. Enter the job details. You must provide values for several parameters:

On the General tab, specify the job name and number of instances:

Name — workloadwithalpine
Completions — 1

On the Container Settings tab, specify the container Name and the Image to use.

Name — alpinecontainer
Image — mycluster.icp:8500/admin/alpine4gpfs

On the Volumes tab, specify the following values:

Name — spectrumscalevolume
Volume — spectrumscalevolume
Path — /gpfs

4e. Review the workloadwithalpine job details. The workloadwithalpine application with an attached volume should be running.

If you click the application name, you can see that instances of the workloadwithalpine application is created.

Each instance of the application creates a directory of the same name in attached volume. Data generated by the applications is visible on Spectrum Scale clients, as shown in the following example:

[root@dghostrhel2 ScaleCfCHostpath]# pwd
/gpfs/gpfs0/ScaleCfCHostpath[root@dghostrhel2 ScaleCfCHostpath]# ls -1
thisisgpfs
workloadwithalpine-kt4sn_17

Using Storage from Spectrum Scale with IBM Cloud Private via the NFS driver

Prerequisites for this example:

IBM Spectrum Scale Cluster with NFS enabled protocol nodes
IBM Cloud Private V2.1.0

When you use this configuration method, you can use a IBM Cloud Private worker that is different than the IBM Spectrum Scale client node. IBM Spectrum Scale file systems can be exported via NFS in a highly available manner, and you can use NFS exports via Cluster Export Services (CES) as storage volumes. You can export either a directory or fileset from IBM Spectrum Scale File system via NFS. If you use load balancing, you can use DNS instead of an individual CES IP address. You should use DNS for load balancing. For more information about using CES with Spectrum Scale, see https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_ces.htm.

Before you begin, confirm that NFS is running on a host:

[root@dghostrhel2 ~]# mmces service list -a
Enabled services: NFS
dghostrhel2: NFS is running

Also obtain the list of CES IP addresses in Spectrum Scale:

[root@dghostrhel2 ~]# mmces address listAddress         Node                          Group      Attribute
--------------------------------------------------------------------
192.168.122.229 dghostrhel2                   none       none

1. Create the Spectrum Scale NFS export that the persistent volume uses

1a. Create a fileset that is named scaleCFCnfs.

[root@dghostrhel2 ~]# mmcrfileset gpfs0 scaleCfCnfs

The output looks like this:

Fileset scaleCfCnfs created with id 2 root inode 7767.

1b. Link the fileset to the volume:

[root@dghostrhel1 ~]# mmlinkfileset gpfs0 scaleCfCnfs -J /gpfs/gpfs0/scaleCfCnfs

The output looks like this:

Fileset scaleCfCnfs linked at /gpfs/gpfs0/scaleCfCnfs

1c. Display the fileset details:

[root@dghostrhel2 ~]# mmlsfileset gpfs0

The output looks like this:

Filesets in file system ‘gpfs0’:
Name Status Path
root Linked /gpfs/gpfs0
ScaleCfCHostpath Linked /gpfs/gpfs0/ScaleCfCHostpath
scaleCfCnfs Linked /gpfs/gpfs0/scaleCfCnfs

1d. Create the NFS export.

[root@dghostrhel2 ~]# mmnfs export add /gpfs/gpfs0/scaleCfCnfs -c '*(Access_Type=RW,Squash=no_root_squash,SecType=sys,Protocols=3:4)'

The output looks like this:

The NFS export was created successfully.

1e. Display the export details.

[root@dghostrhel2 ~]# mmnfs export list

The output looks like this:

Path Delegations Clients
— — — — — — — — — — — — — — — — — — — — — — 
/gpfs/gpfs0/scaleCfCnfs NONE *

2. Create the Persistent volume

2a. Log in to the IBM Cloud Private management console.

2b. From the navigation menu, click Platform> Storage.

2c. Click Create PersistentVolume.

2d. Enter the persistent volume details. For this example, use the following values:

 Name — spectrumscalenfsstorage
 Label — storage=spectrumscalenfs
 Capacity — 1
 Unit — Gi
 Access Mode — ReadWriteMany
 Reclaim Policy — Retain
 Storage Type — NFS
 Key — server ; Value — your_CES_IP_address 
 Key — path ; Value — /gpfs/gpfs0/scaleCfCnfs

2e. Click Create.

2f. From the PersistentVolume tab, review the status of the new volume. Before you can use the storage, its status must be Available.

3. Create the Persistent volume claim.
Follow the steps to create a persistent volume claim in the “Using Storage from IBM Spectrum Scale with IBM Cloud Private via Hostpath driver” section
Note: Use name=spectrumscalenfsvolume and Label ’storage=spectrumscalenfs’

4. Deploy the application
Follow the steps to deploy an application in the “Using Storage from IBM Spectrum Scale with IBM Cloud Private via Hostpath driver” section.
Note : Use volumename=spectrumscalenfsvolume

You can view the new batch job on the Batch page.

The data generated by applications is visible on the IBM Spectrum Scale nodes.

[root@dghostrhel2 scaleCfCnfs]# pwd
/gpfs/gpfs0/scaleCfCnfs[root@dghostrhel2 scaleCfCnfs]# ls -1
thisisgpfs
workloadwithalpine-617vv_5

Using IBM Spectrum Scale for storage in IBM Cloud Private

IBM Spectrum Scale

Storage for Containers

Before you begin

Using Storage from Spectrum Scale with IBM Cloud Private via the Hostpath driver

Using Storage from Spectrum Scale with IBM Cloud Private via the NFS driver

Written by Deepak Ghuge