Fundamentals of IBM Cloud Storage Solutions

Falk Pollok, Parijat Dube

Introduction

With over 40,000 Github stars and adoption in cloud platforms from IBM Cloud Kubernetes Service, Amazon Elastic Container Service for Kubernetes (EKS) over Azure Kubernetes Service (AKS) to obviously Google Kubernetes Engine (GKE), Kubernetes has become one of the most prominent cloud technologies. Besides these public cloud examples it has similarly advanced in on-premise and hybrid cloud deployments from IBM Cloud Private (ICP) to GKE on-prem and Redhat’s OpenShift. Thus, it comes at no surprise that it is also heavily used within IBM Research where we have successfully adapted it for AI workloads. However, microservice architectures generally require processes and hence containers to be stateless which is for instance demanded in point 6 of the Twelve Factor App guidelines. Sending data over the network is one of the main bottlenecks to avoid in any data-centric workload from Big Data jobs such as Spark-based data science jobs to training in deep learning and deep reinforcement learning up to neural network visualization. Storage solutions in Kubernetes have seen lots of changes over recent versions leading to the current solution of Persistent Volume Claims and Flex Drivers as well as specialized storage drivers like Intel’s VCK. Rather than focusing on a historical perspective we will in this article follow a more hands-on approach and present different storage options as well as a brief future outlook on this evolving field.

We will first look at the three traditional storage options cloud object storage (COS), file storage (NFS) and block storage (ext4 via iSCSI) as the main topic of this article and then briefly touch upon AI-specific considerations and evolving technologies.

Basic Volume Provisioning Mechanisms in Kubernetes

Provisioning Volumes with Persistent Volumes and Persistent Volume Claims

There are two important objects for Kubernetes volumes, Persistent Volumes (PV) and Persistent Volume Claims (PVC), which facilitates a separation of concerns: Administrators can provision PVs and developers can request storage via PVCs. Volumes can either be provisioned statically by manually creating these objects or they can be provisioned dynamically through a storage class, in which case the developer only creates a PVC that refers to a specific storage class and Kubernetes will handle the provisioning dynamically through storage drivers.

Originally, volume plugins had to be in-tree, i.e. they had to be compiled with Kubernetes which was alleviated by so called Flex volumes which are the current state-of-the-art in productive used and will be used in this blog post. Flex volumes allow external volume plugins, but still generally require root node access since they require dependencies to be installed on the workers.

The Cloud Object Storage driver we discuss below, for instance, depends on s3fs binaries and will deploy them by launching a daemonset that runs one pod on each worker node that will then open a tunnel into the worker itself (which requires privileged access) to copy its binaries.

A better approach in the near future will be using CSI drivers which will run completely containerized and thus not depend on advanced privileges. Furthermore, it is an independent standard that also applies to other cloud orchestrators (COs) like Docker and Mesos and it will be used through the same Kubernetes primitives mentioned above (PVs, PVCs and storage classes), so most of this blog post series should be applicable even after the transition.

File Storage

File storage is the easiest to deploy, since it is supported by default — we do not need to install drivers and can directly provision a Persistent Volume Claim with it. When you list your storage classes (and grep for file storage in particular) you will find that there are different QoS classes:

$ kubectl get sc | grep ibmc-file
default                    ibm.io/ibmc-file    24d
ibmc-file-bronze           ibm.io/ibmc-file    24d
ibmc-file-custom           ibm.io/ibmc-file    24d
ibmc-file-gold             ibm.io/ibmc-file    24d
ibmc-file-retain-bronze    ibm.io/ibmc-file    24d
ibmc-file-retain-custom    ibm.io/ibmc-file    24d
ibmc-file-retain-gold      ibm.io/ibmc-file    24d
ibmc-file-retain-silver    ibm.io/ibmc-file    24d
ibmc-file-silver           ibm.io/ibmc-file    24d

The bronze, silver and gold classes are so called SoftLayer endurance storage (roughly meaning it determines IOPS by size and provides support for snapshots and replication) — you can find their exact specs in the storage class reference. Custom storage allows one to specify with fine granularity how many IOPS are required. The underlying hard disk type is then determined by the IOPS/GB ratio (<= 0.3 SATA, >0.3 SSD). Retain determines whether your files are kept when you delete the corresponding PVC.

Once you have determined your storage class you can create a PVC as follows:

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfspvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 41Gi
  storageClassName: ibmc-file-gold
EOF

We will first finish showing how to create PVCs for each storage type and then demonstrate how to mount them inside a pod.

Block Storage

Block storage is slightly more involved, but also very easy to deploy, since its helm charts are already provided in the IBM Cloud registry. As a result, we can just add the repository and run the following commands to obtain a new set of storage classes:

helm init
helm repo add ibm  https://registry.bluemix.net/helm/ibm
helm repo update
helm install ibm/ibmcloud-block-storage-plugin

If you want to search all available helm charts, run helm search. We can then provision a PVC just like before:

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: regpvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: ibmc-block-gold
EOF

Cloud Object Storage

S3, the Simple Storage Service, originated as Amazon’s fifth cloud product over a decade ago. According to SimilarTech it is used by over 200k websites, it is also the central storage component for Netflix (which developed S3mper to provide a consistent secondary index on top of an eventually consistent storage) as well as Reddit, Pinterest, Tumblr and others. Arguably, more important than the success of an individual product is its adoption as an API. With cloud object storage solutions from many different cloud vendors adapting it, it has been one of the first vendor-independent cloud standards. IBM’s Cloud Object Storage is S3 compatible and can thus be used with any S3-compatible tooling. In the following, we will setup a COS instance, present important tools and discuss how to comfortably support it within a Kubernetes cluster through FLEX drivers.

Setup of COS

Search for “Cloud Object Storage” in IBM Cloud, click on the “Object Storage” infrastructure service.

Click on “Create”. You can use default parameters.

Afterwards, you need to create new credentials. Please make sure to set {"HMAC":true} as inline configuration parameter. This will make sure that an access key ID and secret access key pair are created that we can use for a Watson Machine Learning manifest as well as managing our COS buckets with the AWS CLI.

Afterwards, you can switch to the Buckets tab and create a new bucket. Please make sure that this name needs to be globally unique.

Once a bucket exists, we can manually add files to it. The image depicts the way to do so via the GUI.

For example, we could add the MNIST dataset.

Cloud Object Storage Tooling

AWS CLI

Once you have a COS instance, you can access it through the S3 part of the AWS CLI. Besides the standard file commands cp (copy), ls (list), mv (move) and rm (remove) known from unixoid operating systems it allows us to sync files, make buckets (mb) and remove them (rb) [interested readers find the less used commands presign and website in the documentation]. In practice, this will look as follows:

• Empty COS bucket

aws s3 rm --recursive --endpoint-url https://s3-api.us-geo.objectstorage.softlayer.net s3://<bucket_name>

• List COS bucket content

aws s3 ls --endpoint-url https://s3-api.us-geo.objectstorage.softlayer.net s3://<bucket_name>

• Copy file to COS (also works to download COS files if parameters are flipped)

aws s3 cp --endpoint-url https://s3-api.dal-us-geo.objectstorage.service.networklayer.com <local_file> s3://<target_bucket>

• Upload all files from current directory to COS

aws s3 cp --endpoint-url https://s3-api.dal-us-geo.objectstorage.service.networklayer.com --recursive . s3://<target_bucket>

To specify the credentials to use you can define the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY either globally or just in front of the command, e.g.

AWS_ACCESS_KEY_ID=<AWS_ACCESS_KEY> AWS_SECRET_ACCESS_KEY= <AWS_SECRET_ACCESS_KEY> aws s3 ls --endpoint-url https://s3-api.us-geo.objectstorage.softlayer.net s3://<BUCKET_NAME>

Alternatively, you can modify ~/.aws/credentials to include different keys following the pattern

[<PROFILE_NAME>]
aws_access_key_id = <AWS_ACCESS_KEY>
aws_secret_access_key = <AWS_SECRET_ACCESS_KEY>

The entry where <PROFILE_NAME> is default is used if you do not specify a profile in the CLI command. Other entries can be used as follows:

aws s3 ls --endpoint-url https://s3-api.us-geo.objectstorage.softlayer.net --profile <PROFILE_NAME> s3://<BUCKET_NAME>

Mounting COS to file system via s3fs and goofys

Instead of always specifying all parameters and the exact bucket name and performing all operations through the AWS CLI, it is more convenient to mount the bucket as a folder onto the existing file system. This can be done via s3fs or goofys. To mount a bucket via goofys just create a file ~/.aws/credentials that contains the credentials like this:

[default]
aws_access_key_id = <ACCESS_KEY>
aws_secret_access_key = <SECRET_ACCESS_KEY>

Download goofys with wget http://bit.ly/goofys-latest and then mount the volume via

./goofys-latest --endpoint=https://s3-api.us-geo.objectstorage.softlayer.net <BUCKET_NAME> ~/testmount

(assuming you have created the mount target via mkdir ~/testmount and made goofys executable via chmod +x goofys-latest)

You can unmount the volume via sudo umount ~/testmount/

Alternatively, you can use s3fs. Create a credentials file ~/.cos_creds with:

<ACCESS_KEY>:<SECRET_ACCESS_KEY>

Make sure neither your group nor others have access rights to this file, e.g. via chmod o-rwx ~/.cos_creds You can then mount the bucket via

s3fs dlaas-ci-tf-training-data-us-standard ~/testmount -o passwd_file= ~/.cos_creds -o url=https://s3-api.us-geo.objectstorage.softlayer.net -o use_path_request_style

Note that s3fs can optionally provide extensive logging information:

s3fs dlaas-ci-tf-training-data-us-standard ~/testmount -o passwd_file= ~/.cos_creds -o dbglevel=info -f -o curldbg -o url=https://s3-api.us-geo.objectstorage.softlayer.net -o use_path_request_style &

More information about s3fs can e.g. be found here.

In simple test environments it might be sufficient to mount the folder as a host volume, e.g. into Minikube via minikube mount ~/testmount:/cosdata [Note: This will spawn a daemon that will keep running so either put it in background or continue in new terminal] To test you could then ssh into Minikube (minikube ssh) and list the files with ls /cosdata Then in turn you can mount the directory from within the Minikube VM into a pod via hostPath binding:

kubectl create -f proxymounttest.yml

with proxymounttest.yml:

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
spec:
  containers:
  - name: ubuntu
    image: ubuntu:14.04
    imagePullPolicy: IfNotPresent
    stdin: true
    stdinOnce: true
    tty: true
    workingDir: /cosdata
    volumeMounts:
      - mountPath: /cosdata
        name: host-mount
  volumes:
    - name: host-mount
      hostPath:
        path: /cosdata

Of course, you could achieve the same through a PVC. Both approaches lead to the result that you can exec into the pod (kubectl exec -it ubuntu -- /bin/bash) and run ls /cosdata to list the folder just like we did in the Minikube VM.

However, for Kubernetes clusters in production it is more desirable to properly mount volumes via drivers and hence we will discuss in the following how to use a Flex driver.

Cloud Object Storage: Driver Setup

Now that the Helm charts are available in IBM Cloud it suffices to use them to obtain storage classes for COS, i.e.

helm repo add stage https://registry.stage1.ng.bluemix.net/helm/ibm
helm repo update
helm fetch --untar stage/ibmcloud-object-storage-plugin
helm plugin install ibmcloud-object-storage-plugin/helm-ibmc
helm init
kubectl get pod -n kube-system | grep tiller
# Wait & check until state is Running
helm ibmc install stage/ibmcloud-object-storage-plugin –f
                  ibmcloud-object-storage-plugin/ibm/values.yaml

You can then list the new storage classes: kubectl get sc

Cloud Object Storage: Manual Driver Setup

IBM has released an open source COS plugin for Kubernetes. It is currently the most difficult to setup of all classes, but it is not unreasonable to expect that helm charts for COS are soon added to the IBM repository after which the setup will be as easy as block storage. Still, it is a good demonstration, since it shows how to load custom drivers into IBM Cloud. The manual setup needs custom images, so you need credentials for either an IBM Cloud container registry or for a private Docker registry. The steps below assume that you have set the environment variables DOCKER_REPO, DOCKER_REPO_USER, DOCKER_REPO_PASS and DOCKER_NAMESPACE to your registry credentials as well as docker namespace.

Let us afterwards clone and compile the driver:

git clone https://github.com/IBM/ibmcloud-object-storage-plugin.git
cd ibmcloud-object-storage-plugin/deploy/binary-build-and-deploy-scripts/
./build-all.sh

We then need to login to our registry and push the images we just built.

docker login --username=${DOCKER_REPO_USER} --password=${DOCKER_REPO_PASS} https://${DOCKER_REPO}
docker tag ibmcloud-object-storage-deployer:v001 $DOCKER_REPO/$DOCKER_NAMESPACE/ibmcloud-object-storage-deployer:v001
docker tag ibmcloud-object-storage-plugin:v001 $DOCKER_REPO/$DOCKER_NAMESPACE/ibmcloud-object-storage-plugin:v001
docker push $DOCKER_REPO/$DOCKER_NAMESPACE/ibmcloud-object-storage-deployer:v001
docker push $DOCKER_REPO/$DOCKER_NAMESPACE/ibmcloud-object-storage-plugin:v001

Now we need to deploy the plugin and driver. As a first step, we need to adapt the YAML descriptors to refer to our docker registry and namespace. Since we do this via sed and macOS does not use GNU sed by default, we use a variable CMD_SED to point to the right command. On macOS this assumes you installed GNU sed as gsed, e.g. via brew. We also need to create a docker-registry secret on the Kubernetes cluster so it can authenticate against the container registry and pull the images. Finally, we can deploy the plugin, provisioner and storage class.

operating_system=$(uname)
if [[ "$operating_system" == 'Linux' ]]; then
    CMD_SED=sed
elif [[ "$operating_system" == 'Darwin' ]]; then
    CMD_SED=gsed
fi
# Replace image tag in yaml descriptors to point to registry and namespace
$CMD_SED -i "s/image: \"ibmcloud-object-storage-deployer:v001\"/image: \"$DOCKER_REPO\/$DOCKER_NAMESPACE\/ibmcloud-object-storage-deployer:v001\"/g" deploy-plugin.yaml
$CMD_SED -i "s/image: \"ibmcloud-object-storage-plugin:v001\"/image: \"$DOCKER_REPO\/$DOCKER_NAMESPACE\/ibmcloud-object-storage-plugin:v001\"/g" deploy-provisioner.yaml
# Create secret, then deploy daemonset and plugin
kubectl create secret docker-registry regcred --docker-server=${DOCKER_REPO} --docker-username=${DOCKER_REPO_USER} --docker-password=${DOCKER_REPO_PASS} --docker-email=unknown@docker.io -n kube-system
kubectl create -f deploy-plugin.yaml
kubectl create -f deploy-provisioner.yaml
cd ..
kubectl create -f ibmc-s3fs-standard-StorageClass.yaml
kubectl create -f ibmc-s3fs-standard-StorageClass.yaml

Provision Cloud Object Storage PVC and Test Pod

With the driver it is now easy to create a PVC, but unlike the previous approaches we need a secret to hold our COS credentials. Hence, we create the secret first:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
type: ibm/ibmc-s3fs
metadata:
  name: test-secret
  namespace: default
data:
  access-key: <AWS_ACCESS_KEY>
  secret-key: <AWS_SECRET_ACCESS_KEY>
EOF

Please note that the credentials need to be Base64 encoded, so please encode them via echo -n "<SECRET>" | base64

We can then create the PVC itself:

kubectl apply -f - <<EOF
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: s3fs-test-ds-pvc
  namespace: default
  annotations:
    volume.beta.kubernetes.io/storage-class: "ibmc-s3fs-standard"
    ibm.io/auto-create-bucket: "true"
    ibm.io/auto-delete-bucket: "true"
    ibm.io/bucket: "<unique bucket_name>"
    ibm.io/endpoint: "https://s3-api.us-geo.objectstorage.softlayer.net"
    ibm.io/region: "us-standard"
    ibm.io/secret-name: "test-secret"
spec:
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 40Gi
EOF

Now that all PVCs have been created we can create a pod that mounts all of them:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: s3fs-test-pod
  namespace: default
spec:
  containers:
  - name: s3fs-test-container
    image: anaudiyal/infinite-loop
    volumeMounts:
    - mountPath: "/cos"
      name: s3fs-test-volume
    - mountPath: "/block"
      name: block-test-volume
    - mountPath: "/nfs"
      name: nfs-test-volume
  volumes:
  - name: s3fs-test-volume
    persistentVolumeClaim:
      claimName: s3fs-test-pvc
  - name: block-test-volume
    persistentVolumeClaim:
      claimName: regpvc
  - name: nfs-test-volume
    persistentVolumeClaim:
      claimName: nfspvc
EOF

AI-Specific Storage Solutions: Intel vck

Intel vck (formerly KVC — Kubernetes Volume Controller) is being developed by Intel AI and custom-tailored for AI workloads. It provides access to a diverse set of data sources through a Custom Resource Definition the inner workings of which have been outlined in a blog post by them. Instead of IBM Cloud we tried this in a local DIND Kubernetes cluster. In order to set it up execute the following:

kubectl create namespace vckns
kubectl config set-context $(kubectl config current-context) --namespace=vckns
git clone https://github.com/IntelAI/vck.git && cd vck
helm init
# Wait until kubectl get pod -n kube-system | grep tiller shows Running state
# Modify helm-charts/kube-volume-contoller/values.yaml to use valid tag from https://hub.docker.com/r/volumecontroller/kube-volume-controller/tags/
helm install helm-charts/kube-volume-controller/ -n vck --wait --set namespace=vckns
kubectl get crd
export AWS_ACCESS_KEY_ID=<aws_access_key>
export AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>
kubectl create secret generic aws-creds --from-literal=awsAccessKeyID=${AWS_ACCESS_KEY_ID} --from-literal=awsSecretAccessKey=${AWS_SECRET_ACCESS_KEY}
kubectl create -f resources/customresources/s3/one-vc.yaml
# Content of resources/customresources/s3/one-vc.yaml:
apiVersion: vck.intelai.org/v1alpha1
kind: VolumeManager
metadata:
  name: vck-example1
  namespace: vckns
spec:
  volumeConfigs:
    - id: "vol1"
      replicas: 1
      sourceType: "S3"
      accessMode: "ReadWriteOnce"
      nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - <SOME_K8S_WORKER_NODE_NAME>
      capacity: 5Gi
      labels:
        key1: val1
        key2: val2
      options:
        endpointURL: https://s3-api.us-geo.objectstorage.softlayer.net
        awsCredentialsSecretName: aws-creds
        sourceURL: "s3://<BUCKET_NAME>/"
kubectl create -f resources/pods/vck-pod.yaml
# Content of resources/pods/vck-pod.yaml:
apiVersion: v1
kind: Pod
metadata:
  name: vck-claim-pod
spec:
  affinity:
    # nodeAffinity and hostPath below were copied from output of
    # kubectl get volumemanager vck-example1 -o yaml
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: vck.intelai.org/vckns-vck-example1-vol1
            operator: Exists
  volumes:
    - name: dataset-claim
      hostPath:
        path: /var/datasets/vck-resource-<LONG_ID>
  containers:
  - image: busybox
    command: ["/bin/sh"]
    args: ["-c", "sleep 1d"]
    name: vck-sleep
    volumeMounts:
    - mountPath: /var/data
      name: dataset-claim

You can afterwards exec into the pod with kubectl exec -it vck-claim-pod sh and list the bucket content with ls /var/data.