Deploy Mayastor on GKE

Harsh Thakur
Harsh Thakur
Published in
5 min readMay 18, 2020
OpenEBS

Today I’ll be writing about another one of OpenEBS storage engines. I’ll be going through how to deploy Mayastor on GKE. If you’re new to OpenEBS, take a look at the concepts about cStor and how you can deploy cStor in my previous post .

Note: This storage engine is in alpha stage and should not be used for production workloads.

About the components

The control plane has a deployment named moac(Mother Of All containers) which is a controller based off of CSI(Container Storage Interface) spec and the data plane consists of a daemon set.

This daemon set consists of a component named Nexus which is similar to cStor target pod and Nexus’s children are equivalent of cStor pool pods. I highly recommend going through the docs to understand further and read about other components involved.

Setup

Similar to how things are done with cStor, we’ll be setting up our cluster with OS image as Ubuntu as that let’s us start iSCSI service. It is necessary for each node to have 2vCPU as the mayastor pod requires 1vCPU for it’s operations and the rest can used used by workloads.

Next, you can set labels for your nodes in the metadata section or you can do it manually through kubectl later.

We can now ssh into the nodes by running:

$ gcloud compute ssh <node name>

There are couple of things to do now.

  • Start the iSCSI service
sudo systemctl enable iscsid && sudo systemctl start iscsid
  • Set the hugePages. This is a Linux feature which lets OS use pages of more than the default size. Mayastor requires 2MB pages and there should be atleast 512 of those. Configuring more pages might not always help as those pages might not be used by the OS.
echo 512 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
  • Restart the Kubelet service. Unfortunately, this needs to be done or the hugePages aren’t detected which causes the daemon set to fail
sudo systemctl restart kubelet

Deploying components

Now that we’ve setup the nodes, we can get to deploying the components required by Mayastor . Since this is in alpha, we’ll have to apply them manually and there isn’t a helm chart yet. To begin with, clone the repository and move to the folder named deploy.

First, we have to create the namespace named mayastor, since other manifests of other components are configured to be deployed there.

kubectl create -f namespace.yaml

Next, we deploy the moac and the daemon set.

kubectl create -f moac-deployment.yaml
kubectl create -f mayastor-daemonset.yaml

Upon checking, all the components seem to be running fine.

PS: I configured kubectl to mayastor namespace.

Storage Ops

We can now create a storage pool and a new storage class.

Let’s create a disk and attach it to the worker node.

$ gcloud compute disks create disk1 --size=10GB --zone=us-central1-c
$ gcloud compute instances attach-disk gke-cluster-1-default-pool-1b15dcb6-5ct0 --disk disk2

We can proceed to deploying the storage pool. Only parameters that need to change are the node name and the disks. Notice how this is different from cStor as we don’t have to create Storage Pool Claims anymore.

$ kubectl create -f msp-node1.yaml

Now that we have a storage pool, we can create a Storage Class.

$ kubectl create -f sc.yaml

Developers can now use this Storage Class in our Persistent Volume Claims.

Deploying workloads

In the pvc.yaml, change the value of storage to 1Gi and apply it.

We can see that created a Persistent Volume.

If you look at the mayaStor volume created, it’ll display the state to be healthy.

We can now deploy any workload utilizing this volume.

We use the spec provided for Flexible I/O tester which helps us measure the performance.

After creating this pod, we can now test the performance of I/O.
Upon running the test as suggested in the docs:

$ kubectl exec -it fio — fio — name=benchtest — size=800m — filename=/volume/test — direct=1 — rw=randrw — ioengine=libaio — bs=4k — iodepth=16 — numjobs=1 — time_based — runtime=60

Disk utilization is 99.83% and R/W speed is around 430kB/s. In my test, Mayastor numbers are limited since 1 CPU is used by Mayastor and I have only one CPU allocated to the workload.

Summary

One of the reasons I find Mayastor interesting is that Nexus can be used to transform data which opens the door for lots of exciting uses.

I would like to thank the amazing community this project has and especially Jeffry Molanus from the team for clarifying all the things I mentioned in this blog.

Thanks for reading and feel free to drop a comment or a suggestion ! :)

--

--

Harsh Thakur
Harsh Thakur

I explore tech. Currently occupied in cloud native landscape.