Storage engine benchmark: Mayastor vs cStor
OpenEBS is a CNCF project focused on Container Attached Storage. cStor has been the storage engine of choice for a while but recently the team has created Mayastor(which is in alpha). Today, we’ll look at the performance of both of these.
If you want to learn how to deploy these engines, please look at my previous post on deploying cStor and Mayastor.
Setup
This was done on a GKE cluster with 3 worker nodes. Worker nodes had:
- Ubuntu as base image
- Local SSD of size 375GB attached
- 8 vCPU each
- 30GB RAM each
Basically, it was 3 of n1-standard-8 instances Google Cloud Platform offers.
Both the storage engines will run over iSCSI protocol. I’ve increased the number of replicas with each storage engine to look at the difference in performance when a replica increases.
Mayastor
After creating the Mayastor storage pool, I created 3 storage classes. Each storage class defines the number of replicas needed.
Next, I created a PVC with each of the storage class and deployed the benchmark tool which is Flexible I/O tester.
Let’s look at the stats when Mayastor is deployed with one replica. The command used for all the tests was this:
kubectl exec -it fio -- fio --name=benchtest --size=800m --filename=/volume/test --direct=1 --rw=randrw --ioengine=libaio --bs=4k --iodepth=16 --numjobs=1 --time_based --runtime=60
As you can see, it has high disk utilization and R/W speed is roughly 96MB/s .
Now let’s see when we have 2 replicas, the change in performance.
R/W speed has dropped to 75MB/s.
Let’s add another replica.
We can see that R/W speed has dropped to 67MB/s which is not bad compared to two replicas.
cStor
Similar to above setup, I created 3 storage classes based on the number of replicas.
Then, I created a PVC and deployed the benchmark tool claiming the PVC as before.
R/W speed was around 7.4MB/s , which means Mayastor outperformed cStor by approximately 13 times.
After adding another replica.
There wasn’t much difference, R/W speed dropped to 7.2 MB/s.
With 3 replicas, we have :
R/W dropped to 6.8MB/s .
The performance drop in both engines is due to synchronous replication which ensures consistent replicas. Although, asynchronous replicas aren’t directly supported, it is possible to achieve it in cStor by taking a snapshot which also let’s us know the time at which the snapshot was taken.
Why the difference?
Mayastor was written in Rust with performance in mind. It utilizes DPDK and advanced polling and queuing techniques to improve performance. If you’re still curious, go through this.
Summary
Clearly, Mayastor looks very promising for a 0.1 release. I believe once features like snapshots, replication with parity and others get added, it will not only make this engine the obvious choice but also have everyone considering Container Attached Storage for their workloads.
Thanks to the OpenEBS team for all the support they’ve provided since the beginning.
