MayaOnline — A Synergized Tool to manage storage for stateful workloads on Kubernetes

August 27, 2018

In October 2017, the term MayaOnline was coined for the first time at MayaData Inc. With the growing usage of our open-sourced OpenEBS Container Attached Storage (CAS) engine for stateful workloads in Kubernetes, there was a need to visualize, monitor and manage the storage components of the various Kubernetes clusters, where our customers were deploying OpenEBS engine. We began our initial experiments on building a prototype for an enterprise SaaS platform and launched an alpha version in December 2017 at the Kubecon event.

The initial version of the product had the following features:

  • Import an existing OpenEBS-enabled K8s cluster.
  • A dedicated Prometheus engine which collected the storage capacity and I/O metrics from the OpenEBS volumes in a time-series format.
  • A dedicated alert-manager which configured the rules for the generation of alerts whenever the metrics reached the threshold limits.
  • A dedicated Grafana service with industry-standard dashboards for visualizing the metrics.
  • A dedicated chat-ops server which was integrated with Slack so that the cluster configuration and real-time alerts could be sent to the administrator’s slack channels.

We opened the portal for reviews and our users provided encouraging feedback which led us to launch the beta version by mid of May 2018 with additional features:

  • Integration with Weave Scope for better visualization and easy sifting through the various K8s resources including storage support in Weave Scope.
  • Support for OpenEBS-enabled OpenShift clusters.
  • A multi-tenant Prometheus and Grafana engine.
  • Additional dashboards for aggregate metrics.
  • A dedicated chatbot called MuleBot.
  • Enhanced user interface.

In the last couple of months, OpenEBS has evolved tremendously. A new open-source initiative i.e. Node Disk Manager (NDM) has been started for common management of K8S disks (physical and virtual) for on-premise and cloud deployments. OpenEBS 0.7 (read blog) now supports cStor — a faster and more resilient storage engine for enterprise workloads, in addition to its existing engine Jiva. Then there is Litmus which introduces chaos to test the resiliency of stateful workloads in K8s (read Chaos Engineering).

Our users are really excited with these developments and we are providing them with a new version of MayaOnline which integrates all of the above for providing data agility (read MDAP). Some of the cool new features include:

  • End-to-end storage visualization.
  • Ability to take snapshots and restore data from OpenEBS volumes.
  • Launch Litmus chaos tests on K8s applications.
  • Enhanced user interface.

Let me talk more about each of them.

End-to-End Storage Visualization

MayaOnline beta already had Scope integration where the user could get a holistic view of all the K8s storage constructs and resources like Persistent Volumes, Persistent Volume Claims and Storage Classes which she has configured to satisfy the storage needs of her workloads. The new MayaOnline goes one step further and integrates NDM into scope. Now a user can view/manage the connectivities between the PVs and the core storage components like StoragePools and Disks !! This provides the right perspective to the user to understand the complex relationships between the various resources which in turn facilitates easy management and debugging.

No other product in the market provides this level of control to the users.

Visualization between K8s Stateful app, PV, CAS components, StoragePool and Disks.

Snapshots and Restore

Ability to take application-consistent snapshots has always been the need of the hour for the developers and they expect all storage solutions to make provisions for the same. Along with the E2E visualization, a MayaOnline user can now browse/navigate to the volume of his choice and take snapshots with a single click of a button. What’s more, the snapshots themselves can be viewed as K8s resources and support control operations like creating clones and restoring them !!

Trigger snapshot control
Create clone control
Visualize snapshots as K8s resources

Litmus Integration

Chaos engineering is the buzzword these days and we at MayaData joined the bandwagon by developing Litmus — a-first-of-its-kind easy-to-develop Behavioral-Driven-Development Test framework where chaos tests can be written in plain English language !! Application developers and testers alike can use Litmus to induce real-time chaos like node failures, network failures, pod failures to name a few. MayaOnline integrates with Litmus and provides an easy interface for the developers and users to develop, modify, launch and view the results of Litmus tests to see how their applications fare in real-time chaos constraints.

Litmus Git repo configuration
Browse K8s Namespaces
Choose application for chaos
Run chaos test(s)
View chaos test results

Enhanced User Interface

As a part of our continuous efforts to improve the user experience, we ran through rapid design sprints with extended teams and sought feedback from external users. This led to a rebranding effort and the resultant new screens are more intuitive and user-friendly.

Connect new cluster from MayaOnline
Dashboard

In summary, MayaOnline is a single place for all the tools that are required for a developer/administrator to use K8s clusters and stateful applications. In the near future, we will be adding more DevOps-centric features like GitOps integration, CI-CD pipelines, blue-green deployments along with a fully functional end-to-end monitoring and management portal.

To conclude, we’d like to offer our sincere and humble thanks to you, our users and well-wishers without whom, this journey wouldn’t have been possible and we will strive to excel at meeting your expectations.

If you have not already tried out MayaOnline, please do so at https://mayaonline.io/. We would love to hear about your experiences.

Cheers !!