Multi-tenant SaaS using Kubernetes Operators
Kubernetes multi-tenancy has been in discussions lately. For the specific case of Kubernetes Custom Resource Definitions (CRDs) and Operators, the discussions are ongoing at the moment in Kubernetes API Machinery SIG. Interesting ideas such nested API servers, recursive namespaces, per namespace CRD schema evolution, etc. are being discussed. Continuing this multi-tenancy theme, in this post we present how we are leveraging Namespaces and Platform-as-Code approach for building multi-tenant SaaS solution on a Kubernetes cluster.
This is a customer story where our goal was to build an e-learning solution on Kubernetes. The architecture demands ability to run multiple Moodle (Open source e-learning platform) instances on a single Kubernetes cluster, where each Moodle instance serves a separate client (tenant). A typical Moodle Stack includes Moodle software and MySQL backend.
To build this solution we are using Platform-as-Code (PaC) approach which is the process of aggregating multiple Kubernetes Operators for the purpose of creating and managing platforms declaratively using native Kubernetes interfaces. Here we are adding Moodle Operator and MySQL Operator to a vanilla Kubernetes cluster. We have built a Moodle Operator and are using MySQL Operator from the community. The Moodle Operator introduces Moodle CRD which supports installation of custom Moodle plugins through Spec definition. It internally uses custom resources from MySQL Operator.
We decided to leverage Kubernetes Namespaces to create isolation between Moodle stacks for different clients as shown below.
A namespace is created per tenant and entire Moodle platform stack is created in each namespace. The stack consists of a Moodle Custom Resource instance and a MysqlCluster Custom Resource instance. The Operators for Moodle and MySQL are deployed in the default namespace and are configured to watch all the namespaces so that they can react to object creations in any namespace. You can try this setup on Minikube by following these steps.
1. Namespace support in Operators: Primary multi-tenancy mechanism in Kubernetes is ‘Namespace’. A namespace provides logical boundary for creating various Kubernetes resources. When implementing an Operator, it is important to ensure that the Operator is able to work with namespaces. This amounts to essentially three things.
- First, deciding in which namespace the Operator itself is deployed.
- Second, deciding which namespaces the Operator should be monitoring.
- Third, ensuring that the Operator correctly handles namespace definition in the metadata property of Kubernetes objects.
For our use-case, both the Operators are deployed in the default namespace and they monitor creation of Custom Resources in all namespaces. They also correctly handle namespace definition specified in metadata of related custom resources.
a) it should support namespaces as identified in point 1 above;
b) it should be possible to take backups of MySQL instances;
c) it should be possible to connect to the MySQL database server created by a MySQL Custom Resource instance from a Moodle Custom Resource instance.
Both Oracle and PressLabs MySQL Operator seemed to satisfy requirements a) and b). However, for requirement c), we kept on running into an issue with Oracle’s Operator. With PressLabs’s Operator this was not the case. Overall we like the PressLabs Operator and have decided to use it in our solution.
3. Using Custom Resources: When working with a community Operator, one challenge you will face is figuring out the details of using various Custom Resources introduced by the Operator. For instance with the PressLabs Operator, we had to figure out that first a Secret object needs to be created and then its name needs to be passed in the Spec of the MysqlCluster Custom Resource. One needs to read through an Operator’s Github documentation to figure out this information. If you are using multiple Operators, this becomes a tedious process. We are developing KubePlus platform toolkit that will readily provide different kinds of information about Custom Resources introduced by different Operators in your cluster, in easy to find manner directly through ‘kubectl’. This information can be seen as a man-page for a specific custom resource which provides information required to consume it, such as pre-reqs, configuration options, supported operations, etc.
4. Custom Resource interdependency: When using multiple Custom Resources to build a Platform stack, you will need to figure out inputs/outputs for each Custom Resource and how they relate to each other to build a stack. For example, in our solution the Spec definition of Moodle Custom Resource includes specification of MySQL service name, username, and password. The values for these variables are based on the corresponding values used when creating a MysqlCluster Custom Resource instance.
In the situations where soft multi-tenancy is sufficient, such as in the described case of using a single cluster for running multiple instances of the same stack for different tenants, Kubernetes Namespaces provide adequate isolation mechanism. This type of Kubernetes use-case can also be described as Cloud-Enabled SaaS where Kubernetes is used to create multi-tenant hosting of a classic install-anywhere software. We have demonstrated that Platform-as-Code approach is beneficial for building such Cloud-Enabled SaaS.
If you enjoyed this post, you may like our upcoming Platform-as-Code handbook. To receive it, stop by our booth at South by SouthWest (SXSW) Startup crawl next week if you are in Austin. If not, register for our Webinar and we will send the handbook to you.