Stay simple and foolish with KubeStellar
By Jun Duan, Cheuk Lam, Mike Spreitzer, Paolo Dettori, Andy Anderson
There are a variety of solutions for the problem of workload scheduling across multiple Kubernetes clusters. Some of the solutions make the problem more complex and the some make the problem simpler.
KubeStellar is one that makes it simpler, as simple as a no-brainer.
This blog is part of a series of posts from the KubeStellar community.
The Simplified View
Below is a simplified view of KubeStellar. It can be seen as a black box, exposing APIs to two groups of roles, the vendors and the users.
In this simplified view, the lower side is the APIs exposed to the vendors. The vendors are supposed to bring their Kubernetes clusters, get the clusters onboard using the easy-to-use automation offered by KubeStellar, and use the vendor-facing APIs to tell KubeStellar how to reach the clusters.
The upper side is the API exposed to the users. Here ‘API’ is in its singular form, because there is only one radically simple API. The users can specify their requirements using this user-facing API.
The Vendor-facing APIs
Here is an example for one of the vendor-facing APIs, a Location object.
apiVersion: scheduling.kcp.io/v1alpha1
kind: Location
metadata:
name: aisle1
labels:
aisle: "1"
spec:
instanceSelector:
matchLabels:
aisle: "1"
...Let’s look at the spec.instanceSelector field of this Location. As mentioned in the opening of this article, this is where the vendors tell KubeStellar how to reach the clusters.
As you can guess: This Location tells KubeStellar that “the cluster is located in aisle 1”.
Really simple, right? If the you are interested in more details hidden behind the three dots of the spec, feel free to checkout the definition of the Location API. In addition, there is another SyncTarget API. SyncTarget API objects serve as mirrors of the Kubernetes clusters, and these objects are what selected by the aforementioned spec.instanceSelector field of Locations.
The User-facing API
Here is an example of the user-facing API, an EdgePlacement object.
apiVersion: edge.kcp.io/v1alpha1
kind: EdgePlacement
metadata:
name: optimized
spec:
namespaceSelector:
matchLabels:
optimizer: turbonomic
locationSelectors:
- matchLabels:
aisle: "1"
...The group and kind of the API contain the word ‘edge’, because KubeStellar was initially born for edge computing. The outlook, however, is not limited to the edge but has more broader horizon including other multi-cloud and hybrid cloud use cases.
Let’s focus on the spec of this EdgePlacement. As mentioned in the opening, this is where the users specify their requirements. In the context of multi-cluster scheduling, a requirement consists of two things: First, what is going to be scheduled; second, schedule to where.
As you can guess: The spec.namespaceSelector field expresses the ‘what’ part, by selecting a set of namespaces. The spec.locationSelectors field expresses the ‘where’ part, by selecting a set of Locations. The two fields together convey the requirement in full, which is “for each of the selected namespaces, schedule their contained stuff to each of the selected Locations.”
Really simple, right? Actually there is some other details which is worth your attention. What the spec.namespaceSelector field selects is a set of namespaces. In other words, the workload are namespaced objects in the selected namespaces. In addition to that, KubeStellar also allows users to specify non-namespaced (aka cluster-scoped) workloads. These details are hidden behind the three dots of the spec, but can be found here in the definition of the EdgePlacement API.
An Example Use Case
The vendors tell KubeStellar where are the clusters, the users tell KubeStellar what goes where. With these been told, KubeStellar behaves like a pure actuator for scheduling decisions, or a human without brain. Now multi-cluster scheduling literally becomes a ‘no-brainer’.
How about introduce some ‘brain’ to make the scheduling decisions? Here is where Turbonomic comes into play. Turbonomic is an optimization tool. It has many use cases listed in this page. One simplest use case is to identify which cluster(s) are overloaded, then makes scheduling decisions that aim to balance the workloads across the clusters.
Combing KubeStellar and Turbonomic, we have an example use case as shown below.
This use case implements a continuous optimization loop. Turbonomic collects information from the clusters, and makes scheduling decisions based on the collected information. KubeStellar accepts the scheduling decisions, and executes them against the clusters.
There is a recorded demo for this use case. In the demo, Turbonomic discovers that the CPU usage is imbalanced across two clusters. It makes a decision to move some workload from one cluster to another. KubeStellar then picks up the decision and executes it. After that, the CPU usage gets much more balanced among the two clusters.
Note that the demo adopts GitOps, meaning that Turbonomic writes the scheduling decision by creating a PR to a git repository on GitHub, and KubeStellar pulls the decision from the git repository once the PR is reviewed, approved, and merged. In addition to that, the manifests used in the demo (including the Locations, SyncTargets, EdgePlacements, and the workloads) are all available here.
Recap
The design of KubeStellar makes a choice to stay simple and foolish, while the choice itself is smart. It cares about execution of scheduling decisions, and only cares about it. It follows the ‘do one thing well’ philosophy. It enables modularity and interoperability.
The example use case is a synergy between KubeStellar and Turbonomic. But KubeStellar can certainly be integrated with any automation that can make scheduling decisions, or any person who has a brain like you. So don’t hesitate to checkout the documentations and code today, and get started with KubeStellar!
