How we manage clusters by extending the Kubernetes API
At BESTSELLER we run multiple Kubernetes clusters in multiple clouds, which gives us some assurance that even if one provider or one region is degraded we are still able to serve our customers. But multiple clusters, multiple clouds and multiple teams can be a bit difficult to grasp as an engineer. That is why we decided to use the extendability of the Kubernetes API to create a cluster-registry.
In this post, we will cover how to combine Custom Resource Definitions (CRD) with Admission controllers to gain control of your custom Kubernetes Resources.
What is a CRD and Admission Controller
To understand CRDs, we need to understand the basic concept of resources in Kubernetes.
- A resource is an API endpoint where you can store API objects of any kind.
- A custom resource allows you to define your own API objects, and thus creating your own Kubernetes kind just like Deployments or Statefulsets.
In short, the Custom Resource Definition is where you define your Custom Resource that extends Kubernetes’ default capabilities.
While the CRDs extend the Kubernetes functionality, Admission controllers govern and enforce how the cluster is used. They can be thought of as a gatekeeper that intercepts (authenticated) API requests and may change the request object or deny the request altogether.
There are two types of Admission controllers; validating and mutating. Mutating admission webhooks are invoked first and can modify objects sent to the API server to enforce custom defaults. After all object modifications are complete, validating admission webhooks are invoked which runs logic to validate the incoming resource. In case the validation webhook rejects the request, the Kubernetes API returns a failed HTTP response to the user.
Creating our cluster specification
Let’s start with the easiest part, creating our custom resource definition, in this case, our cluster specification template.
From the simplified example above we have defined a new API group extreme.bestseller.com and in that group, our CRD clusters.extreme.bestseller.com is stored.
We have defined 3 fields in our clusters specs, Node Count, Cloud and Contact Person where the first two are required.
Implementing the actual CRD is as easy as:
kubectl apply -f ourcrd.yaml
Time to create our first cluster object! More yaml coming up.
Apply it to our cluster:
kubectl apply -f firstcluster.yaml
Now we can get our clusters with `kubectl` just as any other Kubernetes kind:
> kubectl get clustersNAME CONTACTPERSONdestinationaarhus-techblog Peter Brøndum
With our cluster spec and storage in place, it is time for the fun part.
The Admission Controller
The admission controller, in this case, a mutating webhook, consists of two elements:
- A MutatingWebhookConfiguration, which defines which resources is subject to mutation and which mutating service to call.
- An admission webhook server, which does the mutation.
First up is the MutatingWebhookConfiguration. We can divide this into two blocks. The first is clientConfig. Here we configure which admission webhook service to call (can be an external service as well). Next is the rules, where we specify that mutation can only happen on Create and Update requests to the Kubernetes API and only on our Cluster resources.
With this in place, we need to create the actual mutation logic.
Before we deep dive into the code, I have chosen to write this in `Go` as it has a native client for Kubernetes. That being said, you could do this in the language of your choosing. The only requirement is to create a web server that serves a TLS endpoint and accepts and responds with JSON.
This will be a simplified example, and I have tried to squeeze everything into one file. In essence, what we are aiming at is to:
- Receive a JSON request, in Kubernetes terms an AdmissionReview.
- Do our mutation logic.
- Return a JSON response, again in the format of an AdmissionReview, which tells Kubernetes what to mutate.
Yes! you are correct, it is actually Kubernetes that does the mutation.
To the code!
In short, the example creates a single HTTP endpoint. When called, it will unmarshal the body into our cluster specification (along with default Kubernetes stuff) and check if a Contact Person is present. If not, I, Peter Brøndum, will be set as a contact. Then it will marshal it back to JSON and send the response to Kubernetes.
This response is used by Kubernetes to do the actual mutation.
Let’s see in Action
I have deployed the Webhook and the MutatingWebhookConfiguration. Let’s prepare a new cluster spec. Notice that we do not add a contact to this cluster!
When we apply this spec, we don’t see any difference, as long as the webhook sends a status 200.
> kubectl apply -f cluster02.yamlcluster.extreme.bestseller.com/destinationaarhus-techblog02 created
But when we list the clusters, I am the contact.
> kubectl get clustersNAME CONTACTPERSONdestinationaarhus-techblog Peter Brøndumdestinationaarhus-techblog02 Peter Brøndum
It worked! (surprise) But let’s check the logs of our webhook.
2020/10/28 12:54:31 No contact, Mutate me!10.244.0.1 - - [28/Oct/2020:12:54:31 +0000] "POST /mutate?timeout=30s HTTP/1.1" 200 214
In the above, we see that our webhook was reached when we applied the cluster and that no Contact Person was set. From there, it responded with an AdmissionReview telling Kubernetes to mutate.
As you can see from the above examples, it is quite easy to extend Kubernetes’ functionality by creating your own custom resources. Even creating custom logic and behaviour of the resources is doable. And this does not have to be custom resources, it could be used to influence other key components in the cluster.
To be fair, there is quite a lot from our setup in BESTSELLER, I did not cover. But the basics on how we keep track of our clusters are there. Instead of assigning me as a contact on each and every cluster, which would be a pain, we call our CI/CD pipeline and mutate the status, amongst other things, on the cluster resources in Kubernetes. This way, when a cluster is changed, our CI/CD will run a bunch of jobs to setup and configure the specific cluster. When finished the pipeline updates our custom cluster resource in Kubernetes once again and mutates the status.
By Peter Brøndum
Tech Lead, BESTSELLER
My name is Peter Brøndum, I work as a Tech Lead and Scrum Master in a platform engineering team at BESTSELLER. Our main priority is building a development highway, with paved roads, lights and signs, so our colleagues can deliver value even faster.
Besides working at BESTSELLER, I — amongst other things, am automating my own home, and yes, that is, of course, running on Kubernetes as well.