How we manage clusters by extending the Kubernetes API

Peter Brøndum
Dec 1, 2020 · 5 min read

At BESTSELLER we run multiple Kubernetes clusters in multiple clouds, which gives us some assurance that even if one provider or one region is degraded we are still able to serve our customers. But multiple clusters, multiple clouds and multiple teams can be a bit difficult to grasp as an engineer. That is why we decided to use the extendability of the Kubernetes API to create a cluster-registry.

In this post, we will cover how to combine Custom Resource Definitions (CRD) with Admission controllers to gain control of your custom Kubernetes Resources.

What is a CRD and Admission Controller

To understand CRDs, we need to understand the basic concept of resources in Kubernetes.

  • A resource is an API endpoint where you can store API objects of any kind.
  • A custom resource allows you to define your own API objects, and thus creating your own Kubernetes kind just like Deployments or Statefulsets.

In short, the Custom Resource Definition is where you define your Custom Resource that extends Kubernetes’ default capabilities.

While the CRDs extend the Kubernetes functionality, Admission controllers govern and enforce how the cluster is used. They can be thought of as a gatekeeper that intercepts (authenticated) API requests and may change the request object or deny the request altogether.

https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers

Admission Controller Phases

There are two types of Admission controllers; validating and mutating. Mutating admission webhooks are invoked first and can modify objects sent to the API server to enforce custom defaults. After all object modifications are complete, validating admission webhooks are invoked which runs logic to validate the incoming resource. In case the validation webhook rejects the request, the Kubernetes API returns a failed HTTP response to the user.

Creating our cluster specification

Let’s start with the easiest part, creating our custom resource definition, in this case, our cluster specification template.

for in-depth details on CRDs click here

From the simplified example above we have defined a new API group extreme.bestseller.com and in that group, our CRD clusters.extreme.bestseller.com is stored.

We have defined 3 fields in our clusters specs, Node Count, Cloud and Contact Person where the first two are required.

Implementing the actual CRD is as easy as:

kubectl apply -f ourcrd.yaml

Time to create our first cluster object! More yaml coming up.

Apply it to our cluster:

kubectl apply -f firstcluster.yaml

Now we can get our clusters with `kubectl` just as any other Kubernetes kind:

> kubectl get clustersNAME                         CONTACTPERSONdestinationaarhus-techblog   Peter Brøndum

With our cluster spec and storage in place, it is time for the fun part.

The Admission Controller

The admission controller, in this case, a mutating webhook, consists of two elements:

  1. A MutatingWebhookConfiguration, which defines which resources is subject to mutation and which mutating service to call.
  2. An admission webhook server, which does the mutation.

First up is the MutatingWebhookConfiguration. We can divide this into two blocks. The first is clientConfig. Here we configure which admission webhook service to call (can be an external service as well). Next is the rules, where we specify that mutation can only happen on Create and Update requests to the Kubernetes API and only on our Cluster resources.

Kubernetes will only accept a ssl encrypted endpoint, which I will not cover this in this article, but we are in luck, other people have made simple scripts that can help us e.g giantswarm

The webhook

With this in place, we need to create the actual mutation logic.

Before we deep dive into the code, I have chosen to write this in `Go` as it has a native client for Kubernetes. That being said, you could do this in the language of your choosing. The only requirement is to create a web server that serves a TLS endpoint and accepts and responds with JSON.

This will be a simplified example, and I have tried to squeeze everything into one file. In essence, what we are aiming at is to:

  1. Receive a JSON request, in Kubernetes terms an AdmissionReview.
  2. Do our mutation logic.
  3. Return a JSON response, again in the format of an AdmissionReview, which tells Kubernetes what to mutate.

Yes! you are correct, it is actually Kubernetes that does the mutation.

Simplified flow of our webhook.

To the code!

In short, the example creates a single HTTP endpoint. When called, it will unmarshal the body into our cluster specification (along with default Kubernetes stuff) and check if a Contact Person is present. If not, I, Peter Brøndum, will be set as a contact. Then it will marshal it back to JSON and send the response to Kubernetes.
This response is used by Kubernetes to do the actual mutation.

Let’s see in Action

I have deployed the Webhook and the MutatingWebhookConfiguration. Let’s prepare a new cluster spec. Notice that we do not add a contact to this cluster!

When we apply this spec, we don’t see any difference, as long as the webhook sends a status 200.

> kubectl apply -f cluster02.yamlcluster.extreme.bestseller.com/destinationaarhus-techblog02 created

But when we list the clusters, I am the contact.

> kubectl get clustersNAME                           CONTACTPERSONdestinationaarhus-techblog     Peter Brøndumdestinationaarhus-techblog02   Peter Brøndum

It worked! (surprise) But let’s check the logs of our webhook.

2020/10/28 12:54:31 No contact, Mutate me!10.244.0.1 - - [28/Oct/2020:12:54:31 +0000] "POST /mutate?timeout=30s HTTP/1.1" 200 214

In the above, we see that our webhook was reached when we applied the cluster and that no Contact Person was set. From there, it responded with an AdmissionReview telling Kubernetes to mutate.

Final Words

As you can see from the above examples, it is quite easy to extend Kubernetes’ functionality by creating your own custom resources. Even creating custom logic and behaviour of the resources is doable. And this does not have to be custom resources, it could be used to influence other key components in the cluster.

How we use this at BESTSELLER

To be fair, there is quite a lot from our setup in BESTSELLER, I did not cover. But the basics on how we keep track of our clusters are there. Instead of assigning me as a contact on each and every cluster, which would be a pain, we call our CI/CD pipeline and mutate the status, amongst other things, on the cluster resources in Kubernetes. This way, when a cluster is changed, our CI/CD will run a bunch of jobs to setup and configure the specific cluster. When finished the pipeline updates our custom cluster resource in Kubernetes once again and mutates the status.

By Peter Brøndum
Tech Lead, BESTSELLER

About me

My name is Peter Brøndum, I work as a Tech Lead and Scrum Master in a platform engineering team at BESTSELLER. Our main priority is building a development highway, with paved roads, lights and signs, so our colleagues can deliver value even faster.
Besides working at BESTSELLER, I — amongst other things, am automating my own home, and yes, that is, of course, running on Kubernetes as well.

Destination AARhus-TechBlog

Engineers sharing knowledge and insights into current IT challenges, development and features.

Destination AARhus-TechBlog

A team of IT and digital experts from Greater Aarhus, Denmark — sharing ideas, knowledge and insights. Are you a passionate engineer? Check out the many career opportunities of our business network here: https://destinationaarhus.com/vacant-positions/

Peter Brøndum

Written by

Doing cloud native stuff, restoring old bicycles and automating. All while brewing amazing cups of coffee.

Destination AARhus-TechBlog

A team of IT and digital experts from Greater Aarhus, Denmark — sharing ideas, knowledge and insights. Are you a passionate engineer? Check out the many career opportunities of our business network here: https://destinationaarhus.com/vacant-positions/