Writing a very basic kubernetes mutating admission webhook

My findings when attempting to write a (very) simple kubernetes mutating admission webhook.

Okay.. so, webhooks…?

Admission webhooks help you do some really cool stuff, there are two kinds of webhooks, validating and mutating. We will concentrate on the mutating admission webhook in this post.

Mutating admission webhooks allow you to “modify” a (e.g.) Pod (or any kubernetes resource) request. E.g. you can modify a Pod to use a particular scheduler, add / inject sidecar containers (think LinkerD sidecar), or even reject it if it doesn’t meet some some security requirements, etc. etc. — all without having to write a full fledged “micro” service to do this. The webhook can live anywhere, in practice, k8s just needs to know where that is.

Setup

The setup is easy, but important, all you really need to make sure is that the MutatingAdminssionController is enabled in the k8s api-server. To check if your k8s cluster has this enabled, you can use

kubectl api-versions | grep admissionregistration

For development, I can recommend using Kubernetes-In-Docker (KinD), all you need is Docker and KinD. KinD doesn’t auto-enable these, you can use this KinD configuration (kind.yaml)

---
kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
kubeadmConfigPatches:
- |
  apiVersion: kubeadm.k8s.io/v1beta2
  kind: ClusterConfiguration
  metadata:
    name: config
  apiServer:
    extraArgs:
      "enable-admission-plugins": "NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,PersistentVolumeClaimResize,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
nodes:
- role: control-plane

Spin up the KinD cluster with

kind create cluster --config kind.yaml

I trust you’ll be able to configure kubectl etc. to now use this cluster going forward.

This is it, the setup is done. Well done!

Deployment

I’d like to start here, as this is the easiest part of this “tutorial” (if you will). Deploying an admission webhook is in practice the same as deploying any other service onto a k8s cluster.

All we need is

• a Service
• a Deployment
• a MutatingWebhookConfiguration

The first two are simple, so I won’t go into those deeper, you can see what I’ve used on Github.

Please do think about a couple what-ifs before using webhooks in a production environment:

• what if the request to your webhook fails (conn drop, conn reset, no dns, etc.)
• what if the deployment failed and the pod is stuck in a crash-loop
• what if the webhook has become mission critical and must be functional 100% of the time
• what if you create a circular dependency (my favourites!)
• what if you have a 3214 node cluster with many 10s of thousand resources & requests all depending on this webhook

Note: Webhooks are only called over SSL/TLS so your webhook must have a valid signed certificate (we do this further down the line)

The MutatingWebhookConfiguration is where we tell k8s which resource requests should be sent to our webhook. The configuration consists of the following properties:

• apiVersion (at the time it is: admissionregistration.k8s.io/v1beta1)
• kind (must be: MutatingWebhookConfiguration)
• metadata (the usual: name, annotations, labels)
• webhooks (a list of type webhook)

The webhook (type) consists of these properties:

• name
• clientConfig
• ____ caBundle (we will get this from the k8s cluster itself)
• ____ service to send the AdmissionReview requests to
• rules ( a list of rules that define which resource operations should be matched, these rules make sure that k8s resource requests are sent to your webhook )
• namespaceSelector (the usual: matchLabels: {“label_name”: “label_value”}
• …

there are many more that can or should be used, for a simple non-production webhook to play with, the above will suffice.

The full list of properties can be seen here.

A rule consists of the following:

• operations (a list of operations to match, in our case ["CREATE"])
• apiGroups (in our case, empty [""])
• apiVersions (in our case, this is ["v1"])
• resources (in our case, this is ["pods"])

apiGroups, apiVersions and resources are all (kind of) dependent on each other, in this example it’s quite easy as Pod is part of the core api group so it doesn’t need specifying, a empty [""] is matching the core api group.

Here an example:

---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  name: mutateme
  labels:
    app: mutateme
webhooks:
  - name: mutateme.default.svc.cluster.local
    clientConfig:
      caBundle: ${CA_BUNDLE}
      service:
        name: mutateme
        namespace: default
        path: "/mutate"
    rules:
      - operations: ["CREATE"]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
    namespaceSelector:
      matchLabels:
        mutateme: enabled

the ${CA_BUNDLE} above refers to the actual CA bundle retrieved from the k8s API, replace it with your own; you can get your cluster’s CA bundle with

kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}'

As webhooks can only be called over HTTPS (SSL/TLS) you will need to generate a new ssl key & certificate for it. Doing this is somewhat involved, so probably best to take a look here:

and/or

• https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/deployment/webhook-create-signed-cert.sh

The webhook

Pasting the entire code here is probably not very useful, so I’ll concentrate on the essentials that helped me understand a few things.

Conceptually, what a webhook has to do is relatively easy, it receives a AdmissionReview, and responds with an AdmissionReview :) — yes, most blog posts I’ve found concentrate on the Request and Response objects only, which can be a bit confusing.

In essence, this is what needs to happen ..

• the K8S api server will send a AdmissionReview “request” and expects a AdmissionReview “response” — yes, this can get confusing, but i.e. it is something like this

{
  "kind": "AdmissionReview",
  "apiVersion": "admission.k8s.io/v1beta1",
  "request": {...}
}

• the AdmissionReview consists of AdmissionRequest and AdmissionResponse objects
• the webhook needs to “unmarshal” theAdmissionReview from JSON format into some kind of object so it can read the AdmissionRequest and modify the AdmissionResponse object within it
• the webhook i.e. creates its own AdmissionResponse object, copies the UID from the AdmissionRequest object and replaces the AdmissionResponse object within the AdmissionReview with its own (overwrites it)
• responds with a AdmissionReview object in JSON

{
  "kind": "AdmissionReview",
  "apiVersion": "admission.k8s.io/v1beta1",
  "request":  { <<ORIGINAL REQUEST>> },
  "response": { <<OUR RESPONSE>>     }
}

Where to start?

We will start with a basic web server, that supports SSL/TLS, and can read and respond in JSON format.

In practice, you can use whatever programming language you’d like for this, I have used Go, but this can easily be done in Python or any other compiled & interpreted language. Ideally though, use a language that already has K8S libraries so you don’t have to create our own object types; Go (naturally) has these, but there are also at least Python libraries you could use.

Here’s what I did to accept requests on port 8443 with SSL/TLS set up to use a key & cert.

The above example creates a server s with 2 handlers, one for / and one for /mutate which is the endpoint that will be called by k8s (this is what we specified in the MutatingWebhookConfiguration); it listens on port :8443 and we use theListenAndServeTLS method to serve requests over SSL/TLS.

I split up the logic of http request/response from the actual processing of the AdmissionReview request as it’ll be later easier to test the function/s independently; so the /mutate handler really only does

• take the JSON http request and read in the BODY
• send the BODY to the Mutate function in the mutate package (m) — the unmarshalling from JSON into the appropriate object structure, modification and marshalling back to JSON is done here
• respond with a JSON message, either one that describes anError or a AdmissionReview

So far so good, the potentially more challenging part is next.

pkg/mutate/mutate.go

The mutate package does the actual processing of AdmissionReview requests by …

• unmarshalling a received JSON payload into a AdmissionReview object
• using the AdmissionRequest object to decide what we should do
• creating the JSONPatch
• creating a new AdmissionResponse
• updating the AdmissionReview with our new AdmissionResponse
• marshal the finalAdmissionReview into JSON and return it

API Resources, Types, etc.

Something that I’m still struggling with is finding the correct repository, version and file/s even that I need to do things with k8s (or its resources).

Here’s what I’ve found out so far ..

• Pod and other core resource types (which are used to unmarshal JSON into) can be found in kubernetes/api/blob/master/core/v1/types.go
• Metadata objects (which usually are part of main object like Pod), seem to be kept in the apimachinery repo, it doesn’t have a description so :shrug: apimachinery/blob/master/pkg/apis/meta/v1/types.go
• Other resources and their definitions seem to reside in kubernetes/api/{extensions,node,autoscaling,..}; the admission types are located here: kubernetes/api/blob/master/admission/v1beta1/types.go

However, that’s not how they’re imported (at least not in Go); for our example, we need these imports:

import (
  v1beta1 "k8s.io/api/admission/v1beta1"
  corev1 "k8s.io/api/core/v1"
  metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

I believe they must be k8s.io/ imports and not github.com/ imports, even though the repos are hosted on Github.

Marshal & Unmarshal

These terms always get me confused, but I’ll try to explain

• marshal: when you convert a object into a JSON (or other, e.g. protobuf) equivalent representation
• unmarshal: when you convert JSON (or other) into your own object’s representation

For the webhook we need to

• unmarshal the JSON AdmissionReview into a Go AdmissionReview object, I called it ar
• check if there’s a embedded raw object and if so, again, unmarshal it into the object type that we expect, in our case a Pod, which I called simply pod if that fails, we should not try to continue and return an error
• marshal the JSONPatch map into a valid JSONPatch, which, needs to be JSON
• marshal the final Go AdmissionReview object back into a JSON AdmissionReview

this is why we needed to find those imports earlier, so we can create objects of the correct types and unmarshal JSON into them or marshal them into JSON.

The types we use/need are

• AdmissionReview (v1beta1.AdmissionReview)
• AdmissionRequest (v1beta1.AdmissionRequest)
• Pod (corev1.Pod)
• PatchTypeJSONPatch (v1beta1.PatchTypeJSONPatch)
• AdmissionResponse (v1beta1.AdmissionResponse)
• Status (metav1.Status) to set the AdmissionResponse.Result

Why do we create a JSONPatch?

This is because the mutating webhook does not modify the k8s resource, but responds with a JSONPatch, telling k8s how to modify the object for us. I found this quite counter intuitive, since we’re creating a Mutating Admission Webhook but don’t actually mutate anything.

More about JSONPatch expressions and how they work can be found here, but essentially they consist of an operation (op), a path and a value, e.g.:

{
  "op": "replace",
  "path":  "/spec/containers/0/image",
  "value": "debian"
}

In our case, it instructs how and what operations it should apply to the resource that was requested. You can have more than one operation/instruction, e.g. a list of operations [{"op": ..},{"op": ..},{"op": ..}] and I believe they’ll be executed in the same order as added to the list/array/slice. Something for you to try and find out ;).

I used the following to create a JSONPatch list (if a Pod contains more than 1 container) for the webhook, it replaces any container image to use debian instead of the originally requested ..

// resp is the AdmissionReview.Response 
p := []map[string]string{}
for i := range pod.Spec.Containers {
    patch := map[string]string{
        "op": "replace",
        "path": fmt.Sprintf("/spec/containers/%d/image", i), 
        "value": "debian",
    }
p = append(p, patch)
}
// parse the []map into JSON
resp.Patch, err = json.Marshal(p)

This is it!

You can find the all the code on Github, I’d encourage you to try it without looking first :) … except for the details on the SSL ca, key, cert signing. I’ve also added links to resource that I believe will be helpful to understand what’s going on.

Kudos to the IBM-Cloud blog here on Medium that I used as inspiration for this post and also as part reference when I got stuck. It’s more complete but also somewhat more complicated (at least at the time it seemed like it).

alex-leonhardt/k8s-mutate-webhook

Use a IDE / Editor that can help with code completion, dependencies, etc. — I used VSCode, but there are also good Vim plugins.

Resources

• https://godoc.org/k8s.io/api/admissionregistration/v1beta1#MutatingWebhook
• https://godoc.org/k8s.io/api/admission/v1beta1
• https://godoc.org/k8s.io/kubernetes/pkg/apis/admission
• https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/#mutatingwebhookconfiguration-v1beta1-admissionregistration-k8s-io
• https://github.com/morvencao/kube-mutating-webhook-tutorial

Kudos

https://medium.com/ibm-cloud/diving-into-kubernetes-mutatingadmissionwebhook-6ef3c5695f74