Build a Kubernetes Dynamic Admission Controller for Container Registry Whitelisting

Photo by Patrick Robert Doyle on Unsplash

What is a Kubernetes Admission Controller?

Kubernetes admission controllers are plugins that act as gatekeepers by intercepting authenticated requests to the Kubernetes API server and processing them before any object data gets persisted to the etcd datastore.

The admission controllers can mutate/ change a request object (Mutating Admission Controllers) or allow/disallow a request (Validating Admission Controllers) and sometimes do both. The Mutating admission controllers get executed before the Validating admission controllers to allow the Validating admission controllers to act on any changes the Mutating admission controllers may have made to an incoming request. If any admission controller disallows a request, the entire request gets rejected, and an error message is returned.

Admission Controller Phases — High Level

The kube-apiserver binary comes pre-compiled with more than 30 useful admission controllers (the entire list here), some of which operate by default while the others can be enabled/ configured by the cluster administrator. In this list, there are two special “dynamic” controllers — the ValidatingAdmissionWebhook and the MutatingAdmissionWebhook. These two controllers don’t implement any fixed logic; instead, they allow us the flexibility to implement and execute our custom logic via webhooks each time a Kubernetes resource gets created, updated, or deleted in our cluster.

Admission Controller Phases — Detailed

Why do we care about it?

Admission Controllers can help improve the Security, Governance, Cost, and Configuration Management of a Kubernetes cluster. It can help organizations adhere to best practices, meet legal requirements, and enforce company policies on their Kubernetes cluster. For example, one may use the built-in PodSecurityPolicy admission controller to prevent containers from running as the root user or ensure the container root filesystem is mounted read-only. Similarly, the built-in LimitRanger admission controller can help enforce or add pod resource requests and limits. Custom admission controllers can help extend these functionalities further to realize additional use cases that may not be covered by built-in controllers like:

• Enforce or add specific labels or annotations to various objects. For example, each Pod must be assigned to a project or cost-center using annotations or must at the very least have an app label.
• Allow container images from whitelisted/ corporate registries only while denying pulling from unknown image registries.
• Ensure Pods in production are not using the latest tag.
• Enforce a minimum number of replicas of a Deployment for High Availability.
• Inject sidecar containers into Pods.

Sounds great? How do we build our own?

I am glad that I got you interested in it. We will build an Admission Controller to demonstrate one of the use cases we discussed previously — to allow Pods to be created using container images from whitelisted registries only and deny pulling from untrusted image registries.

Use case — Do not allow Pods to be created when pulling from untrusted registries.

To accomplish the above use case, we will build a Validating Admission Controller and register it with our Kubernetes API Server. Before that, we need to first understand the structure of the request we will receive from the Kubernetes API server through a registered webhook, and what would be the structure of our response back to the Kubernetes API server. The Kubernetes API server uses an object of type AdmissionReview both for sending a request to our custom application and to receive a response. For every request, inside the top-level AdmissionReview structure, there is a request node of type AdmissionRequest. The request node encapsulates the details of the original request sent to the API server. Inside this request node, we are interested in the object field that contains the JSON payload of the Kubernetes object like a Pod, Deployment, etc., that is being created/ updated, or deleted.

AdmissionReview Request object example

In the case of a Validating Admission Controller, our application has to receive an AdmissionReview object, process it to determine whether to allow/ disallow the request and return our decision by populating an AdmissionReview structure with a response node of type AdmissionResponse within it. Inside the response node, we convey our decision using the allowed boolean field. We optionally can also include an HTTP status code and a message to relay additional information back to the client.

AdmissionReview Response object example for Validating Admission Controller

If we are building a Mutating Admission Controller, we send back any mutations/ changes using a JSONPatch type object as part of the response node of the AdmissionReview response. The original request will be modified using this JSON patch.

AdmissionReview Response object example for Mutating Admission Controller

Here is the documentation to the entire structure of AdmissionReview for your reference — https://github.com/kubernetes/api/blob/master/admission/v1/types.go

Cluster Setup

I will be using an Amazon EKS based Kubernetes cluster for this demo, but feel free to use any other Kubernetes distribution if you like. However, if you are using a different Kubernetes distribution, you may have to perform an added step to enable the ValidatingAdmissionController plugin on your Kubernetes cluster following the instructions here.)

Install and configure the aws cli, eksctl, kubectl, docker, OpenSSL, and Node.js locally on your computer to follow along with the rest of the article.

We create a brand new EKS cluster using the eksctl tool. It takes a few minutes to spin up the cluster. Once created, eksctl adds the cluster credentials in ~/.kube/config, so the kubectl tool is ready for immediate use.

> eksctl create cluster

Code Walkthrough

Clone the repo https://github.com/dhiman-halder/whitelist-registry to download the entire source code used in this article.

app/server.js

We have a Node.js application that will listen on port 443 for incoming HTTPS POST requests. In the POST method, we expect to receive an AdmissionReview object as an input. Our application will inspect the definition of the Pod that is being created. If any of the container images referenced in the pod spec does not reference one of the whitelisted registry names, our application will reject the request and return an error to disallow the operation; else, will allow it. The response is also wrapped in an AdmissionReview object. Our application reads the whitelisted registry names from an environmental variable.

server.js

gencerts.sh

Our Node.js application, which will act as a server, must be exposed over HTTPS with a server certificate configured on it. The gencerts.sh script in the source code repo can generate the necessary TLS certificates and keys for it. As part of the script, we first establish a Certificate Authority (CA) to be able to self-sign our server certificate. We generate a CA private key and a CA certificate in PEM format. Next, we create a server-side private key. Then we build a Certificate Signing Request (CSR) with the Common Name (CN) whitelist-registry.default.svc. The Common Name (CN) indicates to a client that it can trust the server available on that DNS name. For Kubernetes API Server to call our server application, the common name (CN) on the server certificate must match the Kubernetes service name in the format service.namespace.svc, i.e., whitelist-registry.default.svc in our case. Finally, we create the server certificate in PEM format using the Certificate Signing Request and self-sign it using our CA key and CA certificate. We will not run our Node.js application container as the root user as a security best practice, hence we change file permission on the certificates and keys to allow a non-root unprivileged user to read them. We copy the CA certificate, the server certificate, and the server key to the app folder to bundle them inside the docker container image when it gets built.

Dockerfile

We will build our docker image from the Node 14 LTS version. We create a working directory to hold our application code, TLS certificates, and keys inside the image. We install the app dependencies using npm and bundle our app’s source code within the image. We bind our application to port 8443. Finally, we start our application using CMD instruction; and run the process as a non-root user as a security best practice.

Dockerfile

webhook-deploy.yaml

Here is the Deployment plus Service definition for deploying our containerized Node.js application to our Kubernetes cluster. We have externalized the set of whitelisted registries using an environment variable WHITELISTED_REGISTRIES that we configure on the Pod spec. The service must listen on 443 since it’s not configurable when used as a webhook service. You have to update the container image reference on the Pod spec in the template below to match yours before running it.

webhook-deployment.yaml

webhook-registration.yaml

We will register our Node.js application as a Validating Admission Controller with our Kubernetes server using a Kubernetes object of kind ValidatingWebhookAdmissionConfiguration. If we were registering a Mutating Admission Controller instead, we would be using an object of kind MutatingWebhookAdmissionConfiguration. In this case, as shown below, we will name our webhook com.foo.whitelist-registry; it can be named anything as long as the name is unique across all the webhooks in the cluster. Under the rules section, we specify under what conditions this webhook gets invoked, in this case, we plan on calling our webhook only when a Pod is being created. Under the ClientConfig section, we will specify how the Kubernetes API Server can find our Node.js server application. We will expose our application through a service named whitelist-registry in the default namespace. The caBundle refers to a PEM encoded CA bundle that the Kubernetes API Server as a client can use to validate the server certificate on the Node.js application. Before registering this webhook, we will create a Base64 encoded version of the ca.crt file and replace the CA_BUNDLE placeholder with it.

ValidatingWebhookConfiguration for registering the webhook

Build

Generate TLS Certificates and Keys

We run the gencerts.sh script to generate and copy TLS certificates and keys.

> bash gencerts.sh

Build a Docker Container Image

We will log in to Dockerhub, build our Docker container image locally, tag, and then push the image to Dockerhub using the below commands. Do adjust the Docker Account name in the commands below to match yours.

> docker login
> docker build -t whitelist-registry:1.0 .
> docker tag whitelist-registry:1.0 dhimanhalder/whitelist-registry:1.0
> docker push dhimanhalder/whitelist-registry:1.0

Deploy Webhook Server Application

We will deploy our Node.js application and expose it through a ClusterIP service. Do not forget to update the container image reference in the template to match your docker account name.

> kubectl apply -f webhook-deploy.yaml
service/whitelist-registry created
pod/whitelist-registry created

Register Webhook with Kubernetes API Server

We create a Base64 encoded version of the ca.crt file and replace the CA_BUNDLE placeholder within the webhook-registration.yaml with it.

> cat ca.crt | base64
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMyakNDQWNJQ0NRQ0dlUlJjZWI3...THVUQU5CZ2txaGtpRzl3MEJBUXNGQURBdk1TMHdLd1lEVlFRRERDUlgKYUdsMFpXeHBjM1FnVW1WbmFYTjBjbmtnUTI5VRFLS0tLS0K

Finally, we will register the webhook with the Kubernetes API Server.

> kubectl apply -f webhook-registration.yaml
validatingwebhookconfiguration.admissionregistration.k8s.io/com.foo.whitelist-registry created

Test!

We can test our Whitelist-Registry Validating Admission Controller using two pod definitions included in our source code repo. The first one is test-pod-1.yaml which uses a container image from docker.io.

test-pod-1.yaml using a trusted registry

If we try to create a pod using this pod definition, our request is allowed since the container image comes from one of the “trusted” whitelisted registries! Note that one side effect to this Validating controller is that we must explicitly specify the docker.io registry host even when pulling from a docker public repo.

> kubectl apply -f test-pod-1.yaml
pod/test-pod-1 created

Here is another example — test-pod-2.yaml. In this case, the container image comes from xyz.io.

test-pod-2.yaml using an untrusted registry

If we try to create a pod using this pod definition, we get the below error message.

> kubectl apply -f test-pod-2.yaml
Error from server: error when creating "test-pod-2.yaml": admission webhook "com.foo.whitelist-registry" denied the request: nginx image comes from an untrusted registry! (xyz.io/nginx:latest). Only images from docker.io,gcr.io are allowed.

Cleanup

We deregister the webhook and delete the webhook server application.

> kubectl delete -f webhook-registration.yaml
validatingwebhookconfiguration.admissionregistration.k8s.io "com.foo.whitelist-registry" deleted
> kubectl delete -f webhook-deploy.yaml
deployment.apps "whitelist-registry" deleted
service "whitelist-registry" deleted

Conclusion

I hope you have enjoyed building your Admission controller for Kubernetes. While we can keep writing such controllers to realize different use cases, it is probably very inefficient since we have to write too much code. In the next post, we will be looking at the Open Policy Agent Gatekeeper project that provides a way to realize similar use cases through policy configurations instead of writing code.

References

• https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/
• https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/

Found this article useful? Hit that clap button. Really like it? Hold the clap, give it two, or fifty! Follow Slalom Technology and read more articles on thought leadership in Technology.