Last mile security with Admission Controllers in Kubernetes
PROBLEM: Make sure all the application containers run as non root user ID X or they don’t run in privilege mode or anything that is controlled via the resource (Pod/Deployment etc) definitions such as securityContext has values as prescribed by the security controls you want.
A proper threat modelling reveals potential places to secure. If it’s not for the security by design principles Tom Cruise could have easily avoided the below stunts
To jolt your memory, there were many layers of authentication/authorization: pass code, voice recognition, retina scan etc before mainframe with sensitive information could be accessed and data stolen.
Iteration 1: We follow the security pattern and scan the helm charts during deployment in the pipeline. Problem solved. Note that we do enforce them to be specified in the definition. So even if the Dockerfile has non root user, we will reject them. We necessarily require yaml to define runAsUser to be X, for example.
Assuming Tom is authenticated and authorized to run the deployments directly on the cluster, he actually gets past all the pipeline checks and run the container as root and probably with privileged mode by using kubectl command from his laptop. We need to do better.
Iteration 2: The last mile security is very important. You don’t keep an open safe in a room which is heavily guarded only from outside.
Tom did get inside the room (cluster) bypassing all the scans outside the room (pipeline checks). Inside the room there are further intrusion detection system like pressure sensitive floor, temperature monitor which prevents his admission on the floor.
K8s has a concept of Admission Controllers which kicks in after authentication/authorization of requests to API server and controls the admission of resources in the cluster at runtime. It provides among other controllers admission webhooks which can be configured to put additional checks to allow admission of resources in the cluster. In particular we can use ValidatingAdmissionWebhook, which validates the resource objects (Pods, Deployments etc) on request matches. We can check the pod resource object one more time before it is actually going to be persisted in the etcd store and refuse its admission if it doesn’t have the securityContext’s runAsUser with value X. This operation of validating is very near (last mile) to the actual persistence in K8s and hence more robust than the far away static pipeline checks.
There are various kinds of Admission Controllers. Here is a list of built-in controllers. For a quick example, try deleting default namespace in your cluster. You will most likely get an error — Error from server (Forbidden): namespaces “default” is forbidden: this namespace may not be deleted.
This is because of NamespaceLifecycle admision controller.
Admission Controller also gives us a chance to change the resources on the fly via MutatingAdmissionWebhook. At runtime when we see there is no runAsUser specified and know that it must be set to X, we can mutate the Pod definition, adding runAsUser to X and pass down the resulting mutated object down kubernetes api pipeline. The ValidatingAdmissionWebhook runs after the mutating one, so as to be sure that mutation is still a valid one.
MTLS in Istio: Ever wondered how Istio injects envoy sidecars to your pods? It’s via mutating webhook. It provides your service mesh facilities like mtls which are possible only due to the proxies intercepting the traffic. While we are at it, following istio’s recommendation to put versions and app as labels in the resources definitions, we can validate them if deployments etc have those labels or not in a validating webhook.
To summarize, we can use webhook admission controllers to mutate and validate the resource objects at runtime and enforce the security closer to the entity in question. There are many other Admission Controller as we saw before each providing a unique service for a better security and governance. If you are on IBM Cloud here is a list of controllers enabled by default.
Hands-on: A webhook admission controller demo. Enforcing container image security on IBM Cloud
