Diving into Kubernetes MutatingAdmissionWebhook

Morven Cao
Apr 10, 2018 · 16 min read

Admission controllers are powerful tools for intercepting requests to the Kubernetes API server prior to persistence of the object. However, they are not very flexible due to the requirement that they are compiled as a binary in thekube-apiserver that is configured by the cluster administrator. Starting in Kubernetes 1.7, Initializers and External Admission Webhooks are introduced to address this limitation. In Kubernetes 1.9, Initializers stays in alpha phase while External Admission Webhooks are promoted to beta and split into MutatingAdmissionWebhook and ValidatingAdmissionWebhook.

MutatingAdmissionWebhook together with ValidatingAdmissionWebhook are a special kind of admission controllers which process mutations and validations. This validation is done by matching the rules that are defined in the MutatingWebhookConfiguration(explained below).

In this article, I will dive into the details of a MutatingAdmissionWebhook and write a working webhook admission server step-by-step.

Benefit of Webhooks

Kubernetes cluster administrators can use webhooks to create additional mutating and validating admission plugins to the admission chain of apiserver without recompiling them. This provides developers with the freedom and flexibility to customize admission logic on multiple actions such CREATE, UPDATE, and DELETE on any resource. The number of applications that can be used are vast. Some common use cases include:

  • Mutating resources before creating them. Istio, a representative example, injects an Envoy sidecar container to target pods to implement traffic management and policy enforcement.
  • Automated provisioning of StorageClass. Observes creation of PersistentVolumeClaim objects and automatically adds storage class to them based on predefined policy. Users that do not need to care about StorageClass creating.
  • Validating complex custom resource. Make sure custom resource can only be created after its definition and all dependencies created and available.
  • Restricting namespace. On multi-tenant systems, avoid resources created in reserved namespaces.

Besides the user-cases listed above, many more applications can be created based on the power of webhooks.

Webhooks vs Initializers

Based on feedback from the community and use cases in alpha phase for both External Admission Webhooks and Initializers, the Kubernetes community decided to promote webhooks to beta and split it into two types of webhooks (MutatingAdmissionWebhook and ValidatingAdmissionWebhook). These updates make webhooks consistent with other admission controllers and enforce mutate-before-validate. Initializers can also implement dynamic admission control by modifying Kubernetes resources before they are actually created. If you're unfamiliar with Initializers, please refer to the article: https://medium.com/ibm-cloud/kubernetes-initializers-deep-dive-and-tutorial-3bc416e4e13e.

So what’s the difference between Webhooks and Initializers?

  • Webhooks can be applied on more actions, including 'mutate' or 'admit' on resoures 'CREATE' 'UPDATE' and 'DELETE', whereas Initializers can't 'admit' resources for 'DELETE' requests.
  • Webhooks are not allowed to query resources before created, while Initializers are capable of watching the uninitialized resources by the query parameter ?includeUninitialized=true, which makes the resource creation progress transparent.
  • SinceInitializers persist the 'pre-create' states to etcd, higher latency and increased etcd burden will be introduced accordingly, especially when apiserver upgrades or fails. Webhooks, however, consume less memory and computing resources.
  • Webhooks provide more robustness on failures than Initializers. Failure policies can be configured in Webhooks configuraton to avoid hanging onto resources that are created. Buggy Initializers, on the other hand, may block all matched resources attempted to be created.

Besides the differences listed above, Initializerhas some significant issues currently as open issues with long expected development time including quota replenishment bug. Promotion of Webhooks to beta may be a signal that more support for it in the future, but that depends. If stable behavior is preferred, I suggest you choose Webhooks.

How MutatingAdmissionWebhook works

MutatingAdmissionWebhook intercepts requests matching the rules defined in MutatingWebhookConfiguration before presisting into etcd. MutatingAdmissionWebhook executes the mutation by sending admission requests to webhook server. Webhook server is just plain http server that adhere to the API.

The following diagram describes how MutatingAdmissionWebhook works in details:

The MutatingAdmissionWebhook needs three objects to function:

  1. MutatingWebhookConfiguration

MutatingAdmissionWebhook needs to be registered in the apiserver by providing MutatingWebhookConfiguration. During the registration process, MutatingAdmissionWebhook states:

  • How to connect to the webhook admission server
  • How to verify the webhook admission server
  • The URL path of the webhook admission server
  • Rules defining which resource and what action it handles
  • How unrecognized errors from the webhook admission server are handled

2. MutatingAdmissionWebhook itself

MutatingAdmissionWebhook is a plugin-style admission controller that can be configured into the apiserver. The MutatingAdmissionWebhook plugin get the list of interested admission webhooks from MutatingWebhookConfiguration. Then the MutatingAdmissionWebhook observes the requests to apiserver and intercepts requests matching the rules in admission webhooks and calls them in parallel.

3. Webhook Admission Server

Webhook Admission Server is just plain http server that adhere to Kubernetes API. For each request to the apiserver, the MutatingAdmissionWebhook sends an admissionReview(API for reference) to the relevant webhook admission server. The webhook admission server gathers information like object, oldobject, and userInfo from admissionReview, and sends back a admissionReview response including AdmissionResponse whose Allowed and Result fields are filled with the admission decision and optional Patch to mutate the resoures.

Tutorial for MutatingAdmissionWebhook

Writing a complete Webhook Admission Server may be intimidating. To make it easier, we’ll write a simple Webhook Admission Server that implements injecting nginx sidecar container and volume. The complete code can be found in the kube-mutating-webhook-tutorial. This project refers to Kunernetes webhook example and Istio sidecar injection implementation.

In the following sections, I’ll show you how to write a working containerized webhook admission server and deploy it to a Kubernetes cluster.

Prerequisites

MutatingAdmissionWebhook requires a Kubernetes 1.9.0 or above with the admissionregistration.k8s.io/v1beta1 API enabled. Verify that by the following command:

kubectl api-versions | grep admissionregistration.k8s.io/v1beta1

The result should be:

admissionregistration.k8s.io/v1beta1

In addition, the MutatingAdmissionWebhook and ValidatingAdmissionWebhook admission controllers should be added and listed in the correct order in the admission-control flag of kube-apiserver.

Write the Webhook Server

Webhook Admission Server is just a simple http server that adheres to the Kubernetes API. I'll paste some pseudo code to describe the main logic:

sidecarConfig, err := loadConfig(parameters.sidecarCfgFile)
pair, err := tls.LoadX509KeyPair(parameters.certFile, parameters.keyFile)

whsvr := &WebhookServer {
sidecarConfig: sidecarConfig,
server: &http.Server {
Addr: fmt.Sprintf(":%v", 443),
TLSConfig: &tls.Config{Certificates: []tls.Certificate{pair}},
},
}

// define http server and server handler
mux := http.NewServeMux()
mux.HandleFunc("/mutate", whsvr.serve)
whsvr.server.Handler = mux

// start webhook server in new rountine
go func() {
if err := whsvr.server.ListenAndServeTLS("", ""); err != nil {
glog.Errorf("Filed to listen and serve webhook server: %v", err)
}
}()

Explanation for the above code:

  • sidecarCfgFile contains sidecar injector template defined in ConfigMap below.
  • certFile and keyFile key pair that will be needed for TLS communication between apiserver and webhook server.
  • Line 19 starts https server listening on 443 on path ‘/mutate’.

Next we’ll focus on the main logic of the handler function serve:

// Serve method for webhook server
func (whsvr *WebhookServer) serve(w http.ResponseWriter, r *http.Request) {
var body []byte
if r.Body != nil {
if data, err := ioutil.ReadAll(r.Body); err == nil {
body = data
}
}

var reviewResponse *v1beta1.AdmissionResponse
ar := v1beta1.AdmissionReview{}
deserializer := codecs.UniversalDeserializer()
if _, _, err := deserializer.Decode(body, nil, &ar); err != nil {
glog.Error(err)
reviewResponse = toAdmissionResponse(err)
} else {
reviewResponse = mutate(ar)
}

response := v1beta1.AdmissionReview{}
if reviewResponse != nil {
response.Response = reviewResponse
response.Response.UID = ar.Request.UID
}
// reset the Object and OldObject, they are not needed in a response.
ar.Request.Object = runtime.RawExtension{}
ar.Request.OldObject = runtime.RawExtension{}

resp, err := json.Marshal(response)
if err != nil {
glog.Error(err)
}
if _, err := w.Write(resp); err != nil {
glog.Error(err)
}
}

The serve function is a simple http handler with http request and response writer parameters.

  • First demarshal the request to AdmissionReview, which contains information like object, oldobject and userInfo...
  • Then call Webhook core function mutate to create patch that injects sidecar container and volume.
  • Finally, demarshal the response with admission decision and optional patch, the send it back to apiserver.

For the part of mutate function, you get the free rein to complete it in your preferred way. Let's take my implementation as an example:

// main mutation process
func (whsvr *WebhookServer) mutate(ar *v1beta1.AdmissionReview) *v1beta1.AdmissionResponse {
req := ar.Request
var pod corev1.Pod
if err := json.Unmarshal(req.Object.Raw, &pod); err != nil {
glog.Errorf("Could not unmarshal raw object: %v", err)
return &v1beta1.AdmissionResponse {
Result: &metav1.Status {
Message: err.Error(),
},
}
}

// determine whether to perform mutation
if !mutationRequired(ignoredNamespaces, &pod.ObjectMeta) {
glog.Infof("Skipping mutation for %s/%s due to policy check", pod.Namespace, pod.Name)
return &v1beta1.AdmissionResponse {
Allowed: true,
}
}

annotations := map[string]string{admissionWebhookAnnotationStatusKey: "injected"}
patchBytes, err := createPatch(&pod, whsvr.sidecarConfig, annotations)

return &v1beta1.AdmissionResponse {
Allowed: true,
Patch: patchBytes,
PatchType: func() *v1beta1.PatchType {
pt := v1beta1.PatchTypeJSONPatch
return &pt
}(),
}
}

From the code above, the mutate function calls mutationRequired to detemine whether mutation is required or not. For those requiring mutation, the mutate function gets the mutation 'patch' from another function createPatch. Pay close attention to the little trick in function mutationRequired, we skip the pods without annotation sidecar-injector-webhook.morven.me/inject: true. That will be mentioned latter when we deployment applications. For complete code, please refer to https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/webhook.go.

Create Dockerfile and Build the Container

Create the build script:

dep ensure
CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o kube-mutating-webhook-tutorial .
docker build --no-cache -t morvencao/sidecar-injector:v1 .
rm -rf kube-mutating-webhook-tutorial

docker push morvencao/sidecar-injector:v1

And create Dockerfile as dependency of build script:

FROM alpine:latest

ADD kube-mutating-webhook-tutorial /kube-mutating-webhook-tutorial
ENTRYPOINT ["./kube-mutating-webhook-tutorial"]

Before actually building the container, you need a Docker ID account and change the image name&tag(in Dockerfile and deployment.yaml) to yours, then execute:

[root@mstnode kube-mutating-webhook-tutorial]# ./build
Sending build context to Docker daemon 44.89MB
Step 1/3 : FROM alpine:latest
---> 3fd9065eaf02
Step 2/3 : ADD kube-mutating-webhook-tutorial /kube-mutating-webhook-tutorial
---> 432de60c2b3f
Step 3/3 : ENTRYPOINT ["./kube-mutating-webhook-tutorial"]
---> Running in da6e956d1755
Removing intermediate container da6e956d1755
---> 619faa936145
Successfully built 619faa936145
Successfully tagged morvencao/sidecar-injector:v1
The push refers to repository [docker.io/morvencao/sidecar-injector]
efd05fe119bb: Pushed
cd7100a72410: Layer already exists
v1: digest: sha256:7a4889928ec5a8bcfb91b610dab812e5228d8dfbd2b540cd7a341c11f24729bf size: 739

Create Sidecar Injection Configuration

Now let’s create a Kubernetes ConfigMap, which includes container and volume information that will be injected into the target pod.

apiVersion: v1
kind: ConfigMap
metadata:
name: sidecar-injector-webhook-configmap
data:
sidecarconfig.yaml: |
containers:
- name: sidecar-nginx
image: nginx:1.12.2
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
volumeMounts:
- name: nginx-conf
mountPath: /etc/nginx
volumes:
- name: nginx-conf
configMap:
name: nginx-configmap

From the above manifest, another ConfigMap including nginx conf is required. Here we put it in nginxconfigmap.yaml.

Then deploy the two ConfigMaps to cluster:

[root@mstnode kube-mutating-webhook-tutorial]# kubectl create -f ./deployment/nginxconfigmap.yaml
configmap "nginx-configmap" created
[root@mstnode kube-mutating-webhook-tutorial]# kubectl create -f ./deployment/configmap.yaml
configmap "sidecar-injector-webhook-configmap" created

Create Secret Including Signed key/cert Pair

Supporting TLS for external webhook server is also required because admission is a high security operation. As part of the process, we need to create a TLS certificate signed by the Kubernetes CA to secure the communication between the webhook server and apiserver. For the complete steps for creating and approving the process forCSR, please refer to https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/.

For simplicity purposes, we referred to the script from Istio and created a similar script called webhook-create-signed-cert.sh to automatically create the cert/key pair and include it in a Kubernetes secret.

#!/bin/bash
while [[ $# -gt 0 ]]; do
case ${1} in
--service)
service="$2"
shift
;;
--secret)
secret="$2"
shift
;;
--namespace)
namespace="$2"
shift
;;
esac
shift
done

[ -z ${service} ] && service=sidecar-injector-webhook-svc
[ -z ${secret} ] && secret=sidecar-injector-webhook-certs
[ -z ${namespace} ] && namespace=default

csrName=${service}.${namespace}
tmpdir=$(mktemp -d)
echo "creating certs in tmpdir ${tmpdir} "

cat <<EOF >> ${tmpdir}/csr.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${service}
DNS.2 = ${service}.${namespace}
DNS.3 = ${service}.${namespace}.svc
EOF

openssl genrsa -out ${tmpdir}/server-key.pem 2048
openssl req -new -key ${tmpdir}/server-key.pem -subj "/CN=${service}.${namespace}.svc" -out ${tmpdir}/server.csr -config ${tmpdir}/csr.conf

# clean-up any previously created CSR for our service. Ignore errors if not present.
kubectl delete csr ${csrName} 2>/dev/null || true

# create server cert/key CSR and send to k8s API
cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
name: ${csrName}
spec:
groups:
- system:authenticated
request: $(cat ${tmpdir}/server.csr | base64 | tr -d '\n')
usages:
- digital signature
- key encipherment
- server auth
EOF

# verify CSR has been created
while true; do
kubectl get csr ${csrName}
if [ "$?" -eq 0 ]; then
break
fi
done

# approve and fetch the signed certificate
kubectl certificate approve ${csrName}
# verify certificate has been signed
for x in $(seq 10); do
serverCert=$(kubectl get csr ${csrName} -o jsonpath='{.status.certificate}')
if [[ ${serverCert} != '' ]]; then
break
fi
sleep 1
done
if [[ ${serverCert} == '' ]]; then
echo "ERROR: After approving csr ${csrName}, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2
exit 1
fi
echo ${serverCert} | openssl base64 -d -A -out ${tmpdir}/server-cert.pem


# create the secret with CA cert and server cert/key
kubectl create secret generic ${secret} \
--from-file=key.pem=${tmpdir}/server-key.pem \
--from-file=cert.pem=${tmpdir}/server-cert.pem \
--dry-run -o yaml |
kubectl -n ${namespace} apply -f -

We then execute it and a Kubernetes secret including cert/key pair is created:

[root@mstnode kube-mutating-webhook-tutorial]# ./deployment/webhook-create-signed-cert.sh
creating certs in tmpdir /tmp/tmp.wXZywp0wAF
Generating RSA private key, 2048 bit long modulus
...........................................+++
..........+++
e is 65537 (0x10001)
certificatesigningrequest "sidecar-injector-webhook-svc.default" created
NAME AGE REQUESTOR CONDITION
sidecar-injector-webhook-svc.default 0s https://mycluster.icp:9443/oidc/endpoint/OP#admin Pending
certificatesigningrequest "sidecar-injector-webhook-svc.default" approved
secret "sidecar-injector-webhook-certs" created

Create the Sidecar Injector Deployment and Service

The deployment brings up 1 pod in which the sidecar-injector container is running. The container starts with special arguments:

  • sidecarCfgFile pointing to the sidecar injector configuration file mounted from sidecar-injector-webhook-configmap ConfigMap created above
  • tlsCertFile and tlsKeyFile are cert/key pair mounted from sidecar-injector-webhook-certs Secret create by script above
  • alsologtostderr v=4 and 2>&1 are logging arguments
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: sidecar-injector-webhook-deployment
labels:
app: sidecar-injector
spec:
replicas: 1
template:
metadata:
labels:
app: sidecar-injector
spec:
containers:
- name: sidecar-injector
image: morvencao/sidecar-injector:v1
imagePullPolicy: IfNotPresent
args:
- -sidecarCfgFile=/etc/webhook/config/sidecarconfig.yaml
- -tlsCertFile=/etc/webhook/certs/cert.pem
- -tlsKeyFile=/etc/webhook/certs/key.pem
- -alsologtostderr
- -v=4
- 2>&1
volumeMounts:
- name: webhook-certs
mountPath: /etc/webhook/certs
readOnly: true
- name: webhook-config
mountPath: /etc/webhook/config
volumes:
- name: webhook-certs
secret:
secretName: sidecar-injector-webhook-certs
- name: webhook-config
configMap:
name: sidecar-injector-webhook-configmap

The service exposes the pod defined above labeled by app=sidecar-injector to make it accessible in cluster. This service will be referred by the MutatingWebhookConfiguration in clientConfig section and by default spec.ports.port should be 443(default https port).

apiVersion: v1
kind: Service
metadata:
name: sidecar-injector-webhook-svc
labels:
app: sidecar-injector
spec:
ports:
- port: 443
targetPort: 443
selector:
app: sidecar-injector

Next we deploy the above Deployment and Service to cluster and verify the sidecar injector webhook server is running:

[root@mstnode kube-mutating-webhook-tutorial]# kubectl create -f ./deployment/deployment.yaml
deployment "sidecar-injector-webhook-deployment" created
[root@mstnode kube-mutating-webhook-tutorial]# kubectl create -f ./deployment/service.yaml
service "sidecar-injector-webhook-svc" created
[root@mstnode kube-mutating-webhook-tutorial]# kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
sidecar-injector-webhook-deployment 1 1 1 1 2m
[root@mstnode kube-mutating-webhook-tutorial]# kubectl get pod
NAME READY STATUS RESTARTS AGE
sidecar-injector-webhook-deployment-bbb689d69-fdbgj 1/1 Running 0 3m

Configure webhook admission controller on the fly

MutatingWebhookConfiguration specifies which webhook admission servers are enabled and which resources are subject to the admission server. It is recommended that you firstly deploy the webhook admission server and make sure it is working properly before creating the MutatingWebhookConfiguration. Otherwise, requests will be unconditionally accepted or rejected based on failurePolicy.

For now, we create the MutatingWebhookConfiguration manifest with the following content:

apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
name: sidecar-injector-webhook-cfg
labels:
app: sidecar-injector
webhooks:
- name: sidecar-injector.morven.me
clientConfig:
service:
name: sidecar-injector-webhook-svc
namespace: default
path: "/mutate"
caBundle: ${CA_BUNDLE}
rules:
- operations: [ "CREATE" ]
apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods"]
namespaceSelector:
matchLabels:
sidecar-injector: enabled

Line 8: name - name for the webhook, should be fully qualified. Mutiple mutating webhooks are sorted by providing order. Line 9: clientConfig - describes how to connect to the webhook admission server and the TLS certificate. In our case, we specify the sidecar injector service. Line 15: rules - specifies what resources and what actions the webhook server handles. In our case, only intercepts request for creating of pods. Line 20: namespaceSelector - namespaceSelector decides whether to send admission request the webhook server on an object based on whether the namespace for that object matches the selector.

Before deploying the MutatingWebhookConfiguration, we need to replace the ${CA_BUNDLE} with apiserver's default caBundle. Let's write the script webhook-patch-ca-bundle.sh to automate this process:

#!/bin/bash
set -o errexit
set -o nounset
set -o pipefail

ROOT=$(cd $(dirname $0)/../../; pwd)

export CA_BUNDLE=$(kubectl get configmap -n kube-system extension-apiserver-authentication -o=jsonpath='{.data.client-ca-file}' | base64 | tr -d '\n')

if command -v envsubst >/dev/null 2>&1; then
envsubst
else
sed -e "s|\${CA_BUNDLE}|${CA_BUNDLE}|g"
fi

Then execute:

[root@mstnode kube-mutating-webhook-tutorial]# cat ./deployment/mutatingwebhook.yaml |\
> ./deployment/webhook-patch-ca-bundle.sh >\
> ./deployment/mutatingwebhook-ca-bundle.yaml

Finally we can deploy MutatingWebhookConfiguration:

[root@mstnode kube-mutating-webhook-tutorial]# kubectl create -f ./deployment/mutatingwebhook-ca-bundle.yaml
mutatingwebhookconfiguration "sidecar-injector-webhook-cfg" created
[root@mstnode kube-mutating-webhook-tutorial]# kubectl get mutatingwebhookconfiguration
NAME AGE
sidecar-injector-webhook-cfg 11s

Verification and Troubleshooting

Now it’s time to verify sidecar injector works as expected and try to see how to troubleshoot if you encounter issues. Typically we create and deploy a sleep application in default namespace to see if the sidecar can be injected.

[root@mstnode kube-mutating-webhook-tutorial]# cat <<EOF | kubectl create -f -
> apiVersion: extensions/v1beta1
> kind: Deployment
> metadata:
> name: sleep
> spec:
> replicas: 1
> template:
> metadata:
> annotations:
> sidecar-injector-webhook.morven.me/inject: "true"
> labels:
> app: sleep
> spec:
> containers:
> - name: sleep
> image: tutum/curl
> command: ["/bin/sleep","infinity"]
> imagePullPolicy: IfNotPresent
> EOF
deployment "sleep" created

Pay close attention to the spec.template.metadata.annotations as there is a new annotation added:

sidecar-injector-webhook.morven.me/inject: "true"

The sidecar injector has some logic to check the existence of the above annotation before injecting sidecar container and volume. You’re free to delete the logic or customize it before build the sidecar injector container.

Check the deployment and pod:

[root@mstnode kube-mutating-webhook-tutorial]# kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
sidecar-injector-webhook-deployment 1 1 1 1 18m
sleep 1 1 1 1 58s
[root@mstnode kube-mutating-webhook-tutorial]# kubectl get pod
NAME READY STATUS RESTARTS AGE
sidecar-injector-webhook-deployment-bbb689d69-fdbgj 1/1 Running 0 18m
sleep-6d79d8dc54-r66vz 1/1 Running 0 1m

It’s not there. What’s going on? Let’s check the sidecar injector logs:

[root@mstnode kube-mutating-webhook-tutorial]# kubectl logs -f sidecar-injector-webhook-deployment-bbb689d69-fdbgj
I0314 08:48:15.140858 1 webhook.go:88] New configuration: sha256sum 21669464280f76170b88241fd79ecbca3dcebaec5c152a4a9a3e921ff742157f

We can’t find any logs that indicate webhook server got admission request, seems that request hadn’t been sent to sidecar injector webhook server. So there is a possibility that the issue is caused by configuration in MutatingWebhookConfiguration. Do a double check of MutatingWebhookConfiguration and we find following content:

namespaceSelector:
matchLabels:
sidecar-injector: enabled

Control sidecar injector with namespaceSelector

We have configured ‘namespaceSelector’ in MutatingWebhookConfiguration, which means only resources in namespace matching the selector will be sent to webhook server. So we need label the default namespace with sidecar-injector=enabled:

[root@mstnode kube-mutating-webhook-tutorial]# kubectl label namespace default sidecar-injector=enabled
namespace "default" labeled
[root@mstnode kube-mutating-webhook-tutorial]# kubectl get namespace -L sidecar-injector
NAME STATUS AGE sidecar-injector
default Active 1d enabled
kube-public Active 1d
kube-system Active 1d

We have now configured the MutatingWebhookConfiguration resulting in the sidecar injection occurring at pod creation time. Kill the running pod and verify a new pod is created with the injected sidecar.

[root@mstnode kube-mutating-webhook-tutorial]# kubectl delete pod sleep-6d79d8dc54-r66vz
pod "sleep-6d79d8dc54-r66vz" deleted
[root@mstnode kube-mutating-webhook-tutorial]# kubectl get pods
NAME READY STATUS RESTARTS AGE
sidecar-injector-webhook-deployment-bbb689d69-fdbgj 1/1 Running 0 29m
sleep-6d79d8dc54-b8ztx 0/2 ContainerCreating 0 3s
sleep-6d79d8dc54-r66vz 1/1 Terminating 0 11m
[root@mstnode kube-mutating-webhook-tutorial]# kubectl get pod sleep-6d79d8dc54-b8ztx -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/psp: default
sidecar-injector-webhook.morven.me/inject: "true"
sidecar-injector-webhook.morven.me/status: injected
labels:
app: sleep
pod-template-hash: "2835848710"
name: sleep-6d79d8dc54-b8ztx
namespace: default
spec:
containers:
- command:
- /bin/sleep
- infinity
image: tutum/curl
imagePullPolicy: IfNotPresent
name: sleep
resources: {}
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-d7t2r
readOnly: true
- image: nginx:1.12.2
imagePullPolicy: IfNotPresent
name: sidecar-nginx
ports:
- containerPort: 80
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/nginx
name: nginx-conf
volumes:
- name: default-token-d7t2r
secret:
defaultMode: 420
secretName: default-token-d7t2r
- configMap:
defaultMode: 420
name: nginx-configmap
name: nginx-conf
...

We can see that sidecar container and volume have been injected into sleep application successfully. Until now, we have working sidecar injector with MutatingAdmissionWebhook. With namespaceSelector we can easily control whether the pods in specified namespace will be injected or not.

But there is a problem for this, with the above configurations, all of the pods in default namespace will be injected with a sidecar, this may be not expected for some cases.

Control sidecar injector with annotation

Thanks to flexibility of MutatingAdmissionWebhook, we can easily customized the mutating logic to filter resources with specified annotations. Remember the annotation sidecar-injector-webhook.morven.me/inject: "true" mentioned above? It can be used as an extra control on sidecar injector. I have written some code in webhook server to skip injecting for pod without the annotation.

Let’s give it a try. In this case, we create another sleep application without sidecar-injector-webhook.morven.me/inject: "true" annotation in podTemplateSpec:

[root@mstnode kube-mutating-webhook-tutorial]# kubectl delete deployment sleep
deployment "sleep" deleted
[root@mstnode kube-mutating-webhook-tutorial]# cat <<EOF | kubectl create -f -
apiVersion: extensions/v1beta1
> kind: Deployment
> metadata:
> name: sleep
> spec:
> replicas: 1
> template:
> metadata:
> labels:
> app: sleep
> spec:
> containers:
> - name: sleep
> image: tutum/curl
> command: ["/bin/sleep","infinity"]
> imagePullPolicy: IfNotPresent
> EOF
deployment "sleep" created

And then verify the sidecar injector skipped the pod:

[root@mstnode kube-mutating-webhook-tutorial]# kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
sidecar-injector-webhook-deployment 1 1 1 1 45m
sleep 1 1 1 1 17s
[root@mstnode kube-mutating-webhook-tutorial]# kubectl get pod
NAME READY STATUS RESTARTS AGE
sidecar-injector-webhook-deployment-bbb689d69-fdbgj 1/1 Running 0 45m
sleep-776b7bcdcd-4bz58 1/1 Running 0 21s

The output shows that the sleep application contains only one container, no extra container and volume injected. Then we patch the sleep deployment to add the additional annotation and verify it will be injected after recreated:

[root@mstnode kube-mutating-webhook-tutorial]# kubectl patch deployment sleep -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar-injector-webhook.morven.me/inject": "true"}}}}}'
deployment "sleep" patched
[root@mstnode kube-mutating-webhook-tutorial]# kubectl delete pod sleep-776b7bcdcd-4bz58
pod "sleep-776b7bcdcd-4bz58" deleted
[root@mstnode kube-mutating-webhook-tutorial]# kubectl get pods
NAME READY STATUS RESTARTS AGE
sidecar-injector-webhook-deployment-bbb689d69-fdbgj 1/1 Running 0 49m
sleep-3e42ff9e6c-6f87b 0/2 ContainerCreating 0 18s
sleep-776b7bcdcd-4bz58 1/1 Terminating 0 3m

As expected, the pod has been injected with extra sidecar container. Now, we got working sidecar injector with mutatingAdmissionWebhook and its coarse-grained control by namespaceSelector and fine-grained control by additional annotation.

Conclusion

MutatingAdmissionWebhook is one of easiest ways of extending Kubernetes with new policy controls and resources mutation and a host of other options.

This feature will enable more workloads and support more ecosystem components, including Istio service mesh platform. Starting with Istio 0.5.0, Istio has been refactored to support their auto injection code with MutatingAdmissionWebhook replacing initializers.

Reference

IBM Cloud

Understand how to bring elastic runtimes to the Enterprise…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store