Trendyol Tech
Published in

Trendyol Tech

Photo by Clemens van Lay on Unsplash

Getting Started to Write Your First Kubernetes Admission Webhook Part 2✨

In the previous post, we mostly had talked about writing Kubernetes Admission Webhooks by using the operator-sdk tool and created our first Mutating Admission Webhook against our custom resource type. But, this time we are going to create another type of Admission Webhook called “Validating” for core types such as Deployment, Pod, etc. instead of the custom resource type because we may not always have some kind of custom resource type. Also, we are going to use “kubebuilder” to scaffold the project template. Why? Because change is always good. We are using this method while we are developing the webhooks in our platform team at Trendyol 💪

Kubebuilder is a framework for building Kubernetes APIs using custom resource definitions (CRDs).

Similar to web development frameworks such as Ruby on Rails and SpringBoot, Kubebuilder increases velocity and reduces the complexity managed by developers for rapidly building and publishing Kubernetes APIs in Go. It builds on top of the canonical techniques used to build the core Kubernetes APIs to provide simple abstractions that reduce boilerplate and toil.

As I mentioned earlier, we are familiar with these tools from the Kubernetes Operator pattern but these tools are also helpful for creating Kubernetes Admission Webhooks.

There is really good documentation available to learn about the kubebuilder, but in this post, we are focusing on the special section of the book “Webhook for Core Types”. We also mentioned earlier, kubebuilder & operator-sdk don’t support webhook scaffolding for core types, we have to use the library from controler-runtime to handle it. There is an example in controller-runtime.

Now, let's move into the demo section to make our hands dirty with the kubebuilder tool this time.

Demo

In this demo, we are going to validate Pods against labels that we marked as required. If necessary annotations exist on Pod, then we allow the Pod creation, if not, we don’t.

Prerequisites

  • Minikube v1.18.1
  • Kubebuilder v2.3.2
  • kubectl v1.20.5
  • Go v1.16.3

I’m going to do this demo on the macOS environment, so, you can use “brew” which is a package manager for macOS to install all the tools above.

ValidatingAdmissionWebhook for Pod

Since we are working with operator-sdk, kubebuilder commands might be familiar to us.

Let’s create the directory for the project

$ mkdir -p pod-validatingwebhook
$ cd pod-validatingwebhook

Before initializing the project template with kubebuilder we should create go modules first:

$ go mod init pod-validatingwebhook
go: creating new go.mod: module pod-validationwebhook

To initialize the project template we should run the following command:

$ kubebuilder init --domain developer.guy --license none --owner "developer-guy"
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.5.0
Update go.mod:
$ go mod tidy
Running make:
$ make
go: creating new go.mod: module tmp
go get: added sigs.k8s.io/controller-tools v0.2.5
/Users/batuhan.apaydin/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go build -o bin/manager main.go
Next: define a resource with:
$ kubebuilder create api
# Look at the directory structure, you should see similar to the following
$ tree -L 2 .
.
├── Dockerfile
├── Makefile
├── PROJECT
├── bin
│ └── manager
├── config
│ ├── certmanager
│ ├── default
│ ├── manager
│ ├── prometheus
│ ├── rbac
│ └── webhook
├── go.mod
├── go.sum
├── hack
│ └── boilerplate.go.txt
└── main.go

Now, we are ready for creating ValidatingAdmissionWebhook for Pod, but let’s create an API first.

$ kubebuilder create api — group core — version v1 — kind Pod — resource=false — controller=false ← be careful here we specified resource and controller creation as false here because we don’t need them.Writing scaffold for you to edit…
Running make:
$ make
go: creating new go.mod: module tmp
go get: added sigs.k8s.io/controller-tools v0.2.5
/Users/batuhan.apaydin/go/bin/controller-gen object:headerFile=”hack/boilerplate.go.txt” paths=”./…”
Error: go [list -e -json -compiled=true -test=false -export=false -deps=true -find=false -tags ignore_autogenerated — ./…]: exit status 1: go: github.com/developer-guy/pod-validationwebhook: package k8s.io/api/core/v1 imported from implicitly required module; to add missing requirements, run:
go get k8s.io/api/core/v1@v0.17.2
# if you get the same error above 👆, just get the dependency and run make again like the following:
$ go get k8s.io/api/core/v1@v0.17.2
$ make
go: creating new go.mod: module tmp
go get: added sigs.k8s.io/controller-tools v0.2.5
/Users/batuhan.apaydin/go/bin/controller-gen object:headerFile=”hack/boilerplate.go.txt” paths=”./…”
go fmt ./…
go vet ./…
go build -o bin/manager main.go

After that, create the ValidatingAdmissionWebhook.

# Remember here, programmatic-validation is for ValidatingAdmissionWebhook, and the --default for the Mutating one.
$ kubebuilder create webhook --group core --version v1 --kind Pod --programmatic-validation
Writing scaffold for you to edit...
api/v1/pod_webhook.go

Once you open the project, you should notice that some kind of compile-time error exists in the main.go and “api/v1/pod_webhook.go” files.

main.go
api/v1/pod_webhook.go

So, in order to fix this, we are going to follow the section within the kubebuilder’s book that I mentioned above “Webhook for Core Types”.

Let’s change the content of the pod_webhook.go with the following code, and then, we will explain the code a little bit.

https://gist.github.com/developer-guy/c64b62740d06576a368323f425597a92

One more thing we need to do. Open the main.go file and change the lines between 55 and 59 with the following to register our webhook server.

https://gist.github.com/developer-guy/9c3b3ba038dc2dd0064d8bdb4023a812

Now, we are ready from the code perspective. Let’s move on with the deployment of the webhook.

Before building and pushing the image, remove the “COPY controllers/ controllers/” at #L15 from the Dockerfile because we are not using the controller here.

$ make docker-build docker-push IMG=devopps/pod-validationwebhook:v1
go: creating new go.mod: module tmp
go get: added sigs.k8s.io/controller-tools v0.2.5
/Users/batuhan.apaydin/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
/Users/batuhan.apaydin/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go test ./... -coverprofile cover.out
? github.com/developer-guy/pod-validationwebhook [no test files]
? github.com/developer-guy/pod-validationwebhook/api/v1 [no test files]
docker build . -t devopps/pod-validationwebhook:v1
[+] Building 24.2s (16/16) FINISHED
...
docker push devopps/pod-validationwebhook:v1
The push refers to repository [docker.io/devopps/pod-validationwebhook]
45ed4c050068: Pushed
417cb9b79ade: Pushed
v1: digest: sha256:63f3aaf383f34af74027802316f2c58ee750076c2b03046775ad39be818a80c3 size: 739

Once the image push success, we need to do a couple of things before moving onto the deploy section, please remove the lines between 3 and 9 webhookcainjection_patch.yaml which is under the config/default folder because we are not creating MutatingAdmissionWebhook here, and remove line 16 from the kustomization.yaml which is under the same directory. Also, remove line 2 from the kustomization.yaml which is under config/rbac directory, and, you’re all set.

The last thing that we should do here is enabling and deploying the cert-manager, in order to do that we should edit the “config/default/kustomization.yaml” file by uncommenting the sections marked by [WEBHOOK] and [CERTMANAGER] comments.

We know that kubebuilder also uses a cert-manager to manage TLS management for our webhook, so, we should install a cert-manager in our cluster first, let’s do this.

$ minikube start
😄 minikube v1.18.1 on Darwin 10.15.7
✨ Using the virtualbox driver based on user configuration
👍 Starting control plane node minikube in cluster minikube
🔥 Creating virtualbox VM (CPUs=3, Memory=8192MB, Disk=20000MB) …
🐳 Preparing Kubernetes v1.20.2 on Docker 20.10.3 …
▪ Generating certificates and keys …
▪ Booting up control plane …
▪ Configuring RBAC rules …
🔎 Verifying Kubernetes components…
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v4
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use “minikube” cluster and “default” namespace by default
$ helm repo add jetstack https://charts.jetstack.io
"jetstack" has been added to your repositories
$ helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--version v1.2.0 \
--create-namespace \
--set installCRDs=true
NAME: cert-manager
LAST DEPLOYED: Mon Apr 5 22:53:17 2021
NAMESPACE: cert-manager
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
cert-manager has been deployed successfully!
In order to begin issuing certificates, you will need to set up a ClusterIssuer
or Issuer resource (for example, by creating a 'letsencrypt-staging' issuer).
More information on the different types of issuers and how to configure them
can be found in our documentation:
https://cert-manager.io/docs/configuration/For information on how to configure cert-manager to automatically provision
Certificates for Ingress resources, take a look at the `ingress-shim`
documentation:
https://cert-manager.io/docs/usage/ingress/
$ kubectl get pods --namespace cert-manager --watch
NAME READY STATUS RESTARTS AGE
cert-manager-85f9bbcd97-pqmnv 1/1 Running 0 3m46s
cert-manager-cainjector-74459fcc56-nnpjg 1/1 Running 0 3m46s
cert-manager-webhook-57d97ccc67-8vjff 1/1 Running 0 3m46s

Now, we are ready to deploy the webhook.

$ make deploy IMG=devopps/pod-validationwebhook:v1
go: creating new go.mod: module tmp
go get: added sigs.k8s.io/controller-tools v0.2.5
/Users/batuhan.apaydin/go/bin/controller-gen “crd:trivialVersions=true” rbac:roleName=manager-role webhook paths=”./…” output:crd:artifacts:config=config/crd/bases
cd config/manager && kustomize edit set image controller=devopps/pod-validationwebhook:v1
kustomize build config/default | kubectl apply -f -
namespace/pod-validatingwebhook-system created
role.rbac.authorization.k8s.io/pod-validatingwebhook-leader-election-role created
clusterrole.rbac.authorization.k8s.io/pod-validatingwebhook-proxy-role created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/pod-validatingwebhook-metrics-reader created
rolebinding.rbac.authorization.k8s.io/pod-validatingwebhook-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/pod-validatingwebhook-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/pod-validatingwebhook-proxy-rolebinding created
service/pod-validatingwebhook-controller-manager-metrics-service created
service/pod-validatingwebhook-webhook-service created
deployment.apps/pod-validatingwebhook-controller-manager created
certificate.cert-manager.io/pod-validatingwebhook-serving-cert created
issuer.cert-manager.io/pod-validatingwebhook-selfsigned-issuer created
Warning: admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
validatingwebhookconfiguration.admissionregistration.k8s.io/pod-validatingwebhook-validating-webhook-configuration created

Yaay!!! Seems everything is working, but let’s test the webhook by creating Pod which has no required annotation on it.

# test with invalid Pod
$ kubectl run --generator=run-pod/v1 nginx --image=nginx
Found existing alias for “kubectl”. You should use: “k”
Alias tip: k run — generator=run-pod/v1 nginx — image=nginx
Flag — generator has been deprecated, has no effect and will be removed in the future.
Error from server (missing annotation example-mutating-admission-webhook): admission webhook “vpod.kb.io” denied the request: missing annotation example-mutating-admission-webhook
# test with valid Pod
$ kubectl run --generator=run-pod/v1 nginx --image=nginx --overrides='{ "apiVersion": "v1", "metadata": {"annotations": { "required-label":"foo" } } }'
Flag --generator has been deprecated, has no effect and will be removed in the future.
pod/nginx created

Now, we proved that everything is working as we expected 🎉🎉🎉

References

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
developer-guy

developer-guy

I do mostly Go, Kubernetes, and cloud-native stuff ⛵️🐰🐳