5 Ways of Managing TLS Certificates for your Kubernetes Admission Webhooks
Written by Erkan Zileli & Furkan Türkal & developer-guy
In Trendyol, we are using Kubernetes Admission Webhooks quite heavily. Because Kubernetes is an extensible platform and we love to extend it by writing our own Kubernetes Admission Webhooks and Operators according to our business requirements. If you decide to do the same thing within your organization, I wrote two blog posts (part 1 and part 2) about how to start writing your own Kubernetes Admission Webhooks. I recommend you take a look at those before continuing to read this one. Because this blog post is just aiming to give you a quick overview of which options do you have while managing your TLS Certificates for your Kubernetes Admission Webhooks.
While writing your own Kubernetes Admission Webhook, at some point, you will notice that you have to manage your TLS Certificate for it in order to talk with the k8s API server over TLS because this is a must process that you can see the details from here. As I mentioned above, there is more than one option that you can use to manage your TLS certificates. So I decided to make a blog post and bring ways together to give you a better quick starter.
Table of Content 🔮
· cert-manager CA Injector and BotKube 📝🤖
∘ BotKube
∘ Installation
∘ cert-manager
∘ Installation
∘ 1. Create Certificate resource and inject it to WebhookConfiguration
∘ 2. Create a Secret with a certificate and inject it to WebhookConfiguration
· Helm Hook and Certificator ⎈⚡️
· NewRelic k8s-webhook-cert-manager 🚀
· webhook-create-signed-cert.sh 📜
· Custom Admission Webhook Server Init Container 📦
· Conclusion 🎬
cert-manager CA Injector and BotKube
In this setup, we are going to generate certificates with a cert-manager CA Injector and inject them into our WebhookConfigurations and to be informed about any certificate change with the help of BotKube.
BotKube
According to their website, BotKube is a messaging bot for monitoring and debugging Kubernetes clusters and, we are gonna use it to get information about and monitor any certificate changes.
Installation
To install it, first, get a Slack token for BotKube as they specified on their documentation, and be sure that you installed BotKube to your channel you want to be notified from. Then use your token with the configuration below.
The rest is easy. Just run the following command to install the BotKube with Helm.
That’s it. You can check the BotKube logs on the monitoring namespace to ensure that BotKube is running successfully. Now let’s dive into the cert-manager part.
cert-manager
cert-manager builds on top of Kubernetes and OpenShift to provide X.509 certificates and issuers as first-class resource types. Also, the cert-manager CA Injector is another Deployment that comes with the cert-manager installation. It basically injects certificates to any specified Webhook Configuration.
Installation
You can find various installation methods for cert-manager but today we are going to use Helm.
Now we installed cert-manager and are ready to automate Webhook certificates. CertManager allows us to generate WebHook certificates in few similar ways. Let’s discover them.
1. Create Certificate resource and inject it to WebhookConfiguration
Let’s just apply these resources to the Kubernetes and talk about them.
We have created an Issuer to sign self-sign Certificates. Usually creating this once enough. You don't have to duplicate this for each WebHook.
Another resource is Certificate which should be created per WebHook. When you created these you got a Secretname as my-webhook-certificate-secret. This secret contains tls.crt, tls.key and ca.crt and your TLS Certificate is ready to use.
After that, you should annotate your WebhookConfiguration with the Certificate name you create. The annotation cert-manager.io/inject-ca-from provides a way to injects the ca.crt from the Certificate's Secret.
Applying this to the Kubernetes, you can see you caBundle on your WebhookConfiguration. Let's see.
And this is how you automate your webhook certificates with CertManager.
2. Create a Secret with a certificate and inject it to WebhookConfiguration
Another alternative way, you can create your own certificate as a Secret resource and ask if CertManager to inject this into your WebhookConfiguration. Create a Secret with a specific annotation that cert-manager understands this secret contains the custom certificate and is injectable to any WebhookConfiguration. Annotate your WebhookConfiguration to inject this certificate to the caBundle. Let's try it.
You can choose any options when using the cert-manager. Now let’s look at how cert-manager and BotKube integrate with themselves. Here are some of the BotKube messages sent to the Slack channel you specified.
When you integrate BotKube and create a Certificate resource
When you updated a Certificate resource
Helm Hook and Certificator
If you are already using Helm to deploy your application, this option would be the better option for you because you can easily integrate TLS management for your Kubernetes Admission Webhook using Helm Hooks and Certificator project. If you want to get more detail about these, I wrote another article about how you can use them, you can reach out from here.
In a nutshell, the Certificator project as the name suggests handles the TLS Certificates creation process, it basically uses Kubernetes Certificates API.
The Certificates API enables automation of X.509 credential provisioning by providing a programmatic interface for clients of the Kubernetes API to request and obtain X.509 certificates from a Certificate Authority (CA). This is also another topic in Kubernetes, you can reach out the details of the request signing process from here. Certificator project abstracts that process and creates necessary stuff to create CertificateSigningRequest resource and the resource itself then stores the certificates in a Kubernetes Secret and patches the caBundle field of your admission webhook. But this resource has to be created before the actual application is deployed. So, this is where the Helm Hooks comes into the picture.
At most basic form, the Certificator project can be used as Job resource because it is a CLI application and managed by Helm as you can see from the following definition:
As you can see from the above manifest definition, there are some annotations in order to use Helm Hooks. This will help us to create a Job before deploying the application then if it succeeds, it will be deleted by Helm, please do not forget that the resources that a hook creates are currently not tracked or managed as part of the release. So, to manage your TLS certificates for your Kubernetes Admission Webhooks, just add that definition above to your templates and start using it, that's it 🚀🤩
NewRelic k8s-webhook-cert-manager
The Certificator project that I mentioned above is created based on the k8s-webhook-cert-manager project created by NewRelic. k8s-webhook-cert-manager project is a shell script and the Certificator project is just a re-implementation of it with Golang, both projects do the same thing.
This is a detailed list of steps the script is executing:
- Generate a server key.
- If there is any previous CSR (certificate signing request) for this key, it is deleted.
- Generate a CSR for such key.
- The signature of the key is then approved.
- The server’s certificate is fetched from the CSR and then encoded.
- A secret of type TLS is created with the server certificate and key.
- The k8s extension API server’s CA bundle is fetched.
- The mutating webhook configuration for the webhook server is patched with the k8s API server’s CA bundle from the previous step. This CA bundle will be used by the k8s extension API server when calling our webhook.
Download this project and run the following command in order to create a TLS certificate:
webhook-create-signed-cert.sh
If you want to create TLS certificates manually and load them into your application somehow, this option can help you or even you use Helm you can use built-in functions to store them in a Kubernetes Secret, for more detail, please see. This option is a bit manual because it is not creating or patching the webhook, it is just for creating self-signed X.509 certificates and that’s all.
You can follow up on this tutorial to learn how you can use it for your project.
Custom Admission Webhook Server Init Container
I think this is the fanciest option among others. I found this option from the Velotio Blog and highly recommend you to take a look at the other posts also, but here is the article that I mentioned in the beginning, you can learn all the details about this option from there because I just want you to give a basic idea about how you can use it.
The main function of this init container will be to create a self-signed webhook server certificate and provide the CA bundle to the API server via mutation/validation configs as we did in the previous options but this time we are using the init container thing to do it. This init container will run a simple Go binary to perform all these functions.
Conclusion
We talked about the options that might help us to make that tedious process easier. You can choose one of them and integrate it with your project because all the options are really effortless to implement and use. I also would love to hear about the other options that you might be already using within your organization or individual projects, so please share them with us in the comments. 😋🙏
I hope this post will give you the quick and best options to start to create TLS certificates for your Kubernetes Admission Webhooks.
See you in the next blog posts, please stay tuned 😇🙋🏻♂️
