Trendyol Tech
Published in

Trendyol Tech

Photo by ANIRUDH on Unsplash

5 Ways of Managing TLS Certificates for your Kubernetes Admission Webhooks

Written by Erkan Zileli & Furkan Türkal & developer-guy

In Trendyol, we are using Kubernetes Admission Webhooks quite heavily. Because Kubernetes is an extensible platform and we love to extend it by writing our own Kubernetes Admission Webhooks and Operators according to our business requirements. If you decide to do the same thing within your organization, I wrote two blog posts (part 1 and part 2) about how to start writing your own Kubernetes Admission Webhooks. I recommend you take a look at those before continuing to read this one. Because this blog post is just aiming to give you a quick overview of which options do you have while managing your TLS Certificates for your Kubernetes Admission Webhooks.

While writing your own Kubernetes Admission Webhook, at some point, you will notice that you have to manage your TLS Certificate for it in order to talk with the k8s API server over TLS because this is a must process that you can see the details from here. As I mentioned above, there is more than one option that you can use to manage your TLS certificates. So I decided to make a blog post and bring ways together to give you a better quick starter.

Table of Content 🔮

· cert-manager CA Injector and BotKube 📝🤖
BotKube
Installation
cert-manager
Installation
1. Create Certificate resource and inject it to WebhookConfiguration
2. Create a Secret with a certificate and inject it to WebhookConfiguration
·
Helm Hook and Certificator ⎈⚡️
·
NewRelic k8s-webhook-cert-manager 🚀
·
webhook-create-signed-cert.sh 📜
·
Custom Admission Webhook Server Init Container 📦
·
Conclusion 🎬

cert-manager CA Injector and BotKube

In this setup, we are going to generate certificates with a cert-manager CA Injector and inject them into our WebhookConfigurations and to be informed about any certificate change with the help of BotKube.

BotKube

According to their website, BotKube is a messaging bot for monitoring and debugging Kubernetes clusters and, we are gonna use it to get information about and monitor any certificate changes.

Installation

To install it, first, get a Slack token for BotKube as they specified on their documentation, and be sure that you installed BotKube to your channel you want to be notified from. Then use your token with the configuration below.

https://gist.github.com/developer-guy/fa9b90ea77a642858c2697841b7b4ca6

The rest is easy. Just run the following command to install the BotKube with Helm.

https://gist.github.com/developer-guy/b847b4e1e934a26d20b0a210c64b1175

That’s it. You can check the BotKube logs on the monitoring namespace to ensure that BotKube is running successfully. Now let’s dive into the cert-manager part.

cert-manager

cert-manager builds on top of Kubernetes and OpenShift to provide X.509 certificates and issuers as first-class resource types. Also, the cert-manager CA Injector is another Deployment that comes with the cert-manager installation. It basically injects certificates to any specified Webhook Configuration.

Installation

You can find various installation methods for cert-manager but today we are going to use Helm.

https://gist.github.com/developer-guy/d544ae1f299c74cc1baa738c0a853719

Now we installed cert-manager and are ready to automate Webhook certificates. CertManager allows us to generate WebHook certificates in few similar ways. Let’s discover them.

1. Create Certificate resource and inject it to WebhookConfiguration

Let’s just apply these resources to the Kubernetes and talk about them.

https://gist.github.com/developer-guy/7c70cfa63f5cdeb1c5e53466341d0b9f

We have created an Issuer to sign self-sign Certificates. Usually creating this once enough. You don't have to duplicate this for each WebHook.

Another resource is Certificate which should be created per WebHook. When you created these you got a Secretname as my-webhook-certificate-secret. This secret contains tls.crt, tls.key and ca.crt and your TLS Certificate is ready to use.

After that, you should annotate your WebhookConfiguration with the Certificate name you create. The annotation cert-manager.io/inject-ca-from provides a way to injects the ca.crt from the Certificate's Secret.

Applying this to the Kubernetes, you can see you caBundle on your WebhookConfiguration. Let's see.

https://gist.github.com/developer-guy/180f3fc316c5038c5b73f89b35af51b0

And this is how you automate your webhook certificates with CertManager.

2. Create a Secret with a certificate and inject it to WebhookConfiguration

Another alternative way, you can create your own certificate as a Secret resource and ask if CertManager to inject this into your WebhookConfiguration. Create a Secret with a specific annotation that cert-manager understands this secret contains the custom certificate and is injectable to any WebhookConfiguration. Annotate your WebhookConfiguration to inject this certificate to the caBundle. Let's try it.

https://gist.github.com/developer-guy/454cd8d0355e2bc0c17c93ccd88841c9

You can choose any options when using the cert-manager. Now let’s look at how cert-manager and BotKube integrate with themselves. Here are some of the BotKube messages sent to the Slack channel you specified.

When you integrate BotKube and create a Certificate resource

When you updated a Certificate resource

Helm Hook and Certificator

If you are already using Helm to deploy your application, this option would be the better option for you because you can easily integrate TLS management for your Kubernetes Admission Webhook using Helm Hooks and Certificator project. If you want to get more detail about these, I wrote another article about how you can use them, you can reach out from here.

In a nutshell, the Certificator project as the name suggests handles the TLS Certificates creation process, it basically uses Kubernetes Certificates API.
The Certificates API enables automation of X.509 credential provisioning by providing a programmatic interface for clients of the Kubernetes API to request and obtain X.509 certificates from a Certificate Authority (CA). This is also another topic in Kubernetes, you can reach out the details of the request signing process from here. Certificator project abstracts that process and creates necessary stuff to create CertificateSigningRequest resource and the resource itself then stores the certificates in a Kubernetes Secret and patches the caBundle field of your admission webhook. But this resource has to be created before the actual application is deployed. So, this is where the Helm Hooks comes into the picture.

At most basic form, the Certificator project can be used as Job resource because it is a CLI application and managed by Helm as you can see from the following definition:

https://gist.github.com/developer-guy/5f23eab3366dd206e880964f5b6c8b49

As you can see from the above manifest definition, there are some annotations in order to use Helm Hooks. This will help us to create a Job before deploying the application then if it succeeds, it will be deleted by Helm, please do not forget that the resources that a hook creates are currently not tracked or managed as part of the release. So, to manage your TLS certificates for your Kubernetes Admission Webhooks, just add that definition above to your templates and start using it, that's it 🚀🤩

NewRelic k8s-webhook-cert-manager

The Certificator project that I mentioned above is created based on the k8s-webhook-cert-manager project created by NewRelic. k8s-webhook-cert-manager project is a shell script and the Certificator project is just a re-implementation of it with Golang, both projects do the same thing.

This is a detailed list of steps the script is executing:

  • Generate a server key.
  • If there is any previous CSR (certificate signing request) for this key, it is deleted.
  • Generate a CSR for such key.
  • The signature of the key is then approved.
  • The server’s certificate is fetched from the CSR and then encoded.
  • A secret of type TLS is created with the server certificate and key.
  • The k8s extension API server’s CA bundle is fetched.
  • The mutating webhook configuration for the webhook server is patched with the k8s API server’s CA bundle from the previous step. This CA bundle will be used by the k8s extension API server when calling our webhook.

Download this project and run the following command in order to create a TLS certificate:

https://gist.github.com/developer-guy/bd306741b3a12d6311776121e9e94445

webhook-create-signed-cert.sh

If you want to create TLS certificates manually and load them into your application somehow, this option can help you or even you use Helm you can use built-in functions to store them in a Kubernetes Secret, for more detail, please see. This option is a bit manual because it is not creating or patching the webhook, it is just for creating self-signed X.509 certificates and that’s all.

https://gist.github.com/developer-guy/dd5f8df4f73b3f51819248a6f2e98c48

You can follow up on this tutorial to learn how you can use it for your project.

Custom Admission Webhook Server Init Container

I think this is the fanciest option among others. I found this option from the Velotio Blog and highly recommend you to take a look at the other posts also, but here is the article that I mentioned in the beginning, you can learn all the details about this option from there because I just want you to give a basic idea about how you can use it.

The main function of this init container will be to create a self-signed webhook server certificate and provide the CA bundle to the API server via mutation/validation configs as we did in the previous options but this time we are using the init container thing to do it. This init container will run a simple Go binary to perform all these functions.

https://gist.github.com/velotiotech/b63c6cf34d7327d0a2a39cd4cb11e152#file-webhook_pod_spec-yaml

Conclusion

We talked about the options that might help us to make that tedious process easier. You can choose one of them and integrate it with your project because all the options are really effortless to implement and use. I also would love to hear about the other options that you might be already using within your organization or individual projects, so please share them with us in the comments. 😋🙏

I hope this post will give you the quick and best options to start to create TLS certificates for your Kubernetes Admission Webhooks.

See you in the next blog posts, please stay tuned 😇🙋🏻‍♂️

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
developer-guy

developer-guy

I do mostly Go, Kubernetes, and cloud-native stuff ⛵️🐰🐳