Photo by Jason Pofahl on Unsplash

Manage Kubernetes Admission Webhook's certificates with cert-manager CA Injector and Vault PKI📝 🔐⛵️

Authors: Furkan Türkal developer-guy Erkan Zileli

📦 Table of Contents

· ⛵️ Kubernetes Admission Controllers
· 📝 cert-manager and CA Injector
· 🔐 Vault PKI (Public Key Infrastructure)
· 💻 Installation
· 👀 How to monitor certificates?
· ✨ How to accomplish hot-reloading your HTTP server with renewed certificates without having downtime?
· 🎯Conclusion

The certificate management process has always been a problem for people who want to manage their certificates for their applications in various environments. Because there are always challenges, such as how to store them, where to store them, how to revoke or renew them, etc., today, we'll talk about managing your certificates in a Kubernetes platform for your Kubernetes Admission Webhook applications in a more automated fashion. We had talked about five ways of managing certificates in our previous post; if you want to learn more about them, you can reach out to our last blog post. In the previous post, we also mentioned the cert-manager CA Injector, and we have used a self-signed issuer at that post, but today we'll be using the Vault issuer by using Vault's PKI secret engine.

Without further ado, let's jump into the details of all of the technologies we will use in this guide.

As I mentioned above, we'll be talking about managing certificates on behalf of the applications of what we called Kubernetes Admission Webhooks. Still, this approach is not tied to these types of applications only. So you can use the same method for your other kind of applications as long as they are working on Kubernetes.

Kubernetes Admission Controllers ⛵️

An admission controller is a piece of code that intercepts requests to the Kubernetes API server before the persistence of the object but after the request is authenticated and authorized. There are two special controllers: MutatingAdmissionWebhook and ValidatingAdmissionWebhook. These execute the mutating and validating (respectively) admission control webhooks configured in the API. In addition to compiled-in admission plugins, admission plugins can be developed as extensions and run as webhooks configured at runtime. Admission webhooks are HTTP callbacks that receive admission requests and do something with them. Since a webhook must be served via HTTPS, we need proper certificates for the server. These certificates can be self-signed (instead: signed by a self-signed CA), but we need Kubernetes to instruct the respective CA certificate when talking to the webhook server. In addition, the common name (CN) of the certificate must match the server name used by the Kubernetes API server, which for internal services is `<service-name>. "<namespace>.svc.`

https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/

Now that we understand the Kubernetes Admission Webhooks and the need for certification let's continue by explaining how the cert-manager can help us manage these certificates.

cert-manager and CA Injector 📝

Certificate lifecycle management is a critical requirement for a Kubernetes Admission Webhook. As we mentioned, the cert-manager comes into play to simplify that process by saving engineers time and improving security by negating the risk of human error. Cert-manager is well known and more like a de facto standard software for managing certificates on Kubernetes environments. Cert-manager is a popular open-source tool that automates issuing certificates on-demand using Kubernetes APIs and renewing the certificates before they expire. It uses Kubernetes CRDs heavily to do that. In addition, it provides valuable abstractions for TLS certificates and various types of Issuers in the form of Custom Resource Definitions, making it super easy to use and manage from an end-user perspective. We'll be talking about issuers in detail when we install a cert-manager. Still, it is worth mentioning that ​the first thing you'll need to configure after you've established a cert-manager is an issuer which you can then use to issue certificates. The cert-manager comes with several built-in certificate issuers. Vault is one of them, and this is the issuer that we'll demonstrate today. The Vault Issuer represents the certificate authority Vault- a multi-purpose secret store that can be used to sign certificates for your Public Key Infrastructure (PKI). To learn more about them, you can reach out to the official documentation from here.

So far, so good, we've created certificates for the application. Still, there is one more thing left that we need to do: provide CA (Certificate Authority) information in the Kubernetes Admission Webhook registration manifest, which refers to the caBundle property of the configuration. This is where CA Injector comes into the picture. CA Injector helps configure the CA certificates for Mutating Webhooks, Validating Webhooks, and Conversion Webhooks. An injectable resource MUST have one of these annotations: cert-manager.io/inject-ca-from, cert-manager.io/inject-ca-from-secret, or cert-manager.io/inject-apiserver-ca, depending on the injection source. To learn more about them, you can reach out to the official documentation from here.

Vault PKI (Public Key Infrastructure) 🔐

In a nutshell, the Vault PKI secrets engine can streamline generating dynamic X.509 certificates. By using this secrets engine, services can get certificates without going through the usual manual process of generating a private key and CSR, submitting to a CA, and waiting for a verification and signing process to complete. This is done while also providing an authentication and authorization mechanism to validate. To learn more about them, you can reach out to the official documentation from here.

Installation 💻

This guide assumes that we have a webhook application called config-admission-webhook. It lives under the platform namespace behind a Kubernetes service called config-sidecar-injector-service.

First, we need to install Vault on Kubernetes with dev mode enabled. Because we don't want to deal with sealing/unsealing and managing store stuff. To do so, we'll be using Helm to install Vault because HashiCorp provides an excellent chart for Vault. By the way, you can use any Kubernetes clusters to achieve the same hands-on. In this case, we'll be using Minikube. So, let's start by creating our Kubernetes cluster.

$ minikube start -p demo

Then, install Vault by making use of Helm.

Once you deploy Vault, please ensure that you have a running pod named vault-0 in your default namespace before moving into the next step.

$ kubectl get pods
NAME               READY STATUS RESTARTS AGE
vault-0             1/1  Running  0      32s

The next step will be configuring the Vault to enable the PKI secrets engine.

Once you deploy Vault in dev mode enabled, your root password will be “root.” We’ll be using the commands provided in the official documentation of the Vault website. You can reach out to the commands and details on this page.

Before doing that, we should log in to Vault by making use of the following command:

$ kubectl exec vault-0 -- vault login root
Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run “vault login” again. Future Vault requests will automatically use this token.
Key Value
— — — — -
token root
token_accessor cDLx7PbVXcY3ibzweBBgki0h
token_duration ∞
token_renewable false
token_policies [“root”]
identity_policies []
policies [“root”]

First, start an interactive shell session on the vault-0 pod. The commands we will execute after that will be inside the pod until we warn you that you should go back to your terminal.

$ kubectl exec -ti vault-0 -- /bin/sh
/ $

And run the commands within the following gist:

https://gist.github.com/developer-guy/0b128945dbc14f6bdd6009d6f648d4f3

As I said before, we need to retrieve a token from Vault that can work with the PKI secrets engine. This is where Kubernetes Authentication methods came into play.

Then run the exec command to start an interactive shell, then run the following commands within the gist to configure the Kubernetes Authentication method:

https://gist.github.com/developer-guy/d5cfd97f781b3a1f0812544a4ee99560

The next part is installing the cert-manager itself. To do so, run the following commands within the following gist:

https://gist.github.com/developer-guy/d544ae1f299c74cc1baa738c0a853719

Please ensure that all the pods are running within the cert-manager namespace before moving into the next step:

$ kubectl get pods -n cert-manager
NAME                                     READY STATUS RESTARTS AGE
cert-manager-57d89b9548-wmlrl             1/1  Running    0    9m39s
cert-manager-cainjector-5bcf77b697-rsbh9  1/1  Running    0    9m39s
cert-manager-webhook-8687fc66d4–9hfvq     1/1  Running    0    9m39s

Once you set up everything, the next thing is configuring the Vault issuer, lets do that. To do so, you need to apply the following manifest file, but before we need to create a service account within the platform namespace:

$ kubectl create serviceaccount issuer -n platform
$ ISSUER_SECRET_REF=$(kubectl get serviceaccount issuer -n platform -o json | jq -r ".secrets[].name"); echo $ISSUER_SECRET_REF
$ issuer-token-7z8jj

https://gist.github.com/developer-guy/ad2093fc5b78e5fdbf399a03ea8062df

Now, let's create the first certificate. It will be as simple as what we've done so far. We'll be applying the following manifest:

https://gist.github.com/developer-guy/829035a6df9b22c97e401dbdccc0328b

Once you apply the manifest, you should see the status as Ready:

$ kubectl get certificates -n platform config-sidecar-injector-service
NAME                           READY SECRET                      AGE
config-sidecar-injector-service True config-admission-webhook-tls 5s

Also, you can get the certificate's details by using the view-secret kubectl plugin and the cfssl tool.

$ kubectl view-secret config-admission-webhook-tls -n platform tls.crt | cfssl certinfo -cert -

Do not forget to add config-admission-webhook-tls as a volume

Last but least, in this section, we should add an annotation to let cert-manager CA Injector set the caBundle to the webhook configuration. To do so, we'll be editing the webhook configuration like this:

Do not forget, you should leave the caBundle property empty of the webhook configuration.

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  annotations:
     cert-manager.io/inject-ca-from: platform/config-sidecar-injector-service

Once you save the configuration, you should get an output from the following command:

$ yq e ".webhooks[0].clientConfig.caBundle" <(k get mutatingwebhookconfigurations.admissionregistration.k8s.io config-sidecar-injector -oyaml | k neat) | base64 -D | cfssl certinfo -cert -
{
“subject”: {
“common_name”: “config-sidecar-injector-service.platform.svc”,
“names”: [
  “config-sidecar-injector-service.platform.svc”
 ]
},
“issuer”: {
“common_name”: “config-sidecar-injector-service.platform.svc”,
“names”: [
   “config-sidecar-injector-service.platform.svc”
 ]
},
“serial_number”: “568486349258568295357093419160671126608237098261”,
“sans”: [
“config-sidecar-injector-service”,
“config-sidecar-injector-service.platform”,
“config-sidecar-injector-service.platform.svc”
],
“not_before”: “2022–01–04T18:40:41Z”,
“not_after”: “2023–01–04T18:41:10Z”,
“sigalg”: “SHA256WithRSA”,
“authority_key_id”: “4D:13:A6:D2:85:CF:36:98:FD:65:3E:F5:27:A5:38:EF:40:71:90:3E”,
“subject_key_id”: “4D:13:A6:D2:85:CF:36:98:FD:65:3E:F5:27:A5:38:EF:40:71:90:3E”,
“pem”: “ — — -BEGIN CERTIFICATE — — -\nMIID5jCCAs6gAwIBAgIUY5PPPi69t64ZpuNVWFRuPu6o9RUwDQYJKoZIhvcNAQEL\nBQAwNzE1MDMGA1UEAxMsY29uZmlnLXNpZGVjYXItaW5qZWN0b3Itc2VydmljZS5w\nbGF0Zm9ybS5zdmMwHhcNMjIwMTA0MTg0MDQxWhcNMjMwMTA0MTg0MTEwWjA3MTUw\nMwYDVQQDEyxjb25maWctc2lkZWNhci1pbmplY3Rvci1zZXJ2aWNlLnBsYXRmb3Jt\nLnN2YzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL4B3h/Ps2xyumsd\nxkm+Qnl5ZFcRWc0AmEFQRxDNS0z7T/MSjSNoFb+TbuE2hhmbxkFPA45/dUotxp9i\n6ZNMglirzwaxAyI+8MRGkKRoHKqNN/gj8MC9aUqhy38CImbl2AYiGD0jPx/GTj45\nyimUIr3QUTaU9TCQCSigjTzOnG4FIkEp35CPDJg5KM0exD7ItE8TdabwIYwI5BZp\n7o1eJjoOUHf9PufZcgBY0mxaMYVwfKuKz1dq/e/34qGniFduZe0XPMBTUKQxuH6U\nR8gWlKTOl7i+AKT/uATDX22I1TTeKXf6ymGvn4jz52+dY/DfKxlxk88U70XiGAkv\ne+2du38CAwEAAaOB6TCB5jAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB\n/zAdBgNVHQ4EFgQUTROm0oXPNpj9ZT71J6U470BxkD4wHwYDVR0jBBgwFoAUTROm\n0oXPNpj9ZT71J6U470BxkD4wgYIGA1UdEQR7MHmCH2NvbmZpZy1zaWRlY2FyLWlu\namVjdG9yLXNlcnZpY2WCKGNvbmZpZy1zaWRlY2FyLWluamVjdG9yLXNlcnZpY2Uu\ncGxhdGZvcm2CLGNvbmZpZy1zaWRlY2FyLWluamVjdG9yLXNlcnZpY2UucGxhdGZv\ncm0uc3ZjMA0GCSqGSIb3DQEBCwUAA4IBAQBrKZxo1tPLdwn0mZN64mya8P6DsUnf\nsW2rnetsjc5kJYTV6p/8iov/yQPmsnN9bIZSc87wOTa6QF1fL0jlhWVvUz+9DZPv\nwfdYPvv31KQj+9WNquiaXKr/uELTmIYXeD1/ckJx9ZLE0WjUnfMkRxGxsIiB9JYu\n3WzOIqQV5czS2UubrKsvvGmtPpdfK7JJsWyk9Z4Hga78SEPNmErayYk3zjEB5rMK\n6+bVQIP00P/h89iwwjdBcL7DQDdociKQznL/L2Dm2rtUbkVMPi42WAn6xilaGmVJ\n643hBOsPIVBRtI0g2pquGutx00t0kw2LZCS+81rkz7t+9miiy+x7T72c\n — — -END CERTIFICATE — — -\n”
}

You can test your webhook logic by applying some manifest. In this case, I'll be using my sample manifest with –dry-run=server mode. What we expect is that we should not see any certificate error in the output:

$ kubectl apply -f samples/ –dry-run=server
Error from server: error when creating “examples/auto/pod1.yaml”: admission webhook “config-sidecar-injector-service.platform.svc” denied the request: could not find configs

Whee!! 🥳🚀

How to monitor certificates? 👀

A Prometheus exporter for certificates focusing on expiration monitoring. Designed to monitor Kubernetes clusters inside, it can also be used as a standalone exporter.https://github.com/enix/x509-certificate-exporter

Get notified before they expire:

• PEM encoded files by path or scanning directories
• Kubeconfigs with embedded certificates or file references
• TLS Secrets from a Kubernetes cluster

$ helm repo add enix https://charts.enix.io
$ helm install x509-certificate-exporter enix/x509-certificate-exporter

If you want to set Prometheus alerts, you can configure them manually; you can do it by following this page.

https://enix.io/en/blog/avoiding-certificate-expiration-kubernetes-infrastructure

How to accomplish hot-reloading your HTTP server with renewed certificates without having downtime? ✨

There is one more problem in this setup. There is no logic embedded into the webhook that monitors the renewal of the certificates by the cert-manager. Because once the cert-manager renews the certificate, it updates the content of the secret with the details of the new certificates. So, we need to develop some intelligent logic that monitors these changes and updates the server's certificates without causing downtime. This is where all the magic happens!

As you can see, we're using a Kubernetes secret to mount a certificate into our webhook. The webhook starts with a particular certificate. After some amount of time (10m), if you run the command below, you should get an error like the following:

$ kubectl apply -f examples/auto/ --dry-run=server
serviceaccount/auto-sa unchanged (server dry run)
Error from server (InternalError): error when creating “examples/auto/pod1.yaml”: Internal error occurred: failed calling webhook “config-sidecar-injector-service.platform.svc”: Post “https://config-sidecar-injector-service.platform.svc:443/mutate?timeout=30s": x509: certificate has expired or is not yet valid: current time 2022–01–04T19:31:03Z is after 2022–01–04T19:25:35Z

This is the problem that we are trying to solve. Your certificates are renewed and stored in the secret config-admission-webhook-tls, but your webhook still uses the expired one. So what do you do when this happens? Of course, the easiest solution is to restart your webhook. Thus, your webhook uses the renewed certificates after restarting. But you have to restart it continuously. This isn't cool.

We use Kubernetes secrets to store our certificate and mount it into our webhook as a file. This means that when the secret we mounted is updated, we can see this update on the file system. So when our certificate secret is updated, we can update our certificate at runtime without restarting anything. In addition, we can use fsnotify to watch certificate files and update our HTTP server's certificate when any file update event happens on the file system.

But still, there is a problem with fsnotify and Kubernetes. If we watch WRITE events with fsnotify we can't handle the updates because Kubernetes uses symlinks to mount a file.

You can see the events we got are CHMOD and REMOVE. So we can watch REMOVE events to handle the certificate change instead of WRITE events. Please do not make the same mistake. See this article.

The complete code for this solution is below.

Also, certificate update logs happen when your certificate is renewed, and Kubernetes updates the mounted file. After these logs, we can test our webhook by connecting to its shell to see if it's currently using the certificate is up to date with the command below.

You can test this by applying the following manifest in your cluster:

https://gist.github.com/developer-guy/de82fb8e97557ec711ae2dd79ac1d029

So, run the commands below by looking at their certificate dates whether to match within the secret, which includes certificates:

$ DOM="localhost"
$ PORT="8080"
$ openssl s_client -servername $DOM -connect $DOM:$PORT | openssl x509 -noout -dates

As a result, we finally updated our certificate when the cert-manager updated it without any downtime. Now then, we can define short-lived certificates, which makes us more secure than creating long-lived certificates, e.g., ten years and, we automated the certificate update process.

🎯 Conclusion

As you can see that we have used a bunch of awesome tools to end up having an automated, easy-to-use, and observable certificate management process for our Kubernetes workloads. With that, we also have used short-lived certificates for our production-grade Kubernetes workloads which is a good thing from the security perspective of the applications. We also monitored the whole process to avoid production outages that can happen by certificates expirations.

We hope you enjoy the whole story, please stay tuned for the next blogs posts, and do not forget to subscribe and clap 😋🤝