PKI Certs Injection to K8s Pods with Vault Agent Injector
Inject PKI Certs Dynamically to Kubernetes Pods using Vault Agent Injector
Imagine your app running on Kubernetes. Imagine you also have a ton of secrets to be pulled into the Pod before running the Pod. So you have a Vault agent injector running and pulling the secrets from there. Ahh, that's great. You are following the best practice to store and pull your secrets. You are all set, Arent you? Hold on !! How are you getting your SSL / TLS certificates? Kubernetes Secrets again? Nah !! Why don't we use the same vault agent Injector? Is it possible? Yes, why not. So you might have already guessed what this article is all about. In this article, we will be understanding how we can use the Vault Agent Injector to dynamically generate and Inject PKI Certs to the Pod. The Vault Agent Injector alters pod specifications to include Vault Agent containers that render Vault secrets to a shared memory volume using Vault Agent Templates. By rendering secrets to a shared volume, containers within the pod can consume Vault secrets without being Vault aware.
The injector is a Kubernetes Mutation Webhook Controller. The controller intercepts pod events and applies mutations to the pod if annotations exist within the request. This functionality is provided by the vault-k8s project and can be automatically installed and configured using the Vault Helm chart.
What is the entire story all about? (TLDR)
- Using Vault K8s Injector to generate PKI Certificates.
- Generate PKI Certificates Dynamically.
Prerequisites
- A Kubernetes Cluster ( EKS, AKS, Kind, etc )
Story Resources
- GitHub Link: https://github.com/pavan-kumar-99/medium-manifests
- GitHub Branch: pki-agent-injector
Installing Hashicorp vault using Helm Chart
We will install the official helm chart for the vault and unseal it manually. However, this is not the Ideal way for running a vault in Production. You may want to unseal the vault using a KMS Key ( If being installed in AWS ) or a Google KMS key ( If being installed in GCP ).
Helm Install PKI Vault
Once the helm chart is installed, you should find a Stateful set created with the name vault and also a pod named vault-0 created. However the pod is not in a ready state, it is because the vault is not unsealed yet. Let us now unseal the vault by exec-ing into the Pod.
vault_pki_unseal.sh
Let us understand what the init command does
vault operator init -key-shares=1 -key-threshold=1
-key-shares=Number of key shares to split the generated master key into. By default it is 5 and 3 keys are needed to unseal the vault
-key-threshold=Number of key shares required to reconstruct the master key. This must be less than or equal to -key-shares. 3 by default.
Generating Root CA and Intermediate CA
Vault PKI backend can generate its own self-signed CA. But let us create our own CA and then upload it to the vault PKI backed for signing our certificates.
We now have our certificates generated, let us now configure our vault PKI backend to add these certificates as the CA. We will also have to create the vault roles and the vault issuing and CRL URLs.
Configure Vault’s PKI Backend
Let us understand the commands one by one
vault login $VAULT_ROOT_TOKEN = Used to login to vault with the root token.
vault secrets enable pki = Used to enable the vault pki backend.
vault secrets tune -max-lease-ttl=8760h pki = We configure the ttl of the pki backend to be 8760 hours.
vault write pki/config/ca pem_bundle=@pem_bundle ttl=8760h => We are writing the CA certificate to vault and setting its ttl to 8760h
Now we have our PKI certificate uploaded to the vault. Let us now create a Vault PKI role. The role definition sets the conditions under which a certificate can be generated.
Once we have the role created, we should configure the endpoints from which the certificates can be issued and revoked.
Let us now create a vault policy for the corresponding vault PKI role.
Enabling Kubernetes Auth Method
The Kubernetes auth method can be used to authenticate with the vault using the Kubernetes service account token. This will help the vault to inject the vault token into the Kubernetes Pod. For this let us enable the vault Kubernetes backend by exec into the pod, since the token_reviewer_jwt has to be passed from the vault pod.
We now have a Kubernetes auth role created. This role is bound to the service account names issuer and namespace named default and the policy named betttercallpavan_pki which was created earlier.
$ git clone https://github.com/pavan-kumar-99/medium-manifests.git \
-b pki-agent-injector$ cd medium-manifests$ kubectl apply -f pki-pod.yaml
Using Vault Injector with PKI Certificates.
A list of all the annotations can be found here
kubectl exec -it pki-test -c pki-test — /bin/bash -c “ls /vault/secrets”
kubectl exec -it pki-test -c pki-test — /bin/bash -c “cat /vault/secrets/server.crt”
You should now see the certificates and keys present inside the pod. This is how the PKI Certs can be injected automatically without the need for any sidecar or init containers or any other additional changes. So whenever the pod is restarted the certs are automatically regenerated and injected into the Pod.
Cleanup
$ git clone https://github.com/pavan-kumar-99/medium-manifests.git \
-b pki-agent-injector$ cd medium-manifests$ kubectl delete -f pki-pod.yaml
