Automating Vault secrets consumption in Kubernetes via mutation webhook
This blog post is a continuation/improvement on my previous post (vault-consul-high-availability-end-to-end-tls-on-kubernetes post).
Using Vault on Kubernetes is the most secure way for managing and retrieving secrets, however, consuming them on a Pod becomes a bit of a challenge as you need to write a custom wrapper script to login to Vault with the service account token, get the secrets and export each as an environment variable, and then replace the wrapper process with your intended command using Linux (so that the process inherits the env and stays PID: 1).
This approach works but introduces constraints, for a start you have to have a supporting environment setup (be that bash or python or node), you can not apply it to any Helm chart you’ll find because you will need to make changes to either the deployment.yaml file or and to the docker image running in it.
Wouldn’t it be simpler for users and for new installs of helm chart to automatically set it up just by pod annotations?
The following code and knowledge are heavily based on the excellent work Baznai Cloud has done and I’d like to give them all the credit (you should read their blog, it’s excellent!).
Kubernetes Validation and Mutation Webhooks
Kubernetes has a built-in Admission controller that can intercepts requests to the Kubernetes API server prior to persistence of the object but after the request is authenticated and authorized.
There are two special controllers built into the kube-apiserver binary:
- MutatingAdmissionWebhook
- ValidatingAdmissionWebhook
These execute the mutating and validating of requests after authentication and authorization.
Mutating controllers may modify the objects they admit while validating controllers may or may not admit an object to the API server.
These webhooks objects can be installed just like a normal object in kubernetes via kubectl (they consist of the https address for the actual webhook logic).
Once installed, every request will go through this webhook logic.
Vault Secrets Webhook
Every new request to the API server will go through this webhook.
The webhook will check if the object is a Pod and only mutate the object if it contains specific Pod annotations:
when it finds these annotations, it will modify the Pod object as follows:
- add a shared in-memory volume
- add an init container with the vault-env binary and a command to copy vault-env to that shared volume
- change the Pod command to be vault-env <original-command><orignal args>
- add vault environment variables (ROLE, CA_PATH, SECRET_PATH) for easier vault operations.
Important to note:
the webhook does not modify the Pod env vars, the secrets from Vault are never saved to etcd and cannot be viewed on the modified Pod object.
Requirements to make it work:
- the Pod must have a section of env with the environment variable and the value prefixed by vault: or vault-env won’t add the secret from Vault.
- The Pod must have an explicit command, the webhook cannot determine what is the command / entrypoint for a given docker image.
if you need to find the original command you need to run:
docker inspect <image id>How vault-env works:
vault-env is a compile go binary, it reads the Kubernetes service account token from the Pod and uses that to authenticate to Kubernetes backend via Vault, which gives back a vault token that is then used to talk to Vault and get the secrets.
vault-env creates a new environment for the process that needs the secrets,
only secrets that are prefixed in the env section of the vault with vault: are added to that new environment.
then vault-env is calling exec which replace the running process(vault-env) with the given command and the new environment, resulting in PID: 1 with the secrets exported as env-vars.
Installing the webhook:
We are going to install the webhook via a Helm chart.
Before you install this chart you must create a namespace for it, this is due to the order in which the resources in the charts are applied (Helm collects all of the resources in a given chart and it’s dependencies, groups them by resource type, and then installs them in a predefined order (see here — Helm 2.10).
The MutatingWebhookConfiguration gets created before the actual backend Pod which serves as the webhook itself, Kubernetes would like to mutate that pod as well, but it is not ready to mutate yet (infinite recursion in logic).
export WEBHOOK_NS=`<namepsace>`
WEBHOOK_NS=${WEBHOOK_NS:-vault-secrets-webhook}
kubectl create namespace "${WEBHOOK_NS}"
kubectl label ns "${WEBHOOK_NS}" name="${WEBHOOK_NS}"Get the chart:
git clone https://github.com/innovia/kubernetes-mutation-webhook-vault-secrets && cd kubernetes-mutation-webhook-vault-secretsInstall the chart:
helm upgrade --namespace "${WEBHOOK_NS}" --install vault-secrets-webhook helm-chartTesting the webhook with a KV secrets v1:
a kv secret v1 is simply a secret with no version management.
- create a vault secret at secret/test/vault/auto/secret
- Create a policy granting read to the secret
- create a named role
you can use the terminal icon on the top right of the UI:
vault write auth/kubernetes/role/tester
bound_service_account_names=tester
bound_service_account_namespaces=default
policies=test_policy
ttl=1h- note that you will need to copy the ca.pem from the vault-tls secret to each given namespace
$ kubectl get secret vault-tls -o jsonpath="{.data['ca\.pem']}" | base64 -D > /tmp/ca.pem; kubectl create secret generic vault-consul-ca --from-literal=ca.pem="$(cat /tmp/ca.pem)"- apply the new pod
- Check the logs and you should see the secret “SsshIAMSecret”
Testing the webhook with a KV secrets v2:
note: This will only take the latest version and does not pull a specific version fro a secret (I’m planning on adding that soon)
the only difference between v1 and v2 secret are:
annotation for v1:
vault.security/vault-path: "secret/foo"annotation for v2:
vault.security/vault-path: "secret/data/foo"you must match that path in the policy too.
feel free to check the source code:
