HashiCorp Vault: Delivering Secrets with Kubernetes
Introduction
In this blog post we will walk through an example of delivering database credentials from Vault to a Kubernetes pod using the Vault Agent Sidecar Injector.
What does the Vault Agent Sidecar do? From the documentation: “The Vault Agent Injector alters pod specifications to include Vault Agent containers that render Vault secrets to a shared memory volume using Vault Agent Templates. By rendering secrets to a shared volume, containers within the pod can consume Vault secrets without being Vault aware.”
The injector is a Kubernetes Mutating Admission Webhook Controller. The controller intercepts pod events and applies mutations to the pod if specific Vault Agent annotations exist in the resource’s spec. This functionality is provided by the vault-k8s project and can be automatically installed and configured using the official HashiCorp Vault helm chart.
Delivering secrets to Kubernetes pods can also be accomplished via Vault integrated libraries. However, making application changes can be cost prohibitive or time consuming for development teams. The last section of this blog shows a short example of using natively integrated libraries.
Demo
The demo code used for this post can be found here. See the Readme for setup instructions.
See the following guide before running Vault inside Kubernetes in production environments: https://learn.hashicorp.com/vault/getting-started-k8s/k8s-security-concerns
Authentication
Before we can deliver secrets to our applications, they must authenticate with Vault. This is done via the Kubernetes authentication method.
There are several great examples on the web of setting up Kubernetes authentication in Vault. Here is one from the HashiCorp “Learn” website.
I’ll briefly explain how authentication works in our demo setup…
First, we create a Kubernetes service account for Vault that grants it permission to use the token review API. Example file here.
$ kubectl create serviceaccount vault-auth$ kubectl apply — filename vault-auth-service-account.yaml
Next, we set several variables that will be used by Vault to talk with Kubernetes. When our agents/pods try authenticating to Vault they will present their Kubernetes service account JWT tokens. Vault will use these configured values to talk with the Kubernetes API for validating client/pod tokens. File.
# Set VAULT_SA_NAME to the service account you created earlier
$ export VAULT_SA_NAME=$(kubectl get sa vault-auth -o jsonpath="{.secrets[*]['name']}")# Set SA_JWT_TOKEN value to the service account JWT used to access the TokenReview API
$ export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data.token}" | base64 --decode; echo)# Set SA_CA_CRT to the PEM encoded CA cert used to talk to Kubernetes API
$ export SA_CA_CRT=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo)
Next we enable the Kubernetes authentication method in Vault. We configure the auth method with the values from above so Vault can interact with the Kubernetes API.
$ export K8S_HOST="https://kubernetes.default.svc:443"
$ vault auth enable kubernetes$ vault write auth/kubernetes/config \
token_reviewer_jwt="$SA_JWT_TOKEN" \
kubernetes_host="$K8S_HOST" \
kubernetes_ca_cert="$SA_CA_CRT"
Finally, we set up a “role” in Vault. When clients or pods try authenticating with Vault, they will authenticate against this role. If authentication is successful, they will receive a Vault token with the “transit-app-example” policy attached. That policy dictates what paths clients can access in Vault.
$ vault write auth/kubernetes/role/example \
bound_service_account_names=vault-auth \
bound_service_account_namespaces=default \
policies=transit-app-example \
ttl=24hThis diagram explains the authentication workflow.
Vault Agent Sidecar Injection
If your application is not natively Vault aware, we can leverage the Vault Agent Sidecar Injector to pull secrets from Vault upon pod creation.
Documentation: https://www.vaultproject.io/docs/platform/k8s/injector/
More Examples: https://www.vaultproject.io/docs/platform/k8s/injector/examples/
The Vault Agent Sidecar builds on the authentication pieces described above by adding options for mounting secrets (pulled from Vault) as a shared volume in your application’s container. There are also options to run the Vault Agent as an init container or a sidecar container depending on your needs.
Vault Agent Sidecar Injector Diagram
Walkthrough
We will be deploying Vault inside Kubernetes via the official helm chart. To enable the Vault agent sidecar injector see the below changes to the helm values.yaml file.
The file is located here.
injector:
# True if you want to enable vault agent injection.
enabled: true# image sets the repo and tag of the vault-k8s image to use for the injector.
image:
repository: "hashicorp/vault-k8s"
tag: "0.2.0"
pullPolicy: IfNotPresent# agentImage sets the repo and tag of the Vault image to use for the Vault Agent
# containers. This should be set to the official Vault image. Vault 1.3.1+ is
# required.
agentImage:
repository: "vault"
tag: "1.3.2"
Once the demo is fully provisioned and Vault is installed, you should see a pod called Vault-0 as well as vault-agent-injector (in bold below). The Injector uses mutating admission webhooks to inject the Vault Sidecar into applicable pods based on their configured annotations.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
consul-consul-5gfrx 1/1 Running 0 5m54s
consul-consul-6wvm6 1/1 Running 0 5m54s
consul-consul-connect-injector-webhook-deployment-6fdd799-s4cbt 1/1 Running 0 5m54s
consul-consul-server-0 1/1 Running 0 5m53s
consul-consul-server-1 1/1 Running 0 5m53s
consul-consul-server-2 1/1 Running 0 5m53s
consul-consul-sync-catalog-6569bf5c98-4d8n5 1/1 Running 0 5m54s
consul-consul-zk86g 1/1 Running 0 5m54s
k8s-transit-app-7f8fff44f5-6v8jq 4/4 Running 1 2m51s
k8s-transit-app-7f8fff44f5-n88zj 4/4 Running 1 2m51s
k8s-transit-app-7f8fff44f5-ppb5f 4/4 Running 1 2m51s
mariadb-0 3/3 Running 0 5m3s
vault-0 3/3 Running 0 4m9s
vault-agent-injector-69c55564d9-sksf9 1/1 Running 0 4m9sThe available sidecar annotations are documented here.
Next, we add annotations to our application’s deployment spec to enable the Vault Agent Sidecar Injector. The demo deployment spec is located here.
Deployment spec’s annotations:
annotations:
1 vault.hashicorp.com/agent-inject: "true"
2 vault.hashicorp.com/agent-inject-status: "update"
3 vault.hashicorp.com/role: "example"
4 vault.hashicorp.com/agent-inject-secret-config.ini:"lob_a/workshop/database/roles/workshop-app"
5 vault.hashicorp.com/agent-inject-template-config.ini: |
[DEFAULT]
LogLevel = DEBUG
[DATABASE]
Address=127.0.0.1
Port=3306
6 {{ with secret "lob_a/workshop/database/creds/workshop-app" -}}
User={{ .Data.username }}
Password={{ .Data.password }}
{{- end }}
Database=app
[VAULT]
Enabled=True
DynamicDBCreds=False
DynamicDBCredsPath=lob_a/workshop/database/creds/workshop-app
Platform=kubernetes
ProtectRecords=False
Address=http://127.0.0.1:8200
Token=root
KeyPath=lob_a/workshop/transit
KeyName=customer-key
[CONSUL]
DEBUG=FalseIn our example, the application is expecting its configuration file to be located at “/vault/secrets/config.ini”. We use the Vault Agent’s templating feature to pull database credentials from Vault before templating the file (and credentials) to a shared volume.
Point 1 above enables the sidecar injector for the pod(s).
Point 2 enables an active sidecar to keep the pod secrets up to date.
Point 3 configures the Vault Agent to authenticate against the “example” role for the Kubernetes authentication method that we set up in the above sections.
Point 4 tells the Vault agent which location to pull a secret from inside Vault. In this case, we are requesting a short-lived (dynamic) mysql username and password from Vault.
Point 5 creates a configuration template to be executed by the Vault Agent Sidecar. The name of the templated file is given in the annotation: “vault.hashicorp.com/agent-inject-template-config.ini:”. This creates the templated file at “/vault/secrets/config.ini” in the application container (via shared mounted volume).
Point 6 is an example of the templating language that writes the secret into the file. More templating examples can be found here.
Next, we inspect the application pod to see our vault agent sidecar container once deployed.
$ kubectl describe pods k8s-transit-app-7f8fff44f5-rrwkz
. . .vault-agent:
Container ID: docker://a52b366450de972c8d2496405f9b6d19cbe6274f12728124268ae40618b8c9e1
Image: vault:1.3.2
Image ID: docker-pullable://vault@sha256:cf9d54f9a5ead66076066e208dbdca2094531036d4b053c596341cefb17ebf95
Port: <none>
Host Port: <none>
Command:
/bin/sh
-ec
Args:
echo ${VAULT_CONFIG?} | base64 -d > /tmp/config.json && vault agent -config=/tmp/config.json
State: Running
Started: Fri, 13 Mar 2020 10:49:41 -0500
Ready: True
Restart Count: 0
Limits:
cpu: 500m
memory: 128Mi
Requests:
cpu: 250m
memory: 64Mi
Environment:
VAULT_CONFIG: eyJhdXRvX2F1dGgiOnsibWV0aG9kIjp7InR5cGUiOiJrdWJlcm5ldGVzIiwibW91bnRfcGF0aCI6ImF1dGgva3ViZXJuZXRlcyIsImNvbmZpZyI6eyJyb2xlIjoiZXhhbXBsZSJ9fSwic2luayI6W3sidHlwZSI6ImZpbGUiLCJjb25maWciOnsicGF0aCI6Ii9ob21lL3ZhdWx0Ly50b2tlbiJ9fV19LCJleGl0XZXNzPWh0dHA6Ly8xMjcuMC4wLjE6ODIwMFxuVG9rZW49cm9vdFxuS2V5UGF0aD1sb2JfYS93b3Jrc2h
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from vault-auth-token-4wrdq (ro)
/vault/secrets from vault-secrets (rw)
Let’s make sure the application retrieved its credentials by inspecting the application logs. Note: We connect to 127.0.0.1 in this demo as we are leveraging Consul Connect and Envoy sidecar proxies for service mesh capabilities.
$ kubectl logs k8s-transit-app-7f8fff44f5-6v8jq k8s-transit-app2020-03-13 15:49:43,028 - INFO - app - <module> - Application running on Kubernetes
2020-03-13 15:49:43,028 - WARNING - db_client - init_vault_k8s - Connecting to vault server for k8s auth: http://127.0.0.1:8200
2020-03-13 15:49:43,030 - DEBUG - urllib3.connectionpool - _new_conn - Starting new HTTP connection (1): 127.0.0.1:8200
2020-03-13 15:49:43,047 - DEBUG - urllib3.connectionpool - _make_request - http://127.0.0.1:8200 "POST /v1/auth/kubernetes/login HTTP/1.1" 200 685
2020-03-13 15:49:43,048 - DEBUG - db_client - init_vault_k8s - Initialized vault_client: <hvac.v1.Client object at 0x7ffa2f49fdd8>
2020-03-13 15:49:43,048 - INFO - app - <module> - Using DB credentials from config.ini...
2020-03-13 15:49:43,048 - DEBUG - db_client - connect_db - Connecting to 127.0.0.1 with username v-kubernetes-workshop-a-3TXp6M96 and password A1a-xl6971suVmOHUJJe
To wrap things up, take a look at the deployed application. It leverages Vault dynamic database credentials and the “Transit” secret engine to securely encrypt customer data. Use port 5000 to connect.
$ kubectl get svc k8s-transit-app
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
k8s-transit-app LoadBalancer 10.15.255.6 34.68.115.112 5000:32487/TCP 7m33sVault Native Integration
Vault’s HTTP API supports several easy-to-use third party programming libraries. Several are listed here. This introduction workflow is potentially more secure as it does not require a credential on a shared volume.
The following is an example from our demo.
Code is located here.
Enable this feature by setting the following variable to true in the application config. In this case, the application leverages the pod service account JWT to authenticate to Vault without the need for a sidecar.
More Examples:
https://www.hashicorp.com/blog/injecting-vault-secrets-into-kubernetes-pods-via-a-sidecar/
https://www.hashicorp.com/blog/dynamic-database-credentials-with-vault-and-kubernetes/
