Vault: setting up Kubernetes auth and database secrets engine

Implementation details for authenticating services to Vault to retrieve dynamic secrets/credentials.

Source: https://medium.com/@gmaliar/dynamic-secrets-on-kubernetes-pods-using-vault-35d9094d169

Introduction

Take a simple application that needs to connect to a database. The application just needs to know the host, username, and password. Cool, easy enough. Let’s secure the hell out of it.

  • Vault instance in its own Kubernetes cluster.
  • CloudSQL instance which is accessible by both clusters.
  • Reduce the risk when the database user and password are compromised.

Prerequisites

Before we get into the good stuff, I do assume a few things are in place.

  • Kubernetes cluster.
  • MySQL instance (I’m using CloudSQL).
  • VAULT_CACERT — Vault’s CA cert path.

Setup

In my mind, there are two main workflows for setting up. One is for setting a new cluster to communicate with Vault, primarily focused on the authentication method (this happens per cluster). The second is for onboarding a new application with a new database connection (this happens per application).

A New Cluster

Kubernetes

kubectl create serviceaccount vault-auth
kubectl apply -f -<<EOH
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: role-tokenreview-binding
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: vault-auth
namespace: default
EOH
configure/kubernetes.sh --context apps-cluster --setup
vault secrets enable database
configure/vault-secrets-database.sh --enable
k8s_host="$(kubectl config view --minify | grep server | cut -f 2- -d ":" | tr -d " ")"
k8s_cacert="$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 --decode)"
secret_name="$(kubectl get serviceaccount vault-auth -o go-template='{{ (index .secrets 0).name }}')"tr_account_token="$(kubectl get secret ${secret_name} -o go-template='{{ .data.token }}' | base64 --decode)"
vault auth enable kubernetes
vault write auth/kubernetes/config token_reviewer_jwt="${token_reviewer_jwt}" kubernetes_host="${kubernetes_host}" kubernetes_ca_cert="${kubernetes_cacert}"
configure/vault-auth-kubernetes.sh --token-reviewer-jwt $(configure/kubernetes.sh --token-reviewer-jwt) --k8s-host $(configure/kubernetes.sh --k8s-host) --k8s-cacert-base64 $(configure/kubernetes.sh --k8s-cacert) --enable --configure

New App

Kubernetes

kubectl create namespace demo
kubectl --namespace=demo create serviceaccount vault
kubectl --namespace=demo create configmap vault --from-literal "vault_addr=https://vault.local:8200"
kubectl --namespace=demo create secret generic vault-tls --from-file "${VAULT_CACERT}"
configure/kubernetes.sh --context apps-cluster --vault-addr https://vault.local:8200 --vault-cacert $VAULT_CACERT --new-namespace demo
vault write database/config/demo-db plugin_name=mysql-database-plugin connection_url="{{username}}:{{password}}@tcp(demodb.local)/" allowed_roles="demo-role" username="root" password="password"
vault write -f database/rotate-root/demo-db
vault write database/roles/demo-role db_name=demo-db creation_statements="CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}';GRANT SELECT ON *.* TO '{{name}}'@'%';" default_ttl="1h" max_ttl="24h"
configure/vault-secrets-database.sh --db-name demo-db --host demodb.local:3306 --username root --password password --role-name demo-role --configure --rotate-root --role
vault policy write demo-policy -<<EOF
path "database/creds/demo-role" {
capabilities = ["read"]
}
EOF
configure/vault-general.sh --policy demo-db-r
path "database/creds/{{identity.entity.aliases.auth_kubernetes_3626dffe.metadata.service_account_namespace}}-role" {
capabilities = ["read"]
}
vault write auth/kubernetes/role/demo bound_service_account_names=vault bound_service_account_namespaces=demo policies=demo-policy ttl=1h
configure/vault-auth-kubernetes.sh --names vault --namespaces demo --policies demo-policy --role demo

Testing and Workflow

All of those moving parts may have been confusing, connecting the dots may take a while. Don’t worry, I got you! I’ll explain the important stuff, you can refer to Kubernetes or Vault docs for the others.

kubectl -n demo run -it --rm --image=alpine --serviceaccount=vault test -- /bin/sh
apk add --update vim curl bash jq mysql-client
bash
JWT="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"
TOKEN="$(curl --request POST --data '{"jwt": "'"$JWT"'", "role": "demo"}' -s -k https://vault.local:8200/v1/auth/kubernetes/login | jq -r '.auth.client_token')"
  • demo role is the kubernetes auth role (auth/kubernetes/role/demo)
  • The demo kubernetes role was attached to the demo-policy policy
  • The demo-policy policy has read access to database/creds/demo-role database role.
  • The output is our client token. Think of this as your session cookie.
curl --header "X-Vault-Token: $TOKEN" -s -k  https://vault.local:8200/v1/database/creds/demo-role | jq -r .data
mysql -u$USER -p$PASS -h demodb.local

What’s next?

This is cool and all but how do you apply this into a production-ready environment? I get it, I need my application to implement this workflow in a simple automated manner. Check this out:

Conclusion

Yayyy!!! You did it!!!

Currently a Senior DevOps engineer. Previously a Senior Software developer. https://www.linkedin.com/in/jack-lei-0819832b

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store