Manage Kubernetes Secret Using HashiCorp Vault

Published in

NonTechCompany

4 min readMay 27, 2021

Pain point in Managing Kubernetes Secret

IaC

Since secrets will not be stored in the remote repository, you would have to manually create Kubernetes secret by yourself. The problem would arise when you accidentally lost them or you would like to migrate your applications some where else. Thus, you need to recreate all the secrets again manually which is a tedious job.

Security

Kubernetes secret does not have the best security control. Mostly rely on the RBAC within the cluster.

Ease of usage

Imagine when you have to manage the lifecycle of Kubernetes secret by using either YAML file or using Kubernetes command line. Especially for the less technical people or somebody who is not familiar with Kubernetes will have a really hard time create/update/delete/view the credentials. Moreover, a lot of time a single credentials is used across multiple places. Whether multiple Kubernetes cluster or even in different platform. You don’t want to have multiple copy of the same credentials stored in the different places

HashiCorp Vault

HashiCorp Vault has a capability to securely store and centralize all the file/credentials/certificates and etc. In addition, it allow us to define the set of secrets which is consumed by pods in the Kubernetes Cluster and automatically inject or update them.

There are multiple method to consume secret from the Vault such as using sidecar or CSI driver. Today we are going to take a look at how to inject and consume secret from Vault by using sidecar.

The sidecar will be inject by the agent call agent injector. The secret will be retrieve from Vault and inject to pod by sidecar agent

Prerequisite

Hashicorp Vault Instance
Kubernetes Cluster

Deploy Vault Injector Agent

The Vault Helm chart comes with the Vault server and the agent injector. Thus, we have to install on vault agent injector and disable the installation of server statefulset. This agent injector will responsible to inject the vault sidecar to the pods.

helm install vault-agent hashicorp/vault \
--set "injector.externalVaultAddr=${VAULT_HOST}"

Token Reviewer Service Account

The Vault will watch and verify the JWT of service account by calling Kubernetes Token Review API. Therefore, it is necessary that we need to create a service account with sufficient permission which the Vault is going used to authenticate to the target cluster and call the Kubernetes API.

kubectl create sa vault-token-reviewer

Get JWT from service account

SECRET_NAME=$(kubectl get serviceaccount/vault-agent -o jsonpath='{.secrets[0].name}')TOKEN=$(kubectl get secret $SECRET_NAME -o jsonpath='{.data.token}'| base64 --decode)

Get Kubernetes API Host

KUBE_HOST=$(kubectl config view --raw --minify --flattern --output='jsonpath={.clusters[].cluster.server}')

Get Cluster CA Certificate

KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)

Finally Configure the Vault to review token on the cluster

vault write -address="$VAULT_ADDRESS" auth/kubernetes/config \ token_reviwer_jwt="$TOKEN" \
kubernetes_host="$KUBE_HIST" \
kubernetes_ca_cert="$KUBE_CA_CERT"

Configure RBAC on Hashicorp Vault Secret

In order to control the access to a specific set of secret on the Vault. We have to create the policy and role for a particular secret path. For example, we are going to inject the secret on the path “secret/data”.

Install vault-cli

brew install vault

vault login -address=${VAULT_HOST}

Create policy for accessing the secret

vault policy write -address=${VAULT_HOST} secret-read - <<EOF
path "secret/data/my-secret" {
   capabilities=["read]
}

Create role my-secret-role and attach the above policy

vault write -address=${VAULT_HOST} \
auth/kubernetes/role/my-secret-role \
bound_service_account_names=${POD_SERVICE_ACCOUNT_NAME} \
bound_service_account_namespaces=${POD_NAMESPACE} \
policies=secret-read \
ttl=24h

Inject Secret to Pod

Enable new secret engine data in Vault

vault secrets enable -version=2 -address=${VAULT_HOST} -path data kv

Create new secret my-secret under data engine and add username and password as secret data

vault kv put -address=${VAULT_HOST} data/my-secret username=foobaruser password=foobarbazpass

To inject secret to the pod we can simply add the pod annotations to the deployment object

Go to path /vault/secrets inside the pod which is the default path in which the secrets will be injected. You should see the file data-my-secret similar to this

cat /vault/secrets/data-mysecretdata: map[password:foobarbazpass username:foobaruser]
metadata: map[created_time:2021-05-27T08:36:45.575829748Z deletion_time: destroyed:false version:1]

Inject Template

From the above example you can see that the default content of the secret inject is not really usable. Usually you want to format them first before it can be used. For example, if my application need to consume the file content like

username=foobaruser
password=foobarbazpass

We can achieve this by using inject template annotations with go template to make sure that the agent format the file before inject to the pod

Inject Secret as Environment Variable

Sometime the secret need to be injected as an environment for application to consume. We can modify the inject template to contain export command and run source command when container start

Then add the command argument to the deployment

Advance Configuration

For more advance annotations please visit the official site https://www.vaultproject.io/docs/platform/k8s/injector/annotations

Contact

Email: chan.suttichujit@gmail.com

GitHub: https://github.com/NonTechCompany