Vault : Inject secret in application with agent injector

HashiCorp Vault is a powerful open-source tool, that provides a secure and reliable way to store and tightly control access to sensitive information like secrets, tokens, passwords, certificates, and encryption keys. It allows for secure secret management, data encryption as a service, and privileged access management.

Vault’s primary features include:

1. Secure Secret Storage: Arbitrary key/value secrets can be stored in an encrypted form to Vault, which makes it a secure replacement for plaintext credential storage.
2. Dynamic Secrets: Vault can generate secrets on-demand for systems like AWS, SQL databases, or SSH access. This means that clients can access a service without knowing an existing password or key.
3. Data Encryption: Vault can encrypt and decrypt data without storing it. This enables security teams to define encryption parameters and developers to store encrypted data in a location they control.
4. Revocation: Vault provides mechanisms for revoking both dynamic and static secrets.
5. Lease and Renewal: All secrets in Vault have a lease associated with them. When a lease is expired, Vault will automatically revoke that secret.

With its robust access control mechanisms and focus on traceability, Vault helps organizations meet their security and compliance requirements.

Let’s start to secure your sensitives values in Hashicorp Vault to inject them in your application at startup !

At the end of this article your secret securing mechanism will look like this

Vault secret injection architecture

If your cluster is already setup you can go to step 2 and skip the beginning.

First of all we need to be connected to a cluster. Personally, I would choose a cluster runned by my docker daemon, of course its only for development and testing stage. In this tutorial, I will use K3D and show how to setup a cluster within local daemon docker. In any case, choose the environment with which you feel most comfortable.

1. Setup a K3D cluster.

If you don’t already have the package, you need to install it with this tutorial

CLUSTER_NAME=vault-agent-secret-injection
NB_NODE=1
REGISTRY_PORT=5000
REGISTRY_NAME=vault-agent-secret-injection.localhost

k3d registry create -p 0.0.0.0:$REGISTRY_PORT $REGISTRY_NAME --no-help;

k3d cluster create $CLUSTER_NAME \
    -a $NB_NODES \
    -p '80:80@loadbalancer' \
    -p '443:443@loadbalancer' \
    --k3s-arg '--disable=traefik@server:0' \
    --registry-use k3d-$REGISTRY_NAME:$REGISTRY_PORT;

kubectl config use-context k3d-$CLUSTER_NAME;

2. Install Hashicorp Vault helm chart.

Update the STORAGE_CLASS variable with the storageclass name that you have in your cluster

STORAGE_CLASS=local-path

# Vault installation
helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault \
    -n vault \
    --set server.dataStorage.storageClass=$STORAGE_CLASS \
    --set server.auditStorage.storageClass=$STORAGE_CLASS \
    --create-namespace;


# Wait for Vault in running state
while [[ $(kubectl get pods vault-0 -n vault -o 'jsonpath={..status.phase}') != "Running" ]]
do 
    echo "INFO: Wait for vault get running..." && sleep 5
done


# Init Vault / get credentials
kubectl exec vault-0 -n vault -- vault operator init -key-shares=1 -key-threshold=1 -format=json > vault-keys.json

3. Unseal Vault, enable Key / Value and Kubernetes engine

The last command of the previous section creates a new file named vault-keys.json. If you don’t have it you cannot continue, I suggest you to uninstall the chart and try again the previous section.

# Get Vault credentials
VAULT_UNSEAL_KEY=$(cat vault-keys.json | jq -r ".unseal_keys_b64[]")
VAULT_ROOT_KEY=$(cat vault-keys.json | jq -r ".root_token")

# Unseal Vault
kubectl exec vault-0 -n vault -- vault operator unseal $VAULT_UNSEAL_KEY

# Enable Key Value engine
kubectl exec vault-0 -n vault -- /bin/sh -c "VAULT_TOKEN=$VAULT_ROOT_KEY vault secrets enable -version=2 kv"

# Enable Kubernetes engine
kubectl exec vault-0 -n vault -- /bin/sh -c "VAULT_TOKEN=$VAULT_ROOT_KEY vault auth enable kubernetes"

4. Create single secret named APP_SECRET

kubectl exec vault-0 -n vault -- /bin/sh -c "VAULT_TOKEN=$VAULT_ROOT_KEY vault kv put kv/api APP_SECRET=my_secure_secret"

5. Create a policy that will allow to read secret at specific path in Vault KV engine.

You can set the path of your choice but you have to be consistent with the definition of your Kubernetes manifest, we will see in a few steps.
Note that I named my policy “app”

kubectl exec vault-0 -n vault -- /bin/sh -c "VAULT_TOKEN=$VAULT_ROOT_KEY vault policy write app -<<EOF
path \"kv/data/api*\" {
  capabilities = [\"read\"]
}
EOF"

This policy specify rule for read only access to the secrets stored in /kv/api in Vault. If you have any doubt of the path you can port forward Vault and access to the web console and check the location of path in KV engine. Let’s see how to do this

kubectl port-forward -n vault pod/vault-0 8200:8200

Now, you can request http://localhost:8200 and see the login form of Vault.

Vault login form

Put your root_token in the previously generated file vault-keys.json and go to your secret, you will see the path at the top left

secrets in kv engine

6. Once your policy is set, you have to configure authentication between Kubernetes and Vault.

This step requires to create configuration for Kubernetes engine with appropriate certificates and attach a role to authentication system.

I named my serviceaccount which will be used by my application “api-serviceaccount” , this application is in “api” namespace and my policy is named “app”. You can change it to suits your needs.

TOKEN_REVIEWER_JWT_COMMAND=$(kubectl exec vault-0 -n vault -- /bin/sh -c "cat /var/run/secrets/kubernetes.io/serviceaccount/token")

KUBERNETES_PORT_443_TCP_ADDR=$(kubectl exec vault-0 -n vault -- /bin/sh -c "echo \$KUBERNETES_PORT_443_TCP_ADDR")

kubectl exec vault-0 -n vault -- /bin/sh -c "VAULT_TOKEN=$VAULT_ROOT_KEY vault write auth/kubernetes/config \
   token_reviewer_jwt=$TOKEN_REVIEWER_JWT_COMMAND \
   kubernetes_host=https://$KUBERNETES_PORT_443_TCP_ADDR:443 \
   kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"

kubectl exec vault-0 -n vault -- /bin/sh -c "VAULT_TOKEN=$VAULT_ROOT_KEY vault write auth/kubernetes/role/api \
   bound_service_account_names=api-serviceaccount \
   bound_service_account_namespaces=api \
   policies=app \
   ttl=1h"

7. Let’s deploy our application !

This is an example, feel free to get only annotation and the launch command overwritten

First of all, you need to create a serviceaccount that will be attached to our pod and recognized by Vault

apiVersion: v1
kind: ServiceAccount
metadata:
  name: api-serviceaccount
  namespace: api

Vault Agent Injector will check every annotation.

annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/agent-inject-status: "update"
  vault.hashicorp.com/agent-inject-secret-app: "kv/data/api"
  vault.hashicorp.com/agent-inject-template-app: |
    {{`{{ with secret "kv/data/api" }}
    {{ range $k, $v := .Data.data }}
      export {{ $k }}={{ $v }}
    {{ end }}
    {{ end }}`}}
  vault.hashicorp.com/role: 'api'

agent-inject : says to Vault Agent Injector that it must start vault sidecar within our application pod
agent-inject-status : is used to update the secret in the application when it is changed in Vault
agent-inject-secret-app : defines the path in Vault to find the secret
agent-inject-template-app : will create file that contains, in my case “export APP_SECRET=my_secure_secret”. We will source this file at application launch
role : the Vault role that we create before for Kubernetes authentication

We need to attach our service account to the pod

serviceAccountName: api-serviceaccount

Now, we can overwrite the start command for our application

command: ["/bin/sh"]
args: ["-c", ". /vault/secrets/app && ./app"]

You can find bellow the whole manifest of the deployment that I created for this article

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-deployment
  namespace: api
  labels:
    app: api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-status: "update"
        vault.hashicorp.com/agent-inject-secret-app: "kv/data/api"
        vault.hashicorp.com/agent-inject-template-app: |
          {{`{{ with secret "kv/data/api" }}
          {{ range $k, $v := .Data.data }}
            export {{ $k }}={{ $v }}
          {{ end }}
          {{ end }}`}}
        vault.hashicorp.com/role: 'api'
    spec:
      serviceAccountName: api-serviceaccount
      restartPolicy: Always
      containers:
      - name: api
        image: verovec/golang-api-template:latest
        command: ["/bin/sh"]
        args: ["-c", ". /vault/secrets/app && ./app"]
        imagePullPolicy: Always
        resources:
          limits:
            cpu: "1"
            memory: "1Gi"
          requests:
            cpu: "0.1"
            memory: "100Mi"
        ports:
        - containerPort: 8080
        env:
        - name: APP_ENV
          value: dev
        livenessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 30
          timeoutSeconds: 10
          periodSeconds: 10
          failureThreshold: 10

Congrats ! Vault Agent injector is now working as expected. If you follow my configuration and deploy my container image, after port-forward the pod you can request http://localhost:8080 and get this output

{
    "APP_ENV_SECRET":"my_secure_secret",
    "APP_ENV_VALUE":"dev",
    "HOME":"/home/nonroot",
    "HOSTNAME":"api-deployment-78c847444f-5wmgl",
    "OS_RELEASE":"NAME=\"Alpine Linux\"\nID=alpine\nVERSION_ID=3.18.2\nPRETTY_NAME=\"Alpine Linux v3.18\"\nHOME_URL=\"https://alpinelinux.org/\"\nBUG_REPORT_URL=\"https://gitlab.alpinelinux.org/alpine/aports/-/issues\"\n"
}

Everything about this article can be found in this repository. If you enjoy it don’t forget to clap it and put star on Github

GitHub - verovec/vault-agent-secret-injection: This repository shows you how to inject secret in…