Using Vault Secrets Operator in Kubernetes

Proper secrets handling in Kubernetes clusters is crucial for infrastructure security. The default Kubernetes deployment provides only very basic secrets management features. It doesn’t provide automated secrets rotation and a comprehensive audit. According to official K8s docs, Kubernetes secrets are not even encrypted by default!

Therefore, using special secrets management solutions is becoming a standard in enterprise environments. HashiCorp Vault is one of the most popular solutions and is being widely used nowadays. Using Vault properly and keeping the developers user experience smooth requires additional efforts to integrate Kubernetes with Vault. Fortunately, there are many options of achieving this: CSI provider, sidecar injector and 3-rd party operator like External Secrets Operator (ESO). The latter is the most comprehensive option. It is being actively developed and is well supported by the open source community. But if you want to have the integration provided by a single vendor, you should consider the new HashiCorp Vault Secrets Operator (VSO). It could be a good alternative as a native solution from the “first hand”. The official GA was announced in June 2023 and. The difference between all types of integrations from HashiCorp can be found on this page.

This article has the step-by-step guide on how to configure VSO and includes a demonstration how it works. For sake of simplicity this demo-case is not hardened as per the best security practices (e.g. skipped TLS, default namespaces and account) and aims only to demonstrate how Vault secrets can be consumed from Kubernetes.

Prerequisites

To perform this demo we need a working Kubernetes cluster with configured access via kubectl/helm, a working Vault service with privileged access via CLI.

The tested configuration below has these versions of components:

• k3s v1.26.4
• helm v3.12.3
• vault v1.15.0
• vault-secrets-operator v0.3.2

Detailed preparation steps could be also found on the vendors pages: K3s, Helm, Vault or HCP Vault.

Step 1

The first step will be to configure the target secrets in Vault, which we want to access. We will use the standard kv v2 engine. According to HashiCorp all types of secrets engines are supported by VSO.

We need to enable the engine and put a record into it:

➜  ~ vault secrets enable -path=kvv2 kv-v2
Success! Enabled the kv-v2 secrets engine at: kvv2/
➜  ~
➜  ~ vault kv put kvv2/webapp username="web-user" password=":pa55word:"
== Secret Path ==
kvv2/data/webapp

======= Metadata =======
Key                Value
---                -----
created_time       2023-10-15T17:56:15.85914277Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Also we need to create a policy, which we will use in the later steps:

➜  ~ vault policy write webapp-ro - <<EOF
path "kvv2/data/webapp" {
   capabilities = ["read"]
}
path "kvv2/metadata/webapp" {
   capabilities = ["read"]
}
EOF
Success! Uploaded policy: webapp-ro

Step 2

Now we configure the authentication between Vault and Kubernetes. VSO supports a few options for authentication, but we will use the native Kubernetes auth engine.

Put this snippet in the file sa.yaml:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-auth
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: default

And apply:

➜  ~ kubectl apply -f sa.yaml
serviceaccount/vault-auth created
secret/vault-auth created
clusterrolebinding.rbac.authorization.k8s.io/role-tokenreview-binding created

The manifest creates a dedicated service account with a secret and the right binding.

Now we need to extract some credentials into environment variables:

TOKEN_REVIEW_JWT=$(kubectl get secret vault-auth --output='go-template={{ .data.token }}' | base64 --decode) 
KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode) 
KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')

And use them to enable and configure the auth method:

➜  ~ vault auth enable -path=vso kubernetes
Success! Enabled kubernetes auth method at: vso/

Write the mount point config:

➜  ~ vault write auth/vso/config \
      token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
      kubernetes_host="$KUBE_HOST" \
      kubernetes_ca_cert="$KUBE_CA_CERT" \
      disable_issuer_verification=true
Success! Data written to: auth/vso/config

Create the role, with the Vault policy from Step 1 assigned:

➜  ~ vault write auth/vso/role/vso-role \
      bound_service_account_names=default \
      bound_service_account_namespaces=default \
      policies=webapp-ro \
      ttl=24h
Success! Data written to: auth/vso/role/vso-role

You can verify the created role with commands:

vault list auth/vso/role
vault read auth/vso/role/vso-role

Step 3

Install and configure the operator:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault-secrets-operator hashicorp/vault-secrets-operator

The chart creates a few CRDs, which we need to configure. Applying the yaml below, we will get 3 instances of CRDs: connection, authentication and the static secret itself. Don’t forget to provide the address to your Vault service.

---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  namespace: default
  name: vault-connection
spec:
  # address to the Vault server.
  address: http://128.0.0.1:8200
  skipTLSVerify: true
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vault-auth
spec:
  vaultConnectionRef: vault-connection
  method: kubernetes
  mount: vso
  kubernetes:
    role: vso-role
    serviceAccount: default
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vault-static-secret
spec:
  vaultAuthRef: vault-auth
  mount: kvv2
  type: kv-v2
  path:  webapp
  refreshAfter: 10s
  destination:
    create: true
    name: vso-handled

➜  ~ kubectl apply -f crd.yaml
vaultconnection.secrets.hashicorp.com/vault-connection created
vaultauth.secrets.hashicorp.com/vault-auth created
vaultstaticsecret.secrets.hashicorp.com/vault-static-secret created

The VaultStaticSecret instance maps the kv secrets from Vault to vso-handled secret in the default Kubernetes namespace. The beauty of this solution is that apps can work with the secrets as if they were normal Kubernetes secrets. The secrets can be updated or rotated in Vault and after the defined “refreshAfter” (10 sec in this example) it will be reflected in the cluster.

Step 4

Verification.

Read the secret:

➜  ~ kubectl get secret vso-handled -o jsonpath='{.data._raw}' | base64 --decode
{"data":{"password":":pa55word:","username":"web-user"},"metadata":{"created_time":"2023-10-15T17:56:15.85914277Z","custom_metadata":null,"deletion_time":"","destroyed":false,"version":1}}

Update the secret in Vault and check again after the predefined interval, it should be renewed, example:

➜  ~ kubectl get secret vso-handled -o jsonpath='{.data._raw}' | base64 --decode
{"data":{"password":":new-pa55word:","username":"web-user"},"metadata":{"created_time":"2023-10-15T18:14:19.656357706Z","custom_metadata":null,"deletion_time":"","destroyed":false,"version":2}}

Thanks to the operator, the complexity of secrets handling is hidden and developers can enjoy the first-class cloud-native security.