Vault : Externals secrets in Kubernetes cluster

In a previous story I explained how to inject secret with Vault agent injector at application startup. In this article we will se how to use Hashicorp Vault in a different way.

As you know Kubernetes has its own secret manger. Unfortunately, sensitives values stored in it are only base 64 encoded. Even tho each Kubernetes resources can be RBAC’d, some manifest cannot be versioned on Git.

Vault is a powerful secret management tool that provides a secure and reliable way to store and manage sensitive information such as API keys, passwords, and tokens. Using Vault with Kubernetes offers several advantages :

• Centralized Secret Management: Vault provides a centralized platform to manage secrets. This centralization makes it easier to manage and rotate secrets, control access, and audit usage across multiple applications and environments.
• Dynamic Secrets: Vault can generate secrets on-demand for some systems (like databases, AWS, etc.). This means that instead of having long-lived secrets that can be leaked, Vault provides short-lived secrets that are generated when needed and can be automatically rotated.
• Fine-grained Access Control: Vault supports fine-grained access controls specifying who can access what. Policies can be defined to restrict which secrets a particular app or user can access.
• Data Encryption: Vault can encrypt data in transit and at rest. This ensures that secrets are stored securely and only transmitted to authorized entities in a protected manner.
• Audit Logging: Vault maintains detailed logs about access to secrets. This provides transparency about who accessed what and when, aiding in regulatory compliance and security investigations.
• Integration with Kubernetes: Vault has a Kubernetes authentication method that allows applications running inside Kubernetes to authenticate with Vault using Kubernetes service account tokens. This makes it seamless for Kubernetes workloads to retrieve secrets from Vault. My previous article is helpful for this point.
• Secret Versioning: Vault supports versioning of secrets. This allows you to retain a history of secret values which can be invaluable in rollback scenarios or in case of accidental deletions.
• Sealing & Unsealing: Vault has a mechanism to seal itself, which makes its stored data inaccessible. This can be particularly useful during a detected breach. Unsealing requires a specific number of keys (from a defined set of keyholders), ensuring that a single actor can’t unseal the vault on their own. This point will be more explained later.
• High Availability & Scalability: Vault can be set up in a highly available configuration, ensuring that your secrets are always accessible when needed. It’s built to handle the high-demand and high-scale infrastructure.

With its robust access control mechanisms and focus on traceability, Vault helps organizations meet their security and compliance requirements.

In this article we will see how to store your sensitive secrets in self hosted Vault and share it in Kubernetes cluster.

The following diagram shows what this implementation will look like

diagram of external secrets shared between Vault and Kubernetes cluster

First of all you need to install Vault helm chart

helm repo add hashicorp https://helm.releases.hashicorp.com

helm install vault hashicorp/vault -n vault --create-namespace

And external-secrets helm chart

helm repo add external-secrets https://charts.external-secrets.io

helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace

Once it’s done you have to initialize Vault and store credentials in a file. I called it vault-keys.json

kubectl exec vault-0 -n vault -- vault operator init -key-shares=1 -key-threshold=1 -format=json > vault-keys.json

To interact with Vault we need to unseal it.

Sealing / unsealing mechanism

The “sealing” and “unsealing” processes in Vault are mechanisms for ensuring that sensitive data remains protected, even if the underlying storage is compromised.

When Vault is sealed, it is in a state where it cannot access its storage backend, which means it cannot respond to requests to read/write secrets. This is a defense mechanism: in the event of a security breach, an intruder cannot simply access the Vault storage backend to get all the secrets because the Vault will not release them.

To “unseal” Vault means to make it operational so it can handle requests. Vault’s data is encrypted with a master key, but this key isn’t stored anywhere. Instead, when you initialize a Vault instance, the master key is split into multiple shards using the Shamir’s Secret Sharing algorithm. To unseal Vault, a certain number of these shards must be provided.

Once Vault is unsealed, you can use the root token (or other authentication mechanisms) to interact with it and manage secrets.

Lastly, remember to backup your unseal keys and root token securely. Losing access to them might mean you can’t access your Vault’s stored secrets.

To unseal Vault, first of all you have to get the unseal key which is in the vault-keys.json

VAULT_UNSEAL_KEY=$(cat vault-keys.json | jq -r ".unseal_keys_b64[]")

Once it’s done, past this command to unseal Vault

kubectl exec vault-0 -n vault -- vault operator unseal $VAULT_UNSEAL_KEY

Now Vault is unsealed, let’s see how to store a secret in it

Key / Value engine

The secret will be stored with the key / value engine. Here is some benefits about using it

• It allows you to keep multiple versions of a secret. This is particularly useful to track changes and can help in scenarios where a rollback to a previous version of a secret is necessary.
• Every time you write to a key, it creates a new version of that secret.
• The KV secrets engine supports the use of templates for path names, allowing you to map certain paths to specific actions.
• Vault ensures that all data stored in its backend (regardless of whether it’s the integrated storage or an external one like Consul) is encrypted.
• This engine doesn’t specify how the data is stored. You could use in-memory storage, file systems, or distributed systems. The main concern is to ensure the backend storage is secure, resilient, and meets your operational needs.

To write secret in the KV engine, first of all we need to enable it by running these commands

# Get the root key in vault-keys.json
VAULT_ROOT_KEY=$(cat vault-keys.json | jq -r ".root_token")

# Enable KV engine
kubectl exec vault-0 -n vault -- /bin/sh -c "VAULT_TOKEN=$VAULT_ROOT_KEY vault secrets enable -version=2 kv"

Then, we can write the secret at specific path to retrieve it later. The following command write the secret at kv/example, feel free to update it depending on your needs

kubectl exec vault-0 -n vault -- /bin/sh -c "VAULT_TOKEN=$VAULT_ROOT_KEY vault kv put kv/example secret-exemple=my_value"

If you go on Vault’s dashboard you will see, at the correct path, your newly created secret

That’s all for Vault configuration. Now we can focus on the external secret creation

⚠️ Every following Kubernetes manifest should be RBAC’d but it’s not the subject of this topic. You can find how to do it here : https://kubernetes.io/docs/reference/access-authn-authz/rbac/

External Secrets

The “external-secrets” Helm chart is used to deploy the external-secrets Kubernetes controller, which allows you to manage sensitive information (like API keys, tokens, or database passwords) in external secret management systems, such as AWS Secrets Manager, HashiCorp Vault, or Google Cloud Secret Manager. This enables a Kubernetes-native way to fetch and create Kubernetes secrets from these external sources.

The primary goal is to keep secrets outside of the Kubernetes cluster (avoiding etcd) while still making them available to applications running within the cluster.

Advantages:

• Security: Secrets are not stored in the cluster’s etcd.
• Central Management: Manage secrets across multiple clusters from a single source.
• Auditing & Rotation: Leverage native tools of the secret backend (like AWS Secrets Manager) to audit access or rotate secrets.

Usage:

• Once installed, create ExternalSecret that map to the desired secret in your chosen backend.
• The controller will then fetch the external secret and create a corresponding Kubernetes Secret in the specified namespace.

Always ensure that the controller's permissions are appropriately restricted to prevent unauthorized access.

External-secrets needs to know credentials to interact with Vault.

# Get the root key in vault-keys.json
VAULT_ROOT_KEY=$(cat vault-keys.json | jq -r ".root_token")

# Write Kubernetes secret which includes Vault credentials
kubectl create secret generic vault-token -n default --from-literal=token=$VAULT_ROOT_KEY

Now we can create SecretStore which is a CRD that includes the Vault configuration. It needs the correct provider, it’s host, the engine and where to find credentials to interact with it.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: default
spec:
  provider:
    vault:
      server: "http://vault.vault:8200"
      path: "kv"
      version: "v2"
      auth:
        tokenSecretRef:
          name: "vault-token"
          key: "token"

Once it’s done, let’s create our external secret. This manifest set where to find the chosen secrets and how to store it as Kubernetes secret. You can also define extra fields such as refreshInterval

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-secret
  namespace: default
spec:
  refreshInterval: "15s"
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: secret-exemple
    creationPolicy: Owner
  data:
    - secretKey: secret-exemple
      remoteRef:
        key: kv/example
        property: secret-exemple

When you apply those manifest with these commands

kubectl apply -f secret-store.yaml
kubectl apply -f external-secrets.yaml

Your Kubernetes secret will be generated. Repeat the process whenever you want to add a secret to your cluster.

Note that the SecretStore should not be created again as long as you use the same engine.

You can find my corresponding repository to this article here :

GitHub - verovec/externals-secret-from-vault: This repository shows you how to create secret in a…

Don’t forget to make clap if you find this article useful or to add star to my repository, it encourages me to write articles and to know what interest you.

Thanks to all of my readers, I wish you a good experience with Hashicorp Vault secret manager.