Secrets Management Using Vault in K8S
Secrets management is a very critical component of any application security. The secret doesn’t mean just passwords but also access keys, encryption keys, API token etc. In kubernetes, its built-in Secret object is used to store secret data, and the actual data is stored in etcd along with other Kubernetes objects. Placing sensitive info into a secret object does not automatically make it secure. By default, data in Kubernetes secrets is stored in base64 encoding(not encrypted!), which is practically the same as plaintext. Obviously, it won’t be wrong to say that kubernetes secrets and security cannot go hand-in-hand. To fill this gap, there are few good alternatives which provides better security and usability to secrets. One such common and popular open source solution is Hashicorp Vault
HashiCorp Vault
HashiCorp Vault is a secrets management tool to securely store and manage sensitive assets for applications deployed to kubernetes as well as demonstrating how both applications and vault can be configured to provide seamless integration with one another. By using a tool like vault, application teams can offload some of the responsibilities involved when managing sensitive resources to a purpose built tool, while still being able to integrate with their applications.
Vault comes with various pluggable components called secrets engines and authentication methods allowing you to integrate with external systems. The purpose of these components is to manage and protect your secrets in dynamic infrastructure (e.g. database credentials, passwords, API keys).
Secrets engines are the components which store, generate, or encrypt data. Secrets engines are enabled at a path in vault. When a request comes to vault, the router automatically routes anything with the route prefix to the secrets engine. In this way, each secrets engine defines its own paths and properties.
Authentication methods are the components in vault that perform identity validation of vault clients and responsible for assigning a set of policies to an authenticated client. The Kubernetes Auth Method simplified how the applications can interact with vault avoid the need to manage the additional tasks of managing tokens to access vault.
Kubernetes + Vault
The Vault server cluster can be deployed directly on kubernetes via helm charts (recommended). Besides vault stateful pod(s), vault agent injector also runs as a pod which leverages the Kubernetes mutating admission webhook to intercept pods that define specific annotations and inject a vault Agent container to manage these secrets. This can be used by applications running within Kubernetes as well as external to Kubernetes.
Let’s have a brief overview of HashiCorp’s Vault components:
- Vault Server is the core component which provides an API through which clients can interact with and manage the interaction between all the secrets engines, ACL enforcement, and secret lease revocation
- Vault Agent is a client daemon that helps authenticate to the vault server and perform token lifecycle management
- Agent injector is a kubernetes
Mutating Admission Webhook. This controller intercepts pod CREATE and UPDATE requests looking for the metadata annotationvault.hashicorp.com/agent-inject: truein the requests. When found, the controller will inject theinitcontainer and thesidecarcontainer. Theinitcontainer is to pre-populate our secret, and thesidecarcontainer is to keep that secret data in sync throughout our application's life cycle.
This model of running vault in kubernetes is beneficial because:
- Applications remain vault unaware as the secrets are stored on the file-system in their container.
- Existing deployments require no change; as annotations can be patched.
- Access to secrets can be enforced via kubernetes service accounts and namespaces
How it works?
A Service account token (JWT Token) from the pod is passed to the vault server. Vault server authenticate against kubernetes via TokenReview API to get the service account and namespace attached to the JWT token. The Kubernetes API server returns the account details and thereafter, the vault server validates if the service account is authorized to read secrets using the attached policies. After successful validation, vault server returns a token. In subsequent API call including the vault token is passed to retrieve the secrets.
Show Time!
Now, let’s see how to set up vault for the Kubernetes cluster and provision secrets for the applications
Install the vault helm chart
The recommended way to run Vault on Kubernetes is via the Helm chart.
$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories$ helm repo update
Hang tight while we grab the latest from your chart repositories......Successfully got an update from the "hashicorp" chart repositoryUpdate Complete. ⎈Happy Helming!⎈$ helm install vault hashicorp/vault
The vault pod and vault Agent Injector pod are deployed in the default namespace.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
vault-0 1/1 Running 0 18s
vault-agent-injector-5989b47887-x86dd 1/1 Running 0 19sBy default, the vault server will be in a sealed state. Vault knows where and how to access the storage but does not know how to decrypt any of the data from the storage. Unsealing is the process of obtaining the master key required to decrypt and read the data from storage. The unsealing is done using a series of three secret keys (i.e. threshold: 3). Unfortunately if the server is in sealed state, vault will not be able to decrypt the secrets, since it is in a sealed state.
$ vault operator init
Unseal Key 1: j/4cU59R5jDIuJ+ZvXFawjLVdkqdhCAQEqCKgAWvydrZ
Unseal Key 2: 8NNvdYd8ZsqbtTdA7dtz1fGTj4RGOXHUJ6qdJ09xRpGM
Unseal Key 3: RJe+V67bAzkQ1Fpbl9jzRCbu6a4ptYUjj40E1t0Kx77/
Unseal Key 4: MqoTsmKH4Y3mJmqM4Hvh44llkwbxKrt+rkytsu1fKypX
Unseal Key 5: 5XTaGsHGvnbrWokCjnW2+dWgLN/UZCyjvpiPFK2LQgqbInitial Root Token: s.ZXqf3LAEubr8G4FfAhyEdgLAVault initialized with 5 key shares and a key threshold of 3. Please securely distribute the key shares printed above. When the Vault is re-sealed, restarted, or stopped, you must supply at least 3 of these keys to unseal it before it can start servicing requests.Vault does not store the generated master key. Without at least 3 keys to reconstruct the master key, Vault will remain permanently sealed!It is possible to generate new unseal keys, provided you have a quorum of existing unseal keys shares. See "vault operator rekey" for more information.
To unseal the Vault, we need to run vault operator unseal and enter 3 of the 5 keys that are generated. Once done, check the status of the vault
$ vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 3
Threshold 3
Version 1.9.0
Storage Type inmem
Cluster Name vault-cluster-f951f34f
Cluster ID 49b08417-c56b-8fea-82cc-6ad5e62f54c2
HA Enabled falseThe value of Sealed is false that means vault has been unsealed and ready for use. These unseal steps are necessary whenever vault is started or restarted.
Note: For learning or development purpose, vault can be deployed in dev mode in which the server would be automatically initialized and unsealed.
$ helm install vault hashicorp/vault --set "server.dev.enabled=true"Set the secret in vault
Let’s create an example secret for a postgres database connection
$ vault secrets enable -path=demo kv-v2$ vault kv put demo/database/config username=postgres password=pass123
Key Value
--- -----
created_time 2022-01-16T10:01:53.54041676Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1
Since we want to restrict only the relevant application in the Kubernetes cluster to access the secret, we have to define a policy to achieve that, as follows:
vault policy write demo-policy - <<EOF
path "demo/data/database/config" {
capabilities = ["read"]
}
EOFNext, we want to associate this policy with allowed entities, such as a service account in Kubernetes.
Configure Kubernetes Authentication
As stated before, vault can leverage native authentication mechanism from kubernetes and then bind the secret access policy to the service account. This authentication method can be enabled by executing the following commands:
$ vault auth enable kubernetesCreate a kubernetes authentication role named demo/auth.
$ vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \ kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \ kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt$ vault write auth/kubernetes/role/demo-auth \
bound_service_account_names=demo-user \
bound_service_account_namespaces=default \
policies=demo-policy \
ttl=24h
The token_reviewer_jwt and kubernetes_ca_cert are mounted to the container by kubernetes when it is created. The environment variable KUBERNETES_PORT_443_TCP_ADDR is defined and references the internal network address of the Kubernetes host.
The role connects the kubernetes service account, demo-user, and namespace, default, with the vault policy, demo-policy. The tokens returned after authentication are valid for 24 hours.
Create the Service Account
Lets create a Kubernetes service account named demo-user in the default namespace as stated in the vault policy
$ kubectl create serviceaccount demo-userLaunch the application
For demo purpose, let deploy postgres as a statefulset and inject the secret.
# postgres.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
spec:
serviceName: psql-svc
replicas: 1
updateStrategy:
type: RollingUpdate
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "demo-auth"
vault.hashicorp.com/agent-inject-secret-pgpass: "demo/data/database/config"
vault.hashicorp.com/agent-inject-template-pgpass: |
{{- with secret "demo/data/database/config" -}}
localhost:5432:postgres:{{ .Data.data.username }}:{{ .Data.data.password }}
{{- end -}}
spec:
terminationGracePeriodSeconds: 10
serviceAccountName: demo-user
initContainers:
containers:
- name: postgres
image: postgres:10.1
imagePullPolicy: Always
ports:
- containerPort: 5432
env:
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
volumeMounts:
- name: postgredb
mountPath: /var/lib/postgresql/data
readOnly: false
volumeClaimTemplates:
- metadata:
name: postgredb
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: gp2
resources:
requests:
storage: 3Giagent-injectenables the Vault Agent Injector serviceroleis the Vault Kubernetes authentication roleagent-inject-secret-FILEPATHprefixes the path of the file,pgpasswritten to the/vault/secretsdirectory. The value is the path to the secret defined in vault.agent-inject-template-FILEPATHprefixes the file path. The value defines the vault agent template to apply to the secret's data.
Actually this is what happens, the preceding annotations (prefixed with vault.hashicorp.com) dictates vault-agent-injector pod through a predefined mutating webhook configuration to inject one init container named vault-agent-init and one sidecar container named vault-agent, as well as an emptyDir type volume named vault-secrets . Also, the vault-secrets volume is mounted in the init container, the sidecar container, and the app container with the /vault/secrets/ directory. The secret is stored in the vault-secrets volume.
$ kubectl apply -f postgres.yamlWait until the postgres-0 pod reports that it is Running and ready (2/2). Then verify that the secrets are correctly injected into postgres pod
$ kubectl exec -it postgres-0 -c postgres -- sh
# cat /vault/secrets/pgpass
localhost:5432:postgres:postgres:pass123Bingo!
Vault UI
Vault also features a web-based user interface (UI) that enables you to unseal, authenticate, manage policies and secrets engines. When we operate vault in development mode the UI is automatically enabled otherwise we have to enable it by setting ui configuration to true
To access vault UI, run below command
$ kubectl port-forward svc/vault 8200Launch http://localhost:8200 and login using different auth methods (eg: token)
$ vault token create
Key Value
--- -----
token s.kgYNe3OxsVHZDZqdKi8DZOFH
token_accessor DE8UahNlAc1aClGvfGhwc9UV
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]
In this blog, we learned how we can make application running on kubernetes decoupled from the secret store. It provides a spectrum of features to fulfil an organization’s end-to- end secrets management requirements (key management, encryption, rotation, PKI, storage, replication, revocation, logging, monitoring, audit, etc.).
Liked the blog? Please clap and follow.
