Vault Integration with Kubernetes — Access Secrets
This is the first blog of the series about Vault Integration with Kubernetes. For ease of navigation, here are links to all the blog posts in the series (will get updated as new posts go live):
1. Vault Integration with Kubernetes — Access Secrets
2. Vault Integration with Kubernetes — Dynamic Kubernetes Credentials
3. Vault Integration with Kubernetes — Unified Identity Management Service
Introduction
In today’s world, security is paramount and must be a top priority for every organization. Kubernetes is a container orchestration tool that is widely used to manage containerized workloads. However, Kubernetes has some limitations when it comes to security, particularly in the management and protection of sensitive data within the cluster. One way to address this is by integrating Kubernetes with Vault, an open-source secrets management tool that provides secure storage for sensitive data.
What are Sensitive Data in Kubernetes
In a Kubernetes cluster, sensitive data refers to any information that, if compromised, could potentially lead to security breaches or unauthorized access. This includes secrets and service accounts, which are integral components of Kubernetes and contain sensitive information that needs to be protected.
Secrets in Kubernetes are used to store sensitive data such as passwords, API keys, and TLS certificates. They are crucial for applications to securely access external resources, databases, or other services. However, managing secrets in Kubernetes can be challenging due to the need for secure storage, encryption, proper access controls, and regular rotation of sensitive credentials.
Service accounts, on the other hand, are used by pods and applications (internal and external) to authenticate and interact with the Kubernetes API or other cluster resources. Ironically, by default, the service account token is stored as plain text in the Secret resource in Kubernetes. Especially, before Kubernetes v1.21, the service account tokens don’t expire and don’t rotate, which is not recommended, especially at scale, because of the risks associated with static, long-lived credentials.
All the above problems can be solved by integrating Vault in the cluster. By leveraging Vault’s capabilities, you can ensure that secrets and service accounts are not exposed directly in your Kubernetes configuration files or stored insecurely. Instead, Vault provides a secure and centralized location where you can securely store, retrieve, and manage sensitive data. This not only enhances the overall security posture of your Kubernetes applications but also simplifies the management and rotation of secrets.
Access Secrets in Vault from Kubernetes
With Vault, you can easily manage secrets across different environments, including development, staging, and production. Instead of managing secrets in multiple places or hardcoding them into your applications, you can store them in Vault and access them from your Kubernetes pods.
Additionally, using Vault as the secret backend helps organizations meet regulatory compliance requirements. Vault provides a secure way to store and manage sensitive data, ensuring that compliance requirements are met. With Vault, you can audit who has accessed secrets and when, ensuring that your organization is meeting regulatory requirements.
There are several ways for a kubernetes pod to access secrets in Vault:
Agent Sidecar Injector
Instead of writing and maintaining your own code in the container to manually retrieve secrets from Vault, using Vault Agent Sidecar Injector could automate the process of getting and injecting secrets from Vault into Kubernetes pods. The injector is running as a Kubernetes Mutation Webhook Controller and will alter the pod specification based on the annotations present in the manifest.
For example, suppose we have a secret (a github account credentials) in the Vault KV secret store:
$ vault read secret/test/github/myaccount
Key Value
--- ---
username testuser
password w72g$M3E)KxzjfdXdNAfter deploying the Vault Agent Sidecar Injector in the cluster, config the pod spec with the following annotations:
...
spec:
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: vault-role
vault.hashicorp.com/agent-inject-secret-username: secret/test/github/myaccount
vault.hashicorp.com/secret-volume-path-username: /vault/secrets/github-creds
vault.hashicorp.com/agent-inject-template-username: |
{{- with secret "vault-secret-path" -}}
{{ .Data.username }}
{{- end -}}
vault.hashicorp.com/agent-inject-secret-password: secret/test/github/myaccount
vault.hashicorp.com/secret-volume-path-password: /vault/secrets/github-creds
vault.hashicorp.com/agent-inject-template-password: |
{{- with secret "vault-secret-path" -}}
{{ .Data.password }}
{{- end -}}
...The vault.hashicorp.com/agent-inject: "true" annotation enables the secret injection using configured vault role, all the other annotations define the location and content of the files containing the secret data. You can find the details of each supported annotations on Vault official website.
After the pod is up and running, applications in the pod can directly access github using the files injected into the container.
# Inside the pod
$ ls /vault/secrets/github-creds
username password
$ cat /vault/secrets/github-creds/username
testuser
$ cat /vault/secrets/github-creds/password
w72g$M3E)KxzjfdXdNThere is one thing we need to take care of, by default, the Vault Agent Sidecar Injector itself needs authentication with Vault using the service account attached to the pod, which means, it’s still necessary to configure the Kubernetes auth backend in Vault. The service account must be bound to a Vault role and a policy granting access to the secrets desired.
The Agent Sidecar Injector is the most stable solution at the time of writing.
Vault CSI Provider
The Vault CSI provider allows pods to consume Vault secrets by using ephemeral CSI Secrets Store volumes. It is just one of the providers that implement Secrets Store CSI Driver interface.
Similar to using Agent Sidecar Injector, using the CSI Provider also implifies retrieving secrets stored in Vault and expose them to the target pod running on Kubernetes without it being aware of the Vault interaction processes.
After creating a SecretProviderClass resource which defines how to interact with Vault, add a volume for the deployment and attach the volulme to the pod, an example:
template:
spec:
containers:
- name: app
image: my-app:1.0.0
volumeMounts:
- name: 'vault-db-creds'
mountPath: '/mnt/secrets-store' # The secret data can be accessed under this path
readOnly: true
volumes:
- name: vault-db-creds
csi:
driver: 'secrets-store.csi.k8s.io'
readOnly: true
volumeAttributes:
secretProviderClass: 'vault-db-creds' # The SecretProviderClass object nameIt should be noted that thare are some features that the CSI Provider doesn’t support, e.g. at the time of writing, CSI Provider doesn’t support automatically renews, rotates secrets/tokens, and doesn’t support secret templating like what we did in the Agent Sidecar Injector annotations. Those are the reasons that Vault CSI Provider is not as popular as Agent Sidecar Injector.
Vault Secrets Operator
Similar to other common Kubernetes operators, the Vault Secrets Operator provides its own CRDs and offers the application a native way to access secrets data in Vault. The main task of the Vault Secrets Operator is to synchronize secrets between Vault secret engine and Kubernetes secret objects, it also takes care of the secrets expiration and rotation, to make sure that any changes made on the Vault side are replicated to Kubernetes cluster over its lifetime.
For the applications that do not support dynamically reloading when the secret changes, Vault Secrets Operator goes one step further by automatically triggering a “rollout-restart” operation to the deployments/daemonsets/statefulset objects based on the RolloutRestartTarget settings in the SecretSpec.
At the time of writing, the Vault Secrets Operator is still in public beta and not considered production ready yet.
External Secrets Operator
Unlike all the above components that are maintained mainly by Hashicorp, External Secrets Operator is yet another open source Kubernetes operator maintained by the community with a broader supported secrets management services, including not only Vault, but also various cloud services such as AWS Secrets Manager, Azure Key Vault, GCP Secrets Manager, etc. and it could be easily extended to support any external secrets provider you want to use.
External Secrets Operator reads information from external APIs and automatically injects the values into a Kubernetes Secret. Similar to the Vault Secrets Operator, this is more native way for the pods to access secrets data in Vault, there are no additional annotations. External Secrets Operator also supports data templating like the Vault Agent Sidecar Injector does.
In the following, we will show a simple example of how the External Secrets Operator seamlessly integrates with AWS Secret Manager. To begin, we must first create a secret in the AWS environment.
Once the External Secrets Operator has been deployed in the Kubernetes cluster using Helm, the next step is to configure the integration with AWS by creating a secret containing our AWS credentials (should have permissions to read the secrets in AWS). This secret will serve as the secure access point for the External Secrets Operator, enabling it to interact with various AWS services.
kubectl create secret generic aws-creds \
--from-literal=access-key=${AWS_ACCESS_KEY_ID} \
--from-literal=secret-access-key=${AWS_SECRET_ACCESS_KEY}Then, we could create SecretStore and ExternalSecret resources:
cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: secretstore-aws
spec:
provider:
aws:
service: SecretsManager
region: ap-southeast-2
auth:
secretRef:
accessKeyIDSecretRef:
name: aws-creds
key: access-key
secretAccessKeySecretRef:
name: aws-creds
key: secret-access-key
EOF
cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: example
spec:
refreshInterval: 5m
secretStoreRef:
name: secretstore-aws
kind: SecretStore
target:
name: test-aws-secret
creationPolicy: Owner
dataFrom:
- extract:
key: lingxian-test-secret
EOFAfter the External Secrets Operator handles the ExternalSecret resource named “example” based on its definition, a new secret, specified by the target field within the ExternalSecret, is automatically generated by the operator.
$ kubectl get secret test-aws-secret -o "jsonpath={.data}" | jq -r 'map_values(@base64d)'
{
"another-test-key": "another-test-value",
"test-key": "test-value"
}Once we have the secret, we could continue with “Create a Pod that has access to the secret data through a Volume”. External Secrets Operator would also take care of the secrets update, rotation, etc.
External Secrets Operator is especially useful if you are using different secrets management services besides Vault in your organization. This flexibility enables you to leverage the benefits of different secret management services without being locked into a single provider, providing greater flexibility and adaptability to your infrastructure needs.
However, if Vault is already being utilized as a fundamental component for secrets management, it’s worth noting that it also offers a plugin mechanism that supports various providers, i.e. Vault supports secret management services in different cloud providers as its secret backends. The choice is yours to decide whether to opt for the External Secrets Operator or Vault as the primary point of contact for managing secret data.
At the time of writing, the Vault provider in External Secrets Operator is considered stable and is already used in many production environments.
