Vault Integration with Kubernetes — Dynamic Kubernetes Credentials

This is the second blog of the series about Vault Integration with Kubernetes. For ease of navigation, here are links to all the blog posts in the series (will get updated as new posts go live):
1. Vault Integration with Kubernetes — Access Secrets
2. Vault Integration with Kubernetes — Dynamic Kubernetes Credentials
3. Vault Integration with Kubernetes — Unified Identity Management Service

As described ealier, the service account token in Kubernetes provides a distinct identity in the cluster and should also be protected properly.

The applications within the cluster typically attach the service account directly and use the token mounted inside the pod to communicate with the cluster when needed. The kubelet is responsible for token renewal, ensuring that a valid token is always accessible throughout the pod’s lifecycle. However, when it comes to external users or applications, maintaining static credentials like username/password or certificates is not secure. Setting up a remote auth webhook service doesn’t seem appealing either.

Kubernetes Secrets Engine in Vault helps generating dynamic credentials to access Kubernetes. When a client requests a credential, Vault will create a new service account in the Kubernetes cluster in the backend, and configure it with a specific role so it has the required access. The user will get the credentials returned from Vault in the shape of JWT, which can be used to authenticate directly to Kubernetes.

Vault has an internal timer that will track the lifetime of the credentials, and when the assigned time to live(TTL) has expired, it will automatically remove the service account from Kubernetes, which means the user can not use this anymore to access the cluster.

In order to communicate with Kubernetes, Vault itself (running inside the cluster) needs to be configured using a service account with appropriate permissions, e.g. managing the service account tokens, and optionally managing service accounts, roles, and role bindings.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: k8s-full-secrets-abilities
rules:
- apiGroups: [""]
  resources: ["serviceaccounts", "serviceaccounts/token"]
  verbs: ["create", "update", "delete"]
- apiGroups: ["rbac.authorization.k8s.io"]
  resources: ["rolebindings", "clusterrolebindings"]
  verbs: ["create", "update", "delete"]
- apiGroups: ["rbac.authorization.k8s.io"]
  resources: ["roles", "clusterroles"]
  verbs: ["bind", "escalate", "create", "update", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-token-creator-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: k8s-full-secrets-abilities
subjects:
- kind: ServiceAccount
  name: vault
  namespace: vault

Next, enable the Kubernetes secret backend and configure it accordingly. Since our Vault server is running as a pod within the Kubernetes cluster, no additional parameters are required for the configuration if the command is running inside the Vault pod.

vault secrets enable kubernetes
vault write -f kubernetes/config

In this example, as the Vault operator, we will create a role in Vault specifically for the Kubernetes secret backend. When an application requests a Kubernetes token using this role, a service account will be automatically created, and it will be granted permission to list Pods exclusively in the “argocd” namespace.

vault write kubernetes/roles/automated-sa-test \\
    allowed_kubernetes_namespaces="argocd" \\
    generated_role_rules='
{
  "rules": [
    {
      "apiGroups": [""],
      "resources": ["pods"],
      "verbs":["list"]
    }
  ]
}'

Now, with the appropriate Vault token, the application can request a Kubernetes token from the Kubernetes secret backend. A new service account is created in the “argocd” namespace, and the service_account_token returned can be used by the application to authenticate and communicate with the Kubernetes cluster.

$ vault write kubernetes/creds/automated-sa-test kubernetes_namespace=argocd
Key                          Value
---                          -----
lease_id                     kubernetes/creds/automated-sa-test/kxweWvnY6FH4ZDfGRqWVlxLV
lease_duration               24h
lease_renewable              false
service_account_name         v-root-automate-1686309068-tstyva3h6g8ihwokpg7msill
service_account_namespace    argocd
service_account_token        xxxxxx

This role-based configuration ensures that the generated one-off service account has limited access and can only perform the necessary actions within the designated namespace. By fine-tuning the permissions in this manner, we enhance security and maintain control over the application’s interactions with Kubernetes resources.