Vault Integration with Kubernetes — Unified Identity Management Service

12 min readJun 13, 2023

This is the final blog of the series about Vault Integration with Kubernetes. For ease of navigation, here are links to all the blog posts in the series:
1. Vault Integration with Kubernetes — Access Secrets
2. Vault Integration with Kubernetes — Dynamic Kubernetes Credentials
3. Vault Integration with Kubernetes — Unified Identity Management Service

Vault can be leveraged to authenticate the identity of users or applications against trusted sources of identity and then leverage that authentication to control access to data, systems, and secrets. Additionally, an authorized user or application can request a token that encapsulates identity information for their associated entity.

As Vault operator, we could config Vault to be a unified token issuer, so that different applications, either from the same cluster or different clusters, can communicate with each other, in a secure way.

Identity Entity and Entity Alias

Identity Entity in Vault makes it easier for teams and organizations to identify who someone is across multiple providers. Once the various authentication methods are associated with one entity, it is easier to apply policy to that entity rather than per authentication method.

In the diagram below, Alex is the primary account holder who has accounts in different cloud providers and systems. Vault can recognize Alex’s different credentials and identities as one, whether he was using LDAP, Azure, or Google Cloud credentials, Vault still sees it as Alex. For example, imagine that Alex is running an application in a Kubernetes cluster on Google Cloud that gets deployed from GitHub, connects to a database in Azure, and replicates to AWS. Alex can integrate his different cloud identities into an “Alex” entity and centrally create policies granting and restricting access to secrets for the different components to connect securely to one another, all within one workflow and API.

It is possible to create an entity in Vault and associate various entity aliases with it. Each entity alias corresponds to a specific authentication method.

Suppose we have enabled the Kubernetes and JWT auth (using Kubernetes builtin OIDC provider) methods. We have two applications, each using a different authentication method. Additionally, we have created a policy specifically for testing purposes.:

$ vault auth list
Path           Type          Accessor                    Description                Version
----           ----          --------                    -----------                -------
jwt/           jwt           auth_jwt_b46ca563           n/a                        n/a
kubernetes/    kubernetes    auth_kubernetes_7703b26e    kubernetes backend         n/a

$ vault policy read allow_secrets_readonly
path "secret/*" {
  capabilities = ["read", "list"]
}

Now, we could create an entity in Vault and associate it with the allow_secrets_readonly policy:

$ vault write -format=json identity/entity \
  name="lingxian" \
  policies="allow_secrets_readonly" \
  metadata=asset_id="2b17cacc-9e2d-4499-b95a-445fd26a7814"
{
  "request_id": "35ad1cbf-b333-e5b6-663d-6a05adf077bf",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "aliases": null,
    "id": "0512b829-29ea-45eb-1c5b-bae03b31b134",
    "name": "lingxian"
  },
  "warnings": null
}

For the two applications, we need to create two entity aliases. It’s important to note that the format of the entity alias name differs based on the authentication method used. In the case of the Kubernetes authentication method, the name can be either the service account UUID or the service account name, depending on the value specified for the alias_name_source when creating Kubernetes authentication roles. On the other hand, for the JWT authentication method, the alias name depends on the user_claim field specified when creating JWT authentication roles.

# For the application using Kubernetes auth method
$ vault write identity/entity-alias \
  name="default/application-foo-sa" \
  canonical_id=0512b829-29ea-45eb-1c5b-bae03b31b134 \
  mount_accessor=auth_kubernetes_7703b26e
Key             Value
---             -----
canonical_id    0512b829-29ea-45eb-1c5b-bae03b31b134
id              dc739088-7424-05cc-8411-6f65ff162145

# For the application using JWT auth method
$ vault write identity/entity-alias \
  name="system:serviceaccount:default:application-bar-sa" \
  canonical_id=0512b829-29ea-45eb-1c5b-bae03b31b134 \
  mount_accessor=auth_jwt_b46ca563
Key             Value
---             -----
canonical_id    0512b829-29ea-45eb-1c5b-bae03b31b134
id              521b262b-cf60-a4b0-b7ab-df6cea47dd7b

Vault Auth Role

Once the entity and entity aliases have been created, the next step is to proceed with the creation of authentication roles for the applications. These roles will enable the applications to authenticate with Vault and obtain the Vault tokens.

$ vault write auth/kubernetes/role/application-foo \
  bound_service_account_names="application-foo-sa" \
  bound_service_account_namespaces='*' \
  alias_name_source="serviceaccount_name" \
  ttl=24h
Success! Data written to: auth/kubernetes/role/application-foo

$ cat <<EOF | vault write auth/jwt/role/application-bar -
{
  "user_claim": "sub",
  "bound_audiences": "vault",
  "bound_subject": "system:serviceaccount:default:application-bar-sa",
  "claim_mappings": {
    "/kubernetes.io/pod/name": "pod_name",
    "/kubernetes.io/serviceaccount/name": "service_account_name",
    "/kubernetes.io/serviceaccount/uid": "service_account_uid",
    "/kubernetes.io/namespace": "namespace",
    "iss": "cluster_url"
  },
  "role_type": "jwt",
  "ttl": "24h"
}
EOF
Success! Data written to: auth/jwt/role/application-bar

As mentioned earlier, it is important to specify alias_name_source="serviceaccount_name" (instead of serviceaccount_uuid) in the Kubernetes auth role. This ensures that the entity alias name is predictable and allows us to create the entity alias before the application logs in to Vault.

The JWT auth role request body involves several key configurations:

The "user_claim": "sub" setting tells Vault to use the sub claim from the token in the request header as the entity alias name. In a Kubernetes cluster, applications are associated with service accounts, and each service account is issued a JWT token by the cluster. By setting up the JWT authentication method in Vault (which I've already completed), Vault establishes trust with the Kubernetes OIDC provider and allows applications to access Vault using their service account JWT tokens. The sub claim in the service account JWT token follows the format "system:serviceaccount:<namespace>:<service_account_name>".
The "bound_audiences": "vault" configuration ensures that the aud claim in the JWT token contains the value "vault". This requirement ensures that the token is intended for authenticating with Vault specifically.
The bound_subject setting verifies that the sub claim in the token matches the specified value. This helps to restrict the token usage to the intended subject.
The claim_mappings section defines mappings to construct entity alias metadata. For example, "/kubernetes.io/pod/name": "pod_name" means that after a successful login, the entity alias metadata will include a new key named "pod_name". The value of this key will be extracted from the Kubernetes-specific claim "/kubernetes.io/pod/name" found in the request JWT token.

Vault Authentication Using Service Account

Let’s review our progress before moving forward to creating applications in the cluster and logging into Vault:

We have successfully enabled the Kubernetes and JWT auth methods in Vault, allowing Vault to authenticate applications from the Kubernetes cluster.
We have created an entity and associated two entity aliases with it. Additionally, we have assigned the “allow_secrets_readonly” policy to the entity.
Authorization roles have been created for both the Kubernetes and JWT authentication methods

Moving forward, our next step is to create two service accounts and two pods. Each pod will attach one of the service accounts for authentication and access to Vault resources.

$ kubectl create sa application-foo-sa
serviceaccount/application-foo-sa created

$ kubectl create sa application-bar-sa
serviceaccount/application-bar-sa created

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: application-foo
  namespace: default
spec:
  serviceAccountName: application-foo-sa
  containers:
    - image: lingxiankong/kubectl
      name: application-foo-container
      command:
        - sleep
        - "43200"
EOF

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: application-bar
  namespace: default
spec:
  serviceAccountName: application-bar-sa
  containers:
    - image: lingxiankong/kubectl
      name: application-bar-container
      command:
        - sleep
        - "43200"
      volumeMounts:
        - mountPath: /var/run/secrets/tokens
          name: vault-token
  volumes:
    - name: vault-token
      projected:
        sources:
          - serviceAccountToken:
              path: vault
              audience: vault
              expirationSeconds: 900
EOF

When configuring the second Pod to log into Vault using the JWT auth method, we need to follow best security practices and pay attention to the following details:

Instead of relying on Kubernetes to inject the service account token to the default location (/var/run/secrets/kubernetes.io/serviceaccount/token), we explicitly specify an additional path that contains the token content with the desired audience.
The token audience, represented by the aud claim in the token, plays a crucial role in associating the token with a specific service or party. Depending on the services that the Pod interacts with, the audience value may vary. For example, if the Pod sends a token with an audience bound to "service A" to service A, the service A cannot forward the same token to service B and impersonate the Pod. Service B will reject the token because the token expects an audience of "service A". The Kubernetes TokenReview API allows services to specify the accepted audiences when validating a token.

To validate what we have described earlier, we can log into the “application-bar” Pod and examine the decoded default service account token, as well as the explicitly configured token with the special audience.

$ kubectl exec -it application-bar -- /bin/sh

# Inside the Pod container

$ DEFAULT_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
$ CUSTOMIZED_TOKEN=$(cat /var/run/secrets/tokens/vault)

$ echo $DEFAULT_TOKEN | jq -R 'split(".") | .[1] | @base64d | fromjson'
{
  "aud": [
    "https://kubernetes.default.svc"
  ],
  "exp": 1717751473,
  "iat": 1686215473,
  "iss": "https://oidc.eks.ap-southeast-2.amazonaws.com/id/3EBEEBB7D6994DFFE07ED159183DBA6C",
  "kubernetes.io": {
    "namespace": "default",
    "pod": {
      "name": "application-bar",
      "uid": "f0d8780f-dba7-4076-b252-3ad5454b7675"
    },
    "serviceaccount": {
      "name": "application-bar-sa",
      "uid": "f9e1ad27-1472-46f1-bd79-27899e8f06bd"
    },
    "warnafter": 1686219080
  },
  "nbf": 1686215473,
  "sub": "system:serviceaccount:default:application-bar-sa"
}

$ echo $CUSTOMIZED_TOKEN | jq -R 'split(".") | .[1] | @base64d | fromjson'
{
  "aud": [
    "vault"
  ],
  "exp": 1686217036,
  "iat": 1686216136,
  "iss": "https://oidc.eks.ap-southeast-2.amazonaws.com/id/3EBEEBB7D6994DFFE07ED159183DBA6C",
  "kubernetes.io": {
    "namespace": "default",
    "pod": {
      "name": "application-bar",
      "uid": "f0d8780f-dba7-4076-b252-3ad5454b7675"
    },
    "serviceaccount": {
      "name": "application-bar-sa",
      "uid": "f9e1ad27-1472-46f1-bd79-27899e8f06bd"
    }
  },
  "nbf": 1686216136,
  "sub": "system:serviceaccount:default:application-bar-sa"
}

In the two token files, the main difference is the aud (audience) claim. The value of the second token's aud claim is set as "vault" because we intend to use this token to communicate with Vault using the JWT auth method. We can also verify the format of the sub claim mentioned earlier.

Now, let’s proceed with logging into Vault from the “application-foo” Pod by sending an API request using its default service account token, we have previously created a Kubernetes auth role in Vault for this specific service account.

# Inside application-foo, only the default service token is available

$ export VAULT_ADDR='http://vault.default.svc.cluster.local:8200'
$ DEFAULT_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
$ role=application-foo
$ token=$(curl -sSk "$VAULT_ADDR/v1/auth/kubernetes/login" --data '
  {
    "jwt": "'$DEFAULT_TOKEN'",
    "role": "'$role'"
  }' | jq -r '.auth.client_token'); echo $token
hvs.CAESIBxFijBHS2fifwgi0cbxezJhDtXrYHPvHD7cjTAvZDs7Gh4KHGh2cy5Nc3ptN094WERhcnJicEl3eUdQMGYwTXQ

Woohoo! We have successfully obtained a Vault token. This token can be validated using vault command.

vault token lookup --format=json $token
{
  "request_id": "860ac7c5-34ba-8bce-2626-07d7c252898a",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "accessor": "OYvBW1opqnMIjGBrjec2zIIy",
    "creation_time": 1686190412,
    "creation_ttl": 2764800,
    "display_name": "kubernetes-default-application-foo-sa",
    "entity_id": "0512b829-29ea-45eb-1c5b-bae03b31b134",
    "expire_time": "2023-07-10T02:13:32.571386002Z",
    "explicit_max_ttl": 0,
    "external_namespace_policies": {},
    "id": "hvs.CAESIBxFijBHS2fifwgi0cbxezJhDtXrYHPvHD7cjTAvZDs7Gh4KHGh2cy5Nc3ptN094WERhcnJicEl3eUdQMGYwTXQ",
    "identity_policies": [
      "allow_secrets_readonly"
    ],
    "issue_time": "2023-06-08T02:13:32.571393858Z",
    "meta": {
      "role": "application-foo",
      "service_account_name": "application-foo-sa",
      "service_account_namespace": "default",
      "service_account_secret_name": "",
      "service_account_uid": "1a88bd12-3df4-4479-8357-8f62779c6203"
    },
    "num_uses": 0,
    "orphan": true,
    "path": "auth/kubernetes/login",
    "policies": [
      "default"
    ],
    "renewable": true,
    "ttl": 2689838,
    "type": "service"
  },
  "warnings": null
}

From the above token information, we can see the token has been granted a policy “allow_secrets_readonly” and the service account details has been set in the token metadata.

💡 Here is what happens inside Vault: When the application logs in using its service account token and the Kubernetes auth role, Vault will check if the service account is allowed in the role definition and then look up the entity alias associated with the Kubernetes auth method using the name “default/application-foo-sa”. Once the entity alias is found, Vault will grant the policies associated with it and its entity to the token.

We will do the same inside the “application-bar” Pod:

# Inside application-bar, log in Vault using the default service account token

$ export VAULT_ADDR='http://vault.default.svc.cluster.local:8200'
$ DEFAULT_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

$ role=application-bar
$ curl -sSk "$VAULT_ADDR/v1/auth/kubernetes/login" --data '
  {
    "jwt": "'$DEFAULT_TOKEN'",
    "role": "'$role'"
  }'
{"errors":["invalid role name \"application-bar\""]}

$ role=application-foo
$ curl -sSk "$VAULT_ADDR/v1/auth/kubernetes/login" --data '
  {
    "jwt": "'$DEFAULT_TOKEN'",
    "role": "'$role'"
  }'
{"errors":["service account name not authorized"]}

$ role=application-bar
$ curl -sSk "$VAULT_ADDR/v1/auth/jwt/login" --data '
  {
    "jwt": "'$DEFAULT_TOKEN'",
    "role": "'$role'"
  }'
{"errors":["error validating token: invalid audience (aud) claim: audience claim does not match any expected audience"]}

As we can see, attempting to log in with the default service account token would result in failure, as we have not created the necessary Kubernetes auth role in Vault for the “application-bar” service account. Using “application-foo” role doesn’t work neither, as the service account name doesn’t match. Similarly, using the JWT auth method with default service account token would also fail as the aud and sub claims in the token do not match the configured values in the JWT role.

Let’s proceed with the customized token for a successful login to Vault.

# Inside application-foo, only the default service token is available

$ export VAULT_ADDR='http://vault.default.svc.cluster.local:8200'
$ DEFAULT_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
$ role=application-foo
$ token=$(curl -sSk "$VAULT_ADDR/v1/auth/kubernetes/login" --data '
  {
    "jwt": "'$DEFAULT_TOKEN'",
    "role": "'$role'"
  }' | jq -r '.auth.client_token'); echo $token
hvs.CAESIBxFijBHS2fifwgi0cbxezJhDtXrYHPvHD7cjTAvZDs7Gh4KHGh2cy5Nc3ptN094WERhcnJicEl3eUdQMGYwTXQ

Because we have created the JWT role in Vault for the “application-bar” service account, a token has been successfully returned after login using JWT auth method. Check the token details:

$ vault token lookup --format=json $token
{
  "request_id": "d534181a-4a31-9653-f9c0-08336b11ef22",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "accessor": "lRPMlEy1fsqtmlWjsIQv4g9q",
    "creation_time": 1686269306,
    "creation_ttl": 86400,
    "display_name": "jwt-system:serviceaccount:default:application-bar-sa",
    "entity_id": "0512b829-29ea-45eb-1c5b-bae03b31b134",
    "expire_time": "2023-06-10T00:08:26.197437354Z",
    "explicit_max_ttl": 0,
    "external_namespace_policies": {},
    "id": "hvs.CAESIE-gw48XC6oQS664l_q90lirM53byWtsI2EkjdMwRKUdGh4KHGh2cy5NS0xXd3R5aWxWVmZKalpaZEthVUpnQUg",
    "identity_policies": [
      "allow_secrets_readonly"
    ],
    "issue_time": "2023-06-09T00:08:26.197444091Z",
    "meta": {
      "cluster_url": "<https://oidc.eks.ap-southeast-2.amazonaws.com/id/3EBEEBB7D6994DFFE07ED159183DBA6C>",
      "namespace": "default",
      "pod_name": "application-bar",
      "role": "application-bar",
      "service_account_name": "application-bar-sa",
      "service_account_uid": "f9e1ad27-1472-46f1-bd79-27899e8f06bd"
    },
    "num_uses": 0,
    "orphan": true,
    "path": "auth/jwt/login",
    "policies": [
      "default"
    ],
    "renewable": true,
    "ttl": 81001,
    "type": "service"
  },
  "warnings": null
}

The Vault token assigned to service account “application-bar” has the same policy as the one created for “application-foo”. This is because both applications share the same entity that we created earlier. However, the token of “application-bar” differs in that its metadata fields are based on what we defined in the JWT auth role definition. Additionally, we obtain the Pod name in addition to service account details, which provides a major security benefit. With this information, we can easily trace back to the pod whenever the application communicates with other services using the token.

Unified Identity Token

The token we obtained from the previous steps is specifically meant for communication with Vault, rather than between the applications themselves, especially when the applications are running in different Kubernetes clusters or environments. In such cases, the commonly used approach for authentication and authorization is through JSON Web Tokens (JWT).

As mentioned earlier, the Kubernetes service account token is actually in the standard JWT format. However, it is primarily intended for communication with the Kubernetes API and should not be used outside of the Kubernetes cluster.

Not surprisingly, Vault can also act as an OpenID Connect (OIDC) identity provider and issue JWT tokens to authenticated clients.

For the applications to obtain a JWT token from Vault, we need to grant permissions to the application’s Vault identity to access the identity/oidc/token/${oidc_role_name} path. For testing purposes, we can add an additional permission to the existing policy associated with the entity so that we do not need to change the entity itself.

$ cat <<EOF | vault policy write allow_secrets_readonly -
path "secret/*" {
  capabilities = ["read", "list"]
}
path "identity/oidc/token/*" {
  capabilities = ["read"]
}
EOF
Success! Uploaded policy: allow_secrets_readonly

The final step in the configuration process is to create a new OIDC role. This role will define the template string used for generating the JWT tokens. Typically, the client_id parameter corresponds to the target service identity that the application intends to access.

$ template='{
  "entity": {
    "name": {{identity.entity.name}},
    "id": {{identity.entity.id}},
    "metadata": {{identity.entity.metadata}}
  },
  "auth": {
    "type": "jwt",
    "metadata": {{identity.entity.aliases.auth_jwt_b46ca563.metadata}}
  }
}'

$ vault write identity/oidc/role/my-first-oidc-role \
  key=default \
  template="$(echo $template | base64)" \
  client_id="123456"
Success! Data written to: identity/oidc/role/my-first-oidc-role

The minimum configuration required for the applications to request their identity JWT token from Vault has been completed. Let’s take the “application-bar” Pod as an example. Since we have already granted the necessary permissions to generate the identity token on the entity, the “application-bar” Pod should be able to retrieve its own JWT token after logging in Vault. This token can then be used for communication with other applications or services.

# Inside the application-bar Pod, after getting the Vault token
$ curl -sSk \
  -X GET \
  -H "X-Vault-Token: $token" \
  "$VAULT_ADDR/v1/identity/oidc/token/my-first-oidc-role" \
  | jq -r '.data.token' | jq -R 'split(".") | .[1] | @base64d | fromjson'
{
  "aud": "123456",
  "auth": {
    "metadata": {
      "cluster_url": "https://oidc.eks.ap-southeast-2.amazonaws.com/id/3EBEEBB7D6994DFFE07ED159183DBA6C",
      "namespace": "default",
      "pod_name": "application-bar",
      "service_account_name": "application-bar-sa",
      "service_account_uid": "f9e1ad27-1472-46f1-bd79-27899e8f06bd"
    },
    "type": "jwt"
  },
  "entity": {
    "id": "0512b829-29ea-45eb-1c5b-bae03b31b134",
    "metadata": {
      "asset_id": "2b17cacc-9e2d-4499-b95a-445fd26a7814"
    },
    "name": "lingxian"
  },
  "exp": 1686391727,
  "iat": 1686305327,
  "iss": "/v1/identity/oidc",
  "namespace": "root",
  "sub": "0512b829-29ea-45eb-1c5b-bae03b31b134"
}

After logging in to Vault, the Pod can retrieve a JWT token from the identity/oidc/token endpoint. One of the advantages of using the JWT token generated by Vault is that, as a Vault operator, we have the ability to customize the claims within the token. This allows us to generate unified and consistent identity tokens for different applications using different authentication methods. By tailoring the claims to our specific requirements, we can ensure that the identity tokens provide the necessary information for authentication and authorization purposes.

A final note about Vault entity is that you can difinitely skip the explicit creation of entity and entity alias steps, and directly log in Vault with your prefered auth backend. Behind the scenes, Vault will still automatically generate the necessary entity and entity alias for you. You have to rely on the auth roles to attach policies. However, If you are using Kubernetes auth backend without explicitly specifying the alias_name_source="serviceaccount_name", you may end up with a situation where numerous orphan entities are created in Vault. This can result in significantly increased costs associated with your Vault license, because even if you delete and re-create a service account using the same name, Vault considers them as distinct identifiers due to the different resource UUIDs, please refer to “What is a Client” for more information.

Vault Integration with Kubernetes — Unified Identity Management Service

Identity Entity and Entity Alias

Vault Auth Role

Vault Authentication Using Service Account

Unified Identity Token

Written by Lingxian Kong