From whoami to whoarewe with GKE Workload Identity for Fleets
In a previous post we’ve maneuvered ourselves deeper and deeper into the rabbit hole of GKE workload identity federation. We started by looking at why you want to use a workload identity compared to just using service account credentials in the form of secret keys and then explored how it all works under the hood and what this all has to do with the ominous metadata server in our GKE nodes.
Based on the feedback on that post and the many interesting discussions that post had triggered, I wanted to follow up with a short extension that looks at workload identity in GKE fleets in the same exploratory way that we did for the classical Project-based workload identity. This means another rabbit hole that takes from taking apart a gcloud command, an RFC 8693 token exchange, federated tokens to a place where we hopefully have a better understanding of how we can impersonate GCP service accounts from a fleet-scoped Kubernetes service account.
Disclaimer: The experiment presented in this blog post relates to the state of the Fleets Workload Identity Federation feature at the time of writing. Depending on when you read this post things might have changed and work completely differently to how it is described here.
What’s a Fleet Workload Identity?
Fleets are a construct in GCP to simplify the management of multi-cluster topologies by providing a way to define explicit logical groupings of GKE clusters based on the concepts of sameness and trust boundaries. Fleets form the basis for a range of fleet-enabled features in GKE Enterprise such as Config Sync, Policy Controller, Security Posture Dashboard, Multi-Cluster ingress and many more. Another important capability that the fleet entity enables is the ability to define a shared common workload identity pool for all clusters in a GKE fleet.
The main value proposition of a fleet workload identity over a traditional project bound workload identity becomes apparent when we consider a multi-cluster and multi-project topology like the following:
In the diagram above we see the two scenarios. The left side depicts the traditional setup with two projects each with their own workload identity pool. On the right we have a shared workload identity pool for all the clusters in the project. With a shared workload identity pool the two clusters in our example follow the concept of nameness . We assume that for each of the cluster resources that are in the same namespaces and have the same name are considered the same logical entity that spans across the clusters. In the context of workload identity the fleet sameness property is used to allow us to refer to the Kubernetes service accounts with the same name and in the same namespace as a single entity regardless of the underlying clusters that they are hosted in.
Provision Fleet Workload Identities for a GKE Pod
The explorations and command line instructions in this post assume a GKE cluster with workload identity and fleets enabled as described in the docs. The other steps required to enable and use workload identity for fleets are provided with additional information below.
One result of a fleet-based workload identity federation is that you can obtain a single entity to represent the service accounts across different clusters and projects. This means that when we call the GCP IAM API to authorize the Kubernetes SA to use our GCP SA we only need to issue one call that then applies the permission for all service accounts in a specific namespace in all clusters of the same fleet, even if they are hosted in different GCP Projects.
gcloud iam service-accounts add-iam-policy-binding workload-id-demo-gcp-sa@$PROJECT_ID.iam.gserviceaccount.com \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:$FLEET_PROJECT_ID.svc.id.goog[my-namespace/workload-id-demo-k8s-sa]"On the Kubernetes side, the workload identity pool information and mapping to the GCP Service account is not applied in the form of annotations on the service accounts like it was for project-scoped workload identity federation. Instead the mapping is configured in the form of a ConfigMap resource that is populated with the GKE Fleet membership information like so:
kubectl apply -f - <<EOF
kind: Namespace
apiVersion: v1
metadata:
name: my-namespace
---
kind: ConfigMap
apiVersion: v1
metadata:
namespace: my-namespace
name: wlid-demo-config
data:
config: |
{
"type": "external_account",
"audience": "identitynamespace:$FLEET_PROJECT_ID.svc.id.goog:https://container.googleapis.com/v1/projects/$GKE_CLUSTER_PROJECT_ID/locations/$GKE_CLUSTER_LOCATION/clusters/$GKE_CLUSTER_NAME",
"service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/workload-id-demo-gcp-sa@$PROJECT_ID.iam.gserviceaccount.com:generateAccessToken",
"subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
"token_url": "https://sts.googleapis.com/v1/token",
"credential_source": {
"file": "/var/run/secrets/tokens/gcp-ksa/token"
}
}
EOFOn the workload themselves this config map is then mounted together with the projected service account token to enable workload identity federation:
kubectl apply -f - <<EOF
kind: ServiceAccount
apiVersion: v1
metadata:
namespace: my-namespace
name: workload-id-demo-k8s-sa
automountServiceAccountToken: false
---
apiVersion: v1
kind: Pod
metadata:
name: gcloud-demo-pod
namespace: my-namespace
spec:
serviceAccountName: workload-id-demo-k8s-sa
containers:
- name: gcloud
command: ["sleep","infinity"]
image: google/cloud-sdk:slim
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /var/run/secrets/tokens/gcp-ksa/google-application-credentials.json
volumeMounts:
- name: gcp-ksa
mountPath: /var/run/secrets/tokens/gcp-ksa
readOnly: true
volumes:
- name: gcp-ksa
projected:
defaultMode: 420
sources:
- serviceAccountToken:
path: token
audience: $FLEET_PROJECT_ID.svc.id.goog
expirationSeconds: 172800
- configMap:
name: wlid-demo-config
optional: false
items:
- key: "config"
path: "google-application-credentials.json"
EOFVerifying the Workload Identity
Once our demo pod is ready, we can start an interactive shell session:
kubectl exec -it gcloud-demo-pod -n my-namespace -- /bin/bashWe can then verify that we can use the service account identity federation as expected. By obtaining a new access token via the gcloud command and inspecting it via the tokeninfo endpoint.
SA_TOKEN=$(gcloud auth application-default print-access-token --scopes openid,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/cloud-platform)
curl -H "Authorization: Bearer $SA_TOKEN" https://www.googleapis.com/oauth2/v3/tokeninfoAs expected and just like with the project-level workload identity federation this gives us the metadata for the access token that we obtained for our GCP service account including its email and other properties:
{
// ...
"scope": "https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email openid",
"email": "workload-id-demo-gcp-sa@$PROJECT_ID.iam.gserviceaccount.com",
}Understanding what is going on under the hood
Just like in the blog post on project-level workload identity federation you could stop reading here if you were only interested in getting your fleet-scoped workload identity federation working. However, if you’ve made it this far, you might also be interested in performing some digging that will allow us to explore how the workload identity federation came about.
To make our lives a bit easier and allow us to elegantly extract values from json response payloads we first install the jq tool in our pod:
apt-get install jqJust like in the previous blog post, let’s start by looking at the HTTP calls that are made on your behalf in the process of calling the gcloud command to print the access token.
gcloud auth application-default print-access-token --log-httpIn the verbose logging output we can see that a single HTTP POST API call is made to the IAM credentials API to get a token for our GCP Service Account:
uri: https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/dworkload-id-demo-gcp-sa@$PROJECT_ID.iam.gserviceaccount.com:generateAccessToken
method: POST
...For comparison for the project-scoped workload identity federation this consisted of two separate calls to first get the workload identity and then the token from the metadata server:
uri: http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
uri: http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/workload-id-demo-gcp-sa@$PROJECT_ID.iam.gserviceaccount.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platformSo in the case of workload identity in fleets our workload is fetching the impersonation token directly via a POST on the iamcredentials.googleapis.com API. Let’s try to see if we can reproduce this call:
curl -X POST https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/workload-id-demo-gcp-sa@$PROJECT_ID.iam.gserviceaccount.com:generateAccessTokenWe didn’t really expect this call to work first try but at least the error message gives us some good pointers of how to continue
{
"error": {
"code": 401,
"message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"status": "UNAUTHENTICATED",
"details": [
{
"@type": "type.googleapis.com/google.rpc.ErrorInfo",
"reason": "CREDENTIALS_MISSING",
"domain": "googleapis.com",
"metadata": {
"method": "google.iam.credentials.v1.IAMCredentials.GenerateAccessToken",
"service": "iamcredentials.googleapis.com"
}
}
]
}
}The error response is very clear in that this API is expecting an OAuth2 token. But we don’t yet have an access token that is valid for GCP. However we have access to the Kubernetes service account JWT token that we mounted on our pod. Let’s unpack the JWT to see its content:
cat /var/run/secrets/tokens/gcp-ksa/token | jq -R 'split(".") | .[1] | @base64d | fromjson'This will give us an output similar to the one below
{
"aud": [
"$PROJECT_ID.svc.id.goog"
],
"iss": "https://container.googleapis.com/v1/projects/.../locations/.../clusters/...",
"kubernetes.io": {
"namespace": "my-namespace",
"pod": {
"name": "gcloud-demo-pod",
"uid": "..."
},
"serviceaccount": {
"name": "workload-id-demo-k8s-sa",
"uid": "..."
}
},
"sub": "system:serviceaccount:my-namespace:workload-id-demo-k8s-sa",
...
}To translate this Kubernetes service account token into a GCP token we’ll have to look at the ConfigMap resource that we created according to the fleet workload identity documentation. Here we can find a reference to the Security Token Service under sts.googleapis.com. This service can be used to obtain a federated token that can later be used to obtain an actual GCP access token. The STS service is based on an RFC 8693 token exchange so we can easily craft our token exchange request as per the STS documentation and exchange our Kubernetes token in the form of a subjectToken for a GCP token.
TOKEN_EXCHANGE_RESPONSE=$(curl -X POST https://sts.googleapis.com/v1/token \
-H 'Content-Type: application/json; charset=utf-8' \
--data-binary @- << EOF
{
"audience": "$(cat /var/run/secrets/tokens/gcp-ksa/google-application-credentials.json | jq -r '.audience')",
"grantType": "urn:ietf:params:oauth:grant-type:token-exchange",
"scope": "openid https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email",
"subjectToken": "$(cat /var/run/secrets/tokens/gcp-ksa/token)",
"requestedTokenType": "urn:ietf:params:oauth:token-type:access_token",
"subjectTokenType": "urn:ietf:params:oauth:token-type:jwt"
}
EOF)
echo "$TOKEN_EXCHANGE_RESPONSE" | jqThis request gives us a federated token in the following format:
{
"access_token": "ya29.d...",
"issued_token_type": "urn:ietf:params:oauth:token-type:access_token",
"token_type": "Bearer",
"expires_in": 3598
}The contained access_token isn’t yet usable with the Google Cloud APIs so we need to exchange it once more to obtain a service account token via the service account impersonation API. For this we perform the same POST request for iamcredentials.googleapis.com that we found it in the http call log of our gcloud command:
IMPERSONATION_URL="$(cat /var/run/secrets/tokens/gcp-ksa/google-application-credentials.json | jq -r '.service_account_impersonation_url')"
ACCESS_TOKEN_RESPONSE=$(curl -X POST "$IMPERSONATION_URL" -H "Authorization: Bearer $(echo "$TOKEN_EXCHANGE_RESPONSE" | jq -r '.access_token')" -H "Content-Type: application/json" --data '{"scope": ["https://www.googleapis.com/auth/cloud-platform", "openid", "https://www.googleapis.com/auth/userinfo.email"]}')
echo $ACCESS_TOKEN_RESPONSE | jqThe access token contained in the response body can now be used in the tokeninfo endpoint to inspect its content:
curl -H "Authorization: Bearer $(echo "$ACCESS_TOKEN_RESPONSE" | jq -r '.accessToken')" https://www.googleapis.com/oauth2/v3/tokeninfoFinally this returns the same information about the GCP service account that is bound to our pod’s service account as we had previously obtained via the gcloud CLI before:
{
// ...
"scope": "https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email openid",
"email": "workload-id-demo-gcp-sa@$PROJECT_ID.iam.gserviceaccount.com",
}Obviously in any production scenario we would want to rely on the provided SDK functionality whenever possible. This little experiment of going from the Kubernetes service account token via token exchange to the federated token and finally obtaining the access token for the GCP service account was mainly meant for illustrative purposes and curiosity of how things work under the hood.
