Whoami — The quest of understanding GKE Workload Identity Federation

If you’re anything like me then using product features that you don’t fully understand always leaves you with a feeling of unease. Sure, using the feature might even be easy and cheerful at least as long as everything works as expected. We could even leave it at that. However, somewhere in between intrinsic engineering curiosity and the life experience that at some point in the future a deeper understanding will come in handy, we still have the desire to understand and debunk the magic.

For me one of these magic features was the Workload Identity Federation in GKE. I used it for a long time to provide workloads running in my GKE clusters with a service account identity that it can then use to authenticate itself to use other Google Cloud services. At the surface this immediately makes sense and I quickly embraced that downloading and storing GCP service account keys is risky and can be avoided by associating a GKE service account with their GCP counterpart via Workload Identity Federation. It all just worked fine but honestly I never fully understood how it worked under the hood.

A recent conversation with an old friend reminded me of this quest and led to the blog posts you are reading here. It takes us on a journey down the rabbit hole of Workload Identity Federation that includes the API calls behind a popular gcloud command, DNS lookups, link-local IP addresses and and of course iptables.

Disclaimer: The experiment presented in this blog post relates to the state of the Workload Identity Federation feature at the time of writing. Depending on when you read this post things might have changed and work completely differently to how it is described here.

Edit April 2024: GKE now allows you to use the Kubernetes service account directly as a principal in an IAM binding. The approach described here remains valid and serves as the fallback to address known limitations of the direct binding approach.

img credit: https://unsplash.com/photos/two-white-rabbits-JkgVHEFSolA

Setting up our experiment

If you want to experience the journey to Workload Identity Federation first-hand, you can follow along the steps of this blog in your own GCP project. Along the way we will create a minimal zonal cluster and deploy a sample workload in it. For this you have to specify the project ID and zone that you want to use:

export PROJECT_ID=<YOUR PROJECT ID HERE>
export ZONE=europe-west1-b

We then run the following snippet to enable the required API and create the cluster for our experiment.

gcloud services enable compute.googleapis.com container.googleapis.com
gcloud container clusters create demo-gke-cluster --zone $ZONE --num-nodes 1 --project $PROJECT_ID
gcloud container clusters get-credentials demo-gke-cluster --zone $ZONE --project $PROJECT_ID

In a world without Workload Identity Federation

Before Workload Identity Federation existed, you had to go through the following steps to use a GCP service account from within a Kubernetes workload:

1. Create a new service account and key in IAM
2. Create a secret that holds the service account key in either the Kubernetes API, Secret Manager or a third-party secret management solution.
3. Mount the secret on to the pod
4. (Optional but highly recommended) regularly rotate the secret key

In practice this would have looked something like this snippet:

##
## DO NOT USE IN PRODUCTION!
## THIS IS ONLY MEANT TO ILLUSTRATE 
## THE NEED FOR WORKLOAD IDENTITY FEDERATION
##

# Create the service account in GCP
gcloud iam service-accounts create sa-with-key \
    --description="Service Account with Key Export"

# Create and download a service account key file in json format
gcloud iam service-accounts keys create ./sa-with-key-export.json \
    --iam-account=sa-with-key@$PROJECT_ID.iam.gserviceaccount.com

# Create a k8s secret for the key
kubectl create secret generic sa-key --from-file=key.json=./sa-with-key-export.json

# Create a pod with the secret mounted
kubectl apply -f - <<EOF 
apiVersion: v1
kind: Pod
metadata:
  name: sa-key-mount
spec:
  containers:
  - image: google/cloud-sdk:slim
    name: workload-identity-test
    command: ["sleep","infinity"]
    volumeMounts:
    - name: sa-key
      mountPath: "/var/secrets/google"
      readOnly: true
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: /var/secrets/google/key.json
  volumes:
  - name: sa-key
    secret:
      secretName: sa-key

EOF

Once our sample pod is ready, we can start a new interactive session to poke around and see if the service account can be used as expected:

kubectl exec -it sa-key-mount  -- /bin/bash

From within the pod we can try to get an access token for the service account and inspect that token on the tokeninfo endpoint.

SA_TOKEN=$(gcloud auth application-default print-access-token --scopes openid,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/cloud-platform)

curl -H "Authorization: Bearer $SA_TOKEN" https://www.googleapis.com/oauth2/v3/tokeninfo

This should output some metadata about our access token that confirms the GCP service account we initially created

{
  // ...
  "scope": "https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email openid",
  "email": "sa-with-key@<PROJECT_ID>iam.gserviceaccount.com",
}

In a real-world scenario the service account would of course be used to perform more useful tasks such as for example pushing a message to Pub/Sub, generating content with Vertex AI, or trigger a deployment on GKE.

Switching to Workload Identity Federation

Using an exported service account credential in a Kubernetes pod works just fine. But as described in the GKE documentation mounting GCP service account keys as secrets is an explicitly discouraged practice. Exporting the service account keys creates unnecessary risks of leaking credentials that could be used in other places and for malicious purposes. Further, keeping service account keys fresh by following the best practices for regularly rotating the generated service account keys adds additional operational complexity.

The solution to this problem is to abandon the process of exporting service account keys and using Workload Identity Federation instead.

As per the documentation Workload Identity Federation is enabled on an existing GCP cluster by updating our cluster with a workload-pool and adding the GKE metadata server to our default nodepool. By adding the metadata server flag to the nodepool GKE creates a daemonset in your cluster that ensures that a metadata server instance runs on every cluster node.

Note that in production you would probably want to create a new GKE metadata server-enabled nodepool and shift traffic over. For GKE Autopilot clusters Workload Identity Federation is already enabled by default.

gcloud container clusters update demo-gke-cluster \
    --zone=$ZONE \
    --workload-pool=$PROJECT_ID.svc.id.goog

gcloud container node-pools update default-pool \
    --cluster=demo-gke-cluster \
    --zone=$ZONE \
    --workload-metadata=GKE_METADATA

When the cluster is updated and ready we can prepare our Workload Identity Federation by creating the service accounts on both GCP and GKE side and authorizing the GKE SA for the one in GCP

kubectl create serviceaccount workload-id-demo-k8s-sa

gcloud iam service-accounts create workload-id-demo-gcp-sa --description "SA for the GKE Demo Workload"

gcloud iam service-accounts add-iam-policy-binding workload-id-demo-gcp-sa@$PROJECT_ID.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:$PROJECT_ID.svc.id.goog[default/workload-id-demo-k8s-sa]"

kubectl annotate serviceaccount workload-id-demo-k8s-sa \
    --namespace default \
    iam.gke.io/gcp-service-account=workload-id-demo-gcp-sa@$PROJECT_ID.iam.gserviceaccount.com

With the instructions above we created a pair of service accounts in GKE and GCP. These service accounts are linked via the annotation in the GKE service account and the Kubernetes service account is authorized to generate tokens via the policy binding for the GCP service account.

Relationship GCP and GKE Service Account

Let’s again create a new pod with the same container image as before. This time however, we don’t mount the service account secret but instead tell Kubernetes to use the dedicated Kubernetes service account that we just created. If you look closely at the pod spec you also note that we included a node selector that ensures that the node where the pod is scheduled on has the metadata server enabled. This is required for the Workload Identity Federation to work.

kubectl apply -f - <<EOF 
apiVersion: v1
kind: Pod
metadata:
  name: workload-identity
  namespace: default
spec:
  containers:
  - image: google/cloud-sdk:slim
    name: workload-identity-test
    command: ["sleep","infinity"]
  serviceAccountName: workload-id-demo-k8s-sa
  nodeSelector:
    iam.gke.io/gke-metadata-server-enabled: "true"
EOF

Once our new pod is ready, we can start another interactive shell session:

kubectl exec -it workload-identity  -- /bin/bash

We can then verify that we can use the service account identity federation as expected. By obtaining a new access token via the gcloud command and inspecting it via the tokeninfo endpoint.

SA_TOKEN=$(gcloud auth application-default print-access-token --scopes openid,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/cloud-platform)

curl -H "Authorization: Bearer $SA_TOKEN" https://www.googleapis.com/oauth2/v3/tokeninfo

As before this should give us the metadata for the GCP service account including its email and other properties:

{
  // ...
  "scope": "https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email openid",
  "email": "workload-id-demo-gcp-sa@strebel-gke-wlid.iam.gserviceaccount.com",
}

If you’re only interested in using Workload Identity Federation, you could stop here and use the associated GCP service account from your pods happily ever after. However, if you’re interested in understanding what enables this useful functionality under the hood the interesting part is yet to come.

Tracking down the Workload Identity

To understand how the workload gets its access token we want to first look at the gcloud print print-access-token cli command in more detail. As with other gcloud commands you can add the –log-http attribute to see the underlying http calls that are made on your behalf by the gcloud cli.

In our case we want to use the following command in the interactive session within our pod with workload identity and filter for the URIs that are called under the hood:

gcloud auth application-default print-access-token --log-http 2> >(grep uri:) 1> /dev/null

Based on the two URLs in the output we learn that the gcloud command makes two GET requests to the metadata server to:

1. Figure out the associated GCP service account
2. Request an access token for this GCP service account

uri: http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true

uri: http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/workload-id-demo-gcp-sa@$PROJECT_ID.iam.gserviceaccount.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform

Let’s try if we can replicate the requests manually from within our pod. Note that we also send the Metadata-Flavor header along with the request. The metadata service will ask us to include if it is missing.

PROJECT_ID=$(gcloud config get project)

curl -v -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true

curl -v -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/workload-id-demo-gcp-sa@$PROJECT_ID.iam.gserviceaccount.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform

This works just fine. Apparently the metadata.google.internal endpoint does something interesting that also doesn’t need any authentication and returns the access token. But what’s behind this internal hostname? For this we’ll install dnsutils on our example pod and do a quick DNS lookup

apt install dnsutils -y
dig metadata.google.internal

Dig responds with an A record for the IP address of 169.254.169.254 which corresponds to a link local address according to RFC3927 which means the packets aren’t forwarded by the router and instead stay local to the network device. But where are those packets picked up then?

Remember when we asked our pod to be scheduled only on nodes that have a metadata server enabled? Let’s see if we can find any resources in our cluster that correspond to the metadata server.

kubectl get daemonset -n kube-system | grep metadata

# There is a daemonset called gke-metadata-server. 
# Let's look at it in more detail

kubectl get pod -l k8s-app=gke-metadata-server -n kube-system  -o yaml

Here we can see that the metadata server runs with hostPort = true and runs a metadata server on port 988. Let’s take a note of the pod IP and try calling the metadata server pod from the sample pod. Note that we perform the same call as with the metadata.google.internal hostname but replace the hostname with the pod IP and custom port.

PROJECT_ID=$(gcloud config get project)
METADATA_POD_IP=10.x.x.x

curl -v -H "Metadata-Flavor: Google" http://$METADATA_POD_IP:988/computeMetadata/v1/instance/service-accounts/workload-id-demo-gcp-sa@$PROJECT_ID.iam.gserviceaccount.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform

That works just as before. Now the last big mystery is how did we get from the implicit port 80 of plain HTTP to the port 988 that the daemonset is using on the host?

Maybe a look at the iptables can help us shed some light on this? For this we’d have to SSH into the GCE VM that hosts the GKE Node and run the following command to list the iptables prerouting filter:

sudo iptables -t nat -L PREROUTING -n

The response as shown below shows that iptables are in fact responsible for routing traffic to the metadata server that is running on the port 988.

DNAT       tcp  --  0.0.0.0/0            169.254.169.254      tcp dpt:80 /* metadata-concealment: bridge traffic to metadata server goes to metadata proxy */ to:169.254.169.252:988

With this information we can complete our picture and now have an end to end understanding of how our workload talks to the metadata server to get an access token for the associated GCP service account using Workload Identity Federation in GKE.

(Bonus) Changing my Workload identity

To better understand how the metadata server works, we could try to see if we can get an access token for an unrelated GCP service account. Let’s first create a new service account by running the following command in a cloud shell terminal:

gcloud iam service-accounts create unrelated-gcp-sa --description "Unrelated SA"

From what we’ve learned before, we could try to get an access token for the new GCP service account from the metadata server like so:

curl -v -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/unrelated-gcp-sa@$PROJECT_ID.iam.gserviceaccount.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform

The metadata server responds with a 404 Not Found error. If we only query the path for listing the service accounts on the metadata server we can see why that’s the case.

curl -v -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/

The service accounts served by the metadata server seems to be limited to the GCP service account that is set in the annotation of the pod’s service account. Let’s change the annotation on our Kubernetes service account and tell it that we want it to be associated with the new unrelated service account.

kubectl annotate serviceaccount workload-id-demo-k8s-sa \
    --namespace default \
    iam.gke.io/gcp-service-account=unrelated-gcp-sa@$PROJECT_ID.iam.gserviceaccount.com --overwrite

When we run the token request again our error message now changed to a 403 Forbidden. The error message is clear that our workload ID is missing the iam.serviceAccounts.getAccessToken permission that is needed to create an access token. This permission is part of the workload identity user role. Let’s go ahead and create the policy binding for our new GCP service account.

gcloud iam service-accounts add-iam-policy-binding unrelated-gcp-sa@$PROJECT_ID.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:$PROJECT_ID.svc.id.goog[default/workload-id-demo-k8s-sa]"

Finally we’re able to obtain the service account access token for our GCP service account. In this process we learned that both the annotation on the GKE service account and the policy binding are required to get the access token for a given service account which makes sense from both a security and intent declaration perspective.

Where do we go from here?

Hopefully the experiment in this post was able to shed light on the GKE Workload Identity Federation feature and debunk some of the magic and confusion. With that deeper understanding you should be equipped with the necessary knowledge and confidence to retire your service account keys once and for all and embrace the new world of Workload Identity Federation in GKE.