Mapping Kubernetes Service Accounts to GCP IAMs using Workload Identity

GCP Credential Management on GKE just got a whole lot easier

We love Google Kubernetes Engine (GKE) but until recently we did not have a great story around how to give our services running on Kubernetes the desired Google Cloud Platform (GCP) permissions.

We explored a bunch of tooling around GCP service account credential management - leveraging Terraform, Vault, Kubernetes Secrets and KMS encrypted credentials along the way - but nothing gave us the clean, scalable, short-lived token solution we were looking for.

What we really wanted (and asked for many times) was a way for Kubernetes service accounts to act as GCP service accounts — something Google has now finally made possible via Workload Identities.

Let’s walk through an example of how this works.

Setup

• We have a GKE cluster running under a service account:
  gke-cluster-sa@louis-testing-project.iam.gserviceaccount.com
• We have a bucket gs://workload-identity-test
• We have a different service account which can read the bucket:
  bucket-lister-sa@louis-testing-project.iam.gserviceaccount.com

Objective: A specific pod running on our GKE cluster can list objects in our bucket without ever touching a GCP service account key or mounting any secrets.

How Things Worked Before

Let’s try to list the bucket contents without enabling Workload Identity.

kubectl run -i -tty test1 -image gcr.io/cloud-builders/gsutil ls gs://workload-identity-test
> AccessDeniedException: 403 gke-cluster-sa@louis-testing-project.iam.gserviceaccount.com does not have storage.objects.list access to workload-identity-test.

So what happened here? The gsutil CLI running inside the pod hit the node’s metadata endpoint which returned a token for the service account running the node - gke-cluster-sa. This service account doesn’t have permission to read the bucket so we get a 403.

Setting Up Workload Identity

The first thing we need to do is enable Workload Identity on the GKE cluster.

The Identity Namespace, which is statically defined in the Cluster Edit UI, maps the Kubernetes service account name to a virtual GCP service account handle used for Identity & Access Management (IAM) binding (more on this below).

Enable the Metadata Server

The Metadata Server runs on each GKE node and changes the behavior of the metadata endpoint. This also secures the metadata endpoint removing the need for metadata concealment. Let’s enable Metadata Server on our default node pool.

gcloud beta container node-pools update default-pool \
 — cluster=workload-identity-test \
 — workload-metadata-from-node=GKE_METADATA_SERVER \
 — zone=us-central1-b

With the Metadata Server running on our nodes let’s see what happens of we try to list our bucket contents now.

kubectl run -i — tty test2 — image gcr.io/cloud-builders/gsutil ls gs://workload-identity-test
> ServiceException: 401 Invalid Credentials

We’re getting somewhere. With the metadata endpoint hardened we can no longer retrieve tokens for the service account running the node and so we now get a 401 error instead of 403.

Create and Bind a Kubernetes Service Account

Let’s create a specially annotated Kubernetes service account — witest

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    iam.gke.io/gcp-service-account: bucket-lister-sa@louis-testing-project.iam.gserviceaccount.com
  name: witest
  namespace: default

Here we’ve defined a service account witest, in the default namespace and we’ve provided an annotation that defines the GCP service account we want to map to.

Next we need to bind our derived virtual GCP service account handle to the real GCP service account we want to act as.

gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:louis-testing-     project.svc.id.goog[default/witest]" \
  bucket-lister-sa@louis-testing-project.iam.gserviceaccount.com

serviceAccount:louis-testing-project.svc.id.goog[default/witest] is derived from the Identity Namespace defined when enabling Workload Identity along with the namespace and name of the Kubernetes service account we are binding.

Confirm it Works

Let’s try and list our bucket’s contents after configuring everything.

kubectl run --serviceaccount=witest -i --tty test3 --image gcr.io/cloud-builders/gsutil ls gs://workload-identity-test
> gs://workload-identity-test/success/

Superb, this works beautifully. We are able to list the bucket contents with nary a long-lived token in sight. This is a huge feature which makes our credential management and security story much nicer — thanks Google!