Mapping Kubernetes Service Accounts to GCP IAMs using Workload Identity

GCP Credential Management on GKE just got a whole lot easier

Louis Vernon
Jun 18 · 3 min read

We love Google Kubernetes Engine (GKE) but until recently we did not have a great story around how to give our services running on Kubernetes the desired Google Cloud Platform (GCP) permissions.

We explored a bunch of tooling around GCP service account credential management - leveraging Terraform, Vault, Kubernetes Secrets and KMS encrypted credentials along the way - but nothing gave us the clean, scalable, short-lived token solution we were looking for.

What we really wanted (and asked for many times) was a way for Kubernetes service accounts to act as GCP service accounts — something Google has now finally made possible via Workload Identities.

Let’s walk through an example of how this works.

Setup

  • We have a GKE cluster running under a service account:
    gke-cluster-sa@louis-testing-project.iam.gserviceaccount.com
  • We have a bucket gs://workload-identity-test
  • We have a different service account which can read the bucket:
    bucket-lister-sa@louis-testing-project.iam.gserviceaccount.com

Objective: A specific pod running on our GKE cluster can list objects in our bucket without ever touching a GCP service account key or mounting any secrets.

How Things Worked Before

Let’s try to list the bucket contents without enabling Workload Identity.

So what happened here? The gsutil CLI running inside the pod hit the node’s metadata endpoint which returned a token for the service account running the node - gke-cluster-sa. This service account doesn’t have permission to read the bucket so we get a 403.

Setting Up Workload Identity

The first thing we need to do is enable Workload Identity on the GKE cluster.

The Identity Namespace, which is statically defined in the Cluster Edit UI, maps the Kubernetes service account name to a virtual GCP service account handle used for Identity & Access Management (IAM) binding (more on this below).

Enable the Metadata Server

The Metadata Server runs on each GKE node and changes the behavior of the metadata endpoint. This also secures the metadata endpoint removing the need for metadata concealment. Let’s enable Metadata Server on our default node pool.

With the Metadata Server running on our nodes let’s see what happens of we try to list our bucket contents now.

We’re getting somewhere. With the metadata endpoint hardened we can no longer retrieve tokens for the service account running the node and so we now get a 401 error instead of 403.

Create and Bind a Kubernetes Service Account

Let’s create a specially annotated Kubernetes service account — witest

Here we’ve defined a service account witest, in the default namespace and we’ve provided an annotation that defines the GCP service account we want to map to.

Next we need to bind our derived virtual GCP service account handle to the real GCP service account we want to act as.

serviceAccount:louis-testing-project.svc.id.goog[default/witest] is derived from the Identity Namespace defined when enabling Workload Identity along with the namespace and name of the Kubernetes service account we are binding.

Confirm it Works

Let’s try and list our bucket’s contents after configuring everything.

Superb, this works beautifully. We are able to list the bucket contents with nary a long-lived token in sight. This is a huge feature which makes our credential management and security story much nicer — thanks Google!

Louis Vernon

Written by

Security, Operations and Site Reliability Engineering (SOS) at Descartes Labs.