Workload Identity - Secured Way to Access Google Cloud APIs from GKE Workloads.

Let’s see first understand old ways to access GCP APIs like GCP Cloud storage query and other services from Kubernetes workload (deployement/pods)

Using Service account keys as secrets

GCP IAM service account is special type of account which used by applications running on compute engine/gke/cludrun to access GCP APIs.Application developers download service account keys as json and use that key to create Kubernetes Secret .Kubernetes offers the Secret resource type to securely mount private files inside Pods at runtime.

Issue with this approach

• Not only is this process burdensome because for every application you need to download SA keys and create secret for pods
• SA keys have long retention by default so you need to rotate the keys.In the case of a breach or compromise, an unaccounted-for key could mean prolonged access for an attacker.

Node’s IAM service account

Each node in a GKE cluster is a Compute Engine instance. Therefore, applications running on a GKE cluster by default attempt to authenticate using the “Compute Engine default service account” or user managed SA and inherit the associated scopes.

Issue with this approach

• Container architectures are dense by design .It means one pod may use attached IAM Service account to access one or two GCP services but other pods may not need to access those services .
• By default all pods have access to same attached IAM service accounts which is not an ideal solutions when you have many micro services (pods) running on GKE.
• It does not obey principle of least privileges

The new Google Recommended way — Workload Identity

Workload Identity is feature which allows workloads (pods) in your GKE clusters to impersonate Identity and Access Management (IAM) service accounts to access Google Cloud services.

Let’s understand first why it’s a recommended way

1. It Simplifies authentication as we no longer need to create and manage application credentials. Instead, we can use IAM service accounts to authenticate to Google Cloud resources.

• By enforcing the principle of least privilege, your workloads can only have the minimum permissions needed to perform their function. Because you don’t need to assign broad permissions (like when using the node service account as discussed above), you reduce the scope of a potential compromise.
• Now by using IAM service account with required permission assigned by IAM roles or policies to access GKE workload ,which reduces risk of secret management and from attackers accessing application credentials .
• Credentials actually issued to the Workload Identity are only valid for a short time, unlike the 10-year lived service account keys, reducing the blast radius in the event of a compromise.

Now let’s see workload Identity in action using hands on .

In this section I will walk through setup of workload identity on any existing or new GKE cluster to access cloud storage from pods running on GKE.

To setup workload identity follow below steps

STEP1: Create a new GKE cluster with workload Identity enabled

gcloud container clusters create sample-cluster  --zone=us-central1-a \
--workload-pool=deft-sight-402505.svc.id.goog \
--machine-type "e2-medium" --num-nodes "2"  --disk-size "100"

Enable Workload Identity on an existing Standard cluster

gcloud container clusters update sample-cluster \
    --zone=us-central1-a  \
    --workload-pool=deft-sight-402505.svc.id.goog

STEP2: Create an IAM service account for your application or use an existing IAM service account instead

# access-gcs -> Google service account 
# deft-sight-402505 -> GCP Project ID

gcloud iam service-accounts create access-gcs \
    --project=deft-sight-402505

STEP3: Assign IAM roles on GCP Service account .I will assign storage admin role for accessing GCS APIS

# Role -> roles/storage.admin
gcloud projects add-iam-policy-binding deft-sight-402505 \
    --member "serviceAccount:access-gcs@deft-sight-402505.iam.gserviceaccount.com" \
    --role "roles/storage.admin"

STEP4: Get credentials for your cluster

gcloud container clusters get-credentials sample-cluster --zone us-central1-a --project deft-sight-402505

STEP5: Create kubernetes namespace

kubectl create namespace app

STEP6: Create Kubernetes service account

kubectl create serviceaccount gcs-access-ksa \
    --namespace app

STEP7: Allow the Kubernetes service account to impersonate the IAM service account by adding an IAM policy binding between the two service accounts. This binding allows the Kubernetes service account to act as the IAM service account.

gcloud iam service-accounts add-iam-policy-binding access-gcs@deft-sight-402505.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:deft-sight-402505.svc.id.goog[app/gcs-access-ksa]"

STEP8: Annotate the Kubernetes service account with the email address of the IAM service account.

kubectl annotate serviceaccount gcs-access-ksa \
    --namespace app \
    iam.gke.io/gcp-service-account=access-gcs@deft-sight-402505.iam.gserviceaccount.com

Update your Pod spec to schedule the workloads on nodes that use Workload Identity and to use the annotated Kubernetes service account.

Note:- Do this if you already have deployment manifest files

spec:
  serviceAccountName: gcs-access-ksa
  nodeSelector:
    iam.gke.io/gke-metadata-server-enabled: "true"

Apply the updated configuration to your cluster.

Note:- Do this if you already have deployment manifest files

# Replace DEPLOYMENT_FILE with the path to the updated Pod spec.
kubectl apply -f DEPLOYMENT_FILE

Test GKE Workload Identity

Make sure you have completed all steps mentioned above in the section enabling workload Identity

Let’s first create one GCS bucket and upload some test files in that bucket using gcloud for next steps.

# create bucket 
gcloud storage buckets create gs://workload-identity-test
# create few files 
touch file1.txt file2.txt file3.txt
#Upload files to bucket 
gsutil cp file1.txt file2.txt file3.txt gs://workload-identity-test
# check if files are uploaded or not
gsutil ls gs://workload-identity-test

Create bucket using gcloud command

Upload and see files in bucket using gsutil

STEP1: Create a POD yaml file using annotated Kubernetes service account and save as test-wi-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: workload-identity-test
  namespace: app
spec:
  containers:
  - image: google/cloud-sdk:slim
    name: workload-identity-test
    command: ["sleep","infinity"]
  serviceAccountName: gcs-access-ksa # change with your service acccount
  nodeSelector:
    iam.gke.io/gke-metadata-server-enabled: "true"

STEP2. Create pod by running below kubectl command in app namespace

kubectl -n app create -f test-wi-pod.yaml

STEP3. Check if pod is successfully created and in running state. It may take few mins for pods to be running for above image.

Kubectl -n app get pods

STEP 4. Now check if created pod in above step has required permission to list objects .

There are multiples ways to test that.

# Run kubectl exec without interactive container terminal
kubectl -n app  exec  workload-identity-test -- gsutil ls gs://workload-identity-test

You can see all files available in bucket

# Get a terminal of pod and exectute gsutil commands to confirm pod access to Google APIS
kubectl -n app  exec -it workload-identity-test -- /bin/sh

# now in that terminal execute gsutil commands as show below in screenshot 

Finally let’s confirm that which service account pod is using to get access Google APIs like gsutil (for Google cloud storage)

If the service accounts are correctly configured, the IAM service account email address is listed as the active (and only) identity. This demonstrates that by default, the Pod acts as the IAM service account’s authority when calling Google Cloud APIs.