GKE Workload Identity: A Secure Way for GKE Applications to Access GCP Services

Photo by Will Myers on Unsplash

At Zeotap, we run our infrastructure in Google Cloud Platform. We have a CI/CD process in place and all our micro-services are deployed in Google Kubernetes Engine.

Many of our microservices communicate with GCP services such as GCS buckets, Bigquery, etc and they need a way to authenticate themselves to Google cloud.

By default, GKE uses the service account attached to the underlying node pools and by default, the nodes in the node pools use compute engine default service account. We can override this option by specifying the custom service account during the cluster creation process.

The compute engine default service account has editor access to the projects, so initially it looked like a no-brainer. However, it was carrying a wide variety of security issues and we decided not to continue with that. (For more context, read Perils of GCP’s Compute Engine default service account).

Authenticating to Google Cloud with service accounts

As GCP recommended, We used the standard approach of using service accounts to authenticate to Google cloud.

• Create service accounts for each application that access GCP services
• Generate a json key for the service account
• Add the json key as a secret to the cluster
• Mount the secret as a volume to the application container
• Set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to the key file in the secret volume mount

# deployment.yaml
spec:
      containers:
      - name: $NAME
        image: $IMAGE
        volumeMounts:
          - name: google-secrets
            mountPath: /opt/google-secrets
        env:
          - name: GOOGLE_APPLICATION_CREDENTIALS
            value: /opt/google-secrets/$SECRET_NAME.json
      volumes:
        - name: google-secrets
          secret:
            secretName: $SECRET_NAME
spec:
      containers:
      - name: $NAME
        image: $IMAGE
        volumeMounts:
          - name: google-secrets
            mountPath: /opt/google-secrets
        env:
          - name: GOOGLE_APPLICATION_CREDENTIALS
            value: /opt/google-secrets/$SECRET_NAME.json
      volumes:
        - name: google-secrets
          secret:
            secretName: $SECRET_NAME

Security concerns in using service account key

There are many problems with this approach of using service accounts directly to authenticate to the Google Cloud.

• Not so secret: Service account (GSA) json keys are critical identity. It contains a ssh key-pair which has a validity of 10 years. Even though this is mounted as a secret, this key is not encrypted and available as a plaintext inside the pod/container. Anybody that can login to the pod/container can read the credentials.
• Anyone can wreak havoc: Anybody that has access to the keys can wreak havoc to the system and we can’t identify who is the user behind the service account keys, as only service account information will appear in the logs.
• Not bound by organisation: People who have hold of these keys can use them even after they leave the company. Given this is a ssh key pair, there is no IAM access control on who can use this. This leads to potential backdoor access to the infrastructure.
• Key rotation/management headache: As a best practice, we have to rotate the service account json keys periodically. Rotating the json key is not a straightforward activity. We have to generate a new key in IAM, add the key to the cluster as a secret, redeploy the app. This is a manual task, and using more keys results in more tasks.
• Application downtime: When we rotate the keys, we have to redeploy the application with new keys every single time. This causes application downtime.

So, using service account json keys in the GKE is a cumbersome one and always carries a security risk and operational management overhead.

GKE Workload Identity

In 2020,Google announced Workload Identity(GA) feature which solves all the problems listed above.

With Workload Identity, we can configure a Kubernetes service account to act as a Google service account. Pods running with the Kubernetes service account will automatically authenticate as the Google service account when accessing Google Cloud APIs.

1) Pre-requisites

Workload identity can be enabled on a new or an existing cluster.

Enable workload identity on an existing cluster

For existing clusters, there are 2 steps:

1. Enable the workload metadata property on the cluster level

2. Create a new node pool and migrate the running apps to the new node pool

or

Update the workload metadata property in the existing node pool itself.

Step 1: To enable Workload Identity on an existing cluster, execute the following command:

gcloud container clusters update cluster-name \
--workload-pool=project-id.svc.id.goog

This command will update the following property of the cluster:

— workload-metadata=GKE_METADATA

Existing node pools are unaffected; new node pools default to this property.

Step 2:

Create a new node pool:

gcloud container node-pools create NODEPOOL_NAME \
  --cluster=CLUSTER_NAME

or

Modify existing node pool to use workload identity:

gcloud container node-pools update NODEPOOL_NAME \
--cluster=cluster-name \
--workload-metadata=GKE_METADATA

Now all the apps will be communicating with GKE metadata server.

Enable workload identity on a new cluster

gcloud container clusters create CLUSTER_NAME \
  --workload-pool=PROJECT_ID.svc.id.goog 

We can also enable this option in GCP console.

2) Create the Google Service Account and Kubernetes Service account

After we create the GSA with required IAM access, we can use a new manifest file to create the kubernetes service account (KSA).

In the annotation, we are specifying the GSA to be acted upon by the KSA.

# serviceAccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    iam.gke.io/gcp-service-account: $GSA-NAME@PROJECT-ID.iam.gserviceaccount.com
  name: $APPNAME
  namespace: $NAMESPACE

Update our deployment.yaml container spec with the KSA information. This tells the pod to use the KSA.

# deployment.yaml
spec:
      containers:
      - name: $NAME
        image: $IMAGE
        serviceAccountName: $KSA-NAME

3) Create KSA — GSA Binding

Create a IAM policy binding which grants KSA ‘iam.workloadIdentityUser’ role to the GSA.

gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:PROJECT-ID.svc.id.goog[$NAMESPACE/$KSA-NAME]" \
  $GSA-NAME@PROJECT-ID.iam.gserviceaccount.com

Please ensure that the GSA has required IAM permissions to the resources accessed by our application.

4) Deploy the application

Once the binding is done, we can deploy the application in GKE.

Although we can use a KSA for multiple applications, it’s better to maintain a separate KSA-GSA for each application (that requires GCP access) for identification and access control.

Summary

As we seen, GKE Workload identity removes the management overhead of storing, rotating service account keys. With no keys involved, this also prevents individuals from gaining access to the system and possible backdoor entry.

This is a safe, secure, clean approach for GKE apps to access GCP services.

Co-Author:

Sajan is a Senior Devops Engineer at Zeotap. He is one of the key players in building CICD pipelines in Zeotap. He can be reached on his Linkedin profile.