Authentication for Multi-Regional Cloud Run Deployments with Custom Audiences

Cloud Run authentication is a simple and cheerful process. At least until you discover that your ID tokens that you use for authentication should be scoped to a specific Cloud Run service. This scoping is done via the URI of the Cloud Run service. But what if the Cloud Run’s URI is unknown to the client? For example in a scenario where multiple Cloud Run services are deployed behind an external load balancer in different regions for improved resilience. In this case a client cannot know which URI should be used to authenticate against the exposed service. This is exactly where the recently introduced support for custom audience values on ID tokens for Cloud Run come in. In this blog post we will explore this new feature and how to apply it in a multi-regional deployment topology.

Initial Setup

If you want to follow along with the examples described in this post you’ll first need to configure your Google Cloud Project ID and then enable the required services.

export PROJECT_ID=<my-project-id>
gcloud config set project $PROJECT_ID
gcloud services enable run.googleapis.com compute.googleapis.com

We then create two Cloud Run services in two different regions that run the same Cloud Run hello world image. We also extract the URI that is automatically generated for each Cloud Run service in their own variables.

gcloud run deploy hello-us \
--region us-central1 \
--set-env-vars=COLOR=red \
--max-instances 1 \
--no-allow-unauthenticated \
--image us-docker.pkg.dev/cloudrun/container/hello

HELLO_US_URI=$(gcloud run services describe hello-us --region us-central1 --format 'value(status.url)')

gcloud run deploy hello-eu \
--region europe-west1 \
--set-env-vars=COLOR=blue \
--max-instances 1 \
--no-allow-unauthenticated \
--image us-docker.pkg.dev/cloudrun/container/hello

HELLO_EU_URI=$(gcloud run services describe hello-eu --region europe-west1 --format 'value(status.url)')

As we deployed our services with the — no-allow-unauthenticated flag, any invocation requires an ID token for an authorized entity. This can be a user or a service account identity. For testing purposes we can call our cloud run services with the following cURL commands that use the current user’s ID token in the Authorization header:

ID_TOKEN="$(gcloud auth print-identity-token)"

curl $HELLO_US_URI -H "Authorization: Bearer $ID_TOKEN"
curl $HELLO_EU_URI -H "Authorization: Bearer $ID_TOKEN"

Why the ID-token’s audience is important

So technically we could use the same ID token for authenticating against two different Cloud Run services. The reason why the same token worked for both Cloud Run services is within the ID token that we just used. Let’s unpack the token payload to see what is going on.

echo $ID_TOKEN | jq -R 'split(".") | .[1] | @base64d | fromjson'

As per the JWT format this command will split the token string by the “.” marker, extract the second field, base64 decode the payload and load into jq to print it nicely.

{
  "iss": "accounts.google.com",
  "azp": "[...].apps.googleusercontent.com",
  "aud": "[...].apps.googleusercontent.com",
  "sub": "[...]",
  "hd": "my.org.com",
  "email": "user@my.org.com",
  "email_verified": true,
  "at_hash": "[...]",
  "nbf": 1701266801,
  "iat": 1701267101,
  "exp": 1701270701,
  "jti": "[...]"
}

As mentioned in the ID token authentication documentation, the print-identity-token utility generates a token that is valid for all Cloud Run and Cloud Function instances that the authenticated user has access to. This could potentially cause issues if they are leaked in a production use case and used against unrelated services.

For this reason the documentation suggests that identity tokens should be scoped to a specific audience instead. Because we can only use audiences with service accounts we’ll first create a new service account that we then use to invoke our Cloud Run service.

gcloud iam service-accounts create cloud-run-client

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:cloud-run-client@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/run.invoker"

SA_ID_TOKEN_US=$(gcloud auth print-identity-token \
  --impersonate-service-account cloud-run-client@$PROJECT_ID.iam.gserviceaccount.com \
  --audiences "$HELLO_US_URI")

If we again print the payload of our new token:

echo $SA_ID_TOKEN_US | jq -R 'split(".") | .[1] | @base64d | fromjson'

We can see that the audience claim now matches the URI of our hello-us service:

{
  "aud": "https://hello-us-[...]-uc.a.run.app",
  "azp": "[...]",
  "exp": 1701272217,
  "iat": 1701268617,
  "iss": "https://accounts.google.com",
  "sub": "[...]"
}

If we call our Cloud Run services again with the same token on both services only the one where the audience claim matches the service URI is accepted. To call the second Cloud Run service we would have to create a dedicated ID token with the correct audience.

curl $HELLO_US_URI -H "Authorization: Bearer $SA_ID_TOKEN_US" # works
curl $HELLO_EU_URI -H "Authorization: Bearer $SA_ID_TOKEN_US" # error unauthorized

Using custom audiences when the the Service URI is unknown

Having ID tokens scoped to a particular Cloud Run service is effective in preventing the token from being used with other Cloud Run services that the service account is authorized to use but the client application is not intended to.

However in some scenarios a client might not know the Cloud Run service URI because:

• The Cloud Run service is using a custom domain
• The Cloud Run service is deployed behind an regional internal load balancer
• The Cloud Run service is deployed behind a global external load balancer (most likely in multiple regions)

The mechanics for using an alternative audience for authenticating the Cloud Run service is the same across all of these options. In this blog we’ll focus on the third option and show how an ID token with a scoped audience can be used to transparently access multiple Cloud Run instances behind a global external load balancer. The full network architecture looks as shown in the diagram below.

To create all the necessary components you can run the following code snippet. This will create the following resources in your project:

• 1 External IP
• 1 Google Managed certificate by encoding the external IP in a IP-ADDRESS.nip.io format
• 2 Serverless Network Endpoint Groups (NEG) for each region
• 1 Backend Service that contains both NEGs
• 1 Target HTTPS proxy
• 1 Forwarding rule as the frontend for the Load Balancer

gcloud compute addresses create external-cloud-run \
    --network-tier=PREMIUM \
    --ip-version=IPV4 \
    --global

gcloud compute network-endpoint-groups create neg-hello-us \
  --region=us-central1 \
  --network-endpoint-type=serverless  \
    --cloud-run-service=hello-us

gcloud compute network-endpoint-groups create neg-hello-eu \
  --region=europe-west1 \
  --network-endpoint-type=serverless  \
    --cloud-run-service=hello-eu

gcloud compute backend-services create hello-backend \
  --load-balancing-scheme=EXTERNAL_MANAGED \
  --global
   
gcloud compute backend-services add-backend hello-backend \
  --global \
  --network-endpoint-group=neg-hello-us \
  --network-endpoint-group-region=us-central1

gcloud compute backend-services add-backend hello-backend \
  --global \
  --network-endpoint-group=neg-hello-eu \
  --network-endpoint-group-region=europe-west1

gcloud compute url-maps create external-hello \
  --default-service hello-backend

EXTERNAL_IP=$(gcloud compute addresses describe external-cloud-run --format="get(address)" --global)

NIP_DOMAIN="${EXTERNAL_IP//./-}.nip.io" 

gcloud compute ssl-certificates create hello-cert \
       --domains "$NIP_DOMAIN,eu.$NIP_DOMAIN,us.$NIP_DOMAIN"

gcloud compute target-https-proxies create external-hello-https \
  --ssl-certificates=hello-cert \
  --url-map=external-hello
   
gcloud compute forwarding-rules create hello-https \
  --load-balancing-scheme=EXTERNAL_MANAGED \
  --network-tier=PREMIUM \
  --address=external-cloud-run \
  --target-https-proxy=external-hello-https \
  --global \
  --ports=443

With everything in place we can still call our application with the unscoped identity token of the user

curl https://$NIP_DOMAIN -H "Authorization: Bearer $(gcloud auth print-identity-token)"

But if we wanted to use the domain that points to our external load balancer as the audience of the identity token our request fails as unauthorized:

SA_ID_TOKEN_EXT=$(gcloud auth print-identity-token \
  --impersonate-service-account cloud-run-client@$PROJECT_ID.iam.gserviceaccount.com \
  --audiences "$NIP_DOMAIN")

curl https://$NIP_DOMAIN -H "Authorization: Bearer $SA_ID_TOKEN_EXT"

Luckily the new custom audiences feature of Cloud Run allows us to add custom audiences for a Cloud Run service.

gcloud run services update hello-us --region us-central1 \
  --add-custom-audiences=$NIP_DOMAIN
gcloud run services update hello-eu --region europe-west1 \
  --add-custom-audiences=$NIP_DOMAIN

Now that we have a common custom audience for both Cloud Run services behind our load balancer we can use the previous commands again and use the domain pointing to our external load balancer as the audience of our identity token.

SA_ID_TOKEN_EXT=$(gcloud auth print-identity-token \
  --impersonate-service-account cloud-run-client@$PROJECT_ID.iam.gserviceaccount.com \
  --audiences "$NIP_DOMAIN")

curl https://$NIP_DOMAIN -H "Authorization: Bearer $SA_ID_TOKEN_EXT"

(Optional) Homework Exercise

If you made it up to this point in the post you might be interested in getting your hands dirty yourself. An interesting extension of the scenario in this post could be to add the following configuration.

In addition to the a-b-c-d.nip.io route that takes an ID token with the domain as outlined above you might want to add the following routes and config to your existing external load balancer and Cloud Run Services.

• us.a-b-c-d.nip.io that routes all traffic to the hello-us Cloud Run Service only and accepts a us-scoped audience on the ID token
• eu.a-b-c-d.nip.io that routes all traffic to the hello-us Cloud Run Service only and accepts an eu-scoped audience on the ID token