Debugging Google Application Default Credentials
Inspecting gcloud
application default credentials, Google access tokens, and ID tokens through the refresh token grant and token introspection.
Today, the following error surfaced while I was running code accessing Google Cloud Platform (GCP) resources from my local computer:
ACCESS_TOKEN_SCOPE_INSUFFICIENT 403 PERMISSION_DENIED
My identity did have permission to access the resource, but somehow, insufficient scopes were associated with my access token. This, I had to see for myself ;-).
The default credentials file
The code that I was running authenticated itself according to Google’s application default credential flow. Consider reading my post on application default credentials if this does not sound familiar to you:
When running locally, the default credentials flow usually ends up reading credentials from your local application default credentials file. On Mac and Linux, you can find this file at ~/.config/gcloud/application_default_credentials.json
. Provided that you have generated default credentials, the file has contents similar to this:
{
"client_id": "764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com",
"client_secret": "d-FL95Q19q7MQmFpd7hHD0Ty",
"quota_project_id": "your-project-id",
"refresh_token": "your-refresh-token",
"type": "authorized_user"
}
Refresh token grant
The application default credentials file contains a refresh token. As far as I know, it is impossible to inspect the scopes associated with Google’s refresh tokens directly. However, refresh tokens can be exchanged for access tokens, and those can be inspected.
Exchanging refresh tokens for access tokens is called the refresh token grant. A single call to Google’s token endpoint suffices to obtain a fresh access token and ID token:
curl --silent --request POST 'https://oauth2.googleapis.com/token' --header 'Content-Type: application/json' --data-raw "$(jq '. | .grant_type = "refresh_token" ' ~/.config/gcloud/application_default_credentials.json)"
The result of this call should look something like this:
{
"access_token": "an_access_token",
"expires_in": 3599,
"scope": "https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/cloud-platform openid https://www.googleapis.com/auth/accounts.reauth",
"token_type": "Bearer",
"id_token": "an_id_token"
}
Sure enough, in my case, the refresh token in the application default credentials file did not include the https://www.googleapis.com/auth/cloud-platform
scope.
Now… I have no clue how this happened, as the CLI usually includes the cloud-platform
scope by default 🤷♂️. Creating new application default credentials resulted in a refresh token that did include the required scope, solving my issue:
gcloud auth application-default login
Although the above sufficed for me today, I have had to investigate Google’s access tokens or ID tokens further. Tokens can originate from default credentials as above or from another source, such as the metadata service or service account credentials.
Inspecting access and ID tokens
You can inspect access and ID tokens by presenting them to Google’s token introspection endpoint.†
† Google’s token endpoint is meant for debugging and is not meant to be used by production resource servers or other internal servers — it is subject to throttling and does not implement the OAuth token introspection spec.
Access tokens
Store your access token in the ACCESS_TOKEN
variable and review it as follows:
curl --silent --request GET "https://oauth2.googleapis.com/tokeninfo?access_token=${ACCESS_TOKEN}"
The result looks similar to this:
{
"azp": "764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com",
"aud": "764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com",
"sub": "your-sub-id",
"scope": "openid https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/accounts.reauth",
"exp": "1664891540",
"expires_in": "3137",
"email": "your-email",
"email_verified": "true",
"access_type": "offline"
}
ID tokens
Similarly, check ID tokens by running:
curl --silent --request GET "https://oauth2.googleapis.com/tokeninfo?id_token=${ID_TOKEN}"
The result looks something like this:
{
"iss": "https://accounts.google.com",
"azp": "764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com",
"aud": "764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com",
"sub": "your-sub-id",
"hd": "your-domain",
"email": "your-email",
"email_verified": "true",
"at_hash": "some-hash",
"iat": "1664887940",
"exp": "1664891540",
"alg": "RS256",
"kid": "...",
"typ": "JWT"
}
Unlike Google’s refresh and access tokens, Google ID tokens are, nowadays, plain JWT tokens. You can, therefore, also inspect them more directly without making a call to the introspection endpoint. The most accessible approach is to paste your token and review its contents at https://jwt.io/.
Alternatively, decode the payload locally:
echo "$ID_TOKEN" | cut -d. -f2 | base64 -d | jq
The output looks similar to:
{
"iss": "https://accounts.google.com",
"azp": "764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com",
"aud": "764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com",
"sub": "your-sub-id",
"hd": "your-domain",
"email": "your-email",
"email_verified": true,
"at_hash": "some-hash",
"iat": 1664887940,
"exp": 1664891540
}