Obtaining access tokens from the Google Identity service

Three methods for obtaining GCP access tokens

Using user credentials, service account credentials or the metadata service to obtain access tokens from Google’s identity service, for use on the GCP platform.

Photo by Florian Berger on Unsplash

Applications interacting with GCP resources usually obtain access tokens through Application Default Credentials (ADC). ADC start by looking for a credential file with long-lived credentials on your file system, for example in the file pointed at by the GOOGLE_APPLICATION_CREDENTIALS environment variable variable. They then exchange these long-lived credentials for access tokens. ADC may also directly obtain an access token from the metadata service.

Access tokens are the short-lived bearer tokens granting you access to the GCP APIs. This story takes a closer look at the different ways for obtaining access tokens. It looks at the structure of credentials files, and how to exchange them. It also looks at how to retrieve access tokens from the metadata service if you’re on the GCP environment.

This story takes a closer look at the different ways for obtaining GCP access tokens.

For information on ADC, check out my other story:

Authentication on GCP: Application Default Credentials

Exchanging authorized user credentials

There are two type of Google Accounts: user accounts and service accounts. These account types come with different types of credentials: user credentials and service account credentials. User credential files contain secrets allowing you to authenticate with Google’s identity service as a user, in a way similar to what happens when you access GCP through the Console.

There are two types of credentials: user credentials and service account credentials

The secret in user credential files takes the form of a long-lived refresh token that you can use to obtain access tokens using the standard OAuth 2.0 refresh token grant. You usually obtain user credential files by through the gcloud auth application-default login command, which initiates an OAuth 2.0 authorization code grant (with PKCE) and produces a credential file called application_default_credentials.json in your gcloud configuration directory, usually located at ~/.config/gcloud. You can also obtain refresh tokens in other ways, including by registering your own OAuth application or by extracting them from your gcloud credentials database (check out this story to find out how).

User credential files contain refresh tokens that can be used to obtain access tokens using the refresh token grant

User credential files generated through gcloud look as follows:

The client_id and client_secret in this file are the ID and secret of gcloud, and you can see that they’re not all that secret. This is not a problem, as gcloud is not, in OAuth terms, a confidential client.

Exchanging refresh tokens for access tokens is very easy. The refresh token grant only requires a single call to one of Google’s OAuth token endpoints. One way is to POST an application/json request as follows:

• There’s quite many token endpoints: versioned ones, like https://www.googleapis.com/oauth2/v4/token, and the one at https://oauth2.googleapis.com/token that I use here.

The response is an access token and an ID token:

• You can modify the access token scopes by specifying scopes in your request. You can scope down, but cannot increase beyond the scopes of the original refresh token.
• Remember that scopes determine how much of your user’s authority is delegated to access token bearers; scopes can only restrict what your IAM user is allowed to do. Check out the scopes used by Google services here.
• Per the OIDC spec, if you leave out the openid scope, you will not get the id_token as part of the response.
• The scopes shown here are the default ones. The scope above ending on cloud-platform gives you access to pretty much all GCP resources, but not other Google services such as Gmail or Google Drive.
• If you want a refresh token that includes custom scopes, you can specify these when you log in using gcloud auth login application-default --scopes.

Exchanging service account credential files

Service account credential files are JSON files containing long-lived private keys, which you can generate from the GCP Cloud Console or using the gcloud CLI. These keys can be used to obtain access tokens using the OAuth 2.0 assertion grant with JWT assertions.

You may think of the assertion grant as similar to the password grant, except that instead of transmitting keys (~password), the private key is used to sign an assertion which is transmitted. The actual keys never leave your hard disk. The assertion grant is more involved than the refresh token grant, so I’ll not demonstrate it further here. You can find a nice barebones example implementation here.

Service account credential files contain private keys, which can be used to obtain access tokens through the assertion grant

Service account credential files look as follows:

• Service account credential files specify a project, but user credential files do not. Consequently, when using user credential files the project ID often becomes a mandatory parameter to the ADC.

Obtaining access tokens through the metadata service

The metadata service’s token endpoint is the basis of how “magic” authentication happens when your run applications on GCP. This is the case on GCE, but also on Cloud Run, Cloud Build and most other GCP services.

Retrieving an access token from the metadata service is really easy. From pretty much any application running on GCP, you can retrieve a token from the metadata service by issuing a GET request to http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token. The best way to see this for yourself is to run

curl http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token

from Cloud Shell, a script you submit to Cloud Build or a GCE instance!

• You can additionally scope the token by specifying a scopes query parameter.
• The metadata service has a fixed IP address, 169.254.169.254 . The same IP address is used worldwide. You can expect this address to not change.
• It is possible to specify a non-default hostname or IP address for the GCE metadata server by setting the GCE_METADATA_HOST variable.
• The hostname of the GCE metadata server is pre-programmed as metadata.google.internal in some of Google’s ADC implementations. This hostname will resolve from GCP’s internal DNS servers but will not resolve anywhere else. If you are using a custom DNS server, you may want to make sure this address resolves or override the address that’s contacted by setting the environment variable GCE_METADATA_HOST to 169.254.169.254 .
• One way to get the ADC to work with only external configuration could be to reroute calls for tokens to 169.254.169.254, and to provide valid access tokens in response. This is the idea behind the GCE Metadata Server Emulator that is described in this post by Rashid Salmaan and similar in idea to kube2iam (for AWS) and kube-google-iam. You could use this to get ADC to work within Docker containers, but there’s easier ways.
• When changing the value of GCE_METADATA_HOST you may also have to change the value of the variable GCE_METADATA_IP. It looks like the Python libraries, at least, use this value (or 169.254.169.254 if this environment variable is not set) to check if the metadata service is available, before they use the value GCE_METADATA_HOST(or its default value metadata.google.internal) to make actual requests.

That’s it! This story has demonstrated three ways to obtain access tokens for use with GCP. They’re all used as part of the ADC. If you spotted any errors or have remarks, be sure to let me know in a comment!

I work at Data Minded, an independent Belgian data engineering consultancy. Contact us if you’re interested in working with us!