Local/Remote Authentication with Google Cloud Platform

Authors: Theodore Siu, Daniel De Leo

Update (2021–03–24): In consideration for security, especially for local development and direct user facing computers, using service account JSONS may not be the safest. Articles such as the following advocate similar stances. While we still recommend service account JSONS with GOOGLE_APPLICATION_CREDENTIALS for running code in production non-Google Cloud type environments, we recommend using gcloud auth application-default login for local development. We have removed references to service accounts on “local machines” to reflect this change.

Local machine access to GCP resources maybe blocked by improper authentication

Setting up your Google Cloud Platform authentication on your local or remote machine can be a confusing task. When/How should you be using a service account vs a user account? How do you handle multiple projects? Improper authentication can also lead to permission denied errors. Here is an example of a thrown error:

"Permission 'cloudkms.cryptoKeyVersions.useToEncrypt' denied for resource 'projects/myproject /locations/global/keyRings/myRing/cryptoKeys/myKey'.",
"status": "PERMISSION_DENIED"

Google Cloud Platform provides the following ways of authenticating from your local machine:

1. Creating a service account, downloading a corresponding JSON credentials file and setting the environmental variable of JSON file path to GOOGLE_APPLICATION_CREDENTIALS.
2. Using gcloud auth application-default login to authenticate with a user identity (via a web flow) but using the credentials as a proxy for a service account.
3. Running the gcloud auth login command to authenticate with a user identity (via web flow) which then authorizes gcloud and other SDK tools to access Google Cloud Platform. Note that this auth command is also one of the suite of commands run when gcloud init is run.
4. Using GCP service specific methods such as generating Private/Public key pairs for use with Core IoT Auth when your local machine is an embedded device. These service specific examples are out of scope for this article.

Using Service Accounts

Service accounts in GCP should be used when programmatically accessing GCP resources (ie: from a script, app using google.cloud libraries, hitting a GCP API etc..). If you yourself are accessing GCP, don’t use a service account, instead authenticate with your own Google user identity! Creating and using a service account to authenticate on your local machine can be done by executing the following steps:

Using the GCP UI

1. Logging into your GCP console https://console.cloud.google.com and clicking on the side bar for API's and Services >> Credentials.
2. You will be brought to a screen where you can then click on Create Credentials >> Service Account Key. Here you can create a new service account (remember to attach appropriate IAM roles) and download the JSON key file representing the credentials.

Using gcloud command line tool

1. Assuming you have authenticated as a user using gcloud auth login . You can then set your project using gcloud config set project my-project .
2. Create your service account using the command gcloud iam service-accounts create [SA-NAME] --display-name "[SA-DISPLAY-NAME]" where SA-NAME is the name of your service account and SA-DISPLAY-NAME is the display friendly name. Appropriate IAM roles should also be assigned at this time to the service account.
3. Download the JSON key file representing the credentials gcloud iam service-accounts keys create ~/key.json
--iam-account [SA-NAME]@[PROJECT-ID].iam.gserviceaccount.com

Once the key has been downloaded set your environmental variable GOOGLE_APPLICATION_CREDENTIALS to the path of your JSON by running the export command. For example:

export GOOGLE_APPLICATION_CREDENTIALS=path/to/service_acct.json

To set your credentials more permanently, consider adding this command to your startup/bootstrapping script if you are running a VM. The following is an example done in BASH.

echo 'export GOOGLE_APPLICATION_CREDENTIALS = your/path/to/service_account_cred.json' >> ~/.bash_profile

Using a service account and setting the environmental variable GOOGLE_APPLICATION_CREDENTIALS takes the highest precedence on gcloud over all other methodologies. Update (2021–03–24): In consideration for security, especially for local development and direct user facing computers, using service account JSONS may not be the safest. While we still recommend service account JSONS with GOOGLE_APPLICATION_CREDENTIALS for running code in production type environments, we recommend using gcloud auth application-default login for local development.

Using gcloud auth application-default login

The gcloud auth application-default login command creates a credentials JSON named application_default_credentials.json embedded within the environment under a .config/ directory. This approach is very similar to the service account method above except instead of manually configuring a service account and downloading a JSON file of credentials, the user authenticates and the credentials attached to the user are automatically generated. This is not the recommended method as it takes lower precedence than setting GOOGLE_APPLICATION_CREDENTIALS . Update (2021–03–24): In consideration for security for local machines this is the preferred method of authentication.

Authenticating as a User

Running gcloud auth login or following the gcloud init login flow authenticates you as a user- but how is this different from the above two methods which use service accounts?

1. Authenticating as a user should only be done when doing administrative tasks as a user (ie: setting up service accounts, manually running gsutil commands), not when programmatically accessing GCP such as in a Python or Go script.
2. Unlike using the service account methods, authentication is not done via a JSON file but instead is done through tokens.
3. By using gcloud auth login or gcloud init, gcloud, gsutil and bq commands begin running commands as a user account. It is also possible to use a service account with all of these tools. A general guide for both methods of authenticating with these command line tools can be found here.

What Takes Precedence?

1. If running a gcloud, gsutil, and bq command line tools without authentication as a service account (this is the default when running gcloud auth login or gcloud init), the user account will take precedence.
2. If programmatically using GCP resources (ie: in a script with google.cloud libraries), a service account will be invoked and the environmental variable GOOGLE_APPLICATION_CREDENTIALS takes precedence over all other methodologies.
3. In the service account case, if GOOGLE_APPLICATION_CREDENTIALS is not found, secondary application-default credentials will be invoked depending on operating system or the Google Service used. For example, on a Windows OS, gcloud looks for %APPDATA%/gcloud/application_default_credentials.json while on a non-Windows OS, gcloud looks for $HOME/.config/gcloud/application_default_credentials.json. See this link for searching for where your default credentials are listed.
4. On a GCP resource such as App Engine, Compute Engine or Google Kubernetes Engine a metadata database is queried to grab the service’s own credentials.

If none of the conditions are met, GCP fails to authenticate and one will run into permission errors.

Conclusion

Now that we have validated that the correct entity has been authenticated with Google Cloud Platform, are you still running into the same permission issues? If so, stay tuned for part 2 where we discuss GCP service permissions¹.

1. As of publication of this article in April 2019 there is a known IAM policy bug which occurs when deleting a service account and recreating a service account with the same name. The IAM policy binding of the deleted service account is not immediately removed upon deletion. It is required to to manually remove then re-add the service account’s policy binding to properly grant the IAM roles to the service account.