Local/Remote Authentication with Google Cloud Platform
Setting up your Google Cloud Platform authentication on your local or remote machine can be a confusing task. When/How should you be using a service account vs a user account? How do you handle multiple projects? Improper authentication can also lead to permission denied errors. Here is an example of a thrown error:
"Permission 'cloudkms.cryptoKeyVersions.useToEncrypt' denied for resource 'projects/myproject /locations/global/keyRings/myRing/cryptoKeys/myKey'.",
Google Cloud Platform provides the following ways of authenticating from your local machine:
- Creating a service account, downloading a corresponding JSON credentials file and setting the environmental variable of JSON file path to
gcloud auth application-default loginto authenticate with a user identity (via a web flow) but using the credentials as a proxy for a service account.
- Running the
gcloud auth logincommand to authenticate with a user identity (via web flow) which then authorizes gcloud and other SDK tools to access Google Cloud Platform. Note that this auth command is also one of the suite of commands run when
gcloud initis run.
- Using GCP service specific methods such as generating Private/Public key pairs for use with Core IoT Auth when your local machine is an embedded device. These service specific examples are out of scope for this article.
Using Service Accounts
Service accounts in GCP should be used when programmatically accessing GCP resources (ie: from a script, app using google.cloud libraries, hitting a GCP API etc..). If you yourself are accessing GCP, don’t use a service account, instead authenticate with your own Google user identity! Creating and using a service account to authenticate on your local machine can be done by executing the following steps:
Using the GCP UI
- Logging into your GCP console
https://console.cloud.google.comand clicking on the side bar for
API's and Services >> Credentials.
- You will be brought to a screen where you can then click on
Create Credentials >> Service Account Key. Here you can create a new service account (remember to attach appropriate IAM roles) and download the JSON key file representing the credentials.
Using gcloud command line tool
- Assuming you have authenticated as a user using
gcloud auth login. You can then set your project using
gcloud config set project my-project.
- Create your service account using the command
gcloud iam service-accounts create [SA-NAME] --display-name "[SA-DISPLAY-NAME]"where
SA-NAMEis the name of your service account and
SA-DISPLAY-NAMEis the display friendly name. Appropriate IAM roles should also be assigned at this time to the service account.
- Download the JSON key file representing the credentials
gcloud iam service-accounts keys create ~/key.json
Once the key has been downloaded set your environmental variable
GOOGLE_APPLICATION_CREDENTIALS to the path of your JSON by running the export command. For example:
export GOOGLE_APPLICATION_CREDENTIALS =
To set your credentials more permanently, consider adding this command to your startup/bootstrapping script if you are running a VM. The following is an example done in BASH.
echo 'export GOOGLE_APPLICATION_CREDENTIALS = your/path/to/service_account_cred.json' >> ~/.bash_profile
Using a service account and setting the environmental variable GOOGLE_APPLICATION_CREDENTIALS is the recommended method of service account authentication because it takes the highest precedence on gcloud over all other methodologies.
Using gcloud auth application-default login
gcloud auth application-default login command creates a credentials JSON named
application_default_credentials.json embedded within the environment under a
.config/ directory. This approach is very similar to the service account method above except instead of manually configuring a service account and downloading a JSON file of credentials, the user authenticates and the credentials attached to the user are automatically generated. This is not the recommended method as it takes lower precedence than setting
Authenticating as a User
gcloud auth login or following the
gcloud init login flow authenticates you as a user- but how is this different from the above two methods which use service accounts?
- Authenticating as a user should only be done when doing administrative tasks as a user (ie: setting up service accounts, manually running gsutil commands), not when programmatically accessing GCP such as in a Python or Go script.
- Unlike using the service account methods, authentication is not done via a JSON file but instead is done through tokens.
- By using
gcloud auth loginor
gcloud init, gcloud, gsutil and bq commands begin running commands as a user account. It is also possible to use a service account with all of these tools. A general guide for both methods of authenticating with these command line tools can be found here.
What Takes Precedence?
- If running a gcloud, gsutil, and bq command line tools without authentication as a service account (this is the default when running
gcloud auth loginor
gcloud init), the user account will take precedence.
- If programmatically using GCP resources (ie: in a script with google.cloud libraries), a service account will be invoked and the environmental variable
GOOGLE_APPLICATION_CREDENTIALStakes precedence over all other methodologies.
- In the service account case, if
GOOGLE_APPLICATION_CREDENTIALSis not found, secondary application-default credentials will be invoked depending on operating system or the Google Service used. For example, on a Windows OS, gcloud looks for
%APPDATA%/gcloud/application_default_credentials.jsonwhile on a non-Windows OS, gcloud looks for
$HOME/.config/gcloud/application_default_credentials.json. See this link for searching for where your default credentials are listed.
- On a GCP resource such as App Engine, Compute Engine or Google Kubernetes Engine a metadata database is queried to grab the service’s own credentials.
If none of the conditions are met, GCP fails to authenticate and one will run into permission errors.
Now that we have validated that the correct entity has been authenticated with Google Cloud Platform, are you still running into the same permission issues? If so, stay tuned for part 2 where we discuss GCP service permissions¹.
- As of publication of this article in April 2019 there is a known IAM policy bug which occurs when deleting a service account and recreating a service account with the same name. The IAM policy binding of the deleted service account is not immediately removed upon deletion. It is required to to manually remove then re-add the service account’s policy binding to properly grant the IAM roles to the service account.