A Better Way to Use Google Cloud from Colab

Photo by Pablo Arroyo on Unsplash

One of the best things about Colab is low friction. You open a notebook and Colab knows who you are based on your current Google account — there’s no need to separately login to Colab (unless you’re not already logged into Google).

Problem: Google Cloud provides an amazingly powerful array of services, which work great in a Colab notebook, but you need to separately authenticate yourself to Google Cloud. This is not difficult but, till now, many developers resorted to using a service account, which adds a bit of complexity and, even worse, can lead to accidentally checking service account credentials into a repo, which is a serious security concern.

Solution: We simplified the flow for you so that you can now use your Google Cloud project from a Colab notebook in a more straightforward way and hopefully without thinking about creating a service account.

How does it work? In order to understand what’s new, let’s do a before-after comparison. Here is what can be found in some notebooks using a service account and downloading a private key:

# BEFORE
import os
from google.colab import auth
auth.authenticate_user()

PROJECT_ID = "YOUR_PROJECT_ID"
SA_NAME = "YOUR_SERVICE_ACCOUNT_NAME"
SA_EMAIL = f"{SA_NAME}@{PROJECT_ID}.iam.gserviceaccount.com"
!gcloud config set project $PROJECT_ID
!gcloud iam service-accounts create $SA_NAME
!gcloud iam service-accounts keys create ./key.json --iam-account $SA_EMAIL
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "./key.json"

This is not necessary and there are two potential problems here:

• A service account is created. If you’re not the owner of the project, you may not have the permissions for this.
• Service account credentials are manually downloaded at source code level. The private key can accidentally be pushed and made public.

And here’s the simpler, service-account-less method:

# AFTER
from google.colab import auth
PROJECT_ID = "YOUR_PROJECT_ID"
auth.authenticate_user(project_id=PROJECT_ID)

• You are authenticated for gcloud CLI commands (like before).
• You are also authenticated when using Google Cloud Python Client Libraries.
• Your default project is set.

The new version is shorter, simpler, and less error-prone.

If you’re new to Colab, would like to get more details or example notebooks, check out “Using Google Cloud from Colab”.