HOW TO: Access GCP Services from Looker’s Extension Framework

As a software engineering consultant at 4 Mile Analytics, a data and analytics consultancy, one of my core roles involves enabling clients to extend their data products by developing tools and experiences that reveal insights and unlock the ability to take effective action on them. One of the ways we do this is by maintaining libraries of patterns that accelerate development. If these patterns are tied to project infrastructure, we do our best to automate their configuration and deployment using widely adopted Devops and CI/CD tools.

Consultants at 4 Mile are Looker experts, and we use Looker as the data backend for many of our embedded analytics projects. We often leverage the extension framework, which allows for building customized UI that leverages Looker’s Auth and Security Model, while also easily wiring up the connection to the Looker API via its JS SDK. It comes with a lot of great features out of the box, but sometimes we need to securely connect to other third-party services outside of the Looker extension framework.

For example, a common connection we need to make is to BigQuery. Looker natively reads from data warehouses like BigQuery. However, we need to create an API to manage authentication and authorization when we have a use case that includes writing to BigQuery, like training BQML models.

Spinning up an access token server that grants the proper level of privilege is a task we often do to address this problem. Here are a few other examples of use cases that we have found this type of solution useful for.

Connecting to…

• a middleware that links to a database for maintaining application state.
• an asset server for serving things like map visualization image tiles.
• a CMS server that allows for pulling in content added by administrators.

In this tutorial I’m going to walk you through how we leverage Terraform, Container Registry, Github, and Cloud Build to automate spinning up a simple access token service on Cloud Run, with a lightweight CI/CD pipeline.

Solution Overview

Here are the steps we will walk through in this tutorial:

1. Code walkthrough of the access token server
2. Adding service accounts and IAM permissions to your GCP project.
3. Connecting your project’s Github Repo to Cloud Build
4. Creating and securing Terraform service account credentials
5. Terraform configuration walkthrough
6. Applying Terraform

Here is the source code for the token service, including the Terraform IAC configurations, which you will want to copy or fork:
Looker GCP Access Token Service

Below is an architectural diagram outlining the topology of the final solution:

Step 1: Access Token Server Walkthrough

The purpose of this access token server is to provide a Looker Extension Framework Application front-end with an access token that will allow it to make requests to GCP services like BigQuery with the privileges of a service account that we’ve created in GCP.

The general idea behind the access token server is that the /access_token endpoint accepts as parameters:

• Looker API credentials created for a designated API user, for the Looker instance hosting the Extension Framework Application.
• Scopes for API services that the token server’s service account will be able to access. In our case, GCP’s BigQuery API.

See: access_token request handler

The service then validates the Looker API credentials created against the Looker instance that the service has been configured to point to, i.e. the instance hosting the extension framework application.

See: validate Looker credentials

If credentials passed are valid, our service then requests an access token from the GCP service account associated with our service (which you will bind in a future step) using the scopes passed through. For the service account to successfully return a token, its IAM bindings must match up with the api scope passed through. In our case the BigQuery service scope (https://www.googleapis.com/auth/bigquery)

See: get access token

Once you have this token, it is returned in the body of the access token service’s response.

See: return access token

This sequence diagram outlines the process:

Step 2: Adding service accounts and permissions to GCP

To get started, you will need to create the proper service accounts needed to enable Terraform to build our infrastructure, allowing cloudbuild to build and deploy code updates, and grant our token server the necessary permissions to access BigQuery.

Theoretically you could give Terraform the ability to create our token server’s service account, and grant the appropriate IAM permissions, however this would give any developers with write access to our Terraform file’s Github repo the keys to an entire GCP project, so following the rule of least privilege, you will manually create these service accounts with the least amount of access to do their jobs, and bind them to the services created by Terraform.

First, you will need access to a GCP project that you are the owner of, or at least have the ability to manage roles and resources within.

• Click on the Project Menu and select the GCP project where you want to provision the token server.

Next you will create 2 service accounts in the console.

• Go to Service Accounts and create the service accounts.

The service accounts will have the following names and ids

• terraform-automation
• looker-gcp-auth-provider

After adding the names and ids, skip all the other options and create the service accounts.

Next, enable the Cloud Build service manually, which will create the Cloud Build service account that you will bind roles to. Keep in mind that Cloud Build is a paid service requiring a billing account. For the purposes of this tutorial, monthly costs will be negligible, especially if you don’t intend to productionalize the project.

• Go to Cloud Build and enable the API

Now that you have your 3 service accounts created. Before you leave the service account list view, make sure you copy down the user names of each one as they will be needed in the next step.

You now need to assign them IAM roles, which you will also do in the console.

• Go to IAM & Admin to add the proper roles

You will assign the Terraform service account and the Cloud Build service account the following roles, as they work in tandem to build out your resources :

• Cloud Build Editor
• Cloud Run Admin
• Organization Administrator
• Service Account User
• Service Usage Admin
• Storage Admin
• View Service Accounts
• Secret Manager Viewer

• Repeat until all the roles above are added, and then save.

Your token server’s service account, looker-gcp-auth-provider, will only need the following roles:

• BigQuery Data Editor
• Cloud Run Developer

Step 3: Connecting a Github Repo to CloudBuild

Ultimately Terraform will provision our Cloud Build Github trigger, which will fire when code commits are pushed to the main branch of your project. However, again for security, you as project admin will want to use the console to authenticate Cloud Build to read from your project’s Github repository. You will need to create either an empty repo, or copy/fork our open source project before these next few steps.

• First, Go to Cloud Build to expose your repo.

• Then, follow the steps to connect to your project’s repository.
• You will need to manage repositories to add your project to Cloud Build.

• Then in Cloud Build you can connect the repository

Step 4: Creating and securing Terraform Credentials

There are a couple more manual configurations you need to make before you can enable Terraform to provision your resources.

First, you need to create credentials allowing Terraform to securely connect to your project APIs to make changes. And these credentials need to be securely stored.

Create the keys from the service account details page which you can get to by selecting the Terraform service account from the list.

• Go to Service Accounts to begin creating terraform-automation service account creds.

Store the credentials file that is created somewhere secure for use in the next step, which you will need a little context to complete.

Terraform will be applying configuration changes during a cloud build run that will be triggered by code commits to your project’s github repo. Cloud Build will need to be able to access your Terraform credentials so it can connect to GCP resource APIs and make changes. We will cover how this is done when you review the Terraform config, but this is important to know for two reasons.

1. The first time Terraform runs, the Cloud Build job won’t exist yet (because Terraform hasn’t created it) therefore you will need to apply the changes manually using your local Terraform cli.
2. You will need to store the Terraform creds you just created in a secret for Cloud Build to access during its run.

To take care of issue 1, you simply store your Terraform creds securely locally, and make sure that when you add them to your version of the access token server project, that you make sure not to commit them. I named them terraform_credentials.json and that filename is in the source project’s gitignore file.

To take care of issue 2, we will securely store our credentials in Secret Manager.

• Go to Secret Manager and enable it.

• Next you will create a secret in Secret Manager containing the Terraform service credentials.

• Use the name terraform-service-creds and upload the credentials file as the contents, then click CREATE SECRET.
• You will need to give your cloud build service account the Secret Manager Manager Secret Accessor permission for the new secret to be able to actually view it.

• Click on terraform-service-creds and keep track of the full name of the secret as well, as this will need to be included in your Terraform configuration.

In fact here is everything you will need to configure Terraform for future steps :

• Project name (id)
• Project number
• The name of your Terraform credentials secret
• The url of your Looker instance
• The name of your token service account
• The name of the Github account and repo where your token service will be stored

Make sure you have all of this information handy before you proceed to the next steps.

Step 5: Terraform Configuration Walkthrough

Our project’s Terraform configuration files are consolidated into a single directory:
Terraform Files

You will find step-by-step descriptions or comments in each of these files so that you can read through each of them in order to get a sense of what they do.

terraform.tf :
This file contains the variable definitions used in our Terraform config. Read through the variable descriptions as a reference for their use.

terraform.tfvars.sample :
This file contains a sample of the configuration variables that you will set, based on the values you saved in step 4.

• You will create your own file called terraform.tfvars and add your project values. If you have questions about any of these variables, refer to the previous file terraform.tf for reference.

main.tf :
This file is where the bulk of the Terraform instructions reside. Reading the comments from top to bottom will give you a good sense of what resources are being provisioned, and in what order.

Please note that after Terraform applies locally, but before the project is pushed to Github, that you will need to remove the comments regarding creating the Cloud Run service.

outputs.tf :
This file defines what Terraform will output at the end of a successful run. In this case it only logs the public url of our token service, which your Looker extension framework application will use to request an access token.

Step 6: Apply Terraform

• First things first, you will need to have the Terraform cli tool installed on your local machine:
  Terraform Installation Instructions

Assuming you’ve completed all of the previous steps, from the command line, from your project’s Terraform directory, you should be able to run the following commands.

First, initialize Terraform:

terraform init

Next, run the plan command to surface any pre-flight issues:

terraform plan

Finally, run the apply command to provision all the needed resources for our token service (minus the Cloud Run service itself).

terraform apply

Assuming that this all goes smoothly, your project should have resources spun up at this point.

• For the final steps of deploying this service. You will need to remove the comments to expose the Cloud Run service:
  Cloud Run Provisioning
• Commit this change you’ve made to the project, and push to the main branch of the Github repo you are using to host the token service.

This should trigger your Cloud Build job!

• Go to Cloud Build in GCP to find your job:

Once the job is completed successfully, you should be able to click into the build, view the last step, and find your token service’s url at the bottom of the output window.

This URL will remain stable for subsequent version updates of your application, however you could extend this project and map it to a custom cloud domain name.

Step 7: Test your new Access Token Service

Now that you have your new access token service url, you should be able to test requesting a BigQuery token using Postman or a cURL command.

Here is an example of the request in Postman:

Some things to note about this request:

• base_url is the service url you just copied from the cloud build output.
• client _key and client_secret are Looker API credentials, which you can generate for a specific user using these instructions:
  Generating API Keys
• Notice that the scope parameter in the body of the request is set to the Oauth 2.0 scope for BigQuery view and manage permissions, which is the level of permissions you’ve given to our service account. If our service account didn’t have this level of permission, our access token request would fail. If you wanted to modify this example to request access to a different GCP service you could grant the service account access to that service type, and change the scope as well.

If this request succeeds, you should see an access token that can then be used in header requests to make updates directly to BigQuery.

And there you have it!

Next Steps: Secure Access Token Requests in the Looker Extension Framework

Now that you have the access token server setup, you can utilize it in your Looker Extension Framework.

Here is a JS example of an access server token request that could be run from an Extension Framework app. This is coming from the docs for Looker’s Open Source version of a similar access token server:
Extension Framework Token Request

A few things to call out about this example:

• Notice this use of the extensionSDK.createSecretKey function in conjunction with the extensionSDK.serverProxy function. These two functions work together to conceal things like API credentials from the browser when requesting access tokens.
• serverProxy runs an API call on Looker’s servers, and returns the result of the request to the browser. In this case, you make a token request to your access token server, and then get the result back securely, without our API keys being exposed to the user’s browser. Please read the section in these docs under Server Proxy
• createSecretKey allows for the serverProxy function to reference the values of a user’s user_attribute values. In our case the client key and secret for Looker API access. Please read the section in these docs under Secret Keys
• The ACCESS_SERVER_URL is equivalent to our service url, the one that you grabbed from the Cloud Build output in step 6.
• The scope in the example shared is for connecting with Google Sheets. You will need to change the scope to https://www.googleapis.com/auth/bigquery for use with our example.

Once you have an access token from the serverProxy call, you can then include it in header requests to your BigQuery instance directly. It is highly recommended that when making API calls to third parties from an Extension Framework Application that you use the extensionSDK.fetchProxy function. Please read the section in these docs under Fetch Proxy for more details.

Summary:

In this tutorial we covered all that you would need to establish a secure connection to GCP services like BigQuery from Looker, as well as automating the provisioning of the required resources using Terraform. We hope that you can use this as a starting point for configuring your own custom use cases using the Looker Extension Framework, and if you have a project that you’d love some guidance with, please don’t hesitate to reach out!