Terraform Cloud/Enterprise and GCP Workload Identity Federation

Note: this article has been updated, and the initial article was based primarily on integration of GCP Workload Identity and Terraform Cloud OpenID Connect (OIDC) integration. Today there is a cleaner way called Terraform Cloud Dynamic Provider Credentials, which is the same under the hood, but more automated on TFC runner side.

If you come across this article you probably make use of Terraform Cloud or Terraform Enterprise self hosted instance. For a very long time the only way to leverage Terraform Cloud for Google Cloud deployments was to upload a Service Account Json Key. There were more options for self hosted Terraform Enterprise instance while it is hosted on GCP itself, it could make use of a Service Account attached to the hosting Compute Engine instance and impersonate workflow specific Service Accounts. But even that is not ideal since there is no a proper control preventing workflows from impersonation of other service accounts intended to be used by other workflows.

So, what has changed? As a part of TFE Release v202208–1 (647) OpenID Connect (OIDC) was introduced for Terraform Enterprise/Cloud. I got wondering if that would work with GCP Workload Identity Federation which natively supports OIDC as an external identity provider. To make a long story short, it does work!

To add more on that, Terraform Cloud has introduced Dynamic Provider Credentials, which is a simplification layer for setting up OIDC integration with major cloud providers.

Configuration of GCP Workload Identity Federation for Terraform Cloud/Enterprise workflows

To run through the example, start by cloning the repository and changing to the example folder.

git clone https://github.com/GoogleCloudPlatform/cloud-foundation-fabric.git
cd cloud-foundation-fabric/blueprints/cloud-operations/terraform-cloud-dynamic-credentials

Create Terraform Cloud Workflow

If you don’t have an existing Terraform Cloud organisation you can sign up for a free trial account.

Create a new Workspace for a CLI-driven workflow. Workload Identity Federation works for any workflow type, but for simplicity of the blueprint we use CLI driven workflow). Note workspace name and id, we will use them on a later stage (your values will be different).

Go to the organisation settings and note the org name and id (your values will be different).

Deploy GCP Workload Identity Pool Provider for Terraform Cloud integration

NOTE: This is a preparation part and should be executed on behalf of a user with enough permissions.

Required permissions when new project is created:

• Project Creator on the parent GCP folder/organisation

Required permissions when an existing project is used:

• Workload Identity Admin on the GCP project level
• Project IAM Admin on the GCP project level

Fill out required variables, use TFE Org and Workspace IDs from the previous steps (IDs are not the as names):

cd gcp-workload-identity-provider
mv terraform.auto.tfvars.template terraform.auto.tfvars
vi terraform.auto.tfvars

Authenticate using application default credentials, execute terraform code and deploy resources

gcloud auth application-default login
terraform init
terraform apply

As a result a set of outputs will be generated (your values will be different), note the output, we will use it on the next steps.

project_id = "tfc-dynamic-creds-gcp"
tfc_workspace_wariables = {
  "TFC_GCP_PROJECT_NUMBER" = "200635100209"
  "TFC_GCP_PROVIDER_AUTH" = "true"
  "TFC_GCP_RUN_SERVICE_ACCOUNT_EMAIL" = "sa-tfc@tfc-dynamic-creds-gcp.iam.gserviceaccount.com"
  "TFC_GCP_WORKLOAD_POOL_ID" = "tfc-pool"
  "TFC_GCP_WORKLOAD_PROVIDER_ID" = "tfc-provider"
}

As part of the deployment we have the following:

• GCP Workload Identity Pool and Provider.
• Provider has an attribute condition to make sure only OIDC token generated in a specific TFC Organisation can be used.
• Service Account for impersonation, where we only allow impersonation by a specific TFC workflow based the token attribute.
• We grant the Service Account with Storage Admin permissions for the testing step.

Configure Dynamic Provider Credentials for your TFC Workflow

To configure GCP Dynamic Provider Credentials for a TFC workflow, you need to set a set of environment variables:

• TFC_GCP_PROVIDER_AUTH
• TFC_GCP_PROJECT_NUMBER
• TFC_GCP_RUN_SERVICE_ACCOUNT_EMAIL
• TFC_GCP_WORKLOAD_POOL_ID
• TFC_GCP_WORKLOAD_PROVIDER_ID

Go to the Workflow -> Variables page and click the + Add variable button. For variable type select Environment variable. The variable names listed above are the names of the variables that you need to set. The values provided in the terraform output in the previous step are the values that you need to provide for each variable.

At that point we set up GCP Identity Federation to trust TFC generated OIDC tokens, workflow should be able to use Dynamic Provider Credentials to impersonate a GCP Service Account.

Testing the result

To test the setup, we will deploy a GCS bucket from the TFC Workflow created in the previous step. This will allow us to verify that the workflow can successfully interact with GCP services using the TFC Dynamic Provider Credentials.

Let’s jump into tfc-workflow-using-wif directory and configure terraform backend to point to our workspace, use TFE organisation name and workspace name (names are not the same as ids):

cd ../tfc-workflow-using-wif
mv backend.tf.template backend.tf
vi backend.tf

Fill out variables based on the output from the preparation steps:

mv terraform.auto.tfvars.template terraform.auto.tfvars
vi terraform.auto.tfvars

Authenticate terraform for triggering CLI-driven workflow

Follow this documentation to login ti terraform cloud from the CLI and trigger the workflow execution:

terraform login
terraform init
terraform apply

As a result, we have successfully deployed a GCS bucket from Terraform Cloud workflow using Workload Identity Federation.

Once done testing, you can clean up resources by running terraform destroy first in the tfc-workflow-using-wif and then gcp-workload-identity-provider folders.