GCS to BigQuery via Cloud Composer: Part 2 (Setup Cloud Composer Environment)

Zdravstvuyte 👋

In Part 1, we saw an overview of our ETL pipeline. In this post, we are going to focus on setting up Cloud Composer environment. This post is part of GCS to BigQuery Pipeline via Different GCP Services project.

Folder Structure

Below is how my repo for setting up the environment is structured:

📦1-create-environment
 ┣ 📜main.tf
 ┣ 📜providers.tf
 ┗ 📜variables.tf

Source Code

For the construction of the Cloud Composer environment, we will be leveraging Terraform (IaaC).

All the Terraform configurations can be found here for Cloud Composer.

providers.tf

provider "google" {
  project = var.gcp_project_id
  region  = var.gcp_region
}

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0.0"
    }
    archive = {
      source  = "hashicorp/archive"
      version = "~> 2.2.0"
    }
  }
  backend "gcs" {
    bucket = "terraform-state-bucket-gcp-practice-project-aman"
    prefix = "cloud-composer/create-environment"
  }
}

• The gcp_project_id & gcp_region will be coming from our variables.tf.
• prefix variable is the GCS location where our Terraform state file will be placed.

variables.tf