How to set up a service account in GCP for a data engineering project using terraform.

These are the simple steps to take.

1. Create an account with your Google email ID
2. Setup your first project if you haven’t already, name it what ever you want. eg. “my_first_project1”, and note down the “Project ID” (we’ll use this later when deploying infra with TF)

3. Lets navigate to the service account in the IAM & admin and create a service account, for proper understanding of what a service account is, click on link. Once you done creating the service account. you will have this on your dashboard. You will create these roles: storage admin roles, big query admin role.

at this point i believe your key id will be showing no keys, do assign a key, navigate to the actions and click on manage keys and then click on add key and create a new key. keep the json and click create your key will be downloaded

4. Lets head over to our git bash to check if g_cloud sdk is already installed.

if you don't have it go to this link to install it : google cloud sdk

5. once you have the sdk installed, lets search for our key we downloaded. we are trying to configure the environment variable to point to the keys

→ Now lets set the google application environment credentials.

Click on Y, and this will take you the success page that lets you know that authentication is successful

Please, if you are using git bash in windows and you are not able to access the authentication page, switch to command prompt instead.

once you get through to the page above, your local environment will be now able to interact with your cloud environment.

6. Now lets add permissions for our service account click on IAM to add some roles, click on the edit button on the service account you just created.

also you will need to add the compute engine role.

We will now have to enable APIs, why should we enable this?, we have to do this because when the local environment interact with the cloud environment, it does not interact to the resource directly, so this is where the APIs comes in.

Enable these APIs for your project:

https://console.cloud.google.com/apis/library/iam.googleapis.com
https://console.cloud.google.com/apis/library/iamcredentials.googleapis.com

click on the link and enable your APIs.

Then we are good to go in setting up a service account.

Step 7. Terraform Basic file

• Let first of all make directory for terraform, we can call it terraform_1, then navigate to the directory we just created, lets create another directory called keys1 and also navigate to it, now lets open an editor for a json file you can call it my-creds1.json or what ever you like.
• Then copy and paste the private key you downloaded, into that json file, click on ctrl + T and then type in yes for buffer
• Then we will need to go up one directory. type in (cd..). Then start visual studio code (code .), for new users of terraform, ensure you download the extension of terraform. I will recommend you choose the terraform extension from hashicorp
• create a new file called main.tf. navigate to google and search for terraform google provider.

provider "google" {
  project     = "my-project-id"
  region      = "us-central1"
}

• copy this code to your main.tf

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "4.51.0"
    }
  }
}

provider "google" {
  project     = "my-project-id"
  region      = "us-central"
}

• CTRL + S , then navigate to your git bash and type in the command : (terraform fmt) this commands helps in formatting.
• This is suppose to out put the name of the file.
• Lets head over to our google cloud to grab our project id, that will be inserted in the main.tf file.
• Then we will navigate to our git bash and input these commands.

• lets type in terraform init , this will initiate the terraform-google-provider.

Step 8 : Making a bucket

• Navigate to google cloud dashboard → cloud storage → buckets
• go to your browser and search for terraform google cloud storage bucket
• Copy and paste this code on the main.tf

resource "google_storage_bucket" "auto-expire" {
  name          = "auto-expiring-bucket"
  location      = "US"
  force_destroy = true

  lifecycle_rule {
    condition {
      age = 1
    }
    action {
      type = "AbortIncompleteMultipartUpload"
    }
  }
}

• CTRL + S, and then run a (terraform plan) on git bash
• Then (terraform apply). a terraform state file will pop up.
• Now lets navigate to google cloud and refresh the buckets to see our bucket.
• once you are done with the bucket you can now type (terraform destroy)
• If we now head over to our google and refresh. The bucket wont be available again and in your editor a back up file will be created.

This is a simplistic way of how to get going on terraform and creating a resource.