Terraform CDK GCP

Getting started with Terraform CDK in GCP

In last few months we were assigned the task to create some standards around provisioning. We wanted to write some machines that can be onboarded as part of migration. They should be cut from the same cloth while allowing us to spin machines faster. We wrote that code in traditional terraform HCL; however, we tried another strategy which I shall demonstrate here. Another motivation was to write something for Google-Cloud as currently there is no official getting started guide in terraform website for GCP.

Objective

At the end of this blog , we shall achieve the following

1. A code base that can create machines on the request
2. Validation of input variables and converting them to objects
3. Using language to automate task, which is the motive behind CDKs
4. An actual machine should get created
5. Process the file as an object , not like a map as one has to do in standard terraform code

Requirements

There are some hard requirement and language dependencies to use terraform-cdk. I am listing them down below

1. In order to use CDKTF, you need:

• The Terraform CLI (1.1+).
• Node.js and npm v16+.

2. Some prior experience in Tyepscript is preferred

3. Some development libs to process the yaml file

Approach

In order to solve the first set of problem, we went with an approach of GitOps workflow. Here, one can define the resources that they need in YAML and the CDK can provision it for them. However, in an enterprise level project, it has to be approved by the SRE/CloudOps team to ensure no security laps are introduced. We are going to take the following steps to achieve the same

1. Install the CDK and terraform. We already have this documented in terraform’s official page
2. We have kept the code in Github. This code is kept open for use
3. One can define the resources required in the yaml file, under the gcp_projects folder
4. Execute the code and capture the output

Terraform-CDK https://www.terraform.io/cdktf Installation

Once one has node js installed, CDKTF is simply another external program that one has to download. This can be done using NPM

$ npm install --global cdktf-cli@latest #installing the package
$ cdktf help # Verifying the installation 
# Install npm package for GCP
$ npm install @cdktf/provider-google

Setup a new project

CDKTF comes with its own command for scaffolding of a new project. One can start by following this link as well

$ cdktf init --template=typescript --local
# This step should add google provider 
$ cdktf provider add google
# Verify 
$ cat cdktf.json 
{
  "language": "typescript",
  "app": "npx ts-node main.ts",
  "projectId": "32bba6c6-91a9-4deb-879b-33f55335c2f7",
  "sendCrashReports": "false",
  "terraformProviders": [],
  "terraformModules": [],
  "context": {
    "excludeStackIdFromLogicalIds": "true",
    "allowSepCharsInLogicalIds": "true"
  }
}

Install Yaml processing lib

We are trying to process yaml files, we are going to use js-yaml for this purpose. We shall install that and store it in npm package.json file. Also, we wanted to preserve the logs like java-log4j. We shall install another package for the same i.e. log4js

# install js-yaml  
$ npm install --save @types/js-yaml
# Log4js
$ npm install --save log4js

Setup the credentials

Before executing the code, one has to setup the credentials, use gcloud auth login to setup that.

Let’s write some code

I have written some code in Github and I have made it public. I am going to explain the idea here, preprocessing, interface mapping and then creating the terraform objects.

Preprocessing

Let’s pick an example of the yaml file that operators shall supply, I have made an example file that is processed,

project.yaml

test.yaml file

This will be our yaml file that we can process. Let me illustrate the pre-processing code.

preprocessing.yaml

Processing

Here, we just need to process the object i.e. projectInfo. This object shall encapsulate the gcpObjects within. Makes the processing extremely dev friendly.

Output

We have to execute 2 commands , one to ensure the successful compilation and other to execute the code which is the same like terraform deploy.

$ cdktf synth # compile the code and create the intermidiate file in cdktf.out folder
$ cdktf deploy # executes the terraform apply 

Conclusion

We have achieved the success with this method and reduced the processing code. It is a lot more readable than terraform HCL’s variable processing logic. However, it is not without its flaws, one has to know a language like typescript and python, Code quality has to be maintained as any code base, Git practices must be followed like branching and PR, etc.