Designing CI/CD pipeline for folder-based Terraform code using Git Tags and Google Cloud Build
This article assumes that the reader has some experience working with Terraform, GitHub and Google Cloud Build. Let’s quickly refresh our knowledge on Google Cloud Build. Google Cloud Build is a serverless build, test, and deploy platform offered by Google which helps users:
- Build software quickly across all programming languages, including Java, Go, Node.js, and more
- Deploy across multiple environments such as VMs, serverless, Kubernetes, or Firebase
- Access cloud-hosted fully managed CI/CD workflows within your private network
- Keep their data-at-rest within a geographical region or specific location with data residency
To explore Google Cloud Build in more detail, please refer the official product link:
What is “folder-based” Terraform code?
Let us first understand what we mean by “folder-based” Terraform code. In simple terms a folder-based terraform code will have multiple folders each containing some Terraform code. Folders are generally used to isolate and independently manage Terraform code belonging to different applications or services. For services that share terraform outputs among each other, it would be ideal to have all these service folders under the same Git repository. For example, we can group Terraform code for bootstrap, resource management, networking, security and project factory services. All these folders are present at the root of a git repository created for setting up a landing zone. Please refer the screenshot below:
The Challenge
Due to this folder structure, we will have to go inside each folder and perform the required Terraform commands. Since we are using Google Cloud Build as our CICD platform, we need a way to tell Cloud Build which folder it needs to navigate to run terraform init, plan and apply commands.
Following is the summary of alternatives or approaches we can use which are not ideal:
- For N folders we can create corresponding N Cloud Build triggers each having the respective folder path in a user-defined substitution like “_WORKSTREAM_PATH” . The trigger event for these triggers will be “push to branch” for the main or master branch. The problem with this approach is that it is not at all dynamic. As the folder count increases, it gets increasingly difficult to manage the corresponding Cloud Build triggers. Also whenever a commit to the main/master branch is merged, all the N Cloud Build triggers will be executed simultaneously disregarding any possible implications on other services due to this
- Creating N branches named after the N folders is also not a good idea as we would have to manage these N branches separately. Dealing with increasing folder count and branch merges would be a nightmare (just to name a few)
The Solution
If we look closely at the event trigger options available for Cloud Build triggers we can see the following:
What if we explore the “Push new tag” option……?
The “Push new tag” event invokes the Cloud Build trigger whenever a git tag is pushed to the respective git repository for which the Cloud Build trigger has been created. When a git tag is pushed to a git repository, the connected Cloud Build trigger captures the tag name in one of its substitutions called “$TAG_NAME”. Apart from this substitution, Cloud Build also captures the following default substitutions:
To learn more about Cloud Build substitutions, please refer to the link here.
Since Cloud Build captures the tag name in the substitution “$TAG_NAME”, we can name the tag after one of our Terraform folders. This is a very easy and simple way to communicate which Terraform folder the Cloud Build CICD pipeline should use for running the required Terraform commands. For the folder structure diagram shown earlier, git tag names can be of the following format:
- 02-networking-peering-201768
- 00-bootstrap-test
- 03-project-factory-prod
- 02-security-scc
In the Cloud Build trigger we can use the following regular expression to make sure only the required tag names invoke the Cloud Build trigger:
(00-bootstrap|01-resman|02-networking-peering|02-security|03-project-factory)-.*
The same can be seen in the screenshot below:
We can again make sure we get the correct stage/folder name from the git tag in the cloudbuild.yaml using a simple shell command with the following regular expression:
$ echo "$TAG_NAME" | egrep -o "(00-bootstrap|01-resman|02-networking-peering|02-security|03-project-factory)" > EXTRACTED_TAGComplete CI/CD strategy
First we need to create a pull request trigger for each stage/folder. In these triggers the _WORKSTREAM_PATH user-defined substitution would have to be set to the respective stage/folder. For e.g. if we have 5 stages then we would need corresponding 5 pull request triggers. These pull request triggers will perform terraform init, terraform validate and terraform plan in that specific order.
We want separate pull request triggers for each stage/folder to make sure that they are decoupled from each other. If we had a single pipeline/trigger for all stages, error in code for one of the stages would block the entire pipeline for rolling out changes for other stages. Changes for decoupled pipelines can be rolled out independently for each of the stages. Also if we just had a one CICD pipeline for all stages it would take a lot of time to complete its execution. The CICD pipeline execution time would keep on increasing as the number of resources in each service folder increases.
Cloud Build has this handy feature where we can send the build logs to GitHub so you can directly see the execution logs in your pull request. Below screenshot shows the option enabled:
Once we are happy with our terraform plan, we can go ahead and merge the pull request. Merging the pull request will not automatically trigger the terraform apply pipeline. This is because we have created a Cloud Build trigger that gets invoked only when a git tag is pushed instead of pushing a commit to a branch.
Now to run terraform apply on your recently pushed/merged commit, we need to push a tag for the respective stage/folder for which the change pull request was raised. There are 2 ways to do this:
- We can use the GitHub console to publish a release with the desired tag. NOTE: creating a release for pushing a tag is not at all necessary. It is a necessity only if you are using the GitHub console
- We can use git commands to push the desired tag. Use the following command to do so:
$ git tag <tag_name>
$ git push origin <tag_name>As soon as a tag is pushed, a Cloud Build trigger will get invoked which will use the tag name to navigate to the right folder to perform the terraform apply command along with terraform init and terraform plan.
