How to build a CI/CD pipeline for Terraform with Terramate on GitHub Actions

Please find the example discussed in this article at https://github.com/terramate-io/terramate-github-actions-example

Terramate helps implement and maintain highly scalable Terraform projects by adding powerful capabilities such as code generation, stacks, orchestration, change detection, data sharing and more.

It can easily be used with any existing CI/CD tool to provide a seamless Terraform workflow. Its flexibility allows building a simple set of jobs that provides a similar experience to other tools, such as Atlantis.

In this article, we will go through an example that shows how to use Terramate to build a simple yet powerful CI/CD pipeline in GitHub Actions. This example contains two main workflows:

• A preview workflow that runs on Pull Requests to execute a Terraform plan on the changed stacks and posts it as a comment to the PR.
• An apply workflow that runs on Pull Request merge and executes Terraform apply on the changed stacks.

Before diving in, why not join the Terramate Community on Discord? Meet and chat with fellow community members, ask Terramate questions and stay informed about new product releases, community events, and more!

The Terramate concepts used in this example

• Stacks: Terramate uses the concept of stacks to define smaller, runnable Terraform modules that include a subset of the infrastructure resources. Splitting your Terraform code into smaller stacks is always recommended to reduce execution time, isolate code that changes frequently, and reduce the blast radius in case of failure.
• Orchestration: Terramate CLI offers the terramate run command that executes any commands on one, many, or all stacks. terramate run includes many options that allow filtering which stacks to run and in what order to run them.
• Filtering (stack selection): Terramate allows selecting stacks to run based on the following criteria:
• Working directory: It is possible to select a set of stacks based on their filesystem location using the --chdir option to change the working directory.
• Tags: The --tags option allows running a set of stacks that share a particular tag in their metadata.
• Change detection: Terramate integrates with git to detect which stacks include changes in a commit or pull request. Then, using the --changed option, it is possible to filter out all the unchanged stacks, thus reducing the execution times significantly when running terraform plan and terraform apply.
• Ordering: Terramate also allows specifying in which order stacks are executed. This is done in two ways:
• Implicit ordering: Using the filesystem hierarchy of stacks, Terramate executes parent stacks before their child stacks by default.
• Explicit ordering: This can be done by defining before and after attributes in a stack’s metadata. Ordering can be done using a stack’s path or tags.

GitHub Actions

GitHub Actions is a CI/CD tool provided by GitHub and integrated within its source control system. It allows running jobs using either custom commands or pre-built actions from their marketplace.

GitHub Actions workflows are normally triggered by user actions such as pushing to specific branches in your repository, creating or updating a Pull Request, etc.

Breaking down the example

Let’s take a look at the example repository at https://github.com/terramate-io/terramate-github-actions-example to understand the workflows.

Plan workflow

Looking at the .github/workflows/plan.yml file, the first thing to note is the condition, or trigger, for running this workflow. The on attribute defines what events can trigger this workflow.

In this example, we can see the following:

on:
  pull_request:

This tells Github Actions to only execute this workflow on any events that involve a pull request, namely create and update.

Going over the steps, we can see that it does the following:

• Checkout: Clone the repository. Setting fetch-depth: 0 configures the action to fetch all history for all branches and tags, which is required for the Terramate Change Detection to function properly.
• Install tools: Using [asdf](<https://asdf-vm.com/>) to install the required dependencies such as Terraform and Terramate defined in .tool-versions
• List changed stacks: Run terramate list --changed to list changed stacks only. The output of this command (with id: list) is accessed in the next steps (if: steps.list.outputs.stdout) to determine if we will execute any of the subsequent steps, as we do not want to run any further steps if no stacks have changed.
• Configure AWS credentials: Use an AWS-provided GitHub Action to authenticate to AWS using API keys stored as GitHub Actions Secrets. Note: For production environments we highly recommend using identity federation using GitHub OIDC Actions instead of long lived static credentials.
• Run Terraform plan: Run terraform init, terraform validate, and terraform plan all using terramate run --changed. Note that these steps have a condition if: steps.list.outputs.stdout, so they will be skipped if no changed stacks are detected.
• Generate Preview Comment: This step will use terraform show on the changed stacks to print and format the Terraform plan into a human-readable comment. If no stacks are changed, the next step is run instead, just printing No changed stacks into the comment.
• Publish Plans for Changed Stacks: This step uses a GitHub Action from the marketplace that allows publishing “sticky comments” into a GitHub Pull Request. This means that the action will not post a new comment every time the PR is updated, but will rather update the existing comment with the new plan after each change.

You can see an example of a Pull Request that includes adding a few stacks, which showcases how the PR comment looks like: https://github.com/terramate-io/terramate-github-actions-example/pull/2

Apply workflow

Looking at the .github/workflows/apply.yml file, we can spot that we re-use some of the same techniques to execute terraform apply. Before going into the steps, we can see that the condition/trigger for this workflow is different from the previous one:

on:
  push:
    branches:
      - main

The apply workflow is triggered when any new change is pushed to the main branch. When combined with GitHub branch protection on main, this guarantees that this workflow is only triggered when a Pull Request has been approved and merged.

At Terramate we recommend the best practice of always running terraform apply whenever a Pull Request is merged, thus ensuring that the code in main always reflects the status of your infrastructure.

Most of the steps in this workflow will be similar to the Plan workflow:

• Checkout
• Install tools
• List changed stacks
• Configure AWS credentials
• Terraform init

The main difference is simply that an apply --auto-approve is run instead of a plan, and that it’s not necessary to publish the output as comments in the PR

Note about stack ordering

In the example PR, we can see that the VPC stack includes a before statement, that indicates it should run before the EC2 instances stacks.

This is reflected in the Terraform plan, that shows the VPC stack being executed before the EC2 stacks.

In this example, we discussed how to automate Terraform with Terramate in GitHub Actions. If you have any questions or feedback, please join our Discord Community Server.

Resources:

• Example GitHub Actions https://github.com/terramate-io/terramate-github-actions-example
• Terramate Documentation https://terramate.io/docs/cli/

This article originally appeared in Rethinking IaC on August 21, 2023