How to Create a CI/CD Pipeline with Terragrunt, AWS, and Either GitHub Actions or CircleCI

CI/CD Pipelines, GitHub Actions, and CircleCI

CI/CD pipelines are a set of automated processes that help to build, test, and deploy software applications. The acronym CI/CD stands for Continuous Integration and Continuous Deployment. The goal of CI/CD is to continuously integrate and test code changes as they are made and to automatically deploy the code to production if it passes all tests.

GitHub Actions and CircleCI are two popular continuous integration and continuous delivery (CI/CD) platforms that provide developers with the tools and functionality to automate the software development process, including building, testing, and deploying code changes.

Terragrunt

Terragrunt is a tool for building, changing, and versioning infrastructure safely and efficiently. It is a thin wrapper for Terraform that provides extra tools for keeping Terraform configurations DRY (Don’t Repeat Yourself principle of writing once and preventing repetitions), working with multiple Terraform modules, and managing remote state. Terragrunt enables companies to keep Terraform code organized and reusable across multiple environments and teams.

CI/CD pipelines built with Terragrunt allow developers to safely and efficiently manage infrastructure as code by automatically testing, building, and deploying infrastructure changes. This enables teams to quickly and easily roll out new updates and features while maintaining a high level of control and visibility over infrastructure.

Below are two step-by-step tutorials on how to set up a CI/CD pipeline with Terragrunt, AWS, and GitHub Actions or CircleCI.

Tutorial #1 — How to Set Up a CI/CD Pipeline with Terragrunt, AWS, and GitHub Actions

Github Repository of Tutorial

Setup Terragrunt Repository

Put files structure

$ tree -r
.
├── terragrunt.hcl
└── tenant
    ├── us-west-2
    │   └── dev
    │       ├── vpc
    │       │   └── terragrunt.hcl
    │       └── env.hcl
    └── account.hcl

4 directories, 4 files

Explain VPC Terragrunt configuration

Github Base Source URL (for private Terraform module repositories)

In tenant/us-west-2/dev/vpc there is the Terragrunt configuration for the VPC that needs to be deployed.

In the locals constants there are 3 variables for the source URL:

• gh_source_url
• source_url
• base_source_url

The gh_source_url should be used only when the Terraform modules are in private GitHub repositories. In this case, a PAT is required so that Terragrunt can pull the other repositories from GitHub Actions. Readers can learn more about this in the section titled ‘Configure Github Action’.

If users don’t need to have access to private repositories, they can simply remove the gh_source_url variable and replace the current base_source_url variable by the source_url variable:

locals {
  [...]

  ref             = "?ref=main"
  gh_source_url   = "git::https://github.com/TrackIt/tf-modules.git//vpc_example${local.ref}"
  source_url      = "git::git@github.com:TrackIt/tf-modules.git//vpc_example${local.ref}"
  base_source_url = get_env("IS_GH", "FALSE") == "TRUE" ? local.gh_source_url : local.source_url
}

The get_env function from Terragrunt allows users to get an environment variable and set a value by default if it doesn’t already exist. For instance, when trying to access the IS_GH environment variable, it is set to FALSE if the variable doesn’t exist. If this environment variable is set to TRUE it will use the gh_source_url as the base_source_url in the flow.

Create New GitHub Action

The next step is to create a GitHub Action which will plan the dev infrastructure in the tenant/us-west-2/dev folder.

Create folder `.github/worklows`

$ mkdir -p .github/workflows

Create `dev-plan.yml` file in `.github/worklows`

$ touch dev-plan.yml

Configure GitHub Action

Setup workflow (i.e.: on workflow dispatch)

When configuring Github Actions, users have to specify what triggers the Github Action. In this example, the Workflow Dispatch event is being used to manually trigger deployments in the GitHub Action dashboard. Readers can find the available event list here.

The following is the configuration for the Workflow Dispatch:

name: 'Dev Plan'
on:
 workflow_dispatch:

Setup environment variables (credentials, working directory)

To deploy the infrastructure, users have to set up environment variables:

[...]
env:
 WORKING_DIRECTORY: './tenant/us-west-2/dev'
 AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
 AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
 IS_GH: "TRUE"

• WORKING_DIRECTORY: This is the directory where users are going to deploy from. In this example, all dev resources are deployed in the us-west-2 region from the tenant account.
• AWS_ACCESS_KEY_ID: the AWS Secret Key ID used to deploy the infrastructure (the user who deploys the infrastructure needs to have the correct permissions, see IAM permissions). A Github Action secret is also created for this value so that it can be protected.
• AWS_SECRET_ACCESS_KEY: the AWS Secret Access Key corresponding to the Access Key ID. A GitHub Action secret has also been created for this value so that it can be protected.
• IS_GH: This environment variable is required to use private Terraform module repositories. It must be set to TRUE.

Configure Job

Now that the Github Action has been globally configured, the next step is setting up the jobs. In this example, the job being created is going to init and plan Terragrunt.

[...]
jobs:
 plan:
   name: 'Terragrunt'
   runs-on: ubuntu-latest
   steps:
     - name: 'Checkout'
       uses: actions/checkout@master

Setup Terragrunt / Terraform steps
First, users need to install Terraform and Terragrunt:

[...]
steps:
 [...]
 - uses: hashicorp/setup-terraform@v2
   with:
     terraform_version: 1.0.0
     terraform_wrapper: false

 - name: Setup Terragrunt
   uses: autero1/action-terragrunt@v1.1.0
   with:
     terragrunt_version: 0.37.1

Setup Git Global Config (for private Terraform module repositories)
Readers who are not using private Terraform module repositories can skip this step. Otherwise, git global config is set up using a PAT to access those repositories. The PAT can be saved as a secret; in this example, a GitHub Action secret called TF_MODULES_TOKEN has been created.

[...]
steps: 
 [...] 
 - name: 'Setup Git Global Config'
   run: |
   git config --global url."https://${{ secrets.TF_MODULES_TOKEN }}@github".insteadOf https://github

Terragrunt Init and Plan
Users can now interact with Terragrunt to init, plan, or even apply. In this example, only init and plan are being executed but the same can be done to apply infrastructure:

[...]
steps:
 [...]
 - name: 'Terragrunt Init'
   run: terragrunt run-all init --terragrunt-non-interactive
   working-directory: ${{ env.WORKING_DIRECTORY }}

 - name: 'Terragrunt Plan'
   run: terragrunt run-all plan --terragrunt-non-interactive
   working-directory: ${{ env.WORKING_DIRECTORY }}

As seen above, the WORKING_DIRECTORY environment variable is being used to run the Terragrunt commands only in the correct directory. Users can now init and plan resources using GitHub Actions and Terragrunt.

Tutorial #2 — How to Set Up a CI/CD Pipeline with Terragrunt, AWS, and CircleCI

Setup Workflow

By default, CircleCI will automatically test, build and deploy infrastructure changes when the GitHub repository contains a CircleCI configuration.

Create Config File

The CircleCi configuration file must be stored in .circleci/config.yml.
It will contain the definition of the different steps of the build.
An orb (plug-in) is also required to enable access to the AWS CLI.

version: 2.1
orbs:
  aws-cli: circleci/aws-cli@3.1.1
jobs:
  [...]
workflows:
  [...]

Set Up Environment Variables

Environment variables are configured on the CircleCI console under Project Settings -> Environment variables

Set up the following:

• AWS_ACCESS_KEY_ID: the AWS Secret Key ID used to deploy the infrastructure (the user who deploys the infrastructure needs to have the correct permissions, see IAM permissions).
• AWS_SECRET_ACCESS_KEY: the AWS Secret Access Key corresponding to the Access Key ID.
• GITHUB_TOKEN: the GitHub secret token to clone the modules repository

Build Steps for Terraform Plan and Apply

[...]
jobs:
  init_plan:
    docker:
      - image: alpine/terragrunt:1.1.0
    working_directory: ~/infrastructure-live/admin
    steps:
      - checkout: 
          path: ~/infrastructure-live
      - aws-cli/setup:
         aws-access-key-id: AWS_ACCESS_KEY
         aws-secret-access-key: AWS_ACCESS_SECRET
         aws-region: AWS_REGION
      - run:
          name: Terragrunt init
          command: |
            terragrunt run-all init
      - run:
          name: Terragrunt validate
          command: |
            terragrunt run-all validate
      - run:
          name: Terragrunt plan
          command: |
            terragrunt run-all plan
  deploy:
    docker:
      - image: alpine/terragrunt:1.1.0
    working_directory: ~/infrastructure-live/admin
    steps:
      - checkout: 
          path: ~/infrastructure-live
      - aws-cli/setup:
         aws-access-key-id: AWS_ACCESS_KEY
         aws-secret-access-key: AWS_ACCESS_SECRET
         aws-region: us-east-1
      - run:
          name: Terragrunt apply
          command: |
            terragrunt run-all apply --terragrunt-non-interactive
workflows:

With this configuration, CircleCI will execute 2 steps, one for plan and one for apply. This configuration can be upgraded with the addition of a manual validation of the plan if needed.

How TrackIt Can Help

Companies in need of assistance can partner with an AWS-recognized partner like TrackIt to automate infrastructure provisioning by integrating Terragrunt into their CI/CD pipelines. TrackIt can also assist in the integration of AWS services like AWS Elastic Container Service (ECS), AWS Elastic Kubernetes Service, and AWS Elastic Beanstalk to help achieve high availability, scalability, and security.

About TrackIt

TrackIt is an Amazon Web Services Advanced Consulting Partner specializing in cloud management, consulting, and software development solutions based in Marina del Rey, CA.

TrackIt specializes in Modern Software Development, DevOps, Infrastructure-As-Code, Serverless, CI/CD, and Containerization with specialized expertise in Media & Entertainment workflows, High-Performance Computing environments, and data storage.

TrackIt’s forté is cutting-edge software design with deep expertise in containerization, serverless architectures, and innovative pipeline development. The TrackIt team can help you architect, design, build and deploy a customized solution tailored to your exact requirements.

In addition to providing cloud management, consulting, and modern software development services, TrackIt also provides an open-source AWS cost management tool that allows users to optimize their costs and resources on AWS.