The “Best” Terraform CD pipeline with GitHub Actions

Shameless clickbait picture created with DALL-E

Continuous Deployment pipelines for Terraform are an essential component of safe cloud infrastructure management. Let’s take a look at developing a robust, simple, and scalable pipeline for Terraform Deployments using native GitHub Actions.

An initial draft of this article was much more philosophical around the “whys” infrastructure should be deployed in this way. That article may still come but this is a streamlined guide to developing the pipeline I had in mind.

A GitHub repository with this code can be found here.

The Flow

Diagram describing the CD flow for Terraform Deployment

This shows the process we are going to create. Boiled down:

1. The main branch holds the state of all currently deployed Terraform.
2. Engineers make a PR for a branch called feature/<slug>with changes they want to make.
3. A pipeline automatically picks up these changes, runs a terraform plancommand, and returns the plan as a comment in the PR.
   Important: We want to ensure the above plan is exactly what we apply in later steps. The plan is saved as a file and uploaded as a GitHub artifact.
4. Eventually, someone approves the PR. This kicks off another pipeline to apply the previously reviewed plan. This will download the artifact mentioned up a step and apply the changes.

Sounds easy! Let’s build… right? Nope — first we need a valid target for a Terraform deployment.

For this article, I’m going to deploy simple resources to a Google Cloud Platform project and I will discuss this setup below. If you already know how to do this, or have a different target, feel free to skip the next section.

Google Cloud Platform Prep (GCPP 😆)

Authentication

Authenticating to GCP from a GitHub Actions Workflow can be done in two different ways:

1. Use OIDC with Workload Identity Federation to implicitly authenticate the GitHub Actions jobs to Google Cloud. This is the preferred method.
2. Storage a Service Account JSON key as a long-lived secret in the repository settings.

Both options are valid. The first option is the most secure and robust and I would always recommend it for production environments. The second option is typically fine for sandbox or hobby projects.

Project Prep

A GCP Project needs to be provisioned ahead of time, along with a GCS Bucket to hold your Terraform state. This can then be configured in your backend.tf file like:

terraform {
  required_version = ">= 1.2.0"

  backend "gcs" {
    bucket = "<BUCKET_NAME_HERE>"
    prefix = "<DESIRED_PREFIX>"
  }
}

More information about this can be found here.

The Build

The flow we want to create is split into two distinct actions:

1. Pull Request activity (creation and subsequent commits)
2. Pull Request approval

We will be creating a GitHub Actions Workflow for each of these.

All of the GitHub Action files will live in the .githubdirectory of our repository. We are also going to use composite actions for repeatability.

Terraform Plan Composite Action

Walking through the logic first, the steps we need to take look like:

1. Setup the environment. This includes configuring the runner to use the Terraform version we configure, and then set authentication to GCP. The below example uses a Service Account JSON key file for authentication.
2. Initialize Terraform terraform init
3. Create Terraform plan terraform plan . We do this in a way that captures the stdout of the Terraform plan execution into a variable. This enables us to push the output to the Pull Request comment in a future step.
4. Save the plan artifact. This ensures we can execute this exact plan again in the future.
5. Post a new comment on the Pull Request with the output from terraform plan using the Peter Evans action.

The code looks like:

name: 'Terraform setup and plan'
description: 'Setup Terraform and creates plan'
inputs:
  terraform_directory:
    description: 'Directory that holds Terraform code'
    required: true
  terraform_sa: 
    description: 'GCP service account used for Terraform actions'
    required: true
  terraform_version: 
    description: 'Terraform Version'
    required: true
    default: 1.2.9
  github_token: 
    description: 'GitHub token for auth'
    required: true
  google_sa_key:
    description: 'JSON key for GCP service account'
    required: true
  pr_id:
    description: 'Pull request ID'
    required: true

runs:
  using: "composite"
  steps:
    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v2
      with:
        terraform_version: ${{ inputs.terraforom_version }}
        terraform_wrapper: false

    - id: 'auth'
      name: 'Authenticate to Google Cloud'
      uses: 'google-github-actions/auth@v1'
      with:
        service_account: ${{ inputs.terraform_sa }}
        credentials_json: ${{ inputs.google_sa_key }}

    - name: 'Set up Cloud SDK'
      uses: 'google-github-actions/setup-gcloud@v0.6.0'

    - name: Terraform Init
      id: init
      working-directory: ${{ inputs.terraform_directory }}
      shell: bash
      run: |
        terraform init

    - name: Terraform Plan
      id: plan
      working-directory: ${{ inputs.terraform_directory }}
      shell: bash
      run: |
        echo 'plan<<EOF' >> $GITHUB_OUTPUT
        terraform plan -no-color -out=tfplan >> $GITHUB_OUTPUT
        echo 'EOF' >> $GITHUB_OUTPUT

    - name: Save Artifact
      id: save-artifact
      uses: actions/upload-artifact@v3
      with:
        name: ${{ inputs.pr_id }}-tf-plan
        path: ${{ inputs.terraform_directory }}/tfplan

    - name: Comment Plan
      id: comment-plan
      uses: peter-evans/create-or-update-comment@v2
      with:
        token: ${{ inputs.github_token }}
        issue-number: ${{ inputs.pr_id }}
        body: |
          Terraform Plan:

          ```
          ${{ steps.plan.outputs.plan }}
          ```

          Plan saved to GH artifacts.

This is saved at .github/plan/action.yml so we can use it later.

Plan Workflow

The next step is to write a Workflow that uses the above composite action an executes plans on pull_request events. This encapsulates the opening, and commit changes, to a Pull Request.

We will also need an additional step to extract the Pull Request ID that triggered the job. This is a required field for posting the comment on the Pull Request with the terraform plan output.

The code looks like:

name: Terraform Plan

on:
  pull_request:

env:
  TF_SA: <GOOGLE_SA_EMAIL>
  TERRAFORM_VERSION: "1.2.9"
  TF_IN_AUTOMATION: "True"

jobs:
  terraform_plan:
    runs-on: ubuntu-latest
    if: github.event.review.state != 'approved'
    steps:
      - uses: actions/checkout@v3

      - name: Get PR ID
        id: pr-id
        shell: bash
        env:
          GITHUB_REF: ${{ inputs.github_ref }}
        run: |
          PR_NUMBER=$(echo $GITHUB_REF | awk 'BEGIN { FS = "/" } ; { print $3 }')
          echo "PR_NUMBER=$PR_NUMBER" >> $GITHUB_OUTPUT

      - name: Terraform Plan
        uses: ./.github/plan
        with:
          terraform_sa: ${{ env.TF_SA }}
          terraform_directory: "terraform"
          terraform_version: ${{ env.TERRAFORM_VERSION }}
          google_sa_key: ${{ secrets.GOOGLE_CREDENTIALS }}
          github_token: ${{ secrets.GITHUB_TOKEN }}
          pr_id: ${{ steps.pr-id.outputs.PR_NUMBER }}

This lives in the file /.github/workflows/plan.yml .

The directory /terraform holds the Terraform code in my example. This is referenced as the terraform_directory input in our composite action.

You will need to modify the TF_SA environment variable with the email address of the GCP Service Account you are using.

This is the comment output from the job.

Comment output from a Terraform Plan

Terraform Apply Composite Action

Next we need to create the composite action for executing the planned Terraform changes. Let’s again walk through the logic. Steps 1 and 2 are the same as the plan action.

1. Setup the environment. This includes configuring the runner to use the Terraform version we configure, and then set authentication to GCP. The below example uses a Service Account JSON key file for authentication.
2. Initialize Terraform terraform init
3. Download the plan artifact that was generated by the latest terraform plan execution. This is tricky because we are downloading an artifact from a different GitHub workflow. We use a custom action from the legend Dawid Dziurla for this.
4. Execute the Terraform changes using the plan file. We also save the stdout of this execution into a variable so we can push it as a Pull Request comment later.
5. Create a new Pull Request comment with the output from terraform apply using Peter Evans action.

The code looks like:

name: "Terraform setup and apply"
description: "Applys a terraform plan file from an artifact"
inputs:
  terraform_directory: 
    description: 'where to execute terraform'
    required: true
  terraform_sa: 
    description: 'GCP service account for Terraform'
    required: true
  terraform_version: 
    description: 'GCP Terraform Version'
    required: true
    default: 1.2.9
  github_token: 
    description: 'github secret'
    required: true
  google_sa_key:
    description: 'JSON key for GCP service account'
    required: true
  pr_id:
    description: 'Pull request ID'
    required: true

runs:
  using: "composite"
  steps:
    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v2
      with:
        terraform_version: ${{ inputs.terraforom_version }}
        terraform_wrapper: false

    - id: 'auth'
      name: 'Authenticate to Google Cloud'
      uses: 'google-github-actions/auth@v1'
      with:
        service_account: ${{ inputs.terraform_sa }}
        credentials_json: ${{ inputs.google_sa_key }}

    - name: 'Set up Cloud SDK'
      uses: 'google-github-actions/setup-gcloud@v0.6.0'

    - name: Terraform Init
      id: init
      working-directory: ${{ inputs.terraform_directory }}
      shell: bash
      run: |
        terraform init

    - name: Download Plan
      id: download-plan
      uses: dawidd6/action-download-artifact@v2
      with:
        github_token: ${{ inputs.github_token }}
        workflow: plan.yaml
        pr: ${{ inputs.pr_id }}
        name: ${{ inputs.pr_id }}-tf-plan
        path: ${{ inputs.terraform_directory }}

    - name: Terraform Apply
      id: apply
      working-directory: ${{ inputs.terraform_directory }}
      shell: bash
      run: |
        echo 'apply<<EOF' >> $GITHUB_OUTPUT
        terraform apply -input=false -no-color tfplan >> $GITHUB_OUTPUT
        echo 'EOF' >> $GITHUB_OUTPUT

    - name: Comment Apply
      id: comment-apply
      uses: peter-evans/create-or-update-comment@v2
      with:
        token: ${{ inputs.github_token }}
        issue-number: ${{ inputs.pr_id }}
        body: |
          Terraform Apply:

          ```
          ${{ steps.apply.outputs.apply }}
          ```

This is saved at /.github/apply/action.yml .

Apply Workflow

Now we need to write a workflow that uses the above composite action whenever a Pull Request is approved. In this system, approving the Pull Request both the code and the planned changes to our infrastructure.

Just like the plan job, we will use a method to extract the Pull Request ID so we can post a comment.

The code looks like:

name: Terraform Apply

on:
  pull_request_review:
    types: [submitted]

env:
  TF_SA: <GOOGLE_SA_EMAIL>
  TERRAFORM_VERSION: "1.2.9"
  TF_IN_AUTOMATION: "True"

jobs:
  terraform_apply:
    runs-on: ubuntu-latest
    if: github.event.review.state == 'approved'
    steps:
      - uses: actions/checkout@v3

      - name: Get PR ID
        id: pr-id
        shell: bash
        env:
          GITHUB_REF: ${{ inputs.github_ref }}
        run: |
          PR_NUMBER=$(echo $GITHUB_REF | awk 'BEGIN { FS = "/" } ; { print $3 }')
          echo "PR_NUMBER=$PR_NUMBER" >> $GITHUB_OUTPUT

      - name: Terraform Apply
        uses: ./.github/apply
        with:
          terraform_sa: ${{ env.TF_SA }}
          terraform_directory: "terraform"
          terraform_version: ${{ env.TERRAFORM_VERSION }}
          google_sa_key: ${{ secrets.GOOGLE_CREDENTIALS }}
          github_token: ${{ secrets.GITHUB_TOKEN }}
          pr_id: ${{ steps.pr-id.outputs.PR_NUMBER }}

Let’s see it in action!

Terraform Apply being commented back to the PR

Wrap Up

All of the code and examples can be found here.

Now, whenever an engineer opens a PR on this repository it will automatically generate a Terraform plan for the changes and creates an approval gate for executing those changes. This creates strong control and automation for deploying changes.

Systems like this are important to ensure a lifecycle exists for infrastructure changes. Using Terraform as a tool is the right way to start, but building pipelines and approval gates like this is important for scaling.

Feel free to contact me with any comments, questions, concerns, jokes you have regarding this approach.

Caveat

A couple of the techniques used in this demo don’t work well with Public Repositories (for security concerns). I used a bit of cheese to get around this for the demo, however the code shared in the above snippets (and in the repo) are verified in private repositories.