Terraform CICD github; Statefile in S3

Continuous Deployment site wide pipelines for Terraform are a fundamental part of safe Infra foundation. We look at native GitHub actions for robust, simple, and scalable pipeline for Terraform Deployments

Diagram of CD for Terraform

This article will focus on the following:

1. Creating github actions for terraform deployment.
2. Creating State file in S3 bucket.
3. Creating locking machanism for state file.
4. Deployment of IAC ( S3) with all the terraform best practice

Create a GitHub Repo and Configure GITHUB actions

GitHub - chaitu24/tf-cicd

1. The main branch holds the state of all currently deployed Terraform.

AWS Account and AWS Secrets

Step 1:
Create an IAM user

I used ‘admin’ privs. Please use more granular least privs.

On creating the access key, please remember to copy and download the access key and secret (as it will not appear afterward).

Setting Up Secrets in GitHub Actions

Keep the AWS secret access key and access key id that were copied/downloaded in the previous step so they can be used in the upcoming step.

Step 2:

GitHub repo -> Settings -> /settings/secrets/actions -> add “new repo secret”

Step 3:

Create workflow

Create file in repo -> .github -> workflow -> terraform.yml

In my case (https://github.com/chaitu24/tf-cicd/blob/main/.github/workflows/terraform.yml

# This workflow installs the latest version of Terraform CLI and configures the Terraform CLI configuration file
# with an API token for Terraform Cloud (app.terraform.io). On pull request events, this workflow will run
# `terraform init`, `terraform fmt`, and `terraform plan` (speculative plan via Terraform Cloud). On push events
# to the "main" branch, `terraform apply` will be executed.
#
# Documentation for `hashicorp/setup-terraform` is located here: https://github.com/hashicorp/setup-terraform
#
# To use this workflow, you will need to complete the following setup steps.
#
# 1. Create a `main.tf` file in the root of this repository with the `remote` backend and one or more resources defined.
#   Example `main.tf`:
#     # The configuration for the `remote` backend.
#     terraform {
#       backend "remote" {
#         # The name of your Terraform Cloud organization.
#         organization = "example-organization"
#
#         # The name of the Terraform Cloud workspace to store Terraform state files in.
#         workspaces {
#           name = "example-workspace"
#         }
#       }
#     }
#
#     # An example resource that does nothing.
#     resource "null_resource" "example" {
#       triggers = {
#         value = "A example resource that does nothing!"
#       }
#     }
#
#
# 2. Generate a Terraform Cloud user API token and store it as a GitHub secret (e.g. TF_API_TOKEN) on this repository.
#   Documentation:
#     - https://www.terraform.io/docs/cloud/users-teams-organizations/api-tokens.html
#     - https://help.github.com/en/actions/configuring-and-managing-workflows/creating-and-storing-encrypted-secrets
#
# 3. Reference the GitHub secret in step using the `hashicorp/setup-terraform` GitHub Action.
#   Example:
#     - name: Setup Terraform
#       uses: hashicorp/setup-terraform@v1
#       with:
#         cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}

name: "Terraform Infrastructure Change Management Pipeline with GitHub Actions"

on:
  push:
    branches: [ "main" ]
  pull_request:
permissions:
  contents: read

jobs:
  terraform:
    name: 'Terraform'
    runs-on: ubuntu-latest
    environment: production

    # Use the Bash shell regardless whether the GitHub Actions runner is ubuntu-latest, macos-latest, or windows-latest
    defaults:
      run:
        shell: bash

    steps:
    # Checkout the repository to the GitHub Actions runner
    - name: Checkout
      uses: actions/checkout@v3

     # Install the latest version of Terraform CLI and configure the Terraform CLI configuration.
    - name: Install Terraform
      run: |
        wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg
        echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
        sudo apt update && sudo apt install terraform
          
    - name: Terraform Init
      env:
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      run: terraform init

    # Checks that all Terraform configuration files adhere to a canonical format
    - name: Terraform Format
      run: terraform fmt

    # Generates an execution plan for Terraform
    - name: Terraform Plan
      env:
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      run: terraform plan

      # On push to "main", build or change infrastructure according to Terraform configuration files
      # Note: It is recommended to set up a required "strict" status check in your repository for "Terraform Cloud". See the documentation on "strict" required status checks for more information: https://help.github.com/en/github/administering-a-repository/types-of-required-status-checks
    - name: Terraform Apply
      env:
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      run: terraform apply -auto-approve -input=false

Step 4:

Create a S3 bucket for example ‘01terraformstatebucket’ manually in the region ‘US-EAST-1’ -> enable public access

STEP 5:

Create a DynamoDB table for state locking with the Partition key ‘LockID’ and its type ‘String’

STEP 6:

Prepare the configuration files

1. Providers file

tf-cicd/providers.tf at main · chaitu24/tf-cicd

2. backend config

tf-cicd/backend_config.tf at main · chaitu24/tf-cicd

3. main config

tf-cicd/main.tf at main · chaitu24/tf-cicd

Commit the code

   new file:   backend_config.yaml
        new file:   main.yaml
        new file:   providers.yaml


 git commit -m "stateaction"
[main 3a0da80] stateaction
 3 files changed, 42 insertions(+)
 create mode 100644 backend_config.yaml
 create mode 100644 main.yaml
 create mode 100644 providers.yaml

 git push https://chaitu24@github.com/chaitu24/tf-cicd.git --all
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 20 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 830 bytes | 415.00 KiB/s, done.

Bucket gets created

Action’s pipeline