Managing Terraform Modules with GitHub Actions

Automate your Terraform module releases with GitHub Actions

Introduction:

Before we dive into the details, let me clarify the focus of this post. This article is not a comprehensive tutorial on learning Terraform or GitHub Actions. Instead, we will explore a practical use case where GitHub Actions automates the release process of Terraform modules within a mono repository. If you’re already familiar with Terraform and GitHub Actions and want to streamline your module management and release workflows, you’re in the right place!

Welcome to the Terraform Modules Repository, where we’ll show you how to harness the power of GitHub Actions to automate the release of Terraform modules. Let’s get started!

Repository Structure

The repository is structured as follows:

modules/
    - project
    - simple network
    - ...

Each module directory contains, at a minimum, the following files:

• README.md: Module-specific documentation.
• main.tf: The main Terraform configuration file.
• variables.tf: File for defining variables.
• versions.tf: File for specifying Terraform versions.

Conventional Commits and Release Please

One of the powerful aspects of managing Terraform modules with GitHub Actions is the synergy between Conventional Commits and the Release Please action. These two components work together seamlessly to automate module releases and ensure clear communication about the changes introduced in each release.

Conventional Commits: A Common Language

Conventional Commits serve as a standardized and human-readable format for commit messages. They follow a defined convention that includes a type, an optional scope, and a description. For example, a Conventional Commit might look like this:

feat(modules/network): add VPC module

In this commit message:

• feat: indicates that a new feature has been added.
• modules/network: provides context or a scope for the change.
• add VPC module: is a clear description of the change.

The power of Conventional Commits lies in their ability to provide a consistent and structured way to communicate changes in your codebase. They make it easy for both humans and automation tools to understand the purpose and significance of each commit.

Release Please: Automated Versioning and Changelogs

Release Please is a GitHub Actions tool that automates the process of creating new releases for your modules. It relies on Conventional Commits to understand the nature of changes and generate accurate version numbers, changelogs, and GitHub releases.

Here’s how Conventional Commits and Release Please work together:

Commit Messages: Developers use Conventional Commits for their changes, following the established conventions. This means that every commit is tagged with its type, scope, and description, creating a clear historical record of changes.

Automated Release Requests: When changes are pushed to the main branch and compared with the latest tag, Release Please identifies the modified modules. It uses the commit messages to understand the nature of changes in each module.

Semantic Versioning: Release Please applies semantic versioning rules to determine the appropriate version bump (e.g., patch, minor, major) based on the commit types. For example, a commit with fix might trigger a patch version bump, while feat could trigger a minor version bump.

Changelog Generation: Release Please generates detailed changelogs for each module release based on the commit messages. These changelogs provide a comprehensive overview of what has changed in each release.

Pull Requests for Releases: Release Please automatically creates pull requests for each module release. These pull requests contain the changelog, version bump, and other release-related information.

GitHub Releases: Once the pull requests are merged, Release Please creates official GitHub releases, complete with version tags and changelogs. Users can easily access release notes and download specific module versions.

By combining Conventional Commits and Release Please, you can streamline the release process, reduce manual work, and ensure that your module releases are well-documented and easy to understand. It’s a powerful approach that enhances collaboration and transparency in your module management workflow.

Modules PR Workflow

This workflow runs every time a pull request is created with changes in the modules/ directory. Its purpose is to identify the modules affected by the pull request and perform specific tasks for each module:

Step 1: Find Changed Modules

Description: This step identifies the modules that have been modified in the pull request by comparing the changes with the main branch. It does so by performing the following actions:

• Checking out the repository to access its contents.
• Retrieving the list of changed files in the pull request.
• Extracting module names from the changed file paths using regular expressions.
• Removing duplicate module names from the list.

Purpose: This step is essential for determining which modules have been impacted by the pull request. It helps in targeting the specific modules that require updates, ensuring that only relevant actions are performed.

Step 2: Update README for Changed Modules

Description: For each module identified in the previous step, this step generates and updates the module-specific README.md file with Terraform documentation. It achieves this using the terraform-docs GitHub Action, which automates the process of updating documentation.

Purpose: Automating the documentation update process ensures that the README files for modules are always up-to-date. This improves transparency and makes it easier for users to understand module changes.

Step 3: Perform Basic Review of Terraform Code

Description: This step performs a basic review of the Terraform code for each changed module. It executes the following Terraform commands:

• terraform init: Initializes the Terraform environment.
• terraform fmt -check: Checks if the code is correctly formatted.
• terraform validate: Validates the Terraform configuration.

Purpose: The purpose of this step is to ensure code quality and adherence to Terraform best practices. It helps catch formatting issues and configuration errors early in the development process.

Here is the full version of the code:

name: Modules PR Workflow

on:
  pull_request:
    paths:
      - modules/**

permissions:
  pull-requests: write
  contents: write

jobs:
  find-changes:
    name: Find in modules folder
    runs-on: ubuntu-latest
    outputs:
      changed-modules: ${{ steps.changed-modules.outputs.result }}
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 2
      - name: Get changed files
        id: changed-files
        uses: tj-actions/changed-files@v39
        with:
          files_ignore: '.github/workflows/**'
      - name: Get changed modules
        id: changed-modules
        uses: actions/github-script@v6.4.1
        with:
          script: |
            const paths = "${{ steps.changed-files.outputs.all_modified_files }}";
            const regex = /modules\/([^/]+)/g;
            const matches = paths.match(regex);
            if (!matches) {
              return [];
            }
            // Extract the module names from the matches
            const moduleNames = matches.map(match => {
              return match; // Extract the entire match
            });
            // Remove duplicates and return the unique module names
            const uniqueModuleNames = Array.from(new Set(moduleNames));
            console.log(uniqueModuleNames);
            return uniqueModuleNames;

  update-readme:
    name: Update README for changed modules
    runs-on: ubuntu-latest
    needs: find-changes
    strategy:
      matrix:
        module: ${{ fromJson(needs.find-changes.outputs.changed-modules) }}
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 2
          ref: ${{ github.event.pull_request.head.ref }}
      - name: Render terraform docs and push changes back to PR
        uses: terraform-docs/gh-actions@v1
        with:
          working-dir: ${{ matrix.module }}
          output-file: README.md
          output-method: inject
          git-push: "true"

  module-review:
    name: Individual module review
    runs-on: ubuntu-latest
    needs: find-changes
    strategy:
      matrix: 
        module: ${{ fromJson(needs.find-changes.outputs.changed-modules) }}
    defaults:
      run:
        working-directory: ${{ matrix.module }}
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - run: terraform init
      - run: terraform fmt -diff -check
      - run: terraform validate

Release Please Workflow

This workflow is triggered when changes are pushed to the main branch, specifically within the modules/ directory. It is responsible for creating new releases for the modified modules.

Step 1: Find Release Changes

Description: This step identifies the modules that have changed by comparing the latest tag with the main branch. It performs the following actions:

• Checking out the repository.
• Retrieving the latest release.
• Comparing changes between the latest release and the main branch.
• Identifying modules affected by the changes based on file paths.

Purpose: This step determines which modules have seen changes since the last release. It’s a crucial step in automating the release process to ensure that only modified modules are considered for release.

Step 2: Create Pull Requests for New Releases

Description: For each affected module, this step uses the Release Please tool to create a pull request that initiates a new release. Release Please automate various aspects of the release process, including generating a changelog and creating GitHub releases.

Purpose: By automating the release process with Release Please, this step ensures that new module versions are created consistently and with proper documentation, making it easier for users to track changes and updates.

Step 3: Create GitHub Releases

Description: After the pull requests for new releases are merged, this step creates GitHub releases for the modules. It uses the same Release Please tool to manage the release process, including the creation of release notes and version tags.

Purpose: The final step in the release process ensures that the newly created modules are officially published as GitHub releases. Users can then easily access release notes and download specific module versions.

These detailed descriptions provide a deeper understanding of each step’s role in your GitHub Actions workflows for managing Terraform modules. They highlight the importance of automation in simplifying module development and release processes.

Here is the full version of the code:

name: Release please workflow

on:
  push:
    branches:
      - main
    paths:
      - modules/**

permissions:
  contents: write
  pull-requests: write
  packages: write

jobs:
  find-release-changes:
    name: Change finder for modules folders
    runs-on: ubuntu-latest
    outputs:
      module-paths: ${{ steps.change-finder.outputs.result }}
    steps:
      - uses: actions/checkout@v4
      - run: npm install @octokit/action
        name: Install Octokit
      - id: change-finder
        uses: actions/github-script@v6.4.1
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} 
        with:
          script: |
            const { Octokit } = require("@octokit/action");
            const { execSync } = require('child_process');
            const octokit = new Octokit()
            const [owner, repo] = process.env.GITHUB_REPOSITORY.split('/');
            const latestRelease = await octokit.request('GET /repos/{owner}/{repo}/releases/latest', {
              owner: owner,
              repo: repo,
              headers: {
                'X-GitHub-Api-Version': '2022-11-28'
              }
            })
            console.log(`latest release: ${JSON.stringify(latestRelease.data)}`);
            execSync('git pull --tags');
            const status = execSync(`git diff --name-only ${latestRelease.data.tag_name} origin/main`, { encoding: 'utf-8'});
            console.log(status);
            const changes = status.split('\n');
            let modules_paths = new Set();
            for (const change of changes) {
              if (change.startsWith('modules/')) {
                  const library = change.split('/')[1];
                  modules_paths.add(library);
              };
            }
            modules_paths = Array.from(modules_paths);
            console.log(modules_paths);
            return modules_paths;

  validate-output:
    runs-on: ubuntu-latest
    needs: find-release-changes
    steps:
      - run: echo ${{fromJson(needs.find-release-changes.outputs.module-paths)}}

  release-pr:
    runs-on: ubuntu-latest
    needs: find-release-changes
    strategy:
      fail-fast: false
      matrix:
        module: ${{fromJson(needs.find-release-changes.outputs.module-paths)}}
    steps:
      - uses: google-github-actions/release-please-action@v3
        id: release-please
        with:
          path: modules/${{ matrix.module }}
          changelog-path: CHANGELOG.md
          token: ${{ secrets.GITHUB_TOKEN }}
          release-type: terraform-module
          package-name: ${{ matrix.module }}
          monorepo-tags: true
          command: release-pr

  release-please-release:
    runs-on: ubuntu-latest
    needs: find-release-changes
    strategy:
      fail-fast: false
      matrix:
        module: ${{fromJson(needs.find-release-changes.outputs.module-paths)}}
    steps:
      - uses: google-github-actions/release-please-action@v3
        id: tag-release
        with:
          path: modules/${{ matrix.module }}
          changelog-path: CHANGELOG.md
          token: ${{ secrets.GITHUB_TOKEN }}
          release-type: terraform-module
          monorepo-tags: true
          package-name: ${{ matrix.module }}
          command: github-release

Prerequisites for Module Releases

To successfully perform module releases, ensure the following conditions are met:

• There must be at least one folder under modules/.
• At least one tag must be pushed to the repository.
• Releases must be published on GitHub through the pull requests created by the Release Please workflow.

Summary:

In this post, we explored a practical use case for GitHub Actions: automating the release process of Terraform modules within a mono repository. We delved into two key workflows, the “Modules PR Workflow” and the “Release Please Workflow,” each contributing to a seamless module management experience.

The “Modules PR Workflow” streamlines the pull request process by identifying modified modules, updating their documentation, and performing basic code reviews. This automation enhances collaboration among developers and ensures that module documentation is always up-to-date.

The “Release Please Workflow” takes care of the release process, creating pull requests for new module versions and automating changelog generation. It simplifies version management, making it easier for users to track changes and access updated modules.

Final Words:

Automation is the key to efficient software development, and GitHub Actions empowers us to streamline processes like never before. By applying these workflows to your Terraform modules repository, you can save time, reduce manual errors, and provide a better experience for your users.

As you embark on your journey of managing Terraform modules with GitHub Actions, remember that automation is about more than just convenience; it’s about improving the quality and reliability of your codebase. Whether you’re a seasoned developer or just starting, harnessing the power of automation will make your module management a breeze.

Happy Terraforming, and may your Terraform modules always be up-to-date and easily accessible to your users!

Upcoming Medium Post: Workflow Demonstrations

We have prepared a comprehensive demonstration of the GitHub Actions workflows used in this article. In a separate Medium post, we will provide step-by-step details and visual guides to help you understand how these workflows automate Terraform module management. Stay tuned for the upcoming Medium post, where we’ll walk you through the practical implementation of these workflows, showcasing their effectiveness in managing Terraform modules efficiently. Don’t forget to follow our GitHub repository to receive updates and notifications when the Medium post is published.