Using Terraform on GitHub Actions to Manage Team Membership on GitHub

At Cognite, we use the GitHub Terraform provider to manage our organization’s users and teams. This is a convenient way to handle access rights for all GitHub users and their team memberships through code. It provides clarity, visibility, and consistency in the onboarding and offboarding processes. It also ensures an overall better experience for everyone involved. We can control the state of the users, their permissions, and team membership by changing a “single source of truth.” Every change is requested through a pull request (PR), which has to be approved by the relevant parties.

We use Atlantis to run Terraform remotely on a dedicated instance. Atlantis is a tool that listens to webhooks from GitHub and executes Terraform commands remotely. Atlantis is self-hosted, and it runs on our Kubernetes cluster. A webhook is configured on a GitHub repository that triggers a Terraform plan/apply execution whenever a change is made to that repository. The Atlantis instance is paired with the GitHub repository with a secret that needs to be configured on both ends.

This setup works well for us and our main GitHub organization. However, to facilitate collaboration with customers, we needed to replicate this setup to several GitHub organizations created for external collaboration. We wanted to keep these organizations completely separated from our main GitHub organization environment and avoid sharing any use of resources between them.

This motivated us to look for an alternative way to run Terraform on GitHub to save us from the trouble of maintaining several instances of Atlantis for every single GitHub organisation that we create. Based on our research, here’s an overview of the potential challenges and benefits associated with each of the alternatives we considered.

How to run Terraform

There are alternative ways to run Terraform on GitHub Actions that could, to some extent, replace the functionalities provided by Atlantis.

• The Hashicorp setup-terraform GitHub Action could be used to set up a Terraform CLI in the GitHub Actions workflow. This action allows us to run Terraform commands in GitHub actions using the GitHub Actions run syntax. The action also installs a wrapper script around all Terraform calls that exposes STDOUT, STDERR, and the exit code as outputs, which are available to subsequent steps in the workflow.
• Another alternative to run Terraform on GitHub Actions is to use the Terraform installation on the GitHub runner itself. This way, we don’t need the help of the Hashicorp setup-terraform GitHub Action, but we miss out on the benefits provided by the wrapper script. This script is beneficial for us, since we want to use the output of the Terraform commands in order to print it out in a PR comment.

One drawback of using Terraform on GitHub Actions is the lack of directory detection. If there are multiple directories of Terraform in the same repository, we need to set up actions that will execute the Terraform commands per directory. Additionally, the actions will run regardless of whether the state in a directory was changed or not.

How to store the Terraform backend

There are also multiple options for configuring and storing the Terraform backend. Terraform can either use a local or a remote backend storage. However, since GitHub Actions will run on a different instance every time a workflow is started, the local backend storage is not an option for us, as it will be lost once a workflow is completed. We could choose between the following options:

• Store the Terraform state file using a Google Cloud Platform (GCP) storage bucket. In this case, we need to create a GCP service account and a GCP storage bucket that will only be accessible from the service account. We also need to provide GitHub with credentials to this service account.
• Store the Terraform state file using GitHub artifacts. We can persist workflow data on GitHub after a workflow is completed by uploading the data as an artifact to GitHub. While the artifact is easily accessible within the workflow and can be shared between different jobs, it is much less trivial to achieve the same functionality between workflows. The artifact can be downloaded using the GitHub REST API, which will require some scripting. The retention period for a push or a pull request is 90 days, and in the case of a PR it restarts each time someone pushes a new commit to the pull request. The other option is to store the artifact elsewhere, but this is the equivalent of simply storing the state file in a GCP storage bucket.

How to secure the storage of the Terraform backend

Our priority is to maintain a high level of security — in other words, limited access to and protection of the resources in use. Depending on the Terraform backend storage choice, the options for securing the storage will vary.

• To manage access to the GCP resources, we can use IAM roles, and to handle encryption on GCP, we can use GCP’s data encryption options. By default, Cloud Storage always encrypts the data on the server side, before it is written to disk and manages the encryption keys on users’ behalf.
• To gain access to GitHub artifacts, we need to interact with GitHub’s Artifacts API using an access token with read permissions on the “repo” scope. The GITHUB_TOKEN, automatically made available to Actions workflows on pull requests, including those from forking repos, has the necessary permissions and would expose the artifacts to all GitHub users.

How to secure the storage of secrets and credentials used

In order to enable user management of a GitHub organization in the way discussed in this blog post, we rely on using sensitive information such as credentials to establish connections to relevant parties and to grant us permissions for relevant actions. We need to provide the following secrets to the workflow:

• GH_USER_TOKEN — a GitHub personal access token with “repo” scope permissions that grants the ability to conduct certain actions on our GitHub organization, such as creating and configuring team memberships.
• SERVICE_ACCOUNT_KEY — to establish a connection to the storage bucket containing the Terraform state in GCP. We also need to specify the GCP_PROJECT that the bucket is created in.

We entrust the storage and encryption of these secrets to GitHub Secrets. GitHub Secrets are encrypted environment variables that can be created at a repository or organization-level. The secrets are intended to be used in GitHub Actions workflows. The values of the secrets are encrypted before they reach GitHub, and remain encrypted until they are used in a workflow. Repository-level secrets are accessible by all users that have write permissions to the repository. This means that every user with write permissions to the repository that manages team membership on GitHub with Terraform will have access to the secrets.

In order to limit the access to the secrets, we will keep the number of people who have write permissions to the repository to a minimum. In this way, users that only have read permissions to the repository will not have access to the repository secrets. Consequently, they will not be able to make any use of the secrets, even if they fork the repository and try to run a workflow. These users will still be able to create a PR from their forked repository which will be subject to a review before it can be merged.

Final setup

After considering all the alternatives, we decided on a final setup that seemed optimal for our current needs. Here is how the complete workflow looks like:

We use the setup-terraform GitHub action to execute the Terraform commands. The action installs a wrapper script around all Terraform calls that exposes STDOUT, STDERR, and the exit code as outputs, which we want to use in a next step in the workflow to post the output of the terraform plan as a comment to a PR. For the purpose of executing Terraform we created a new GitHub user and a personal token for this user with “repo” scope permissions which enable the user to execute all the necessary Terraform commands. The authentication to GitHub is based on this user token available to the workflow through the GH_USER_TOKEN secret mentioned earlier.

For the sake of reliability and simplicity, we use a GCP storage bucket for the remote state instead of using GitHub to persist workflow data, since downloading the data requires some additional effort, and since the data is only available for 90 days (if no changes are made). The GCP login is performed via a GitHub action using a service account key stored as a secret.

Here is how we configured all of this in a workflow:

There are a limited number of engineers who are given write permission to the repository that manages the team membership on GitHub with Terraform. Additionally, the repository is configured with branch protection to require approval before changes are merged, up-to-date branches, and successful status check for the workflow to make sure everything works as expected.

With this setup, the state file does not contain any sensitive information, and people who don’t have write permission to the repository are not able to extract any sensitive information from the GitHub runner, as they are simply not able to even run a workflow.

The work elaborated in this article was lead, checked and approved by my super skilled tech lead Carlos Lopes Pereira and I would like to acknowledge his help in guiding me.

How are you handling access and team membership for GitHub users across separate organizations? Let us know in the comments below.