Streamlining Terraform at TeamSnap
We’ve been using Terraform at TeamSnap for over 4 years now and have regularly tried to improve our processes to remove the pain points of using Terraform in practice, at scale, and bring back the joy most of us experienced in our “hello world” experiences when we realized how amazing declarative infrastructure-as-code could be.
We’ve been through several different patterns for organizing our Terraform code, but regardless of how all of that is organized, you’re probably going to run into a common problem.
It can take some time and some iterations to build private/custom Terraform modules at a company. After they’re built, they need to be kept up-to-date and improved on. After they’re updated, those changes need to be rolled out to any Terraform code that’s using them.
How can you build in a way where more Terraform modules and more Terraform code that’s calling on those modules feels like it’s making short-term and long-term development easier without swiping the “Tech Debt (or future Toil) Credit Card ©” every time you build or use a module?
Handling Change
At TeamSnap, one of the early improvements we had made was to store all of our custom Terraform modules in a monorepo, and follow a process we had written for how to properly git tag the changes we made, whether it was adding a new module or updating an existing one. Part of that manual release process was to update the CHANGELOG.md file. Not only was this tedious and painful to collaborate on if multiple people were making changes at once, but we still had to remember to update everything that was using the modules and hope the changes didn’t break anything. Needless to say, the reality was that we’d rarely update anything downstream when we’d make changes, and we’d just use the latest version of the module when we were creating something new.
Here’s a example of what that change log looked like:
Here’s an example of what our Terraform deployment code looked like that referenced the module:
Introduction to Conventional Commits and Semantic Releases
Let’s solve for 2 problems at once:
- Multiple people making changes to our modules should be easy.
- Tracking those changes should be foolproof and painless.
Step 1: Ditch the module monorepo.
This worked well enough for us for a while as a small team with small infrastructure, but eventually grew to be a real problem. With our monorepo, all our Terraform deployment code was referring to that single repo URL so a change/new git tag on the monorepo didn’t necessarily mean an update was relevant to the deployment code. By using separate repos for each module, we knew that any update to a module repo meant that any Terraform code using that repo needed the update. That’ll come up again in a few paragraphs… bear with me.
Step 2: Adopt conventional commits.
Some of you may be thinking, “why would you manually write a change log when everything is in git already?” Well, git commits can be a pain to sort through if you’re just trying to understand which new changes apply to whatever downstream dependency you’re trying to update. It’s doable, but it’s not painless. It certainly helps if the entire team taking care to write descriptive commit messages, so lets start there.
“Conventional commits” were introduced to our team by our Senior Platform Director, Trey Tacon, and was quickly adopted due to how big of an impact it made on our quality of life. Check out conventional commits in depth here, but lets focus on the single-sentence description they wrote:
A specification for adding human and machine readable meaning to commit messages
Take special note of “… and machine readable meaning …” and keep that in the back of your mind, because this is about to get interesting.
To wrap up this section, the general gist is to write your commit messages like this:
- git commit -m "feat: [ISSUE-123] cool new feature"
- git commit -m "fix: [ISSUE-345] squash that bug"
One immediate benefit of adopting this style on our team is that it encouraged better commit habits and “story telling” around what a PR was actually trying to accomplish.
Now let’s get to the fun part, automation.
Step 3: Adopt semantic releases with GitHub Actions.
For those who aren’t familiar with semantic versioning, check out semver.org. The idea is to establish conventional meaning around versioning, and this snippet from semver.org covers the general concept:
Given a version number MAJOR.MINOR.PATCH, increment the:
MAJOR version when you make incompatible API changes,
MINOR version when you add functionality in a backwards compatible manner, and
PATCH version when you make backwards compatible bug fixes.
It’s time to revisit that “… and machine readable meaning …” phrase we read earlier. By adopting conventional git commits, we’re able to use an open source node module to read our commits and automatically generate a changelog, organized in git releases, with each pull request automatically tagged and released appropriately upon merge to the main branch.
Rhetorical question: That sounds amazing, but must be a pain to configure, right?
Serious answer: It’s so easy you have no excuse not to go add it to your repos, right now. Check this out.
Create two files in any repo you want:
- A GitHub Actions workflow. Naming things is hard, lets just call it
.github/workflows/release.yml. You can copy and paste these contents with no changes,secrets.GITHUB_TOKENis a special token GitHub manages for you, for exactly this type of purpose, where we want to automatically author git releases. Read more about thesemantic-releaseNPM module here:
2. Create a .releaserc file in the root of the repo with these contents:
That’s it. You’re done. As long as you write conventional commits in your pull requests, you’ll have automatic tags/releases with generated release notes that looks like this:
We’ve solved for releasing changes quickly and easily. How do we handle the downstream?
When I say downstream, I’m talking about all the Terraform code that’s using our custom (and in private repos) Terraform modules. The main “disadvantage” of easily iterating on your Terraform modules is that your downstream quickly becomes out of date. The more places you were able to use your module, the more places you need to keep up-to-date, which quickly gets out of hand.
Dependabot for Private Terraform Module Repos
Dependabot is an amazing tool that can automatically discover which dependencies in your code have newer versions available to upgrade to, and will automatically open detailed pull requests with the version updated for the outdated dependency it found. If you’re not already familiar with it, read this blog post by Github for an overview.
We use Dependabot at TeamSnap for all of our repos, whether its GO, Ruby, NodeJS, or Elixir, but usually we’re tracking open source dependencies. Can we leverage it for our Terraform module dependencies, even though we’re using git releases instead of a Terraform registry and our git repos are private? YES!
Create a file named .github/dependabot.yml in your Terraform repos that use your private modules. Give it these contents, but we’re going to need to take ONE manual step before it’ll work:
.github/dependabot.yml exampleHere’s what the Dependabot pull requests will look like, showing all the new changes you should be aware of as well as links to individual commits. This image is a real pull request for TeamSnap’s staging environment Terraform code, which relies on our private CloudSQL module:
The manual step
Dependabot needs to be granted access to private GitHub repositories, as described in this GitHub blog post.
Additionally, you’ll need to create a personal access token with Repo scopes, and add it as an organization secret for dependabot in GitHub (named DEPENDABOT_REPO_SCOPE if you use the .github/dependabot.yml we shared earlier).
You can find the secrets configuration page for Dependabot if you replace <YOUR_ORGANIZATION> with your actual organization name in this URL: https://github.com/organizations/<YOUR_ORGANIZATION>/settings/secrets/dependabot. If you don’t use that direct link, take care that you’re adding the secret for Dependabot specifically, and not for Actions or Codespaces.
Conclusion
This setup has made TeamSnap’s Terraform module version management and downstream updates painless, regardless of how many modules we create or how many Terraform repos depend on our modules.
