Unit Testing Terraform
Unit testing Terraform is a very different animal than unit testing, say Java. Anyone who is familiar with testing software is likely familiar with the test pyramid. The most isolated, granular and plentiful suites of tests are the unit tests. As we move up the test pyramid, each layer introduces more complexity, and as a consequence, they provide fewer, but more integrated and more expensive tests. Tests closer to the base of test pyramid (e.g. unit tests) can also be executed much faster than tests higher up on the test pyramid. As an example, a large suite of unit tests is expected to give near immediate feedback, taking just seconds to a few minutes to execute.
The quick feedback attributed to unit tests allows unit tests to be integrated into the build life cycle as a standard practice. Evidence of this can be found in build life cycle management tools, like Maven, where “test” is a build phase that by default executes a test goal tied to whichever unit test framework is included in the project’s dependencies. Indeed, the speed at which unit tests execute and the feedback they provide is at the heart of their value. Now, to muck that all up: enter Infrastructure as Code (IaC) with Terraform.
What is IaC with Terraform, Anyway?
Before we dive into the problems with unit testing Terraform, it’s worth discussing what IaC is and its benefits. IaC is really just applying software engineering practices and principles to infrastructure. With IaC, infrastructure is described as source code. The benefits are that IaC can be checked into source control, infrastructure can be easily reproduced by “building” IaC source code, the source code can be tested (sort of), linted, and integrated into a managed build life cycle. Such is the promise of IaC and Terraform, as it is a relatively well adopted IaC platform. And the promise is great. But the reality is that it’s also immature compared to general purpose (GP) programming platforms like Java. And there are also some gotchas.
Terraform Unit Testing “Gotchas”
One of the biggest gotchas (as was alluded to above) is unit testing. Unit testing IaC is well, different than unit testing GP languages. Take Java and JUnit, for example. For many systems, achieving 90% coverage with high quality unit tests and getting feedback from those tests on every build is not only possible, in many organizations, it’s commonplace. That’s in part because it’s realistic to account for nearly every logical path for each class, run those tests in near isolation and get feedback within a minute or two. None of that is realistic with with Terraform or really IaC, in general.
Accounting for branches with Terraform is difficult and some particulars with the language makes coverage impractical and not very valuable, anyway. Sure, you can work hard to try to have coverage for the sake of coverage. But if achieving your desired coverage target is not telling you anything useful, then it’s just a lot of time spent for little to no realized value. What’s more is that since infrastructure tests validate against provisioned infrastructure, then the more tests you have, the longer you have to wait for feedback. Whereas a Java unit test suite with robust coverage might run in seconds, a Terraform unit test suite with could take hours or more to complete, thus diminishing its value.
The final gotcha is the support of test frameworks. You basically have a couple of options when it comes to testing Terraform: Terratest and kitchen-terraform. Both are integrated with Terraform and neither is great. I suppose a third option could be just using InSpec, but that is a solution that’s really not very well integrated with Terraform. And tests that are not well integrated have a tendency to be omitted from the development process.
OK, How Can We Test Terraform, Then?
Terraform infrastructure tests spin up, tear down and/or update infrastructure to test different scenarios. Spinning up, tearing down and updating infrastructure takes time. When each test takes takes potentially minutes or more, having a large suite of tests that are packaged as part of the development process and which are run with great frequency, it’s important to target the most valuable tests for that test suite. This doesn’t mean tests aren’t delivered with various levels of granularity. They are! But it does mean that testing for every possible scenario diminishes the value of the feedback provided by the tests.
Instead of a test pyramid in the traditional sense, something else is needed to fill the holes and ensure that adequate testing and validation are being performed on infrastructure at multiple layers. That something else is affectionately referred to as the Swiss Cheese Model. The idea is basically that tests exist at various layers: modules, services, systems, monitoring, etc. with the understanding that there are holes in each layer. However, tests and validation are included at each layer ensure that each hole left open from one layer is covered by another. It’s discussed in more detail here: https://www.thoughtworks.com/talks/infrastructure-design-patterns-xconf-eu-2017
Available Testing Frameworks
The aim of Infrastructure as Code is to bring to infrastructure all of the practices that are enjoyed by general purpose programming languages, like Java. So, why shouldn’t it enjoy the same tooling? Java has automated unit test support via JUnit or Spock with Groovy or <insert your automated unit test framework here>. Why not leverage what already exists?
My team had a preference for Spock with Groovy. So, we worked on a Java Terraform API abstraction integrated with Maven. Using our Java Terraform API, we could invoke Terraform’s init, apply, destroy and out commands to help facilitate our tests. The bonus was that with Maven integration, we also got build life cycle and dependency management support (more on that in a future post). Now, we truly can write Terraform tests like we write Java, or rather, Spock Groovy tests with all the trimmings.
