Unlocking the Power of Terratest — Part 1

How to use terratest to create meaningful and insightful tests for terraform code.

Introducing Terratest

As a DevOps engineer, there is no shortage of tools to use. We have tools for building, testing, monitoring, automation… heck we even have tools for building other tools. So when faced with the decision of whether or not to use a new tool, it’s important to understand, not only how to use it, but when to use it, and more importantly, what are the problems we can solve by using this tool.

Terratest has a pretty simple premise. For the unfamiliar, Terratest is an open-source testing framework created by Terragrunt that includes helper functions for testing terraform code by providing a simple, reusable, and easy-to-use interface for creating and running automated tests on cloud resources. It’s able to apply terraform code, provision the resources, run the test cases, and then destroy everything. Terratest is most beneficially used for testing terraform modules to make sure that the modules work as expected before being consumed to create real infrastructure. That is why terratest will most often be used to spin up short lived, temporary resources to run tests against, and then immediately destroy all resources.

For anyone that’s worked with terraform, the obvious advantage of using an approach that actually deploys resources is that a simple terraform plan won’t always catch every possible error. There are some things that are on the providers side that terraform can’t know until it runs the apply. So having a tool to run a terraform apply and destroy is useful in its own right.

But if all we cared about was running a successful terraform apply and destroy, it wouldn’t warrant learning and implementing a new tool just for that. We could just as easily run the apply and destroy in a pipeline anytime we make a change to a terraform module. The real power of Terratest comes from the ability to run tests on our terraform code that will give us real insight into the functionality of our infrastructure.

There’s a common pitfall of testing for basic or obvious tests that don’t really provide any use. For instance, testing a terraform module that creates an s3 bucket to make sure it creates the s3 bucket is almost as bad as testing whether 1 == 1. We can trust terraform to create what we tell it to. And if there is a syntax error in the terraform code, it will be obvious when the apply fails and tells us what mistake we are making.

Tests should be insightful and designed to catch instances when the apply will still complete successfully, but the final infrastructure is not what we are expecting. That’s the sweet spot where terratest can truly shine.

In part one of my Unlocking the Power of Terratest series, I will explore how terratests can be used to verify the interconnectivity between resources and the final functionality of complex modules.

This article won’t spend a lot of time on the HOW aspect of running terratests, for that there are countless tutorials online to choose from. Rather, this article is designed as more of a guide of WHEN to use terratest and what kind of tests it should be used for.

Interconnectivity and Functionality of Resources

Let’s say we have a terraform module that creates an ECS cluster and runs tasks on that cluster. If you’ve ever created an ECS cluster using EC2 as the launch type, you’ll know we need to create a decent amount of infrastructure to run ECS on top of. Not only do we need to create the ECS cluster, ECR repository, ECS service, and ECS task definition, but we also need to create the EC2s that ECS will use to place tasks on. And that includes an ASG, launch template, capacity provider, etc. And we’ll also need to create a load balancer to send traffic to our tasks running on ECS. For that we’ll need an ALB, target group, listeners, etc. Not to mention all the additional resources you’ll need to properly configure your networking and security, such as IAM roles and security groups.

Yes, terraform is fully capable of creating each of these resources and we can trust that terraform will create what we’ve declaratively defined. But what about how these resources interact with each other? A terraform apply will tell us that all the resources have been created, but what terraform can’t tell us is whether our EC2 instances have joined the ECS cluster, or if the tasks running on ECS are in a running state, or if our app is even accessible through the load balancer. These are behaviors that happen outside of the resource creation.

The interconnectivity between resources and their functionality is just as important as the creation of the resources. If not properly tested, we could end up with a broken environment that passes a terraform apply flawlessly.

This is exactly the type of terraform module that could benefit from using terratest to validate its functionality. Let’s look at some examples of how we would do that:

Instance Count

Let’s say we want to make sure that the EC2 instances are joining the cluster properly. There are a few issues that could cause our instances to not be able to join the cluster, such as an issue with the ECS agent or IAM permissions. We want to make sure that any changes we make to the module don’t affect the ability of our instances to join the cluster.

For this, we will create a test case in our terratest to make sure that our cluster’s instance count is as expected.

This test functions as an important check on the terraform module to make sure that the EC2s and autoscaling group are properly associated with the cluster. It uses the AWS SDK to get the instance count of the cluster and assert that it matches the expected value. Terraform alone would tell us that the instances have been created, but only terratest can tell us that the instances have joined the cluster.

Running Task Count

ECS is designed to run tasks on the cluster, but there are a multitude of issues that could cause our tasks to fail before reaching a steady state. It could be related to permissions, network connectivity, or even an issue with the application the task is running. Terraform can tell us that the ECS service is created and starting tasks, but terraform has no idea if those tasks are constantly failing or stopping. Enter terratest:

This terratest function uses the the AWS SDK to get the running task count of the service and assert that it matches the expected value. If there is an issue with how the task is configured, terratest will inform the user.

Network Health Checks

Another important area that requires testing is the networking. It’s great that terraform can verify that it’s built all our resources, but that won’t tell us that our resources are properly talking to each other. In our ECS example, we are creating a load balancer to send traffic to our tasks running in the cluster. A good way to verify that our endpoint can reach the application is to make sure we have health checks that aren’t failing. That is something we can test with terratest.

This terratest function will check the health of our target group to make sure that the target is healthy. This will help us verify that traffic can properly flow to our ECS cluster.

Conclusion

I hope that through the ECS example it has become clear how terratest can play an important role in our testing suite.

What we’ve learned is that while terraform can provide a lot of information about our module from a successful apply in a testing environment, there are many more factors that affect the functionality of our modules. Testing the final product of a module allows the user to know that all the separate components of the infrastructure our properly configured.

While this kind of testing can be done through other methods and at different stages in the deployment pipeline, designing our tests to run on each terraform module by directly calling the module can make sure that the building blocks of our infrastructure are all functioning properly. That way, when the modules ultimately get consumed, we can trust that the infrastructure they build will be fully operational and act as expected.

By performing thorough testing, users can have confidence in the stability and reliability of the module, and ensure that it will continue to function properly over time.

This was the first installment in my 3 part series on terratest. In part 2, I will discuss the second common use case where terratest can help users test their terraform modules.