A Jump Start on Terraform Testing

Published in

The Startup

9 min readOct 4, 2020

Within this article, I aim to quickly outline and inform you of some of the different forms of testing that can be implemented on your Terraform code.

We will cover:

Static Testing/Analysis
Linters
Basic Security Testing
Unit Testing
Integration Testing
Property Testing
E2E Testing
Example code and links

Static Testing/Analysis
What is Static testing?
Static testing is a software testing technique by which we can check the defects in software without actually executing it. Often you can find small issues that you may have missed when eyeballing your code.

What are the benefits of static testing/analysis?

Fast.
Stable.
No need to deploy resources.
Very easy to use.

Hashicorp provides you with a way to perform static analysis with the command — terraform validate. This command will allow for a quick check to make sure you have not made a trivial mistake with spelling, missing a bracket, or what I believe to be more useful; checking for unused variables.

All you will need to do here is run your terraform validate command in your code directory.

Note- If you want to do something with your output text you can always format to JSON with the flag -json .

What is a typical example of where this type of testing can help us?Duplication. To give an example:

I have followed a common Hashicorp standard of creating a proivider.tf file to store my provider information:

provider "aws" {
access_key = var.AWS_ACCESS_KEY_ID
secret_key = var.AWS_SECRET_ACCESS_KEY
region = var.region
version = “~> 2.56”
}

However, I have also copied code that already has a provider block nested within it. Therefore, I now have two provider blocks within my Terraform working directory, one in my provider file and one in my main file. If I run a validate command, this becomes clear:

$terraform validateError: Duplicate provider configuration
on provider.tf line 1:
1: provider “aws” {A default (non-aliased) provider configuration for "aws" was already given at main.tf:3,1–15. If multiple configurations are required, set the "alias" argument for alternative configurations.

Linters

What is a Linter?
A linter is a tool to analyse code for programmatic or stylistic errors. Although a linter is also another form of static analysis, for this article, I wanted to keep them separate to give an easier way to break down these sections. Also, it may help to keep these definitions separate for other tools or technologies.

After installing tflint you will be able to run tflint in the needed directory. Linters can pick up errors that would otherwise not be picked up by other methods of static testing. An example of this would be for provider-specific fields/resources; this can be shown with the below example.

resource "aws_instance" "testing_instance" {
ami = “ami-0ff8a91507f77f867”
instance_type = “t1.2xlarge” # invalid type!
}

If we do some static testing on this code — like terraform validate:

$ terraform validate
Success! The configuration is valid.

If we run our linter:

$ ./tflint
1 issue(s) found:Error: "t1.2xlarge" is an invalid value as instance_type (aws_instance_invalid_type)on main.tf line 3:
3: instance_type = "t1.2xlarge" # invalid type!

You can see that our linter finds the issue before having to attempt to run a terraform plan. Static testing combined with linting is a great way to test without having to create any infrastructure. These are ideal steps to include before committing to your SCM and should be considered mandatory.

Security Testing

What security aspects are we considering?
To better understand this, let us do back to why we are using infrastructure as code (IAC).

There are many benefits to IAC; it can allow someone to create an intricate network, spanning data centers, regions and subnets in the Cloud. One of the most common issues when starting to develop resources in the Cloud is that you expose your instances/hosts to the outside world, this is one of many security issues we want to check before deploying our code.

Tfsec is another tool we can use for static security analysis. If you check the README.mdof the tfsec Github repository, you will find that there is a large number of potential security flaws.

In the below image, you can see that tfsec find three sections within my code that have a fully open ingress security group. Open security groups are a common mistake that many make; this is partly due to the fact they have not set up correct routing to their instances.

Again tfsec is yet another form of static analysis; as aforementioned, I wanted to keep them separate to give an easier way to break down these sections.

What is a Unit Test in Terraform?

Within Terraform, a unit test can be simplified to, the testing of a Terraform module. Unit tests are used as a way to build confidence on a small, and logical grouping of your code — a unit.

The reason unit testing can be difficult in Terraform is because Terraform is full of dependencies. Think about a basic use-case of creating an EC2 instance on AWS. You are using an AWS provider; you are reliant on AWS’s API. It is for this reason that you will need to deploy your resources to test them; you cannot do this statically, this is one of the reasons that many will find it hard to describe unit tests in Terraform.

Many people would say that there are only integration tests in Terraform because of the reason I mentioned above — you are always reliant, or integrated with other units/groups of code/tools.

Good things about unit tests:

Still pretty quick — generally under 10 mins.
Gives a high level of confidence in individual units.

Weaknesses of unit tests:

You need to deploy your resources.
Requires you to write non-trivial code.

Terratest is a prevalent tool that is used to help write unit tests. Terratest is a Go library that makes it easier to write automated tests for your infrastructure code. It provides a variety of helper functions and patterns for common infrastructure testing tasks.

Every team can make use of unit testing with Terratest; it is straightforward to implement, especially when you only have a few modules to test.

The code I will be testing will create an EC2 instance and write ‘Hello, World!’ on a web server (see the end of the article). The unit test will look for the connection and check that the body says ‘Hello, World!’

There is a vast amount of information on Terratest — I encourage you to have a look yourselves.

Integration Testing

What is a Terraform Integration Test?
Integration tests within Terraform are tests that utilise multiple modules that work together, some of which may have dependencies on each other.

What are the benefits of integration testing:

Mostly stable (with retry logic)
You have high confidence in individual units working together.

Integration tests generally go through the following stages (imagine you have two Terraform modules):

Apply module 1.
Apply module 2.
Run validations to make sure everything is working.
Destroy module 1.
Destroy module 2.

Remember — integration testing will also mean you have to provision the resources!

An example; you may have some code that looks to deploy instances on AWS. You create a VPC module as well as an instance module. The VPC will be created before you provision the instances within the created VPC. Therefore, the integration test could test that the EC2 instances are reachable, and making use of VPC module.

After running our tests, we would hope to see that our tests have passed:

TestStagingInstances 2020–08–20T22:34:52+01:00 logger.go:66: Destroy complete! Resources: 8 destroyed.
--- PASS: TestStagingInstances (192.15s)
PASS
ok      github.com/mSm1th/terraform_testing 192.151s

When writing an integration test with Terratest, you have the option to write code in stages. This use of stages allows for more flexibility when testing locally.

Property Testing

What is a property/properties Test?
Property testing is used to inspect and validate specific properties that exist on your infrastructure. You are testing if the infrastructure conforms to a particular specification, whether that is a service is running, a file has the correct permissions or other similar actions.

Most of the tools that accomplish this are making use of domain-specific language (DSL), as a result of this, the code is pretty straightforward and easy to understand.

The downside of property tests is that you are testing things are there, not necessarily that they work, as opposed to some of the other tests that were previously noted.

I find that some of these tests don’t provide as much value. For example, if I am using Terraform Kitchen, following an extensive example, you can end up with a directory like:

ms@home:~/kitchenTerraform/example1$ tree.
└── test
├── assets
│ ├── aws_key
│ └── aws_key.pub
├── fixtures
│ └── wrapper
│ ├── main.tf
│ ├── outputs.tf
│ └── variables.tf
├── Gemfile
├── Gemfile.lock
├── integration
│ └── extensive_suite
│ ├── centos_attributes.yml
│ ├── controls
│ │ ├── inspec_attributes.rb
│ │ ├── operating_system.rb
│ │ ├── reachable_other_host.rb
│ │ └── state_file.rb
│ ├── inspec.yml
│ └── ubuntu_attributes.yml
├── kitchen.yml
├── main.tf
├── outputs.tf
├── test_it.sh
└── variables.tf7 directories, 19 files

Another interesting point is that you can use Terratest to do property testing as well, by making use of shell scripts. You can run a script on the host, passing in arguments if you wish and check the return code of the script.

E2E Testing

End to end testing in Terraform is what you may expect. You are testing all of your code from beginning to end. In most cases, you will be deploying this test in a pre-prod environment or an environment that is a near replica of your production environment.

In theory, you could write all of your tests in the same way in which you wrote the integration tests and then run multiple sets of integration test, which will ultimately amount to your E2E testing. However, this seems rarely done.

What is the main reason for this? It goes back to the test/testing pyramid.

E2E tests take a long time. It usually is more time effective to do more unit tests with some integration tests thrown in, than full E2E testing.

E2E tests are (generally speaking):

Too slow and too brittle

Being too slow is easy to understand; deploying your entire architecture from scratch and then un-deploying it can take a long time.

The feedback loop is too slow. Suppose you compare this to unit tests or even static analysis. These methods are quicker to write, provide rapid feedback and generally, add more value.

The term brittle may seem a bit odd. Within Terraform, there are many components, the more components/resources you create, the more likely you are to run into a ‘flaky’ error.

Some tests will fail due to unforeseen errors, which we could easily misconstrue as errors within our code. You can get rid of many errors with retries and the like, however when you increase the number of components and the amount of time taken to provision you are just more likely to run into flaky errors.

So how do we E2E test?

The simple answer is to have an environment which you do not discard or demise. You can run your code E2E on the server and validate that the changes. You are attempting to replicate your production environment.

In conclusion

There are lots of things to look into for testing in Terraform. A lot of the way people test comes down to the scale and criticality of what they are building.

If it is small sections of infrastructure or simple applications on the cloud, people may eyeball code and run a terraform validate against the code. Some may think it is a good idea to run your tests in a container for consistency and quickly clearing up.

Hopefully, this article has given a brief insight into testing in Terraform and convinced you to give some of these tools a try. Also, if you are keen to learn more about Terraform I highly recommend the book, Terraform Up & Running by Yevgeniy Brikman, it helps round out ones understanding of Terraform.

Example Code and links

I have made all the code (TF and Go code) available at the below GitHub repository, while the code is not perfect it should help you to test basic examples.

https://github.com/mSm1th/terraform_testing

Links of tools discussed in this article:

Some other testing articles that are worth a read: