Automating IaC Integration Tests with Terraform, GitHub Actions and AWS.

Published in

Globant

7 min readAug 28, 2020

Written with the assistance of: Brayan Andres Bautista Calderon and Jorge Ernesto Guevara Cuenca. Special acknowledgement for their help and support.

Whenever you are developing a new application you try to minimize the risk of failure, testing the app functionality, security breaches or compliance. If you are working on a development framework that has achieved a certain level of complexity, the tests ought to be automated using CI/CD pipelines deploying and testing applications in various stages up to production. All of this is very important from the application side but, What about the infrastructure side supporting the application?. This side of the equation has to be considered as well given that you can have the most robust, secure and reliable application but if the underlying infrastructure (servers, DB, networking, firewalls, etc.,) has important flaws, your application can become worthless.

Having your infrastructure as code or IaC, has a few advantages including the possibility of automating the testing of your infra before deploying your application, decreasing the risk of issues going into different stages and most importantly before going into production.

There is quite a few IaC tools around, but for this example we are going to be using Terraform. This tool has been written with Golang which allow us to use the language testing package, terraform libraries and other Go libraries such as Terratest. You can use terratest collection of helper functions and patterns for common infrastructure testing tasks.

Infrastructure Scenario

We have a simple web application deployed on AWS Fargate, there is one public subnet with an ALB, receiving external users requests and routing them to the Fargate containers in the private subnet. The Elastic Container Registry stores the docker application image, that is used by Fargate to run the web app.

GitHub Actions

GitHub Actions is a GitHub integrated feature that provides you with an execution environment integrated into every step of your CI/CD workflow. We use GitHub Actions to automate the testing of the CI/CD pipeline, you can use a wide variety of Actions developed by the community allowing you to integrate your pipeline with different cloud vendors including AWS.

GitHub Actions Concepts:

Actions are individual tasks that you combine as steps to create a job. You can create your own actions, use actions shared from the GitHub community, and customize public actions. To use an action in a workflow, you must include it as a step.
Runner is any machine that executes GitHub Actions, you can use a runner hosted by GitHub or host your own runner called self-hosted runner. A runner waits for available jobs. When a runner picks up a job, it runs the job’s actions and reports the progress, logs, and final results back to GitHub. Runners run one job at a time. GitHub provides 2000 min/month of free GitHub hosted runner executions which allows you to leverage CI/CD capabilities without having to spend on an external service.
Event is a specific activity that triggers a workflow run. For example, activity can originate from GitHub when someone pushes a commit to a repository or when an issue or pull request is created. You can also configure a workflow to run when an external event occurs using the repository dispatch webhook.
Job is a set of steps that execute on the same runner. You can define the dependency rules for how jobs run in a workflow file. Jobs can run at the same time in parallel or run sequentially depending on the status of a previous job.

Pipeline Structure

In this section we describe the pipeline created using GitHub Actions, each one of the steps are marked with a green check if they are successful, or a red x if the step failed. The first step Set up job is created automatically by GitHub Actions, the following steps are described next.

GitHub Actions uses a yaml file located inside .github/workflows folder within the repo that instructs the runner a job that contains: actions to use, environment variables, when to execute the pipeline (Events), commands to execute etc., our pipeline has a single job split in multiple steps, performing both static and integration tests:

Run actions to setup the tools required to run the tests such as terraform and Go. The checkout action allow the container to copy and check the files in the repository.

Run Terranovax/aws-ecr-deploy@v1 is a community action that allows to build and push a docker container image into AWS ECR service. Here we use predefined secrets that are AWS access keys with proper permissions in the account to access AWS services and features we need. The build image has our example web application and will allow us to test every part of the infrastructure.

Run Go Static Tests to run basic terraform lint commands and validate the output of terraform plan.

Deploy Infrastructure to apply the changes from the previous stage terraform plan output.

Run Go Integration Tests here we use a Go function from a terratest library that allow us to make an HTTP GET call to our Application Load Balancer, this test return an status code from the URL e.g. 200 to check if the request has succeeded. Any other return fail from the Go test.

Integration tests with terratest functions

Writing the tests with Go allow us to use terraform - terratest libraries and functions. At the beginning of the code we need to import the libraries we are going to use:

After importing the libraries we create a main testing function to prepare the terraform working directory and set a waiting pre-warm period for the ECS Fargate task to complete the new build image deployment. Finally we send the expected test attributes (HTTP status code, webpage body) to the testURL function:

The testURL function is one of the main pieces within the integration tests, this is going to simulate a user making a request to our application URL, checking the webpage body contents and returning the HTTP status code, this allows the pipeline to test end to end functionality from all underlying infrastructure parts:

Results

After the pipeline is completed we can look for the results checking the logs in the GitHub Actions job logs. Here we can see the test got the expected HTTP status code and body from the web app so the test result was successful:

TestHttpMicroserviceValidity 2020-07-22T22:58:02Z retry.go:72: Calling http://tectech-insider-ALB-111111111.us-east-1.elb.amazonaws.com/
TestHttpMicroserviceValidity 2020-07-22T22:58:02Z http_helper.go:32: Making an HTTP GET call to URL http://tectech-insider-ALB-111111111.us-east-1.elb.amazonaws.com/
TestHttpMicroserviceValidity 2020-07-22T22:58:02Z microservice_integration_test.go:49: Got expected status code 200 from URL http://tectech-insider-ALB-111111111.us-east-1.elb.amazonaws.com/
--- PASS: TestHttpMicroserviceValidity (190.34s)
PASS
ok   microservice_test 190.354s

Receiving an unexpected body or status HTTP code results in test failure:

TestHttpMicroserviceValidity 2020-07-22T14:16:01Z retry.go:72: Calling http://tectech-insider-ALB-111111111.us-east-1.elb.amazonaws.com/
TestHttpMicroserviceValidity 2020-07-22T14:16:01Z http_helper.go:32: Making an HTTP GET call to URL http://tectech-insider-ALB-111111111.us-east-1.elb.amazonaws.com/
TestHttpMicroserviceValidity 2020-07-22T14:16:01Z microservice_integration_test.go:49: Got expected status code 200 from URL http://tectech-insider-ALB-111111111.us-east-1.elb.amazonaws.com/
    microservice_integration_test.go:54: 
         Error Trace: microservice_integration_test.go:54
                         microservice_integration_test.go:39
                         test_structure.go:24
                         microservice_integration_test.go:35
         Error:       "Error body page" does not contain "Welcome to tech insiders"
         Test:        TestHttpMicroserviceValidity
         Messages:    Body should contain expected text
--- FAIL: TestHttpMicroserviceValidity (191.29s)
FAIL
exit status 1
FAIL microservice_test 191.303s

Conclusion

A simple pipeline allow us to automate IaC testing, it is the perfect addition to application testing so all the involved pieces have the intended behavior and performance. Having static tests is a very basic starting point to check code healthiness, however adding integration tests for your infrastructure decreases the level of uncertainty and risks associated specially before going live with an app into production.

References

Link to the code: https://github.com/braybaut/tech-insiders-tdi
GitHub Actions Documentation: https://docs.github.com/en/actions
Terratest Documentation: https://terratest.gruntwork.io/docs/