Describing our Automated Test Strategy

After my previous post describing how I use the Test Automation Pyramid, and a Twitter discussion with Richard Bradshaw who pointed me in the direction of his talk “Pyramids are Ancient — Test Automation Strategy”, I decided it was about time that I reflected on my current team’s automated test strategy.

I’ve asked myself the following questions: what automated tests do we write? Why do we write them? What value do they provide us?

Background: What Does Our Team do?

We are a team of consultants who are responsible for writing security controls in AWS. These controls are set up in every AWS account used by teams within the organisation we currently work in. Our customers are anyone who uses AWS within the organisation.

The majority of our controls utilise AWS Config Rules which trigger AWS Lambda Functions, written in Python, and then report a compliant or non-compliant status depending on the AWS resource the rule is evaluating. We deploy this infrastructure using Terraform, which adds another layer of complexity.

What Automated Tests do we Write?

Our automated test strategy can be summarised with the following illustration (yes, I’m using my favourite, the test automation pyramid):

Let me explore these in more detail:

• Bottom Layer: at the bottom layer of our pyramid we have our unit tests. As I mentioned above, the majority of our code is written in Python and is deployed in the form of AWS Lambda Functions. We ensure that the critical functionality of each function is tested using Python’s unittest framework.
• Middle Layer: here we have two different types of tests. Our Terraform Tests ensure that we have configured our Terraform correctly to support the functionality of our controls. We also have what I’ve described as boto3 tests. These tests make a boto3 API call to AWS to ensure that the resources we have deployed are running correctly.
• Top Layer: our top layer, and by far our most expensive layer of tests, are our end to end tests. These tests typically trigger an action in AWS that will trigger one of our Lambda functions to run. We then have to analyse the results that our function produces and assert an expected outcome, depending on the action we have performed.

What Value do These Tests Provide?

I’ve explained what tests we write and where they fit into the Test Automation Pyramid, but why do we write them?

I decided to scribble my thoughts on why we write the different types of tests we do within our team. What value is provided by each of the blocks of our pyramid?

At the top of our pyramid, our End to End tests are slow-running and are difficult to maintain. We tend to only write these for our most critical pieces of functionality. I’d argue that AWS has not been built with testing in mind (throttle limits on APIs, time it takes for logs to appear), but that is an entirely different topic.

They do still provide value. They test each and every layer of our individual security controls and they simulate the experience that we are providing our customers. The information they provide us (and auditors!) is really important.

Next up is the middle layer. Our Terraform/boto3 tests allow us to ensure that we have configured our declarative Terraform code properly. Have we created our own resources to be compliant (e.g. not exposed our EC2 instances to the public)? Has Terraform Enterprise deployed our infrastructure correctly? This is arguably the first layer that Terraform can be tested at, so these tests act as unit tests for our Terraform code.

By using boto3 we can also ensure that all of our individual Terraform resources are working together correctly. These are non-intrusive checks that allow us to (resonably) quickly determine the state of our deployed features. For example, has CloudWatch been enabled, for monitoring purposes, on one of our EC2 instances?

I don’t think I need to spend too long describing the benefits of unit testing in this blog.

In the context of our team, we practice TDD which allows us to ensure that we’re developing the functionality we have agreed as a team and with our stakeholder. The unit tests act as scaffolding to our functional code, meaning any functional changes are safeguarded by these tests. We also have a relatively slow deployment process, so having these checks to run before we kick that off is often invaluable.

Final Words

I hope you found this insight into the value that the different layers of tests provide to our team useful. Test strategies like this cannot be copied and pasted and they are heavily dependent on your team’s context.

If you’re looking for advice on how to develop a test strategy of your own, whilst considering the different layers you should be testing at, then I’d recommend giving Richard Bradshaw’s talk (which I also linked above) at SauceCon a watch.