Define Testing Strategy using the Testing Pyramid

Colin But
Colin But
Aug 4, 2018 · 6 min read

For a software project to succeed you need a successful strategy to testing. One way of doing this is to use the Testing Pyramid. Whether you are starting out a new software development project or working on an existing project, it is important to have a test strategy in place.

We all know it’s bad to not have any test at all but it is also bad to have useless tests. Partial tests, intermittent tests that fail for no reason, hard to maintain tests are just some examples of bad dev testing. We all know there’s different tests to put in place. For example, unit tests, integration tests, end-to-end tests, UI tests, API tests, component tests, service tests, system tests, and the list goes on.

So you start to put in tests for your code. But where do you start? How do you tackle it?

This is where a Test Strategy comes in.

In this strategy it is best to utilise the Testing Pyramid to help you define and categorise the different types of tests.

Testing Pyramid

A testing pyramid is a pyramid of where all the different types of tests fits. The basic pyramid looks like this:

Very simple. 3 levels.

  1. At the bottom, is Unit Tests. These should count for the majority of tests you have for your codebase. They tests the smallest unit of code possible. Normally, only test a single method each.

A more detailed Testing Pyramid may look like this:

Component Tests

At the same level as Integration Tests you may have what is called Component Tests along side it. These are tests which tests the different components of a software system. Of course, you need to define within your system what constitutes your ‘Components’. This assumes your software is a modular distributed software system with separate ‘Components’, SOA, or Microservices.

API Tests

Sitting in its own level and in-between the top 2 levels is API tests which are by the name of it tests the API endpoints of your system. Assuming you have APIs of any sort, be it RESTful or non-RESTful. These tests are common because a lot of web apps these days have so-called backend web services which expose its functionality (CRUD like operations) via endpoints. So it is important to test the invoking of these APIs as well. Some people argue these are just Integration Tests. And they’re probably right, they’re merely just a more specific classification of Integration Tests.

UI Tests

Perhaps not so well known until recent years are these tests which tests the ‘UI’ aspect of your system (if your system has an UI). If you’re system is just a backend web service exposing APIs then obviously you don’t really need these and they’re likely not valid. i.e. You don’t have an UI!

Manual Tests

At the very top of the pyramid now sits a cloud or bubble of ‘Manual Tests’. These are tests which is performed manually. They test the whole system in a black box way. They are important as they test the system as a whole which each of the individual area tests cannot do.

Who should do what?

Knowing which tests to write is a good start. Next step is probably agree on who is responsible for what.

In a team, you have developers, QAs, DITs, and maybe even SDETs. For a team to function properly you probably need clear responsibilities assigned.

  1. QAs are obviously responsible for manual tests. Sometimes, SDETs do these tests too. They perform them alongside QAs.

Avoid the Ice-Cream Cone Anti-Pattern

Don’t do this.

This is bad. This basically is the inverse of the Testing Pyramid. It is the opposite of what the Testing Pyramid suppose to achieve.

You won’t achieve much with this approach. You could think of this as a bad strategy. Having a big portion of manual testing compared to that of Unit Tests cries for disaster. Development time increases. As it takes much longer to do things manually than having an automated test suite at hand which you can easily run within a small time frame (if you have fast tests that is).

One of my previous company i worked for, we had this anti-pattern in place. I didn’t notice it at first until i left that company only later on to come across a blog post which mentions that. Only then i realise… I’d wish i pointed that anti-pattern out to my former employers earlier so we could have saved a lot of time in development which would have resulted in quicker time to market for our business features.

We had little to none Unit Tests let alone Integration Tests. We did however had a large number of E2E tests with a bit of API tests. But mostly our testing was done by hand manually.

If only…

So the bottomline is… avoid this as much as possible.


Takeaways

  1. Write Unit Tests (this is MUST!, and it’s an absolute basic requirement!)