The Test Pyramid: Greatest Wonder of the (Software Testing) World

A New World

The swift and rather needed move towards Agile in the software world had led to a lot of benefits. However, it also imposed pressure on the software tester, having to deal with the accelerated turnover of development pushed forward to production. A testing strategy needs to be in place.

It is easy to figure out that repeatedly manual testing a whole system is rather unattainable. Consequently, developers and testers have to contribute effort to automated tests.

The easiest mistake that a team can make is to only rely on end to end tests. Although these tests are focused on the actual users and their relevant interaction with the system, they do not provide a quick feedback mechanism on what is wrong and where to find the underlying flaw. This is where the concept of a test pyramid comes to the rescue.

The Age of Pyramids

First brought up in the book “ Succeeding with Agile” by Mike Cohn, the Test Pyramid provides a design to help out in visualising the right balance of effort that should be allocated to the varied types of automated tests.

At the depths of the pyramid lies the foundation of good old unit tests. These provide a comforting assurance that each unit of code is operating as expected. One of the reasons behind preaching for the largest amount of tests to be at this layer is that these provide the quickest feedback. Unit tests are found within the system’s repository and can be set to fail a build if any of the tests fail. This prevents them from being ignored if failing, much to the contrary of tests up high in the Pyramid.

The middle layer provides verification of services and APIs, as well as the interaction between them. The amount of effort should be less than unit tests since a level of logic is ‘hidden’, therefore finding the root cause of a bug will take longer with a failed verification from this level.

Finally, on top of this structure, we have the UI tests. These tests are the closest to the actual users, and therefore cannot be defined as redundant or worthless. UI tests give a concrete piece of mind when an actual user journey is verified to be working as expected. However, as mentioned previously, going up the Pyramid means also that the ‘time to root cause’ is longer, as there is a lot of logic that needs to be dug up. A set of good UI tests can be hard to maintain unless extra effort is spent to set up a solid and reliable testing framework.

Needless to say, this topic has been blogged and discussed countless times, so my input on it is definitely not about questioning its validity. Different agile teams may have different priorities or responsibilities towards a shared application. Therefore, saying that a universal test pyramid is suitable for all the teams is quite misleading. Each team would have a different opinion on where its testing efforts are being spent, and this would reflect on their idea of the right pyramid.

“Is one pyramid not enough?”

Not really. For instance, we can think of an agile team which produced most of its commitment towards creating integrations with payment gateways. Its client would then be a team which build the front end part so that these payment methods can be consumed by the end users. We can agree that their ideal test pyramid would include a large number of unit tests, followed by a set of service layer/integration tests. Going up the pyramid, they would not be so interested in UI tests as this is not part of their own work. Consequently, they do not hold the right knowledge to design these automated tests.

Likewise, the team which would build up the visual aspect of the payment methods would spend an amount of its automation effort to verify the correctness of its UI for any regressions.

“So? Each one his own pyramid?”

No that’s not what I mean. I believe that within an organization, communication is key to everything, and this is true as well in respect to this topic. Testers and developers working in different areas of a common application should discuss and build up the pyramid of test automation together, with each team covering the right areas of the system. The responsible teams for each area hold the best knowledge for each decision, and should not be overruled by other QA members working within other parts of the system.

Conclusion

There is still a lot of speculation about how the slaves built the actual Pyramids of Egypt. What’s for sure is that they built it just how the Pharaoh wanted it to be. Make sure that when you build YOUR Pyramid, you build it how YOU need it to be, not how others in your organization suggest it to be.The more information you have on what you need to cover, the better you are at designing your own Test Pyramid.

Originally published at https://testautonation.com in 01/2018.