Prism’s Testing Pyramid

Published in

prismapp

6 min readDec 1, 2016

Introduction

As more and more companies are moving faster, they want their latest, greatest software features to be available to their users as quickly as they are built. However there are several issues blocking them from moving ahead. One key issue is the massive amount of time it takes for someone to certify that the new feature is indeed working as expected and also to assure that the rest of the features will continuing to work.

In spite of this long testing cycle, we still cannot assure that our software will not have any issues. In fact, often our assumptions about the user’s needs or behavior are probably wrong. But this long testing cycle only helps us validate that our assumptions are correct.

How can we break out of this rut & get thin slices of our features in front of our users to validate our assumptions early on?

THE TESTING PYRAMID

Most software organizations today already know about the “Testing Pyramid”. The testing pyramid is a concept developed by Mike Cohn, described in his book Succeeding with Agile.

Its essential point is that you should have many more low-level unit tests than high level end-to-end tests running through a GUI. Unit and integration tests are probably the best things you can do. Those tests make you more confident when making changes, and they contain lots of good information on what the code does. The bulk of your tests are unit tests at the bottom of the pyramid. As you move up the pyramid, your tests gets larger, but at the same time the number of tests (the width of your pyramid) gets smaller. We can split percentage for each test parts, for example: 50% unit test, 30% integration test, 20% e2e test. The main idea is to retain the pyramid shape proportion. Thus, it should be smaller (in term of quantities) as it gets higher in the pyramid.

Now, let’s consider these anti-patterns:

Inverted pyramid/ice cream cone. The team relies primarily on end-to-end tests, using few integration tests and even fewer unit tests.

Hourglass. The team starts with a lot of unit tests, then uses end-to-end tests where integration tests should be used. The hourglass has many unit tests at the bottom and many end-to-end tests at the top, but few (almost nothing) integration tests in the middle.

Cupcake. Testing isn’t a team effort, everybody is involved but nobody is communicating. This results in duplication — the same scenario ends up being automated at many different levels.

Dual Pyramid. Focus on the testing pyramid is going as it is supposed to be. However, it lacks of collaboration between teams. This scenario will lead to a long feedback cycle due to invisible walls between them.

On this post, I will explain the most software organizations suffering (based on my knowledge and experiences) that I call “Inverted Testing Pyramid/Ice Cream Cone problem”.

The Inverted Testing Pyramid (Ice Cream Cone)

Most software organizations today spend maximum time and effort to manually checking software. Some invest in automation, but mostly building slow, complex, fragile end-to-end GUI test. Very little efforts are spent on building a solid foundation of unit & integration tests.

Over-investment in end-to-end tests is a slippery slope. Once you start on path, you end up investing even more time & effort on testing which gives you diminishing returns. They end up with majority (80–90%) of their tests being end-to-end GUI tests. Some efforts are spent on writing so-called “integration test” (typically 5–15%.) resulting in a shocking 1–5% of their tests being unit/micro tests.

Why is this a problem?

The base of the pyramid is constructed from end-to-end GUI test, which are famous for their fragility and complexity. A small pixel change in the location of a UI component can result in test failure. GUI tests are also very time-sensitive, sometimes resulting in random failure (false-negatives)
To make matters worst, most teams struggle automating their end-to-end tests early on, which results in huge amount of time spent in manual regression testing. Its quite common to find test teams struggling to catch up with development. This lag causes many other hard-development problems.
Number of end-to-end tests required to get a good coverage is much higher and more complex than the number of unit tests + selected end-to-end tests required.
Maintain a large number of end-to-end tests is quite a nightmare for teams.

Prism’s Testing Pyramid

To deliver great features to be available to users as quickly as they are built and certify that the new feature is indeed working as expected and also to assure that the rest of the features will continuing to work.

We at Prism compiled several best practices into a testing pyramid that well suited with our company. We are trying our bests to keep the pyramid shape by avoiding anti-patterns, and trial & error experiments.

First, we adapt the origin testing pyramid that developed by Mike Cohen and we split the percentage based on Google often suggests a 70/20/10 split: 70% unit tests, 20% integration tests, and 10% end-to-end tests.

After that, we breakdown integration test and end to end testing more specific. Integration testing separate to 3 parts: component (8%), integration (7%), and API (5%). End to End testing separate to 2 parts: End to End GUI (8%) and Exploratory (2%).

As we can see, almost all of testing are automated. But, we still need human interaction for testing (exploratory testing) because human behaviors are different from machines’.

In the next blog post, I will explain all parts on our testing pyramid in more depth. Specifically, segregation of duties and responsibility between QA and Engineer. Last but not least, Yes We Are Hiring!

Notes: There are no silver bullets. Our testing pyramid were designed to suits us. It might or might not be suitable for yours.

References:

Prism’s Testing Pyramid

Written by Rifky Ali