The Pyramid of Automated Testing
Automated tests are the heart and soul of any CI / CD flow. Over the last few years testing went from 100% manual to more and more automated. In the current age of technology, mobile and DevOps are the real driving forces of the change software testing. Release cycles are more frequent and the iteration is accelerated. The competition is fiercer than ever to deliver value fast for clients and end users. This means that all processes in development and operations must be geared towards efficiency. One of the most time-consuming activity and a virtual minefield when it comes to human errors.
One of the most time-consuming activity and a virtual minefield when it comes to human errors. Test often and fail early, this is the type of mindset you need to have.
What type of test should I automate?
There is no silver bullet for test automation, but the testing pyramid is good starting point for most teams to know how to allocate resources.
The pyramid of automated testing distinguishes three types of tests:
- Unit test
- Integration test
- End-to-end test
These tests are different from each other in size, complexity, speed, and application method. To uncover as many bugs as possible before the product is delivered to the end user a combination of Unit, integration, and E2E test should be carried out.
Which test shall I do most often?
The testing pyramid shows the right balance between the 3 type of tests. At the bottom of the pyramid is the unit test, the majority of your test should be this type. As you are moving up on the pyramid the tests are getting larger and less frequent. Developers as Google suggest 70/20/10 distribution, 70% unit test, 20% Integration and 10% end-to-end test and our dev team has a similar test ratio as well.
Unit test is a software testing method where an automated piece of software takes a small unit of your product and tests its behavior in isolation from the rest of the code. A unit is the smallest testable part of a software. The purpose of unit testing is to determine that individual units of the software are fit for use. The majority of the bugs are discovered during this procedure.
What are the advantages of unit test?
- Unit tests are fast. Doesn’t take long to build and takes a blink of an eye to returning results .
- Unit tests are reliable. Small test systems are less likely to have errors.
- Isolates failures. This makes easier to locate and fix bugs.
Besides the obvious advantages of Unit testing it has its own shortcomings. The test has small code coverage, therefore, it has to be repeated multiple times to cover all the units. Furthermore the results are only valid in the isolated test environment. To understand how multiple units of a software product are working together it is highly recommended to run a series integration test.
Integration testing is a logical continuation of unit testing. An integration test takes two or more units of your software and tests their behavior as a whole system. The purpose of integration tests is to verify that independent units can coherently work together.
There are two ways of carrying out integration test:
- Bottom-up method
- Top-down method
At the bottom-up method directly follows the unit tests. It tests progressively higher-level combinations of units. While the top-down method starts by testing the highest level combinations of units and gradually progresses towards lower level units.
What are the advantages of integration test?
- Discover incoherency in the code and dependencies
- Relatively fast
- Discover bugs that are the results of interactions of software units.
Integration tests can examine bigger and more complex subsets of the system then unit test. However, to understand how fit your software is for their intended purpose, you should carry out end-to-end tests.
The end-to-end test can be considered as the most complex integration test, where all the existing features are connected. The purpose of end-to-end testing is to verify that an application is functioning as expected, to ensure data integrity between connected features and systems. Real-life scenarios are used test the application.
For example, a simple end-to-end test for a social media app might involve the following steps:
- Logging into the app
- Accessing news feed
- Liking posts
- Post a picture
- Adding a friend
- Accepting a friend request
- Opening the chat
- Sending a message
- Closing the chat
- Logging out of the app
Key take away
If you are new to automated testing, the pyramid is a good place to start. Unit tests, integration test, and E2E test both have their pros and cons. It is better to use a combination of them to identify issues at every level. There is no definite answer how many you should do from each, it is different for every product. A rule of thumb is that most of your test should be unit test, much less should be integration tests, and you should carry out E2E tests relatively rarely compare to the other two.