How to test your mobile apps efficiently? A five-level pyramid testing strategy.
Testing is probably one of the hottest topics in software development. Everybody agrees on the need of quality testing and a certain amount of tests (or of code coverage). But what’s the scope, the target and the purpose of your tests? On which environment should you run them and how do you deal with your dependencies?
These are some of the questions we tried to answer by providing a testing strategy for our mobile applications.
Problematic: Deal with the context and the dependencies
Defining a clear scope and an environment for your tests should be the first step in the process. Imagine yourself testing your e-commerce application manually. Do you want to test the order process including the payment service provided by a third party company? Or do you want to focus only on the code developed by your team abstracting the payment service?
There is certainly no one correct answer. But these two identified tests should not happens at the same time and will definitively not have the same costs (in terms of implementation and maintainability).
Based on the Test Pyramid presented by Martin Fowler, we have defined our own pyramid consisting of 5 levels:
For each levels we have defined:
- Test targets: Which components are actually tested and what’s the purpose of the test?
- Test scopes: It will increase at each stage of the pyramid. Making your tests more dependent on local and external dependencies.
- Test types: The technique to implement tests
Each test level has its’ own challenges
In this section we will go over each level of the pyramid, from the bottom to the 🔝
Unit Tests
Unit tests will check and validate the functionality of the smallest pieces of your code. The scope is the tested class and the function itself nothing more.
Most of the specific scenarios and edge case have to be tested here and won’t be necessarily tested in the upper layers.
The external dependencies of the app, such as a REST client, must have to be mocked.
The local dependencies of the class, such as a repository, must also have to be mocked.
All the current modern languages provide frameworks for unit testing, like XCTest for iOS and JUnit for Android.
TDD or not, unit tests should be part of the definition of done. They have to be written during the implementation of the feature.
Integration Tests
The integration tests verify the interaction and the integration of small modules and interfaces within a single component.
The external dependencies of the app, such as a REST client, must have to be mocked.
Local dependencies inside your tested component can be used for real, depending on the purpose of the test. Local dependencies outside the component must have to be mocked.
The same techniques, frameworks and rules are reused from the unit tests.
Application Tests
Application tests focus on the whole application. We can go through all the layers of the tested application. They ensure your app is working as expected without any external disturbances. It will make you confident that at least the logic inside your application is working.
The external dependencies continue to be mocked. It means that your app has to work in a completely artificial environment!
Because of their cost in terms of implementation and maintenance (take care of flaky tests), you should identify your test cases carefully. From that level and the next ones, we recommend to test only the scenarios that are not testable with unit or integration tests.
The implementation is often realised through UI Tests (XCUITest, Espressoor appium for cross-platform and blackbox strategy) or with Snapshot tests.
From this point all the way to the top, implementing new test cases requires delegated Tasks. Implementing new test cases should be considered as its own feature.
System Tests
We are almost at the top of our pyramid 😃 !
From this point, we are allowed to use some external dependencies such as the back-end application of the app.
Any sub-systems that are not directly part of the configuration, like an external payment service for example, must be mocked.
You will have to consider to run your tests on a dedicated testing environment or directly on a staging environment. Both options have pros and cons and won’t be discussed here.
UI Tests and Unit Tests are probably the two most used techniques for the implementation. But considering the environment question above, they will be more difficult to set up and to maintain than the tests at lower levels.
Consumer Driver Contract Testing (CDCT) is a third option. It is a very elegant technique which should quickly detect any regressions between the front-end and the back-end. Check the pact.io documentation for more information.
System Integration Tests
Finally, here we are! We can make complete use of any dependencies of our app with the System Integration Tests 🥳
Hold on, don’t celebrate just yet… With great power comes great responsibility. You will learn the price of depending on external services for your tests. Some of them will give you access to a dedicated test environment which will help you. Having access only to the staging or even worse only to the productive environment of a third party service could make your system integration tests impossible.
Despite their complexity, System Integration Tests are a level you should not ignore. Especially, if your initial plan was to reduce the number of your manual tests.
UI Tests and Consumer Driver Contract Tests are the preferred techniques.
The most important rule will be to clearly define which scenarios you want to test here in which context and in which environment.
What has this strategy brought us?
In the Mobile App Development division of The Federal Office of Information Technology, Systems and Telecommunication (FOITT), we have decided to focus on what we can control: The lower levels until the Application Test included.
We try to have a high code coverage and to write relevant unit and integration tests. It’s the responsibility of the development team and it is written by them. These tests are fast and run in our CI pipeline on every pull request created.
The frameworks stay as close as possible to the development technology used by the developers. Because it should not be a constraint to write tests.
Application Test scenarios are decided on by the whole team because they have a bigger impact on our work. They take time to be executed. That’s why they run in a Nightly Build strategy on a cloud platform made up of real devices. It gives us more confidence that no regression has been added in the previous day. We also had the experience that most of the crashes reported by QA during manual testing could be detected in an fake environment at an early stage.
For the System Tests and the System Integration Tests, we would like to use pact.io in the future because it’s already in place for our web toolchain. We would also like to avoid to maintain a dedicated test environment and directly use the staging environment, with all the constraints this involves.
We continue to do a lot of manual tests and we would like to reduce them. But we don’t want to replace all of our manual tests by automated tests. We prefer to increase our coverage and our test cases in the lowest levels. Keeping the System Integration Tests, the System Tests and even the Application Tests for typical critical scenarios of our app.
And what about you? What is your experience in testing in your company or for your side project? Let us know!
We also recommend you check this Github thread with a lot of testimony from the biggest tech companies.
This article was written by Raphael Guye, Bryan Reymonenq and Marco Stähli.
