Mobile testing at Monese

How we’re running 600 automated tests… in 3 hours

While many companies are struggling with carrying out testing fast enough, the QA team here at Monese has managed to successfully ramp up our automated mobile testing to quite an impressive speed before each release.

Here is how we do it.

Mobile automation framework

At Monese, we use the CI framework Jenkins to run our tests on real mobile devices. We can run tests based on the following factors:

• the type of phone
• user’s Monese plan
• suite name
• build (develop or release)
• test environment

All test results are reported to TestRail.

Main features

Every action taken on the framework is recorded onto logs, including some diagnostics data. We keep video of each failed test to help fix any changes made on that screen without rerunning tests manually. Given this information, we are able to analyze the reason why a test failed and the behaviour that caused this.

We also use Allure Framework for test reporting. It shows us precise information of which, where and why tests failed, along with information on what was expected to happen and what actually occurred.

When we need to generate random user data and account information for testing purposes, we utilize API and DB calls. This is done instead of running the entire user interaction on the test itself. We leave a test suite that creates the accounts to test this specific user interaction while all the other test suites use the API and DB calls. In turn, this is another huge time-saver when running tests.

Appium allows us to write one single test which runs for both iOS and Android. This is to avoid code duplication. Writing tests for both environments is essential. But it is easy to make mistakes when both environments need to test the same features. Code duplication is difficult to support and maintain. It is also easy to forget to test or update parts of a test in one of the environments. To avoid all these duplication issues, we use Appium and Google Page Object Pattern to write our tests.

If you can parallel, do it

To speed up the test execution process, we run tests in parallel (one device — one test of the suite). Whatever suite we run is sent and tested on all possible phones. We can also automatically detect the next available device to begin the next test of the suite systematically. This is both cost-effective and extremely timesaving.

For testing purposes at Monese, we currently test on 12 connected phones (6 Androids and 6 iOS) — all while using one Mac mini. We have no limits (within reason) on the number of devices we employ. This means we could easily double or increase our phones to 40, and in turn, boost our speed, while still only utilizing one Mac mini! It is a simple piece of hardware handling all this simultaneous testing with incredible velocity.

We get live reporting of results as test suites are running. This helps the team quickly identify run health in general and stop execution if there are too many failures.

The result: 600 tests executed in an astonishing 3 hours.

This article was written by Aleksei Medvedjev, QA Automation Architect at Monese.