Run UI tests 5X faster on Firebase test lab
The story began when we decided to have a mature CI (continuous integration) environment for our Android app. At the beginning, the number of UI test cases was small. We created some test suites (groups of tests) where each one grouped some related tests. We used only one CI machine to run our tests on Firebase test-lab. Everyone was happy.
But when the code base became bigger and the number of UI tests exceeded 150 tests, we noticed that running all UI tests, after each push on Git, is very time consuming. It took about 30~40 mins to receive the result from Firebase test lab.
Then we started increasing the number of CI machines, where each one handled a group of UI tests, to achieve parallelism. So we reduced the waiting time. But we paid much more! When we reached 8 machines we figured out that increasing the machines means that we will waste a lot of money.
But we needed to get over the 2 problems at the same time. We sought waiting for a very short time, to receive tests result, without paying too much.
We successfully developed a solution which uses only one CI machine and receives the tests result in 6~8 mins for a massive number of UI test cases.
Reducing the waiting time the dumb way
The configurations, which assigns each test suite to each CI machine, were manually maintained. Every single machine runs a single test suite on a single testing device.
Adding new tests means either adding them to one of the suites which will increase the waiting time for that suite or adding a new machine, running in parallel, with a newly created test suite which doesn’t increase the time but increases the monthly cost of the CI service.
Reducing the waiting time the smart way
The configurations, which assign some tests to each testing device, are now automatically maintained. Each single device runs a very few tests.
Imagine the situation when we run each single test case on a single testing device, we will end up having the overall testing time equals the longest time taken by a single test case. Did you say we need to apply some parallelism?
The solution is based on:
1. Implementing an annotation processor which lists all the UI tests at the build time.
2. Execute shell scripts in parallel where each script is concerned with a very few tests on a single testing device.
The last paragraph explains why a small group of tests/device is sometimes better than a single test/device.
But you will use more testing devices and you will pay more for Firebase, won’t you? Fortunately the cloud testing payment policy is time based. It’s something like x$/device/hour. So the cost of a single device operating for 1 hour is the same as the cost of 60 devices where each one operates for 1 min.
For instance, if you have 30 tests where each one takes around 2 mins, either running them on a single device or on 30 devices will cost the same. But the more-devices solution is way better because you will get the result of all tests in almost 2 mins rather than 60 mins if you applied the single-device solution.
But, in practice, you somewhat will exceed 2 mins. The CI machine is used to execute shell scripts, in parallel, to distribute tests among the testing devices. Specifying 1 test/run requires a lot of machine threads which may ultimately consume more time than the case of specifying 3–5 tests/run. So depending on the available machine threads, the number of tests per run is calculated.
Check this git repository which shows in a step-by-step manner how you can smoothly integrate this solution to your project.
If you liked this article, give it some claps 👏 (did you know you could go up to 50?) and follow me on Medium. Happy testing!
This solution was done with assistance from members of Blink22 team.
