Our automation experience II
How to run UI test in parallel.
We want to run our tests quickly and as often as we can.
We decided to split our tests and run it in parallel in multiple devices. That’s called Sharding. We found four different Gradle plugins to do this.
I created a github project with some tests as example. https://github.com/PabloMGrela/ShardingTests
Spoon was our first option to achieve our objective. There is a Gradle plugin to integrate it into your android project.
You can see the example in this branch of my example project.
Spoon has lots of options to work with your test suite, also has a nice UI report with the total time of the execution.
But, we didn’t found the solution to our main problem. Flaky tests. So sadly, we discard Spoon.
Then we found Fork, a spoon fork (awesome naming 😂). I don’t want to lie, Fork was difficult to implement because I didn’t read carefully the docs AND the error handling is not the best Fork feature.
I had several issues saying that “No devices found” and “No tests found” in test apk. I didn’t met the naming convention for test class, it should end with “Test” (or you can change the regex in Fork gradle settings). Also I needed to kill the gradle daemon each time before run the tests. (Today march 6th, is not compatible with AGP 3.6.1)
You can see the example here.
We are able to handle flaky tests and set how many times do you allow the retry.
Furthermore Fork allow us to record a video of each test and it has an awesome jenkins plugin that allows us to check if a flaky test is too much flaky.
We also tried composer (example), with its gradle plugin, but we had the same problem as we faced with Spoon, we cannot handle flaky tests. Also, right now the project is archived and the last update was one year ago…
I would say that composer has the best UI report so far.
Finally we found Gordon. Check it on the repo example.
It was really easy to implement and allows flaky tests, but doesn't support latest AGP (3.6.1). After trying Fork, we miss more data in the reports, support video or screenshots its an awesome feature that it's not available for Gordon.
When a UI test fails in one device, Gordon retries it in a different one.
Right now we are using Gordon to shard and run our test suite. It tooks less than half hour to finish instead of 3/4 hours, so we can run it several times each day and allows us to found the issues really quick.
We spend a lot of time to implement our testing strategy, but now we can work with less pressure about break anything, decreasing our bug amounts and increasing our app reliability.
Thanks for reading!
