Photo by Bill Jelen on Unsplash

Speed up UITest Automation in iOS

Do you work on a large codebase with a huge UITest Automation suite? This might sound familiar. You raise a PR. Then you wait. And wait. And wait! Eventually, hours later you find out if there’s a failure or not. Maybe you moved on to something else in the meantime. Uh oh, now you’re context switching. Plus, the expensive hardware resources are held up running the tests!

While working on a project, I faced something similar. I was able to reduce the UI Tests execution time from over 2 hours to 45 minutes using the solution which we are about to discuss.

In this article, we’ll go through how to reduce UITest automation execution time to essentially solve the two below problems.

1. Slow UITests holding up PRs leading up to longer turnaround time on failures disrupting our dev and release process timelines.
2. Longer UITests execution times holding up our CI/CD hardware resources like agents/nodes.

So what do we do? We will try to reduce the test case execution time using below two techniques.

1. Sharding: Get a list of all the tests cases to be executed and distribute their execution over many agents using sharding.
2. Build Options: Use Fastlane(or even xcodebuild command) to build the project once and use the derived data/XCTestRun file to only execute test cases without building.

To put this in perspective of the CI/CD pipeline, see the below diagram.

Stage 1 and Stage 2 in a CI/CD run serially and jobs in stages can run in parallel.

Now, let’s discuss each stage in little more detail. We’ll not get down to low-level details of each and every step in this article but still give a good idea of the solution. Testing

1) Stage 1

In this stage, we build the code for testing. Also we collate all the test cases in the suite and then distribute them over n files. (n represents the number of agents u may want to engage to parallelize the test case execution).

Stage 1: Job 1: Here, we compile our code for testing only(does not run tests). There are build options available, both in xcodebuild and fastlane for the same

If using xcodebuild :- build-for-testing

If using Fastlane scan :- build_for_testing

This should build our app for the testing and create .xctestrun file in the DerivedData which can be used in stage 2.

Stage 1: Job 2: While CI job is busy compiling code for testing, in a parallel job, we will collate and shard test cases into buckets.

So how do we approach the collation of test cases in the first place? There are two ways to achieve it.

Option A: Using Test plans: The test plan is a JSON file with the .xctestplan extension that provides a way to run a collection of tests with different test configurations. The details of how test-plan work is beyond the scope of this article but there are various articles on the web you can find.

Now, assuming having a basic understanding of how test plans work, we need to add one test plan to the project.

1. Add TestPlan to a project via Xocde — > Product — > TestPlan
2. Click on “+” on the bottom right to attach the test plan to a target.
3. Make sure that the “Automatically include new tests” checkbox is unchecked

Xcode setting

This leads to the testplan JSON structure looking like below.

testplan JSON structure

Now since this is a JSON structure, you can use Ruby/Bash or any language to parse it easily and get a list of all tests to be executed. This solution handles the disabled tests quite gracefully.

Option B: Using Stencil/Sourcery: Stencil is a simple and powerful template language for Swift and Sourcery is a code generator for Swift language. Together we use it to read all the Tests.swift files and extract the list of all test cases from them. The sourcery command looks something like this.

$ ./bin/sourcery — sources <sources path> — templates <templates path> — output <output path>

The sample template that can be used as an argument.

{% for type in types.classes|based:”XCTestCase” %}
{% for method in type.methods %}
{% if method.parameters.count == 0 and method.shortName|hasPrefix:”test” %}
{{type.name}}/{{method.name}}
{% endif %}
{% endfor %}
{% endfor %}

Output of sourcery

The disadvantage of using stencil and sourcery combination is that it doesn’t handle disabled tests straightforwardly. (To overcome it, both option A and B can be used together but with a slight variation. This time, check the “Automatically include new tests”. This changes the TestPlan JSON to have “skippedTests” array which you can exclude from the above-collected list).

Now that we have collated the test cases in Stage 1, we need to distribute them as well into n buckets. You can choose any algorithm for it but the basic one is something like below.

The disadvantage of this approach is that certain tests with higher execution times may accidentally come together which may still prolong the execution time.

Advanced Tip:- you can progressively add execution times next to the test cases generated using stencil. This can help you smartly group the test cases together.

2) Stage 2:

Phew!! That was a lot to do! But the hard work is done. Now we have two outputs from Stage 1.

1. n files containing XCTestCase names to be run on n agents.
2. .xctestrun file or the ‘Derived data’ having build information.

If using Fastlane scan :-

test_without_building :- Test without building, requires a derived data path.

xctestrun :- Run tests using the provided .xctestrun file.

In each agent, use the above build options to execute UI test cases. Additionally, use only_testing build option to pass in one of the sharded files from stage 1 containing testcases.

only_testing :- Array of strings matching Test Bundle/Test Suite/Test Cases to run.

If using xcodebuild :-

We can either use the test-without-building option which needs derived data from Stage 1, or use -xctestrun to provide the .xctestrun file directly as input. Use -only-testing option to pass sharded files.

Conclusion

Following the above steps, I could honestly reduce the execution time from over 2 hours down to 45 min which was a huge win for the project leading up to less wait times for PR and optimizing usage of hardware resources.

There are further optimizations that can be done which can vary from project to project for example.

1. While running a UITest, the restart is what takes a lot of time. We can try avoiding many re-starts in the app.
2. Preset the data for a UITest when there are pre-requisites required to test a flow.
3. Parallelizing UITests on a single agent if they are data independent.

Good Luck!