Parallelizing UI Tests

With regards to testing, one of the most interesting new features presented at the WWDC is by far the possibility to run multiple concurrent simulators which in combination to the xcodebuild enhancements allows to run test sessions in parallel. There’s no need to remark that this is a turning point development when it comes to UI Tests, which suffer to become appealing especially because (among other things) of their slowness.

By running in parallel you can not only execute the same tests on different devices simultaneously, but also different tests on the same device. Say you have 10 tests suites, you can split them running 5 on “iPhone 6 simulator 1” and 5 on “iPhone 6 simulator 2”.

If your wondering how many simulators you can run at once you’ll be amazed with what xcodebuild tells you in the command line logs:

Running tests concurrently on multiple destinations. Maximum concurrent test device destinations is unlimited

Mind-blowing.

Steps involved

To see how much faster this really gets I tried parallelizing SBTUITestTunnel’s UI test.

The steps involved are pretty simple:

  • Create N additional ui test target and associated scheme
  • Split your test cases among the N targets
  • Create N copies of the simulator you want your tests to be run
  • Pass the simulator’s ids to the destination parameter of xcodebuild

An example

Test targets and splitting

For simplicity let’s start with N=2 and let’s say you have 4 files with UI tests cases:

file1UITests.swift → UI test target 1
file2UITests.swift → UI test target 1
file3UITests.swift → UI test target 1
file4UITests.swift → UI test target 1

You create a secon UI test target and split them evenly

file1UITests.swift → UI test target 1
file2UITests.swift → UI test target 1
file3UITests.swift → UI test target 2
file4UITests.swift → UI test target 2

Simulators cloning

Now, for each simulator you want the tests to run create a copy (WindowDevices and Simulators Create a new Simulator). So if you want to test on iPhone 6 you’ll create a second iPhone 6 simulator.

xcodebuild

Invoke xcodebuild using the new -destination parameter

xcodebuild
-workspace SBTUITestTunnel.xcworkspace
-scheme SBTUITestTunnel_UITests
-sdk iphonesimulator
-destination 'platform=iOS Simulator,id=<<IPHONE_SIM_1>>'
-destination 'platform=iOS Simulator,id=<<IPAD_SIM_1>>'
test &
xcodebuild
-workspace SBTUITestTunnel.xcworkspace
-scheme SBTUITestTunnel_UITests2
-sdk iphonesimulator
-destination 'platform=iOS Simulator,id=<<IPHONE_SIM_2>>'
-destination 'platform=iOS Simulator,id=<<IPAD_SIM_2>>'
test &

This will happen

Benchmarks

The initial results are very promising even with the test cases of the library which are all rather short.

SBTUITestTunnel tests on iPhone and iPad complete in 32m with Xcode 8 and just a bit under <10m with the new xcodebuild (splitting test over 2 targets) running parallel tasks.

Further improvements

Yes, leveraging XCTest’s new firstMatch API we can expect even faster results. I’m gonna investigate this area soon and post a new blog with new benchmarks. Stay tuned!