Get the Most Out of UI Tests With XCode Test Plans

As Trendyol IOS Team, we use both manual and automated user interface tests to make sure our app truly does what we expect from it. Our goal is always to maximize automation coverage to keep the tests of new features and regression test period short, minimize stress and manual efforts.

There are too many memes about the 100% code coverage on the internet, It's really hard to catch this goal since our company constantly delivers new features to its users. We use the built-in XCTest framework to write our UI tests, the challenging thing about them is they require more maintenance than unit tests since the app’s UI changes more frequently.

At that point, It is very important that we get the most out of the tests we have. To achieve that we should be able to use different combinations of tests in different situations for different purposes. Schemes also can be used to create different configurations but that can easily become unmanageable.

In this article, we’re going to take a look at the benefits of using Xcode Test Plans which allows us to run individual test suites with different execution styles, localizations, locations, and sanitizers.

What is an Xcode Test Plan?

Xcode test plan provides a way to run a collection of tests with different test configurations. The test plan is a JSON file with the .xctestplan extension that you can add and manage through UI or from the source code. You can convert an existing scheme to use a test plan. You should add a plan to the scheme. It has three main parts:

• Test targets: More than one test target can be added to a single plan. You can configure each target’s execution and automatically include new test options separately.
• Shared settings: Shared settings(default options) will be overridden by specific configurations. These are all the settings you would normally find in the scheme editor and also some extra settings; launch arguments, localization settings, screenshot settings, attachments, text execution order, runtime sanitizers, thread checker, and memory management.
• Configurations: Here you can override the default options and create different configurations. A test plan runs the selected tests multiple times, once for each of the configurations.

This is how an empty test plan looks like

Advantages of Use XCode Test Plans

Run Specific Tests from Command Line

There are many advantages to use it but this one can’t be achieved for xcodebuild without test plans or third parties unless you don’t write endless if conditions and add -only-testing: parameters according to that in your pipeline script. Most of the other test frameworks have the functionality to run tests according to the path. So traditional separate them into folders method not working here. There is no built-in functionality for xcodebuild but now thanks to test plans we can easily group and run them with -testPlan parameter from the command line.

We use JIRA, Bitbucket and Slack for organizing our tasks and use Jenkins to manage our pipelines. With test plans, we are able to trigger only related tests for the tasks development scope or other teams can easily trigger our tests with Jenkins API remotely. If they need to check a specific part of client behavior, we provide them test plans which contain all needed test scenarios.

Before/After Use Test Plans

Localizations:

Currently, Our app only has a Turkish language but soon there can be many other options. We will create a configuration for each language we support to make our tests compatible and it will be as easy as snapping a finger.

Locations:

We have instant delivery channels for groceries and meals inside our app. They both use google maps so users can easily select their location on the map. Test plans have options to configure locations. You can make simulators run in a different location for each configuration by selecting from default locations or with create gpx files for desired locations.

GPX File

Test Execution:

You can specify the execution of your test’s order alphabetically or randomly for different configurations. You can also set different timeouts for them.

If you are using Xcode 12, there are 2 new options to set default and maximum execution time allowance to guard against long-running or hanging tests.

A small piece from Map UI Tests

Running tests with different sanitizers/diagnostics:

There are three main sanitizers :

• Address Sanitizer identifies potential memory-related corruption issues.
• Thread Sanitizer detects race conditions between threads.
• Main Thread Checker verifies that system APIs that must run on the main thread actually does run on that thread.

Normally both address and thread sanitizers can’t be enabled at the same time. Using a test plan we can create two configurations, one with the address sanitizer enabled and a second with the thread sanitizer. Our tests will run twice, once for each of the sanitizers.

Summary

It is quite challenging to keep coverage high and add new tests day by day. Meanwhile, we need to use what we have in the best way possible.

I hope you find Xcode Test Plans useful for that purpose on your Xcode projects. Thank you for reading and Feel free to comment and contact me if you have any questions.