UI Automation for the Trainline iOS App: Tooling and Performance

In my previous post I discussed some questions around UI automation testing for iOS applications. I looked into questions regarding stability and the use of stubbing data as a base for most UI tests.

Choosing the right tools

The most important part of doing automation is to choose the right tools. There is plenty of choice on the market for UI automation testing tools for iOS. Appium, Calabash, KIF — these are just some of the big players you can choose from, not to mention a large number of smaller ones and one native tool — Apple UI Automation. All of these tools have some benefits but also some disadvantages.

The main challenge is this: there is no ideal solution.

Below, I will share with you my thoughts and conclusions about some of them — and I really hope this will help you choose the most appropriate tool for your team when the time comes. To keep these learnings manageable, I will focus on some key, well-known tools as well as the native Apple solution — not, for obvious reasons, the Apple legacy JavaScript implementation, but rather the modern Xcode UI Tests framework based on Swift (you guys know what I’m talking about, right?).

Also, I really must emphasise that all of the points and thoughts below are based on empirical evidence: we have actually used all of these solutions at Trainline at various times. All of this is real experience and learning.

Appium

This is a really good tool that has a lot of benefits and good thinking behind it. It’s open-source, free to use and quite easy to setup. A huge benefit of this tool is its cross-platform nature. So, if you have iOS and Android apps, with similar features and UI, and you want to use one tool to write automation for both of the apps, then Appium could be for you.

Here are some of the other benefits:

• tagging for tests scenarios
• abstraction level for Scenarios using Gurkin
• quite easy to setup
• cross-platform
• black-box approach, completely external to project.

Here are couple of issues which we found to be drawbacks:

• very slow after the latest iOS release.
• Java based — so it’s not native for iOS developers

Calabash

Here are some plus points:

• tagging for tests scenarios
• abstraction level for Scenarios using Gurkin
• easy to setup
• have cross platform capability
• able to run tests on Real devices
• relatively fast for tests run

But here are some unfortunate minuses:

• requires recompiling the project specifically for tests to run
• invasive as it requires the injection of the Cucumber library with HTTP Server in order to implement communication with the Ruby part
• Ruby-based — so, again, it’s not native for iOS developers

KIF

This is quite an old tool which is, by nature, much closer to Xcode UI Tests rather then to the Appium or Calabash approach.

Pros:

• very close to development, as it uses Objective-C as the main language
• fast to run
• easy to setup

Cons:

• very invasive as it literally runs inside your app
• no cross-platform capabilities

Also I need to mention that this tool uses UnityTests style, more commonly known as JUnit test style, for writing tests. This could be a pro or a con depending on your point of view.

Xcode UI Tests

This is the native solution from Apple. More precisely, the second generation native solution. The first implementation from Apple was based on JavaScript and didn’t integrate properly with the IDE. Instead, you had to use the Instruments application to access those scripts without any syntax highlighting or auto-completion. So it was a really messy, unstable and unmanageable way of writing UI automation for iOS.

Xcode UI Tests, on other hand, is quite a mature and thought-through tool. With some references to the past JavaScript UI automation in its structures, you can use, or access, elements and search controls, but they have been rewritten in Swift and propositioned by Apple as the main tool for UI automation on iOS.

Like the previous tool, Xcode UI Tests uses JUnit style for test writing and works as set of unit tests by adding a new bundle to your project, and it integrates perfectly with Xcode and Xcode Bots.

Pros:

• fast — it’s actually really fast.
• no need to integrate anything or do any additional setup — works out of the box
• uses Swift as its base language - perfect for native developers.
• great integration with Xcode from writing tests to getting feedback from test runs.
• black-box testing, as test runs on top of a sand-boxed app instance
• stability: from my experience tests run more stably and more predictably in comparison with other tools.

Cons:

• no tagging, harder implementation for CI to run a specific subset of tests.
• no cross-platform capabilities

Requirements and choice problem

So, once we had evaluated all the available tools and listed the key pros and cons, in order to actually choose the right tool, we really needed to think about our specific requirements: what were the significant factors which would impact most on us? This is what we identified:

1. A team approach: We wanted to have developers closer to UI automation. As I mentioned in my previous article — UI automation is a team effort, all team members should contribute to it.
2. Performance: We needed something fast, in order to connect test runs into the continuous delivery pipeline. This is very important for the overall quality of the product. If you have your test runs before you ship anything to customers, you significantly reduce the risk of shipping something bad.
3. Stability: We needed something reliable, something that would not fall apart with the next iOS release or stop working if Xcode were to update to new version.
4. Smooth integration: We needed something that integrates natively and smoothly into the development environment with the minimum of setup and overhead.

We decided not to restrict ourselves to a cross-platform solution, but rather, take something really optimal and specific for iOS and make the best out of it.

Taking all of this into account, it was quite obvious that Xcode UI Tests — was the one and that we should try to incorporate it in our development practices more heavily. It is based on a native for the team language — Swift, allowing a black-box testing approach to a sandboxed app with the minimum of impact on our main app code. It’s the fastest available tool on a market that integrates natively with no context switching for developers. It uses the same tools we use day-to-day: - Xcode, showing you the output of your tests in the same IDE you use for coding and in the same style with which we were already familiar from unit tests.

Although it doesn’t have a tagging system or natural language support for writing tests scenarios, it is still the best available tool that the team can use. And, as you will see below, those disadvantages could be mitigated with the use of certain techniques.

Performance boosting tips

OK, so we chose Xcode UI Tests. Now for some more details around how it’s all set up here at Trainline.

The basics, such as Xcode setup or adding UI Test targets to the application, can be found in the Apple documentation, and there is also a handful of tutorials online. So this post will instead focus on some aspects related to the performance of the tests.

Here are some tips, based on the experiences of our team, for using Xcode UI Automation on a real app over some period of time, which will help to speed tests up and improve the stability of test runs:

• log out any active users between test runs and clear data. This makes all tests run independently on a fresh setup of the application, so you eliminate any possibility of failures caused by some garbage being left over by a prior test
• use stubbing data for most of the tests. This will allow you to minimise any slowness with real service latency of responses and potential variations in responses
• write tests “wisely”, meaning that it may be prudent to combine a few scenarios in one in order to reduce the amount of runs with the same coverage
• do not over-complicate your tests. Tests should be quite deterministic and not do too much. You really need to find some middle ground on what tests should look like
• split tests into groups. This will allow you to do more careful selections if needed on CI to run a group of tests in a specific area, such as a payment system, for example — it should just be a matter of pointing your CI to specific folder
• select some sanity tests to run on each commit on CI. Do not try to run everything every time as it will take a long time and not really bring a lot of value
• use expectations to make sure you are interacting with elements that exist - this will minimise occurrences of false failures of tests

Another trick worth considering is to disable any animations within the app. You will be surprised how much time you gain by doing this! The downside of this, of course, is that there becomes a point when you can argue that this is not really running the real app and you will be testing modified code and I would agree with this. However, generally speaking, this is not as bad as it may sound, as we are only disabling UIView animations and so no functionality is being changed. So if you are ready to compromise a little bit on that in order to gain a performance boost for your tests, this is a neat trick!

CI integration

Another important question is how to integrate tests into the delivery pipeline. If a test suite is not integrated and, instead, left on a manual execution cycle, it gets forgotten and starts to rust, so they end up being a complete waste of effort and time. So in order to really benefit from UI automation tests, they have to be included in the continuous delivery pipeline and in the development process overall. And really importantly to understand why we do that, the main reason being to minimise the possibility of delivering a broken product. Making sure that automated tests are running constantly on any change we do should guarantee better stability and minimise any possibility of some simple, but costly, error slipping through. Also UI automation is the perfect tool to offload manual testing overhead to free up some valuable QA time to do exploratory testing.

Taking into account all of the above, we began with a simple setup of running all tests we have on anything we were about to merge to the development branch. This proved to be a good step to start from: while there were not many tests to start with, it made sense to have them all running often, so we could see how stable they were and what performance was like. We agreed that to get anything merged into the development branch, it should have UnitTests and UI Automation passed.

But with your test suite growing, it is inevitable that it will at some point take too long to run all of the tests every time. And this is where the idea of splitting the tests into multiple suites came to the fore.

Ideally you want to have a selective run when working on a feature or are fixing some issue and you just need to make sure that the area you are affecting is not broken, or that the feature you are implementing works as expected. But as you will see, there are certain limitations and problems around this. So for some time we decided to implement a simple solution of having two tests bundles: one with some sanity tests to cover most of the app functionality in a few tests on the most important key areas; and the second — all of the tests. With this, we can run a small subset of tests on every commit and before merging to development, and a large set of tests to run over night on the main development branch to check if all the work done during the day is still stable.

Although the Xcode UI Tests tool does not have a tagging system to define a group of tests, unlike Calabash, for instance, we still have few options to implement such functionality.

Using Schemes

The simplest way is to set up different schemes inside your project. Basically, for a specific scheme you can choose what to run, specify tests, bundle of the tests or suite to run on CI. And this is quite simple to do, you do not need to duplicate code or targets.

You just need to add new scheme in the “Manage schemes…” menu or duplicate existing one with tests. And you will be able to specify what tests to run when this scheme is selected. It should be dead simple to configure CI job to run this custom scheme. And you will have just subset of tests that got run there with results for them.

This setup lead to one complication though — this is all manual process, and in case you need to change anything when number of tests grown up significant, it will be really easy to lose context there. So maintainability cost will be quite significant with such approach.

Using the command line

Luckily, Apple introduced a different way of doing that with the new Xcode 8 on the last WWDC. It’s a slightly more complex way that requires some more setup on CI and some work with scripts, but it gives a more automatic and flexible way of doing that.

I’m talking about two new console commands that allow us to split builds from the test run.

One is build-for-testing which allows us to do only a build of a project and produce some resources that could be shared across a few build agents to run tests.

xcodebuild build-for-testing -workspace 
                 -scheme 
                 -destination

And another one: test-without-building. This command allows you to only run the test if the application was already built with the previous command and we have the generated resources.

xcodebuild test-without-building -workspace 
                 -scheme 
                 -destination

Also, there is one more new addition to the family of commands for console runs of the tests or, rather, an addition to the normal xcodebuild one.

xcodebuild test -workspace 
        -scheme 
        -destination 
        -only-testing:TestBundleA/TestSuiteA/TestCaseA
        -only-testing:TestBundleB/TestSuiteB
        -only-testing:TestBundleC
        -skip-testing:TestBundleD/TestSuiteD/TestCaseD

So, by using -only-testing and -skip-testing we are able to set up more flexible and targeted ways of running a subset of tests on CI and it could be done per feature, or just to achieve some parallelisation for running tests which will become quite important, at some stage, as the test bundle grows and thus the time it takes to run all the tests becomes bigger.

Summary

In conclusion, I would like to say that, at Trainline, we as a team went on quite a long journey trying different variations of tools and setups before ending up with Xcode UI Tests. In short, our journey looked like this:

KIF -> Calabash -> Appium -> Calabash -> Xcode UI Tests

Yes, we even tried to implement Calabash more than once. The second attempt was more successful and we used it for some time. What we learned here is to continue searching and trying. It’s impossible to achieve something without trying. We failed a few times, but we kept looking and trying and now we have a highly maintainable, stable, performing solution for our UI automation requirements which we keep improving.

<< Previous: Stubbing for UI Automation on the Trainline App