Emcee — Open Source Tool for iOS UI Testing Infrastructure

A short intro to our tool for running iOS UI tests.

Emcee means a master of ceremonies.

Some Key Facts

• We have 900+ native (XC UI based) UI tests for our iOS app
• We run non-flaky UI tests, around 200, on every pull request
• We run all UI tests many times throughout the day
• We test against 3 iOS versions, and we use only iOS Simulators
• There are about 30 build machines (Mac Mini) in our build farm.

We need a fast, flexible, scalable, and stable way to run our UI tests. We decided to run tests in parallel using our Mac Mini machines. We haven’t found any off-the-shelf tool for doing that, so we created our own — Emcee. It is open source and available on GitHub.

What Does Emcee Do?

Emcee is a command line tool that we run from our CI scripts. In fact, it allows us to run all our UI tests using a single command line invocation.

Our pipeline for running tests is simple: we prepare Xcode build artifacts using `xcodebuild build-for-testing`. Then we run Emcee, and after running all tests it produces outputs, like Junit reports, etc.

When you start Emcee during a build, it automatically spawns remote workers for itself. Then the workers will start taking jobs from the main Emcee process (the one you start on your pull request). Once all jobs are finished, Emcee terminates, and all workers quit as well.

Currently, Emcee uses SSH to deploy itself to the build machines. We run UI tests side-by-side with TeamCity build jobs.

Also, Emcee eliminates the need to maintain the daemon state, deploy the service, or perform any administration of the test queue. Should Emcee ever crash, it brings itself back to life when next build starts. If any of the worker Macs is on maintenance, it automatically picks the next working machine and run tests on it.

Runtime Dump

Emcee is capable of locating all UI tests in a given xctest bundle. We call this feature runtime dump, and though currently it requires support on the project side, it provides quite a bit of flexibility:

• It allows you to get information about all tests in the xctest bundle.
• You can then filter available tests and run only selected ones.
• Emcee uses runtime dump to validate the inputs, i.e. fails if you attempt to run a test that doesn’t exist. It’s used to split tests to run across multiple build machines.

We use runtime dump a lot. For example, we have infrastructure tests, regular UI tests, skeleton UI tests that don’t test anything, but provide a placeholder for a test to be written, some app performance tests, etc. We filter these types of tests using runtime dump and run them on PR and throughout the day; we have special configurations to run infrastructure tests and performance tests since they are not part of regression test suite. We use runtime dump to find all affected UI tests on PR and run them on pull request in addition to stable UI tests we are executing.

Read more technical details on our GitHub Wiki.

Queue and Workers

As I noted above, the queue instance is run on each pull request, and then it launches its own set of workers. Below is an animation showing how the queue feeds the workers with UI tests.

Animation of how Emcee workers run UI tests.

Buckets

Emcee has several ways to generate the queue contents. Internally, it employs the concept of buckets, each bucket may have N tests. Buckets are used to start tests on simulators. Emcee will run all tests in a bucket on a simulator without restarting the test runner app.

The method of splitting tests into buckets is called schedule strategy. Currently, Emcee supports following strategies:

• Individual: each bucket has exactly one test in it. Downside: since we are running XC UI Tests, there is an overhead of launching test runner app for each test.

In the individual schedule strategy, each bucket wraps a single test.

• Equally divided: we divide all tests equally among all workers. Downside: some worker may (and definitely will) become available earlier than others, effectively staying idle, while the slowest worker will slow down the whole UI test run.

In equally divided strategy, all buckets have almost the same number of tests.

• Progressive: the number of tests in bucket varies depending on the position of the bucket inside the queue. If the bucket is in the head of the queue, it will have more tests in it. Buckets in the tail of the queue will have fewer tests in them. While there is some increased overhead of launching the test runner app, load balancing between the workers is better: whenever a worker becomes available, it picks the next bucket from the queue, which is will highly likely to have have fewer tests in it. Essentially, all workers finish more or less simultaneously.

In the progressive strategy, buckets have various tests in them, depending on bucket’s position in the queue.

Emcee Needs Your Attention!

Emcee is an open source project. While we’ve been using it for a long time, for almost a year now, we’d love to see that it has an impact outside Avito! Feel free to open an issue on GitHub and we will do the best to help you start running Emcee in your environment.

We will continue posting about Emcee and UI testing at Avito. Stay tuned!