Emcee — the test runner for iOS developers

Avito is the most popular classified in the world with millions of daily active users and hundreds of business scenarios inside the mobile application.There are thousands of UI tests in the Avito iOS app. We heavily count on them as a good substitute for the manual testing process, although we still have some manual test cases.

We created the Emcee test runner to support this complexity and scale of autotesting. It allows us to horizontally scale the process of iOS autotesting across physical machines using iOS simulators. Emcee is available on our Github repository.

Tests state of the union

The number of tests in the Avito mobile application is constantly growing. The majority of them consist of fast component UI tests. This kind of test combines the interactivity of XCUI tests with the speed of unit tests. In fact, these are application screens in isolation but with interactive checks of their features. The network layer is mocked, and there is an option to open and test just the necessary screen of the mobile app. We run UI tests per pull request with two versions of iOS — the oldest supported and the newest one. We launch 3000 tests for every developer’s pull request. And we have 30–40 PRs per day.

But we have already abandoned running the XCUI e2e tests on pull requests for a year now: they are too coupled to the backend (end-to-end), and quite often it can cause flakiness and 50x errors. For example, a new version of backend service can be rolled out, or the database can start the process of updating during a test run. However, we run XCUI tests before releases and every night, with reports being delivered to the team in the morning. There are about 600 end-to-end tests left in our test model.

The numbers

If we just run our component tests suite on a single Mac mini with a single simulator launched, it would last 6 hours. In fact, the build takes 15 minutes on CI.

Doing the same thing with our 600 end-to-end tests will take more than 40 hours because tests are run with four different versions of iOS. But the build with all these regression tests lasts 50 minutes because everything is parallelized across our farm of build machines.

Emcee, the test runner

The available speed of our builds is achieved by using our Emcee test runner. It has some crucial for our task options. Emcee:

• Manages iOS simulators which are launched on our Apple computers farm. We have ~100 Mac minis in data centers.
• Organizes a queue of tests.
• Sends analytics for test execution to Grafana.
• Performs the required number of restarts using different strategies.
• Runs tests only where they can pass. For example, only on computers with the installed simulator and Xcode of the required version.
• Allocates the maximum possible resources for executing a set of tests, depending on the priority.
• Integrates test run results into our internal infrastructure: test management system and test reporting system.

What do I need to try Emcee?

First of all, you’ll need the desire to learn new things and the will to boost your test runs. We’ve done a lot of work to make it easy for you.

Then, Mac computers. We use Mac minis because they can be stuffed into racks. You can also use laptops if you have free ones lying around. You can also rent bare metal Apple machines from providers.

And finally, you need your iOS application with tests.

We’ve written a short guide how to get started quickly.

Who is Emcee for?

Feel free to try Emcee if:

• You want to reduce the time to market of your application.
• Your autotests runs have started to take a significant amount of time on the regression stage and pull requests, and now you’re looking for a way to speed them up.
• You want to rely heavily on autotests instead of manual testing, and you understand that many tests will be required, but you could not find a tool to run them.
• You’re fond of cool technologies and want to collaborate with Avito engineers.

What do people who already use Emcee say?

Anton, Bolt:

We obligatorily run UI tests when a feature branch is merged into the master branch or on pull request. When we had about 150 tests in the suite, it took about 10 minutes to run them on a single Mac mini. In our case, this was ok. However, when the number of tests grew to 500+, the run’s time increased to 30–35 minutes. We realized that this was too long to get feedback on the application state.

We considered moving to the cloud to run tests faster, but the cost of using MacOS instances pushed us to look for alternatives. As a result, we came to the Emcee runner from Avito: we set up a farm of 5 machines, on which we run 5 simulators at each of them. As a result, the time of test runs decreased to 10–12 minutes.

Evgeniy, Ivi.ru:

When we started implementing UI tests to speed up regression testing actively, we encountered that they took quite a long time to complete. At some point in time, the test run began to take an hour and a half! Emcee made it possible to parallelize tests on several Mac minis and significantly reduce the run time.

Andrey, Space307:

The iOS autotest project in our company started in an environment where the application had already grown, and regression was a bottleneck for releases. We released the application no more than once a month and were overwhelmed with regression.

To run tests, I wanted to build a reliable in-house infrastructure based on an out-of-the-box and preferably open source solution. We found only one such tool — Emcee. I had to tinker a little with the implementation, but you set it up once, and then everything works like clockwork. You have to just take it and connect new nodes and measure the acceleration of test runs.

We have about 800 e2e iOS tests that run in 2–3 hours at this moment. Our release cycle has been reduced from 4–5 weeks to 2. It has become much easier for testers to allocate time for additional activities in the form of process improvement and development of test documentation.

We are ready to support you

We are ready to take the time and support you with the implementation, setting up test runs, and we are prepared to give some advice on how to organize your own test farm.

Catch us on our landing page or join our Telegram chat.