Guide to mobile app autotesting tools

Published in

Bumble Tech

18 min readAug 15, 2018

… for though it has many omissions and contains much that is apocryphal, or at least wildly inaccurate, it scores over the older, more pedestrian work in two important respects. First, it’s slightly cheaper; and secondly it has the words DON’T PANIC printed in large friendly letters on its cover.
— The Hitchhiker’s Guide to the Galaxy

My name is Arseny Batyrov and I work in the QA department at Badoo, where I am primarily involved in manually testing web applications. I also run courses on manual and automated testing for mobile apps.

Before creating a new course, I was considering which tools to tell my students about. I combed through the internet for articles that offered comparisons and, strangely enough, I didn’t find a suitable source of information. That’s when I decided to create one myself.

I had three aims:

1. Classify the tools in the autotesting stack. We do it to understand how they relate to each other and how well they work together;

2. Explain which tools are popular today on the market;

3. Discuss the most popular tools of each type and compare them based on several parameters.

The fruit of my labours is this guide to the most popular and simplest-to-master tools for autotesting mobile applications.

For more information, please feel free to use these extra resources:

● When choosing a tool, take a look at this comparison.
● If you want to know how automation is set up on mobile devices, check out this classification.
● If you want to earn more money, get to grips with this popular tool.

Disclaimer: I am not a testing guru and this article is not an exhaustive list. If you spot a mistake and want to add something to the guide, feel free to leave me a comment and I will update the article with any helpful new information.

App and tests

For starters, let’s work out what tools we’ll be working with.

There are two important elements which aren’t part of the automation stack: the app and the tests. All the automation tools access the app, and the app interacts with the user and other apps via one or more interfaces: GUI, API, a network interface, CLI and several others.

API (application programming interface) is the basic interface for interacting with other programs.

GUI (graphic user interface) is a graphics interface used for interacting with the user.

Net (networking interface) operates via a network and is used both by both advanced users and programs.

Tests may use all of these interfaces to interact with an app. In the case of manual testing, the intermediary between the tests and the app is the tester: it is the tester who converts the text of the test cases into actions in respect of one of the app’s interfaces.

To achieve automation, the tester needs to be replaced by tools able to interact with one or more app interfaces. Utilities are also required to run and generate a test set.

All these instruments are referred to collectively as the ‘autotesting stack’. To understand how they interact in the stack, they need to be classified. My proposed classification is arbitrary and is primarily required with a view to understanding the tools and their compatibility.

In all, there are four groups of tools: drivers, wrappers, frameworks and combined tools. Let’s consider them in more detail.

Classification of tools

Driver

Autotesting utilities, like other programs, can only interact with an app via a programming interface. There are special programs for working via other interfaces, namely drivers.

A driver is a tool that makes one of the app interfaces available via API.

Each interface, apart from API itself, requires its own driver. For example, if you send the command “Click on Menu” to the driver for the GUI, it receives this command via the API and forwards it to the app being tested. The command is then converted into a click on the menu button graphic. Drivers are not required (or are virtually not required) to interact with the app API because the interaction is programmed, but they are essential for working with other interfaces.

The most complicated drivers are those for the GUI, since this interface is very different from the customary ways in which the program communicates by code. At the same time, the GUI is the most relevant for automated testing of mobile apps, since it has to be used most often in integration testing. The most popular drivers for the GUI in respect of mobile testing are UIAutomator and Espresso for Android, and XCUITest for iOS.

Wrapper

In cases where the driver’s functionality is complicated and inadequate or inconvenient, an additional level is added which I shall refer to as a ‘wrapper’.

A wrapper is a program which interacts with the app via one or more drivers, increasing ease of use and expanding the driver’s capabilities.

A wrapper may perform the following functions:

1. Behaviour modification (without altering the API)
For example:
● Additional protocolling
● Data validation
● Waiting to perform an action for a set period of time

2. Increased convenience and/or level of API abstraction via:
● Using syntactic sugar — convenient names for functions, shorter calls to them; a unified style of writing tests

● Implicit driver management– for example, when a driver is initialised automatically without requiring every action to be spelled out manually

● Simplifying complex commands, such as selecting events from a calendar or working with scrollable lists

● Implementing alternative programming styles, such as the procedural style or fluent

3. Consolidation of API drivers

In this case the wrapper provides a single interface for working with several drivers at once. For example, this allows you to use the same code for tests on iOS and Android, just like in the case of the popular wrapper, Appium.

Framework

At the other end of testing is the launch framework. For the purposes of the present article I shall refer to this as a ‘framework’ for short.

A framework is a program for generating, running and collecting the results from a set of tests.

The tasks that make up the framework include:

● Generating, grouping and sorting a set of tests
● Setting parallelisation (optional)
● Creating fixtures
● Running tests
● Collecting results
● Generating completion reports (optional)

You can see that these functions are not only related to the testing of mobile applications — they may also be applied successfully when testing desktop and web applications. The thing is that a framework is not supposed to provide interaction between tests and apps; it only works with tests, and the app type doesn’t matter.

If drivers and wrappers can be visualised as being between tests and apps, then a framework is above tests, organising the running of them. Therefore, it is important not to confuse the terms ‘driver’ and ‘framework’. Of course, some frameworks have their own drivers for working with apps, but this is by no means a necessary requirement. The most well-known frameworks in mobile testing are xUnit and Cucumber.

Combined tools

Finally, one more group of tools used for automated testing of mobile apps are what may be referred to as combined tools, which combine frameworks and drivers (not only mobile drivers) and even development capabilities. Xamarin.UITest, Squish and Ranorex. all support the automated testing of iOS, Android and web apps, and the latter two also support desktop app testing.

So, now that we have classified the tools, we need to define the most popular ones in each category and compare them.

Survey

To determine the most popular and most commonly used tools, I conducted surveys (with a Russian speaking groups) on several sites, communities and channels for QA engineers, asking three simple questions. I didn’t limit the answers people could give so that I didn’t have to choose between different types of tools. I also gave people the option of specifying a different answer, which produced a rather long train of different utilities which don’t match the classification. The results don’t claim to be statistically accurate, but they do illustrate the trends in the mobile app automated testing industry excellently, as things stand in January 2018.

As you can see from the results, the leading utilities are those based on WebDriver: Appium and Selenium. The most popular frameworks are JUnit and Cucumber — the latter being the more popular, which is surprising, since it is in an entirely different category from the other. Official drivers are more popular than unofficial ones for all platforms, evidently due to the quality of support and greater capabilities than in the case of drivers developed by third parties.

The three most frequently used programming languages are Java, Python and Ruby (Java leading by a wide margin). I attribute the fact that Ruby made it into the top three to the popularity of Cucumber.

Finally, the spread in terms of platforms is largely as expected: Android is ahead of iOS by a wide margin, followed by mobile web. The only surprising thing is the answers concerning desktop applications for Windows in the final survey, but some combined tools allow you to test mobile and desktop applications simultaneously.

Having got to the bottom of which tools are popular, we now move on to comparing the most significant ones. First of all, I have produced a comparative table for each tool type, showing the capabilities of the tools in question. I have tried to gather the most up-to-date and reliable information regarding each tool, but it is possible I have omitted something. If you spot a mistake in the description, please do tell me about it in the comments.

Comparison of tools

Drivers

There aren’t that many drivers in mobile testing, and they are often developed by the same companies that develop the operating systems. Android has two official drivers: UIAutomator, which at present is on version 2.0, and Espresso. They are both part of the Android Testing Support Library, are developed by Google and are well documented. Besides them, Robotium and Selendroid are projects developed by third-party companies. All four products in one way or another operate on the Android Instrumentation Framework, a basic API which Android provides for interacting with the system.

To begin with, let’s consider the drivers from Google. Both tools are able to work with WebView and hybrid applications, both support development in Java and Kotlin, and both work equally well with emulators and real devices.

UIAutomator

UIAutomator supports Android versions from API level 18 (Android 4.3) upwards. It does not require its own code to be implemented into the project — i.e. it can interact with apps which have already been compiled. Moreover, when working with UIAutomator, the capabilities of the Android system may be used to the full: switching on geolocation, activating a system app, rotating the device, pressing the home button or taking a screenshot for example. That is why this tool is often used for functional end-to-end testing, as a stand-alone or with wrappers.

UIAutomator doesn’t have its own recorder for tests, but it does have UI Automator Viewer, a utility which allows you to obtain data on elements of the app running on the emulator or real device, and displays locators for them. In the same place it displays a hierarchy for all the elements, which is very convenient when using them in tests.

Espresso

Espresso, in turn, is mainly intended for white-box testing and was created as a tool for developers. It supports older APIs, starting with API level 10 (Android 2.3.3), but it does require access to the source code in order to run. Therefore, Espresso can’t be used alone with other apps and the Android system. However, it does have a recorder, which allows you to record simple script and use it at the initial stage of automation.

Overall, if you just need to test an app (aside from the issue of interaction with the system) and if you want to/have the option of working with source code, the best tool to use is Espresso. Moreover, Espresso offers useful functions such as the automatic synchronisation of tests vis-à-vis the UI app, and you don’t have to write all sorts of wait commands.

If, however, you need to test an app in conjunction with other apps or system functions and you only have access to .apk, go for UIAutomator.

Incidentally, these tools may also be used together, as they are part of the same library. You can even combine commands for both tools in a single test.

Selendroid and Robotium

Both Selendroid and Robotium were released before the official drivers and are still around to this day.

Robotium supports Android API versions starting from API level 8, and is able to work with WebView starting from API level 15. As for Selendroid, it works with a limited list of API versions, namely versions 10 to 19. Both tools are only able to access a single app, do not require access to the source code, and support work on emulators and real devices. In Robotium’s case, tests need to be written in Java, while Selendroid supports the WebDriver protocol which offers the option of using virtually any popular programming language.

Selendroid comes with the Inspector utility, which you can use to view the hierarchy of elements and record simple record-and-play-back tests. As for Robotium, it offers the Robotium Recorder plug-in for IntelliJ IDEA and Android Studio with similar functionality.

Overall these tools are developing much less vigorously than the drivers from Google, and their target audience is much narrower. Nevertheless, the results of the survey show that some companies continue to use them.

XCUITest

For a long time, iOS used the UIAutomation driver for interaction with the app. (Among other things, this caused confusion due to its name being similar to the name of the Android driver.) However, from iOS 10 onwards, Apple discontinued support for this driver and it was replaced with the XCUITest driver, which is part of the XCTest package.

The XCUITest driver supports iOS starting from version 9.0, while tests for it are written in Objective-C and Swift, as are the apps themselves. In order to test an app, access to its source code is not required, but from Xcode 9 onwards the driver is able to test several apps, including system apps, simultaneously. ‘Off the shelf’ XCUITest only allows you to launch tests on simulators, however using some third-party tools you can also force it to work with real devices.

XCUITest has its own recorder, integrated directly into the Xсode interface. You can use it to record simple UI tests, and also to find UI elements and their properties.

Wrappers

Appium

Appium is the best-known wrapper at the present time. It allows you to test an app virtually regardless of platform, type and system version. Of course, this approach has several advantages and drawbacks.

Appium supports a large number of drivers, and not only mobile ones:

iOS
● XCUITest
● (deprecated) UIAutomation

Android
● (beta) Espresso
● UIAutomator 2.0
● (deprecated) UIAutomator
● (deprecated) Selendroid

Windows Driver (for desktop Windows apps)
Mac Driver (for desktop Mac apps)

Support for such a variety of drivers is achieved in a rather interesting way: Appium uses the WebDriver interface version universally known as Selenium WebDriver. Besides supporting a large number of platforms, this approach also has other advantages:

● Users have the option to write tests in any language which supports WebDriver, which includes virtually all popular programming languages. Moreover, this allows you to free tests from using the app’s own programming languages. This is most relevant for iOS, as tests on XCUITest can only be written in Xcode. With Appium, however, in this case you can use any language and any suitable development environment.

● transition to hybrid and web app testing: the WebDriver protocol has (almost) become a standard for web automation.

● You can use any test framework — in one way or another, almost all of them work with the WebDriver protocol, which means that they won’t experience problems connecting to Appium.

● There’s no need to add anything to the app code because each platform uses drivers that don’t require access to the code. Ease of deployment aside, this means that you can test the app build that users will see, and not a special test build.

We are mainly interested in support for mobile apps, so we’ll focus in more detail on the implementations of the UIAutomator 2.0, Selendroid and XCUITest drivers.

The simplest one is UIAutomator 2.0, which Appium interacts with directly, sending it the necessary commands. UIAutomator 2.0 works with Android versions 5.0 and higher. With 4.2 to 5.0 you can use UIAutomator 1, while the Selendroid driver provides interaction with older versions. For the purposes of interaction with XCUITest and for circumventing several limitations, WebDriverAgent (WDA) from Facebook is used. WDA runs as part of a simulator or a real device and sends commands via API XCUITest.

Appium’s drawbacks are a product of its advantages:

● The tests fail more often than those written for native drivers due to errors in the code of the wrapper itself. This is particularly relevant for iOS, since WDA is added.

● Appium is not capable of finding and comparing images in the apps, and doesn’t work directly with alert notifications in Android.

● There’s limited support for Android API < 17, but this might be rectified by using Espresso as a driver.

Nevertheless, the Appium wrapper is very popular and is developing vigorously, so many problems may be resolved by the community in due course.

WebDriverAgent

We move on to the WebDriverAgent wrapper, which Appium uses for working with iOS. In reality, this is an implementation of the server part of the WebDriver protocol which allows you to manage devices on iOS. What’s more, the functionality available is pretty comprehensive: you can run and stop an app, use gestures, and check whether elements are visible on the screen.

The wrapper works both with simulators and with real devices. For the purpose of discovering elements, there is an inspector interface which opens up in the browser. The wrapper itself is supported by Facebook and Appium commands and is developing quite vigorously. At the same time, it can also be used separately from Appium, if you prefer.

Calabash

The next very popular wrapper is Calabash for Android and iOS. This tool was developed by Xamarin but the company discontinued support in 2017, so now only the community offers support.

Each OS has its own wrapper: Calabash iOS or Calabash Android. They both support WebView testing and Ruby/Jruby programming languages. Calabash Android does not require access to the app source code, but Calabash iOS does require a connection to the Calabash framework app code. Also, in order to work with views outside the native iOS app, an additional tool is used, DeviceAgent, which allows you to detect these views and to interact with them. Theoretically, this means that you can also test in WebView, but in practice it’s better to stick to those views iOS gives to your app: various overlays, designed for confirmation, sending letters and pasting photographs. Calabash Android supports work with WebView, but on a quite limited scale: it’s only capable of tapping, entering text and delivering alerts.

Overall, Calabash is a reasonably stable and fast tool, which has useful functions for getting an app into shape (“back doors”) and which supports integration with Cucumber “off the shelf”. But due to a lack of official support, problems may arise when using it, and the community cannot guarantee they will be solved quickly.

Earl Grey

Earl Grey is an implementation of sorts of Espresso for iOS, and it has also been developed by Google. In this case everything is as standard for iOS wrappers: it must be added to the project in Xcode, tests can only be written in Objective-C and Swift, and you can only test one app; it cannot see external views. However, testing on real devices is supported. In itself it is an attractive wrapper, and more or less regular support is available, but for whatever reason it is not popular with testers.

Frameworks

Frameworks are the least related to mobile device testing, because they work with tests and integrate with any drivers and wrappers. I’m not going to consider them in detail (there are hundreds of pieces on the internet on this subject), but I will confine myself to a superficial comparison.

xUnit and TestNG

The most popular frameworks are those which are part of the xUnit family. They were created as tools for unit testing, and the first service of their kind was JUnit. At the same time, they are capable of working not only with test modules, but also with any others. Thanks to their universality, the xUnit frameworks are used everywhere and dominate the field of web app testing. JUnit only works with Java, but there are currently implementations of these frameworks for virtually every popular programming language.

Somewhat different from this group is the TestNG framework, which has more varied auxiliary functions.

Cucumber

Also popular are BDD frameworks, first and foremost Cucumber. Unlike xUnit and TestNG, in this case the tests and steps are generated on the basis of documentation and are written in Gherkin (which is similar to natural language). I should clarify that Cucumber is focused on acceptance testing, and carrying out automated function testing on it is quite complicated.

Combined tools

Xamarin

Xamarin is a service for the mobile development and testing of apps, which has its own farms with mobile devices and tools for automated testing. Development is mainly in C#, and it has its own recorder.

Ranorex

Ranorex is a tool for the automation of virtually any apps. It is able to integrate with Selenium and test mobile apps on emulators and real devices. It is only available for Windows and it uses C# and VB.NET as programming languages for testing. It also has a recorder for tests.

Squish

Squish is also able to automate web, mobile and desktop apps, supports BDD, and has its own recorder and IDE. You can use Python, Perl, JavaScript, Tcl or Ruby to write tests.

The main advantage of this solution is the full testing cycle: there is no need to configure separate utilities and their interaction. However, all these tools cost money, often have closed source code and are rarely used for mobile apps testing. Therefore, I cannot recommend them without reservations.

Conclusion

This is, of course, far from a complete list of the tools available for mobile app functionality testing. This article has not covered KIF, Frank, SilkMobile, TestComplete and many other utilities. Instead, it is intended as a guide to the main tools, and I hope it can help people understand the mobile app autotesting stack and feel confident they won’t make a mistake when choosing servers. If you have something to add to the topic, please write to me in the comments, which I promise to read, and I’ll add anything interesting to the article.

Finally, and for your convenience, I have put all the tools in a single table and made a list of useful links. You can find these materials in the ‘Cheat sheets’ section below.

Acknowledgements

Huge thanks to the entire Badoo team for help in preparing and editing this article — you’re the best! Particular thanks go to Alex Khozya, Vitaliy Kotov and Viktor Karanevich.