7 best device farms to test your iOS and Android applications
Cloud device farms are an important element of the Mobile DevOps pipeline and allow a great reduction in labor costs for testing, as well as increasing the coverage of devices.
Back in the old days, when the smartphone industry market was ruled by Nokia and Microsoft, one characteristic pain of mobile development was discovered — broads range of devices by features and modifications of OS. Testers had to test the applications not only on different versions of the OS, but also on different physical devices.
After the release of iOS, the range of phone models was small, therefore eliminating the problem with them. However in the world of Android, the problem of fragmentation has risen to its maximum. There are thousands of various models on the market, with more coming out on the daily basis, while apps and games are expected to work flawlessly on each one of them.
Additionally there are different versions of firmware on these models… leading to the realization that manually you need a bunch of man-hours to test each individual release.
Since the problem with the diversity of the device fleet has arisen in ancient times, Nokia has already begun to offer its developers a service for renting smartphones. It was possible to put the application on the device and in the remote control mode to go through the key scenarios. It was free, while having long wait times to access devices, but this test gave the developers hope that the software will work correctly on different smartphones and therefore no user problems would appear.
The initiative was good, and device farms, received its further development — the OS itself have begun to provide tools to simulate user actions. That is, the tester or developer writes a script that pretends to be human. Adding to that convenient DevOps tools to build and automatically run these scripts — and you can forget about the dark picture, with lots of man-hours for each release.
Today we will talk about popular cloud device farms: Firebase Test Lab, Samsungh Remote Test Lab, AWS Device Farm, Sauce Labs, Xamarin Test Cloud, Perfecto.
How it works
The built-in automation of UI-testing appeared relatively recently: iOS 9.0 (XCTest UI) and Android 4.3 (UI Automator, although Espresso worked with Android 2.2).
On the lower level are all implemented quite simple: together with the application, a special script is launched, which causes the OS to simulate the work of a real user (touch, gestures with your fingers and pressing the buttons of the device).
There are several popular subsystems for executing scripts: Appium, Calabash, Espresso, Robotium, UI Automator for Android, XCTest for iOS, which in turn support one or more programming languages — Ruby, C #, Java, Python, Swift.
Since the problem with the range of the devices is relevant first of all for Android, we’ll get to know the Google and Samsung farms first.
Samsung Remote Test Lab
The first in line is the Samsung Remote Test Lab service.
This service is technologically outdated and would not be worth mentioning in our article, if not for one thing. Samsung is the leader and one of the lawmakers in the Android smartphone market, so early access to the flagship innovations will allow you to test the operation of your application even before the devices appear on the market. Plus there is access to devices based on Tizen: Z smartphone line and Gear smartwatch.
Work with the service is as follows: you reserve the device and run a special Java application that provides remote access to the screen and input devices (touchscreen, buttons). At the moment, there are 25 models of smartphones and tablets, each of the models in several copies and modifications.
Automation is done at the level of manual recording of the sequence of events, and the application has to be installed manually.
In general, the possibilities are limited, but it’s also completely free.
And the part is that Samsung Remote Test Lab supports a convenient mode of remote debugging! So we can safely recommend this service as an additional farm for manual testing on Samsung devices.
Firebase Test Lab for Android
The next service is developed in Google and is called Firebase Test Lab for Android. In General, Firebase is good for teams, specializing in development for Android, and the farm is just one of the tools.
Currently, not many device models are available (about 30, the list below in the screenshot), but there is also the ability to run on emulators.
Test Lab is included in a single subscription to the service Firebase and can be completely free for the start. Unlike the Samsung service, Firebase Test Lab is easy to integrate into a DevOps pipeline.
Test scenarios can be implemented using the Espresso, Robotium, UI Automator 2.0 and Robo tools. Screenshots are made during script execution.
In general it is a good solution for Android-development of small projects using native tools.
Specialized Professional Farms
Samsung is not the only option for Android-developers, so we will continue our review with professional farms that support iOS, have a large bunch of models and cost money.
AWS Device Farm
AWS Device Farm has nearly 400 devices available (about 100 models), prices from 0.17 dollars per minute, there is also an Unlimited (!) option and 1000 first minutes free.
There is a high quality of service and integration APIs for DevOps pipeline. For scripting, you can use Appium (iOS + Android), Calabash (iOS + Android), Espresso (Android), Robotium (Android), UI Automation (iOS) and XCTest (iOS) and several others.
Xamarin Test Cloud
The next professional service is Xamarin Test Cloud. More than 2500 (not a typo) of real devices!
iOS, Android and a full set of features are supported (screenshots, automated scripts, video, and also promise remote debugging and recording coming in the future). Price start from 99 dollars a month.
The service is ideal for developers of cross-platform solutions (Xamarin, React Native), and projects with a wide user audience (high coverage of the models). Supports automated scripts based on Calabash and Xamarin.UITest.
And we finish our review with two multi-functional services that offers comprehensive services for testing a wide range of software: mobile and hybrid applications, as well as mobile websites.
Meet Sauce Labs. One veteran of the market of automated testing. Jason Huggins, a Selenium creator, put his hand to this service.
Sauce Labs is a mature service for mature teams. Prices start from $149 per month, there are native and hybrid applications for iOS and Android and the ability to organize your own private cloud or to test manually.
There is support for integration with DevOps pipelines and running on emulators/simulators, although there are no more than 20 device models.
In other words, the fans of Selenium will like this service.
And we finish our review with of one of the oldest farms of devices from the Perfecto company . In the days of Symbian and Windows Mobile, the company started to offer their devices for rent. The prices were high, however it was possible to quickly run the application and make sure that it works (or does not work).
There are about 20 different models available for iOS and more than 50 for Android. As a framework we suggest to use Appium. You can manually test for free, while automation will cost from $299 per month.
We have reviewed the most interesting cloud devices farms that allow us to remove the annoyance of manual testing. If you have not decided yet where to start, then you can focus on the following:
- if you are single developer and write in Java/Kotlin for Android, then feel free to use Firebase Test Lab (for free);
- if you want to connect remote debugger — there is only Samsung Test Lab (for free);
- if you are looking for a service with the maximum device coverage — we recommend Xamarin Test Cloud (from $ 99 per month).
In any case there are a sufficient number of farms on any technology stack and wallet, so you can easily find a suitable solution.
Now you can finally release the exhausted QA engineers from the manual testing routine. It’s time to automate.