Automated mobile app testing at Elements
Author: Danny E.
As a company that creates high-grade apps, we want to deliver a certain level of quality to our customers. One way to achieve this is through testing. We have been testing our apps manually since we started creating apps, but to achieve a greater level of quality and efficiency, we have been looking into automated testing, specifically for mobile apps. With automated testing a big part of manual testing can be automated, mostly usable for regression testing. Using automated testing, newly developed functionality can easily and efficiently be re‑tested.
Because of its complexities, perfect automated mobile app testing is still considered the holy grail in the world of mobile app development.
With all this in mind, we started to search for the tool to fit our needs: easy to use and cross-platform (iOS and Android) support. We have looked into several solutions, both open-source and commercial. Of the open-source solutions we have tested Calabash and Appium. SeeTest is the only commercial product we have tried extensively in this comparative research.
Calabash
Calabash is easy to use with Cucumber, a simple and natural language to describe tests in more or less plain text English. Cucumber is written in a business-readable domain-specific language and serves as documentation, automated test and development aid — all rolled into one format.
We have used Calabash to try automated testing with some of our projects, and found out that it mostly works fine for Android, but not as well for iOS. The reason for this is that on iOS, Calabash seems reliant on the use of accessibility labels and doesn’t really have any other good options of selecting the proper objects to write your test cases with. As we were looking for a good solution that covers both Android and iOS we started looking at Appium.
Appium
Appium (supports both iOS and Android) currently seems to have a big community supporting it and is being actively developed on. It serves as a layer between Selenium, a testing framework for web applications, and the mobile devices you want to automate your tests on (or mobile simulators, for that matter).
The Appium automation tool generates test scripts based on recorded taps and other gestures made on real devices or simulators. Using the WebDriver protocol, Appium supports many popular web languages, such as Python, PHP, Ruby and JavaScript with Node.js. This is great, as it can reach a large audience, but it also has a downside: The Appium libraries for all these languages are maintained individually and independently from the main Appium automation tool.
For example, during our tests we have used Appium in combination with Python and the problem we encountered is that the generated Python code turned out to be outdated or at least incompatible with the main Appium automation tool. Most likely this will be fixed in future versions, but for now (and it has been like this for a while) you will basically need to rewrite the test scripts yourself and just use the object inspector and recording functionalities of Appium to help you with the manual test script writing. Also the documentation is not always up-to-date.
Another thing we noticed while checking out Appium, is that for both Android and iOS, most of the objects inside apps are only accessed using XPath and references by ID, instead of another descriptive identifier. There are some object types that can be recognized by name, like buttons, but for most others you will just need to tell Appium the index of the object (for example the first checkbox on the screen or the fourth button). This can be very error prone, as apps change constantly during development. We definitely want to minimize our efforts of keeping the test scripts up-to-date, and by using names instead of indices this is easier to maintain.
Although the automation tool crashed frequently, overall Appium seems like a reasonable solution and you can eventually get it all to work. Our conclusion is that it has a high learning curve and requires a lot of manual scripting and research to get things rolling.
SeeTest
Besides the aforementioned open-source solutions, we also looked into one of the commercial mobile test automation products. Recommended by one of our clients, we also tried the SeeTest test automation tool, developed by Experitest.
As we started with the SeeTest 30-day trial you notice immediately that they tried to make it a complete package that works right after installation — and it does. Hooking up iOS devices, Android devices, it just works right away.
With devices hooked up you can start recording your tests with the help of an object inspector. The objects in the app that you use in your tests can be added to the very useful object repository. The purpose of this is that you can assign different ways per platform (iOS, Android, BlackBerry and Windows Phone) how to find that specific object. For example, a submit button of a form is accessed differently on iOS compared to Android, but in your repository this is one object. In your test script you reference to this repository object to minimize the time it takes to write scripts for multiple platforms. Also, if the way you need to access a certain object changes during the app development cycle, you will only need to change the object in the repository and all scripts that use it will update accordingly.
Just like Appium, SeeTest can generate test scripts in several languages, and they actually work out of the box. Subsequently, the generated test scripts can be used to execute the tests in a continuous integration environment.
One downside we found is that the test execution seemed to be a bit slower compared to Appium. Also Seetest’s automation tool crashes occasionally.
Conclusion
Overall SeeTest seems way ahead of its open source competitors, but it also comes with a price tag, as it is a commercial product. The (annual) fee does include support with a response time of 72 hours, which is quite nice. Our experience is that Experitest helps out with basically everything, from integrating the tool in your environment to best practices. Their support and delivery of a complete package is what differentiates them from the other products we looked at.
Although SeeTest is far from the perfect solution for automated mobile app testing, we decided to implement SeeTest as it seems the solution that fits our needs best at the moment.
So, in short, we are now using SeeTest in our mobile development cycle to deliver even higher quality apps. We will post our further experiences with SeeTest on this blog, so be sure to keep an eye on our website.
Originally published at www.elements.nl on August 31, 2015.
