Mobile Automation Frameworks

Advantages and disadvantages — getting the best of each

Intro

We live in a rapid world, where everything becomes mobile. If you are any kind of service provider or seller, you need to be available for your customers on mobile platforms or you will lose your relevance.

Mobile devices can run mobile (web) sites, native apps or hybrid apps. each of these has its place and justification, but native apps utilize the device operating system resources and features the best way, because they are directly communicating with them. Most big companies or mobile oriented startups develop native mobile apps, i.e. iOS and Android apps.

when you test mobile apps, you need to have access to (physical) devices (simulators are simply not accurate enough for customer scenarios — having limited and sterile storage) and testing scenarios can take quite some time.

User Interface Automation

As web automation made a revolution in UI (User Interface) testing of web sites, the same thing happened to mobile apps UI testing in recent years.

Selenium was a big success and still is one of the biggest WEB UI automation solutions. It allows you to develop a super quick and effective UI automation using open source implementations, while getting the support of a worldwide community of developers. Selenium can read and interact with the DOM (Data Object Model) of the web page and perform actions on the page web elements. There is no dependency on the web site developers/releases and you can use your preferred programming language.

On the basis of Selenium, a mobile solution was built and called Appium. Appium can test all types of mobile apps: web based, native and hybrid. Similar to Selenium you can code in almost any language you want, you get a worldwide community support and you have no critical dependency on app developers/releases, therefore making a black box testing, interacting with the mobile elements it can read.

Mobile operating system providers (Apple — iOS and Google — Android) are providing their own native testing framework (iOS’s is called XCTest/XCUITest & Android’s is called espresso). It “talks” directly with the mobile elements and is integrated as part of the development environment, therefore test code needs to be written in the app development language (iOS in Swift, Android in Java/Kotlin).

The way Appium works is that it performs as a mediation layer between the client running the test code and the native testing framework performing the actions. Appium client is installed on your development environment (i.e laptop) and Appium server is installed on the mobile device and then “talks” with the native automation framework.

So, which automation framework should you use, Appium or native?

Why use Appium and not Native?

1. Open source + worldwide community support. Others in the world have the same struggle you have, so most likely, you will find solutions and workarounds on the web (just Google it). Java is the most common language so you will find many results for it.
2. Use your preferred language (Java, JavaScript, Python, php, you name it). No need to invest time in learning a new language.
3. You can have one Appium repository that handles the testing of both iOS and Android (at least the basic Appium driver layer can be the shared).
4. Black box testing — advantages:

• You have no dependency on app releases and you’re not touching the app code repository.
• The Appium repository is completely in your (QE) responsibility — so you can utilize it as you want. You can add API verifications as part of tests and you can add SDKs to any service. Sky's the limit.

Why use Native and not Appium?

1. Because Appium setup is hard and it takes some time to establish a reliable Appium setup. You need to establish at least 95% stability before you can calmly depend on its results and run it night and day. Native setup is almost seamless.
2. Because Native testing is MUCH MUCH faster. how fast? 2–8 times faster depending on type of test. It happens because the way it works is so different. Apart from Appium being another layer in the middle, it actually scans the DOM and waits for an element to be found. Native testing checks for a coded attribute condition. it doesn’t scan the screen elements over and over again like Appium does.
3. Because Native testing can make significant shortcuts: It can start a test at a specific page without going through the entire flow to get to it and it can run on mock data and not need to test both backend and frontend all the time. These 2 options make tests super fast and reliable.
4. White box testing — advantages:

• Since app developers need to be involved, they need to supply IDs for elements and sometimes even add coded attribute conditions. This makes test writing much easier and improves the quality of the app through test driven design & development.
• Tests can be part of the app development repository. This means every app developer can run tests as part of his/hers final round before sending the ticket to QA. A test can run in a simple right click, as simple as running unit tests.
• Because tests can be part of the app repository, it is easier to integrate them into a CI-CD (Continuous Integration/Delivery) flow.

What is the right thing for me?

You didn’t expect to get a clear and definite recommendation, did you?

Appium allows you to own and control everything, including the ability to have deep testing of backend along with testing the interface, but when you have difficulties to reach an element and need app development help, the app developers are basically doing you a favor, since it’s your toy and not theirs. This independence can also cause reduced code quality and efficiency if you’re developing alone. Tests could be too long/unreliable to be part of a non human interaction CI acceptance level gate.

On the other hand, native testing requires you to strongly collaborate with app development teams with super clear definition of who does what and a clear test modification and approval process. You get many more people involved and can get higher coverage easier, but there could be control and responsibility arguments.

Some companies prefer test automation to be done by app developers and have no dedicated automation team. It’s definitely doable, but it brings questions as of how good can the same execution organization criticize/check itself. Clear procedures are needed to address these issues and it doesn’t necessarily say that developers will QA the product better than people who specialize in QA-ing the product.

To conclude,

If you’re part of a small development team, that wishes to support multi-layer testing (with/without backend/mock data) and wants all developers to get involved in testing, if you have just one platform to support and you’d like a full package CI-CD — I’d go with native testing.

If you need to make API level verifications along with UI testing, you can’t easily get mock data and you need to support multi-platform OS — I’d go with Appium.

Big companies with a variety of needs and conditions can do both and get both benefits. Appium as a nightly heavy duty full end to end utility and native as a CI-CD simplified solution that involves main app developers and maintains test driven design and development.