Appium vs Native Frameworks: A Comparison

By Kouros Aliabadi and Rohan Janjua

This post is a summary of our experiences and not necessarily the whole picture. We will provide some links and build upon this in future posts to help you in your own research.We hope that this will be a good starting point for anyone wishing to make the choice between some of the leading mobile automation approaches.

There are alternative automation tools out there but for the context of this blog post we will restrict ourselves to this Appium and the default native tools for iOS and Android: XCUITests and Espresso respectively.

Appium:

Appium has been the defacto tool for mobile automation over the last few years, providing a selenium like experience. Up to now it has been one of the most professionally viable choices due to a number of reasons.

1. Firstly it is language agnostic, coming with support for a wide range of popular languages. If your language of choice has a webdriver client, you are able to use Appium. This is down to the shared jsonWireProtocol. This is of great convenience as it allows test developers to pick up the tool quickly. Languages supported include, but are not limited to Java, C#, JavaScript, Python, and ruby.
2. It is very simple to pick up and get started with easily. Similarly to the previous point, Appium has a lot of similarities to Selenium webdriver. The benefit of this is that if the test developers are used to writing selenium webdriver web tests then Appium should be relatively simple to pick up and quite intuitive.
3. Appium enables testing of multiple platforms from the the same test code base. For those working in a centralised test team or a team that works across both iOS and Android, it can reduce the amount of boilerplate code required to work with the test infrastructure and emulators.
4. Additionally in the experience of some of our test engineers, the support for native apps utilising webviews is better on Appium than most other competitors. The support provided by Appium can be invaluable when it comes to automating hybrid apps.

There are also some disadvantages to using Appium:

1. In our experience, Appium tests run a lot slower than tests written in either XCUITests or Espresso
2. The test code doesn’t live with the dev code. Now this may be up for debate, but we firmly feel that test code and dev code should live close to one another.
3. For testing cross platform apps there will always be some degree of technical complexity involved with Appium. You either have to:
4. Maintain 2 sets of PageObjects/ScreenObjects that adhere to a single contract so they can be called from only one set of tests.
5. Write 2 sets of tests
6. Ensure developers use the same id’s across both apps

Native tools:

The two main platforms for app development now have very competitive UI automation tools of their own: XCUITest and Espresso. The maturity of these two tools has improved dramatically and in a recent talk at WWDC 2017Apple have made it clear that they are very active in working on improving their UI automation tools.

1. There is a fundamental advantage to using XCUITest and Espresso: they are created by the platform providers: Apple and Google. These tools will always be ahead of the curve for iOS and Android testing. Any new feature from Appium would in most cases be built on top of an existing native tool’s functionality.
2. Another major benefit in our eyes is that you will include the test code with the source code of your project. This is perhaps up for debate slightly but we will discuss this in a future post. For now we will just state that we believe that including your test code within the same project and in the same language that your developers use has great benefits. It gives more incentive for collaboration within the team and allows everyone to easily contribute to this aspect of software development.
3. Sometimes the android experience is supposed to be different to the iOS experience, by writing your tests for each platform you don’t have to complicate your test framework to handle these situations.
4. Much less flaky. Native tools just are less flaky. It may have something to do with a smaller number of moving parts and less layers of communication between the test code and the instrumentation, but tests written with native tools do seem to be much less flaky.
5. You can use a much larger toolset than if you use a tool like Appium. For example with Android you have access to both the UiAutomator library and the Espresso library.

Espresso:

Google’s Android testing tool Espresso has some neat features of its own that are worth mentioning.

1. Espresso is very fast, we’ve never seen something so fast in UI automation. It’s like the Flash of UI automation, no other tool comes close to it’s speed.
2. You don’t have to worry about waiting for things to happen or finish in your test code as Espresso handles this for you, excluding some exceptional cases. This basically takes the most brittle thing about UI automation out of the equation.
3. Espresso takes a ‘grey box’ testing approach. Not quite black box, not quite white. With espresso the tester has a lot more control over the application than in a black box tool like appium.
4. Espresso allows the tester to make use of tools like Robolectric, a framework for running the Android SDK without starting up a simulator or plugging in a device. What this means is that your tests start up faster, and run faster too.

There is a framework similar to Espresso, also created by Google, for iOS testing which is worth a mention here: EarlGrey. We haven’t used it, but if it brings some of the benefits of Espresso to iOS it is well worth exploring.

There are also some disadvantages to using Native test tools:

1. If you are developing a cross platform app, you will potentially have to maintain twice the number of tests compared to if you used a tool like Appium.
2. If you are developing a cross platform app, you will need to know two languages.
3. More complexity can be involved in this approach. This is a side effect of the extra power and access that a tool like Espresso can give the tester. For example Espresso automatically waits for events to complete in the application under test using an IdlingResource. However for some cases the tester will have to implement the IdlingResource manually.
4. With a tool like Espresso, the tester can put the application into a state that it may not be able to reach from a user’s perspective. Again, this is the other side of the extra power you get.