Framework selection
In the last part of this series, I’m looking at how to choose a specific framework. Since there are a lot of frameworks available,I have tried to narrow the selection to the following:
This selection includes 2 Android, 2 iOS, and 5 frameworks that support both platforms. For a better overview, I have put them in a table.
Android-only framework selection
Espresso and Robotium are similar to frameworks that are part of the application’s source code, except that Espresso is a direct part of the source code from the start of the project, whilst Robotium needs to be actually added to the project. Both frameworks are used to write tests in Java so they require at least basic programming experience. Since Google develops the Android OS with long-term support and functionality, it can be expected that they will offer durable and long-standing support. In terms of outputs and integration with CI, these frameworks are very similar. If I had to choose from these two a favorite, it would be Espresso, largely because it can be expected to have long-term support.
Choosing an iOS-only framework
For EarlGrey and XCTest, the situation is similar to Android, and these two frameworks are part of the application’s source code. Both use Swift / Objective-C tests, and some programming skills are needed. The first difference is that XCTest is a native part of the source code from the start of the project, whilst EarlGrey must be added to the project. EarlGrey is developed by Google, which uses it primarily to test iOS versions of applications such as Youtube or Gmail. XCTest is an Apple product. Since both Apple and Google are two big companies, we can assume the long-term support of these frameworks in the future. If I had to choose one of these two, I would go to XCTest because it is a native Apple framework.
Choose the framework for both platforms
Of the five frameworks I originally considered, I had to remove Ranorex and TestComplete because they do not support Mac OSX, which is one of the key requirements in our company. Another framework that I dropped out of this list was Experitest due to a price model that did not suit us. Finally, there were two frameworks, namely Appium and Calabash, whose advantages and disadvantages are now to be summarized.
Appium pros
- Supports multiple languages Java, C #, Ruby, Python, JavaScript. Therefore, the user is not bound to one language but has a choice.
- It is developed as one joint project for both Android and iOS.
- It includes a test recording tool that is suitable for speeding up work.
- It contains native support for selenium to automate the web browsers.
Appium cons
- It supports BDD but does not create predefined steps ready for use versus Calabash.
- In the case of the use of uncommon languages, the problem is to find the tutorial because even though Appium has a large community, the vast majority is focusing on Java and Python.
Calabash pros
- The element search console brings a lot of features. Thanks to this, we are able to execute and verify a series of commands without the need to run the test.
- It allows you to search for web elements in your app using css selectors, making it possible to search everything in webview.
- It already has predefined BDD (Behavior-driven development) commands thanks to Cucumber, which sometimes differ for Android and iOS, but we can write tests without using ruby code.
Calabash cons
- Supports only Ruby language.
- Developed in two separate Calabash Android, Calabash iOS projects.
- It does not contain a test recording tool that can facilitate test development work.
- For web tests, you need to install selenium gem and modify the structure of the project.
- Prior to the recent Xamarin, he announced the ending of the Calabash support that moves to open source so it will matter how the community faces.
What works for us?
This framework selection took place in our company well before the official Calabash support and its open source transition was completed. So, at that moment, after considering all of these points, we decided to choose Calabash, mainly because of the interactive console and the element search system. Another reason was the native support of Cucumber, which attracted us mainly because such written tests are understandable to the extent that they can be entered in a system such as JIRA. But if I had to make the choice today, given the uncertainty surrounding the passage of Calabash to open source, I would choose Appium.
