Recommendations for choosing a test automation framework
Focus on the platform
If you have the option of choosing a framework for testing mobile apps, I would recommend considering two basic things. The first, is the selection of mobile platforms that you want to test. The term mobile platform is only relevant for Android or iOS. The second, is the choice of the environment where the tests will run, like for example Windows, Mac, Linux, etc. This recommendation can be best explained in the following example.
On a smaller Android project, the company decides to automate part of the tests as if it is only Android, so it will choose a framework that is specialized in this — like, for example Espresso, or Robotium. Everything goes well until the customer decides to make an iOS version of the application that the tests also have to cover. These tests, unfortunately, can’t be written through Espresso because it’s been made purely for the Android platform. I suggest to solve this issue by selecting a framework for iOS, like EarlGrey, or KIF, where the remaining tests you would write separately. The second option, is that from the outset, the choice of framework is deliberately one that supports both Android and iOS, such as Calabash or Appium, with which tests will be rewritten.
The first solution would be recommended for when we’re confident that the project is intended for a shorter period of only 3 to 6 months. For example, if the whole project lasts for 5 months and the end is only 2 months then it would be easier to only write tests for the iOS platform. This sort of solution would allow us to maintain Android tests, and thus cover the Android platform. On the other hand, if we decide to go for a framework that supports both platforms, we would give up the Android test and start from scratch, and the testing coverage would be that much smaller.
On the other hand, if you know the project will be going for a longer period of time, for example, a year, I would recommend choosing a framework for both platforms at the cost of losing the Android platform. This approach has the advantage of having only one framework that can handle both Android and iOS. Additionally, the testing maintenance is much easier because all of the tests are written in one framework, and it takes away the need to control two different frameworks. I would like to mention another advantage, and this is the unified output format of tests that would be very difficult to achieve in two different frameworks.
Now, I’m moving to select the test run environment. Since tests need to run somewhere, it’s necessary to determine which operating system the framework will support. The most commonly supported environments are Windows, Linux or macOS. I will sum up this issue in more detail in the last part of this series.
With Android frameworks, the situation is much easier, because run tests do not require any special connection between the test application and the environment in which the tests will run. For example, a framework like Espresso is implemented directly as part of the application’s source code, and test run is executed via Android studio or a terminal that can be used in any of the above environments.
In the case of iOS devices automation, the situation is a little more complicated, as they are in most cases bound to macOS. The main reason for this, is that in order for tests to run on physical iOS devices, they need Xcode and can only be used on macOS. For example, the Calabash framework that supports both platforms, can be used to test Android apps without any problems on Windows, Linux or MacOS, but if I wanted to use it to test iOS applications, I have to use only macOS.
Next time, I’ll show you the next selection criteria, like the skills needed for automation, readability and output level, and integration with CI.
