Focus on skills, outputs and CI
The skill needed for automation
The skill needed for automation is different from framework to framework. For example, frameworks like Ranorex or Espresso contain tools that can be used to record tests. The test recording tools work, in principle, to record the interaction with the application, and then to convert it into the code.
Here is a test generated with Espresso:
@Test
public void mainActivityTest() {
ViewInteraction appCompatButton = onView(
allOf(withId(R.id.setCoordButton), withText("Button"),
childAtPosition(
childAtPosition(
withClassName(is("android.widget.LinearLayout")),
0),
3),
isDisplayed()));
appCompatButton.perform(click());
ViewInteraction linearLayout = onView(
allOf(childAtPosition(
IsInstanceOf.<View>instanceOf(android.widget.FrameLayout.class),
0),
isDisplayed()));
linearLayout.check(matches(isDisplayed()));
ViewInteraction appCompatEditText = onView(
allOf(withId(R.id.latText),
childAtPosition(
childAtPosition(
withClassName(is("android.widget.LinearLayout")),
0),
0),
isDisplayed()));
appCompatEditText.perform(click());
ViewInteraction button = onView(
allOf(withId(android.R.id.button2),
childAtPosition(
allOf(IsInstanceOf.<View>instanceOf(android.widget.LinearLayout.class),
childAtPosition(
IsInstanceOf.<View>instanceOf(android.widget.LinearLayout.class),
1)),
0),
isDisplayed()));
button.check(matches(isDisplayed()));
}For example, elements can be selected from the generated code to be used to create tests, or create a test structure that can be further modified to save time. If the framework does not offer such a tool, it doesn’t mean that it’s worse — because even though it may be possible to speed up the test process, it’ll always be necessary to manually modify and maintain these tests.
Since testing can’t be done by a person who doesn’t have at least a basic knowledge of programming, it’s necessary to consider whether the company wants to automate only briefly and superficially, and thus hires a junior tester, or takes it more seriously, and will give it to the programmer or junior programmer who will be more expensive than a tester, but be able to suggest the architecture greater detail, and will think of things that a tester without programming knowledge wouldn’t.
The knowledge of programming, and the programming language in which the tests will be created, is another thing to consider when making your choice. Most market frameworks offer the option of using only one, e.g. Espresso uses Java, but there are also those that offer a selection of languages such as Appium, which allows you to write tests in Java, Ruby, Python, etc. However, I should point out that if a framework supports many languages, most guides will be written for the most common ones like Java and Python, and the less-used languages will be much less documented.
I should still mention one small but substantial factor in the documentation, and that is the community. It is advisable to verify which community is behind the individual frameworks and, possibly, to find out how long the individual frameworks have existed on the market.
Readability and level of test output
When choosing a framework, I would recommend that you study the outcome of the test, because that will form the basis of how quickly it can be evaluated. Some frameworks provide only the basic text output level, for example, to the terminal, while some can generate output to json or html and move the visual page up one level. For this purpose, Gherkin is perfectly suited for the integration with a variety of frameworks such as Calabash, Appium, Espresso, etc. Gherkin serves to write tests in the form of short, meaningful sentences. That is why I would consider choosing whether or not to integrate this language. With Gherkin there are not only more readable test outputs, but these outputs can then be easily copied into the bug tracking tool, as this helps to save our time with creating bug report.
Integration with CI
The last point to consider when choosing a framework, is how the tests will run. Tests can be run manually, or using a planning tool, like for example, Cron or LaunchControl, or through Continuous Integration, Jenkins, Gitlab CI . Each way has its use:
- I run the tests manually most often when I develop or modify the tests, or if some tests failed and I need to see what happened in the test.
- I used to use the scheduler abundantly when the automation started in INLOOPX, and it was necessary for the tests to run regularly on certain days and hours.
- Later on in the company, we moved to the CI which has the advantage over a regular planner, that the test can run on certain versions of applications. For example, a smoke test is triggered before the application is sent for testing, or a whole set of tests will be triggered when the application has bigger changes.
Because the CI tools provide the greatest amount of variability and setup, in selecting test frameworks, I recommend finding out which CI tools they support. In addition, I also recommend that the framework support parallelization. Under parallelization in this case, I think it’s possible to run tests on multiple devices at the same time, or at least to think about this when designing the tests, so that the framework supports parallelization, but the test architecture doesn’t allow it. This issue will be dealt with in more detail in another article.
Since the main selection criteria I have more or less exhausted, it’s time to look at the actual selection of the tool, which I’ll take you through in the last article of this series.
