Sep 1, 2021
Mobile test automation practices —
Part 2: preconditions, elements verification and independent steps
This is Part 2 of an article on the mobile test automation practices we use at Bumble Inc. In Part 1 we looked at the role of automation in our processes, gave details of the framework, and examined in detail three practices we employ when creating autotests.
Here in Part 2, we cover the following topics:
- Practice no. 4. Verification of element state transitions
- Practice no. 5. Guaranteed preconditions setup
- Practice no. 6. Simple and complex actions, or Independent steps in tests
- Practice no. 7. “Optional” elements verification (which is mandatory to do 🙂)
- Summary
- Bonus.
I would like to stress that the examples given in both Parts of this article are equally relevant for those just starting to roll out automated testing in their project, and those who are already actively using it. Also, it shouldn’t matter which framework you use. We generalise all our recommendations so that it’s easier for you to apply them.
Spoiler alert: We will share a test project with all practices located on Github at the end of this article.
Practice no. 4. Verification of element state transitions
This practice is probably one of the most important practices for mobile automation. This is because it is extremely rare to see static elements in applications that display on-screen immediately, and that are always accessible on-screen. More often we have a situation where it takes a certain amount of time to load elements.
For example, in the case of slow internet, elements received from a server are displayed with a significant delay. And try to verify them before they appear and the tests will fail.
Thus, before trying to verify elements we need to wait for them to appear on-screen. Obviously, this is not a new problem and standard solutions exist. For example, Selenium has different types of “wait” methods, and Calabash has the “wait_for” method.
Why we need our own bike/Why we didn’t go for a standard solution
When we began to build our automation framework we used a standard “wait_for” method to wait for elements to appear or change state. But then we ran into a problem. Sometimes, once every 2–3 weeks, our tests froze. We couldn’t understand how or why this was happening, or what we were doing wrong.
After we added logs and analysed them, it turned out that the cases of frozen tests were linked to the “wait_for” method, which is part of the Calabash framework. “wait_for” uses the timeout method from the Ruby Timeout module, which is implemented on the main thread. The tests froze when this timeout method was nested within other methods: both within ours and within the Calabash framework methods.
For example, let’s take the case of a profile page being scrolled down to the “block user” button.
We can see that the “wait_for” method is used here. The screen is scrolled down, then there is a wait for the animation to complete, and finally, there is verification that the “block user” button is displayed.
Let’s take a look at the implementation of the “wait_until_no_animation” method.
The “wait_until_no_animation” method is also implemented with “wait_for”. It waits for the animation on-screen to finish. This means that “wait_for”, called inside “wait_for”, calls other methods. Just imagine that there are also “wait_for” calls inside Calabash methods. As the chain of “wait_for” methods inside “wait_for” methods lengthens, the risk of the test freezing increases as well. For this reason, we opted not to use this method but to devise our own solution.
The requirements for our own solution matched those for standard ones. We needed a method that would repeat verification until the required condition is met, or until the set timeout expires. Should our verification be unsuccessful within the fixed time frame, our method would need to raise an exception.
Initially, we created the “Poll” module with one “for” method, which repeated the standard “wait_for” method. Over time our own implementation allowed us to extend functionality as much as we needed to. We added methods, waiting for specific values of set conditions. For example, “Poll.for_true” and “Poll.for_false” clearly wait for the code being run to return “true” or “false”. In the examples below I show how the various Poll module’s methods are used.
We have also added various parameters for methods. We shall take a look at the “return_on_timeout” parameter in more detail. In essence, when using this parameter, our “Poll.for” method will not raise an exception, even if the required condition is not met but instead returns the result of the executed code block.
I can hear you asking two questions: “How does it work?” and “Why do we need it?” Let’s start with the first question. If in the case of the “Poll.for” method, we were to wait for 2 to become greater than 3, then we would always get a timeout error.
But if we add our “return_on_timeout” parameter and, similarly, wait for 2 to become greater than 3 then, once the timeout has finished, 2 will still not become greater than 3, but our test will not fail. Instead, the “Poll.for” method will return a result of the executed code block.
Why do we need this? We are using the “return_on_timeout” parameter to verify the state transitions of our elements. But this must be done very carefully as it could conceal real instances of tests failing. Used incorrectly, the tests will continue to run when the required conditions are not met when they should have raised an exception.
Element state transitions (various options)
Now let’s move on to the most interesting bit. Let’s talk about how to verify various state transitions, and which state transitions they are. Allow me to introduce our test object — the black square:
It can only do one of two things: appear on your screen or disappear from your screen.
The first state transition option is called “Should appear”. This is where in state 1 our test object is not displayed, and in state 2 it appears.
If it appears, then verification is successful.
The second state transition option is called “Should disappear”. This occurs when in state 1 our test object is displayed, and in state 2 it disappears.
The third option is not as obvious as the previous two because in this case, essentially, we are verifying that the state is unchanged. This is called “Should not appear”. This occurs when in state 1 our test object is not displayed on-screen, and sometime later in state 2 it still does not appear.
You have probably guessed the fourth option. This is called “Should not disappear”. This is when in state 1 the object is displayed on-screen, and sometime later in state 2 it is still there.
State transitions verification implementation
So, now we have identified all possible element state transitions. How can we verify them? Let’s review the implementation of options one and two first, and then — options three and four.
In the case of the first two options, it’s all pretty simple. To verify the first one, we just need to wait for the element to appear, using our “Poll” method:
To verify the second one, we need to wait for the element to disappear:
However, in the third and fourth cases, things are not so straightforward.
Now to consider the option “Should not appear”:
Here, firstly we check that the element is not displayed on the screen.
Then, using “Poll.for” with the “return_on_timeout” parameter, we wait for the element to appear. Here, the “Poll.for” method will not raise an exception if the element in question does not appear but will return “false”. The value obtained from “Poll.for” is saved in the “actual_state” variable.
Next, the “assert” method is used to verify that there has been no change in the status of the element.
To verify the “Should not disappear” option we use similar logic — waiting for the element to disappear from the screen (rather than appear on it):
Verifications of these four state transition options are relevant for many mobile application elements. Since we have many engineers developing tests there is always a possibility that someone will forget to include some options when creating new verifications. For this reason, we have moved the implementation of verifications for all state transition options to a single method:
“yield” is the code for the block passed to the method in question. In the examples above this was the “elements_displayed?” method. But it could be any other method that returns the state of a required element. Ruby documentation.
Thus, we can verify any element state transition option by calling a single method, which makes life considerably easier for the whole testing team.
Conclusions:
- It is important to include all four state transition options when verifying UI elements
- It is helpful to move this verification to a generic method.
We recommend using a full system of possible state transition options for verifications in tests. What do we mean by ‘full’? Imagine that when an element is present the state is “true”, and when it is not the state is “false”.
We are building a matrix of all permutations. If a new state were to appear on the scene, the table could be extended, and new permutations obtained.
Practice no. 5. Guaranteed preconditions set up
Task definition
As you can probably guess from the header, the task in this section is to explore how to configure preconditions before starting a test execution.
Examples of solutions
Here are two examples. The first is switching off the location service in the iOS settings. The second is generating a chat history.
This is what the first example of implementing a method for switching off the location service in iOS looks like:
We are waiting for the switch element to appear on-screen. Once it does, then we verify its state. If it does not match up with the desired state, we change it.
After this, we close the settings and launch the application. And it can happen that all of a sudden we encounter a problem: for some reason the location service remains on. How can this be? After all, we have done everything we could to switch it off. It would seem that this is a problem with how the system settings in iOS work. When you leave the settings quickly (and, in the case of a test, this occurs instantly, once you click on the switch element), its new status is not saved. However, problems may also occur when configuring preconditions in our application.
Now to the second example, namely generating a chat history before starting to run a test. This is what implementation of the method looks like:
We use QaApi for sending messages via “user_id” and we send the required number of messages in the loop.
Next, we go into the chat and perform the necessary verifications related to loading the messages history. But sometimes not all the messages are displayed in the chat. Sometimes we continue to receive them while we are in the chat, but not before we have opened it. This is related to the fact that the server cannot send the necessary number of messages instantly. In some cases, there is a delay in delivering messages.
Let’s identify the common problem in both these examples. When the preconditions are set incorrectly, tests fail in subsequent verifications, irrespective of the fact that the applications themselves are operating as they should. When we start to examine the reasons for this we realise that the tests fail due to incorrect preconditions setup. Such cases are frustrating not only for testers but also for developers because the latter analyse tests results as they are developing additional functionality.
How can we resolve this problem? Add verifications to the methods for setting preconditions, confirming that a required action has been performed successfully before proceeding with test execution.
So, in this case, our method for switching off the location service will look like this:
Using the “Poll.for” method we ensure that the status of the switch element has changed before we move on to the following action as part of the test. This allows us to avoid the problems arising from the location service staying switched on from time to time.
In the second example, once again it is our QaApi methods that help us out.
Before sending messages, we receive the current number of messages in the chat, and afterwards, we ensure that the required number of messages has been sent. Only after this verification succeeds will the test continue to run. In this way, when we open the chat in the application, we see all the necessary messages and can perform the verifications we need.
Takeaways
We always recommend the use of guaranteed preconditions set up in tests by adding verifications that required actions have been performed successfully. This allows us to save the time usually needed to investigate test failures in those cases where setting preconditions has not worked since the test will fail immediately at the preparation stage.
In terms of a general recommendation, we would advise you to add to your tests verifications that all presets are set when asynchronous actions are performed.
You can read up in more detail about the problems described in this section in this article by Martin Fowler.
Practice no. 6. Simple and complex actions, or, independent steps in tests
Simple actions
Based on the conclusions stated in the previous section, it might seem that we should add verifications, confirming that all actions have been performed in tests before moving on to the next actions. But in fact, this is not the case. In this next section, we will talk about implementing steps for various actions and verifications. This is a very general formulation that is appropriate for any step in our tests. For this reason, as usual, we will consider specific examples.
Let’s start with a test to search for and send GIF messages.
Initially, we need to open a chat with the user to who we want to send a message:
Then we need to open the field for entering GIF messages:
Next, we need to enter a search query for GIF images, make sure that the list is updated, send an image from the list, and make sure that it has been sent.
Overall, this is what the scenario looks like:
Let’s focus on the step responsible for searching for a GIF:
Here, as in almost all other steps, this is what we do:
- First, we wait for the necessary page to open up (ChatPage)
- Then we save a list of all accessible GIF images
- Next, we enter a search keyword
- Then we wait for the list to update (after all, we said that it is useful to add verification that actions have been performed successfully before proceeding further).
Everything would appear to have been implemented correctly. After the search has been completed, we check that the list of images has been updated, and only then do we send one of them. But we face a problem if, for example, we want to write a test to verify that, after having entered an identical search query, the list of images will not update. In this case, we need to create a separate step for entering a search query for GIF images. This step will duplicate the existing one to a major extent.
A similar problem occurs in cases when a given action could lead to various results, and, in the step, we only handle one of them in the form of verification that this certain action has been performed. This leads to difficulties with reusing such steps and, therefore, to both a slow-down in test development and to duplication of code.
How can this be avoided? As you may have noticed, our search-for-GIF-images step actually included three actions:
- Saving the existing list
- Searching
- Verification that the list has been updated.
The solution to the re-use problem will be to divide this step into three simple, independent steps.
The first step saves the existing list of images:
The second step — searching for GIFs — allows you to type in a search keyword:
In the third step, we wait for the list to update:
At this point, this is what our initial scenario looks like:
In this way, we can use the first two steps even in a case where the list does not update in the test. This helps us save development time since we can reuse simple steps in different tests.
But, here too, there are a few minor issues. Steps cannot always be created to be simple and independent. These we call “complex”.
Complex actions
When we speak about complex actions we mean those actions which include transitions between different screens in an application, or the handling of changes in the state of any screen. Let’s consider such a situation, based on the example of voting in a mini-game.
A mini-game is a screen where the user is offered profiles of different people who have written to them. You can either reply to the message or skip those users. We shall call the act of skipping “Voting No”.
We need to write a test that will “vote No” N times, will close the game screen, and will then re-open it and verify the user has the correct progress in the game.
“Voting No” is a simple action but if we create a simple step for this action, then in order to vote N times we need to use this step the same number of times at the level of the scenario. Reading such a scenario is inconvenient so it makes sense to create a more complex step using the parameter “Number of votes” which would be able to vote the required number of times.
This step would appear to be quite easy to implement. The only thing you need to do is vote on the page the required number of times.
But this is where we encounter a problem: sometimes the test votes too quickly, i.e. it presses the button before the previous button press has been processed. In this case, the application will not send the new vote to the server. At the same time, our step will be completed successfully. At the next step, when we want to make sure that the user has the correct progress in the game, the test will fail because the previous step has not performed its task, and has cast an insufficient number of votes.
As in the case with preconditions set up, this is where we need to get to the bottom of test failures occurring at the step where the test and the application work correctly, but the previous step worked incorrectly. Of course, no one likes such situations so this is a problem we need to solve.
After each vote is cast, we add verification of a change in progress in the mini-game. Only after this will we attempt to vote again. In this way, all the votes will be processed and we will eliminate test failures related to incorrect progress in the game.
Takeaways
We are creating independent steps for simple actions. This allows us to reuse/create scenarios faster and easier than if we needed to rewrite steps that are similar to one another. For steps for complex actions, we add verifications as to whether the action has been performed, before proceeding to the next actions.
To generalise these recommendations, we would advise you to identify independent methods for simple actions in tests and to add presets verification to complex actions.
Practice no. 7. “Optional” elements verification
By “optional” elements we mean those elements which could either be displayed or not on one and the same screen depending on certain conditions. Here’s an example with dialogues for user’s actions confirmation, or ‘so called’ alerts.
You have probably seen similar dialogues in various mobile applications. In our two applications, there are over 70 of them. They appear in various places in response to various actions by users. What “optional” elements do they contain?
Task definition
Let’s analyse the screenshots above.
- Screenshot 1: header, description and two buttons
- Screenshot 2: header, description and one button
- Screenshot 3: description and two buttons.
Thus, the “optional” elements in these dialogues are the header and the second button. The locators for the elements on all the alerts are identical — irrespective of where they are shown in the application, or which user action they appear after. For this reason, we want to implement verifications for all types of alerts once, on their base page, and then inherit a class for each specific alert from it so that we don’t need to duplicate the code in question. Creating these kinds of verifications is something we will look at in this section.
Examples of solutions
Let’s begin with how we call a method for verifying each of the dialogues:
In all these examples we call the “verify_alert” method, using lexemes for verifying the necessary elements. At the same time, as you can see, we don’t send a lexeme for the second button inside “WaitForReplyAlert” class since it should not be there, nor do we send a lexeme for the header inside “SpecialOffersAlert” class.
Here is the implementation of the verify_alert method:
First, we wait for the mandatory elements to appear on-screen. Next, we check that their texts match with the lexemes sent to the method. In the case of “optional” elements, if the lexeme was sent to the method, then we verify the text. Otherwise, we do nothing.
What is the problem with this approach? The problem is that we miss out on verification of the “optional” element not displaying when it should not be displayed. This can lead to confusion. For example, the following alert may appear:
The user doesn’t know which button to tap because both buttons appear to close the dialogue. This looks like a critical bug. Even if the application does not actually crash, this needs to be fixed. Tests need to be changed so that they pinpoint such problems.
To this end, we change this verification in the tests. From this…
to this.
We have changed the if-condition and have added verification of the second state. If we do not send a lexeme for the “optional” element, it means that this element should not be on-screen, which is what we are verifying. If there is some text in the “title”, then we understand that the element with this text should be there, and we verify it. We decided to move this logic to the shared method, which has been called “wait_for_optional_element_text”. It is used not only for alerts but also for any other screens in the application that contain “optional” elements. We can see that the if-condition from the example above is to be found within the new method:
The implementation of the “verify_alert” method has also changed:
We are still waiting for the mandatory elements to appear on-screen. After this, we verify whether their text matches the lexemes sent to the method. In the case with “optional” elements, we now use the “wait_for_optional_element_text” method, which helps us to make sure that these elements are not displayed when they should not be displayed.
Takeaways
We’d like to draw your attention to the fact that there is no such thing as “optional” verification but rather, there are “optional” elements — and all their states must be verified. Moving verification of the “optional” elements states to the shared method allows us to easily reuse it for various screens in the application.
Generic recommendation: we advise you to use a full system of possible elements’ states for verifications in tests and to extract shared methods for similar actions.
You will have noticed that we made similar recommendations earlier in this article. The point is that the examples in which they were applied differ.
Summary
To summarise, here are our key recommendations on mobile test automation from seven practices we have described:
- Since verifications are the reason why we write tests, always use a full system of possible states
- Do not forget to add presets verification for asynchronous actions
- Extract shared methods for reusing similar code — both in the steps and in the methods on the pages
- Implement simple test objects
- Implement independent methods for simple actions in tests.
This advice might seem obvious to some, but we want to stress the fact that it can be (and should be) applied in various situations. If you have additional helpful recommendations to add to the list, please feel free to do so in the comments!
A little bonus
We have prepared a test project where all the practices described in this article are represented. We encourage you to follow the link, study and apply them: Mobile Automation Sample Project.
