Android Application Testing with Firebase Robo Test
Testing an app to ensure its quality is very important for app developers. Nowadays, it is very common for companies to have a dedicated team to manually test their apps before it is released. Even if the engineers wrote proper unit and instrumentation tests for the app, some bugs or edge cases may still surface when users are using the app.
Firebase Robo Test
Firebase Robo Test is a testing tool that performs testing by analyzing the UI structure of the app and then simulating real user interactions. It differs from a normal Android unit and instrumentation test in that it does not require any mocks; it tests the app as a whole. Due to this, it offers several benefits over traditional testing as it is able to:
- Test against real backend API and wait for responses before continuing.
- Test obfuscated code.
- Test the flow of the app from one screen to the other.
- Make use of the Firebase Test Matrix to simulate user interactions on multiple device configurations at once.
Setting Up and Running Robo Test
Before we setup our Robo test, we need to have a Firebase project. If you do not have one yet, you can follow this link to create a project.
There are 2 ways to run Firebase Robo Test: from the Firebase project console’s Test Lab, or gcloud command line. Since we are talking about automation, I will focus on the later.
The gcloud command line tool is useful for test automation because we can execute gcloud commands from our CI/CD infrastructure. But, before we use the gcloud command, we must first setup the gcloud command line tool. You can find the setup guide here https://cloud.google.com/sdk/gcloud/.
Now, we can use the following command to run Firebase Robo test:
$ gcloud beta firebase test android run \
--type robo \
--timeout 3m \
--device model=walleye,version=28,locale=en_US, orientation=portrait \
--app /path/to/apk_file.apkOr we can also use a yaml file to setup our Robo test:
$ gcloud beta firebase test android run \
/path/to/robo.yml:args robo.yml file will look like this:
args:
type: robo
timeout: 3m
device:
- model: walleye
version: 28
locale: en_US
orientation: portrait
- model: walleye
version: 27
locale: ja_JP
orientation: portrait
app: app/build/outputs/apk/debug/app-debug.apkRunning the above command will give us the following result:
Uploading [app/build/outputs/apk/debug/app-debug.apk] to Firebase Test Lab...
Raw results will be stored in your GCS bucket at [https://console.developers.google.com/[REDACTED]]Test [matrix-dpg1wuq7kikza] has been created in the Google Cloud.
Firebase Test Lab will execute your robo test on 2 device(s).
Creating individual test executions...done.Test results will be streamed to [https://console.firebase.google.com/project/[REDACTED]].
13:34:34 Test matrix status: Finished:2
Robo testing complete.More details are available at [https://console.firebase.google.com/project/[REDACTED]].
┌─────────┬───────────────────────────┬──────────────┐
│ OUTCOME │ TEST_AXIS_VALUE │ TEST_DETAILS │
├─────────┼───────────────────────────┼──────────────┤
│ Passed │ walleye-27-ja_JP-portrait │ -- │
│ Passed │ walleye-28-en_US-portrait │ -- │
└─────────┴───────────────────────────┴──────────────┘
Robo test can also be run with android app bundle. To do this, supply --app flag with your .aab file.
Guiding Robo Test
Robo Script
Robo script is a json file that we can use to outline the steps that the Robo test has to perform when being run. It contains an array of actions. Here is a simplified robo script for our sample app:
[
{
"eventType": "VIEW_CLICKED",
...
"elementDescriptors": [
{
"className": "androidx.appcompat.widget.AppCompatEditText",
...
"resourceId": "com.malvinstn.myapplication:id/username",
"contentDescription": "",
"text": ""
},
...
]
},
{
"eventType": "VIEW_TEXT_CHANGED",
"timestamp": 1561810701547,
"replacementText": "myusername",
...
"elementDescriptors": [
{
"className": "androidx.appcompat.widget.AppCompatEditText",
...
"resourceId": "com.malvinstn.myapplication:id/username",
"contentDescription": "",
"text": ""
},
...
]
},
{
"eventType": "VIEW_TEXT_CHANGED",
"timestamp": 1561810707923,
"replacementText": "password",
...
"elementDescriptors": [
{
"className": "androidx.appcompat.widget.AppCompatEditText",
...
"resourceId": "com.malvinstn.myapplication:id/password",
"contentDescription": "",
"text": ""
},
...
]
},
{
"eventType": "VIEW_CLICKED",
"timestamp": 1561810729044,
"replacementText": "Sign in or register",
...
"elementDescriptors": [
{
"className": "com.google.android.material.button.MaterialButton",
...
"resourceId": "com.malvinstn.myapplication:id/login",
"contentDescription": "",
"text": "Sign in or register"
},
...
]
},
...
]You can take a look at the full robo script here.
We can use Android Studio to easily create robo script. It is located in Tools | Firebase | Test Lab | Record Robo Script and Use it to Guide your Robo Test. This will launch the application and then record the UI interaction with the app.
Starting a recording from an emulator can be really slow. I suggest using a physical device if you would like to record a robo script.
After all the steps in Robo Script have been performed, Robo test will then continue to simulate user interactions just like a normal Robo test until timeout is reached or there are no more actions to perform.
Another benefit of creating a robo script, is that we can reuse it to identify issues in our app through Google Play Console’s pre-launch reports.
To specify the robo script file in gcloud command, use --robo-script argument:
$ gcloud beta firebase test android run \
...
--robo-script /path/to/roboscript.jsonor if we are using a yaml file:
args:
...
robo-script: /path/to/roboscript.jsonRobo Directives
Robo directives is an alternative way to guide Robo test. It is a list of “commands” that Robo test will adhere to. At the moment, there are 3 types of directives that we can utilise: text, click, and ignore. Out of these 3, only text can have a value.
To use robo directives in gcloud command, use --robo-directives argument:
$ gcloud beta firebase test android run \
...
--robo-directives text:textview_resource_id=value,click:view_resource_name=Or if we are using a yaml file:
args:
robo-directives:
"ignore:facebook_login_button" : ""
"ignore:menu_share" : ""You can mix robo directives with robo script. When robo script has been run, Robo test will continue to simulate user interactions with robo directives as a guide.
Robo Test Results
Suppose we have a simple app that allows user to login, add fruits into the shopping cart, and then checkout. We are going to let Robo test to crawl this app for us.
After Robo test is successfully run, we will be able to view the test report on the Firebase Test Lab console. Inside the report, we can click into each tab to get more details about each of the specific stats. On the top right corner, we can click on “Test results” to see the GCS bucket that stores the actions.json log file, logcat, videos, and screenshots of the test.
In the Robo tab, we will find the stats of the crawl and the result of the robo script. It also generates a nice graph that shows us the path that Robo test took. Since the graphs can be really big, I will put the sample graph in an external link here.
Robo test also provides screenshots and videos of the test so that we can find and determine the root cause of our problem. These are also useful to ensure that our app looks consistent across different devices.
When we test with physical device, we will also get performance stats. These stats contain the standard android performance metrics such as time to initial display and graphics stats.
Performance reports also provides the device performance data over time such as CPU, memory, and network activities. We can scroll to each action performed by Robo test and see the device state at that particular time.
When Should We Use Robo Test?
Even though Robo test might sound promising, there are a couple of considerations that need to be had before using it. Robo test is meant to be used to find performance and user experience issues, not to test for correctness.
One of the reasons for this is that when there is a failure in executing one of the steps of a robo script, Robo test does not fail the entire test. Instead, it will continue with regular Robo crawl and interact with the app until timeout is reached or there are no more actions to perform.
If we want to check the execution status of robo script, we will need to look into actions.json file located in GCS bucket and find the correct line.
...
, {
"startTimeSeconds": 16.865,
"endTimeSeconds": 16.865,
"roboscriptFinished": {
"reason": "ELEMENT_NOT_FOUND"
}
}
...Another reason is flakiness. Robo test can be very flaky because they do not rely on any mocking mechanisms. If we are building an app that requests data over the internet, then Robo test will use the API as is. As we know, network connection can be very unstable.
Since Robo test runs on Firebase Test Lab, it may limit some of the use cases of our app and we will have to account for those cases. For example, we will not be able to use third party login services other than Google login. We would have to add ignore statements to prevent Robo test from interacting with third party login buttons. In addition, we will also need to prevent Robo test from going out of our app, such as opening share sheet, or accessing the camera and gallery.
In summary, Robo test is meant to be a layer on top of our existing unit and instrumentation tests. It is a very good tool to simulate user interactions and to automatically validate our app against different multiple devices at once, but it has limitations as well. If we are talking about testing pyramid model, then Robo test would be the apex of that pyramid.
