Gotta Test ’Em All: How to Test OutSystems Mobile Apps
A while ago I wrote an article where I discussed the pivotal role of mobile app testing in driving app quality and the impact on user adoption and user satisfaction. I went over the complexity of mobile app testing due to the enormous range of different mobile devices, operating system versions, and network conditions. Today I want to walk you through testing mobile apps with OutSystems and AWS Device Farm.
The number of variables when testing OutSystems mobile apps, as with any other mobile technology, is huge, even at a surface level. That said, you should always consider looking deeper. After all, a malfunctioning app is just asking to be uninstalled on the spot.
There’s only so much you can test without reaching for the dreaded thing that can destroy a perfectly built scenario: a real device. Testing on real devices is a Pandora’s box logistically and is something that haunts the dreams of any mobile developer. My team and I have lost a fair amount of sleep over it. We had to find a solution, a way to test without being buried under a mountain of mobile phones.
There are many solutions out there such as Visual Studio App Center, Perfecto, or Saucelabs. But Amazon Device Farm has turned out to be the antidote for our nightmares. An AWS testing framework that lets developers upload and run tests on real Android and iOS devices in the AWS cloud, Device Farm allows you to perform automated tests and to have remote access sessions on specific devices with their own configurations, which means that before connecting to the device, we can configure its state.
AWS also provides an SDK that can be used to interact with all AWS services. This way, we can connect the Device Farm (or any other service) with our internal dashboards.
Also, AWS has launched Direct Device Access for Private Devices. With this new feature, developers can use individual devices in their private test set as if they were directly connected to their local machine via USB.
Device Farm also supports a wide range of testing automation frameworks like Appium, Calabash, XCTest, and many others, where you can write your own tests.
So yeah, it’s a pretty impressive tool, especially when you see it working.
Getting Your Hands Dirty: Amazon Device Farm and OutSystems
So, now I’m going to walk you through using AWS Device Farm to test OutSystems apps. To see it in action, we need to create the tests first, of course! We’ll be using a simple OutSystems application, and we’re going to test the login page on an Android device. For the technical details about how to set up your tests, have a look at these test samples on GitHub; you can also follow other testing tutorials.
1. Machine Setup
Install the automation test framework that best suits you. For this article, we will stick with Appium. Like Appium, some frameworks support more than one programming language. So make sure you install everything. We chose Python as our programming language.
2. Test Setup
Start by creating your testing project. Before you send all your tests to the Device Farm, I highly recommend running the exact tests in your local test environment first. It’s easier to find the root cause of an issue locally. It’s also less expensive. So in your main test file, add the following desired capabilities to your test.
desired_caps['platformName'] = 'Android'
desired_caps['deviceName'] = 'aPhone'
desired_caps['appPackage'] = '<app_identifier>'
desired_caps['appActivity'] = ".MainActivity"3. Test Planning and Phasing
Typically, you don’t create a test for everything. Ideally, you’ll isolate each part of the app that you want to test. So, before you start coding your test, you must come up with a plan. Sit down, relax, and try out your application, searching for the main functions you want to test.
4. Test Creation
Now that you have a plan, you are ready to start setting up your tests. Let’s begin by creating a test file in the tests folder and coding a test case. When coding your test, prefix your test method with the word “test”; this helps the test framework identify what method our test contains.
We are implementing interaction tests, so everything is sequential. First, we’ll start the test. Next, we’ll wait for an event/item to display on the screen. When the one we expected is displayed, we’ll click it, and then we’ll wait again for the next one to appear on the screen. You get the idea: start the test, wait, click, wait. Sometimes, we may need to use a sleep condition just to make sure that a specific event happens or a specific item appears on the screen; otherwise, we may not notice it.
import os
import unittest
from appium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ECclass TestClass(unittest.TestCase):
def setUp(self):
self.driver =webdriver.Remote('http://127.0.0.1:4723/wd/hub', {}) def test_case(self):
... def tearDown(self):
self.driver.quit()
if __name__ == '__main__':
unittest.main()
How do I know that something is on the screen? How do I click it? Okay, this is the tricky part. Remember the article that I wrote about building native apps with OutSystems MABS a while back? If yes, you already know that OutSystems applications are hybrid. This means that some changes we make when creating our OutSystems applications are mapped to HTML. So if you always set a data attribute with a label, it helps you identify your application elements in the test case, and it’s easier to find the element with XPATH.
In the first scenario, as seen in the following examples, we’re trying to find an image. We added an attribute with a value that represents the image (in this case, it’s “SuccessImg”), and we searched for it with XPATH (//img[@data-test-id=”SuccessImg”]). When dealing with a list, we need to be extra careful. To find a specific element on a list, for example, the third one, we have to make sure that we have the index of the value. Here, we need to look for the attribute “data-test-id” with the value “MyAttrId-2”.
I know, I know; there are some specific scenarios where we can’t test a specific function of our OutSystems mobile application in a Chrome web browser. Most of these cases happen because there is a direct dependency to some native plugin, which must be installed in the application. For that specific scenario, we have to connect our mobile device to our computer, open Chrome, and type chrome://inspect/#devices in the URL. This will open a page that shows all the devices that are connected to your computer.
Now, inspect your device, and start digging into your HTML. Search for the buttons, anchors, or links that you will need to navigate your app. A good way of identifying your app buttons is to use the HTML Id field, but, if for some reason that specific button doesn’t have an Id, you can use XPATH instead.
Don’t forget: iOS devices can only be inspected using Safari on Macs and by enabling the web inspector on the device. Android can be inspected by both PCs and Macs using Chrome and enabling the developer tools on the device.
5. Test Bundling
We’ve created our test, and now we’re ready to submit it to the Amazon Device Farm. How can we do this? It’s straightforward: running a command we can create a zip file containing our test bundle. This test bundle is important because it contains the test and libraries that will be executed by the AWS Device Farm. To submit the tests:
1. In the AWS Console, create the project where you will perform your tests and a new run. A run represents a specific app with a specific set of tests on a specific set of devices. The groundwork is done.
2. After this, you should upload your app package and your tests. If you don’t have any, AWS has you covered with two built-in tests. In this example, we’ll be using our own.
3. Now the fun begins: select the devices that you want to test and specify the device state (WiFi, NFC, GPS, Bluetooth). Currently, AWS Device Farm has 178 Android and 162 iOS devices. For Android, there are 139 distinct devices (Motorola, Samsung, Wiko, and so on) operating with 23 different Android versions. For iOS, there are 26 distinct devices (iPad 2, iPhone 8, iPod Touch 6th Gen, and so on) operating with 26 different iOS versions.
4. Go time! Review, run, and view the results! Each run produces a report with the device logs, test logs, screenshots, videos, and more.
Wrap Up
Device Farm is so helpful. We can now continuously deliver new features, improvements, and bug fixes with a high confidence level. Our developers now develop new features and test them right away.
This tool also helped us with our support cases. As you might know, it’s unfeasible to have all the different devices and different OS versions. With this tool, we don’t have to lose a night’s sleep over this; every year, AWS Device Farm adds new devices to their service. So, each time we receive a support case for something that’s not working as expected on a Lava Iris, Ulefone, or Mlais device, we can request remote access to Device Farm, and we can access and test the app on the device in real time.
I challenge you to give it a try, and now it’s up to you! This isn’t as straightforward as low-code, but it’s also not as complicated as it may seem. The return on your effort will be worth it. And, don’t forget we used the AWS Device Farm with Appium, but you can use other device farms. Also, as I mentioned previously, you have everything explained here, here, and here. Let us know how this solution worked for you!
