Running Autotests on Real Devices with Amazon Device Farm and Browserstack

The article is also available in Russian at the link.

Currently, test automation is pretty much popular, and this is not a surprise. A large number of regressions on various projects is a resource problem, and automation helps in the best way possible. However, if automated testing of websites on various browsers with Selenium is not too expensive, mobile apps test automation requires certain investments.

The very principles of testing say that you need to check the product as close to the conditions of its actual use as possible. In the case of mobile apps, QA engineers need to test on real devices. And here we face two problems.

The first of them is the absence of the device farm of your own. Many applications need to be tested on a fairly large set of devices with different configurations. That means these devices must be available in reality, since emulators and simulators do not always provide the necessary approximation to the actual conditions of use. And if it’s not a problem to check everything on your own smartphone, as well as run an automated test on it, then when you need five, ten or more devices to check, testing becomes very expensive, because each device needs to be purchased, and time needs to be spent on installing builds.

The second problem is the difficulty of running automated tests even with a large number of devices available. To run all the regression tests, you need to connect various devices and monitor the process of passing: this, of course, makes life easier, but the verification process is still stretched.

So what can we be done? Is there a solution to these problems, and how automated testing can be performed on a large number of real devices with minimal time spent? The answer to this question and a guide to action I offer in this article.

In today’s world, it doesn’t matter if you don’t have a large number of real devices to test. Many companies offer their own stack of devices, and today we will look at two of them: Amazon Device Farm and BrowserStack. Both have been known for a long time for providing access to test devices, but few people know that they also allow you to conduct automated tests on several real devices at the same time. Let’s check how to do this below.

Amazon Device Farm

To get started with Amazon Device Farm, follow the link https://aws.amazon.com/device-farm/ and click Create an AWS Account. I don’t see any point in fully describing registration on the service, as everything is described in sufficient detail. If you already have an account, you can skip this step.

Once signed up, you will be taken to the Device Farm page (the page is also available from the AWS console via the menu item of the same name), where you are asked to create a new project for automated testing. To create a project, click Create a new project.

Then enter the project name and click Create Project.

After that, the page for creating and passing test runs opens.

Now let’s perform a few more manipulations with our tests.

For tests, we use the Appium + Java + Junit configuration, and further actions will also be described based on this configuration. Your configuration may differ.

To use ADF, you need to create an archive with our tests that will run on various devices. To do this we need to connect some plugins in the project file Pom.xml, namely: Apache Maven JAR Plugin and Apache Maven Assembly Plugin. To enable it, specify the following in the Pom file:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <version>2.6</version>
            <executions>
                <execution>
                    <goals>
                        <goal>test-jar</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <version>2.10</version>
            <executions>
                <execution>
                    <id>copy-dependencies</id>
                    <phase>package</phase>
                    <goals>
                        <goal>copy-dependencies</goal>
                    </goals>
                    <configuration>
                        <outputDirectory>${project.build.directory}/dependency-jars/</outputDirectory>
                    </configuration>
                </execution>
            </executions>
        </plugin>

        <plugin>
            <artifactId>maven-assembly-plugin</artifactId>
            <version>2.5.4</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>single</goal>
                    </goals>
                    <configuration>
                        <finalName>zip-with-dependencies</finalName>
                        <appendAssemblyId>false</appendAssemblyId>
                        <descriptors>
                            <descriptor>src/main/assembly/zip.xml</descriptor>
                        </descriptors>
                    </configuration>
                </execution>
            </executions>
        </plugin>

    </plugins>
</build>

Make sure that you are using the latest versions of plugins. Update the plugins if necessary.

After Maven connects all the plugins, you need to use them to create the jar files of our test, as well as an archive with them. To do this, in IntelliJ IDEA, open the Maven sidebar, open the Plugins folder, open the jar folder, and click on jar:jar, and then on jar:test-jar.

As a result of plug-ins execution, two jar files will be generated in the target folder in the root folder of the project.

Next, in IDEA, in the assembly folder, you need to run a plugin called assembly:assembly.

After executing the plugin, a zip file with our tests will appear in the same target folder. We will need it in the future.

Back in the ADF. On the page with the created project, select “Create a new run”.

Next, we will configure our test run. At the first stage, you need to download the tested APK. To do this, select the button with Android/iOS and download the APK in the suggested field.

Once the APK has been downloaded, all the capabilities of this app will be available. Make sure that everything is correct, then click Next step.

The next step is to configure your test directly. To do this, in the drop-down window, select the frameworks that were used to write it (I have Appium + Java + JUnit; if you have a different configuration, then choose it from the list).

Then you will be prompted to download the test. As a test, we use the zip archive that was created earlier. Once the test archive is loaded, you can select optional configurations such as Video Recording and App Performance Data Capture (I recommend doing this, since further debugging of tests will be somewhat easier), and proceed to the next stage.

At the next stage, we add the devices that we want to run our app on for autotests. You can choose a ready-made configuration that Amazon offers, or click “Create new device pool” and specify a set of devices yourself, specifying its name and description

After the devices are selected, we go to the next stage, where the device configuration is set. Device Farm allows you to add additional data, add other applications (for example, to test interaction), specify the location of the device, and much more. Once the desired configuration for your devices is selected, we proceed to the final stage.

At the final stage, the maximum operating time for the device is set, after which the device is forced to stop. I recommend taking time with a small margin. If your tests are supposed to take 20 minutes, then you should take 30–35 minutes per device. After selecting the time, we complete the configuration and start the test run.

Once we have confirmed, the tests start running on each of the devices we have selected. To see the details, go to the test run by clicking on it. Then you can track your overall progress in passing tests, and stop them if necessary.

After passing the tests successfully (or unsuccessfully), you will see a message indicating the end of execution. We will be able to study the logs for each device, watch a video recording of test execution, and explore other parameters, such as performance.

BrowserStack

In order to run your autotests on Browserstack, you first need to register at browserstack.com. The resource provides a free trial version. By clicking on “Free trial” or “Get started free” proceed to create your account.

After that, we will be asked to create a free account. You can do this using your Google account, or using the registration form. Then you should click on the “Sign me up” button and we will get to the main page. Further, by clicking on the “App Automate” link, go to the page with autotests.

To run your autotests using the Browserstack on the current page, you need to click “Get started with App Automate”

After that, we will be asked to choose a framework with which our test is launched. For my tests I choose Appium, but you can also choose Espresso or XCUI.

After clicking, we will have access to the settings for the test. You should choose the language in which the tests are written. I will have it Java, although this is not so important for further actions.

You will also need to download the APK or IPA of your application, as well as the device on which all this will be tested.

As a result of the selected configurations, will be generated a code that will need to be added to your project, after which you can start executing tests.

After that we run the tests in the same way as if we were running them from a real device.

The test will start executing and after it is fully executed on the Browserstack, we will be shown the execution result. In order to see the result, we need to go to the Dashboard and click on the test results that appear. Next, you will need to select the session of the running test.

In the test results, the logs of the test launch and its passage are available, as well as a video of how the test was performed.

If you need tests to run on multiple devices, then they should also be selected and written in capabilities.

Conclusion

At the moment, if you do not have your own devices for automated testing, or you want to improve this process, then such quite affordable and effective options as device farms can help with this. It won’t take long to figure out how to run your tests, but as a tool, it will be of great help in your test automation tasks. Which one to choose is a matter of ease of use and price, everyone can choose the best option for themselves.

If you or someone you know needs to develop a high-quality mobile application, web service or to help in solving business needs, feel free to contact Effective, we are ready to collaborate with you in any way: both on the whole project and outstaff. You can contact us via e-mail contact@effective.band