Running Automation Tests on AWS Device Farm using Appium and TestNG

I have given a task of running my mobile automation tests on AWS Device Farm. We used to run our tests locally on both Android and iOS devices. There are several drawbacks of running tests locally such as: device maintenance, device connectivity issues, devices keep disconnecting, limited number of devices, etc.

There are several cloud testing providers out there such as Saucelabs, Browserstack, Testingbot and many others. I have given this task to try AWS Device Farm because we already use AWS services anyway, then why not try their cloud testing solution. I will say in the beginning that AWS Device Farm is not perfect yet, it doesn’t work out of the box (Such as changing hub URL and Desired Capabilities), but it has a potential for being a solid cloud testing solution since it is backed by Amazon.


Let’s get started.

AWS Device Farm Setup

First, you need to signup for AWS Device Farm, they will give you free 1,000 minutes to get started.

After you signup create a new project and let’s get back to it later.

Tests Packages Setup

Usually, on some cloud testing provider such as saucelabs, the test code will run on your computer while instructing cloud devices to do something, but in Device Farm you need to package all your tests to be run on the cloud.

Here are things that I used for my testing framework:

  • JAVA
  • Maven
  • TestNG
  • Appium

AWS already have somehow complete documentation how to set this here. But I will explain it again.

  1. Packing the Tests

I mention in the beginning that all the tests need to be packed and sent to the cloud server, in order to do this maven JAR plugin to pack all the tests into a single JAR file. Add this to your pom.xml file header under version tag.

<packaging>jar</packaging>

Add this plugin to your pom file

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.6</version>
<executions>
<execution>
<goals>
<goal>test-jar</goal>
</goals>
</execution>
</executions>
</plugin>

2. Packing Dependencies

Next you need to build all dependencies into a single jar file with maven dependency plugin

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.10</version>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/dependency-jars/</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>

So now you will have two JARs files in your target folder. One containing your tests files and the other containing dependencies.

3. Zip Everything!

Create a new file src/main/assembly/zip.xml and put this inside.

<assembly
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
<id>zip</id>
<formats>
<format>zip</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<fileSets>
<fileSet>
<directory>${project.build.directory}</directory>
<outputDirectory>./</outputDirectory>
<includes>
<include>*.jar</include>
</includes>
</fileSet>
<fileSet>
<directory>${project.build.directory}</directory>
<outputDirectory>./</outputDirectory>
<includes>
<include>/dependency-jars/</include>
</includes>
</fileSet>
</fileSets>
</assembly>

And put this in your pom.xml

<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.5.4</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<finalName>zip-with-dependencies</finalName>
<appendAssemblyId>false</appendAssemblyId>
<descriptors>
<descriptor>src/main/assembly/zip.xml</descriptor>
</descriptors>
</configuration>
</execution>
</executions>
</plugin>

4. Stop and Check!

Compare your repository with mine here https://github.com/danggrianto/aws-devicefarm-appium/tree/0.1. When you follow this guide, you should be at least have the same settings.

5. Base Test

Let’s write a simple test to run on device farm. The purpose of this test is just to make sure that device farm able to upload the app and the tests then take a screenshot. That’s all. TL;DR: It is not a test.

public class BasicTest {
private static AppiumDriver<WebElement> driver;

/**
* Run before each method
*
@throws MalformedURLException
*/
@BeforeMethod(alwaysRun = true)
public void setUp() throws MalformedURLException {
DesiredCapabilities capabilities = new DesiredCapabilities();
URL appiumURL = new URL("http://127.0.0.1:4723/wd/hub");
driver = new AndroidDriver<WebElement>(appiumURL, capabilities);
}

/**
* Run After each test method
*/
@AfterMethod(alwaysRun = true)
public void tearDown(){
driver.quit();
}


@Test
public void basicTest(){
takeScreenshot("basicTest");
assertTrue(true);
}

/**
* Take Screenshot
*
@param name file name
*
@return true if successful
*/
private boolean takeScreenshot(final String name) {
String screenshotDirectory = System.getProperty("appium.screenshots.dir");
File screenshot = driver.getScreenshotAs(OutputType.FILE);
return screenshot.renameTo(new File(screenshotDirectory, String.format("%s.png", name)));
}
}

Source Code: https://github.com/danggrianto/aws-devicefarm-appium/tree/0.2

6. Package Everything

Go to command line and run this command

mvn clean package -DskipTests=true

After you get a BUILD SUCCESS message, check your target directories for zip-with-dependencies.zip file.

To double check things run this command to see what is inside your JAR file.

jar tf target/automation-1.0-SNAPSHOT-tests.jar

Note: automation is my package name, so the name is depending on your pom.xml, so it will vary.

These are the output of that command on my computer.

META-INF/MANIFEST.MF
META-INF/
tests/
META-INF/maven/
META-INF/maven/danggrianto/
META-INF/maven/danggrianto/automation/
tests/BasicTest.class
META-INF/maven/danggrianto/automation/pom.xml
META-INF/maven/danggrianto/automation/pom.properties

Notice that tests/BasicTest.class already included, so we are good to go.

Run Tests On Device Farm

  1. Go back to Device Farm Site.
  2. Select Project that you created before above.
  3. Create A New Run

4. Upload APK file. I put sample apk on github if you don’t have any apk to test.

5. On Configure a test page, pick Appium Java TestNG and upload zip-with-dependencies.zip file.

6. Select Device. You can use default Top Devices.

OR create a new device pool with your preferred devices. To make test faster just pick one android device.

7. Specify device state. I just leave it on default settings.

8. Review and start run.

9. Grab a coffee and wait! :)

10. Check the result.

Woohoo… Congratulation, your test passed.

Screenshot for confirmation that the test, really pass.

Conclusions

Like I mentioned above device farm is not perfect yet there are several issue with it such as:

  • Logging. There are some logs but so far I haven’t been able to get the log of my test. Just appium log and device log.
  • Cannot pick tests. There is no options to pick which tests to run, Device Farm is going to run whatever tests file you package on the zip file.
  • Outdated Appium: When I wrote this blog they only support appium 1.6.3 while the latest version is 1.6.4

Though Device Farm is not as easy as Saucelabs but it has a lot of potentials.

On my next blog, I will show how to setup this tests with jenkins. next tutorial is here.