Mobile Test Automation Using AWS Device Farm

Utilizing the Java SDK to integrate AWS Device Farm efficiently into a CI pipeline.

Server-side execution model

When executing tests, AWS (at least in their public cloud) uses an approach which I call server-side execution. This means that the test code runs entirely on their infrastructure. The alternative to this is what providers like Browserstack or Saucelabs use where the test code runs on the user’s machine, called client-side execution, and interacts with the cloud service via a web service. In general, client-side execution is easier to implement; if you already know how to execute tests locally, the configuration changes you need to make are minimal.

Executing tests via the web console

The easiest way to start running tests on AWS Device Farm is through their web console. It is a tedious process if you need to do this frequently (we will talk about using their API in just a few moments), but going through the web console is an excellent way to familiarise ourselves with the different steps in the process.

What next?

Congratulations! You successfully executed your first Appium tests on AWS Device Farm. Let’s look at some of the challenges and see how using a custom test spec can make our lives easier. Then, we will showcase how to use their API to automate the process of scheduling tests.

Custom environments for the rescue

If you take another look at the custom environment specification above, you can see how we can utilize this feature to remove this restriction. The base64 encoded information includes our test configuration. This allows us to customize test runs by uploading a relatively small YAML file (the test spec) rather than re-creating the much larger test package. We encode the configuration during the creation of the test spec. When it’s executed we decode it back into a file which is then passed to the execution as a parameter.

Further improvements

We can take this approach even further and remove the need for a test package altogether. Instead, we configure the custom environment to fetch the necessary code and configuration from an external source and build it entirely on the AWS instance.

Executing tests via API

Now that we’ve gone through the steps of manually creating a test execution let’s see how we can use the excellent AWS Java SDK (there are SDKs for other programming languages too) to automate the process.



Besides the device filters, there is some additional configuration you can define for AWS test executions (those were the settings we skipped during the Specify device state on the web console):

Maven integration

For a slightly different way of integrating AWS Device Farm (but using the same Java SDK), you can have a look at my Maven plugin. It provides goals for each step (uploading app package, uploading test package, scheduling a run etc.) which allows you to separate them. For example, you can upload a new app package every time there is a new build and upload a new test package every time there is a change in your test code. The test execution itself can then be independent of both of these and just re-use the latest (or any specific) app and test packages.

Jenkins plugin

There is also a Jenkins plugin for AWS Device Farm but it is not as flexible as the solutions described in this article.


In this write-up, we showcased a PoC-style demo of how to run tests on AWS Device Farm in just about the same way as locally or on client-side execution platforms like Browserstack. Getting your tests running on AWS Device Farm and especially integrating them into a build process in an efficient way is not a straightforward process (for all but the most basic requirements), and I hope the examples can help others who are facing similar challenges.

Better Programming

Advice for programmers.

Martin Schneider

Written by

Software architect and basketball coach from Austria currently living in Singapore

Better Programming

Advice for programmers.