Continuous Testing using Jenkins and AWS Device Farm

This is part 2 of tutorial how to run mobile automation tests using java, testNG, appium on AWS Device Farm, for the first part read here.

In this tutorial I am going to show you how to run those tests that we wrote on the first part of my tutorial on jenkins to have Continuous Testing.

Install Jenkins Plugin

Assuming you already have jenkins running then you need to install this plugin.

1. Go to your jenkins

2. Click Manage Jenkins

3. Manage Plugins

4. Click Available Tabs

5. Filter for aws-device-farm

6. Install and Restart your jenkins (if needed)

IAM User Permission on AWS

Go to your AWS Console to configure Permission for Device Farm. Read the documentation here. Below just a copy from the official AWS documentation.

  1. Open the IAM console at
  2. Choose Policies.
  3. Choose Create Policy. (If a Get Started button appears, choose it, and then choose Create Policy.)
  4. Next to Create Your Own Policy, choose Select.
  5. For Policy Name, type a name for the policy (for example, AWSDeviceFarmAccessPolicy).
  6. For Description, type a description that helps you associate this IAM user with your Jenkins project.
  7. For Policy Document, type the following statement:
"Version": "2012-10-17",
"Statement": [
"Sid": "DeviceFarmAll",
"Effect": "Allow",
"Action": [ "devicefarm:*" ],
"Resource": [ "*" ]

8. Choose Create Policy.

To assign the access policy to the IAM user

  1. Open the IAM console at
  2. Choose Users.
  3. Choose the IAM user to whom you will assign the access policy.
  4. In the Permissions area, for Managed Policies, choose Attach Policy.
  5. Select the policy you just created (for example, AWSDeviceFarmAccessPolicy).
  6. Choose Attach Policy.

Copy your Access Key ID and Secret Access Key

Configure Jenkins Plugin

Go back to your jenkins and open Manage Jenkins > Configure System and go to AWS Device Farm Section

Enter your Access Key ID and secret Access Key to the Input field.

Create and Setting Up Jenkins Job

Click New Item to create a new Job

Pick Freestyle Project

Choose Source Code

Go to Build section and pick Invoke top-level Maven targets (assuming if you already have maven plugin installed and configured)

Set Goals as:

clean package -DskipTests

Note: You don’t have to use maven plugin, you can just use execute shell and run your command.

Go to Post-build Actions and select Run Tests on AWS Device Farm

Enter your Project name that you created in the first tutorial.

Pick Device Pool that you want to run.

Enter location of the APK on the jenkins. (Note: You can use file parameter to upload your apk file or you can copy artifact from other build. I assume that you already know how to do this)

Do other configuration as you wish.

Save the configurations and run the tests.

Run The Test

Yes, run those tests. Grab a coffee and wait.

Test Results

When the tests are completed jenkins is going to report it. It will be similar like this.

Clicking Full Report will send you to AWS Device Farm detailed report.


On my tests, I have a lot of setup and configurations such as user credentials, device informations etc that I put on the properties file. However, when packing the files into JAR file those properties file is not included. The next tutorial will show you how to include those properties file.