Appium Parallel Tests
Up to Appium 1.7 release, parallel test execution has always been a painful experience. Without using any commercial device cloud solutions such as SauceLabs, setting up an in-house device cluster for Appium test usually ends up with building a Selenium grid as shown in this picture:
For each target real or virtual device we want to connect, a dedicate Appium server instance must be created. All Appium server instances need to be linked into a Selenium grid. Appium client test scripts will start parallel WebDriver sessions towards the Selenium grid, which will further dispatch requests to each corresponding Appium server and target device.
Such infrastructure is hard to maintain and not scalable due to the facts that:
- JSON configuration files are required for the Selenium hub node as well as for each Appium server instance so that it can serve as a grid node.
- Each Appium server needs to be started with the right parameter particularly defined for the target device connected.
- Users need to implement failure recovery mechanism to bring dead Appium server instance back to life. Default Selenium grid implementation does not provide such features.
Fortunately, a new improvement has been introduced in Appium 1.7 release to support running parallel tests on a single Appium server instance. Now the infrastructure can be simplified as below:
In case of Android test using UIAutomator2 driver, multiple Android devices can be connected to the same Appium server on different systemPort
. When creating AndroidDriver
in client test scripts, we can use a new desired capability systemPort
to specify which device to run.
A new feature coming from Xcode 9 is to allow running multiple iOS simulators on the same machine. Appium 1.7 release takes the advantage of this feature to make iOS parallel test using XCUITest
driver work in a similar way, different WebDriverAgent ports can be specified using desired capability wdaLocalPort
when instantiating the IOSDriver
. Each wdaLocalPort
is associated with a target iOS device.
Now let’s have a look at some examples on parallel tests of Android native apps, iOS native apps and mobile web apps.
Parallel Android Native Tests
First, we start two Android AVDs with different hardware profiles and Android versions:
$ adb devices
List of devices attached
emulator-5556 device
emulator-5554 device
Emulator emulator-5554
has Android version 8.0
, and emulator emulator-5556
is on Android 7.1.1
.
Now let’s create a maven project in Intellij IDE and add dependencies of Appium Java client library and TestNG to its pom.xml
file:
<dependencies>
<dependency>
<groupId>io.appium</groupId>
<artifactId>java-client</artifactId>
<version>5.0.4</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.11</version>
<scope>test</scope>
</dependency>
</dependencies>
Two desired capabilities are important for parallel tests of Android native apps:
udid
the device idsystemPort
forappium-uiautomator2-driver
, set a different system port for each Appium server instances.
systemPort
specifies the port Appium uses to connect to the appium-uiautomator2-server
. Default value is 8200
. To avoid conflicts when running tests in parallel, one should select different port values between 8200
and 8299
. A single Appium server utilizes different systemPort
to split the traffic flowing towards different devices connected.
We use TestNG here to implement our test case and run it in parallel on different Android emulators. Define a setup
method with @BeforeTest
annotation and a teardown
method with @AfterTest
annotation in TestNG code to instantiate WebDriver session and shut it down before and after each test:
@BeforeTest(alwaysRun = true)
@Parameters({"platform", "udid", "systemPort"})
public void setup(String platform, String udid, String systemPort) throws Exception { URL url = new URL(APPIUM_SERVER_URL); String[] platformInfo = platform.split(" "); DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(MobileCapabilityType.AUTOMATION_NAME, "uiautomator2");
capabilities.setCapability(MobileCapabilityType.PLATFORM_NAME, platformInfo[0]);
capabilities.setCapability(MobileCapabilityType.PLATFORM_VERSION, platformInfo[1]);
capabilities.setCapability(MobileCapabilityType.DEVICE_NAME, "Android Emulator");
capabilities.setCapability(MobileCapabilityType.UDID, udid);
capabilities.setCapability(AndroidMobileCapabilityType.SYSTEM_PORT, systemPort);
capabilities.setCapability(MobileCapabilityType.APP, "/Users/henrrich/Documents/work/jsta/appium/demo-apps/demo.apk");
capabilities.setCapability(MobileCapabilityType.ORIENTATION, "PORTRAIT");
capabilities.setCapability(MobileCapabilityType.NO_RESET, false); driver = new AndroidDriver<MobileElement>(url, capabilities); driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
}@AfterTest(alwaysRun = true)
public void teardown() throws Exception {
if (driver != null) {
driver.quit();
}
}
The setup
method is parameterised with platform
, udid
and systemPort
, whose values are used to configure desired capabilities platformName
, platformVersion
, udid
and systemPort
.
The app we are going to automate is the sample app from AWS device farm. The app includes various examples of Android UI widgets, which is perfect for demo purpose. Here we add a TestNG test method implementing a login and logout flow:
@Test
public void testLoginAndLogout() throws InterruptedException {
navigateToCategory("drawer_login_page"); MobileElement usernameInput = (MobileElement) driver.findElementByAccessibilityId("Username Input Field");
MobileElement passwordInput = (MobileElement) driver.findElementByAccessibilityId("Password Input Field");
MobileElement loginButton = (MobileElement) driver.findElementByAccessibilityId("Login Button"); usernameInput.clear();
usernameInput.sendKeys(CORRECT_USER_NAME); passwordInput.click();
passwordInput.sendKeys(CORRECT_PASSWORD); loginButton.click(); Assert.assertEquals(getMessage(), LOGIN_SUCCESS_MESSAGE); driver.findElementByAccessibilityId("Alt Button").click();
Assert.assertTrue(loginButton.isDisplayed() && usernameInput.isDisplayed() && passwordInput.isDisplayed());
}
The goal is to run this login and logout test in parallel onto the two Android AVDs we have launched. So we define a test suite in testng.xml
as below:
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" ><suite name="AndroidNativeSuite" verbose="1" parallel="tests" thread-count="2">
<test name="Android native app test on Android 7">
<parameter name="platform" value="Android 7.1.1"/>
<parameter name="udid" value="emulator-5556"/>
<parameter name="systemPort" value="8200"/>
<classes>
<class name="AndroidNativeParallelTests" />
</classes>
</test>
<test name="Android native app test on Android 8">
<parameter name="platform" value="Android 8.0"/>
<parameter name="udid" value="emulator-5554"/>
<parameter name="systemPort" value="8201"/>
<classes>
<class name="AndroidNativeParallelTests" />
</classes>
</test>
</suite>
Class AndroidNativeParallelTests
contains the testLoginAndLogout
test method as well as setup
and teardown
methods defined previously. In suite AndroidNativeSuite
we run the login and logout test in parallel in two separated test threads with different parameter sets, one for emulator-5556
and one for emulator-5554
.
Let’s run the suite, and we can see tests are running concurrently on two emulators with different hardware profiles and Android versions, but with only one Appium server.
Parallel iOS Tests
As mentioned before, thanks to the feature from Xcode 9, running multiple iOS simulators on one MAC host becomes feasible. Similar to the Android demo, we boot up two iOS simulators, one is iPhone 6 (iOS 10.3.1), and the other is iPhone 7 (iOS 11.2).
Device UDIDs are listed:
$ xcrun simctl list | grep Booted
iPhone 6 (D2D2D9FB-21E0-4198-B938-F2FA798B57A9) (Booted)
iPhone 7 (D8B5AD32-0108-4CCC-90EF-04577893870E) (Booted)
Running parallel iOS tests with a single Appium server requires the following desired capabilities:
udid
simulator UDID, as retrieved fromxcrun simctl list
command.deviceName
simulator nameplatformVersion
simulator OS versionwdaLocalPort
unique wdaPort, as WDA defaults to 8100.
wdaLocalPort
is the key to the magic of parallel test here. As we know, Appium relies on WebDriverAgent to automate iOS apps under the hood. WebDriverAgent contains an application server internally which terminates the incoming JSON Wire protocol requests and control the app under test. When running parallel tests, traffic are splitted to different wdaLocalPort
and handled by the WebDriverAgent listening on that port to avoid conflicts.
We define corresponding parameters to the setup
method in our TestNG test, so their values will be used to establish the IOSDriver
session on a particular target device.
@BeforeTest(alwaysRun = true)
@Parameters({"platform", "udid", "deviceName", "wdaLocalPort"})
public void setup(String platform, String udid, String deviceName, String wdaLocalPort) throws Exception {URL url = new URL(APPIUM_SERVER_URL);String[] platformInfo = platform.split(" ");DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(MobileCapabilityType.AUTOMATION_NAME, "XCUITest");
capabilities.setCapability(MobileCapabilityType.PLATFORM_NAME, platformInfo[0]);
capabilities.setCapability(MobileCapabilityType.PLATFORM_VERSION, platformInfo[1]);
capabilities.setCapability(MobileCapabilityType.DEVICE_NAME, deviceName);
capabilities.setCapability(MobileCapabilityType.UDID, udid);
capabilities.setCapability("wdaLocalPort", wdaLocalPort);
capabilities.setCapability(MobileCapabilityType.APP, "/Users/henrrich/Documents/work/jsta/appium/demo-apps/TaskApplication.app");
capabilities.setCapability(MobileCapabilityType.ORIENTATION, "PORTRAIT");
capabilities.setCapability(MobileCapabilityType.NO_RESET, false);driver = new IOSDriver<MobileElement>(url, capabilities);
}
The testng.xml
will contain two tests binding to the two iOS simulators launched.
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" ><suite name="IOSNativeSuite" verbose="1" parallel="tests" thread-count="2">
<test name="IOS native app test on iPhone 7">
<parameter name="platform" value="iOS 11.2"/>
<parameter name="udid" value="D8B5AD32-0108-4CCC-90EF-04577893870E"/>
<parameter name="deviceName" value="iPhone 7"/>
<parameter name="wdaLocalPort" value="8100"/>
<classes>
<class name="IOSNativeParallelTests" />
</classes>
</test>
<test name="IOS native app test on iPhone 6">
<parameter name="platform" value="iOS 10.3.1"/>
<parameter name="udid" value="D2D2D9FB-21E0-4198-B938-F2FA798B57A9"/>
<parameter name="deviceName" value="iPhone 6"/>
<parameter name="wdaLocalPort" value="8101"/>
<classes>
<class name="IOSNativeParallelTests" />
</classes>
</test>
</suite>
Running the suite will start parallel iOS tests on two simulators. The WDA server for iPhone 7 is listening for the incoming requests on port 8100
and another WDA server for iPhone 6 is listening on port 8101
.
Parallel Web App Tests
By the time this article is written, parallel Safari/WebView sessions on iOS simulators are not working in Appium due to an Apple bug.
In addition to udid
and systemPort
capabilities, running parallel web app testing for mobile Chrome browser requires setting up one more desired capability chromeDriverPort
. This capability specifies on which port Chrome driver will work. When automating web applications in Chrome browser concurrently on different Android devices with one Appium server, different chromeDriverPort
need to be selected to avoid traffic conflicts.
The code example here shows the setting of desired capabilities for mobile Chrome browser.
@BeforeTest(alwaysRun = true)
@Parameters({"platform", "udid", "chromeDriverPort", "chromeDriverPath"})
public void setup(String platform, String udid, String chromeDriverPort, @Optional String chromeDriverPath) throws Exception {
URL url = new URL(APPIUM_SERVER_URL);String[] platformInfo = platform.split(" ");DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(MobileCapabilityType.AUTOMATION_NAME, "Appium");
capabilities.setCapability(MobileCapabilityType.PLATFORM_NAME, platformInfo[0]);
capabilities.setCapability(MobileCapabilityType.PLATFORM_VERSION, platformInfo[1]);
capabilities.setCapability(MobileCapabilityType.DEVICE_NAME, "Android Emulator");
capabilities.setCapability(MobileCapabilityType.UDID, udid);// systemPort is optional when testing android web app with driver "Appium"capabilities.setCapability("chromeDriverPort", chromeDriverPort);if (chromeDriverPath != null) {
capabilities.setCapability(AndroidMobileCapabilityType.CHROMEDRIVER_EXECUTABLE, chromeDriverPath);
}capabilities.setCapability(MobileCapabilityType.BROWSER_NAME, "Chrome");
capabilities.setCapability(MobileCapabilityType.ORIENTATION, "PORTRAIT");
capabilities.setCapability(MobileCapabilityType.NO_RESET, false);driver = new AndroidDriver<MobileElement>(url, capabilities);
}
To be noted that we use Appium
driver here instead of uiautomator2
, since uiautomator2
driver does not yet support mobile web app testing in Appium. systemPort
desired capability works together with uiautomator2
driver, so it is not required in mobile web app testing here.
We also declare parameter chromeDriverPath
as optional. If it is missing, the default chromedriver
from local Appium installation will be used, otherwise the specified chromedriver
binary will be used for automating the mobile Chrome browsers of corresponding versions.
Given the same two Android AVDs are launched, we define the testng.xml
as below:
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" ><suite name="WebAppSuite" verbose="1" parallel="tests" thread-count="2">
<test name="Web app test on Android 7">
<parameter name="platform" value="Android 7.1.1"/>
<parameter name="udid" value="emulator-5556"/>
<parameter name="chromeDriverPort" value="9516"/>
<parameter name="chromeDriverPath" value="/Users/henrrich/Documents/work/jsta/appium/chromedriver"/>
<classes>
<class name="AndroidWebAppParallelTests" />
</classes>
</test>
<test name="Web app test on Android 8">
<parameter name="platform" value="Android 8.0"/>
<parameter name="udid" value="emulator-5554"/>
<parameter name="chromeDriverPort" value="9515"/>
<classes>
<class name="AndroidWebAppParallelTests" />
</classes>
</test>
</suite>
For emulator-5556
, its ChromeDriver works on port 9516
while the ChromeDriver of emulator-5554
is listening on port 9515
. emulator-5556
comes with a Chrome browser of version 55
, so we set chromeDriverPath
parameter to point to an external ChromeDriver (version 2.28) which is compatible with Chrome 55
.
Run the test suite and we can see concurrent mobile Chrome tests on two Android AVDs.
The complete demo project source code can be downloaded from git repository:
$ git clone https://gitlab.com/huanghe389/appium_parallel_execution.git
By He Huang from SetPace
Visit https://www.setpace.se/ for courses in Appium and other Test Automation Frameworks