Appium Parallel Tests
Up to Appium 1.7 release, parallel test execution has always been a painful experience. Without using any commercial device cloud solutions such as SauceLabs, setting up an in-house device cluster for Appium test usually ends up with building a Selenium grid as shown in this picture:
For each target real or virtual device we want to connect, a dedicate Appium server instance must be created. All Appium server instances need to be linked into a Selenium grid. Appium client test scripts will start parallel WebDriver sessions towards the Selenium grid, which will further dispatch requests to each corresponding Appium server and target device.
Such infrastructure is hard to maintain and not scalable due to the facts that:
- JSON configuration files are required for the Selenium hub node as well as for each Appium server instance so that it can serve as a grid node.
- Each Appium server needs to be started with the right parameter particularly defined for the target device connected.
- Users need to implement failure recovery mechanism to bring dead Appium server instance back to life. Default Selenium grid implementation does not provide such features.
Fortunately, a new improvement has been introduced in Appium 1.7 release to support running parallel tests on a single Appium server instance. Now the infrastructure can be simplified as below:
In case of Android test using UIAutomator2 driver, multiple Android devices can be connected to the same Appium server on different systemPort. When creating AndroidDriver in client test scripts, we can use a new desired capability systemPort to specify which device to run.
A new feature coming from Xcode 9 is to allow running multiple iOS simulators on the same machine. Appium 1.7 release takes the advantage of this feature to make iOS parallel test using XCUITest driver work in a similar way, different WebDriverAgent ports can be specified using desired capability wdaLocalPort when instantiating the IOSDriver. Each wdaLocalPort is associated with a target iOS device.
Now let’s have a look at some examples on parallel tests of Android native apps, iOS native apps and mobile web apps.
Parallel Android Native Tests
First, we start two Android AVDs with different hardware profiles and Android versions:
$ adb devices
List of devices attached
emulator-5556 device
emulator-5554 deviceEmulator emulator-5554 has Android version 8.0, and emulator emulator-5556 is on Android 7.1.1.
Now let’s create a maven project in Intellij IDE and add dependencies of Appium Java client library and TestNG to its pom.xml file:
<dependencies>
<dependency>
<groupId>io.appium</groupId>
<artifactId>java-client</artifactId>
<version>5.0.4</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.11</version>
<scope>test</scope>
</dependency>
</dependencies>Two desired capabilities are important for parallel tests of Android native apps:
udidthe device idsystemPortforappium-uiautomator2-driver, set a different system port for each Appium server instances.
systemPort specifies the port Appium uses to connect to the appium-uiautomator2-server. Default value is 8200. To avoid conflicts when running tests in parallel, one should select different port values between 8200 and 8299. A single Appium server utilizes different systemPort to split the traffic flowing towards different devices connected.
We use TestNG here to implement our test case and run it in parallel on different Android emulators. Define a setup method with @BeforeTestannotation and a teardown method with @AfterTest annotation in TestNG code to instantiate WebDriver session and shut it down before and after each test:
@BeforeTest(alwaysRun = true)
@Parameters({"platform", "udid", "systemPort"})
public void setup(String platform, String udid, String systemPort) throws Exception { URL url = new URL(APPIUM_SERVER_URL); String[] platformInfo = platform.split(" "); DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(MobileCapabilityType.AUTOMATION_NAME, "uiautomator2");
capabilities.setCapability(MobileCapabilityType.PLATFORM_NAME, platformInfo[0]);
capabilities.setCapability(MobileCapabilityType.PLATFORM_VERSION, platformInfo[1]);
capabilities.setCapability(MobileCapabilityType.DEVICE_NAME, "Android Emulator");
capabilities.setCapability(MobileCapabilityType.UDID, udid);
capabilities.setCapability(AndroidMobileCapabilityType.SYSTEM_PORT, systemPort);
capabilities.setCapability(MobileCapabilityType.APP, "/Users/henrrich/Documents/work/jsta/appium/demo-apps/demo.apk");
capabilities.setCapability(MobileCapabilityType.ORIENTATION, "PORTRAIT");
capabilities.setCapability(MobileCapabilityType.NO_RESET, false); driver = new AndroidDriver<MobileElement>(url, capabilities); driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
}@AfterTest(alwaysRun = true)
public void teardown() throws Exception {
if (driver != null) {
driver.quit();
}
}
The setup method is parameterised with platform, udid and systemPort, whose values are used to configure desired capabilities platformName, platformVersion, udid and systemPort.
The app we are going to automate is the sample app from AWS device farm. The app includes various examples of Android UI widgets, which is perfect for demo purpose. Here we add a TestNG test method implementing a login and logout flow:
@Test
public void testLoginAndLogout() throws InterruptedException {
navigateToCategory("drawer_login_page"); MobileElement usernameInput = (MobileElement) driver.findElementByAccessibilityId("Username Input Field");
MobileElement passwordInput = (MobileElement) driver.findElementByAccessibilityId("Password Input Field");
MobileElement loginButton = (MobileElement) driver.findElementByAccessibilityId("Login Button"); usernameInput.clear();
usernameInput.sendKeys(CORRECT_USER_NAME); passwordInput.click();
passwordInput.sendKeys(CORRECT_PASSWORD); loginButton.click(); Assert.assertEquals(getMessage(), LOGIN_SUCCESS_MESSAGE); driver.findElementByAccessibilityId("Alt Button").click();
Assert.assertTrue(loginButton.isDisplayed() && usernameInput.isDisplayed() && passwordInput.isDisplayed());
}
The goal is to run this login and logout test in parallel onto the two Android AVDs we have launched. So we define a test suite in testng.xml as below:
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" ><suite name="AndroidNativeSuite" verbose="1" parallel="tests" thread-count="2">
<test name="Android native app test on Android 7">
<parameter name="platform" value="Android 7.1.1"/>
<parameter name="udid" value="emulator-5556"/>
<parameter name="systemPort" value="8200"/>
<classes>
<class name="AndroidNativeParallelTests" />
</classes>
</test>
<test name="Android native app test on Android 8">
<parameter name="platform" value="Android 8.0"/>
<parameter name="udid" value="emulator-5554"/>
<parameter name="systemPort" value="8201"/>
<classes>
<class name="AndroidNativeParallelTests" />
</classes>
</test>
</suite>
Class AndroidNativeParallelTests contains the testLoginAndLogout test method as well as setup and teardown methods defined previously. In suite AndroidNativeSuite we run the login and logout test in parallel in two separated test threads with different parameter sets, one for emulator-5556and one for emulator-5554.
Let’s run the suite, and we can see tests are running concurrently on two emulators with different hardware profiles and Android versions, but with only one Appium server.
Parallel iOS Tests
As mentioned before, thanks to the feature from Xcode 9, running multiple iOS simulators on one MAC host becomes feasible. Similar to the Android demo, we boot up two iOS simulators, one is iPhone 6 (iOS 10.3.1), and the other is iPhone 7 (iOS 11.2).
Device UDIDs are listed:
$ xcrun simctl list | grep Booted
iPhone 6 (D2D2D9FB-21E0-4198-B938-F2FA798B57A9) (Booted)
iPhone 7 (D8B5AD32-0108-4CCC-90EF-04577893870E) (Booted)Running parallel iOS tests with a single Appium server requires the following desired capabilities:
udidsimulator UDID, as retrieved fromxcrun simctl listcommand.deviceNamesimulator nameplatformVersionsimulator OS versionwdaLocalPortunique wdaPort, as WDA defaults to 8100.
wdaLocalPort is the key to the magic of parallel test here. As we know, Appium relies on WebDriverAgent to automate iOS apps under the hood. WebDriverAgent contains an application server internally which terminates the incoming JSON Wire protocol requests and control the app under test. When running parallel tests, traffic are splitted to different wdaLocalPort and handled by the WebDriverAgent listening on that port to avoid conflicts.
We define corresponding parameters to the setup method in our TestNG test, so their values will be used to establish the IOSDriver session on a particular target device.
@BeforeTest(alwaysRun = true)
@Parameters({"platform", "udid", "deviceName", "wdaLocalPort"})
public void setup(String platform, String udid, String deviceName, String wdaLocalPort) throws Exception {URL url = new URL(APPIUM_SERVER_URL);String[] platformInfo = platform.split(" ");DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(MobileCapabilityType.AUTOMATION_NAME, "XCUITest");
capabilities.setCapability(MobileCapabilityType.PLATFORM_NAME, platformInfo[0]);
capabilities.setCapability(MobileCapabilityType.PLATFORM_VERSION, platformInfo[1]);
capabilities.setCapability(MobileCapabilityType.DEVICE_NAME, deviceName);
capabilities.setCapability(MobileCapabilityType.UDID, udid);
capabilities.setCapability("wdaLocalPort", wdaLocalPort);
capabilities.setCapability(MobileCapabilityType.APP, "/Users/henrrich/Documents/work/jsta/appium/demo-apps/TaskApplication.app");
capabilities.setCapability(MobileCapabilityType.ORIENTATION, "PORTRAIT");
capabilities.setCapability(MobileCapabilityType.NO_RESET, false);driver = new IOSDriver<MobileElement>(url, capabilities);
}
The testng.xml will contain two tests binding to the two iOS simulators launched.
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" ><suite name="IOSNativeSuite" verbose="1" parallel="tests" thread-count="2">
<test name="IOS native app test on iPhone 7">
<parameter name="platform" value="iOS 11.2"/>
<parameter name="udid" value="D8B5AD32-0108-4CCC-90EF-04577893870E"/>
<parameter name="deviceName" value="iPhone 7"/>
<parameter name="wdaLocalPort" value="8100"/>
<classes>
<class name="IOSNativeParallelTests" />
</classes>
</test>
<test name="IOS native app test on iPhone 6">
<parameter name="platform" value="iOS 10.3.1"/>
<parameter name="udid" value="D2D2D9FB-21E0-4198-B938-F2FA798B57A9"/>
<parameter name="deviceName" value="iPhone 6"/>
<parameter name="wdaLocalPort" value="8101"/>
<classes>
<class name="IOSNativeParallelTests" />
</classes>
</test>
</suite>
Running the suite will start parallel iOS tests on two simulators. The WDA server for iPhone 7 is listening for the incoming requests on port 8100 and another WDA server for iPhone 6 is listening on port 8101.
Parallel Web App Tests
By the time this article is written, parallel Safari/WebView sessions on iOS simulators are not working in Appium due to an Apple bug.
In addition to udid and systemPort capabilities, running parallel web app testing for mobile Chrome browser requires setting up one more desired capability chromeDriverPort. This capability specifies on which port Chrome driver will work. When automating web applications in Chrome browser concurrently on different Android devices with one Appium server, different chromeDriverPort need to be selected to avoid traffic conflicts.
The code example here shows the setting of desired capabilities for mobile Chrome browser.
@BeforeTest(alwaysRun = true)
@Parameters({"platform", "udid", "chromeDriverPort", "chromeDriverPath"})
public void setup(String platform, String udid, String chromeDriverPort, @Optional String chromeDriverPath) throws Exception {
URL url = new URL(APPIUM_SERVER_URL);String[] platformInfo = platform.split(" ");DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(MobileCapabilityType.AUTOMATION_NAME, "Appium");
capabilities.setCapability(MobileCapabilityType.PLATFORM_NAME, platformInfo[0]);
capabilities.setCapability(MobileCapabilityType.PLATFORM_VERSION, platformInfo[1]);
capabilities.setCapability(MobileCapabilityType.DEVICE_NAME, "Android Emulator");
capabilities.setCapability(MobileCapabilityType.UDID, udid);// systemPort is optional when testing android web app with driver "Appium"capabilities.setCapability("chromeDriverPort", chromeDriverPort);if (chromeDriverPath != null) {
capabilities.setCapability(AndroidMobileCapabilityType.CHROMEDRIVER_EXECUTABLE, chromeDriverPath);
}capabilities.setCapability(MobileCapabilityType.BROWSER_NAME, "Chrome");
capabilities.setCapability(MobileCapabilityType.ORIENTATION, "PORTRAIT");
capabilities.setCapability(MobileCapabilityType.NO_RESET, false);driver = new AndroidDriver<MobileElement>(url, capabilities);
}
To be noted that we use Appium driver here instead of uiautomator2, since uiautomator2 driver does not yet support mobile web app testing in Appium. systemPort desired capability works together with uiautomator2 driver, so it is not required in mobile web app testing here.
We also declare parameter chromeDriverPath as optional. If it is missing, the default chromedriver from local Appium installation will be used, otherwise the specified chromedriver binary will be used for automating the mobile Chrome browsers of corresponding versions.
Given the same two Android AVDs are launched, we define the testng.xml as below:
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" ><suite name="WebAppSuite" verbose="1" parallel="tests" thread-count="2">
<test name="Web app test on Android 7">
<parameter name="platform" value="Android 7.1.1"/>
<parameter name="udid" value="emulator-5556"/>
<parameter name="chromeDriverPort" value="9516"/>
<parameter name="chromeDriverPath" value="/Users/henrrich/Documents/work/jsta/appium/chromedriver"/>
<classes>
<class name="AndroidWebAppParallelTests" />
</classes>
</test>
<test name="Web app test on Android 8">
<parameter name="platform" value="Android 8.0"/>
<parameter name="udid" value="emulator-5554"/>
<parameter name="chromeDriverPort" value="9515"/>
<classes>
<class name="AndroidWebAppParallelTests" />
</classes>
</test>
</suite>
For emulator-5556, its ChromeDriver works on port 9516 while the ChromeDriver of emulator-5554 is listening on port 9515. emulator-5556comes with a Chrome browser of version 55, so we set chromeDriverPathparameter to point to an external ChromeDriver (version 2.28) which is compatible with Chrome 55.
Run the test suite and we can see concurrent mobile Chrome tests on two Android AVDs.
The complete demo project source code can be downloaded from git repository:
$ git clone https://gitlab.com/huanghe389/appium_parallel_execution.gitBy He Huang from SetPace
Visit https://www.setpace.se/ for courses in Appium and other Test Automation Frameworks
