How to Run iOS Apps in AWS Device Farm
This blog post describes how to prepare, build, and run iOS apps in AWS Device Farm.
The article is intended for mobile engineers and QEs, and provides the documentation of the solution that should make our life easier if there is a need to replicate it in the future.
The story of our challenges
We have an app that measures the performance of scheduled tasks on various mobile devices. We need to be able to run our app on as many physical devices as possible and at any time. The app needs to be able to login to our system and interact with it.
Due to the requirement for physical devices, we sought a platform with a good variety of physical devices and a way to run the app. Fortunately, the search was simple. Many similar services exist, such as AWS Device Farm, Google Firebase Labs, Azure DevLabs, Sauce Labs, and more. Another option was to build our mobile farm, but that would avert our attention from our primary goal. AWS Device Farm is a service providing managed access to many physical mobile devices and web browsers for testing. The essential part for us is that there is no need to maintain those devices or browsers. It is straightforward to use, it has devices that we need, and we are already using other services in AWS; therefore, AWS Device Farm was our final choice.
Another challenge of ours was determining how to pass credentials to mobile devices that are running in the cloud. Simple solutions like using environment variables in a Docker container are not applicable as mobile devices cannot run Docker containers. Furthermore, we wanted to avoid compiling the credentials directly into the app. Thus we searched for a solution how to pass values from the command line when starting the test.
The app preparation
The following paragraphs describe the steps needed to build a simple iOS XCTest UI. Feel free to skip this part if you already have your test that you would like to run. The AWS Device Farm does not support launch arguments; thus, our biggest challenge with the iOS UI test was passing values to test from the command line without recompiling the test.
New UI Test creation
- In XCode switch to “Test Navigator”
2. Click on the “+” button at the bottom left and select “New UI Test target”.
3. Give your test target any name. The standard convention is <app-name>UITest
Setup for passing values from the command line
- Create a TextField/TextView in the StoryBoard/SwiftUI. The newly created view must be visible and focusable to be visible for the XCUI test.
- Add accessibility identifiers to new fields.
- Go to app scheme setting > Test > Arguments > Environment variables > Add new variable with default value
An example of XCTest UI uses environment variables and passes the values from the command line to the running app.
Build IPA files for AWS
This step assumes that you have working signing for your application. There is a need to provide an application IPA file and tests the IPA file. I will show how to create the IPA files via UI and also the command line.
Classic UI style
- In Xcode select target device ‘Generic iOS Device’
- Then click Product -> Archive
- Wait for the build to finish, select the archive and click on “Distribute App”. The next steps are visible in the images below.
Steps needed to do local IPA file Export
4. Once you export the IPA files, you can close the list of archives
5. Click Product -> Build for test
6. Create /tmp/Payload folder. Any folder of your choice can replace the /tmp/ folder: mkdir -p ‘/tmp/Payload’
7. Go to “Derived Data” folder. You can obtain the path from “Xcode -> Preferences -> Locations” settings.
8. Once in “Derived Data” folder go to “your-app” -> Build -> Products -> Debug/Result-iphoneos
9. Copy folder *UITests-Runner.app to the newly created Payload folder
10. Zip the Payload folder and rename the zipped folder to uitests.ipa
Highly recommended way: Command line
Just use the following gist code to fulfil all manual steps above::
The project creation in AWS Device Farm
This section describes how to create a project in AWS Device Farm and configure it for the run. We use a browser to manually complete the setup below, but I strongly encourage you to check the Boto3 documentation for Device Farm for a way to automate what I am going to show.
Firstly login to AWS in your browser, then go to the “Device Farm” section and click on “Create a new project”. The next step is for providing “Project Name”, so fill in anything you prefer and click “Create project”.
I would suggest going to “Project Settings” and optimizing the “Execution Timeout” after the project creation. I believe that for initial testing, the value of 5 minutes should be more than enough. I recommend creating something like “Tiny Device Pool” for the initial test setup.
The run configuration
Here we set up the run on the device itself, and the result of the setup is scheduled to run against the device pool of your choice.
1. Go to your project and click “Run”.
2. Select the mobile platform (Android & Apple) and upload the IPA file that you created in the above step “Build IPA files for AWS”. Wait for upload and analysis to finish and then click next.
3. You should see the “Configure your test” screen. Select “Test” to be XCTest UI. Upload the UItest.ipa from step “Build IPA files for AWS” and wait for upload and analysis to finish.
4. Change the selection of “Choose your execution environment” to “Run your test in a custom environment”. This gives you the possibility to select “Default TestSpec for iOS XCTest UI” or another that you prefer more and allows you to click on “Edit”.
5. Add the following line to “pre_test:commands:”.
pre_test:
commands:
- /usr/libexec/PlistBuddy -c “add :<your UI test target>:EnvironmentVariables:PIN string ‘<pin value>’” $DEVICEFARM_XCUITESTRUN_FILE6. Optional Steps: In case you are not sure about the current content of the $DEVICEFARM_XCUITESTRUN_FILE then add:-cat $DEVICEFARM_XCUITESTRUN_FILE. The other option is that you can remove the added values from the run file by delete command below:
post_test:
commands:
— /usr/libexec/PlistBuddy -c “delete :<your UI test target>:EnvironmentVariables:PIN’” $DEVICEFARM_XCUITESTRUN_FILE7. Once done with modifications click on “Save as New” or “Save” based on whether you are editing the default or your TestSpec.
8. In the next step, choose your new device pool. After this, you can set up anything specific for your app on the next screen “Specify device state” and click on “Next”.
9. One of the last settings is to specify the expected time for your test. Once you do so, click on “Confirm and start run”.
The final words
If you find anything missing in the article or have a question about more details, please leave a comment. This blog post is a filter and sorted collection of information that we found in various discussion forums, official documentation, and our own experience. I want to say a big thanks to Chris Ismael, Jakub Snor, Robert Reed and Zdenek Papez, for their help with the article’s review.
