How we test AppsFlyer SDK — part 1

Intro

I joined AppsFlyer at the beginning of 2020 to lead the mobile SDK automation team. By doing so I also joined the amazing quest to answer one all-important question: What is the best way to perform mobile test automation for a mobile SDK product?

In order to answer this question, we need to understand who our end-users are: mobile app developers who implement our SDK abilities within their app code, across different plugins (Unity, React…etc), operating systems, and devices brands.

Another difficulty we have is that our testing usually requires “real” devices. The main challenge here is how to test multiple scenarios on multiple devices simultaneously. Basically, we need a low maintenance solution that fits everything.

In this blog post, we will outline the journey of designing and building our mobile SDK testing automation system, sharing the decisions we took, and things we learned along the way.

This blog post will have 5 parts:

1. Creating apps for SDK testing, and building apps from repository
2. Testing devices and automation commands
3. Executing scenarios on our apps with rest API
4. Validating executables using a validation service
5. Connecting the dots — automation architecture

Let’s get started!

Creating apps for SDK testing

First, before even talking about automating the tests, we had to find the best, and most efficient way to test our SDK manually.

So we needed to come up with a simple app design to hold test cases.

With a simple struct of test scenarios held in the TableView/RecyclerView UI, we pass the clicked test item to the test ViewController/Activity, using a singleton class as the test manager that holds the test struct.

The TestViewController will call the executeTest in testManager passing itself in order to get the result to UI.

Building apps from repository

Once we have the app, we hold it in a GitLab repository. We require the ability to build it dynamically for each SDK version release, so the SDK version is a parameter that we definitely need.

Since we are running real devices, we use a local machine with a hub to connect devices to it (we will get to that soon).

And, in order to build the apps in runtime (Xcode/Gradle) we need executable script (py-script) to build the apps with the relevant SDK version.

The script has some caching logic to save some time if the app is already built.

All that is left now is to make sure that this parameter is passed by our Jenkins in the job that we have for building the apps, and to call the script from within the executable shell.

Jenkins will run the commands for:

Echo python3 cocoapods_init_ios.py $tag $version $env

Parameters will be passed to the py-script:

parser = argparse.ArgumentParser()
parser.add_argument('-v', '--version', help='version (4.8.0)', required=False)
parser.add_argument('-e', '--environment', help='release / test', required=False)
parser.add_argument('-t', '--tags', help='@AppsFlyer_Install_Organic', required=True)
args = parser.parse_args()

In our script, we need to build the iOS and Android apps with the relevant SDK. To do so, we need to change the Gradle and pod files within the script using the parameters that we received and call the build commands.(We can see the Android and iOS build commands in the snippet below.)

os.system("./gradlew assembleDebug") // ----android 
command_to_build_archive = "xcodebuild -workspace {xcworkspace_file_name}.xcworkspace -scheme {app_name} clean archive -archivePath build/{app_name}".format(
    app_name=app_name, xcworkspace_file_name=xcworkspace_file_name) // ----iOS

Now, as a proof of concept this was great. Next, in order for it to work in a real-life scenario, we needed to delete the Gradle/pod file for every new SDK we got, and rebuild it with the new SDK version parameter. (And when we had a build with the same SDK version we did not rebuild the app.)

Jenkins running the py-script to build the apps. (In a case where the app has an existing SDK version, we skip the build.)

Result

And now you have it:

• An application that we can run SDK scenarios on.
• A repository for the apps.
• A script that builds the app with latest SDK on our local machine.

Next challenge: How to run this app on our test device using automation?

In part 2, we will talk about test devices and how to automatically run commands.