Collecting Insights from Test Automation

Telemetry/analytics collected during the automation run can help us to learn about the overall health of the application.

Every product has a large amount of automation code written for it these days.

This can be included with the few test cases running in BATS which get executed for every sanity run, the daily smoke tests, or the entire regression suites that might be running over the weekend. These test counts might be ranging from a few test cases to hundreds of them

There is a lot of data that we may not be collating from these test executions. Some of this information if properly collected and curated, can be used to collect and prepare various reports, which in turn can help us to prepare a benchmark to determine the overall health of the application over a period of time.

Goal

In this post, we would describe the approach we took to collect the various analytic/telemetry details from every test execution we run, and how we collate the data collected from our daily test executions.

These reports can be used to capture and generate various reports including and not limited to performance, crashes, and various other dashboards.

Getting Started

Ours is an Android SDK project and our automation is primarily an instrumentation-based framework, which uses Espresso, UIAutomation, and JUnit at its core level.

To collect the analytics/telemetry and performance-related information, we need to get some of the details like:

1. Details about the application under test.
2. Details about the device on which the tests are being run.
3. Test details include the execution status of the test, the total time needed for the test to complete, etc.
4. Failure reasons like the stack traces of Crash/ANRs seen on the application.

Once the above information is collected, we can push the analytics/telemetry data to a backend NoSQL database like MongoDB, DynamoDB, Cloud Firestore, etc.

With a NoSQL database, we can collect additional newer metrics which we feel might be useful to us any time during our future releases, with very few changes to our application.

From the collected information on the NoSQL database, we can make use of various charts/dashboard generating applications like Grafana or MongoDB Charts to maintain the release-specific or application-specific testing metrics and dashboards.

So let us dive into the details of how we are going to collect the above information and later push the data to the backend.

Collecting Application/Device details:

As we are using an instrumentation-based framework, we have very easy access to some of the build configuration parameters including and not limited to the below examples:

1. Application details like — Application version, bundle-id, android SDK used, etc.
2. Device details — Device model, OS version, OS API, etc.

Collecting Test details:

JUnit provides us with an excellent interface to process the test results at various stages via TestWatcher.

Using TestWatcher, we can hook into the various flows of the test execution as below:

1. When the Test begins.
2. When the Test gets completed with status as passed, failed, or skipped.

Using the above two options, we can get the below details about our tests:

1. Test details including the Test-Name, Class-Name, etc.
2. Test execution status like Pass/Fail/Skip.
3. Total time required to complete each test.

Sample telemetry/analytics collected during a test suite execution.

Till now, we have collected the details about the test under execution and the device-related telemetry information on which the tests are executed.

TestWatcher also provides us with the stack trace of the failure.

However this only contains the reason as to why the test has failed and in the case of automation, generally, it is the JUnit Assert which causes the failure either because the expected screen or the behavior was not present.

For an instrumentation test, when the application crashes, the test fails it abruptly, without executing the next automation statement.

While Android provides us with ways to continue the tests in their own individual sandbox environments by providing us with Android Test Orchestrator, how do we collect crash-related information?

Collecting Crash/ANR details:

AndroidJUnitRunner helps us to run the instrumented JUnit tests on Android devices.

Similar to the TestWatcher, we can hook onto our overall test execution status, and when the instrumentation tests fail, we can get the exception details including the extended Crash details from the OnException method.

Unlike a crash which would cause the instrumentation test to fail abruptly, the ANRs would only cause the test to fail based on the next JUnit Assert in the code, as the expected application screen may not be showing up.

As a result, when the test case fails, TestWatcher would only throw the exception from the JUnit Assert within the automation test.

For the ANRs we would need to overwrite our TestWatcher during the failures to collect the ADB logs and look for the ANR-related logs in them. With this, we would be able to collect the exception-related stack trace.

Pushing the collected analytics/telemetry data to the backend:

This overridden implementation of the TestWatcher can be consumed as a simple JUnit TestRule.

The data can be pushed to the analytics/telemetry using the below design:

JUnit Rule to collect and save the analytics/telemetry data to a NoSQL database.

On top of the default telemetry information which we are collecting as shown above, with a NoSQL database in the backend, we can always collect any other additional metrics like memory usage in terms of Proportional Set Size (PSS) or battery statistics, by collecting them at various stages of test execution.

We can then add this new information simply as an additional document, on top of the existing analytics/telemetry information already being collected.

A local NoSQL database can be configured using MongoDB, and at the end of test execution, the captured result of a failed test case, along with its stack trace of the failure can appear as below:

Similarly, the data can also be pushed to the Cloud Firestore, and the application related analytics/telemetry data can appear as below:

The columns from left to right are: App Version, Device OS version on which the Tests were run, and Device Model name on which the tests were run.

The columns from left to right are the Date on which the tests were run, the name of the test classes which were run, and the test status of the classes under consideration.

The columns from left to right are the test status of the classes under consideration, the name of the individual Tests, and the device and the test-related analytics/telemetry which was collected.

Analytics/Telemetry data analysis:

Once the analytics/telemetry data is collected, we can use it to generate reports for a given applications test reports as below:

1. Total Crashes/ANR/Failure-related statistics in a given release.
2. Comparison of the test data with the existing baseline w.r.t devices, releases, OS version, and various other statistics.
3. Collect additional data for the applications like memory usage, battery usage, the time taken for each activity/application screen to load, etc.

Using MongoDB Charts, we can even prepare dashboards as below:

Failure related statistics

Performance results comparison

Memory usage in terms of Proportional Set Size (PSS) for an application when comparing two different OEMs

Conclusion:

While this particular implementation was for Android applications, this idea can be extended to any platform or language.

With this way of collecting and collating data, we can have a benchmark for the application’s performance and this data can be used as a historical reference and also collect newer metrics with very few changes to the code.