Google Play expands its pre-launch testing capabilities
Discover how the free test crawl can now be guided to explore deep-links, fill in forms and even play games
Testing is one of the keys to achieving great quality in your app or game. However, it’s often one of the most time-consuming and frustrating aspects of development work. As a Product Manager at Google, I know how important it is for us to understand the challenges testing creates. So we’ve been working on new features and capabilities in both the Google Play Console and Firebase Test Lab — all with the goal of making testing our pain, and not yours. Basic app testing now requires mercifully little work on your part and more advanced testing will use less of your time, allowing you to focus on what you do best: delivering great app and game features.
In this post, I’m going to walk you through some new features in the Google Play Console pre-launch report and Firebase Test Lab. I will explain these features in the context of a fictitious app in the category of fitness tracking. In this app, users sign up, log in, create an avatar, invite buddies with whom to train, join a local gym, and set and track their goals.
Pre-launch report
The Google Play Console pre-launch report is designed to provide an early warning about quality problems for every app published to the pre-production tracks in the Google Play Console. The report is based on an automated crawl of your app on a diverse range of physical devices, using the tools and infrastructure of Firebase Test Lab. The test devices range from high-end devices such as the Google Pixel 2 to low-end devices such as the Samsung Galaxy J1. The testing is done by a crawler (which we call Robo) that explores your app by inputting taps, swipes and text as someone using your app would.
After your app has been crawled, the pre-launch report can tell you about a range of issues in your app such as crashes, security vulnerabilities and startup time. It also takes screenshots of each screen it crawls to help you spot UX issues.
To help you debug issues, you can replay the video of each crawl on each device on which it was run and see exactly what Robo did. When you review performance data such as network, memory and CPU metrics, the video is synchronized to the chart’s timeline to help you spot where exactly in your app a crash or drop in performance happened. For crashes, we also provide a stack trace. And you can see the full contents of the logcat for all runs.
Pre-launch report doesn’t require instrumentation of your app or SDKs, it is free and comes out-of-the-box for every app published to any of the pre-production tracks (which were formerly known as the alpha and beta test channels).
A new overview
A challenge many developers face when confronted with large volumes of app health and performance data is deciding how to prioritize the issues that are reported. To help with this, Google Play launched a new landing page that digests all issues found into a single launch recommendation.
In this report on the latest version of our fitness app, you can now see a simple and clear recommendation at the top of the page. In this case, crashes, performance and accessibility issues were found and the recommendation is to resolve errors and review warnings before releasing this version. Below the recommendation is a table that lists and prioritizes all the issues that were found in the pre-launch report.
Demo loops and monkey actions
The intelligent Robo crawler that powers pre-launch report and Firebase Test Lab does a great job of examining the Android framework part of the app. But, like many apps, our fitness app example uses OpenGL for features such as the creation of a fitness avatar. Previously the crawler couldn’t test apps and games that used OpenGL, but we’ve added several ways for you to now access these parts of your app.
To guide the crawler through your game, you can write one or more demo loops. Demo loops are automated journeys through your app that might focus on a game level or a game character. Demo loops are declared in your manifest and triggered at the start of the crawl but are not called during normal app use. Once the crawler has run the demo loop, you get the standard test results as well as information on rendering issues, allowing you to investigate any issues with frame rate.
However, if you don’t use demo loops, pre-launch report uses another feature called monkey actions. If the crawler detects that it has reached an OpenGL canvas in your app or game, it will start performing random taps and swipes. While the behavior of monkey actions may not be as sophisticated as that recorded in a demo loop, it can still detect issues you didn’t encounter in your own tests. Together, these features form a significant step towards meaningful automated game crawling.
Getting behind the sign in
The developers of our imaginary app care about keeping their users’ data safe and secure, so the app uses custom login or Google Sign-In. When the crawler sees a Sign-In with Google button, it uses a Google account included on the test device. If your app doesn’t use Google Sign-In, you can configure the pre-launch report with a test account username and password that the crawler will enter when it encounters the login screen.
Digging into deep links
Deep links are a way to invoke functionality within your app while bypassing the normal entry flow. For example, someone might search the web and follow a deep link from the search results to relevant content in your app.
Our fitness app uses deep links to encourage people to sign up to local gyms. In the case of this app, the process works by sending the user a Firebase notification alerting them to a special offer. They click the link in the notification and are taken straight to the gym sign up screen. This flow is very important because money changes hands and so should be tested in every release.
With the latest version of the pre-launch report, anyone can now test up to three deep links. The crawler works as usual for five minutes, then closes the app and visits each deep link in turn, crawling for a further 30 seconds. Any issues found during this extra crawl are included in the report as normal.
Extra control from Robo scripts
Sometimes crucial parts of an app are only reached through a complicated flow, such as our app’s gym sign up form. It turns out that automated crawlers (and even some humans) are not very good at entering the correct text to bypass this type of form.
Robo scripts guide the crawler to get it past complicated parts of an app. They can also be used to ensure the crawler follows a particular path through the app. Robo scripts take the form of a sequence of actions recorded as a JSON file. The scripts can be recorded using a tool built into Android Studio. With the app running locally, you record a series of taps, swipes and text input. The completed script is then uploaded to pre-launch report via the Settings tab. The script then guides the beginning of the crawl, carrying out the pre-recorded sequence to get the app into some state that you think will be interesting to test. Once it’s completed the Robo script, the crawler starts working normally.
Usability and accessibility
The pre-launch report now also uncovers accessibility and usability issues such as small touch targets, contrast, and voice-over annotations.
Small touch targets are a problem for many people using apps, including those using your app one-handed while on the go or those with dexterity impairments. Having good contrast to ensure legible text is also important to people generally, but particularly for those with sight impairments. Some of those who are visually impaired rely on TalkBack, so the report also highlights when TalkBack voiceover annotations are missing or confusing.
Pre-launch report as a launch decision tool
Developers have told us that they take the pre-launch report very seriously. In a recent survey, 86 percent said that they consider pre-launch report results to be important, or very important, when determining whether to launch a new app. When developers have ignored the report’s recommendation and launched, in the vast majority of cases the crashes found in the pre-launch report were later experienced by people using the app. Those people had a poor experience, and they may have left bad reviews or poor ratings, they may have chosen not to spend money in the app, or simply uninstalled it.
The pre-launch report is part of the standard Google Play Console features and available for free to everyone who publishes APKs to the testing tracks (previously alpha and beta). You can find it in the Google Play Console under ‘Manage Releases’.
Do more in Firebase Test Lab
The pre-launch report helps catch problems that could adversely affect people using your apps when you upload your apps for user testing to Google Play. However, you may want to catch more problems earlier in the development cycle, because issues found early are easier and cheaper to fix. Doing this as quickly as possible is where Firebase Test Lab comes in.
If you sign up for Firebase Test Lab you have access to even more powerful tools, an even more extensive range of devices and the ability to write instrumentation tests. So, for example, you can integrate with several development tools such as Android Studio, Fastlane, Google Cloud SDK, and of course the Firebase Console. These tool integrations enable you to test your apps and games at all stages of the application lifecycle with a broader array of devices, giving you more comprehensive results.
But, as we all know, creating and maintaining tests can be hard. It’s because of this that we’ve looked at ways of improving how automated tests are specified.
OpenGL testing
In our fitness app, users create an avatar. This involves uploading a photo from which a caricature is created. Then, when they choose a focused activity, such as weight training, the avatar’s persona is updated.
People using the app then guide their avatar around the gym activities as they work on their fitness.
The avatar feature in our example app is written in OpenGL, for a more immersive and fun experience for people. Previously it was hard to test apps and games that use OpenGL, but Firebase Test Lab has introduced a couple of ways in which you can now do this more easily.
The first is the previously mentioned demo loop capability, which was recently added to the Google Play Console pre-launch report. In Firebase Test Lab you can run the same demo loops on a wider range of devices than those used in the pre-launch report.
To further explore game testing Firebase has been collaborating with NetEase, one of the largest games development companies in China, to create tests for games that run in Firebase Test Lab. As part of this collaboration, NetEase open-sourced their test environment, AirTest, along with some of the frameworks they use for developing tests.
With these frameworks, you create tests that add actions and assertions that refer to game elements. These include actions such as touching a game element or waiting for a game element to become visible, or asserting that a game element is visible.
As with demo loops, these tests can be run locally, but they can also be run in Firebase Test Lab.
Instrumentation tests
Our app includes a goal creation screen where users create fitness goals and share them with all their fitness buddies. This part of the app is written using the Android framework. Typically, creating tests for this type of screen has involved laboriously crafting tests in one of the many testing frameworks, such as Espresso or UI Automator, and then keeping those tests updated as the UI evolves.
Espresso Test Recorder, part of Android Studio, simplifies this process by enabling you to quickly record Espresso tests by interacting with a device running your app locally. In addition to recording actions, you can add assertions against any screen elements. These tests are created in either Java or Kotlin.
So now, if your UI changes, you can go back and easily record new versions of the tests.
With the Espresso Test Recorder, you can modularize your testing. You can create test snippets for commonly used functions, such as signing in or filling out a search form. This way you can more easily create a battery of tests for your app.
The more devices, the better
Our app needs to run well on the full range of devices that people take to the gym. How do we make sure that the app runs well on all those devices? Pre-launch report offers free crawler testing on about 10 devices representing a wide range of capabilities and OS versions. You can go even further with Firebase Test Lab, using a broader range of devices, covering an even wider range of screen sizes, Android API versions, OEMs, and chipsets.
Testing locally to go global
Our fitness app is distributed globally and therefore needs to be tested for different languages. Both Firebase Test Lab and the Play Console enable you to do this. With Firebase Test Lab, you can run your tests on any locale you wish, while pre-launch report automatically runs for the most popular languages with those using your app.
Test results
As with the pre-launch report, Firebase Test Lab provides comprehensive test results, including crashes and ANRs, video and screenshots, logs, performance metrics (which are tied to the videos), and a map of the activities the crawler found.
The video and screenshots feature is especially important to help the design team verify that their app looks good and runs well across all those screen sizes, orientations, and locales. And in the performance tab in both products you can see what’s happening in your app during CPU spikes or frame rate drops.
Pre-launch reports are generated for free within the Play Console for all apps that are published to any of the test tracks. To get started for free on Firebase Test Lab, use the Firebase Spark billing tier.
Final word
It’s now easy to test comprehensively.
Pre-launch report helps you discover crashes, performance issues, security vulnerabilities and UX and accessibility issues. All with zero configuration required from you: publish an APK to the Google Play Console’s pre-production testing tracks and a report is generated. You can go further by providing custom login credentials, Robo scripts, deep links, and demo loops to test your app better.
If you want to test more often and more comprehensively, Firebase Test Lab adds the capability for you to create and run your tests on more devices, and run these tests from your continuous integration system. If you are interested in learning more about pre-launch testing check out the session from I/O 2018.
What do you think?
Do you have thoughts on pre-launch testing? Let us know in the comments below or tweet using #AskPlayDev and we’ll reply from @GooglePlayDev, where we regularly share news and tips on how to be successful on Google Play.
