Google Play expands its pre-launch testing capabilities

Discover how the free test crawl can now be guided to explore deep-links, fill in forms and even play games

Suzanne van Tienen
Jul 13, 2018 · 12 min read
Image for post
Image for post
Image for post
Image for post

Pre-launch report

The Google Play Console pre-launch report is designed to provide an early warning about quality problems for every app published to the pre-production tracks in the Google Play Console. The report is based on an automated crawl of your app on a diverse range of physical devices, using the tools and infrastructure of Firebase Test Lab. The test devices range from high-end devices such as the Google Pixel 2 to low-end devices such as the Samsung Galaxy J1. The testing is done by a crawler (which we call Robo) that explores your app by inputting taps, swipes and text as someone using your app would.

A new overview

A challenge many developers face when confronted with large volumes of app health and performance data is deciding how to prioritize the issues that are reported. To help with this, Google Play launched a new landing page that digests all issues found into a single launch recommendation.

Image for post
Image for post

Demo loops and monkey actions

The intelligent Robo crawler that powers pre-launch report and Firebase Test Lab does a great job of examining the Android framework part of the app. But, like many apps, our fitness app example uses OpenGL for features such as the creation of a fitness avatar. Previously the crawler couldn’t test apps and games that used OpenGL, but we’ve added several ways for you to now access these parts of your app.

Getting behind the sign in

The developers of our imaginary app care about keeping their users’ data safe and secure, so the app uses custom login or Google Sign-In. When the crawler sees a Sign-In with Google button, it uses a Google account included on the test device. If your app doesn’t use Google Sign-In, you can configure the pre-launch report with a test account username and password that the crawler will enter when it encounters the login screen.

Digging into deep links

Deep links are a way to invoke functionality within your app while bypassing the normal entry flow. For example, someone might search the web and follow a deep link from the search results to relevant content in your app.

Extra control from Robo scripts

Sometimes crucial parts of an app are only reached through a complicated flow, such as our app’s gym sign up form. It turns out that automated crawlers (and even some humans) are not very good at entering the correct text to bypass this type of form.

Image for post
Image for post

Usability and accessibility

The pre-launch report now also uncovers accessibility and usability issues such as small touch targets, contrast, and voice-over annotations.

Image for post
Image for post

Pre-launch report as a launch decision tool

Developers have told us that they take the pre-launch report very seriously. In a recent survey, 86 percent said that they consider pre-launch report results to be important, or very important, when determining whether to launch a new app. When developers have ignored the report’s recommendation and launched, in the vast majority of cases the crashes found in the pre-launch report were later experienced by people using the app. Those people had a poor experience, and they may have left bad reviews or poor ratings, they may have chosen not to spend money in the app, or simply uninstalled it.

Do more in Firebase Test Lab

The pre-launch report helps catch problems that could adversely affect people using your apps when you upload your apps for user testing to Google Play. However, you may want to catch more problems earlier in the development cycle, because issues found early are easier and cheaper to fix. Doing this as quickly as possible is where Firebase Test Lab comes in.

OpenGL testing

In our fitness app, users create an avatar. This involves uploading a photo from which a caricature is created. Then, when they choose a focused activity, such as weight training, the avatar’s persona is updated.

Image for post
Image for post
Image for post
Image for post

Instrumentation tests

Our app includes a goal creation screen where users create fitness goals and share them with all their fitness buddies. This part of the app is written using the Android framework. Typically, creating tests for this type of screen has involved laboriously crafting tests in one of the many testing frameworks, such as Espresso or UI Automator, and then keeping those tests updated as the UI evolves.

Image for post
Image for post

The more devices, the better

Our app needs to run well on the full range of devices that people take to the gym. How do we make sure that the app runs well on all those devices? Pre-launch report offers free crawler testing on about 10 devices representing a wide range of capabilities and OS versions. You can go even further with Firebase Test Lab, using a broader range of devices, covering an even wider range of screen sizes, Android API versions, OEMs, and chipsets.

Image for post
Image for post

Testing locally to go global

Our fitness app is distributed globally and therefore needs to be tested for different languages. Both Firebase Test Lab and the Play Console enable you to do this. With Firebase Test Lab, you can run your tests on any locale you wish, while pre-launch report automatically runs for the most popular languages with those using your app.

Image for post
Image for post

Test results

As with the pre-launch report, Firebase Test Lab provides comprehensive test results, including crashes and ANRs, video and screenshots, logs, performance metrics (which are tied to the videos), and a map of the activities the crawler found.

Image for post
Image for post

Final word

It’s now easy to test comprehensively.


What do you think?

Do you have thoughts on pre-launch testing? Let us know in the comments below or tweet using #AskPlayDev and we’ll reply from @GooglePlayDev, where we regularly share news and tips on how to be successful on Google Play.

Google Play Apps & Games

Tips, trends, and industry thoughts for app and game…

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store