When Mobile App A/B Testing Gives Really Valuable Results

Everyone seems to be obsessed with A/B testing these days. We constantly see advice about testing different button colors, testing variations of shopping cart icon overlays, trying new calls to action on your landing page, and so on.

While random A/B testing may lead to improvements in your app’s user conversion rate or boost downloads — we once increased the downloads of our app MyDay a few times over by simply altering the order of screenshots, — in most cases random A/B testing won’t give you any tangible results. According to AppSumo, only one out of eight A/B tests actually deliver valuable results. In other cases, such tests often aren’t worth the effort.

So how can you run mobile app A/B testing and get results that can really help you to improve your app’s performance? That’s what we’ll explore in this article.

When and why does A/B testing fail?

A/B tests are often considered failures when they fail to deliver statistically significant results.

When results aren’t statistically significant, it means that they don’t lead you to any solid conclusions about which of the tested alternatives is best. It’s easy to produce insignificant results when you run tests randomly on some part of your product, making small design tweaks here and there according to no particular logic. Although some are of the opinion that such tests still may have some value, running A/B tests on literally everything is simply unproductive.

But that’s only one side of the coin.

Another unpleasant situation is when an A/B test shows results that are statistically significant, but are not what you expected. For example, let’s say you’ve invested lots of resources and made drastic changes to your product, only to discover that these “improvements” lose out to old “unimproved” variants in an A/B test. When this happens, it’s generally because there wasn’t enough data to prove the efficacy of such changes in the first place — no KPI analysis, no user research, no market research.

A/B testing will show you the results you seek only when the tests are meaningful, and targeted at those elements of your product that can potentially improve your app’s performance and help you reach your business goals.

How can we make A/B testing meaningful?

[Image source: TheNextWeb]

First, thoughtfully construct your A/B test

Mobile A/B testing generally consist of two parts:

1. Testing of in-app content (UI and UX, usability, onboarding experience, calls to action and so on). In-app A/B testing is meant to improve the overall user experience and engagement with the app.
2. Testing of app store pages, ads, and landing pages. App store and landing page A/B testing is focused on boosting the user acquisition rate.

As we’ve suggested, blindly running A/B tests for each small part of your product can be meaningless, but you can test really valuable improvements when you know what exactly you need to improve.

To design a valuable A/B test, follow these three steps:

1. Define your goals

Keeping in mind your app’s business objectives is crucial when defining key metrics to improve through A/B testing.

For the vast majority of mobile apps, the most important metric is user conversion rate. A “user conversion” is a specific action a user takes that gets you closer to your app’s monetization goals. You can define different user conversion events for different stages of interaction with your product.

Here are just a few examples of possible user conversion events:

1. Pre-install conversion:

• clicking on an ad
• signing up for a subscription
• going to the app’s app store page
• downloading the app from an app store

2. In-app conversion:

• registering
• accomplishing certain in-app actions that keep users engaged (like uploading a profile photo or sharing content)
• entering credit card information
• making an in-app purchase
• sending app invites

2. Define your KPIs and measure current performance

Key performance indicators (KPIs) are the metrics that reflect your app’s success. But should “success” be defined as a number of installs, time spent per use session, or something else? It all depends on your business objectives.

In the context of A/B testing, well-defined KPIs will show you where to focus your tests. Measure your KPIs using app analytics tools, find potential weak spots in your product, prioritize improvements, and then work to eradicate the weak spots with A/B testing insights.

Need more downloads? Try different variations of your app’s store page and your landing page. Do users tend to churn, or is the average in-app session time too low? Identify screens where users typically drop off and run A/B testing there.

Read more about KPIs and app analytics tools in our dedicated app analytics and metrics guide.

3. Remember that good UI and UX are the keys to success

If you’re just casually testing random variations of a landing page, shopping cart menu, or message screen, then your results are likely to be haphazard as well. Each variant you include in your A/B tests should be designed with your target users in mind, and should be based on extensive research and on principles of good design.

You can waste a lot of time trying to improve your product’s KPIs by changing something for the sake of change, — or you can expertly improve your KPIs with one or two thoughtful design tweaks inspired by meaningful data about your users.

When A/B testing is done right

Here are some real-life cases when a sensible A/B test delivered substantial results and helped to increase user conversion rate.

Runkeeper: Engaging users by reorganizing a start screen

Runkeeper is an activity tracking app for iOS and Android that launched back in 2008. In 2013 they decided to encourage more users to track activities other than running. They constructed an A/B test around the hypothesis that changing the start screen might encourage users to track their walking, cycling, and other activities.

[Source: Localytics]

Changing the start screen is risky, since it may turn away some users that are accustomed to the old version. However, Runkeeper’s new start screen worked: the icon-based menu spurred users to try tracking features for activities other than running. As a result, the number of logged non-running activities increased by 235 percent.

Škoda: Experimenting with app store screenshots

Škoda managed to increase their Škoda Little Driver app’s install rate by 50 percent by simply rearranging the order of their app store screenshots.

They based their hypothesis on research that posited that user attention span has been reduced to around eight seconds. This suggests that the average user doesn’t look through all screenshots or read long descriptions. Škoda developed three screenshot variations to test their efficacy against each other. For the first variation, they placed a screenshot that describes the main function of the app first in the image order. The second variation included an alternate first screenshot, and the third included an alternative background image depicting sky, fields, and road.

[Source: Bamboo Group]

The first variation showed the best results, increasing app installs by 50 percent. Although this result was pretty much expected, Škoda meade another valuable insight from their A/B testing: distracting backgrounds can have a negative effect on user conversion.

Fab: Looking for the right call to action

Fab, an online retail community, tested how a clearer “Add To Cart” button might affect the number of items that users add to the cart.

Their original design featured a small shopping cart accompanied by a “+” symbol and no text. Alternates used in the A/B test were buttons with “Add To Cart” and “+ Cart” text on them.

[Source: Wishpond]

The results showed that a clearer and more direct “Add To Cart” call-to-action increased clicks by 49%. That this seemingly trivial change can have such a large impact underscore the fact that our best common-sense judgments don’t always lead to the most efficient business outcomes.

Spreadshirt: The cleaner the better

Spreadshirt is an online retail company that lets people sell and buy custom T-Shirts and T-Shirt prints.

Spreadshirt wanted to increase the number of sellers on their site. To achieve this, they changed their infographic that illustrates how a person can become a seller on the marketplace.

Their original infographic was informative, but perhaps a bit confusing:

[Source: Wishpond]

Their alternate variant was rather minimalist, but clearly illustrated the process of becoming a seller:

[Source: Spreadshirt]

And the results? The number of users who clicked the “Start selling now” button increased by 606 percent!

Conclusion

The A/B tests we highlighted here delivered significant results because they were built on app analytics data and smart product design improvements.

First, the developers identified weak spots in their product that they wanted to improve. Second, they designed their test variants with the user in mind, attempting to make their product clearer, more intuitive, and more engaging.

The biggest A/B testing mistake is blindly running tests without sufficient research to back up your proposed improvements. When conducting mobile application A/B testing, you must understand what your users want, rely on sensible product design, and continue analyzing and testing to achieve truly meaningful results.

Originally published at yalantis.com.