How we use Firebase Audiences to A/B test the onboarding experience in our iOS apps

We know there is a subset of our users who do not use our apps to their full potential. The big question for us was why?

We worked with our User Research department to set up a number of one-to-one sessions that formed a qualitative study into how our users use the apps, and most importantly navigate around them. One of the key takeaways from these sessions was that our users were simply not aware that some features existed. Once the idea of the feature was mentioned, they were instantly able to use it effectively. We needed to improve the way we educate.

After some deliberation, we decided to test the effect of a number of onboarding screens. These are tutorials that highlight some really useful functionality within the apps.

Onboarding tutorials

So far so good?

Onboarding tutorials in Dark Mode

When we release new features, we love to A/B test their effectiveness, and if the onboarding tutorials work, they could be a game changer. The problem is that we test using Firebase’s AB Testing solution and Remote Config, which requires an internet connection to enrol a user in a test.

One of the things we love about the new onboarding experience is that it is just that, a beautiful experience to welcome a new user to the app, not a delayed experience dependent on internet connection. This would also mean that any user who opened the app without an internet connection (they do exist) would be instantly removed from our test groups.

So… What next?

A few different plans were put forward, but one really stood out. An on-device enrolment A/B testing framework that uses Firebase’s Audiences to provide results.

Audiences in Firebase

Firebase defines their Audiences as:

“Audiences let you segment your users in the ways that are important to your business. You can segment by event (e.g., session_start or level_up) and by user property (e.g., Age, Gender, Language), and combine events, parameters, and properties to include practically any subset of users.”

Sounds perfect and the concept is simple, but as with everything the devil is in the detail.

First we set up an analytics event to be sent to firebase called apps_enrolled_ab_test with two parameters: test (e.g. onboarding) and variant (e.g. A or B).

If it is the first time the app has been opened, we check that the user is eligible to enrol in the test, select a variant, store it locally and send the analytics event to Firebase, effectively adding the user to the appropriate Audience. Depending on the variant, we either do or do not display the onboarding screens.

Setting up an Audience in Firebase

Once a user is in an Audience, we can then filter our usual Firebase events and compare the impact on page views and interaction with features that are a part of the onboarding process.

Filtering events by audience in Firebase

The result is a straightforward way to A/B test functionality without relying on the user’s internet connection (or lack thereof).

There are some downsides to this approach that need to be kept in mind when analysing the results:

• Enrolment in either audience is based on random choice, which means the amount of users in a particular variant could be skewed significantly in either direction.
• There is also no knowledge of user demographic when sorting users, which could lead to an inflated percentage of users from a particular demographic in a variant, again, skewing results.
• Finally, we have no remote control over the test, the biggest downside here being that we would need to release a new version of the application to stop the test if there are any issues.

As with any test or experiment, as long as these potential pitfalls are kept in mind, we can ensure we end up with the correct analysis.