Get Started with AB Testing and Remote Configuration in Firebase

Stick figure holding up a green and red flag with an A and a B written on them
Feature flags

I have been wanting to try A/B Testing since I first heard of it many years ago. It just seems such a logical thing to do, to try two things side-by-side, compare them and see which is best.

But getting started is hard. Too hard. I think the tooling and know-how haven’t been easy to find.

I have found that Firebase solves the tooling part well, and in this post, I will try to help with some know-how.

Remote Config is not just for AB testing

First, I want to advertise Firebase and Remote Config a little (note: I am not affiliated at all).

I want to show that even if your team isn’t experienced enough to do A/B testing, Remote Config can still be very valuable.

If you are working on a small project or on your own, by all means, you probably do not need this. As your system grows though and more configuration is added, this becomes an invaluable tool.

Remote Config allows you to define settings server-side and pull them down to clients. This allows you to release with the capability of enabling or changing functionality or content afterwards. What makes Remote Config especially powerful is that these settings can be segmented.

The segmentation is natively connected with Google Analytics and allows you to give different settings over different audiences, such as in a specific country, region, age group and much more.

This can be used to change things such as the number of ads to display, changing links to information pages or the email addresses.

Of course, not all of these need to change, perhaps ever, in which case, keep them hardcoded. I have found, however, that values that are defined outside the app, specifically URLs to documentation or URLs for banners, are good candidates for being configured remotely.

In our team, we control most documentation URLs, service maintenance messages, and even part of our top page UI data set as part of Firebase Remote Config.

As a team member put it today when asked:

It’s good for being able to change common data fast from the backend, such as lists and curated data.

Feature flagging

Remote Config can also be used for feature flagging — i.e., enabling or disabling functionality based on configuration.

This is super useful when you are building some functionality that has a significant impact. Why? Because you can continuously commit, merge and publish without actually releasing the in-progress work!

I have written about keeping pull request size down before, which reduces effort in testing (as it can be done in small batches) and helps developing incrementally.

What usually happens when we don’t feature flag is that we end up having large branches that get out of sync, causing merge nightmares.

A merge a day keep the conflicts away! Adding in a feature flag can be as simple as this, a tiny snippet from my latest project:

Example of how to use a release flag for a page implementation in Flutter

Of course, you can make this even neater with factories and dependency injection. Read on to see an example of this.

Why use Firebase Remote Config?

I admit, when I saw Remote Config the first time, I thought “I can build that myself”.

However, it took me about 2–4h working out how to use Firebase Remote Config that was already there, documented and tested.

I promise, writing, hosting and running a configuration service on your own, will take at least 10x that time.

Setting up Remote Config and AB testing

Working with Remote Config in our team, I found that it is important to have a way to be able to test with it.

The big drawback with the Firebase Remote Config in its raw form is that it isn’t very easy to test on a single device. There are a few ways you can do this:

  1. You can control your property using a Google Analytics audience, but this takes hours if not days to update
  2. You can use a user property, this is the solution I am using
  3. You can get the device identifier and use this with an AB test. This is effective but, it requires a device identifier which can be hard to get unless you are already using Cloud Messaging
Graph showing different symbols of user properties, events and devices going into remote config and coming out as different configurations
Remote Config can be controlled by a number of methods, user properties, events, device ids and more

To toggle my Remote Config, I’m using a user property which is easy to use for conditions.

I added a hidden functionality in my app where a user can tap to toggle an “experimental mode” user property. This works beautifully, it is easy to debug in the Firebase debug event view and works great with the Firebase configuration condition.

So to break this down, I will show:

  1. Creating the user property
  2. Switching the user property
  3. Reading the Remote Config (and propagate it immediately)
  4. Configuring Remote Config and A/B tests

Using a user property for testing configuration

Firebase and Google Analytics comes with something called user properties. With Audiences in Firebase you can segment users based on events for example but user properties allow us to set long term properties to specific users specifically. They also take effect much faster than events.

As an example, a user that has made a purchase can be added to the buyers audience for the next thirty days. User properties instead are useful for properties that seldom change, such as age group.

This is not mentioned in the documentation for user properties, but it is important you make sure to define the user property before it can be used.

It takes a few hours for it to pop up after having been used the first time as well. You can add the property under custom definitions in the Firebase console.

Add the user property to custom definitions before use

Switching the user property

After a user property has been added we can start configuring it for users.

It is easy to do so using the Google Analytics library for Firebase.

However, I will complicate this a little. I wanted a way to toggle the “test settings” off and on, in any specific device. This way I can test the new functionality by entering or leaving this test mode.

I allow the user to switch this setting inside the app by tapping in a secret location (if you download my app, knock yourself out to my experimental widgets 😀)

The code for the widget controlling the tapping is shown below. It is only a small wrapper to handle the tapping and countdown:

Widget to toggle the experimental mode by tapping

The service shown below hides away the setting of the user property and the saving of the state between app sessions:

Service to help abstract the persistent storage of experimental mode and integration with Firebase through Google Analytics

Loading the remote configuration

You will want to read the remote configuration as your application starts up. Worst case, your user will use your default values. You should try to set these for all configuration settings as a fallback and keep them relatively up-to-date for new releases.

It can be tricky with Remote Config throttling. Throttling means that even if fetching the settings, the library might quietly just feed you the cached values. It took me some time to figure this out.

await remoteConfig.fetchAndActivate();I/flutter (17598): Current time: 2021-08-15 17:54:41.459275
I/flutter (17598): Remote config last fetch 2021-08-15 17:36:51.968 with status RemoteConfigFetchStatus.success

Given the classes I shared above, I got around the caching problem by reducing the cache time to zero.

In the Flutter SDK for remote config, the naming is a little different from the native libraries. I found it a bit hard to understand what each property in the settings does but finally, found that the minimumFetchInterval is the setting to modify to “force” settings to be read.

My code for loading the configuration looks like this:

Example of forcing the refresh of remote config settings if the cache is stale after switching experimental mode

Conditional remote configuration and A/B testing

That’s a mouthful of a header, but it’s easier than it sounds!

Now that you have your Remote Config flag in the code, kicking off the A/B test is as simple as creating it, tweaking your segment and starting it!

Adding a new AB test in the Firebase console

Though it depends on the size of your user base, you should let your test run around two weeks if not longer to get some solid data out. You can measure success based on a number of different events and in the end, the result will look something like this:

Example of an AB test after running for a while

If you only need to conditionally set a flag for testing, as above, with the experimental flag, you do that by adding a conditional group in the Remote Config console:

Adding a conditional group for the experimental mode
Using the conditional group to feature flag the functionality

Conclusion

I have only just started to use this in my own development of JReader and in our team building FACY but I feel it’s already helping us to bias towards iterative development.

Without using feature flags it is very easy to end up doing waterfall delivery. Big features blocking releases as they cannot be realised until completed.

Using the feature flags we are able to roll out functionality internally before releasing publically and testing on production data. This is incredibly useful as there is no test better than using the real thing.

Hope this helps and let me know if I can expand on anything else!

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
David Dikman

David Dikman

Active reader, developer, manager, entrepreneur & sporadic writer here and at https://greycastle.se. Currently working at https://styler.link.