Get Started with AB Testing and Remote Configuration in Firebase
I have been wanting to try A/B Testing since I first heard of it many years ago. It just seems such a logical thing to do, to try two things side-by-side, compare them and see which is best.
But getting started is hard. Too hard. I think the tooling and know-how haven’t been easy to find.
What is A/B testing? — A Step by Step Introduction (Pt. 1)
Optimizely defines A/B testing (also known as split testing or bucket testing) as a method of comparing two versions of…
I have found that Firebase solves the tooling part well, and in this post, I will try to help with some know-how.
Remote Config is not just for AB testing
First, I want to advertise Firebase and Remote Config a little (note: I am not affiliated at all).
I want to show that even if your team isn’t experienced enough to do A/B testing, Remote Config can still be very valuable.
If you are working on a small project or on your own, by all means, you probably do not need this. As your system grows though and more configuration is added, this becomes an invaluable tool.
Remote Config allows you to define settings server-side and pull them down to clients. This allows you to release with the capability of enabling or changing functionality or content afterwards. What makes Remote Config especially powerful is that these settings can be segmented.
The segmentation is natively connected with Google Analytics and allows you to give different settings over different audiences, such as in a specific country, region, age group and much more.
This can be used to change things such as the number of ads to display, changing links to information pages or the email addresses.
Of course, not all of these need to change, perhaps ever, in which case, keep them hardcoded. I have found, however, that values that are defined outside the app, specifically URLs to documentation or URLs for banners, are good candidates for being configured remotely.
In our team, we control most documentation URLs, service maintenance messages, and even part of our top page UI data set as part of Firebase Remote Config.
As a team member put it today when asked:
It’s good for being able to change common data fast from the backend, such as lists and curated data.
Remote Config can also be used for feature flagging — i.e., enabling or disabling functionality based on configuration.
Introducing feature flags to Ruby on Rails - Greycastle
During a meeting a while back I suggested using feature flags to help to release a critical change to our payment…
This is super useful when you are building some functionality that has a significant impact. Why? Because you can continuously commit, merge and publish without actually releasing the in-progress work!
I have written about keeping pull request size down before, which reduces effort in testing (as it can be done in small batches) and helps developing incrementally.
What usually happens when we don’t feature flag is that we end up having large branches that get out of sync, causing merge nightmares.
A merge a day keep the conflicts away! Adding in a feature flag can be as simple as this, a tiny snippet from my latest project:
Of course, you can make this even neater with factories and dependency injection. Read on to see an example of this.
Why use Firebase Remote Config?
I admit, when I saw Remote Config the first time, I thought “I can build that myself”.
However, it took me about 2–4h working out how to use Firebase Remote Config that was already there, documented and tested.
I promise, writing, hosting and running a configuration service on your own, will take at least 10x that time.
Setting up Remote Config and AB testing
Working with Remote Config in our team, I found that it is important to have a way to be able to test with it.
The big drawback with the Firebase Remote Config in its raw form is that it isn’t very easy to test on a single device. There are a few ways you can do this:
- You can control your property using a Google Analytics audience, but this takes hours if not days to update
- You can use a user property, this is the solution I am using
- You can get the device identifier and use this with an AB test. This is effective but, it requires a device identifier which can be hard to get unless you are already using Cloud Messaging
To toggle my Remote Config, I’m using a user property which is easy to use for conditions.
I added a hidden functionality in my app where a user can tap to toggle an “experimental mode” user property. This works beautifully, it is easy to debug in the Firebase debug event view and works great with the Firebase configuration condition.
So to break this down, I will show:
- Creating the user property
- Switching the user property
- Reading the Remote Config (and propagate it immediately)
- Configuring Remote Config and A/B tests
Using a user property for testing configuration
Firebase and Google Analytics comes with something called user properties. With Audiences in Firebase you can segment users based on events for example but user properties allow us to set long term properties to specific users specifically. They also take effect much faster than events.
As an example, a user that has made a purchase can be added to the
buyers audience for the next thirty days. User properties instead are useful for properties that seldom change, such as
This is not mentioned in the documentation for user properties, but it is important you make sure to define the user property before it can be used.
It takes a few hours for it to pop up after having been used the first time as well. You can add the property under
custom definitions in the Firebase console.
Switching the user property
After a user property has been added we can start configuring it for users.
It is easy to do so using the Google Analytics library for Firebase.
However, I will complicate this a little. I wanted a way to toggle the “test settings” off and on, in any specific device. This way I can test the new functionality by entering or leaving this test mode.
I allow the user to switch this setting inside the app by tapping in a secret location (if you download my app, knock yourself out to my experimental widgets 😀)
The code for the widget controlling the tapping is shown below. It is only a small wrapper to handle the tapping and countdown:
The service shown below hides away the setting of the user property and the saving of the state between app sessions:
Loading the remote configuration
You will want to read the remote configuration as your application starts up. Worst case, your user will use your default values. You should try to set these for all configuration settings as a fallback and keep them relatively up-to-date for new releases.
It can be tricky with Remote Config throttling. Throttling means that even if fetching the settings, the library might quietly just feed you the cached values. It took me some time to figure this out.
await remoteConfig.fetchAndActivate();I/flutter (17598): Current time: 2021-08-15 17:54:41.459275
I/flutter (17598): Remote config last fetch 2021-08-15 17:36:51.968 with status RemoteConfigFetchStatus.success
Given the classes I shared above, I got around the caching problem by reducing the cache time to zero.
In the Flutter SDK for remote config, the naming is a little different from the native libraries. I found it a bit hard to understand what each property in the settings does but finally, found that the
minimumFetchInterval is the setting to modify to “force” settings to be read.
My code for loading the configuration looks like this:
Conditional remote configuration and A/B testing
That’s a mouthful of a header, but it’s easier than it sounds!
Now that you have your Remote Config flag in the code, kicking off the A/B test is as simple as creating it, tweaking your segment and starting it!
Though it depends on the size of your user base, you should let your test run around two weeks if not longer to get some solid data out. You can measure success based on a number of different events and in the end, the result will look something like this:
If you only need to conditionally set a flag for testing, as above, with the experimental flag, you do that by adding a conditional group in the Remote Config console:
Without using feature flags it is very easy to end up doing waterfall delivery. Big features blocking releases as they cannot be realised until completed.
Using the feature flags we are able to roll out functionality internally before releasing publically and testing on production data. This is incredibly useful as there is no test better than using the real thing.
Hope this helps and let me know if I can expand on anything else!