Testing Your A/B Tests: How to QA Your Experiments With Firebase on Android
Firebase is a platform providing various tools that can be really useful to improve and grow your mobile app. One of those tools is their A/B testing tool. With it, you can quickly set up and run A/B tests in your app in order to test different versions of your app and its features, to eventually improve it and provide a better experience to your users.
In this article, I will explain how you can quickly test your A/B test, and make sure you go to production with the set-up you want. Firebase lets you run different experiments (Notifications, Remote Config, In-app messaging). This article takes a Remote Config experiment as an example, but I imagine this is applicable for most types of experiments.
Before we get started, here’s the documentation for the A/B Testing feature of Firebase, which I recommend to have a look at if you need more information after having read this article.
Why testing an A/B test?
Testing every aspect of your application is essential to ensure you’re releasing a stable and performant application. It’s the way to make sure your application works as expected, features are implemented according to the specifications, no bugs have been introduced by accident, etc
Testing your A/B test follows the same logic. You want to ensure that your users are going to experience the app the way you expect them to, depending on which group they are in and which variant of the app they are seeing. If your test relies on some condition for your users to be assigned a variant, you want to check that a user sees that variant only when that condition has been met.
Depending on what you are testing, you have to be sure you can trust the data you’re gathering before making decisions because an error in your data can be costly. If you discover an error in your test set-up after having launched it, all your results will be invalid and you will have wasted precious time. Or even worse, you might not realize there is an error in the set-up, and end up making decisions based on invalid data, which could make you lose your users, cost you money, etc…
Thankfully, testing an A/B test is quite easy with Firebase. It essentially consists of forcing a variant of your test to a Firebase Instance ID, which you can do from the Firebase Console.
Getting your Firebase Instance ID
The Firebase Instance ID provides a unique identifier for each app instance and a mechanism to authenticate and authorize certain actions, such as sending FCM messages, or in our case deciding which variant to give you during an A/B test.
More information about the Instance ID can be found here.
In order to get your Instance ID in your app, you simply need to add the following code, which will retrieve the instance of the FirebaseInstanceId:
FirebaseInstanceId.getInstance().instanceId.addOnSuccessListener {
val instanceId = it.token
}In the app I’m working on, I’ve added a Dev Settings page in the app, only available if the build is not for production (so for dev or QA purposes). From that page, you can see your Instance ID, and I’ve added options to copy it to the clipboard or send it via an email.
I recommend to do something similar in your app, this can greatly help your QA team in retrieving the Instance ID if they have to QA your A/B tests.
Testing the test
The way to test your experiment will consist of forcing a variant to an Instance ID. This way you can check all of your variants one by one and test them all. Also, if you have conditions for your test, you can see if you only receive the forced variant once the condition has been met.
One important thing to know is that you can only test A/B tests that are in draft. Once an experiment has been started in Firebase, there is no way to force a variant for any specific users. So be sure to do your testing before starting the experiment.
Head over your Firebase Console, and look for A/B Testing in the Grow section.
Look for the experiment you wish to test in the Draft section. Once you expand the Details section, you will see a Test Devices field.
From there, click on Manage test devices. This will open a modal in which you can add one or several Instance ID(s). For each of them, you can then choose which variant of your experiment you wish to see. Click on save, and voilà. You should now see the selected variant in your app!
After this, you can go back to that modal and change the variants as many times as you want, while the experiment is in a draft. If you’re running an experiment with Remote Config though, the change in your app might not be immediate.
This depends on the minimumFetchIntervalSeconds you have set up in your app when configuring the Remote Config tool, which dictates how long the app should way before checking for new Remote Config values on Firebase. The default value is 12 hours. When developing the app, you might want to set this to just a few seconds, in order to be able to change from variant to variant without having to wait in between. Be careful not to release this in production though!
You can use the following method to set the minimum fetch interval:
FirebaseRemoteConfigSettings.Builder()
.setMinimumFetchIntervalInSeconds(minimumFetchIntervalSeconds)In my app, I’ve set the minimumFetchIntervalSeconds to 10 seconds when the build is for dev or for QA.
You can read more about this set up here.
Conclusion
I hope that this article will have provided you with some useful information on how to successfully QA your A/B tests with Firebase before launching an experiment. Just like every other part of your app, it’s very important to test your app and its features before releasing it to your users, to ensure you provide them with the best experience possible.
I’m planning on writing another article to go more in detail into how to set up an A/B test in your Android app, and why you would want to do this in the first place, so stay tuned!
Thanks for reading, happy coding!
