Running an A/B Test in Your Android App With Firebase
An A/B test is a powerful tool for a team working on a mobile application. It allows you to split your users into various groups and give a different version of your app to each one simultaneously, without external factors impacting one version more than another. You can use it to compare the performances of the different versions based solely on how good they are at satisfying your users' needs.
In this article, I will take you through how to set up an A/B test in your Android app, using Firebase, and hopefully convince you to start an experiment if you’ve never done so before!
Why should you run an A/B Test?
Running an A/B test is a great way to understand the impact of each change you introduce in your app, and make sure they are improving your conversion rates and other key metrics. I believe you should try to run a test every time you’re introducing a change in the key features of your app.
For example, I’ve recently run two experiments in the app I’m working on, on two crucial parts of the app. The first was on the registration flow. We thought that modifying the order in which we asked information about our user could improve the number of users going through the whole funnel, but were hesitating between two different versions. We ran a test between those versions versus the previous, and we saw that both versions had a significant improvement on the previous version and that one of the two was performing a bit better than the other one, which allowed us to choose one.
The other experiment was on the most important part of our app, the key feature people use our app for. We thought we could improve the experience of users on that screen, and came up with an alternative solution with our UX team, about which we were quite excited. We ran the A/B test for a few weeks and discovered that the solution we thought of didn’t actually improve the performance at all. Thanks to this experiment, we avoided releasing a change that would not have had a positive impact on our users, and we decided to spend more time on the subject and come up with another solution, one that would actually improve the experience of our users.
Firebase tools: Remote Config and A/B tests
The Firebase platform lets you run A/B tests of different types. You can for example try different types of Notifications, which could help you find out which texts or icon makes users want to click on the push more. You can also test In-App Messages, and discover which message would drive more engagement.
The 3rd type of experiment is the one that interests us here: Remote Config. This tool enables us to control parts of the app directly from the Firebase console. It lets you define a series of parameters, and set a value for each in the Console. From the app, you then query the Firebase Remote Config SDK and retrieve the value of each parameter. Based on the value, you can adjust the corresponding feature in the app. For example, you can choose to show or hide a feature, show different versions of the same screen, modify the text of a message in the app, the color of a button, etc…
The way the A/B test tool from Firebase works with Remote Config is that it will split your users into different groups, and give a different value for the chosen parameter depending on which group the user belongs to. You’re then able to track the performance of each value in the Firebase Console and decide which one you want to keep. Once you have chosen, you can put a stop to the test, and roll out the chosen variant to all your users by setting the winning value in the parameter in Remote Config. This means that all your users will immediately get the best performing variant, without needing to update their app.
Before we take a closer look at the implementation, it’s important to note that there are many other tools that let you perform A/B tests on Android applications. Firebase is a good choice as it’s free, easy to implement, and you most likely will already be using some other of its tools (like the Analytics, Crashlytics, etc…). I did however use another analytics tool to perform deeper analysis on the performance of the A/B test, but I’ll get back to this later.
Setting up Remote Config in the app.
In case Firebase isn’t set up in your project at all, you’ll want to follow this guide, which will take through the steps of creating your Firebase project, registering your app, and adding your Firebase Configuration file.
Once Firebase is set up, you’re ready to use Remote Config.
First, you’ll want to add the dependency:
implementation ‘com.google.firebase:firebase-config-ktx:19.2.0’ implementation ‘com.google.firebase:firebase-analytics-ktx:17.4.4’The analytics dependency is needed in order to be able to use it when analyzing the results of your test. In addition, you may want to use some analytics event as a trigger for your A/B test. Using the ktx library will allow you to use the Kotlin extensions of the Firebase library.
You will need to set up in-app default values. Those values will be used in case your app can’t connect to the Firebase backend, or if the parameter you’re querying is not set in the backend. For this, you’ll need to create an XML file. Go to your res folder, and create an xml folder if you do not have one already. Create a new XML file in this folder (named remote_config_values for example). Here’s what the content of the file should look like:
You will pass this file to your RemoteConfig object with the method setDefaultAsync. You’ll use Firebase.remoteConfig to retrieve your Remote Config instance. This instance will be used for configuring Remote Config in your app and fetch the values set in the Console.
The first thing you'll want to do is to set the minimal fetch interval, which is the minimum amount of time you want to wait before fetching new values from the backend. The default value is 12 hours, which means that the next time you'll try to fetch new values, the same values will be returned if the previous time was less than 12 hours ago. Otherwise, the latest values in the console will be retrieved.
There is a server-side quota on how many queries your app can do per hour. The Firebase team recommends against choosing a small interval value, as this could potentially make your app hit the quota. However, when developing your app, you probably won't want to wait for hours before being able to change your values and test them. Therefore, when working on a development build, you can choose a much lower value, 10 seconds for example. In my app, I have a dev flavor and a prod flavor, depending on who I'm building the app for. I'm using this flavor to decide which minimal fetch interval to set.
Here’s what we discussed so far will look like in your app:
Here, I’ve created an interface called RemoteConfigManager, which will contain the method used to retrieve the value from the backend. I’m implementing this interface in RemoteConfigManagerImpl (with no method so far), which is class I’ll be accessing in my app whenever I want to get a value from the backend.
With our RemoteConfig instance configured, we can now get a parameter from the backend! If you do not yet have parameters in your Firebase Console, head over to the Console and to Remote Config. You’ll be asked to add a new parameter, with a key and a value. In my example above, the key was my_parameter_key. In the default config file, I’ve put the value default_value, but now I want to put a real one to check we properly get the value from the backend. Let’s put remote_config_value. You can also add a description, which is a good way to keep track of the different parameters you’ll add.
When you’re done adding values here, you need to click on Publish Changes in order to validate everything and make them accessible in your app.
Let’s go back to the code now, and try to get the value in the app.
You’ll need to fetch and activate the values with the Remote Config object. You can use the method fetchAndActivate() which will asynchronously perform those operations and returns a Task with the result, to which you can add listeners, and finally retrieve your value.
Because this is done asynchronously, you may want to take a moment and think of the best user experience to give your users. For example, you could show a loading screen while the value is fetched if the value drastically has an impact on the way your UI looks.
Setting up the A/B test in the Firebase Console
Now that you have Remote Config enabled in your app, and you know how to retrieve values, you can start an experiment on Firebase. Head over to A/B Testing in the Console, and click on Create an experiment. Select a Remote Config experiment.
Enter the details of your test (name, etc…), and move to the Targeting section, where you have the choice to add an Activation event. If you set up one, it means that only the users that have performed this event will enter your experiment. All the other users will not be taken into account in your results.
Here’s an example of why this can be important: Imagine you’re running a test on a social media app that has a registration and then features like post a photo. You want to improve the registration with an experiment. The main metric you will look at could be the increase in the percentage of new users completing the registration, but you also want to have a “big picture” metric like the percentage of users posting a photo.
If you do not use an activation event such as “start registration”, all of your users, including those that have registered in the past, will be in your test, and their photo posts will be in your results. But, what you really want, is only the posts from the new users completing the registration now, as they are the ones that will be affected by your experiment. I encourage you to take a moment and think about the experiment you have in mind, and whether you should add an activation event.
Next, in the Goals section, you’ll be asked for a success metric, such as performing another event, or a retention improvement… As I was mentioning at the beginning of the article, I personally think that the analysis part of an A/B test on Firebase has limitations, and therefore I’m performing deeper analysis with another Analytics tool, which is Amplitude. It’s a very powerful tool, that offers a free plan up to a certain amount of monthly events, so I encourage you to check it out. In my example of the registration flow experiment, one thing I would like to know for example (and that Firebase can’t tell me) is the completion of each step during the experiment.
However, if you have simple needs in terms of analytics (ex: clicking on a button or not, buying something or not, an amount of money spent, a retention rate…) then you could only be using Firebase.
Finally, in the Variants section, you get to link your experiment to the Remote Config that you set up earlier. You should be seeing the parameter you have created before, but you can also create a new one then. You can now define several variants of your parameter and split your users into a group for each.
The Control group should be the same version of the app you currently have. You could name the value control for example. Then you can create a new Variant for each version you want to test.
You can now hit Review and create your Draft of an experiment. Once it is in draft, you can check that your experiment is behaving as expected using the test device feature, which can force a specific variant to your device. You can read more about this here:
Back in your app, you can now implement all your different variants, and display them to your users based on the value returned by Remote Config.
To conclude…
I hope that this article will have helped you get started running A/B Test in your Android app. Regardless of the size of your team, it is worth spending time and resources on an A/B Test culture as it is a very valuable tool to use in order to understand better how your users behave in your app, and how to improve their experience to the best it can be.
I’d love to hear your thoughts, comments, or suggestions in the comment section! What type of experiment have you been running?
Thanks for reading, happy coding!
