Geek Culture
Published in

Geek Culture

A/B Testing In Android

As the name suggests either A or B. A/B testing is a way to improve your app by testing a feature on a subset of users.

Photo by Luke Chesser on Unsplash

What is it?

Let’s understand it with an example. Suppose you are working on an app that needs some changes on the dashboard. Your old dashboard has a very old UI, you want to change it with some material design and some really cool animation. Now you don’t wanna take the risk to change everything without knowing users' feedback on the new dashboard. For that what we can do is, we can show a subset of users our new dashboard and to the rest, the old dashboard. Once you get some good feedback on the new dashboard, you can switch all your user to the new dashboard. And if your new dashboard is not very impactful for users, you can improve it without affecting a lot of users.

Key capabilities

  • Test and improve your product experience
  • Find ways to re-engage your users by using the Notifications composer
  • Safely roll out new features
  • Target “predicted” user groups

How to do it?

Select a suitable A/B testing platform, here we are gonna use Firebase Remote Config.

Firebase Remote Config

Firebase Remote config is a cloud service that lets you change your app behavior without reinstalling the build. When using remote config you create a variable with default value in your app which maintains the behavior and appearance of your app. Now when you want to change this behavior, you can override the default value in-app with remote config and your app will behave as you want. This way you can update your app anytime with a negligible impact on performance.

Key capabilities

  • Quick roll-out
  • You can use Remote Config to provide variations on your app’s user experience to different segments of your user base by app version
  • A/B testing

How to Integrate Remote Config?

Step 1: Create a project on firebase console.

Step 2: Download google-services.json file in your app folder.

Step 3: Add some dependencies in app/build.gradle file

implementation platform('')
implementation ''
implementation ''

Step 4: Add google service dependency in project level build.gradle file

classpath ''

Step 5: Get the Remote Config singleton object

remoteConfig = Firebase.remoteConfig
val configSettings = remoteConfigSettings {
minimumFetchIntervalInSeconds = 3600

Step 6: In app default value for remote config object
You can set in-app default parameter values in the Remote Config object, so that your app behaves as intended before it connects to the Remote Config backend, and so that default values are available if none are set in the backend.


Step 7 : Set parameter values in the Remote Config backend

To add remote config object in firebase server, open firebase console, in left you will see remote config under engage , click on that and you will see a page like above. Click on add parameter button here. You will see a screen like below.

Add parameter key same as what you added in default config XML file. Click on add parameter button and publish changes.

Step 8 : Fetch and activate values

Wohoooo it’s all done, Great job guys , you made it. But it’s not done yet. Now let’s talk about how we do A/B testing.

Create Firebase Remote Config Experiments with A/B Testing

Step 1: Create an experiment

Select A/B testing tab from left on firebase console. Now click on create experiment button. You will see a popup like below, click on remote config.

You will see a window like below with 4 sections- Basic, Targeting, Goals, Variants.

  • In Basics add your experiment name and description.
  • In Target you can choose your audience for this experiment like for which country user you want to show it, then for new build or old build, for which language etc. Then you can select percentage of user whom you want to show your experiment.
  • From the dropdown list select a built-in objectives (engagement, purchases, revenue, retention, etc.), Analytics conversion events, and other Analytics events. When finished, click Next.
  • In the Variants section you’ll choose a control group and at least one variant for the experiment. Use the Choose or create new list to add one or more parameters to experiment with. You can create a parameter that has not previously been used in the Firebase console, but it must exist in your app for it to have any effect. You can repeat this step to add multiple parameters to your experiment.
    To add more than one variant to your experiment, click Add another variant.

Click Review to save your experiment.

Step 2: Validate your experiment on a test device
For each Firebase installation you can retrieve the installation auth token associated with it. You can use this token to test specific experiment variants on a test device with your app installed.

  • Get the installation auth token as follow:
FirebaseInstallations.getInstance().getToken(/* forceRefresh */ true)
.addOnCompleteListener { task ->
if (task.isSuccessful) {
Log.d("Installations", "Installation auth token: " + task.result?.token)
} else {
Log.e("Installations", "Unable to get Installation auth token")
  • Go in A/B testing tab from console , go on drafts and click on 3 dot menu and click on manage test devices.
  • Enter auth id, which you received above.
  • Run the app and confirm that the selected variant is being received on the test device.

You can read more about it from here:

At Healthifyme we use A/B testing for almost every feature. Recently we changed our dashboard.

New Dashboard , Old dashboard

Now because we were doing a big change, we should be able to revert it back if it doesn’t go good, we wants to know our users feedback. So we rolled it out for almost 25k users. And below were the outcomes:


  • First food track — 46% in the test set against 41% for the control
  • Water track after onboarding — 39% in the test set against 34% for the control
  • Connecting a steps tracker after onboarding — 9.5% against 6%


  • App launch D3 retention : 31.60% vs 26%
  • D3 retention on trackers: 32% vs 27.7%; D7 retention: 24% vs 18% for control


  • Average tracking events per user: 14.5 in the test set against 12.5 for the control

Which was a good win for our experiment, and we can rollout it to more users.

It’s not necessary we should be using these for changes like revenue or something which makes a big impact.We can do it for anything as small as a text change too and know their impact independently.

Sometimes it can happen when you don’t see a big impact of a experiment on your app and on your users, but it’s a path for your learning. Something is not always nothing.


Hope this will help you in your next development.

If you have any feedback please write me back at claps are really appreciated to help others find this article 😃 .



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store