Android A/B testing Made Simple With Cadabra Library

Dmitry Si
Dmitry Si
Dec 1, 2020 · 6 min read
Image for post
Image for post
Photo from by Brian Wangenheim

A/B testing is not easy: designing an experiment and analyzing data properly is a massive task, and technical implementation adds even more complexity to the mix. In this article, I will show how to streamline that last part of the A/B testing process on Android to focus on what matters the most — your data.

A/B testing complexity

Let’s start with the complexity associated with A/B testing before I offer the solution. So, what are the problems?

Solution requirements

If we were given enough time to design a complete solution to the aforementioned problems how would it look like? Here is my proposed library requirements. (Scroll to the last paragraph to get straight to the code samples)

Solution

With all these requirements in mind, I’ve created a library let’s see how that works (links to the library repo and maven below)!

Here is an example of the minimal A/B test configuration, for a simple test.

Image for post
Image for post
Both layouts and content can be changed by Cadabra

(I’ll cover more use-cases in a separate article, but the repo already contains the sample project, and the documentation describes the solutions for the most common cases)

A simple experiment with automatic resources resolution

All we need to do is

enum class AutoResourceExperiment : Variant {STRANGE, CHARM}
CadabraAndroid.initialize(this)
CadabraAndroid.config
.startExperiment(
AutoResourceExperiment::class,
RandomResolver(AutoResourceExperiment::class))

With auto-resources resolving layouts and strings that have suffixes _strange or _charm (according to the declared variants) will be automatically resolved by the special context wrapper, with the additional safety layer of defaulting to the explicitly provided resource if the desired one is not found.

val ct = cadabra.getExperimentContext(AutoResourceExperiment::class)// use `_strange` by default and `_charm` if Variant CHARM is active
showAlertDialog(
ct.getStringId(R.string.dialog_title_strange),
ct.getLayoutId(R.layout.dialog_layout_strange))

The experiment context is bound to a single experiment so it won’t accidentally pick up the wrong resource if variant “Strange” is active for another experiment, but it’s a good practice to give variant longer more specific names.

An experiment with custom data

Should the experiment require more complex parameters than resources, they can be provided as Variant fields

enum class FancyExperiment(
val screenLayout: Layout,
val screenContent: Content,
val itemsLimitPerPage: Int
) : Variant {
CHARM(Layout.WIDE, Content.COMPLETE, 10),
STRANGE(Layout.COMPACT, Content.FAVORITES, 20),
}

And later retrieved via Cadabra singleton

val aVariant = cadabra.getExperimentVariant(FancyExperiment::class)when (aVariant.screenLayout) {
WIDE -> setContentView(R.layout.activity_wide)
COMPACT -> setContentView(R.layout.activity_compact)
}
loadContent(
favoritesOnly = aVariant.screenContent == Content.FAVORITES,
numberOfItems = aVariant.itemsLimitPerPage
)

Experiments activation

For the real-life scenario, we’d need some service to control the experiments, like Firebase, which is supported out of the box

CadabraAndroid.config
// register experiments without starting
.registerExperiment(FancyExperiment::class)
// load experiments config from Firebase
.startExperimentsAsync(FirebaseConfigProvider())

FirebaseConfigProvider is part of the library, and any other services including custom in-house endpoints can be added via extensible providers API.

Note the explicit registration step, which prevents accidental starting of the experiment that’s not ready yet or has been disabled already.

Design

The key design aspects of the library

The Startup

Medium's largest active publication, followed by +752K people. Follow to join our community.

Dmitry Si

Written by

Dmitry Si

Software developer. Most recently Android Java/Kotlin engineer. Former manager, desktop and embedded software creator. https://github.com/fo2rist/

The Startup

Medium's largest active publication, followed by +752K people. Follow to join our community.

Dmitry Si

Written by

Dmitry Si

Software developer. Most recently Android Java/Kotlin engineer. Former manager, desktop and embedded software creator. https://github.com/fo2rist/

The Startup

Medium's largest active publication, followed by +752K people. Follow to join our community.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store