A/B testing made easy with Polymer

Open source behavior to run experiments in your Polymer application for A/B testing and staged rollouts

White-ground kylix (wide-bowled drinking cup), found in a tomb at Delphi. Photo by Dennis Jarvis

You want to add this shiny new green button. But your colleague strongly believes it should be red. How do you know who is right? In another scenario, you implemented the essence of a new feature. Although there are rough edges, you want to roll it out to a selected group of lead users to gather their feedback. How do you do this?

Experiments to the rescue

Both scenarios, A/B testing and staged rollout, are solved by implementing two variants into your application, and then dynamically selecting them. The algorithm, also called oracle, could pick a variant randomly (A/B testing) or based on the user ID (staged roll out).

There are tons of A/B testing services out there but we found them too heavy for many small scale experiments. Hence, we created — and open sourced — a Polymer behavior that allows for simple experimentation.

Implement your first experiment

First, you need to add the behavior to your project:

$ bower install app-experiment-behavior

Next, add the behavior to your component:

behaviors: [
  Polymer.AppExperiment.Behavior(
    'experiment-component:greeting',
    [ 'world', 'universe' ]
  )
],

The parameter experiment-component:greeting defines the name of the experiment and world and universe are the keys of the two variants.

Next, you need to define the property experimentViewer, which contains the viewer:

properties: {
  experimentViewer: String
}

Finally, you can use the experiment property, exposed by the AppExperiment behavior:

<template is="dom-if" if="[[_is(experiment, 'world')]]">
  Hello world
</template>
<template is="dom-if" if="[[_is(experiment, 'universe')]]">
  Jo, universe!
</template>
...
_is: function(val1, val2) {
  return val1 === val2;
}

See the demo for a complete example (c0de).

Smarter oracles

By default, the AppExperiment behavior assumes that you store the ID of the viewer (i.e. the user looking to your element) into the experimentViewer property. The behavior will “randomly” pick a variant based on the hash of the viewer ID.

Sometimes you want to do things a bit smarter. Let’s say we want to show our bleeding edge feature only to users in the tenant early-adopter. Our viewer object, stored in property experimentViewer, contains a.o. the name of the tenant, e.g.:

{
  id: 'joe',
  tenant: 'early-adopter'
}

With that in mind, we can define our own oracle that will pick a variant based on the tenant:

const oracle = function(viewer, values) {
  if (viewer.tenant === 'early-adopter') {
    return 'bleeding-edge-variant';
  }
  return 'stable-variant';
};

Finally, we need to pass the oracle when initializing the behavior:

properties: {
  experimentViewer: Object
},
behaviors: [
  Polymer.AppExperiment.Behavior(
    'experiment-component',
    [ 'stable-variant', 'bleeding-edge-variant' ],
    { oracle }
  )
],

You can find the AppExperiment behavior on Github.

Happy coding!

Want to learn more about Polymer? Have a look to all our Medium posts on Polymer.

Photo: Dennis Jarvis