“selective focus of noodle” by Sarah Boyle on Unsplash

Scoping A/B Test Code in React

A journey to discover a better way to implement A/B testing in React

When Vrbo™ began its journey replatforming to a tech stack built on Node with React, we were full of hope. Our eyes glistened with the reflection of blank files in our code editors. This would be the end of spaghetti code.

Of course, line by line our repo grew, and one by one our dreams started to fade. We knew we would soon write our first A/B tests and we started to ponder how they could be contained but powerful, nimble but scalable. Alas, A/B tests are the natural nemeses of clean code. By their nature, they create long winding strands of logic intended to deliver a clean slurp of an experience but instead often leading to a Lady and the Tramp-style face full of dog breath.

Given a test between three widgets, we have to consider variations in…

• Visibility (showing/hiding a given widget)
• Attributes (color, size, text applied to a given widget)
• Test exposure event (this may fire on page load or as the result of a subsequent action)
• Interaction (events that occur as the result of a user action)

A little housekeeping

This post is about showing and hiding elements based on test assignment in a way that is scalable, readable, and maintainable. It is not about building or integrating a testing platform. To use the strategies described here, we assume you already have a testing platform that buckets users into tests and that you maintain said assignment on a state object that looks something like…

abTests: {
    ABC_WIDGET_TEST: {bucket: 1},
    XYZ_WOGGLE_TEST: {bucket: 0}
}

It also assumes you call a function to log a test exposure, which we’ll call logTestExposure. For the sake of this exercise, we will equate one page view with one test exposure even though reality may, and often does, dictate a more controlled solution.

An okay solution

In our old code base, test code was a tangle of logic scattered between content layer, presentation layer, and event handlers. To create a test between widgets A, B, and C, we might have code that looks like…

// selector
getWidgetColor = (abTests) => {
    return abTests.ABC_WIDGET_TEST.bucket == 0 ? ‘red’ : ‘blue’;
};
// component
onMount() {
 logTestExposure(‘ABC_WIDGET_TEST’);)
}
render() {
    const {abTests} = this.props;  
    const useTriangle = abTests.ABC_WIDGET_TEST.bucket == 2;
    return() {
        {useTriangle &&
            <TriangleComponent color={this.props.widgetColor} />
        }
        {!useTriangle &&
            <CircleComponent color={this.props.widgetColor} />
        }
    }
}

In this case…

• Visibility: Logic lives in the component
• Attribute: Logic lives in a selector
• Test exposure event: Lives in the component
• Interaction: Not shown here for the sake of brevity, but it can end up in the component, action handlers or middleware

Pros:

• Easy to write, no planning necessary

Cons:

• Logic ends up all over the place
• No consistency from test to test

Insidious in this approach is the fact that what’s shown here is just one way to organize your logic. There could be a thousand other equally valid ways and re-deciding how to structure your code with each new test adds overhead for the test author as well as maintainers later on.

An okay-er solution

With the shiny newness of the new web platform, we thought if we just contained test-specific logic to selectors, we could prevent the headache of hunting down test code in a million different files.

With this approach, the code above might be refactored to look like…

// selector
getWidgetColor = (abTests) => {
    const {bucket} = abTests.ABC_WIDGET_TEST;
    return bucket == 0 ? ‘red’ : ‘blue’
};
getWidgetShape = (abTests) => {
    const {bucket} = abTests.ABC_WIDGET_TEST;
    return bucket == 3 ? ‘triangle’ : ‘circle’;
};
// component
onMount() {
    logTestExposure(‘ABC_WIDGET_TEST’);
}
render() {
    return() {
        <ShapeComponent 
            color={this.props.widgetColor}
            shape={this.props.widgetShape}
        />
    }
}

In this case…

• Visibility: Logic lives in a selector
• Attribute: Logic lives in a selector
• Test exposure event: Still lives in component

Pros:

• Logic for determining test variant is scoped to the selector (mostly)

Cons:

• Child components grow in complexity as they must support all possible attribute combinations
• Selectors get unwieldy as test-specific logic intermingles with non-test-specific logic
• Component (or other action handlers) still have to determine when to fire test exposure event

An actually kind of good solution

After writing a few tests using the selector-driven solution, it occurred to us that we were repeating a lot of patterns that could be further abstracted into their own component. The solution we found leveraged a React wrapper component that encapsulates any child code and renders it conditionally based on a user’s test bucket. In this example, we’ll call our wrapper component AbTestContainer.

Code for it looks like…

// component
import AbTestContainer from ‘../AbTestContainer’;
render() {
    return() {
        <AbTestContainer
            testName={‘ABC_WIDGET_TEST’}
            variant={0}
        >
            <RedCircle />
       </AbTestContainer>
       <AbTestContainer
           testName={‘ABC_WIDGET_TEST’}
           variant={1}
       >
           <BlueCircle />
       </AbTestContainer>
       <AbTestContainer
           testName={‘ABC_WIDGET_TEST’}
           variant={2}
       >
           <BlueTriangle />
       </AbTestContainer>
    }
}

Now…

• Visibility: Logic lives in parent component
• Attributes: Logic lives in parent component, branching to unique components or reusing existing attributes from selectors
• Test exposure event: Fires automatically on mount in the wrapper component

Pros:

• Easy to read
• Concentrates logic in parent component
• Easy to clean up once test is complete
• Still allows for flexibility when required

Cons:

• Requires some overhead to set up wrapper component initially
• Stray logic can still appear in selectors and components (but less so than other solutions)
• Added complexity around unit tests for the parent component

Where the magic happens

The key to this approach is the code behind the AbTestContainer component. The gist of its logic is… If a user is in test x bucket y, show the content wrapped in <AbTestContainer testName={‘x’} variant={y} />. If a user is not in the test, or the test data is undefined, show the control content which is wrapped in <AbTestContainer testName={‘x’} variant={0} />.

It looks something like…

function mapStateToProps(state) {
    return getAbTestData(state);
}
function mapDispatchToProps() {
    logTestExposure: (testName) => {
        // do some logging
    }
}
@connect(mapStateToProps, mapDispatchToProps)
class AbTestContainer extends PureComponent {
    static propTypes = {
        testName: PropTypes.string.isRequired,
        variant: PropTypes.number.isRequired
    }
    static defaultProps = {
        testName: ‘’,
        variant: 0
    }
    componentDidMount() {
        logTestExposure(this.props.testName);
    }
    render() {
        const {testName, variant, abTests} = this.props;
        const {bucket} = abTests[testName] || {};
        const isControlOrVariantInTest = variant === bucket;
        const isControlFallback = !bucket && variant === 0;
        const showComponent = isControlOrVariantInTest || isControlFallback;
        if (showComponent) {
            return this.props.children;
        }
        
        return null;
    }
}

An even better future

As a concept, we’ve toyed with the idea of making our AbTestContainer component even more robust by adding support for multiple tests using the same component and in-line support for prop overrides.

Code for that might look like…

// component
import AbTestContainer from ‘../AbTestContainer;
render() {
    return() {
        <AbTestContainer
            testName={‘’ABC_WIDGET_TEST’’}
            variant={[0, 1]}
            propOverrides={{
                0: {color: ‘red’},
                1: {color: ‘blue}
            }}
        >
            <Circle color={‘white’} />
        </AbTestContainer>
        <AbTestContainer
            testName={‘’ABC_WIDGET_TEST’’}
            variant={2}
        >
            <BlueTriangle />
        </AbTestContainer>
    }
}

Now…

• Visibility: Logic lives in parent component
• Attribute: Logic lives in parent component
• Test exposure event: Fires automatically on mount by default in wrapper component

Pros:

• Concentrates even more logic in the parent component
• Reduces code lines overall by adding a quick and dirty way to override props
• Reduces dependence on selectors
• Still allows flexibility when required

Cons:

• Increased complexity for wrapper component
• Decreased readability

In Closing

Since our team has adopted the “actually kind of good solution”, the speed and ease of building new tests has improved and overall satisfaction with it is high. The reward has been a quick two-step test implementation, less stray code, automatic exposure logging, and easier clean up. And that makes a codebase I want to work in.