RapideX: Rapido’s in house experimentation platform :Part — 1

WHY We Need An Experiment Platform

As any data driven org we rely on our data to generate signals to feed to our diagnostic framework and decision support system for problem discovery in our products. It becomes critical input for us to tweak / enhance our strategy resulting in new product feature release. While this strategic intervention is critical to improve our product offerings it cannot be rolled out to the entire operational geography / user-base for obvious reasons.
These are hypotheses which need validation by data on a smaller geography / user-segment — which leads to A/B testing.

Over the years as our operating scale grew , so did data points, product features , analytics complexities.
Managing A/B testing at scale across domains in a generic way was becoming a challenge which started resulting in slower conclusion and time-to-market for new features.
Before investing in building an in house experimentation platform , we searched available SaaS / open source offerings and could not find any suitable solution matching our need.Most of the were tailor made for content driven experiments (UI only) or lacked the granularity / flexibility to be able to provide a generic framework for integrating app as well as service.
Hence the decision of building one in house.

Evolution Of Experimentation at Rapido

Manual Experiment Phase ( Pre 2021)

Before 2021, the number of experiments were limited and we used to manage ideation, configuration, and conclusion in a much more naive way . Though it was working for us then we could see things will start falling apart soon as we were growing. The pandemic slowed our growth but we knew that was temporary.

Automation in Experiment Creation but Still no Platform (2021–2023)

During 2021, we tried our hand for the first time to build an in-house experiment platform. We were able to mobilize things , able to integrate with domain services to configure experiments. 2 things didn’t go well for us -
1. Make our architecture & approach generic enough to be decoupled and make it easy for every domain service to integrate easily.
2. And hence adoption.

Retrospectively , it’ll be fair to say that we parted ways with this initiative and went into hibernation.

Inception Of RapideX (2023):

From the later part of 2022, when the effect and constraints of the pandemic was almost vanishing and we were experiencing multi-fold growth as an organisation , we knew A/B testing is going to get mainstream .

We named it Rapidex — Experimentation at Rapido scale!

Design Principles for Rapidex:

1. Should be able to Conceive, Configure & Execute Hypothesis in a generic way within the platform — SILOS ARE BAD , Often leads to Counter-Intuitive & Conflicting .

2. Should be able to run Experiments matching our event scale — SPEED OF INNOVATION & PRODUCT STRATEGY VALIDATION should not get bottlenecked .

3. Should be able to reduce friction between stakeholders with data-points & conclusions.

4. Should be able to capture and translate experiment results for Causal Inference determination.

5. With improved UX & Integration strategy, ease the entry-barrier and hence build CULTURE OF EXPERIMENTATION within the Org.

These are derived from problems we wanted to solve from our past learning.

HOW? Approach of Designing Rapidex

At the very basic, a hypothesis is represented with 2 entities within RapideX -

1. Experiment — Hypothesis that needs validation in order to determine if it’s significant to be adopted as business / product strategy. It mainly consists of the following:

• Product : What product category the experiment is meant for. It is service in Rapido context.
• Domain : Business domain selected for experiment. eg: Pricing, Supply
• Actor : Persona targeted for experiment. Captain / Customer.
• Experimentable Feature : domain functionalities to be experimented.
• Population Condition : Criteria to identify sample population the experiment should run on.
• Sample Split strategy : Defines how to split the sample population into 2 or more test(aka treatment) & control groups.

2. Execution — defines when & how the experiment is supposed to run. Consists of the following :

• Experiment ID — Identifier of an experiment construct.
• Geo — Location where the experiment is targeted to run
• Scheduled Time Window — time window for which the experiment is supposed to run.

This approach gave us flexibility to be able to conceptualise an hypothesis in the form of an experiment_id and then be able to configure its execution specific details in execution_id. An experiment can have multiple executions allowing the A/B testing to run in parallel in different geo-temporal slices.

TL;DR — if Experiment is a class, executions are the objects.

One major aspect of causal inference is Stable Unit Treatment Value Assumption which in short ensures no interference and no variance in one treatment (aka test) groups outcome hence forming non-biased observations during causal analysis.

We’ll touch base on the implementation of this in upcoming posts.

Another aspect of design was to be able to run quasi-experiment alongside true-random experiment because in practical sense, hypothesis often demands to be able to label user-segment in a custom way, deviating from true-random labelling. Netflix’s approach inspired us here.

Trustworthiness in Experiment is a major factor for its overall effectiveness as well as adoption.
Our approach was simple, we’ll take pre and post value comparison of associated metrics as one of the major KPI for any experiment’s success definition. One thing to note here is — the success of the experiment does not mean the hypothesis was correct and gave us an anticipated result in our favour. Success of an experiment is defined as — “to be able to correctly label data and draw inference from the experiment outcomes whether it’s a win for hypothesis or null-hypothesis”.
The Difference-in-Difference method helped us here.

In upcoming posts, we’ll dig deeper on the engineering side of RapideX and will discuss architecture and typical system design nuances.