How Meesho’s dedicated A/B Infra enables us to experiment and scale new features
A common dilemma which we often face during the beginning stages of any product development is that we think we understand our users better from a business point of view. However, in reality, users’ needs tend to differ based on various factors such as user demographic, type of business entity and time itself.
For instance, Meesho’s user base from Tamil Nadu could be purchasing more products from the saree category as compared to the northern part of India, where kurti sets are sold more. And it becomes an integral part for us to analyse this difference proactively and customise the app features accordingly.
To understand how a new feature is perceived by a variety of audiences, a dedicated A/B service allows us to make data-driven decisions before rolling out the feature to all users.
Within Meesho, there are different areas where we conduct A/B testing; starting from the login screen to order fulfilment and extending beyond to enhancing the supplier experience as well. Now let’s look at some of our different use cases.
Use Case #1 — We’d like to ask the users to sign up or login at different stages on the app, to understand which variant has more user retention rate.
For one set of users, we would display the login popup when the user adds an item to cart. For another set, we would show the login screen immediately on opening the app. By testing how users in each variant interact on the app, we can understand which variant is more successful in enabling users to login/sign up.
Use Case #2 — We’d want to show different screens to people from different parts of the country.
Based on the location the user has entered during signup and from his/her order history, the users from different states will be segmented into different audience groups in our system configuration. The API for banners will then fetch the relevant audience group from the A/B service and will be displaying the respective state-wise banners on the user’s app.
Audience Types
In our current system, we can create a new audience type using the following three methods:
- Dynamic — A defined remote configuration placed on our system will help us experiment on dynamic user entity as described in Use Case #2. Currently applicable only for users (both new and existing). We’ll listen to Kafka events for information regarding new user signups.
- Automatic — Here we’d utilize RedShift, a data warehouse service via relevant queries and refresh intervals. Applicable for any entity type across the Meesho ecosystem.
- Manual — Uploading a manual list (CSV file) of audience types and entities into the Admin panel to run experiments. This method is also applicable for any entity types.
Architecture Overview
The below architecture allows each feature (client service) on the app to access the predefined set of audience data from this A/B service. The multiple variants of a feature that is ready for deployment, would be rolled out to the selected audience segment to gather more insights as to which variant is successful.
Some Challenges We Overcame
Following are some of the roadblocks that we overcame due to an in-house A/B service:
- In our older A/B testing model, we were only able to consider the user’s metadata as an entity. However, in our current infra we could consider any values as entities (catalogs, pin codes, reviews, and so on) and conduct experiments on a much larger scale.
- Earlier the banners, challenges and other events were rolled out as common to all Indian users. Apart from feature experiments, our current model of audience segmentation allows us to customise the app even better by targeting different sets of users (use case #2).
- Instead of rolling out a new feature to all users, we could now conduct experiments on multiple variants in production and determine a winning variant with real-time user data.
Multiple tech teams across Meesho use the A/B infra to configure and fetch audience entities so that they can test their respective features parallelly without any conflict. This makes it one of the highest throughput services within Meesho, with over ~31000 TPS at peak load.
Meesho has always been a data-driven organisation. By building our own AB service, we have been able to launch all new features as experiments to our users. We track the way these features impact our core metrics over a few weeks. Once we are sure that the feature improves these metrics, we roll it out to all our users. AB testing is thus a fundamental building block for all growth at Meesho.
Co-authored by Ashok Lingadurai.
Tech recruiter Karthik Venkatasamy reveals the only way to survive Meesho’s unmatched growth
Read all about Hridyansh Sahu’s story of accelerated growth — from intern to SDE- I
Are you following us on social media yet? You might want to head over to Life at Meesho’s Instagram, Facebook, Twitter, and LinkedIn pages if you are a fan of fun and positivity.
