A/B test : Why to “(A)Always (B)be Testing” for NHB products?

Products For Next Half Billion
9 min readOct 30, 2022

--

This Article explores the importance of conducting A/B tests for Products meant for the Next Half Billion users and the nitty gritty of setting them right to have the maximum impact from the tests to learn most about users and craft best possible experiences!

Sample Representation of an A/B Test

Let’s start with exploring who are these NHB users!

Who are NHB Users?

People who have come online for the first time via their mobile phones, since 2017, with 500+ million yet to come online in the next 5 years.
In India, they represent the lower 60 percent of India’s income distribution and have traditionally been underserved, excluded, and disempowered from becoming a part of the mainline technology and digital products.
Read more about what does the term NHB mean in the article titled “What do we mean by the Term NHB?” here: Link

What is an A/B Test?

An A/B test is a process that helps us take a call on what would be the best for the user and would enable them to perform the intended action successfully and while we are at it, it helps drives business priorities and revenues as well.

Theoretically, it refers to a randomised experimentation process wherein two or more versions of a variable (web page, page element, app etc.) are shown to different segments of website/app visitors at the same time to determine which version leaves the maximum impact and drives business metrics.

Why is an A/B Test needed?

Benefits of A/B Testing

An A/B test is needed whenever it is difficult to make choices conclusively as to what would provide better user experience and help the user to achieve their goal in the minimum amount of time while ensuring that they follow the best process.

We should consider an A/B test as it gives a comprehensive view of what would be best for the user quantitatively and qualitatively and helps us take the right call!

Why is A/B Testing important for NHB users?

NHB users are the users who have recently come online hence for them established principles, interactions and flows may not work.

Is it important to test and validate each existing flows and hypothesis that is drawn from the current literature that exists about product management and is influenced by existing global products.

Persona, Behaviours change pretty much every 15 km in NHB landscape!

As these users are new, their behaviours, their ways to interact with the products are different and hence it’s always important that we don’t settle till the flows are optimised to the brim!

Current products are not vernacular, do not spark confidence, don’t have the UI/UX fit for these users and hence its important that we build products aimed at these users that can help them adopt products and technology faster and stick to it!

And nothing better than A/B test can come to rescue here to know what would be the best for the users!

Additionally, NHB offers a second order thinking problem as well, which is that with each 50 kilometer, you can expect the dialect, language, societal structure, access to internet, language change. Hence, its important that we optimize the product for each Target group on the app contributing to above a certain NHB persona.

Also, A/B tests offer us a possibility to test out the aspects user research before implementing them completely and going full throttle into the same!

Hence, test everything and then build your product. Even if things don’t work out, you know what went wrong and what not to do in future!

What can you A/B test?

You can A/B Test any and every component of the product before taking call of what to finalise for your user. In case of NHB audience, its very important to test various components that exist and see if user are able to follow the same and perform.

Product Element you can test:

1. Copy

Copy [Content] of A varies from B

Sometimes effective copy can convey the value of what your product wants the user 100x better than an ineffective one and enhance user experience!

2. Flow

Some flows work for few set of users, others work well for majority! Important to test critical flows like the onboarding one to understand what can boost your login to install!

3. Design

A GIF vs Video vs Static Image: Elements at the same place in the user flow can convey different elements to different depths! A/B test what conveys the value of the product most effectively and then shape up

4. Process

Tier 1 audience prefers card payment but tier 2 or onwards [ NHB users] prefer to use cash yet before they build the trust on the ecosystem. Hence, its important to trust the process we use here!

5. Third Party Services

Different third party services give better results for different TG sets! Experiment before choosing. Example OTP providers who can deliver messages effectively in NHB areas, are very few with varying success rate. Backend infra platform for live comms also work differently with different TGs due to location — A/B test your providers before taking the call!

You can A/B test between OTP providers to enable what can help
You can A/B test between 100ms and Agora to measure what streaming platform would work better for you!

6. Color

Yes, it turns out, we can A/B test color as well. Google A/B tested the CTR of the shade of the blue for CTR button and had a variant which made 200 million dollars more for google every year! Yes, just the color of the CTA!

Read more at: https://www.theguardian.com/technology/2014/feb/05/why-google-engineers-designers

Famous Google A/B test!

7. Anything and everything!

A/B test everything important and take data driven decisions!

Different types of A/B tests

Multivariate testing

Multivariate testing (MVT) is a form of A/B testing where variations of multiple page variables are simultaneously tested to analyse which combination performs the best out of all the possible permutations.

Split Variant Testing

In split testing, you test a completely new version of an existing page to analyse which one performs better. You should use split testing when you want to test the entire design or copy of the existing page without touching the existing page.

Multipage Testing

In multi-page testing, you test changes to particular elements, say the CTA button, and across multiple pages, as shown in the image below:

How to perform an A/B Test?

Performing an A/B test can generally be split across 5 different steps post the observation step.

Observation Step
So, lets say you have an observation backed by data is that sale of a particular element has dipped in your product!

Hypothesis Step
Hypothesis would be that probably the flow leading to that element is not very visible to the right TG or the copy is not able to effectively convey the complete value of the same!

Research
Next step would be to seek all possible data and perform user research via calls, surveys and understand more user feedback around each hypothesis and understand which one can be the most probable cause and then shortlist them in priority order!

Experiment Outline
Create an experiment around testing what you believe are high priority cases to understand what is not working or how you can boost the sales at best. For example now, Create minimum 2 variants of the current flow and the new best flow you believe can help discovery! Define which set of users are allocated what and how long would you run the experiment!

Launch
Now is the time to take help from design, tech and launch the a/b test with proper defence and target metrics in place along with near to real-time tracking setup already!

Measure
Now is the time to taste success! Measure how effectively were you able to convey the value to the user basis how much delta you see in the final sales across the variants! if it works, scale the experiment. If not, test another hypothesis or variant till you are able to solve and understand, what could drive that up!

Steps to Perform an A/B test

A/B testing mistakes to avoid

Creating too many variants which makes it difficult to come to a conclusion if the results are very close

Running the variant for too short a time hence preventing any conclusive data to form

Not defining metrics before launching which prevents us to effectively track which variant worked the best

Not launching enough A/B is the biggest mistake!

A/B testing challenges

  1. Running the Test for right duration
    It’s important that we run the test for the right duration to ensure that we measure the important metrics of retention and allow significant users to use the same
  2. Splitting the audience in the right set
    Making sure that there is no bias while defining the target and control group and keeping this as randomised as possible for best results!
  3. Setting the right experiment outline
    It is important to define the right set of variants and have the objective always in sight. Understand your users and use that to craft the variants to chose between best backed by data and inference.
  4. Defining right target metric
    Most of the A/Bs are set with how we would conclude which would be the winning variant and how we would measure the success of the event
  5. Defining right defence metric
    It’s important to set what metrics should not be impacted via the a/b!

Conclusion

A/B test should be used effectively and designed on various components regularly to ensure that the NHB audience gets best exposure on solving their problems via tech.

A/B Test is not a one time thing. You can use it continuously to enhance the user experience and keep improving business metrics bit by bit without making a change that may cause more harm by not measuring it properly and launching without testing.

A/B test gives us a clear picture of what we should do and if yes, what was the impact of the same and if not, where the dip is and what can be improved!

Right form of A/B test hence should be: Always be Testing

Any suggestions for us, do share in comments!

Follow US on Linkedin

  • Follow us on Linkedin to stay updated about interesting insights about Products for NHB
  • Join the WA group to stay updated and join the community to contribute
  • Check our Medium link here to stay updated about interesting product work and research happening for NHB

Team behind the Project
Rukmum Vatsalya — Product Designer
Amar Srivastava — Product Manager
Vikash Singh — Product Designer

Team NHB

--

--

Products For Next Half Billion

We are a community of product managers and designers who are building products for the next half billion users!