AB Testing with GrowthBook at Evermos
What people said about the experiment?
“Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day.” ― Jeff Bezos
“One of the things I’m most proud of that is really key to our success is this testing framework. At any given point in time, there isn’t just one version of Facebook running. There are probably 10,000.” — Mark Zuckerberg
The Importance of A/B Testing at Evermos
Evermos is the best social commerce platform of choice for resellers and dropshippers, to empower and build a mutual cooperation economy and collaborate with Indonesian SME and MSME brands.
In today’s highly competitive e-commerce landscape, data-driven decision-making is crucial for maintaining a competitive edge and driving growth. At Evermos, a leading social commerce platform in Indonesia, understanding customer behavior and optimizing the user experience is paramount. One of the most effective methods to achieve this is through A/B testing.
What is A/B Testing?
A/B testing, also known as split testing or bucket testing, is a method used to compare and evaluate the performance of two or more variations of a particular element, such as a web page, an advertisement, or a product feature. It involves presenting different versions of the same element to different groups of users and analyzing the results to determine which variation performs better.
AB testing is a powerful technique that allows Evermos to make data-driven decisions, eliminate guesswork, and optimize their strategies for better outcomes. By testing different variations and measuring the impact on key performance metrics, such as conversion rates (CVR), click-through rates (CTR), or engagement levels, AB testing provides valuable insights into user preferences and behavior.
The primary objective of AB testing is to identify the most effective option or combination of elements that can maximize desired outcomes. By conducting controlled experiments, businesses can understand how variations in design, content, functionality, or user experience impact user behavior and engagement. This empirical approach helps them refine their marketing campaigns, improve website performance, and enhance overall customer satisfaction.
Why A/B Testing is Critical for Evermos
Data-Driven Decision Making:
- Objective Insights: A/B testing provides objective data on what works and what doesn’t, reducing reliance on guesswork and intuition.
- Evidence-Based Changes: Decisions backed by data ensure that any modifications to the platform are likely to yield positive results.
Improved User Experience:
- Personalization: By testing different elements, Evermos can tailor experiences to meet user preferences, enhancing satisfaction and loyalty.
- Usability Enhancements: Identifying and implementing the most user-friendly interfaces and features improves overall user interaction with the platform.
Increased Conversion Rates:
- Optimizing Sales Funnels: A/B testing can reveal which variations lead to higher conversion rates, helping to optimize the sales process from discovery to purchase.
- Reducing Bounce Rates: By understanding what keeps users engaged, Evermos can reduce bounce rates and increase the time users spend on the platform.
Risk Mitigation:
- Gradual Rollouts: Instead of overhauling the platform based on assumptions, A/B testing allows for gradual, tested changes that minimize the risk of negatively impacting user experience.
- Cost Efficiency: Testing changes before a full-scale implementation ensures resources are allocated to strategies that are proven to work, saving time and money.
Competitive Advantage:
- Staying Ahead: Continuous experimentation and optimization keep Evermos ahead of competitors who may not leverage A/B testing as effectively.
- Market Adaptation: Quickly responding to market trends and user behavior changes helps Evermos stay relevant and appealing to its target audience.
How is AB testing at Evermos?
In conducting ab testing, we look for the part in the funnel where small improvements can have the biggest impact.
currently Evermos uses Growthbook to conduct ab testing and adopts the Growthbook ab testing anatomy [1], where detailed ab testing anatomy can be seen in the image below
Hypothesis
AB testing hypothesis is a statement or assumption that is formulated before conducting an AB test to evaluate the effectiveness of different variations or changes being tested. The hypothesis provides a clear objective and expectation for the experiment and serves as a guide for designing the test and interpreting the results. The following is to come up with several hypotheses to support the experiments.
After the list of hypotheses is obtained, then prioritization is based on effort, ROI, etc.
AB testing hypotheses typically consist of two parts:
Null Hypothesis (H0):
- The null hypothesis states that there is no significant difference or effect between the variations being tested. It assumes that any observed differences in the test results are due to random chance.
- Example : There is no significant difference in the add-to-cart rate between the control group (current design) and the variation ATC group (enhanced “Add to Cart” button)
Alternative Hypothesis (H1 or HA):
- The alternative hypothesis suggests that there is a significant difference or effect between the variations being tested. It is the opposite of the null hypothesis and often specifies the direction or nature of the expected difference.
- Example : There is a significant difference in the add-to-cart rate between the control group (current design) and the variation ATC group (enhanced “Add to Cart” button)
The following is an example of implementation in growthbook
Audience and Assignments
Choose the audience for the experiment. To increase the detectable effect of the experiment, the chosen audience should be as close to the experiment as possible. For example, If the experiment focuses on a new user, we should select the audience with new user registration and no transactions yet. If we include all users, we would have users who could not see the experiment, which would increase the noise and reduce the ability to detect an effect. Once you have selected your audience, you will randomize users to one variation or another[1].
Variations
In A/B testing, a variation refers to a version of the webpage, app, or any other element being tested that differs from the control or original version and an A/B test can include two or more variations. Key Concepts of Variations in A/B Testing [1] :
- Control Group : The control group is the original version of the element being tested. It serves as the baseline against which all variations are compared.
- Variation Group : Each variation group includes a modified version of the original element. These modifications can range from minor changes, like the color of a button, to major redesigns of a webpage
The following is an example of implementation in growthbook
Tracking
Tracking records the events and behavior of two or more groups, where the events and behavior are metric success data. Then GrowthBook creates a query builder to get data in each group. The following is a dummy example of query builder results in Growthbook.
Result
With A/B testing we use statistics to determine if the effect we measure on a metric of interest is significantly different across variations. The results of an A/B test on a particular metric can have three possible outcomes: win, loss, or inconclusive. With GrowthBook we offer two different statistical approaches, Frequentist and Bayesian. By default, GrowthBook uses Bayesian statistics. Each method has their pros and cons, but both will provide you with evidence as to how each variation affected your metrics[1]. The following is a dummy example of ab testing results on Growthbook.
Conclusion and Benefits
The A/B testing experiment on UI enhancement for the “Add to Cart” button on the product detail page has the potential to positively impact Evermos by increasing user engagement, driving higher conversion rates, enhancing customer satisfaction, improving business performance, and fostering a culture of data-driven decision making. By continually optimizing the user experience based on empirical evidence, Evermos can stay competitive in the e-commerce landscape and sustain long-term growth and success.
What went well at Evermos?
There are several conditions that are already went well at Evermos
- Some stakeholders already understand AB testing, so they are data driven in determining decisions.
- There are initiatives or projects that go through an ab testing process before going to production.
- Several engineers already understand the implementation of ab testing in applications.
- Good and clear collaboration between teams (engineering, product, business and data) in carrying out ab testing.
What can be improved at Evermos?
There are several conditions that can be improved at Evermos
- A/B testing knowledge is not evenly distributed so there are several initiatives that do not go through A/B testing.
- The timeline is tight so there are initiatives that have not gone through A/B testing.
- There is room for improvement in the documentation, so that every learning in ab testing can be recorded.
References
[1] version 1.0, “The Open Guide to Successful AB Testing with a focus on GrowthBook” open-guide-to-ab-testing.v1.0.pdf (growthbook.io)
