How A/B Experiments Drive Growth at SurveyMonkey
Since we went public in the Fall of 2018, SurveyMonkey has seen a 15% increase in the growth rate year-over-year for self-serve users. Over the last 5 years as an engineer on the Experimentation team, I’ve seen our A/B testing grow to become a huge contributing factor to this success year after year.
If you’re constantly making changes to a platform with over 17 million users — without measuring — it can be nearly impossible to know which changes are effectively driving growth and which ones are potentially hampering it. The key to knowing what works and what doesn’t is to integrate A/B testing into your rollout plan.
What is A/B testing?
A/B testing is an experimentation method in which one group of users is shown one version of the product while another group is shown a second version. The “A” version is usually identical to the current product without any changes (called the “Control” group) while the “B” version represents a change to the product. Running experiments like this is an effective way to improve the product by first testing to see if a subset of users will react favorably to a change before we roll it out to our entire population of users.
For example, say all the buttons in the product are yellow. A hypothetical A/B test could be to continue to show the “A” group of users the yellow buttons while the “B” users would be shown green buttons. Once we have enough data, we measure to see if users engaged more with the yellow or green buttons. Based on that data, we decide which version to roll out to a wider audience.
Experiments come in all shapes and sizes. You can run experiments on the copy used to describe your product, the design elements used to illustrate what it does, the in-product flows the user will navigate through, an entirely new feature or product, and so much more.
3 types of experiments we’ve done at SurveyMonkey
1. Show different versions of copy to find the one that best matches users expectations
Experiment: Late last summer we ran a test where we changed the copy used to describe our Audience collector. We hypothesized that using more explicitly detailed copy to describe the product would enable users to more easily determine if it was the right product for them. We identified which triggers our users engaged with the most and used those to test our hypothesis.
Result: With the completion of this experiment we observed a significant lift in Audience collectors created presumably because the addition of the word “targeted” along with a clearer description of our demographic offerings resonated with our customers. Simply put, the words you use to communicate what your product does play a critical role in how the user perceives its value and whether or not they decide to make a purchase.
Tip: Try your best to put yourself in the customer’s shoes and deliver them the information they need the most at that point in their journey, in the most succinct way possible. Maybe even try user testing with customers, showing them the feature and seeing what words they use to describe it, or get ideas from people in different roles on your team. Anyone can have a great idea!
2. Introduce a feature one small part at a time to ensure success
Over the last 20 years we’ve seen many surveys flow through our platform with varying levels of success. We ran experiments around our survey creation flow and decided to introduce a new mode called “Build It For Me.” With this feature we were able to distill the core elements of what makes a great survey and surface them to the user to guide them along their survey creation journey.
Experiment: Along with the changes to the initial creation flow, we introduced the “Genius Assistant” to provide the creator with real-time actionable feedback as they made changes to their survey. To test this concept before fully incorporating it into the product, we decided to do an A/B test, keeping the control as is and exposing “Genius Assistant” to users in the variant B.
Result: We found that adding these additional guidelines significantly increased survey deployment rates in the treatment group. It turns out that if people are investing time and energy in your product they probably trust you to guide them on how to use it efficiently. We were able to use this positive result to help steer the product direction for future experiments.
Tip: Sometimes an isolated feature could be successful on its own, but when combined with other parts of the product, can provide a poor experience for your users. If you’re skeptical of your results, conduct an A, B, C test to test the control (A), the added feature (B), and the added feature with the other changes (C).
3. Switch up content order to see how it affects user decision making
Sometimes even a simple layout change can yield outsized results.
Experiment: In this experiment we tested to see if reversing the order of packages displayed on the pricing page would have an effect on the average order value of purchases. In the control group packages were ordered from least to most expensive; in the treatment group packages were displayed from most to least expensive.
Result: We found the average order value increased significantly by about 8% in the treatment group, led by a 33% increase in purchases of our Premier package. The increase in order value can likely be attributed to what’s known in psychology as “anchoring bias.” Wikipedia defines anchoring bias as “… a cognitive bias where an individual depends too heavily on an initial piece of information offered (considered to be the “anchor”) when making decisions.” In the context of this experiment it can be hypothesized that since most of our customers use languages where they read from left to right, the treatment group saw the most expensive package first, mentally “anchored” on that price, and this anchoring had an effect on their eventual purchase.
Tip: Psychology plays a huge role in how we make decisions. I recommend reading the works done by Cialdini and Kahneman to learn more.
Conclusion
I’ve had the joy of watching our tooling and processes evolve as we’ve continued to see the benefits of this approach. I’ve also enjoyed watching the compounding effects of our changes on users’ engagement rates and their subsequent reflection in the company’s performance. At SurveyMonkey we believe that cultivating a culture of curiosity is crucial in maintaining a healthy growth rate for the company. We also believe that data-driven decision making leads to better outcomes than those based on intuition alone. Throughout the years our experimentation process has proven to be an effective way to introduce both minor and major improvements to the product that we know our customers will love.
Whether you work on a software product like SurveyMonkey or in any other industry, approaching changes with an experimentation-first mindset can bring the same benefits to your company too!
If you have any further questions about how to run a good experiment you can reach me at chriss@surveymonkey.com. Also, we’re hiring! So, check out our career page and come join the team!
