My Experiences with A/B Testing

Albert Yip
Aug 30, 2019 · 6 min read

Introduction

At Hootsuite, a fair portion of our decision-making is built upon experimentation. Determining what features should be implemented or what redesigns should take place is at the core of A/B testing. Value is derived from this decision-making process in the form of increased signup rates and customer retention by providing a more consistent customer journey. Product friction, one of the main barriers of entry to customers, is tackled in a data-driven approach to increase the percentage of active customers that return on a per week basis. It is also used to reduce customer churn, the percentage of customers who stop using the product in a given time period.

How do I experiment?

Getting into the nitty gritty

To perform our A/B testing at Hootsuite, we use Optimizely X, an experimentation platform that provides A/B testing tools, to handle our experimentation logic. This logic encompasses what metrics we want to measure, how to bucket customers into certain variations, what customers we want to target, and how to successfully roll out these changes without interrupting the customer journey.

Linking the LinkedIn

Let’s take an example where we are conducting an experiment that is interested in how a customer engages with LinkedIn, specifically within the early stages of the product. In the following picture, we see there is a homepage banner meant to guide the customer and get them to add a profile picture, connect to a possibly related source and follow 5 hashtags.

Metrics & Tracking

The case against experimentation

Throughout this article, I’ve advocated for the use of experimentation and A/B testing to understand and maximize customer success, allowing them to overcome the hurdles and friction points of the product. However, we could also affect customers in a negative way with too much experimentation. A constantly changing feature base or customer interface puts a lot of stress on how the customer can use the product (“Where did my favourite buttons go?”, “How can I access this workflow?”, “Why did they choose this ugly guide?”). Customers are very resistant to change. They like to preserve their understanding of the product and would rather build upon that knowledge.

About the Author

Albert is a Software Developer co-op on the Product Growth Retention Team at Hootsuite. He is currently a third year Computer Science & Mathematics Double Major.

Hootsuite Engineering

Hootsuite's Engineering Blog