A/B Testing: Learning How Our Users Shop
This article was originally published on October 13th, 2015
Last quarter, Refinery29 enacted a series of experiments to try to improve the shopping experience on our mobile site. Here’s a little ditty of how all of it happened.
How We Do Shopping
Strictly speaking, there is no “shopping” on Refinery29 where users purchase products through checkout on our site (that’s a different business model). What we do offer is that while users read articles that features products, we give them the opportunity to click on that item to be taken to another site to learn more and even purchase. Many readers come to Refinery29 to get shopping advice– our readers trust our recommendations. Even though Refinery29 is a content site, we create a lot of desire and curiosity about the products we feature and everyone wins when we can make the learning and buying experience easier.
Our Goals For Testing
Refinery29′s primary goal is to get people to consume our content. While we want to make it easier for people to connect to products they want to buy, we wouldn’t want to dramatically decrease the completion rate of articles to do so. Nor would we to prevent users from reading a subsequent recommended article because any time we take you out of your ‘reading’ flow there is a chance that you might never come back.
With this in mind, we set a goal to improve CTR (click through rate) to a retailer’s product page without affecting the completion rate of a story or video on R29 or disrupting the momentum of a user going to the next story… and the next one, and the next one, etc.
How We Did It
With small, iterative changes like these, it’s important to learn as you go and pivot as needed.
- Step 1: Stake the territory. We decided to prioritize the iterations on mobile web as that consists of the lion’s share of of our site traffic.
- Step 2: Create a baseline. We assessed the current performance of our mobile shopping experience to understand what we had to beat and set goals for improvement.
- Step 3: Test & Hypothesize!
- Test 1 = In order to click out to a shopping product’s page, a user had to tap 2x on the “Buy” button. The first one took users to a larger view, the second one took users to the retailer’s site. In test one, eliminated the detailed view of the product and made it so that the user only had to click once.
- Hypothesis 1 = Requiring only one tap action to be taken to product’s buy page will improve CTR to our partner retailer sites.
- Test 2 = Many of the product images have multiple “Buy” buttons associated with them, which means that the length of a story on mobile web could get quite long, forcing the user to have to scroll for quite some time to complete it. In test two we would eliminate the multiple “buy” buttons, effectively shortening the article’s total length.
- Hypothesis 2 = Shortening the height of the article will improve completion rate of articles.
Test Results
With these two tests we ended up improving the CTR on shopping products by a TON and we were able to get article completion rate to a better spot.
Next steps
Think of new tests to improve this experience and the next steps for the “shopping” experience for R29.
- Small ideas: Testing a new approach to credits, creating a “browser” frame so it is easier to go back to the original article, testing how different buttons change the CTR.
- Bigger ideas: How and if we should have a dedicated product page, whether to integrate video shopping or a universal cart.
- BIG ideas: How to integrate shopping into third party platforms, which is the way that a lot of content is going to be consumed in the future. What can we do to connect readers to the products they want within the distributed platforms like Apple News, Facebook Instant Articles and Snapchat?
Key Takeaways
For people who work in product development, I can hear you yawning. None of the above changes are earth shattering and this process sounds really familiar.
However, the interesting question is how to apply these principles to the content world, which for the most part is the land of “splashy” rather than iterative hypothesis driven development. My conclusion is that any plan that is concrete, rational, and broken out in actionable and understandable steps is inevitably more compelling and likely to succeed even in the fast and reactive world of media. Sometimes it takes that kind of patient, focused and measured development — and a lot of unsexy A/B tests that move things step by step — that can really transform a company.
– Jen Ku, Director of Product
