How to Get More Out of A/B Testing Than a Win or Loss
A 2-step Guide for UX Designers and Product Managers
You just found out your new design lost an A/B test. It’s that one you spent hours of user research on; the one with every pixel in its place.
First you’re in shock. Then you start denying it and re-checking the data. You may even pull someone aside to vent about it. You’re basically going through the seven stages of grief for the loss of an A/B test.
I’ve been there and chances are, if you’re a UX designer or product manager for a company that believes in A/B testing, you have too.
At Lucid, we have around 25 scrum teams and at any given time, each one is running 5 to 10 A/B tests. I was once the UX designer on a team that ran over 50 A/B tests in three months. We pretty much test everything, sometimes even bug fixes!
In this blog post, I’m not going to explain what A/B testing is or why you should do it (there are already a lot of good resources on that). Instead, I want to tell you about 2 things I’ve learned over the course of hundreds of A/B tests that can allow you to get more out of A/B testing than wins and losses (and to feel less grief).
Doing these 2 things can have a greater impact than one individual test ever could!
1. Document your A/B Tests
This is a fancy way of saying “write a summary of your tests down.” We’ve established a pretty good template for A/B test documentation at Lucid, which I’ll cover later. First, let’s highlight the business and personal benefits that come from good A/B test documentation.
Business Benefits of A/B Test Documentation
Higher Quality Tests in the Future
Learning or remembering details about past A/B tests and why something won or lost leads to deeper understanding and context. In turn, that understanding sparks new ideas, causes stronger hypothesis, and prevents running similar tests twice. Seem obvious? Yeah, I thought so too.
“We ran something similar in the past and it lost.”
“Susan had some good thinking around this but recently left the company.”
“We had reasons for doing it this way, but I can’t remember all of them.”
Have you ever heard statements like this while sharing your work? The conversations either go one of two ways. If you can easily get additional context and understanding, the conversation will lead to an improved test or a new test idea completely. If you cannot get additional context, the conversation will lead to frustration and a feeling the test could probably be better (if only someone had spent 10 to 15 minutes to document the work).
A/B test documentation also helps new hires or new team members get up to speed faster and create higher quality work.
Happier, More Informed Stakeholders
Lucid is rapidly growing. Here’s a link to our open positions in case you are interested in applying! With that growth comes the need to streamline how we keep stakeholders informed about what we are working on. Good A/B test documentation allows stakeholders to spend their time providing feedback instead of working to understand the premise and connect the dots. If stakeholders and managers are aware of documentation, they can easily share it out to other people they think it will benefit and facilitate collaboration.
Personal Benefits of A/B Test Documentation
Does it seem like your documentation mainly benefits other people and teams within the organization? Real value comes when documentation is expected from all teams and there is a giving and receiving of insights.
Here are some of the personal perks you’ll notice as you take time to write and engage with A/B test documentation:
You’ll Literally Get Smarter
The brain can only hold so much information. Research on forgetting shows within 24 hours, people generally forget an average of 70 percent of new information. Documentation is good because it simply helps you remember what you once knew! Quickly jog your memory on a past A/B test by pulling up the documentation and make smarter recommendations based off it.
I was the UX designer for the Lucidchart mobile app at one point in time. When I first switched to that team, I remember realizing a test we ran on the web could apply and be successful in the mobile app as well. The test specifically dealt with the order in which users name a document and share a document. I was able to brush up on the data in the documentation and felt pretty smart when I presented the idea to my product manager.
You’ll Become More Trusted
Taking the time to document for stakeholders conveys thoroughness in your process. It shows managers you take the time to do things right.
You’ll Have Content for More Creative Initiatives
Your documentation will be helpful when you need case studies for new hire interviews, content for workshops and presentations in the UX field, and pieces for your portfolio. Just recently I was able to dig up some past A/B test documentation to support a project Lucid put together for students of Human-Computer Interaction Design at Indiana University.
Lucid’s A/B Test Documentation Template
A group of product managers and UX designers crafted a template for A/B test documentation at Lucid a few years back. It has changed and improved slightly over the years; I hope it can be a helpful resource or starting point for you.
Not only is the content of A/B test documentation important, but also where the documentation lives and how it is written. In general, good A/B test documentation is:
- Easily accessible. At Lucid, we use Confluence to store A/B test documentation and everyone within the organization can access it. In order for an A/B test to have a greater impact, it needs to live somewhere others can search for or stumble upon it.
- Clear / concise. If documentation is too lengthy or hard to follow, the likelihood someone else will read or learn from it is so low you probably should not have even bothered with it.
2. Review Multiple A/B Tests at a Time
At Lucid, we try to hold larger A/B test review meetings and look at batches of tests together.
Why Hold this Type of Meeting
Find and Learn From Bigger Picture Themes
When you look at multiple tests together, you can pick up on themes you otherwise wouldn’t be aware of or feel more confident about a theme. Themes can range in topic from implementation trends to interaction design principles to learnings about users.
One of the themes that kept popping up in an A/B test review meeting was “familiarity”. We marked several winning tests with “familiar to another product.” Had we not looked at several of the tests together, we may have been able to pull out this trend. However, it would not have been such a resounding, strong trend we felt comfortable designing around it in the future. It was helpful to know some of our most successful tests mimicked the way another product like Google did something. In future projects, I kept this theme in mind as I designed (not to reinvent the wheel unnecessarily) and still keep it in mind today!
Develop Stronger Test Hypotheses
We invite multiple stakeholders and managers to this meeting and having extra eyes on the tests often helps pull out new hypotheses as to why a test won or lost. In one of these meetings, someone pulled out a detail I had not considered as a possible culprit for a losing test: the wording. I had spent a lot of time crafting it with a different stakeholder and felt pretty confident it was better than T-A. However, she had good points and I’m glad she was in the meeting to bring my attention to something I would have overlooked.
Better Understand Users
In our meetings, we try to think about what all of the test results tell us about our users. This is something Matt Snyder, our director of UX, is really good at. What do all the test results and themes tell us about users, their personalities, their tendencies?
Keep Design Assets and Thinking Up-to-date
Most of the time, UX designers follow along as an A/B test runs. However, if your team is running a lot of A/B tests, these meetings help keep everyone informed on the actual status of an A/B test. If the tests affect components used in a design system, it’s important for UX designers to be aware if the test won or lost and how to implement similar designs in the future.
How to Hold this Type of Meeting
Push to hold this type of meeting at least once a quarter or when there are at least 10 A/B tests to review. The meeting generally lasts 1 to 1.5 hours. Those invited include the UX designer and product manager closest to the tests, their managers, and 1–2 other key stakeholders. I work with my product manager to co-own the agenda and rely heavily on him to explain the test results to the group. To make it more fun, consider holding the meeting over lunch!
Meeting Prep
Before the meeting starts, prepare by making a spreadsheet that includes all of the A/B tests you want to go over and all the links to the A/B test documentation (this is another area where documentation comes in handy).
Meeting Agenda
- Explain the goal of meeting. Make it clear that the group is trying to review as many A/B test results as possible in order to come out with better hypotheses and bigger picture themes.
- Review the first A/B test and 1) highlight the differences between the A arm vs. the B arm, 2) state the results of the test and which metrics determined this, 3) allow time for the group to share hypotheses, 4) repeat these steps with all the other A/B tests.
- Save 10 to 15 minutes at the end to discuss shared themes and user learnings across all of the tests.
Advice for the Notetaker
Start writing down potential learnings that come up as the group is discussing each individual test, even if they aren’t technically “themes” yet. I like to create a column for a potential theme and add checkmarks to any tests to which it applies. At the end of the meeting, our spreadsheet is a lot more complex!
In Summary
Documenting your A/B tests and regularly reviewing them in batches can have a greater impact on your business than any single A/B test result could! I’ve witnessed those 2 things leading to higher quality tests, more informed stakeholders, smarter employees, stronger test hypothesis, and a better understanding of users.
These things have helped me personally to become a more trusted coworker, to feel smarter, and to have the content to contribute to larger creative initiatives. I also feel less grief over the loss of a single A/B test as I’ve come to realize there is more to gain from A/B testing than an individual win or loss!
