Pick The Right Process: A/B Testing vs Prototyping

Written by Andy

For anyone involved in building a product, you know the importance of going fast while minimizing risk. You need to move fast to get your ideas out to customers, to solve your users’ problems, and to beat the competition. But you also need to carefully minimize the risk of building the wrong product or of spending too much time on ineffective features.

By testing your ideas frequently and vigorously with users, you can gather data on what works and what doesn’t, and then move forward with the most significant improvements.

A/B Testing and Prototyping have their place. But picking the right one can make all the difference when it comes to getting your team the right information about your product.

So how do you go about testing though? The most typical way to do it is with A/B testing: release one or more variations while retaining a control and see how they perform. Especially with feature flag management as a service today, we’re seeing more and more adoption among enterprise companies undergoing agile transformations. However, there are limitations to A/B testing:

• Limited variations in each version. Like any good experiment, you want to minimize the number of changes when you compare them against each other or you may not know which of the changes in each version was responsible for how it performed.
• How it performed, not why. While you’ll get data on which version users clicked on more, you won’t know why. And while those may match your hypothesis, it’s hard to know for sure especially if your question isn’t narrow.
• Building different versions can be expensive. Unless the variations are pretty similar, designing and developing each version for release is expensive and time-consuming.
• Global vs. local maximum. This is really the underlying issue here. If you’re making tweaks to a process or have several similar versions you want to taste, A/B testing is the way to go. However, this method is not ideal for when you want to make larger changes, or you are still exploring different directions.

Prototyping can give you first-hand user feedback while mitigating variables.

This is where prototyping and user testing can be more helpful in your process. User research is the qualitative equivalent of A/B testing’s quantitative focus. Instead of showing a fully developed version to millions of users and collecting the data, show a prototype and learn through user research why certain approaches are (or are not) effective for certain users. And unlike traditional approaches to user research, you can now quickly and easily build a full-fledged app instead of having users click through static wireframes. Through a working prototype, you can:

• Use real data. Real data matters to users’ perceptions of their experience. When you ask users what they think about something, they react very differently when it’s personally meaningful to them. Not only will they tell you what they think, they’ll start engaging with it in ways that you may not have anticipated. At a minimum you’ll find some friction points in your flow, but you may find additional paths and use cases you hadn’t even considered.
• Identify edge cases early. Not only does using real data change user perceptions, it can affect the design process as well. Data is invariably messy — it’s not always evenly distributed, it may not be structured correctly, and it may not even be accurate in places. Playing around with the data in the actual user experience is a lot easier to understand in context than viewing the data in JSON or a spreadsheet. And it’s a lot cheaper to make changes now than after you’ve finished (or even started!) coding.
• Experience your ideas in context. When Jeff Hawkins was developing an early handheld device (pre-iPhone days), he carried a block of wood as a prototype to meetings to simulate what it’d be like to hold and use the device in everyday situations. Especially with mobile apps, using the app on a device (and not a web browser on a laptop) out and about is a different experience than in a lab environment.
• Get feedback with minimal investment. Oftentimes you’ll have a grand vision of what rehauling UX could look like, but it’s hard to get the buy-in for big changes. A working prototype allows you to get feedback on what those changes would mean without having to change your production app,. Even if you don’t get the data necessary to make an eventual large investment, you’ll likely find insights on what aspects of that vision are most beneficial.

In situations where a prototype is most helpful, here are some tips to get you started:

• Develop a hypothesis — Like with A/B testing, it’s important to know what questions you want answered from the prototype. Setting up the right questions means you can prioritize how “full-fledged” a prototype to build and which functionality to include or exclude.
• Build and iterate — Building a prototype is like a mini version of building an app but faster and with fewer constraints. Pick a tool to help you quickly get from vision to app faster and still allows you to easily make changes. Even before you start showing users, you’ll likely want to make adjustments so be prepared to do that early and often.
• Show, not tell (Or, it’s not just for feedback) — Prototypes are helpful not just in getting feedback from users or coworkers, but also as a way of sharing your thoughts. Wireframes or even a sketch on the back of a napkin can do a better job of illustrating your ideas than just the written word. Providing a working app gives everyone a clearer idea of your vision.

It doesn’t matter if your product has mass adoption or is still in its infancy, you’ll need to constantly test and evolve. Gaining a firm understanding of your users’ workflows, desires, and goals is what separates elite product managers from the rest. And the secret to cracking that code is through thoughtful and frequent user testing. Utilizing rapid prototyping can help improve iteration cycles, adoption rates, and key success metrics. So use this guide when determining which type of methodology is right for you and when the time is right, pick the right platform to help you design, test, and iterate quickly.