A/B TESTING, The Mechanism to Design Wisdom — Lessons From Disney

Too many UX designers in the digital industry are full of pride — believing they have the authority to envision the perfect product solution to users’ problems. They spend countless hours designing layouts, style-guides, animations, and prototypes. Most won’t know how far they stray from reality and success until ‘product launch day’ many months from now. Some ancient philosophers state that pride is the precursor to shame, and only from humility comes wisdom. A/B Testing is the mechanism to design wisdom — teaching those who dare to listen to data and remove ego, formulate multiple iterative solutions, and fail fast.

My Experience A/B Testing at the Walt Disney Company

While being a Product Designer at the Walt Disney Company, I was fortunate enough to be part of the A/B Testing team — a cross-functional team of product analysts, designers, content strategists, and developers to envision and execute impactful product experiences globally. Our team’s core duty was to optimize the ecommerce funnels for both Disneyland and Walt Disney World. Our daily grind was designing and implementing tests to reduce friction, increase conversion rate, increase revenue, and meet other success metrics (KPI’s) for a guest trying to purchase tickets, etc.

3 Steps to Design Wisdom

Step 1: Remove Ego and Listen to Data

By being given access to data analytics tools for a website/app or platform, collective user behavior is exposed and able to be measured. These behavior metrics can be categorized as such — ‘click through rate, conversion rate, scroll reach, mouse movement, time spent, etc.’ Having metrics to measure in a design execution leaves no room for ego, most importantly, you can now manage it with accuracy.

Step 2: Formulate Multiple Iterative Solutions

Once a team understands the current read on existing behavior, they are then able to formulate solutions in how to augment the design or content towards a business goal (ex. ‘The current conversion rate is 2%, how can we increase it to 4%?). At first glance, 2% to 4% is a small increase — however, for a large platform or company, this can result in millions of dollars in revenue. With Multi-Variate Testing, you can come up with multiple variants in how a design can incrementally change — letting thousands of users decide which incarnation performs the best towards said goal.

Step 3: Fail Fast

A/B testing by nature will have losers — one variant will out perform the others. By removing your ego and letting the users decide which variant is best for them — you don’t need to worry or waste time on seeking perfection. This allows teams to move much quicker and be much more self-aware of the evolution of a product. Once you are able to measure your test, you can remove the failing variants and keep the highest performing one for 100% of traffic. The most important thing is to move fast, fail fast, and learn fast.

Thank you! Leave a comment on your thoughts and opinions on A/B testing and how it could help you or your team attain design wisdom. :)