A/B Testing Conference in Carlsbad, CA
Now updated with slides and videos!
EXPO SUMMIT provides the opportunity to learn about A/B Testing process, analytics, and technology
On February 5, 2019, Walmart Labs in Carlsbad, CA hosted a free A/B testing conference with speakers from Intuit, LinkedIn, State Farm, and Walmart Labs.
We reviewed how Experimentation and Optimization is done and the technology behind it, from experimentation platforms and process to the reporting and analytics performed. Attendees met other industry experts, asked questions and shared their experiences.
If you couldn’t attend, or just want to review the talks, slides and videos are available below (where we could).
Opening Keynote
Claude Jones, Senior Director of Engineering, Walmart Labs
Claude Jones is a seasoned technical leader with over 15 years of experience. He has been with WalmartLabs for 6 years and is responsible for managing a distributed team of 60+ product, user experience, project management, architects, and engineers focused on building custom tools to improve the way Walmart develops and delivers eCommerce experiences across the web, native app, and stores.
Claude has a passion for bringing out the best in others and believes that anything is possible with hard work and dedication. In his spare time he runs the Elevate Foundation, that is an organization focused on helping others in need and giving back to the community.
Speakers
Anthony Tang, Senior Engineering Manager, Walmart Labs
Anthony Tang leads the A/B Testing Platform team responsible for building and maintaining EXPO, the experimentation and optimization platform for Walmart. His team works closely with product analytics and product management to enable and promote A/B testing across the Walmart websites and mobile apps. Anthony joined Walmart Labs in 2012 after working at Yahoo!, and earned his MS in Computer Science from UCSD.
The Evolution of a Platform, Inside EXPO
Designing and building a new A/B testing platform is a challenging undertaking. In this talk, Anthony will discuss the challenges and key learnings, not only technical but also organizational and cultural, in developing, internally selling, and ultimately winning organizational acceptance, of Walmart Lab’s own internal A/B Test Platform “Expo”.
Slides Available
Video Available
Weitao Duan, Manager, Experiment Science, LinkedIn
Weitao Duan manages a team of data scientists in Applied Research team at LinkedIn. His team drives experiment innovation through research and engineering efforts and collaborates with a variety of teams in Data Science and Application/Infra Engineering. He has 5 years of experience in solving complex applied research problems in causal inference. Weitao holds a Ph.D in Industrial Engineering from Northwestern University.
Experimenting with LinkedIn Economic Graph at Scale: Stories and Lessons Learned
A/B testing is utilized to make data-driven decisions at many web companies. Statistics has always been the backbone of A/B testing, and its role is even more crucial in the era of Big Data. General A/B testing frameworks and methodologies have their challenges and pitfalls in practice. In this talk, we will discuss several challenging A/B testing scenarios, such as addressing network effect where one user’s action can influence another and how we solve these at scale.
Slides Available
Video Available
Dylan Lewis, Experimentation Platform Leader, Intuit
Dylan Lewis is the Product Management Leader for the Intuit Experimentation Platform. He drives the product roadmap to ensure that data driven decisions are backed by solid experimentation practices. Most recently he was the Analytics Evangelist focused on helping data workers be able to do the best work of their lives. He brought C-level visibility to the analytics tools, systems, and processes used across Intuit. Prior to that, Dylan was the Director of Experimentation at TurboTax.com, leading the testing program for 12 tax seasons. Dylan joined Intuit in 2005 after working at Visual Sciences. He earned his Master’s in I/O Psychology from SDSU.
Lessons Learned in Experimentation @ Intuit
Dylan will be walking through the lessons learned from optimizing a testing program over more than a decade at Intuit. In his talk, learn how continuous reflection impacted the performance of the testing program at TurboTax and how the lessons learned can be applied in your organization.
Mark Russell, Lead Analyst, A/B Testing Team, State Farm
Mark Russell is the lead analyst of the A/B testing team at State Farm. Mark has a Ph.D. in Human Factors Psychology from Wichita State University and has been at State Farm for 14 years. He started his State Farm career in their usability lab and later moved to digital analytics where he analyzed online user behavior and started their online survey program. Currently, he coordinates A/B testing on State Farm’s online properties and evangelizes the value of digital experimentation across the company.
Risk vs. Reward: Challenges to Adoption of A/B Testing
Mark will discuss the challenges and lessons learned in adding widespread use of A/B testing to a corporate culture with nearly a century of history. In this talk, he will cover issues like educating stakeholders on the basics of experimentation, getting buy-in from business partners to see A/B testing as a solution rather than extra (optional) work, and coordinating downstream analysis of the impact of A/B tests.
Jessica Huang, Online Grocery Product Analyst, Walmart Labs
Jessica Huang is a Product Analyst on the Online Grocery team. She helps Product Managers on new customer onboarding and other initiatives such as Weighted Meats and ADA compliance with A/B testing and other site analytics requests. Previous to Walmart, she worked as a consultant at Accenture on projects such as email marketing analytics and A/B testing. Jessica is a proud UC Berkeley Alumna — Go Bears!
Sonali Sheel, Online Grocery Product Manager, Walmart Labs
Sonali Sheel leads Walmart’s new user experience for Online Grocery. As a graduate of the University of Michigan, Sonali previously worked in Product at ClearCost Health and Practice Fusion. Outside of work she enjoys playing recc basketball and keeping Shawty, her air plant alive.
How Walmart Labs Online Grocery uses A/B testing
Online Grocery offers customers the ability to shop for groceries on-the-go, and choose a time around their busy schedules to pick up in store or have them delivered to their door. In this talk, we will share how Walmart Labs uses A/B testing to measure the impact of new features on the end-to-end customer experience and iterate quickly to optimize our site and mobile app in an omni-channel environment.
Slides Available
Video Available
Breakouts
Omar Jabbar, Senior Product manager — Experimentation, Walmart Labs
Omar has been working in the Experimentation and Optimization space for 10+ years. Currently he is with Walmart’s Testing and Optimization team supporting its new Online Grocery Experimentation program. Previously he was the Head of Experimentation at PayPal where he led the team transition to a federated based Experimentation COE model and helped shape the new in-house Experimentation platform. Prior to PayPal Omar was at eBay for close to 9 years working on the Experimentation team. He managed both Experimentation Test Management & Data Quality teams there. With the help of his teams at eBay he led & built a thriving culture of a Experimentation enabling 1000s of tests a year.
A/B Testing Process
There are many different ways to run an A/B testing program. We will share in this breakout the different processes we use, pain points, and success stories.
Shuling Liu, Director, Testing Optimization & Analytics, Walmart Labs
Shuling is the director of Testing Optimization and Analytics at Walmart Labs. She manages a team of test managers, data scientists and statisticians. Before joining Walmart, she was a director of marketing analytics at eBay with extensive experience in CRM, Social and Loyalty, domains that relied heavily on AB testing and optimization. In addition, she has worked for few other companies such as Allstate Research and Planning Center, American Management System and Chase Bank. She holds a M.S. in Operations Research and Statistics from Rensselaer Polytechnic Institute, as well as a B.A. in Economics and B.S. in Mechanical Engineering from Shanghai Jiao Tong University of China.
A/B Testing Analytics
You’ve run the test, and collected a lot of data, but what does the data tell you? In this breakout, share how you interpret results and learn how others analyze data to get the most out of your A/B tests.
Steve Bitondo, Staff Product Manager for Expo, Walmart Labs
Steve Bitondo is the Product Manager for the A/B Testing platform at WalmartLabs. He has more than 30 years experience in computer software engineering, including stints coding I/O drivers and networking software for HP, and doing front end web development for walmart.com. Steve is one of the original developers of the walmart.com in-house A/B testing platform. He has 3 patent applications, and likes to focus on the intersection of web technology and business decision making. Steve has a BS degree from the California Institute of Technology and a MBA from UCLA.
A/B Testing Technology and Platform
3rd party or in house and what features are important? What challenges do you run into and how can the platform or its usage help? In this session, we’ll discuss A/B testing platform concerns and share what works and what doesn’t.
Schedule
8:30am — 9:30am — Registration/Breakfast
9:30am — 10:00am — Welcoming Keynote — Claude Jones
10:00am — 10:40am — Anthony Tang, The Evolution of a Platform, Inside EXPO
10:40am — 10:50am — Break
10:50am — 11:30am — Weitao Duan, Experimenting with LinkedIn Economic Graph at Scale: Stories and Lessons Learned
11:30–11:40 — Break
11:40am — 12:20pm — Dylan Lewis, Lessons Learned in Experimentation @ Intuit
12:20pm — 1:30pm — Lunch
1:30pm — 2:10pm — Mark Russell, Risk vs. Reward: Challenges to Adoption of A/B Testing
2:10pm — 2:20pm — Break
2:20pm — 3:00pm —Sonali Sheel and Jessica Huang, A/B testing Walmart Groceries
3:00pm — 3:15pm — Break
3:15pm — 5:00pm — Breakout Sessions (Technical, Process, Analytics)
5:15pm — 5:30pm — Closing Remarks
5:30pm — 7:30pm — Networking/Happy Hour
