Disney Streaming Embraces 3 Key Tenets of Experimentation
Collaborators: Diana Jerman, Mark Harrison, Anmeen Leong, Michael Ramm
When conducting experiments that can impact millions of guests, you need tools that are easy to use, help guide good experimental design and build confidence in results that even non-technical team members can understand. We settled on three key qualities for the tools in our experimentation platform: requires limited resources, capabilities and integrations that delivers fast results, and promotes an obsession with customers.
At Disney, experimentation has played a big role in our ever-evolving history and goal of delighting customers. From experiments in animation, storytelling, and robotics, we encourage our teams to look at challenges as opportunities to invent and innovate.
In 2019, Disney Streaming formed a new team — The Experiment-X Team — to develop the tools and culture for running experiments within our partner applications (e.g. Disney+, Star+, Hulu, and ESPN+). Different teams were using different tools at the time. We focused on building one platform that is fast, and supports all teams enabling innovation experiments on millions of customers around the world. We knew the experimentation platform would also need to be easy to use and create confidence for internal teams. We set out to standardize metrics, encourage best practices, and make it easier for all teams to perform experiments at scale
Our team bonded over our frustrations with inadequate investment that created poorly built or poorly designed in-house solutions. In our experience those solutions were clunky, prone to failure or difficult for anyone outside of engineering and data orgs to use. Third party solutions didn’t integrate well enough to generate useful results. We set out to build a powerful, fully integrated tool that was accessible to all, and that will scale and grow with us.
We started with three days of working sessions with stakeholders across the company. We established our mission:
Deliver capabilities that powers innovation with a culture of rapid and iterative experimentation, where performance data informs decisions
We established three core tenets which we are still using to prioritize what we build:
- To let teams create experiments faster, focus on key platform and tooling integrations within our ecosystem
- To let teams deploy and understand results more quickly, develop full-service capabilities that limit the reliance on analytical and engineering resources
- To enable all teams to embrace innovation using experimentation, build an experimentation program that supports all partner applications and pushes our obsession with our customers
We developed a playbook and delivered an MVP platform just 6 months after kicking off our working sessions. The fast turnaround was purposeful. We agreed that we needed to develop something quickly that we can learn from and that our stakeholders would get value from. A year later, after our stakeholder teams had run more than 40 experiments, we launched version 2 of the platform: X². We set up a cross-functional experimentation team to work with stakeholders, helping them integrate with X² and supporting them throughout the testing lifecycle. This included a sophisticated onboarding process, and guidance on experiment design, planning, and analysis.
We organized the cross-functional experimentation team around domains like product, backend engineering, data science and audience intelligence. One crucial team, known as Core Experimentation, develops tools to automate and standardize experiment results. They help create reliable and reproducible results for every experiment, across all partner applications. They help us design and develop features that guide good experimental design, for example in-tool Power Analysis. We will share more on our in-tool Power Analysis and other tools in later posts.
#1 To let teams create experiments faster, focus on key platform and tooling integrations within our ecosystem
Our integrations fall into two main categories: platform (“horizontal”) integrations, and tooling (“vertical”) integrations. A powerful example of a platform integration is with our in-house session service. The session service mints a new session a few different ways: every time a guest enters the site for the first time, when they sign up for the service or when they log in and tell us who they are. This session tells us a variety of other things about our guests, including device type and location. We use these unique attributes to target participants and to ensure consistency in targeting (i.e., an experiment that only targets Android devices for guests entering the site for the first time).
An example of a tooling integration is our in-house landing page content management system (CMS), called Cannonball. Our marketing teams use Cannonball to develop new landing page treatments to test across partner apps (e.g. Disney+ and Star+). With our integration, marketing teams can easily develop, publish and add new landing pages to experiments.
#2 To let teams deploy and understand results more quickly, develop full-service capabilities that limit the reliance on analytical and engineering resources
One of the biggest challenges we wanted to address is how to make experimentation accessible to all stakeholder teams and users. We have consciously established guardrails at this early stage of the program. The reality of building an experimentation program and tooling to support it is that stakeholders, especially those new to experimentation, need some hand-holding. Even so, our vision for the future is to make experimentation widely available to as many stakeholder teams as possible throughout the product development and UX optimization lifecycle.
We set out to identify areas where we relied heavily on our analytics teams and where we might be able to develop features to automate. One of the first areas of collaboration focused on automating results analysis and visualization. We developed ExpAn, the Experimentation Analysis library. Our goals with ExpAn were to improve time efficiency, standardize how people analyze experiments, and increase confidence in experimental findings. ExpAn reads natively instrumented data, performs experiment analysis, writes results into a database, and then X² reads and visualizes the results within the X² experiment results UI. This is more efficient than the manual analysis and standalone results dashboards we used to use, so it saves us and teams a tremendous amount of time.
#3 To enable all teams to embrace innovation using experimentation, build an experimentation program that supports all partner applications and pushes our obsession with customer service.
Two years ago, when we set out on this journey, I’ll admit I was overwhelmed with the scope of what we wanted to achieve and where we should start. We quickly realized that the best starting point was simply to talk to our internal stakeholders. We hosted working sessions and developed a tiger team, made up of a cross-section of the company. Success would be directly tied to how accessible and easy we made experimentation. Experiment velocity was also baked into our business objectives and key results (OKRs), helping to reinforce how important experimentation is. We developed a solid baseline with version 1 of the platform. For X², most of the new optimized features we enabled came directly from feedback and conversations with our end users who leveraged the first version of the platform.
One of my favorite examples of what we learned from launching version 1 of the platform is when teams started using it as a way to control rollouts. A few teams figured out that they could use the platform to slowly release a feature to a subset of subscribers, ensuring a good user experience and limiting the blast radius in the event that something went wrong.
We realized that teams will find creative ways to show us what they need and that it’s important for us to listen and to take action on what we learn. In this case and others, we found a creative way to help teams in the short term but committed to more useful long term solutions. When we developed X², we developed a controlled rollout feature that helped streamline the ad-hoc rollouts people used in version 1.
“There’s really no secret about our approach. We keep moving forward — opening up new doors and doing new things — because we’re curious. And curiosity keeps leading us down new paths.”
— Walt Disney
Today, we’ve onboarded 19 stakeholder teams and have experiments running across key domains including Landing Page, Sign Up Funnel, Commerce, and Quality of Service (QoS). We are investing in our product, engineering and data science teams to enable our stakeholder teams to easily develop, run, and measure experiments. As Disney Streaming continues to grow and compete with other big players in the streaming world, experimentation will be at the forefront of how we continue to delight and engage our customers. We are well aware of the challenge and responsibility we have. It’s what drives us every single day to open new doors and unlock this very powerful tool for our experimenting teams.
In the next part of this introductory series to the Disney Streaming Experimentation platform, we’ll share the key components that make up the platform (the guts). In future posts we’ll focus on specific features that have helped us scale experimentation across all of our partner applications.
Follow us and stay tuned for future posts with more details on the Experimentation Team and what we are focused on … you won’t be disappointed!
—
If you are passionate about experimentation and want to shape the future of Disney Streaming Services, we have openings on our product, engineering and data science teams.
Lastly, if after reading this post you want to learn more about a specific topic, drop your suggestion in the comments section. We love feedback!
