Experimentation Reading List: 5 ways to level up your experimentation skills right now

As Optimizely’s Experimentation Program Manager for our Product organization, I’m constantly looking for content that helps me play bigger with experimentation. If you’re doing the same at your organization, check out what I’m reading now below.

If you’re curious what exactly an experimentation program manager does, check out my blog on How to set your goals to level up your Experimentation Program.

1. Conference Talk: Start at the End: How to Do Research That Has Real Impact by Michael Margolis

As the former User Researcher for Google Ventures, Michael Margolis handled a portfolio of over 300 companies.

His framework for conducting user research is helpful to understand how you can improve your experimentation program by asking the right questions, digging deeper on the areas with the biggest potential impact and keeping what matters to your executive stakeholders top of mind.

2. Blog: 4 Principles for Making Experimentation Count by Lindsay Pettingill

This blog post offers great principles for creating high impact product experiments and demonstrates an experimentation program at scale. At the time, Airbnb was running over 700 experiments per week!

I feel like I could highlight this entire blog, however, here are my top takeaways:

• Don’t just launch a feature or set up an experiment, and wait for the magic to happen. In more cases than not, there will be no magic. This does not mean you are not awesome, but is a reminder that our job is hard.
• Why do hypotheses matter? Without them you’re untethered, easily distracted by what appear to be positive results, but could well be statistical flukes. In this situation it’s easy to make up a story that fits your findings as opposed to doing the hard work of understanding what’s going on.
• Get a sense for base rates before you start an experiment using historical data. Without base rates, you are essentially in the dark about whether you can actually detect the impact of your awesome feature change.
• But if you only focus on the wins you are going to miss a ton of insight, and risk being blind to mistakes. Experiments do not fail — hypotheses are proven wrong.

3. Blog: Learning in Product by Ellen Chisa

Program Management, similar to Product Management, can look very different in each organization and can even look very different depending on what projects you are working on. Ellen explains how to take a step back and ask, “What is the best thing I can be learning to have the biggest impact on the company right now and/or have the biggest impact when moving into the my next role.”

This process is a helpful framework for thinking about how to make the biggest impact to your experimentation program.

4. Book: The Dip, by Seth Godin

Even great programs have dips! When your team is low on steam, push through by staying focused on the impact of your end goal.

Godin reminded me that saying no to projects that aren’t worthwhile (aka when there isn’t a high impact end goal) frees me up to focus on making the biggest impact and where I can be most successful.

5. Webinar: Experimentation at Scale by Claire Vo and Becca Bruggman

Claire, my Executive Sponsor and founder of Program Management (formerly Experiment Engine), and I share actionable learnings to grow your program and get more executive buy-in for experimentation.

Have you read any of the above? Do you have any recommendations for what I should read or watch next? Tweet me at @bexcitement.