This is a photo of me from my Bar Mitzvah. Back then I thought I had officially become a man:
But, of course, I only truly became an adult once I started using Experiment Docs for the Patreon Growth Team. Let me explain.
Growth is sexy, uh, right? It’s fast, right? You find the hacks, you do the hacks, the metrics go boom.
Boy, I wish. Growth is method. It’s discipline. [Proceeds to lose 95% of readers]
Doing real growth is not “one simple trick to increase conversion” or “how we made bajillions from a brilliant hack.” Real growth is process. You’re scientists in a laboratory, trying to figure out how this mysterious particle/cell/protein behaves (your user’s brain when they use your product).
You try ideas. They mostly don’t work, as they shouldn’t. But they give you ideas for new things to try. Over time your ideas start working more often than they used to. Growth is learning steadily, over a very, very long time.
Just like in a laboratory, being rigorous is the key to progress. Science moves forward when teams can build on the transparent, reliable work of those before them.
And that is why when any of us gets a “brilliant” growth idea, we spend about an hour filling out an experiment document.
Yes, it’s bureaucratic. Yes, it feels like a drag. But that discipline is also the reason the growth team at Patreon has been able to conclusively impact the number of creators making over $1,000 a month, as well as overall GMV growth rate.
What’s an Experiment Document?
An experiment doc is a set of short sections that force us to think through why we’re doing an experiment, what we think will happen, how we’re going to measure it, and whether it’s even worth doing.
Here are the sections we use at Patreon:
Hypothesis
What you believe about your users’ brains/behaviors/feelings/motivations. What makes you think this?
This is NOT what you think is going to happen in this experiment, that’s the next section. This is a statement that can be totally decoupled from this experiment.
We believe that many creators are not getting to be financially active because they are not completing their pages fully, because it’s hard to write descriptions, goals, and rewards. Creators feel writers’ block because they have never had to write these before. We believe this based on interviewing sales, support, and customer success teams who interact frequently with creators just starting out on the platform.
Prediction
If you do your idea, what you think will happen.
“We predict that if we [idea], then [metric] will [increase by x%] because [justification].”
Predicted Impact
Is this experiment worth running? In the best case scenario, how much would this move our company’s metrics?
We structure this as two parallel universes — today (Control) and if we rolled this out to 100% (Predicted). This is totally done with educated guesses using benchmarks and clues from our past experiments.
Experiment Design
What’s the maximum percent of users we feel comfortable A-B testing this with? How long are we running this for?
We shoot for the highest percentage we can go for (50–50 split) if we can. We would choose a lower percentage if the area of the product is high-risk (payments-related) or happen to have multiple experiments running in parallel, and want to minimize saturation effects.
Sufficient Test + Spec
What is the simplest, tiniest scope that we could get away with and still have a meaningful experiment.
Fighting feature creep is the key to speed. We always ask ourselves, is that extra idea REALLY going to make or break this experiment? (Usually no). Will this experiment still be meaningful without that extra feature? (Usually yes).
“If This Works We Should”
This is the section for dumping all our brilliant-at-the-time feature creep ideas. We simply promise ourselves that if this works we’ll do that next.
This a mental hack we do to ourselves to keep scope small. I frequently look at this section in hindsight and thank the stars we didn’t spend the time on these extra features that seemed crucial at the time.
Instrumentation
How are we measuring this? What events and properties are missing? This is a critical section to fill out in advance. The last thing you want to happen is to run something for two weeks only to discover you can’t answer a key question because you didn’t record it.
Be explicit about what it’s going to take to measure — from UTM parameters and what you’re going to call them, to events, to what platforms that data will be found in. Use this section to QA your analytics once you launch your experiment.
Results
While results obviously need to wait until after the experiment runs, we frequently populate this with “ideas for segmentations and analysis” ahead of time to remind our future-selves of ways to dig in deeper.
Learnings
What does this tell us about our user model? Was our hypothesis totally invalid? How much did this impact our actual business metrics (not just nice-sounding percentages).
This is the culminating section of the entire experiment. What did we learn, and…
Next Steps
…what are we doing about it? Running more experiments? Doubling down? Doing more qualitative analysis? Retrying slightly differently? The process begins all over again. Growth is never done.
More reasons to use experiment docs
- When you run a jamillion experiments in parallel, it’s really hard to remember why you’re doing something, or what your follow up ideas were.
- When you’re sharing knowledge with Marketing and Core Product, it’s much faster and more credible to share a super-well-documented rigorous experiment, than “eh we tried this I think and it sorta worked.”
- Two years from now, when “future you” thinks of the same idea you disproved today, you’ll be able to dig in and know exactly what you did and what you didn’t try.
Steal Patreon’s experiment document
You’re still reading? Impressive. This post has been nothing but wet blanket advice about discipline and hard work, and you’ve stuck through it. You’re already more likely to succeed than 99% of people who put the word growth in their LinkedIn titles. Kudos to you.
The Patreon Growth Team has made you a copy of our experiment document:
EDIT Oct 31 2018: Since the original publication of this post, we’ve streamlined our experiment doc even more — here is the 2018 edition. Special thanks to Otis Anderson, Cathy Deng, Matthew Crane, & Maura Church.
I wish you many rigorous experiments!
Know someone whose craft would benefit from this post? Click the 💚 below to surface this post to more growth professionals . Cue cheesy instructional animation:
