How Pinterest Supercharged its Growth Team With Experiment Idea Review

Jeff Chang, John Egan | Growth Engineering

Growth teams need to be organized bottom-up to scale well

The Pinterest Growth team has over 100 members, and we’ve run thousands of experiments over the years. It’s difficult to run that many experiments and still maintain a high success rate over time. We’ve found the traditional growth team model of team leads deciding which opportunities to try didn’t scale well as our team grew. Increasing the number of high-quality ideas ready for experimentation is one of the biggest levers for increasing the impact of a growth team, but our leads have less and less time to research and find great opportunities as we continued to scale the team.

To address this, we changed the structure of our Growth team to be bottom-up, meaning everyone — engineers, product managers, engineering managers, designers, etc. — is expected to contribute quality experiment ideas. However, this model immediately runs into the reality that finding great growth opportunities is a rare skill that usually only comes with experience. We developed Experiment Idea Review (EIR) as a way to quickly train team members to find high-quality growth opportunities and continuously build our idea backlog. The rest of this post describes how to run EIR effectively.

An inversion of the traditional growth team model for experiment ideation. Dotted lines represent the ongoing growth of the team. Enabling everyone to generate high-quality experiment ideas scales much better than the top-down model.

Experiment Idea Review Guide

What is Experiment Idea Review?

In a nutshell, EIR is a recurring meeting to pitch ideas. It has two main goals: (1) train team members how to come up with quality ideas and (2) build a sufficient backlog of high quality experiment ideas. EIR can be run at various intervals (e.g., weekly, monthly) depending on how many quality ideas the team needs for a healthy backlog, but the more practice everyone gets, the better they become at originating high-quality ideas.

It’s important to note EIR is not a brainstorm. At Pinterest, we historically used brainstorms to generate growth experiment ideas. We would get everyone together in a room and have each person come up with ideas off the top of their head, which we would then try to feed into our roadmap. However, we’ve learned simple brainstorms tend to only be effective when the goal is to come up with solutions to very specific, scoped problems (e.g., the email open rate for dormant users is good, but the clickthrough rate is low — how might we solve that?). People are able to contribute much more when given sufficient time to research ideas ahead of time versus just coming up with ideas in the moment.

Experiment Idea Review Process

Before an EIR meeting, team members come up with ideas and spend up to an hour filling out an experiment idea doc. A well-written document quickly provides readers with enough detail to make an educated guess of the idea’s potential value, so the doc should be as detailed as possible.

An experiment document should contain the following components:

• Problem statement: What user problem is being solved? (we always want our experiments to benefit users)
• Screenshots & videos: What does the current user experience look like?
• Experiment idea: What are the details of the proposed experiment’s design, setup, and goals?
• Hypothesis: What will the experiment result in, and how will it significantly improve key metrics?
• Opportunity size: How many users per day will see the experimental feature?
• Estimated impact: Based on the opportunity size, what is the expected range of impact?
• Investment: How much engineering/design/etc. time would be required to invest into this idea?
• Precedence: Are there previous related experiments that our company or other companies have run? What can we learn from them? If we’ve run this type of experiment before, why will it work differently now?
• Recommended Rating: Based on all of the above, should we work on this experiment immediately, put it in the backlog, or not prioritize it?

During EIR, you should plan on spending around five minutes for presentation and five minutes for discussion per idea. Leads should give extensive feedback about the idea, and the discussion should conclude with action items. Different teams can use different rating systems, but in general each rating should match an action item. For example, good action items are “work on immediately”, “move to backlog”, and “deprioritize”.

Best Practices

We’ve run into a few issues making EIR effective, and we’ve developed a few key best practices in response to them:

Problem: People don’t bring ideas
Solution: Create a schedule with assignments for specific people to bring ideas on specific dates. Managers need to set expectations that finding great ideas is required to succeed on a growth team. Consider pairing new and experienced team members so they can work on ideas together.

Problem: People don’t know how to come up with ideas
Solution: Teach people during onboarding how to find ideas through competitor audits, user flow walkthroughs, metrics deep dives, and examining past experiment results. Build a library of high-quality idea docs to serve as examples for new team members to learn from.

Problem: Idea quality is lacking
Solution: Note that this is expected when ramping up an EIR process. To accelerate quality improvement, leads should give significant feedback after every idea (there’s a feedback guide later in this post to help with what kind of feedback to focus on). Also, leads should emphasize that one good idea will generate more impact than ten bad ones and will take much less time to implement as well. It’s also important to set expectations around the amount of prep time that should go into researching an idea; typically, we recommend spending between 30 minutes and two hours prepping an idea for EIR.

Problem: Idea document is not detailed enough to give a rating
Solution: Have the leads collect the idea one-pagers ahead of the meeting so they can scan through and spot if any ideas are lacking sufficient detail. If they find any, ask the authors to flesh their ideas out, and reschedule the ideas for review in a later EIR meeting to avoid spending review time twice for the same idea.

Problem: Good ideas are not acted upon
Solution: For really strong ideas, assign owners to work on them right there in the meeting.

Feedback Guide

Here are some common feedback areas for helping team members improve the quality of their experiment ideas:

• Opportunity size too low: If too few users see an experimental feature, the impact will be limited.
• Impact estimate unlikely: It’s common for people to overestimate how much impact a proposed change may have.
• Precedent not listed: New team members usually aren’t familiar with past experiments, so it’s good to share precedent learnings early and often.
• Precedent listed without discussion: “Why will it work differently now?” should be answered for any listed precedents.
• Investment size incorrect: People tend to underestimate how long projects will take. There is significant team overhead for each experiment, so you can’t just consider development time.
• Ratings too lenient: Look at your past experiments and divide the amount of big wins (not including shipped but small win) by the total number of experiments. It’s likely a low percentage, which should encourage people to be critical about what experiments to commit time to.

For more information, watch the talk Jeff Chang gave at the SF Growth Engineering Meetup about bottom-up growth teams and experiment idea review:

Our Results

After running the EIR process for over a year, we’ve seen the following benefits across our Growth team:

• Depending on the subteam, 50–100% of experiments are ideated by non-lead team members
• We’ve scaled our Growth team to 100+ members while continuing to increase the overall experiment impact of each subteam
• New team members are more effective since they’re able to more quickly learn about what the team has tried in the past and own new experiments with increased confidence.

Experiment Idea Review has helped the Pinterest Growth team scale its impact, and team members have enjoyed increased agency in deciding what projects they work on while also having more direct influence on their own personal impact and career growth. If you are an engineer, PM, analyst, or designer who wants to join a high impact bottom-up growth team, please check out https://pinterest.com/careers.

Thanks to the many people on the growth team who helped test and improve this process!