How Pinterest Supercharged its Growth Team With Experiment Idea Review

Jeff Chang, John Egan | Growth Engineering

Growth teams need to be organized bottom-up to scale well

The Pinterest Growth team has over 100 members, and we’ve run thousands of experiments over the years. It’s difficult to run that many experiments and still maintain a high success rate over time. We’ve found the traditional growth team model of team leads deciding which opportunities to try didn’t scale well as our team grew. Increasing the number of high-quality ideas ready for experimentation is one of the biggest levers for increasing the impact of a growth team, but our leads have less and less time to research and find great opportunities as we continued to scale the team.

To address this, we changed the structure of our Growth team to be bottom-up, meaning everyone — engineers, product managers, engineering managers, designers, etc. — is expected to contribute quality experiment ideas. However, this model immediately runs into the reality that finding great growth opportunities is a rare skill that usually only comes with experience. We developed Experiment Idea Review (EIR) as a way to quickly train team members to find high-quality growth opportunities and continuously build our idea backlog. The rest of this post describes how to run EIR effectively.

Image for post
Image for post
An inversion of the traditional growth team model for experiment ideation. Dotted lines represent the ongoing growth of the team. Enabling everyone to generate high-quality experiment ideas scales much better than the top-down model.

Experiment Idea Review Guide

What is Experiment Idea Review?

In a nutshell, EIR is a recurring meeting to pitch ideas. It has two main goals: (1) train team members how to come up with quality ideas and (2) build a sufficient backlog of high quality experiment ideas. EIR can be run at various intervals (e.g., weekly, monthly) depending on how many quality ideas the team needs for a healthy backlog, but the more practice everyone gets, the better they become at originating high-quality ideas.

It’s important to note EIR is not a brainstorm. At Pinterest, we historically used brainstorms to generate growth experiment ideas. We would get everyone together in a room and have each person come up with ideas off the top of their head, which we would then try to feed into our roadmap. However, we’ve learned simple brainstorms tend to only be effective when the goal is to come up with solutions to very specific, scoped problems (e.g., the email open rate for dormant users is good, but the clickthrough rate is low — how might we solve that?). People are able to contribute much more when given sufficient time to research ideas ahead of time versus just coming up with ideas in the moment.

Experiment Idea Review Process

Before an EIR meeting, team members come up with ideas and spend up to an hour filling out an experiment idea doc. A well-written document quickly provides readers with enough detail to make an educated guess of the idea’s potential value, so the doc should be as detailed as possible.

An experiment document should contain the following components:

  • Problem statement: What user problem is being solved? (we always want our experiments to benefit users)

During EIR, you should plan on spending around five minutes for presentation and five minutes for discussion per idea. Leads should give extensive feedback about the idea, and the discussion should conclude with action items. Different teams can use different rating systems, but in general each rating should match an action item. For example, good action items are “work on immediately”, “move to backlog”, and “deprioritize”.

Best Practices

We’ve run into a few issues making EIR effective, and we’ve developed a few key best practices in response to them:

Problem: People don’t bring ideas
Solution: Create a schedule with assignments for specific people to bring ideas on specific dates. Managers need to set expectations that finding great ideas is required to succeed on a growth team. Consider pairing new and experienced team members so they can work on ideas together.

Problem: People don’t know how to come up with ideas
Solution: Teach people during onboarding how to find ideas through competitor audits, user flow walkthroughs, metrics deep dives, and examining past experiment results. Build a library of high-quality idea docs to serve as examples for new team members to learn from.

Problem: Idea quality is lacking
Solution: Note that this is expected when ramping up an EIR process. To accelerate quality improvement, leads should give significant feedback after every idea (there’s a feedback guide later in this post to help with what kind of feedback to focus on). Also, leads should emphasize that one good idea will generate more impact than ten bad ones and will take much less time to implement as well. It’s also important to set expectations around the amount of prep time that should go into researching an idea; typically, we recommend spending between 30 minutes and two hours prepping an idea for EIR.

Problem: Idea document is not detailed enough to give a rating
Solution: Have the leads collect the idea one-pagers ahead of the meeting so they can scan through and spot if any ideas are lacking sufficient detail. If they find any, ask the authors to flesh their ideas out, and reschedule the ideas for review in a later EIR meeting to avoid spending review time twice for the same idea.

Problem: Good ideas are not acted upon
Solution: For really strong ideas, assign owners to work on them right there in the meeting.

Feedback Guide

Here are some common feedback areas for helping team members improve the quality of their experiment ideas:

  • Opportunity size too low: If too few users see an experimental feature, the impact will be limited.

For more information, watch the talk Jeff Chang gave at the SF Growth Engineering Meetup about bottom-up growth teams and experiment idea review:

Our Results

After running the EIR process for over a year, we’ve seen the following benefits across our Growth team:

  • Depending on the subteam, 50–100% of experiments are ideated by non-lead team members

Experiment Idea Review has helped the Pinterest Growth team scale its impact, and team members have enjoyed increased agency in deciding what projects they work on while also having more direct influence on their own personal impact and career growth. If you are an engineer, PM, analyst, or designer who wants to join a high impact bottom-up growth team, please check out

Thanks to the many people on the growth team who helped test and improve this process!

Pinterest Engineering Blog

Inventive engineers building the first visual discovery…

Pinterest Engineering

Written by | Inventive engineers building the first visual discovery engine |

Pinterest Engineering Blog

Inventive engineers building the first visual discovery engine, 200 billion ideas and counting.

Pinterest Engineering

Written by | Inventive engineers building the first visual discovery engine |

Pinterest Engineering Blog

Inventive engineers building the first visual discovery engine, 200 billion ideas and counting.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store