User retention at Pinterest: tackling an ambiguous problem
Revant Kapoor | Technical Lead, Engagement Growth
User retention is an important but nebulous idea. It’s hard to acquire users and even harder to help them feel at home through an engaging and relevant experience. So when some of those users start to leave the platform, your first thought is to re-engage them. It’s important, though, to understand what they liked about Pinterest and help them reconnect with the platform. As Pinterest has gone global with hundreds of millions of Pinners, getting this understanding has become harder. Here I’ll share how we created a framework to make progress on this large and ambiguous problem.
Finding inspiration
As the growth team at Pinterest grew we naturally started focusing on more sophisticated problems like retention.
JK, nothing happens this way! Even though Pinterest now has 600+ engineers, we still operate like a startup. Engineers have the freedom to identify problems, propose solutions and have impact. I raised the opportunity with a few other engineers and started working on it!
I previously worked at Highlight, which was acquired by Pinterest in 2016. When I joined the growth team at Pinterest, we were focused on deepening the engagement of Pinners. Looking at the scale of Pinterests data I thought this would be a great opportunity to use machine learning for churn prediction (the ability to predict when a user is likely to drop off our platform). Once we could predict churn, we could intervene and prevent at least some from doing so. While this actually solved an important problem, in reality, it was the passion for using machine learning to do something interesting that made me pursue this particular idea. We ran a number of experiments targeted toward those with low engagement and who were likely to become dormant. Over time, we learned this was actually too late! Many of them had been “power Pinners” whose engagement had steadily declined over time. What if we were to intervene earlier in their “downgrade” journey to prevent churn? This planted the seed for “downgrade prevention” (a.k.a. user retention).
In a company, inspiration to solve new problems can come from many places — from people excited about using a new technology, personal frustrations people have with the product, or looking at data trends, what competitors are doing, or simply a gut feeling.
If you’re someone who has the inspiration to work on something related at your company, hopefully this framework will help you make a strong case to your leadership. If you’re in a leadership position, you can point people here to present problems to you in a more structured way.
Defining the problem
Defining the problem might seem like the obvious first step, but even some of the best people can forget to do this.
When defining the problem, the first thought that may come to mind is probably, “what are the potential ways we can keep Pinners engaged?”. We do need to address the “what”, and our brain often races to this question, but first we need to answer who we’re solving for, and next, how will we know when the problem is solved (with a measurable success criteria)?
In our case, we defined the target population as: those who visited Pinterest at least six weeks in a quarter but have since reduced their visitation frequency by at least three weeks. So, if a user visited for seven weeks the first quarter of the year but only four in the second, that user would be considered a downgrade. We had to make a lot of decisions to come to this definition, including which visitation rate we should consider (weeks per quarter or days in a month) and the threshold for downgrade (one week, three, or something else?). We then defined our success criteria as: reduction in downgrade rate (If every quarter X% of users downgrade, we would like to reduce that rate by Y%).
This is probably the hardest step when you’re starting with an ambiguous problem, but it’s worth spending time here before rushing into the “what”, and doing so with a group. When we started, we looked at three different definitions of the same thing and had discussions with analysts, data scientists, user researchers, and other engineers and product managers. Eventually you’ll need to work across teams to solve this problem, so don’t be shy in involving people early.
Is the problem worth solving?
Once you’ve defined the problem, the most important thing to answer is the magnitude. In a company there are typically only two reasons to invest in an area: paranoia or opportunity. Paranoia is simply what happens if we don’t solve the problem and opportunity occurs when solving a problem opens a new area of growth. In either case you have to link the problem you’re solving to the topline or bottomline of the company.
In our case it was paranoia. When it comes to visitation rates there will typically be those who upgrade and those who downgrade. And while overall visitation on Pinterest had been continuously growing, we needed to understand what happens to downgraded Pinners in the future. Is it just natural for Pinners to downgrade and then come back later, or is it a one way street? We crunched some numbers and realized a significant portion of those who downgraded eventually churn from the platform (downgrade is monotonic). We realized if we didn’t do anything, a percentage of these Pinners would never come back, which would be a massive missed opportunity because they had previously been engaged.
Once paranoia kicked in, we still needed to set a goal and convey the opportunity. We had a team goal of reducing downgrade rate, and also matched it to the company’s top line metrics. Often when you’re dealing with a new problem you won’t have all the data you need, and so rough calculations are okay. Feel free to make assumptions as long as you have a precedent for them.
Finding some wins
After doing everything above you can now come to the “what” part of it. A key strategy of our growth team is to work in an incremental manner. Even though the strategy work is long term, it’s important to be able to show progress with small wins. In other words, build an MVP, not a finished product. Typically it’s a good idea to start with something that’s worked before.
The bare minimum we needed to run experiments was to at least have a way of targeting the downgraded Pinners. We could target those who had already downgraded but that would only include those who had already made their mind. We ideally wanted to capture Pinners before they had completely downgraded. Fortunately we had worked on churn prediction before, so we built a downgrade prediction machine learning model to test things out.
While we were building the model we also wanted to figure out what to do for these Pinners once we could target them. We thought it would be best to ask them, as they surely would know why they were using Pinterest less. So we did an in-product survey for a sample set, and found the most common reason for not using Pinterest was that they got busy. We were now back to the drawing board and realized these Pinners needed more relevant content that made the experience seem additive and good use of time. We revisited one of our most successful experiments from the past — showing a topic picker with recommended content. Once you’ve shown some success, you can invest more time and resources into a problem.
Identifying a repeatable process
In any new problem space, it’s easy to find initial quick wins, but finding repeatable tactics that build toward a long-term strategy is harder. In order to continue investing in a strategy you need to figure out how to keep reaping gains from it, by finding out areas (not particular solutions), you can work on.
The first step was to get a deeper understanding of our user population. We analyzed data to compare Pinners who downgraded to those with a similar past visitation who didn’t downgrade. For example, we looked at a cohort who visited X weeks in Q1 and compared the behavior of those who downgraded in Q2 to those who didn’t. This analysis helped us learn more about the problem areas and form a strategy. We came up with 3–4 different hypotheses for why these Pinners downgraded and then brainstormed to bring in people from different backgrounds and schools of thought. The important thing is to execute on ideas that will help you test each of the hypotheses first. Hopefully 1–2 of those strategies work out, and if they don’t, at least you have some learnings you can apply into re-tackling the problem.
The best outcome is success, but the worst outcome should be to have a more informed opinion about the problem.
P.S. If you like solving such problems, we’re hiring.
Acknowledgements: I’d like to thank Phoebe Liu, Avantika Gomes and Ramki Venkatachalam for working with me as we figured out some of the details and the rest of the engagement growth team for continuing to forge ahead on this.
