Lessons in growth and increasing signups

Jean Yang | Pinterest engineer, Growth

On the Growth team at Pinterest, we’re constantly brainstorming ways to attract new users. Over the past year we ran a series of A/B experiments which resulted in a significant increase in daily signups and doubling of conversion rates. Here’s the story of what we’ve learned and what’s next.

Why focus on signups?

Pinterest’s mission is to help people discover things they love and inspire them to do those things in real life. The core features of the service, such as Pinning, following boards, messaging, and recommendations only work when we can uniquely identify the current user. Therefore, we focus a lot of our efforts on guiding unauth (unauthenticated) users to sign up so they can go beyond searching and browsing to access all that Pinterest has to offer.

Identifying opportunity through mistakes

The project started with a serendipitous discovery. In May, we opened up a minor feature that had previously required authentication and saw an unexpected drop in signups. Before the change, unauth users who tried to access the feature saw the following signup screen with no additional context.

Despite the lack of information and design, a good amount of users who saw the screen signed up. We looked into whether those who signed up via this page became engaged Pinners and found that the retention rate was on par with the overall retention rate.

These findings inspired us to experiment with adding contextual signup prompts to different parts of the site.

Understanding unauth users

While we value signups, we also want to avoid alienating visitors who may not be ready to sign up yet.

We came up with a set of metrics to ensure we didn’t compromise the user experience in a way that would damage our long term growth potential.

In addition to login and signup rates, we looked at the following:

• Unauth engagement (how many actions an unauth user performs during a session)
• Unauth retention (what % of unauth users return to the site within a week)
• Signup retention (what % of users who sign up return and perform core actions)

Testing different variables

We found it’s okay to become slightly more aggressive as we learn more through experiments and can provide a good user experience. We also learned that the color and text used in the prompts matter.

We ran many experiments to test calls to signup at different points in a typical user flow. A few of our experiments tested similar variations on the same signup prompt.

1. Non-dismissible signup modal

2. Dismissible signup modal with small “x” in corner

3. Dismissible signup modal with small “x” in corner and “Skip for now” button

The first treatment earned 2–3x more signups than the 3rd treatment without reducing retention or engagement significantly. We learned that a bit of an extra nudge to sign up works well if placed at a point in the flow where you’ve already demonstrated enough product value to users.

Color matters

In one experiment, we found white text on a black background outperformed black text on a white background. A simple switch in color scheme netted us 10% more signups daily.

(1)

(2)

Text matters

Changing text can yield large gains, but it can be challenging to predict success, which is why we A/B test everything, such as:

Of these, (A) performed best. The best performers gained 5%+ daily signups over the worst performers. We found emphasizing that there’s more content available worked better than focusing on text around discovery, ideas, and inspiration. Explicit calls to sign up worked well too.

Beware of novelty effects

We always want to be wary of novelty effects when running experiments. A new, well-positioned call to sign up may initially capture a lot of users, but the effectiveness will likely decrease with time. For this experiment, we kept holdout groups running for months.

It’s also important to analyze user behavior by looking at trends over time rather than data in aggregate. For one experiment, we calculated the retention rate a few days in and found that the retention rate was higher in the “enabled” group than the “control” group. A couple of weeks later, the overall retention rate for “enabled” was only a few percent lower than that of “control.”

However, when we graphed the signup retention rate of users against signup date, we saw that the difference in retention rates were initially small but widened over time, with the “enabled” group showing lower retention. (Note: Y axis coordinates intentionally removed.)

Trends like this are common with signup experiments. Users who sign up during the early days of an experiment are often those who were likely to join and remain engaged anyway. The easiest converts sign up early, and those who do so later tend to be less engaged. To analyze whether an experiment is successful, we need to understand tradeoffs and long term trends like these.

What’s next?

We’re continuing to test the ways we provide a great user experience for new and existing users. While we’ve increased daily signups and conversion rates, there’s still a lot of work to be done, such as scaling the translation and localization of features. We’ve run 11 experiments with a total of 53 treatment groups for this project so far, with more in the pipeline.

Stay tuned for more from the Pinterest Growth team. If you’re interested in joining us, we’re hiring Growth Engineers!

Acknowledgments: This work was the result of cross-functional collaboration between members of the Growth, Product Design, Writing, Business Analytics, and Web teams.

Jean Yang is a growth engineer at Pinterest.