Exploring effective user signals
Sophia Feng | Software Engineer, Growth
As a visual discovery tool, one of our key areas of focus is providing users with relevant recommendations at the right time. This means delivering personalized ideas regardless of interest or location for each of our 200M monthly active Pinners, and using signals to make their experience even better. For example, if a Pinner follows the topic “cookies,” it’s likely they’re interested in baking, and so we can make recommendations for other things to bake. Alternatively, if a Pinner hides recipes with meat, we can use that signal to recommend more plant-based recipes.
In late 2017, the Growth Activation team (responsible for new user activation) doubled down on better understanding signals to improve the new user experience. Since, we’ve launched more than 20 experiments across our platforms. Through the success and failures of these experiments , we’ve learned a few key things about using signals that we’ll share in this post, including:
- Shortening the onboarding flow to provide a quick experience isn’t necessarily better.
- Users are more receptive to sharing information when it’s tied to a clear value (e.g. if we know a user is female, we can provide more relevant results in search and recommendations across areas like beauty and style).
- Minimizing unknown signals improves the overall home feed experience.
- Signal collection is an opportunity to educate users about our product and how the signals will deliver value to them.
Introducing a gender step
The first major signal we focused on was gender, because it plays a role in how our recommendation system decides which Pins to serve. Despite the importance of this signal, we realized that gender was missing from a good percentage of users who signed up with Google accounts.
Our first approach was to add an extra step between Google account authentication and before we actually register the user by moving them into the new user onboarding process.
With this treatment we saw about a seven percent increase in new user activation, which shows the positive impact of having gender signal. Unfortunately, we also saw a 30 percent drop in Google signups. We suspect that with traditional social network sign ups, from a user’s perspective, after authenticating an account it’s expected to land “in the product” and not on a sign up page. In our case the additional step of asking for a user’s gender was not only out of place, but it also was confusing to the user. As a result, users with the intent of signing up dropped mid-process.
Collecting signals at the right time
To avoid sign up drops while gaining increased activation rates, the gender step must trigger at the right time, and we have to educate the user why it’s helpful to share. We iterated by redesigning a completely new gender step as part of the new user onboarding flow after signup. The new user onboarding flow traditionally had two simple steps (i.e. topic selection and browser extension steps), because users tended to drop off during longer onboarding. With the addition of a step, we expected to see a drop off in experiment data.
We weren’t surprised to see increased activation and engagement metrics across the experiment dashboard since the previous iteration proved the importance of gender to personalized content. What was surprising was the 11 percent increase in new user onboarding flow completion despite having added an extra step. From this experiment we learned that timing of signal collection is critical. Out-of-context collection is disruptive to the user’s experience.
Educating users
From previous experiments with Google authentication, we wanted to dig into why a shorter onboarding flow isn’t necessarily better. Specifically, we wanted to learn whether metric gains were primarily from the existence of the gender step (and not necessarily from collecting a new signal) or from the gender signal itself improving a user’s topic recommendations and home feed. We applied the same treatment to users who sign up via Facebook (though we tend to know more about signals relevant to them through the Facebook API). Similar to the Google experiment, we saw an eight percent increase in onboarding flow completion, even though this user cohort had 100 percent gender coverage. This confirmed that the gender step itself has a major impact on the user experience.
To help educate users about why it’s helpful to share their preferred gender, the step explains that this signal “helps us show [you] users more relevant content.” Since it’s tied to a clear value, users are more receptive to providing this information. We applied this learning to all other permutations of signup methods (i.e. Facebook, Google, email) and across our platforms (iOS, Android, Web) and saw similar wins.
Next steps
After seeing tremendous success with gender, we asked ourselves which other signals could be helpful for users to share in order to see more personalized ideas right after signing up. We’ve started to explore other signals such as country, locale and age, ensuring that with each signal experience we educate users about the value of providing this information
We continue to see great wins from experimenting with new signals. The increased signals coverage means we have more leverage at personalization beyond topic ranking and ads targeting. Most importantly, our learnings show that we can increase our signals coverage without being disruptive to users. Instead, we can use signals collection as an opportunity to educate users and ultimately make their Pinterest experience more personalized and enjoyable.
Acknowledgements: Special thanks to Shana Hu (designer), Phebe Huang (product manager), Victoria Kwong (software engineer), Yize Li (software engineer), Akul Kapoor (software engineer) and Kennan Davison (intern).
