Product Management for Relevance

I recently presented to the Twitter product managers on our core recommendation algorithms and how their teams could contribute to improving relevance on Twitter. Below are the key takeaways.

There are many magical only-on-Twitter moments. Our team makes it easier for users to find them, even if they’re not on the app at the right time or following the right people. We algorithmically surface personalized content and people recommendations, and deliver them through ‘MagicRecs’ push notifications, email, and modules across the app.

Context

Recommendations relevance on Twitter is a challenging problem

• Hundreds of millions of Tweets are potential candidates to be recommended at any given time
• 140 characters means sparse content per Tweet
• Relevance on Twitter is directly linked to recency of the Tweet. We need to quickly surface recommendations and don’t have much time to collect engagement signals
• Decision making is complex. Should we send this recommendation or wait for a better one?
• Personalized feature generation, such as finding the number of mutual follows, is very expensive for people who have large follow graphs

Relevance considerations while product building

1. Look for opportunities to structure and label data. We can do this implicitly through encouraging engagement or explicitly by increasing use of #hashtags, photo tagging, or interest selection
2. Build in feedback mechanisms. Help our machine learning models learn faster when our relevance prediction is wrong
3. Consider if your product’s needs bias towards precision or recall, and its tolerance for diversity.

Precision means that recommendations surfaced are relevant to the user. The goal is to only serve relevant recommendations, and reduce false positives (showing recommendations that are not relevant).

Recall means that all relevant recommendations are surfaced to the user. The goal is to show all relevant recommendations to a user and reduce false negatives (not showing something that would have been relevant) .

These concepts are inversely associated. It is relatively easy to surface a lot of recommendations with low precision, as well as be precise if only a few recommendations are shown. The goal is to solve for both, but what we optimize for depends on the product application.

For example, with push recommendations we need high precision since they are invasive. When a user has explicitly shown that they are in an exploratory mood, such as through a Trends deep dive, we can lean more towards recall.

You should also consider your product’s tolerance for diversity, or more exploratory recommendations. For example, if we know that a user is interested in Modern Art, we could also show Modern Architecture candidates and see if the user engages. More diversity helps us learn more about the user, but we tradeoff confidence in our precision.

4. We design the algorithms. Our algorithms are constantly evolving with new hypotheses, learnings, and product needs. If you have thoughts of how we can improve Twitter’s user experience through relevance, come talk to us!

Interested in the intersection of machine learning algorithms and compelling user experiences? I’d love to connect. Find me on Twitter at @kwchang, DMs are open.