Pinterest’s Email Programs at Scale with the help of ML

John Egan from Pinterest has done a series of very fascinating talks about email and notifications. This blog is an attempt to organize and learn from one of his talks on scaling email operation at Pinterest. This is a great case study on how ML plays an instrumental role in providing value and driving engagement for the end users.

Email & Notifications at Pinterest

At the start: Emails and notifications weren’t really taken seriously at Pinterest.

Today: Emails and notificaitons are responsible for 20% of the company’s daily active users (DAUs)

How John Egan turned emails and notifications became a core growth drivers for Pinterest.

Pinterest is about you, about your interest and your persona taste, discover new ideas.

Basic stats of Pinterest around 2016:

1. >150m users
2. 100b pins
3. 1b boards
4. 80% on mobile, 40% online millennials, >50% international

Two principles that help drive emails and notifications:

1. What you should put into emails and notifications: usually the misconception is that you can put anything into emails and notification but in reality, you should really focus on using emails and notifications as as an extension of the product’s core value & mission.
2. Any notification type can be sent on any channel: any content can be anywhere else. The content can be in all different channels such as email, push etc.

Email/Nootification at Pinterest really boiled down to 3 basic principles:

A. Coverage: These are the things that you can send out to everybody.

It was used to believe that emails and notifications were only reserved to activate less engaged users. Pinterest classifies users by different user states:

Pinterest used to just focus the bottom three with emails but emails and notifications should be used by everybody. Emails and notifications are powerful in helping dormant users become more active, helping an active user become a power user -> amplifying their engagement and increase stickiness.

These are the different content types (Pins, Boards, Pinners, Topics, Searches) send to the users and based on their level of personalization (Personalized, similar, popular, trending, diverse):

In totality, Pinterest had more than 20 types of email campaigns that they could send to the users on any different day.

Given these diverse content types, levels of persoanlization, it becomes critical to figure out what ‘s the right message to be sent to the right channel at the right time.

The evolution at Pinterest started with the business rules. They just need to figure out what’s the highest value content based on the CTR rate. These are the example rules used:

1. Content: Pick notification to send has had the highest click through rate for that user
2. Frequency: Send frequency based on highest 7 day engagement for that user
3. Time: Send at hour that has the highest open rate for that user for that location

After the business rule, the 3 year roadmap for Pinterest became replacing every single one of the business rules with machine learning models and predictions.

The above three highlighted rules then evolved to become:

1. Content: Built models to predict what the users’ CTR would be for the content
2. Frequency: use the same engagement prediction to figure out at what frequency will be the best for that given user
3. Time: Build models to predict at what hour the user will have the highest amount of engagement

First Example of the machine learning experiment the email and notification team did: “Send more of the emails they like”

In this experiment, the team built a model to figure out what content the user likely to engage more with. The biggest take away with this is that this added millions of active users to the Pinterest platform.

Second Example of the machine learning experiment the email notification did: “send fewer of the ones they don’t”

This helps with inbox deliverability because the emails have become a lot more engaging and drives higher click through rate.

B. Conversion: working on copy optimization

Experiment that helps with driving email conversion was dynamic subject lines & copytune.

This tool will take the input copy and automatically outputs copies in each language.

The journey to optimize the content started with thoughts that this was going to be straightforward but it turned out to be a lot more iterations and fine-tuning was required.

Took the winner from the first iteration which added 1% open rate and ran the second iteration that really iterated on the word selections

Again from the second iteration, added all the learning and experimented different combinations which added 11% in open rate (150 million people receiving

The third test iteration really isn’t that different from the first iteration but the micro-optimization was able to move the needle. In some cases, the same type of optimization was able to drive 40% increase in open rate:

C. Quality: These are the determinants of whether the user is going to have an engaged session once they come back into the product again

One of the core mantra for the team was to move away from the editorial emails with a more machine learning driven email. The editorial team used to put together a selection of 10 high level topic areas for 6 different countries. Each week, a lot of effort went into curating these emails. What the Pinterest team saw was that the open rate was ok but the click through rate wasn’t that great. The conversion between clicks and having a quality session after the click in product wasn’t that great either.

From the editorial team, having a single set of content to be sent to millions of users was probably not a viable nor desirable. So the strategy shifted to focus on a single category and expand from there.

The Pinterest team introduced a recommendation email called “Category recommended pins” (CRP). This email drew from millions of categories. After launch, this email not only quickly replaced editorial but also took a life on its own. The roadmap timeline for the CRP email then became an evolution that looks like:

This then gradually grew and grew until becoming the highest volume email category going forward, replacing the previous reining champion for three years.

Here’s the original video for how Pinterest Scaled their email: