User state-based notification volume optimization
--
Arun Nedunchezhian | Software Engineer, Growth
Notifications play an important role in helping Pinterest’s 335M+ monthly active users discover and engage with inspiring content. It’s crucial to send the right number of notifications with high quality content. If we send too few notifications, Pinners might miss out on great opportunities to discover ideas, whereas sending too many notifications could cause fatigue, and lead them to unsubscribe from notifications completely.
For us, notification budgeting means calculating the optimal number of notifications to be sent in a week for each person, and we train a model to determine that optimal number.
Current system
Our current notification budgeting system is based on “Notification Volume Control and Optimization System at Pinterest”, which was published as a paper in KDD 2018. The system consists of two steps:
- Utility Prediction: Use an ML model to predict a utility value for a given user u
- Budget Allocation: Decide the optimal budget for the week for a given user u
Utility prediction
Let p(a|u, ku) denote the probability of the user ‘u’ performing action ‘a’ this week given notification volume ku. The action ‘a’ can be any metric that we want to optimize. We train a model to predict the probability of action ‘a’ given k notifications for a user u. We make the prediction for values in the range kmin to kmax. From the predicted values, we select the optimal budget ‘b’ in the Budget Allocation step (described below). Our current production system uses daily engagement as the action to maximize for all users since optimizing for it helps us increase our daily active users (DAU).
Daily engagement might be a long shot to reach for low-engagement users. However, converting more low-engagement users to Weekly Active Users (WAU) is a more reasonable goal. The main difference from our existing budgeting system is that we now have separate models for low-engagement and high-engagement users instead of one model for all users. A custom model for low-engagement users with a different metric to optimize will help increase overall engagement for those users underserved by the previous model.
Budget Allocation
The incremental utility of sending the k+1 notification to user u is p(a|u, k+1)−p(a|u, k). In order to achieve the maximum utility with a limited number of notifications, we need to remove the potential notifications with lowest incremental utility.
For each user, we first compute the optimal budget imax in the allowed range [kmin, kmax], and then we gradually increase budget from kmin to imax until the average incremental value of the remaining notifications is below the threshold . The value i, which gives the maximum incremental utility, is chosen as the user’s budget for the week. The algorithm to find the optimal threshold can be found in the KDD’18 paper.
User groups
We want our notifications to increase engagement for users based on their current activity (for example, if they’re returning weekly, our goal is to show engaging content that brings them back daily).
Based on a user’s activity with Pinterest, we assign them one of six possible user states:
The less engaged a Pinner is, the harder it is to develop accurate models, so we have separate models based on how active the user might be.
We cluster the users into two main groups based on either weekly or daily engagement.
Thus based on each user’s level of engagement, we decide which metric to use for the model’s objective function. We then train a gradient boosting decision tree (GBDT) classifier to predict if the user will be active in the next week given a notification budget ‘b’ this week.
Training setup
We frame this as a machine learning problem by having two different objectives to learn.
Although there will be two models optimizing two different objectives, the features used in the current production system are retained. More information about the features used can be found in the KDD’18 paper
The training labels are generated differently even though the same features are used for both models. For WAU, we set the label as 1 if the user has an engagement action on Pinterest at least one day in a week, otherwise 0. For DAU, if the user was active 4 days in a week, we set 4 positive labels and 3 (7–4) negative labels. We then train two separate models for the WAU and DAU user groups using the appropriate labels.
Learning & results
We ran A/B experiments on a significant fraction of users who only subscribed to push notifications.
Our system was able to reduce the volume of notifications sent to low-engagement users who wouldn’t benefit from them without affecting the overall push open rates or any other key metrics. The system was able to send more notifications to users who would benefit from receiving them, increasing WAU counts by 1% in the enabled group. After doing deeper analysis on user state levels, it turns out the system was able to achieve the WAU increase by cutting down notifications to dormant users by 3% and increasing notifications to resurrected users by 17% and marginal users by 7%. Having different objective functions based on user activity states resulted in more efficient notification budget allocation.
Future work
User state based notification volume optimization has yielded very good results in improving user engagement. We’re working on other ways to improve our system, such as:
- Creating additional groups if necessary for optimizing other metrics
- Based on users’ preferred channel (email notifications, Push Notification, or both), moving some user states from WAU group to DAU group
- Considering other metrics like revenue, impressions, clickthrough rate, or engagement to better optimize for highly-engaged users
Acknowledgements:
This is joint work between Yuxiang Wang and Koichiro Narita from the Growth Notifications Team. We would like to thank Bo Zhao for building the first version of the Budgeting System and providing valuable feedback as well as Yunsong Guo for valuable suggestions throughout this project.
