How machine learning significantly improves engagement abroad
By: Yunsong Guo | Pinterest engineer, home feed
Last year Pinterest became a truly global company, with more than half of people using our app outside the U.S. We had invested heavily in making Pinterest personal, fast and reliable for everyone, regardless of language, location or device, which resulted in major retention and engagement gains around the world. For instance, using machine-learned models we increased the number of localized Pins for countries outside the U.S. by 250 percent over the past year. Now each of the more than 150 million people who visit Pinterest monthly see Pins most relevant to their country and language, among other signals. Ultimately, machine learning has allowed us to extend our systems to better understand international content and Pinners, and localize experiences for accelerated growth.
While there are many teams and projects that contributed to our internationalization efforts, in this post, we’ll cover how machine learning improved the home feed experience for Pinners abroad and led to the biggest gain in home feed engagement last year.
The Pinterest home feed
We wouldn’t see any gains if we didn’t first start with one of the highest trafficked parts of the service–the home feed. Each of our 150 million active Pinners are presented with a personalized home feed each time they visit, which requires machine learning application to constantly improve the relevance of ideas. (For more details on the machine learning problem set up, training instance generation, earlier features and a linear model we used in home feed ranking, check out this previous blog post.) Over the years we’ve evolved our ranking models from logistic regression to support vector machine (SVM), and now to gradient boosted decision trees (GBDT). We’ll go through three major improvements in our modeling and feature engineering pipelines in the following sections.
From linear models to gradient boosted decision trees (GBDT)
One central problem we faced was providing Pinners with an endless stream of relevant and engaging content, personalized to their tastes and interests. Home feed addresses this problem by using machine-learned models (aka Pinnability) to rank a large set of candidate Pins based on predicted relevance and presents the most relevant ones. The accuracy of relevance prediction by the Pinnability models is, in part, key to the success of Pinterest. Before, we only used linear models, either logistic regression or SVM ranking with a linear kernel function, to build relevance prediction models. When a relatively small number of features (<100) were used, the linear models were sufficient to explore the feature space and generate a good prediction outcome. However, as we added more features to Pinnability, the linear models were quickly reaching their limits in exploring the ever-complex feature space, so we applied more complex modeling techniques.
From our A/B experiments, we found gradient boosted decision trees (GBDT) provide the best engagement results, so we fully productionized them in home feed early last year. The GBDT models help Pinners discover more relevant, personalized ideas in their home feed. As a result, the number of people saving Pins from their home feed every day increased by more than 10 percent, which was the largest home feed engagement gain last year. The GBDT models especially improved the number of relevant Pins people outside the U.S. discovered and saved by up to 18 percent, depending on the country. We use Apache Hive and Cascading to generate training data and xgboost to train our GBDT models offline on a single AWS r3.8xlarge instance. For larger training data sets, we developed the capacity to distributively train GBDT models on our in-house Spark cluster. Now, we use more than 700 dense features and some sparse features in our production models consisting of 350 trees of depth seven.
Language and country match features
At first our models didn’t contain specific features for Pinners outside the U.S., so people with similar interests from different countries used to get the same relevance prediction score. To improve this experience, we started a project to localize home feed. As you’d expect, Pinners outside the U.S. generally prefer Pins with text in their own language, according to our research. We define a Pin’s language and country to be the main language and country of the external web page the Pin was saved from. We developed accurate language detection models based on the text of the external page and added tens of country/language match features such as whether the Pinner’s language and country are the same as a Pin’s language and country, and if the Pinner’s language is among the top X languages spoken by others who’ve saved the Pin. In total we defined ~40 language- and country-based match features, and added them to our GBDT models. With these model features, we’ve considerably increased the number of Pins saved per international Pinner by 10 to 20 percent, depending on the country.
Another important metric we measure is home feed localness, the percentage of Pins in home feed that are in the Pinner’s first language. Since we measure both localness and the number of people saving Pins, we sometimes observe a tradeoff between more people saving Pins and fewer localized Pins, especially for languages that have relatively fewer candidate Pins to rank. For instance, one production ranking model increased the number of people saving Pins by 4 percent while dropping localness by 8 to 10 percent. We addressed this trade off by developing an adaptive training process for GBDT described in the next section.
Adaptive training of GBDT models
In our training dataset, both Pins saved and Pins clicked are positive examples while impressions that don’t lead to a save or click are negative examples. With a logistic loss function, we use weighted GBDT training by xgboost. We developed an adaptive training process that specifies different instance weighting schemes for different trees during training. For example, in training we could specify that the weight of each instance of a Pin saved is twice as much as a clickthrough instance for the first 100 trees. Then, for the next 100 trees, the weight of each instance of a Pin saved globally is 3x that of a Pin saved in the U.S. We call this process of setting different instance weight for different trees “adaptive GBDT training” (as illustrated in Figure 1).
In practice, we apply three different weighting schemes to generate 350 binary trees in our prod GBDT models. We first train 150 trees with high Pins saved weights, and then use this 150-tree model as a base to train another 100 trees with refined weight settings for Pins saved and clickthroughs. Lastly we add 100 trees with a significantly higher weight of positive international instances, which include Pins saved and clickthroughs by any Pinner outside the U.S. Each tree is generated with 80 percent randomly sampled training instances. For best engagement gains, we gradually decreased the learning rate across the three adaptive training stages.
We found that properly re-weighting instances in the adaptive training process significantly improved the experience for Pinners. Compared to our previous GBDT model, this adaptive training process alone increased the number of people abroad saving Pins from their home feed every day by more than 10 percent.
We’re continuously experimenting with new ways to improve the relevance of the more than 10 billion recommendations we make to Pinners every day, and machine learning models are at the core. In addition to GBDT, we’ve started developing TensorFlow-based Deep Neural Network (DNN) models for home feed relevance ranking. If you’re interested in solving these machine learning challenges, join us!
Acknowledgements: this is joint work with Dmitry Chechik, Xin Liu and Mukund Narasimhan.
We also thank Xiaofang Chen, Derek Cheng, Crystal Lee, Stephanie DeWet, Chris Pinchak, Wangfan Fu, Yun Chi, Jiajing Xu, Grace Huang and Dan Frankowski for fruitful discussions and suggestions.
