Automating Shop the Look on Pinterest

Kunlong Gu | Engineer, Discovery

Every day, people come to Pinterest to discover new ideas — and when they find a product they want to buy, it should be easy to purchase it. In home decor specifically, millions of people come to Pinterest to find inspiration. That’s why in 2017, we launched Shop The Look as a way to help Pinners get to the products they love faster, and for brands to put customers on a path to purchase. Shop the Look identifies specific objects in a larger scene with white dots that connect the Pinner to the checkout page.

We previously experimented with a human-in-the-loop approach to match product links with Pins, but needed a better way to scale across the billions of images we show Pinners. As a solution, we used computer vision to fully automate the process of matching products to scenes. Today, we’re announcing a fully automated experience on home decor scenes on iOS, with Android coming soon. This update increases our Shop The Look Pin coverage by 22.5x across billions of Pins and products, and in early testing has already lifted engagement by 7%.

Here, we’ll share how we leveraged computer vision and the dataset unique to Pinterest (175B+ Pins saved with user-added context) to automate this experience.

From Pin to purchase

Behind the scenes

The data collection stage prepares the data that our machine learning models use for training, while the ML modeling stage prepares our models to identify and localize home decor objects in a scene. This process classifies objects into product categories and represents them with embeddings, which is a digital representation of images so that visually similar images have a shorter distance between their embeddings compared to dissimilar images. (You can find more in our recently submitted paper.) Finally, the serving stage uses our trained model to identify the product category from the query Pin (‘the look’), and then uses visual embeddings to find the closest product candidates to ‘the look’.

Pinterest’s unique dataset

ML Modeling

We also trained an Embedding Model to represent images as mentioned above (which we will talk about in an upcoming blog post — get excited!). The embedding learns from visual similarity across Pin images. In particular, we added this high-quality dataset of 270,000 scene-product matching pairs so the embedding can handle the domain shift from scenes to product images. The domain shift means the objects in the scenes are of various lighting conditions, rotation, and noisy backgrounds while product images usually contain a high-quality front photo with clear white background. Our Embedding Model captures that information well while providing product similarity.

The third model is a re-ranking model that mainly trains on user engagement data and re-ranks the visually similar candidates so the product is optimized for engagement. The re-ranking model also takes in semantic and contextual information, such as the category of the scene image, the boards that frequently include this Pin, and other objects in the scene.


When a user engages on a Pin, we first decompose the scene using the detection model. We restrict the search space by matching the annotation from detection and the category annotations in the shopping corpus. Then we use visual similarity scoring (the distance between embeddings) in the shopping index to generate result candidates. Finally, we apply the reranking model on those candidates to fine-tune the results.


In terms of internal operation, the automation frees people from doing repetitive work so they can spend more time being creative.

In the long term, the scene images are great resources to learn the relationship between objects, i.e. what objects complement each other or go well together in a certain style. We hope to leverage this rich data of object occurrence and build a sophisticated object graph for every object in the world, making Pinterest a personalized stylist for home, fashion and more.

You can expect to see more from us in shopping and visual search in the coming months!

Acknowledgements: Automating Shop the Look is a collaborative effort at Pinterest. Special thanks to Chuck Rosenberg, Andrew Zhai, Dmitry Kislyuk, Raymond Shiau, Eric Kim, Francis Chen, Jeffrey Harris, Angela Guo, Tim Weingarten, Jen Chan, Joyce Zha, Amanda Strickler, and Stephen Carr for the collaboration on this product.