Engineering Shop the Look on Pinterest

By Wonjun Jeong | Pinterest engineer, Product

This week we announced Shop the Look, a new way to shop and buy products inside home and fashion Pins. With more than 150 million people discovering billions of ideas every month on Pinterest, and fifty-five percent using the app to shop and plan purchases, we wanted to make it easier than ever to go from inspiration and action. Shop the Look combines our computer vision technology with human curation to recommend a variety of related products and styles you can bring to life in just a tap. In this post, we’ll share how we engineered Shop the Look.

Image for post
Image for post

Overview

We built two different technologies to make products inside Pins shoppable. First, we’ll explain how we combined computer vision technology with human curation to help people shop on Pinterest. The second approach ingests content from content networks and automatically makes products inside Pins shoppable.

Image for post
Image for post

Approach #1: combining computer vision with human curation

Leveraging our object detection technology, we built new machine-assisted curation tools that help marry human taste with computer vision. Given a fashion Pin, we use object detection to infer areas of the image that contain products and apply our visual search technology to quickly find and parse through millions of products in order to generate a set of candidate products.

Human curators with fashion expertise confirm the most relevant products within the candidate. We can then recommend these products in Shop the Look.

Image for post
Image for post

Approach #2: Ingesting third party data

To make Shop the Look more scalable, we built a feed ingestion system for content networks. Now we can easily ingest external feeds containing image links, product links and XY coordinates.

Our daily workflow uploads feed files to S3 parses and validates the feed. We save any errors back to S3 to notify brands and networks. With the processed feed, the system then creates Pins on behalf of the content network and matches their products with Pins people can buy on Pinterest. After, products are tagged inside the Pin using the XY coordinates within the feed. Our jobs are idempotent and can be rerun at any point without leading to duplicate content or causing the feature to break.

On occasion brands or networks don’t provide XY coordinates. This was a challenge we solved using our object detection technology. First, we detect the objects and coordinates within the source image. We then cross-match the annotations with the same object detection performed on the product and assign the product an XY coordinate on the source image.

Image for post
Image for post

Data model

We built Shop The Look using a graph-based model. To represent the tag relationship, we created a node called PinTag which contains the XY coordinate and an edge with a given fashion or home Pin. We also created edges from PinTag to different Pins with similar products. Here’s an example:

Image for post
Image for post

The graph-based solution allows us to build even more useful features:

Image for post
Image for post

Infrastructure

We built this on top of a specialized Zen cluster we call Zen Canonical Pin. Zen is our in-house graph-based storage system, and a Canonical Pin is a data model representing Pins that share the same link and image hash (like the Pins above). Zen Canonical Pin gives us two key benefits.

Once the data is stored, it’s ready for serving. We built Shop the Look for all our platforms–iOS, Android and web.

Image for post
Image for post

What’s next

Early tests show Pinners engage with Shop the Look Pins 3–4x more than Pins without Shop the Look, save Shop the Look Pins 5x more and visit a brand’s website 2–3x more. Now that we’re rolling out this experience to everyone, we have a few upcoming initiatives:

Shop the Look is now available to Pinners in the U.S. We hope you give it a try!

Acknowledgements: This project was a cross functional effort in collaboration with Rocir Santiago, Joyce Zha, Rahul Pandey, Casey Aylward, Coralie Sabin, Cheryl Chepusova, Cherie Yagi, Kim O’Rourke, Anne Purves, Tiffany Chao, Aimee Bidlack, Aya Nakanishi, Tim Weingarten, Adam Barton and Yuan Tian. People across the whole company helped launch this feature with their insights and feedback.

Pinterest Engineering Blog

Inventive engineers building the first visual discovery…

Pinterest Engineering

Written by

https://medium.com/pinterest-engineering | Inventive engineers building the first visual discovery engine | https://careers.pinterest.com/

Pinterest Engineering Blog

Inventive engineers building the first visual discovery engine, 200 billion ideas and counting.

Pinterest Engineering

Written by

https://medium.com/pinterest-engineering | Inventive engineers building the first visual discovery engine | https://careers.pinterest.com/

Pinterest Engineering Blog

Inventive engineers building the first visual discovery engine, 200 billion ideas and counting.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store