What Pinterest’s ‘Complete the Look’ Means for Ecommerce

The creation of ‘shoppable scenes’.

Pinterest is set to take a very significant step forward in the race for visual search supremacy.

A team from Pinterest, Stanford, and UC San Diego has submitted a paper to the Computer Vision and Pattern Recognition conference with the snappy title, ‘Complete the Look: Scene-based Complementary Product Recommendation’.

In the briefest of terms, the paper outlines the following:

  • A new tool, Complete the Look, will recommend products that complement the context of a picture.
  • The ‘context’ of the picture can mean variations in lighting, decor, location, or even the subject’s body shape.
  • Using Convolutional Neural Networks, Complete the Look can learn ‘compatibility’ from past images and use this as the basis for a compatibility scoring mechanism when new images are introduced.
  • Complete the Look is a response to the nature of the images we share on social media. A lot of revealing information is missed if we focus only on products.
  • This builds on Pinterest’s Shop the Look, which identifies products contained in an image and returns results for visually similar items.
  • Shop the Look has allowed users to buy what they see. Complete the Look crops the main product from the image and searches using the ‘scene’ instead.
  • The tool can act as a virtual stylist for ecommerce brands. A user can submit a photo of a location, for example, and Complete the Look will help identify relevant outfits for them to purchase.
  • A release date is still to be confirmed.

Shop the Look vs. Complete the Look:

Why Complete the Look Matters for Ecommerce

Online communication is increasingly driven by images.

Instagram Stories has 300 million daily active users, to cite just one photo-focused statistic.

This brings with it a challenge, when compared to text-based posts. It is much more difficult to train an algorithm to identify the topic of a photograph than it is to understand the meaning of a Tweet.

Moreover, these social media photos are rarely of the glossy quality we are accustomed to in magazines or catalogues.

Pinterest splits fashion imagery into two categories: scene images (street photos, selfies) and product images (one image on a plain background).

Ecommerce has typically focused on the latter, but will now be able to build on this platform by turning scene images into a curated list of product recommendations.

As noted in the paper,

Existing work focuses on predicting compatibility between product images (e.g. an image containing a t-shirt and an image containing a pair of jeans). However, these approaches ignore real-world ‘scene’ images (e.g. selfies); such images are hard to deal with due to their complexity, clutter, variations in lighting and pose (etc.) but on the other hand could potentially provide key context (e.g. the user’s body type, or the season) for making more accurate recommendations.

All of these factors contribute to the meaning of an image, but have been notoriously difficult to interpret using neural networks.

In fact, these elements of ‘style’ are slippery even for the most skilled of human observers.

The paper refers to “the overall aesthetics of a room”, a vague phrase that reveals how ill-equipped our language is when it comes to the materially intangible. We can say that a chair is a chair, but we struggle to detail why it adds to the ambience of the room.

This is precisely the point of visual search; it allows for expression through imagery when language is limiting.

Complete the Look comes into its own in the interpretation of those images.

If I search using an image that contains a pair of shoes, I may want to know what brand they are or where to buy a pair. However, if I already own the same pair, I would prefer to know which other items might go well with the shoes.

That subtle distinction is substantive for retailers.

The shift is from ‘similarity’ to ‘compatibility’.

Pinterest highlights the benefits of Complete the Look as follows:

“1) scene images contain not only the fashion items worn by the subject (or user), but also rich context like their body type, the season, etc. By exploiting this side-information, we can potentially provide more accurate and customized recommendations;

2) our system can be adopted by users to give fashion advice (e.g. shoes that go well with your outfit) simply by uploading (e.g.) a selfie;

3) our system can be readily adapted to existing platforms to recommend products appearing in fashion images.”

How Can Shoppers Use Complete the Look?

As an example use case, a shopper could share a picture of a wedding venue on Pinterest and receive recommendations based on the look of the location, what other people have worn there, and the upcoming weather.

This can be blended with other data points, such as the user’s past preferences and the searches of similar users on the platform.

Weighting those varied elements to arrive at a short list of summary recommendations remains a work in progress.

As shown below, Pinterest is training the system using binary questions. A recommendation is either correct or incorrect based on the scene contained in the stimulus photo.

The scene is further broken down into a query category, such as ‘Outerwear’ or ‘Curtains’.

This is a helpful insight for retailers, as the organisation of information within a website hierarchy will play a part in how Pinterest serves results.

Complete the Look will launch on Pinterest this year (date still TBC), and it will pave the way for similar technologies.

Undoubtedly, Google will follow suit soon after Pinterest’s product launches, with ‘Lens the Look’ or similar.

The ability to capture and scale sartorial nous could be transformative, especially for sites with sizeable inventory to scan.

In an age of digital assistance, Complete the Look responds to a real challenge for ecommerce.

Furthermore, the value it adds for consumers could be the factor that moves visual search from niche curiosity to mass-market product.

The Startup

Medium's largest active publication, followed by +479K people. Follow to join our community.

Clark Boyd

Written by

Industry analyst and writer for tech, business, and marketing publications. Tropical fruit enthusiast and walker of dogs.

The Startup

Medium's largest active publication, followed by +479K people. Follow to join our community.