
Can Artificial Intelligence make you more fashionable? This is the question that Facebook AI Research unit have been working on answering with their new project “Fashion ++”. The system analyses outfit images submitted by users and then makes minimal edits like rolling up the sleeves on a shirt, taking off a scarf or changing the colour of a top to make it more stylish.
To train the algorithm what constitutes good style, researchers used over 10,000 user-submitted street style images from the online fashion site Chictopia. The algorithm can identify visual differences including colour, texture, shape, silhouette and pattern. The researchers then created “unfashionable” looks by digitally replacing one piece of an outfit with the least similar garment. These inputs were used to develop a “fashionability classifier” — essentially a scale that scores a look on stylishness.
Sounds simple right? Not quite, there were many obstacles the researchers faced when building this model. Some challenges included how to model things that are so small and subtle, how do you train a system and teach it these differences between ‘good’ and ‘slightly better’ outfits? How do you capture style in a computational way?”
“Automatically suggesting minimal edits is challenging because the difference between outfits is subtle — often just a few pixels, which is difficult to capture with traditional computer vision models,” says Kristen Grauman, who is one of the researchers that worked on FAshion ++. “Furthermore, ideal training data would consist of curated pairs of better and worse versions of the same outfit, but that would be much too time-intensive to gather manually. Our approach takes steps to address both issues.”

The main idea is an activation maximization method that operates on localized encodings from a deep image generation network. Given an original outfit, we map its composing pieces (e.g., bag, blouse, boots) to their respective codes. Then we use a discriminative fashionability model as an editing module to gradually update the encoding(s) in the direction that maximizes the outfit’s score, thereby improving its style. The update trajectory offers a spectrum of edits, starting from the least changed and moving towards the most fashionable, from which users can choose a preferred end point. We show how to bootstrap Web photos of fashionable outfits, together with automatically created “negative” alterations, to train the fashionability model. To account for both the pattern/colors and shape/fit of the garments, we factorize each garment’s encoding to texture and shape components, allowing the editing module to control where and what to change (e.g., tweaking a shirt’s color while keeping its cut vs. changing the neckline or tucking it in). After optimizing the edit, our approach provides its output in two formats: 1) retrieved garment(s) from an inventory that would best achieve its recommendations and 2) a rendering of the same person in the newly adjusted look, generated from the edited outfit’s encodings. Both outputs aim to provide actionable advice.

The team reports that in a human perceptual study involving over 100 test outfits and nearly 300 people, 92% of respondents judged outfits as more fashionable after Fashion++ made changes to them. Furthermore, 84% said they thought already-fashionable outfits modified by Fashion++ were similarly or more fashionable.
Generally speaking, AI has been slow to make inroads in subjective fields like fashion, says Devi Parikh, an associate professor at the Georgia Institute of Technology and FAIR researcher. “Creativity is often something that we think of as innately human — that is what makes us intelligent — so it’s a technical challenge to think about how we can get machines to think in the same way,” Parikh says. “Fashion is very subjective, but machines can lead to seeds of inspiration.”
While Fashion ++ is a research project, its easy to see its commercial value and how it may be used in Facebook gadgets like the Portal. For everyday consumers, recommendations for how to edit an outfit would allow them to tweak their look to be more polished, rather than start from scratch or buy an entirely new wardrobe. For fashion designers, envisioning novel enhancements to familiar looks could inspire new garment creations.
The full Fashion++ research paper is available here
