The Algorithm That Sees Beauty in Photographic Portraits

Scientists at Yahoo Labs have developed a machine learning algorithm that distinguishes beautiful portraits from the not-so

Beauty is in the eye of the beholder. But what if the beholder is a machine? Today, Miriam Redi and pals at Yahoo Labs in Barcelona say they’ve created a machine learning algorithm that can pick out “beautiful” photographic portraits from…err…not so beautiful ones.

They say their algorithm is significantly better than other machine-based attempts to recognise beauty and provides some curious insights into the nature of photographic portraiture.

Redi and co begin with a set of 10,000 photographic portraits that have been rated for beauty by humans on a scale from 1 to 10 and annotated with words and phrases that describe each image. These ratings provide a kind of gold standard of aesthetic judgement that any algorithm must aspire to.

The team first analyse each image using face recognition software that determines the age, sex and race of the main subject in each portrait along with the position of the face worked out from the relative coordinates of the eyes, nose and mouth.

They then created algorithms that measured how well each image achieved standard photographic techniques of composition, exposure and lighting. These included the arrangement of objects within the image, the distribution of lighting within the image and the sharpness of the picture. They also measured the exposure quality, contrast and jpeg compression along with the contrast between the face and the background.

Finally, they used the human annotations to classify the emotional polarity of each image, whether positive or negative and assessed the originality of the image compared to others.

The next stage was straightforward number-crunching to determine which of these metrics correlated with the human beauty rating. And the results make for interesting reading.

The good news for many of us is that that the visual qualities of the person in the photographic have little impact on the beauty of the resulting portrait. “We find that race, gender, and age are largely uncorrelated with photographic beauty,” say Redi and co.

Instead, many of the factors that do correlate with beauty relate to the technical quality of the image. “Aesthetic score is related to sharpness of facial landmarks, image contrast, exposure, homogeneity, illumination pattern, uniqueness, and originality,” they add.

Indeed, the single most important factor is the sharpness of the image. But other important factors include the contrast between the face and background. Curiously, exposure quality is negatively correlated with beauty suggesting that photographers can create beautiful images by playing with under and overexposed images.

All this leads to a clear strategy for training a machine learning algorithm to recognise beautiful portraits — just look for those images with sharply focused faces with a high contrast between the face and background.

And that’s exactly what the Yahoo team has done. They trained an algorithm to spot beautiful portraits in this way and say it works with a high degree of reliability on a large number of test images. “We built a classifier that is able to successfully distinguish between beautiful and non-beautiful portraits,” they say.

That’s interesting work that provides an automatic way to pick out beautiful images and also throws some light onto the way humans judge beauty objectively. “Physical/demographic properties such as gender, eye color, glasses, age, and race show very low correlation with image beauty, suggesting that any subject, no matter his/her traits, can be part of a stunning picture,” conclude Redi and co.

The number crunching also reveals some interesting curiosities about the nature of portraits. For example, pictures of females tend to be more memorable, as well as brighter and post-processed, while pictures of males tend to have darker colors. What’s more, men smile less than women in portraits.

It’s not hard to think of ways that Yahoo might use this kind of technology. Imagine an album that automatically picks out the best portraits of you and your family. Or a camera that has this algorithm built in, like the smile detectors common in today’s point-and-shoot models.

The good news is that no matter how bad your pictures today, this algorithm and others like it should be able to help you improve them in future. And if not, there’s always Photoshop!

Ref: arxiv.org/abs/1501.07304 : The Beauty of Capturing Faces: Rating the Quality of Digital Portraits

Follow the Physics arXiv Blog on Twitter at @arxivblog and on Facebook