Algorithm Clones Facial Expressions…And Pastes Them Onto Other Faces

Want to paste your expression onto a picture of your boss? Your wait is over…

One of the great barriers to remote communication is in conveying facial expressions. Imagine a realistic avatar that represents you in video calls, online chats and web conferences. This character appears in all these environments, adding a realistic presence to your voice.

But here’s the thing: there is simply no way of reading your expression and then recreating it accurately on an avatar or on another face. Animators have the same problem in recreating actors’ expressions on cartoon faces. Consequently, expressions in animated films have to be manually controlled and ordinary video calls are doomed to a dim future of badly lit faces in cluttered bedrooms or boardrooms.

If only somebody could solve the problem of expression cloning…

Enter Yihao Zhang at Shanghai Jiao Tong University in China and a few pals who have created an algorithm that clones facial expressions and pastes them onto other faces. The work raises the prospect of accurately reproducing facial movements and expressions on avatars, cartoon characters and more or less any face.

This is not the first time that computer scientists have attempted to paste an expression from one face to another. But previous efforts have never been entirely successful.

The problem arises because these algorithms measure the way a face distorts when it changes from a neutral expression to the one of interest. They then attempt to reproduce the same distortion on another face.

That can work if the two faces have similar features. But when the faces differ in structure, as most do, this kind of global distortion looks unnatural.

Take, for example, a person with a small mouth and a big smile. This requires a significant distortion which may be inappropriate for many other faces. “If we apply this smile expression to a target person whose mouth is already large in the neutral face, the global warping method may lead to an unnatural “super large” mouth,” say Yihao and co.

What’s more, a change in expression uses a different set of muscles to move the face and this alters the topology of the skin surface, creating dimples, wrinkles and so on. These change the balance of light and shadow on the face and this cannot be reproduced by an algorithm that simply distorts the overall shape of the face.

Yihao and co take a different approach using two different techniques. The first involves the kind of warping already tried but on a local scale rather than an entirely global one.

The algorithm divides the face into regions such as the eyes, mouth, nose etc and measures the distortion in each of these areas independently. In particular, it measures any change in the height-to-width ratio of each facial feature.

It then applies the same warping to each part of the target face. And to ensure that this warping is realistic, it constrains the changes using the measured height-to-width ratio. That ensures that the facial feature never becomes too large or small in relation to the rest of the face.

The second technique produces accurate shadows on the distorted face. To do this, Yihao and co have created a computer model of the facial muscle groups involved in different expressions and the way these change the topology of the face. The algorithm works out which groups have been used to create a certain expression and then adjusts the shadows in the distorted face accordingly.

The results are impressive. Yihaos and co show how their new algorithm can clone expressions and paste them accurately on to other faces creating and compare it to existing methods. “Experimental results show that our proposed algorithm outperforms the existing methods,” they say.

That raises the prospect of a new generation of online communication in which avatars can accurately represent your facial expressions as you speak. It should also be a boon for animators wanting an easier way to make cartoon characters more like the actors who voice them.

It could also be entertaining. Yihao and co use their algorithm to paste expressions on to other people and even to a face in an oil painting (top image). There’s surely fun to be had pasting expressions onto pictures of your friends, significant other, boss etc. Those long winter evenings will just fly by.

Ref: arxiv.org/abs/1503.00088 : Facial Expression Cloning with Elastic and Muscle Models

Follow the Physics arXiv Blog on Twitter at @arxivblog and on Facebook