Microsoft & Peking University FaceShifter: High Fidelity, Occlusion Aware Face Swapping

Synced
Synced
Jan 6, 2020 · 3 min read
Image for post
Image for post

Face swapping technologies are something of a double-edged sword in AI research. The ability to realistically switch and manipulate faces presents dangers for misuse in identity theft, fake news and other scenarios; but also widespread opportunities in the billion dollar film, television and computer game industries. Face-swapping’s kinder side has made the tech a popular new tool in the visual and graphic arts communities.

One of the current challenges in SOTA face-swapping is to integrate both realistic and high-fidelity effects — particularly how to extract and adaptively reorganize the identity and attributes of the source and target facial images. A new study from Peking University and Microsoft Research Asia proposes a novel two-phase framework, FaceShifter, that aims for high-fidelity and occlusion-aware face exchange.

Compared to existing face swapping methods, FaceShifter uses much more information from the target image. The model fully and adaptively utilizes and integrates target attributes to generate exchanged faces with high fidelity in the first processing stage.

The researchers proposed a new attribute encoder for deriving multi-level target face attributes and a well-designed generator with Adaptive Attentional Denormalization (AAD) layers to integrate target attributes with identity characteristics display. They also added a second stage to address the challenging face occlusion problem with a new Heuristic Error Acknowledging Refinement Network (HEAR-Net) that can recover anomalous areas in a self-supervised manner without any manual annotation.

Using natural face images from the FaceForensics ++ test images dataset, researchers conducted experiments on FaceShifter and other face swapping tools FaceSwap, Nirkin, DeepFakes, IPGAN, and the latest FSGAN. Human evaluators were asked to choose i) the one having the most similar identity with the source face; ii) the one sharing the most similar head pose, face expression and scene lighting with the target image; iii) the most realistic one. FaceShifter’s results where judged not only more perceptually attractive, but also to have retained a better overall identity display.

Researchers noted that all the other face swapping tools they tested synthesized inner facial areas first then combined that information with the contour of the target face, which could create inconsistent and unnatural appearances.

Moreover, the faces generated by the other methods ignored the shape of the original face, and did not consider key elements in the target image such as lighting and resolution. IPGAN for example has a single-level attribute representation, which degrades resolution and does not accurately retain target face expressions such as eyes closed. The team says FaceShifter solves all these problems.

Image for post
Image for post
Comparison with FaceSwap, Nirkin et al., Deep- Fakes, IPGAN on FaceForensics++ face images.
Image for post
Image for post
Comparison with FSGAN.

The paper FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping is on arXiv.

Author: Herin Zhao | Editor: Michael Sarazen

We know you don’t want to miss any story. Subscribe to our popular Synced Global AI Weekly to get weekly AI updates.

Image for post
Image for post

Need a comprehensive review of the past, present and future of modern AI research development? Trends of AI Technology Development Report is out!

2018 Fortune Global 500 Public Company AI Adaptivity Report is out!
Purchase a Kindle-formatted report on Amazon.
Apply for Insight Partner Program to get a complimentary full PDF report.

Image for post
Image for post

SyncedReview

We produce professional, authoritative, and…

Synced

Written by

Synced

AI Technology & Industry Review — syncedreview.com | Newsletter: http://bit.ly/2IYL6Y2 | Share My Research http://bit.ly/2TrUPMI | Twitter: @Synced_Global

SyncedReview

We produce professional, authoritative, and thought-provoking content relating to artificial intelligence, machine intelligence, emerging technologies and industrial insights.

Synced

Written by

Synced

AI Technology & Industry Review — syncedreview.com | Newsletter: http://bit.ly/2IYL6Y2 | Share My Research http://bit.ly/2TrUPMI | Twitter: @Synced_Global

SyncedReview

We produce professional, authoritative, and thought-provoking content relating to artificial intelligence, machine intelligence, emerging technologies and industrial insights.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store