To tell lies, look into the eyes

Exposing GAN synthesized faces using inconsistent eye specular highlights

Can you tell which are fake? Real human eyes exhibit consistent specular reflections from both eyes, while GAN synthesized ones usually do not. Image source:real image: the Flickr-Faces-HQ (FFHQ) dataset. GAN-synthesized image: http://thispersondoesnotexist.com

GAN models can now be used to create highly realistic synthetic images of human faces down to the minuscule details. Such fake imageries have put our trust to online visual media on the line, corroborated by the recent reports (e.g., from the Verge, CNN, Reuters) of GAN synthesized faces being used as profile images of fake social media accounts. While sophisticated and highly capable, the GAN model only has a limited understanding of what a human face looks like from the numerous face images used as its training data. In particular, it has trouble grasping more abstract concepts about the geometric, physiological, and physical aspects of human faces. These imperfections of the GAN model can be taken advantage of to expose synthesized faces.

Corneal specular highlights for a real human face (left) and a GAN-synthesized face (right). The corneal regions are isolated and scaled for better visibility. Note that the corneal specular highlights for the real face have strong similarities while those for the GAN-synthesized face are different. Image sources: real image: the Flickr-Faces-HQ (FFHQ) dataset. GAN-synthesized image: http://thispersondoesnotexist.com

One such imperfections can be found in the specular reflections of the eyes. The corneas in our eyes are like mirrors — they reflect incoming light from bright objects (e.g., lights, the sun, or any object with a shiny reflective surface) in the environment. The specular reflections in the eyes can be captured in a high-resolution image such as a portrait photograph. When the light sources are fairly faraway from the face so that the different viewing angles of the two eyes becomes negligible, the two eyes see roughly the same scene, and they will exhibit almost identical specular reflections. For the GAN synthesized portrait images, this physical constraint is not respected. What we observe is that the specular reflection highlights from the two eyes in a GAN synthesized image are often incompatible with each other, in terms of the number, location, and geometric shapes.

In our recent work, we explore this visual artifacts of GAN synthesized faces for their detection. We show that such artifacts exist widely and further describe a method to automatically extract and compare corneal specular highlights from two eyes. Our method outputs a numerical score to indicate the consistency of the two specular reflections, with larger values indicating higher levels of consistency (hence more likely to be a real face). We found that roughly 85% of high quality GAN generated faces sampled from http://www.thispersondoesnotexis.com can be exposed with this method.

Consistency analysis of corneal specular reflections for sampled GAN synthesized faces from http://www.thispersondoesnotexis.com. The top row shows the full image, the second row shows the region of the two eyes. The third row is the automatically detected corneal regions (blue circle) and specular reflections from the left (green) and right eye (red). The last row is the IoU score (in the range of [0,1]) between the specular reflections of the two eyes, smaller values indicate lower consistency.

Consistency analysis of corneal specular reflections for sampled real human faces from the Flickr-Faces-HQ (FFHQ) dataset . The top row shows the full image, the second row shows the region of the two eyes. The third row is the automatically detected corneal regions (blue circle) and specular reflections from the left (green) and right eye (red). The last row is the IoU score (in the range of [0,1]) between the specular reflections of the two eyes, smaller values indicate lower consistency.

One can imagine that the inconsistencies of the specular reflections can be fixed with manual processing, but automating the cover-up requires significant changes to the underlying GAN model. Of course, our method has its limitations. Complex lighting environment for real images can cause false positives, especially when the light sources are close to the subject’s face. Nevertheless, it becomes one more hurdle for the GAN models to create realistic forgeries to fool our eyes.