Thoughtful Biometrics for Facial Recognition — Part 2

This article is second in an introductory series leading up to the Thoughtful Biometrics Workshop 8,10,12 March 2021.

Extracting intrinsic information from faces, such as identity, gender, ethnicity, and age, is a task that humans perform routinely and efficiently. Now, machines are beginning to catch on. The availability of robust, low-cost computing systems has created interest in developing automatic face recognition systems and deploying them in a number of applications, including biometric-based access systems.

A technology once only seen in television dramas — automatic face recognition systems are now deployed and utilized in our daily activities. Commercial applications of automatic face recognition are now abundant, including “tag” suggestions on Facebook, as well as the organization of personal photo collections in Google Photo. Moreover, after Apple’s announcement of Face ID, your face may become the norm for unlocking your phone and for daily payment transactions.

Automatic Face Recognition

Automatic face recognition poses a challenging problem in the field of image analysis and computer vision. Thus, research in face recognition is striving to solve fundamental challenges, such as developing face matching methods that are invariant to age, pose, illumination, and facial expressions. Further, research also seeks to utilize advances in technologies like digital cameras and mobile devices to perform face recognition in new applications and scenarios. Finally, researchers are looking to fulfill the increased demands on security in numerous practical applications where human identification is needed.

Old School Face Representation

To identify a face in a digital image, the face recognition system should automatically find the face in the image (if there is one). The recognition occurs by matching the detected face with the face template in a database. Just as in fingerprints, where ridge details were described in a hierarchical order at three different levels, facial features can also be described in a hierarchical order.

Level 1 features are the facial characteristics that can be observed from the general appearance of the face, such as skin color. Level 2 features are the localized characteristics of the face, such as the shape of the face and the relationship among the facial attributes. Finally, level 3 characteristics are the micro-features that can be useful for the discrimination of monozygotic (i.e., identical) twins, such as facial marks.

Face matching is the process of measuring the similarity or dissimilarity between two face images based on the extracted features. Level 1 face features are quite analogous to level 1 fingerprint features. Hence, level 1 face features cannot accurately identify an individual over a large population of candidates. Similarly, much like level 2 features of fingerprints, level 2 face features are the most discriminative features and are predominantly used for face recognition approaches. There are two broad categories of approaches to match the detected face images: Appearance-based and feature-based methods.

Appearance-based methods consider the global properties of the face image intensity pattern, such as Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), and Independent Component Analysis (ICA). Feature-based methods use local features of the face, such as geometric relations between the facial features and local texture features of the face that are invariant to pose and lighting, such as gradient orientations and local binary patterns (LBP).

Meanwhile, level 3 features contain unstructured micro-level features on the face, including scars and other facial marks. These features have been used, along with level 2 features, to identify monozygotic twins.

Deep Face

Recently, deep neural networks have achieved impressive results for many visual recognition tasks, including face recognition. Neural networks are not new; perceptrons were first developed in the 1950s. However, network models with many hidden layers (deep structures) can now be trained due to better regularization strategies, and the availability of large face databases and processing capabilities. Again, rather than handcrafted features, face representations are learned by deep convolutional neural networks (ConvNets). These are trained to classify identities or verify pairs of face images from large-scale training sets of face images.

However, the success of these deep ConvNets approaches relies on sophisticated learning and large-scale training sets. Therefore, the highly successful face systems that have been developed by Facebook and Google raise the question: did they use my personal images in their training database?

For example, Facebook now has 2 billion monthly users who upload about 350 million photos every day — a “practically infinite” amount of data that Facebook can use to train its facial recognition software, according to a 2014 presentation by an engineer working on DeepFace, Facebook’s in-house facial recognition project. Facebook says publicly it doesn’t have any plans to sell its database directly. However, they used everyone's personal images to train their DeepFace.

Privacy Concerns

Face recognition technology has the potential to improve our lives in profound ways, but a surveillance technology that readily identifies everyone based on his or her face has been taboo because of its radical erosion of privacy. Moreover, unconsented use of individuals' online face images to train algorithms or detect their online presence is another privacy concern. Note that these two concerns are two sides of the same coin; millions of people are uploading and sharing photos and personal information online without realizing how the images could be used to develop surveillance products they may not support and later used without their consent to track them.

Based on current investigations, face recognition systems developers are scrapping images of people’s families, photos from private photo apps or public websites, and using them to build surveillance technologies. To add-in salt to the injuries, hundreds of local and federal law enforcement agencies have started using these companies' technologies in the past year without public scrutiny.

Despite All This, Face is Not the Answer

Whereas facial features are intrinsic properties of a face, the appearance (the textured look) of a face is subject to several factors, including the facial pose (or camera viewpoint), illumination, facial expression, and occlusions (sunglasses or other coverings). In unconstrained scenarios where face image acquisition is not well controlled or where subjects may be uncooperative, the factors affecting appearance will confound the performance of face recognition.

Moreover, there may be similarities between the face images of different people, especially if they are genetically related. Such similarities further compound the difficulty of recognizing people based on their faces.

Then there is the racial bias. In the landmark 2018 “Gender Shades” project, an evaluation was done on three gender classification algorithms, including those developed by IBM and Microsoft. Subjects were grouped into darker-skinned females, darker-skinned males, lighter-skinned females, and lighter-skinned males. Most algorithms performed the worst on darker-skinned females, with error rates up to 34% higher than for lighter-skinned. Independent assessment by the National Institute of Standards and Technology (NIST) has confirmed these studies, finding that face recognition technologies across 189 algorithms are least accurate on women of color.

These shocking results have prompted immediate responses, shaping an ongoing discourse around equity in face recognition. IBM and Microsoft announced steps to reduce bias by modifying testing cohorts and improving data collection on specific demographics. A Gender Shades re-audit confirmed a decrease in Black females' error rates and investigated more algorithms, including Amazon’s Rekognition, which also showed racial bias against darker-skinned women (31% error in gender classification). This issue is also central to the documentary Coded Bias that came out last year.

This result corroborated an earlier assessment of Rekognition’s face-matching capability by the American Civil Liberties Union (ACLU), in which 28 members of Congress, disproportionately people of color, were incorrectly matched with mugshot images.

Though “fair” face recognition is a challenging task, the advanced deployment of deep learning is helping many companies and universities address these challenges, and few of them have able to significantly improve their accuracy and fairness based on the recent NIST face recognition Vendor Test (FRVT) ongoing project.

This article is second in an introductory series leading up to the Thoughtful Biometrics Workshop 8,10,12 March 2021. Additional articles can be found as follows:

• A gentle introduction to Thoughtful Biometrics — Part 1
• Thoughtful Biometrics for Facial Recognition — Part 2 (this article)
• Thoughtful Biometrics for Fingerprints — Part 3
• Thoughtful Biometrics for Iris — Part 4
• Thoughtful Biometrics for Voice — Part 5
• Thoughtful Biometrics for DNA — Part 6
• Thoughtful Biometrics Overview — Part 7
• What is Identity Management — Part 8