Trueface Achieves Most Accurate Face Recognition for Masked Faces at Biometric Rally
U.S. Department of Homeland Security (DHS) Maryland Test Facility Biometric Rally 2021
The results are finally in for the 2021 Department of Homeland Security (DHS) Biometric Technology Rally. As promised in our previous blog post, we trained a face recognition model which is optimized for masked face images. Trueface, a Pangiam company, ranked as the #1 most accurate face recognition software for masked faces. In the mask face category, Trueface achieved an overall matching-focused True Identification Rate (TIR) of 98.4%. When face masks were removed, Trueface was able to achieve 100% accuracy across all genders, races, and skin tones, as summarized below:
- 100% TIR for Male and Female demographic groups
- 100% TIR for Asian, Black, and White demographic groups
- 100% TIR for Darker Skin and Lighter Skin groups
Competition Overview
The 2021 Biometric Technology Rally evaluated state-of-the-art biometric identity verification systems for high throughput use cases. Unlike other facial recognition benchmarks like NIST FRVT, whose tests are run only on datasets of existing imagery, the MdTF Rallies collect data in controlled environments meant to reflect the real world. Examples include kiosks, e-gates, and other low friction image-capture systems. Results are therefore much more akin to real-world environments vs results generated in a lab/sandbox.
This year, the rally focused on testing how well biometric systems worked in two key areas:
- in the presence and absence of face masks
- for people in different demographic groups
It is also worth mentioning that the application process is highly competitive; only a handful of the best biometric providers in the industry are selected to participate.
Results Breakdown: Masked Performance
One of the key findings of the report is that, on average, biometric matching systems experienced a 9% reduction in TIR when masks are introduced, identifying only 86% of all test subjects successfully. The Trueface algorithm, which has been trained to work even when the face is partially occluded, only experienced a 0.9% accuracy reduction.
Results Breakdown: Operational Focus vs Matching Focus
Additionally, the Rally sought to measure not only matching systems’ accuracy for self-reported gender and ethnicity of volunteers but also what, if any, were the measurable effects of acquisitions systems on overall performance. The Maryland Test Facility describes the difference in measurement results below:
Operational Focused True Identification Rate (TIR) — the performance of an acquisition and matching system combination.
Matching Focused True Identification Rate (matching-TIR) — the performance of an acquisition and matching system combination, discounting any failures of the acquisition system to submit images for matching.
As can be seen in the left graph above, Trueface achieves a perfect 100% TIR for every gender and ethnicity (with the Ouray acquisition system). Since 2017, we have made it our mission to minimize the implicit bias in facial recognition models (explore our journey to parity of performance with our self-reported Fair Face challenge results here and here). These results are an exciting acknowledgement that we are on the right track to achieving our goal of performance parity.
The graph on the right plots Trueface’s TIR for each ethnicity and gender inclusive of acquisition system failures. We applaud the Maryland Test Facility for testing both matching and acquisition systems together in real-world situations, shedding light on the fact that external factors clearly affect TIR.
We firmly believe that facial recognition will only be whole-heartedly adopted when the benefits of the technology can be realized by all, and these results prove we are on the right track to achieving equitable performance. We’d like to thank the Maryland Test Facility for including Trueface in these tests and look forward to next year’s rally.
Acknowledgment: “This publication is based upon work conducted under the U.S. Department of Homeland Security Cooperative Research and Development Agreement №20-TCBI-013.”
Disclaimer: The views and/or conclusions contained in this document are those of the author(s) and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Department of Homeland Security (DHS), and do not constitute a DHS endorsement of the equipment tested or evaluated.
