DO MASKS HAVE A REAL IMPACT ON FACIAL RECOGNITION ?
Since Covid-19 has slowly become central in our daily lives, we all (or almost) got used to wearing a mask pretty much everywhere outside our houses. To convey this message, public figures have had to play their part as well and wear masks on media appearances. But what is the impact of wearing a mask on applied facial recognition ?
BE MASKED OR NOT BE MASKED, THAT IS THE QUESTION
November US presidential elections could not be a better example of two different approaches regarding masks on public appearances. It is no surprise that Donald Trump has appeared way less with a mask than Joe Biden. This led our team to wonder if such a difference would have an impact both on recall and precision of our results. Based on 450K minutes of videos processed by our algorithm during the US election, Context has analyzed variations on both candidates to compare results with and without a mask.
It appears that surprisingly enough, the presence of a mask is never a sufficient criteria to challenge our confidence threshold to identify both candidates (and we’ll explain why below) but the confidence score itself is more affected for Donald Trump than Joe Biden. In other words, when Donald Trump wears a mask, our algorithm slightly lowers its confidence score as it is almost unchanged for Joe Biden. A first explanation would be that Joe Biden’s upper facial part is more distinctive than Donald Trump’s. Another one could be that Trump’s mask being bigger than Biden’s and covering a wider fraction of his face, it becomes harder to recognize him.
In both cases, performances are slightly lowered but the overall system and recognition result is resilient. Why is that?
A FACE IS SUM OF INFORMATION
To assess this question, it is essential to understand how the algorithm learns : it does not learn to recognize Donald Trump or Joe Biden based on pictures as a whole but on infinite small criteria on each picture. Throughout this process, the network will make a complete use of any part of each picture from the database to refine its learning. To say it simply, it does not only learn based on the shape of the mouth or size of the nose but on many other criteria beyond basic human points of interest. The distribution of criteria improves the system’s resilience. A face becomes a sum of information, for which only a small fraction of it becomes necessary to ensure clear results. Whether the face is hidden by a mask, sunglasses, a cap or even from side view, the algorithm still has enough information to provide the right answer.
The second reason why masks did not affect that much our results is the amount of data our system is trained with. The bigger the dataset, the more comparison points you get, the more the algorithm gets to be trained on various cases and compensates when part of data is lacking.
In cases when your own data is insufficient, one could think that adding pictures of the celebrity wearing a mask to the database could be a good idea to improve recognition in such a situation. But is it really?
PROVIDING MASKED PICTURES TO IDENTIFY MASKED PEOPLE ?
As it would seem the most logical solution, it is actually the riskiest. Adding pictures of specific people wearing masks to the database increases the risk of introducing what we call a wolf in the database. A wolf is an identity that manages to impersonate another identity in the database. Wolves are successfully imitating other identities and leading to increase false positives in results, one could call them impostors.
What is the impact on results then ? Having just a few masked pictures in your database will increase the chances that any presence of a mask in a video content you’ll process will always give the same celebrity result : the one that has pictures with masks in the database. It is a current problem one could face with celebrities wearing caps or sunglasses if these are not distinctive and always-worn items. A good example of this is the tricky case of guitarist Slash : does wearing sunglasses and a top-hat make you Slash ? And what about cases when / if Slash does not wear both items in public ? Will the AI be able to identify him ?
On can then understand that the non-systematic presence of masks in datasets is an issue. As the algorithm has never been trained on masks, the mask is not an identity marker per say but the network never learnt to ignore it and to consider it as anecdotal.
To make this approach efficient enough, one would probably have to systematize adding masked pictures on all celebrities (or a fair amount) of the database. This would be the only way to “ignore” the masks.
THE EFFORT IS NOT WORTH THE RISK
Overall, taking the chances to enrich your database with masked pictures can only be considered if your original results are good enough. If for a masked Joe Biden, the algorithm does not even provide Joe Biden as the main answer, adding masks to the database will not change the game. In other words, such a solution can only improve results that are already good, not turn bad ones to excellent ones.
Another risk has to be highlighted regarding adding pictures of masked public figures: personalized masked. Let us take an example of a political party branded mask : adding such a mask to a celebrity’s database, Donald Trump’s for instance, creates a risk if any random people, let’s say his supporters, wear the same mask in a video. The singularity of the mask, what makes it different from others, becomes the reason why it is misleading. For this reason, any addition of masked pictures in your database would probably have to be neutral-enough so that it does not create misleading hints on who is behind this mask.
In any case, all the elements present above led us to think that the effort of adding masked pictures to the database was not worth the risk, especially when our algorithm was able to maintain enough confidence in the results at this stage.
Last month, Andrew Maximov, a Belarusian technologist living in Los Angeles posted a video on YouTube showing how facial recognition technologies could actually remove masks from faces in images or video content and retrieve the person behind, through social networks profiles. Although it is not proved at this stage that such matches are accurate, this case in point shows that use of masks did not put an end to facial recognition efficiency.
