TSA missed at least three firearms in 2018: Why AI can help.
Humans aren’t made for baggage screening: the cognitive limitations of X-ray screeners. Part 2 of a series on the challenges associated with X-ray security screening. At Synapse Technology, we’re using AI to make this process more effective and efficient.
Imagine if your job was designed to target all of your mental weaknesses.
For X-ray screeners at airports, this is their reality. Security screening is a critical role to securing airports that relies heavily on human perception, judgment, attention, and decision-making. A task that’s constrained by the limitations of the human perceptual system. A task that can be draining and “mind-numbingly boring”.
And if the role wasn’t challenging enough, doing it without pay during the current government shutdown can exacerbate the conditions that lead to lower officer motivation, which can further decrease the level of security and increase employee turnover in a time where passenger traffic is increasing consistently.
All of these aspects of the role can have drastic consequences as they represent weaknesses that “could potentially be exploited by terrorists or criminals seeking to attack the aviation system”.
In an earlier post we discussed how airport security officers have to maintain a challenging balance between their mission of keeping people safe and the practicality of needing to keep the line moving.
We’re specifically focusing on the screener that has to look at an X-ray of your bag to determine if it contains a prohibited item that cannot be allowed to pass through the checkpoint. Screeners have to do this quickly to keep the flow of passengers smooth; they could catch 100% of threats if they manually searched every bag, but then no one would make it to their flight. So it falls on their judgment to decide which bags to allow onwards, which bags to take a closer visual look at (potentially stopping the belt to prevent the bag from continuing out of the machine), and which bags to pull aside for a manual search.
The role and its limitations aren’t just limited to the 9,000+ security checkpoints at airports around the world, but also to security checkpoints in other environments like office buildings, courthouses, military bases, entertainment venues, and more.
In this post we’ll explore the cognitive factors that negatively affect an X-ray screener’s ability to do their job effectively.
The Very Human Limitations Of Security Screeners
Certain limitations are technological. Some threats could never be spotted by the operator even if they had hours to analyze the image because of the way that X-ray machines capture images.
But there are also very consequential limitations on the human mind that hinder screeners’ ability to stay alert and focused on the task at hand.
These limitations include:
- Time constraints
- Distractions and divided attention
- A long (and growing) prohibited items list
- Mental fatigue
- Low frequency with which dangerous items appear
- Low motivation and emotional exhaustion
Let’s explore just how each of these factors contributes to making the screening task more difficult.
One of the biggest factors affecting screeners is the fast pace at which they need to screen bags. TSA operators have roughly 2.5 seconds before the bag is fully in view, after which another bag immediately begins to appear. The operator can stop the belt to buy themselves more time, but they risk holding up the line like a traffic jam.
Since the impact of most of the factors to follow is an increase in reaction time, they result in a corresponding decrease in detection rate (a lower percent of threats that a screener successfully catches).
Screeners are also in the thick of it, within feet of the passengers they’re screening. These officers need to maintain a focus on the monitor to scrutinize each bag closely, at times needing to zoom in on the bag or use certain filters to analyze the image– actions which cost them precious seconds.
Their eyes need to go back and forth between the two monitors displaying bags. The left monitor showing a “top view” of each bag and the right monitor showing a “side view” of each bag. While having the added view provides additional information that can increase detection rates for difficult objects, it also increases the operator’s reaction time. This can lead to operator missing a threat that was visible in the second monitor that the operator did not have enough time to scrutinize closely. When pressed for time screeners will often look at the “top view” as their primary source of information.
Screeners can also be affected by other variables like noise, draughts, temperature, screen reflection, and poor air quality. I’ve encountered screeners that have even cited bad monitors with poor contrast as having a negative effect on their ability to screen.
This problem of distractions has led some airports to isolate their screeners in the controlled environment of a separate room (a process often called “remote screening”), and even explore if autistic individuals would be more effective at the task.
A long prohibited items list
Research shows that the more objects you have to look for, the more time you need to analyze an image that may or may not contain those objects.
Screeners are mentally scanning through a list of hundreds of items when looking at each X-ray. Since they are constrained by time, a longer prohibited items list leads to lower detection rates. This problem worsens every year as screeners contend with a growing prohibited items list, most recently modified with the addition of powders as a restricted class of items.
Novel or emerging threats can also be added to the officers’ mental checklist based on intelligence, such as a new IED design or a 3D-printed gun that could be concealed in unintuitive ways,
If you haven’t inferred by now, the life of a security screening officer can be demanding and draining. Research shows that the performance of an X-ray screener starts to suffer after only 10 minutes and that performance declines exponentially with increasing time.
Security officers typically rotate between certain roles such as being the X-ray screener, opening and searching bags, helping passengers to place their belongings in the bins correctly, and operating the body scanner. These rotations occur every 15–20 minutes by regulation at most airports around the world. However, at some airports a rush of passengers and shortage of officers can result in a screener staying on for 30, 45, or even 60 minutes at worst.
Security screeners will do several X-ray screening rotations during their 8–12 hour shift. End-of-day shifts are likely to have worse detection abilities as a screener becomes more tired during the course of the day.
Low frequency of dangerous items (sporadicity)
Perhaps one of the interesting characteristics of X-ray screening is the inverted relationship between the threat level of an object and how often it appears. The most dangerous of objects like IEDs and guns rarely appear, and operators inherently are biased (perhaps correctly) towards assuming that such an object will not be present in the next bag.
It’s as if someone handed you a Where’s Waldo book and told you that Waldo may or may not be on any given page. In fact he might not be in the book at all. You might give up rather quickly if you had the choice, especially if you had already spent several hours staring at similar images.
This prevalence effect has dangerous consequences — “miss” rates are higher for objects that appear less frequently. Also, if there is a relatively benign item like a liquid present in a bag, a screener may flag it and move on without detecting an explosive that might have also been concealed elsewhere in the bag. Relatively benign objects like liquids can appear rather frequently, so screeners are biased to look for prohibited items they know are likely to be present. Items are also likely to be less threatening and less consequential.
And if a screener misses a dangerous threat it’s possible that no one, including the screener, would ever know. Unless the passenger self-reports or the threat was part of a test. This means that we’ll never truly know how many firearms or other threats might have been missed at security checkpoints.
There are some existing methods to address the prevalence issue. A technology called Threat Image Projection (“TIP”) periodically projects a gun, explosive, or other threat onto a random bag. There’s a good chance that your bag at some point may have looked to a screener as if it had a gun inside! These images keep officers alert to these types of items. But TIP also has its own drawbacks — operators often report that TIP can be slow, obvious (e.g. a giant gun projected in the middle of a tiny bag), and stale (TIP libraries are not refreshed often). TIP is not a silver bullet for tackling the prevalence effect.
Motivation (or lack thereof)
The role itself does not lend itself to positive reinforcement. Screeners are rarely, if ever, rewarded or recognized for finding threats and they may be reprimanded for missing something important in a covert test. Or, as we’ve encountered, screeners may feel a perpetual anxiety that they might miss something important.
In previous years, Bloomberg BNA reported that Homeland Security ranked in last place among federal agencies to work for.
These aspects of the job can have a demotivating effect on screeners, who can suffer from emotional exhaustion, low job satisfaction, and low motivation. Decreased motivation can have a detrimental impact on their screening ability; if screeners are not motivated they will miss threats.
They’ll also be more likely to quit their job. Some airports analyzed by BNA experienced a turnover of 30–80% across five years. In the TSA, annual attrition is about 10–20%, lower for full-time employees and higher for part-time employees. At some international airports attrition can be as high as 30%, and in non-airport environments (such as courthouses or office buildings) as high as 50%.
All of these constraints lead to a role that sets itself up, to some degree, for failure. Airport security administrators will often say that they have to be right 100% of the time while terrorists have to be right only once.
But 100% is an impossible figure to hit. Security screeners are only human, prone to making mistakes and getting tired among other shortcomings. Humans aren’t wired to stare at a screen for several hours per day, looking for items that may or may not be there. No matter how good new hardware might be, humans will always be limited in their ability to reliably interpret its output.
This is why at Synapse we’re using artificial intelligence to develop algorithms that can automatically detect dangerous threats like guns and knives with greater accuracy and speed than human operators. Algorithms don’t get tired and remain unbiased, allowing them to better catch adversarial examples that a non-alert screener might miss.
Adding an AI-based “second pair of eyes” increases detection rate by catching more threats but also eases the cognitive load on human screeners, allowing them to process bags more quickly and be more effective at their job.
Stay tuned for an upcoming post with more details about how we bolster the human decision-making process and alleviate some of these limitations with modern software.