Fig. 1: An airport security screener analyzing X-rays of passenger baggage.

Airport screeners missed at least three firearms in 2018: Why AI can help.

Humans aren’t made for baggage screening: the cognitive limitations of X-ray screeners. Part 2 of a series on the challenges associated with X-ray security screening. At Synapse Technology, we’re using AI to make this process more effective and efficient.

Imagine if your job was designed to target all of your mental weaknesses.

For X-ray screeners at airports, this is their reality. Security screening is a critical role to securing airports that relies heavily on human perception, judgment, attention, and decision-making. A task that’s constrained by the limitations of the human perceptual system. A task that can be draining and “mind-numbingly boring”.

And if the role wasn’t challenging enough, doing it without pay during the current government shutdown can exacerbate the conditions that lead to lower officer motivation, which can further decrease the level of security and increase employee turnover in a time where passenger traffic is increasing consistently.

All of these aspects of the role can have drastic consequences as they represent weaknesses that “could potentially be exploited by terrorists or criminals seeking to attack the aviation system”.

In an earlier post we discussed how airport security officers have to maintain a challenging balance between their mission of keeping people safe and the practicality of needing to keep the line moving.

We’re specifically focusing on the screener that has to look at an X-ray of your bag to determine if it contains a prohibited item that cannot be allowed to pass through the checkpoint. Screeners have to do this quickly to keep the flow of passengers smooth; they could catch 100% of threats if they manually searched every bag, but then no one would make it to their flight. So it falls on their judgment to decide which bags to allow onwards, which bags to take a closer visual look at (potentially stopping the belt to prevent the bag from continuing out of the machine), and which bags to pull aside for a manual search.

The role and its limitations aren’t just limited to the 9,000+ security checkpoints at airports around the world, but also to security checkpoints in other environments like office buildings, courthouses, military bases, entertainment venues, and more.

In this post we’ll explore the cognitive factors that negatively affect an X-ray screener’s ability to do their job effectively.

The Very Human Limitations Of Security Screeners

But there are also very consequential limitations on the human mind that hinder screeners’ ability to stay alert and focused on the task

at hand.

These limitations include:

  • Time constraints
  • Distractions and divided attention
  • A long (and growing) prohibited items list
  • Mental fatigue
  • Low frequency with which dangerous items appear
  • Low motivation and emotional exhaustion

Let’s explore just how each of these factors contributes to making the screening task more difficult.

Time Constraints

Fig. 2: Sample X-rays of baggage. There are two firearms and three knives in this image set. In security X-rays metallic objects show up as blue and organic objects like clothes show up as orange. Everything else takes on a greenish color in the middle. Denser objects are more opaque.

One of the biggest factors affecting screeners is the fast pace at which they need to screen bags. TSA operators have roughly 2.5 seconds before the bag is fully in view, after which another bag immediately begins to appear. The operator can stop the belt to buy themselves more time, but they risk holding up the line like a traffic jam.

Since the impact of most of the factors to follow is an increase in reaction time, they result in a corresponding decrease in detection rate (a lower percent of threats that a screener successfully catches).

Divided attention

Their eyes need to go back and forth between the two monitors displaying bags. The left monitor showing a “top view” of each bag and the right monitor showing a “side view” of each bag. While having the added view provides additional information that can increase detection rates for difficult objects, it also increases the operator’s reaction time. This can lead to operator missing a threat that was visible in the second monitor that the operator did not have enough time to scrutinize closely. When pressed for time screeners will often look at the “top view” as their primary source of information.

Fig. 3: An X-ray screener watching the monitors.

Screeners can also be affected by other variables like noise, draughts, temperature, screen reflection, and poor air quality. I’ve encountered screeners that have even cited bad monitors with poor contrast as having a negative effect on their ability to screen.

This problem of distractions has led some airports to isolate their screeners in the controlled environment of a separate room (a process often called “remote screening”), and even explore if autistic individuals would be more effective at the task.

A long prohibited items list

Screeners are mentally scanning through a list of hundreds of items when looking at each X-ray. Since they are constrained by time, a longer prohibited items list leads to lower detection rates. This problem worsens every year as screeners contend with a growing prohibited items list, most recently modified with the addition of powders as a restricted class of items.

Novel or emerging threats can also be added to the officers’ mental checklist based on intelligence, such as a new IED design or a 3D-printed gun that could be concealed in unintuitive ways,

Fig. 4: An example of a subset of prohibited items from a 2013 TSA list.

Mental fatigue

Fig. 5: Exponential performance decline with increasing time. It’s hard to find another task where human performance drops off so steeply. From: Meuter, Renata & Lacherez, Philippe. (2015). When and Why Threats Go Undetected: Impacts of Event Rate and Shift Length on Threat Detection Accuracy During Airport Baggage Screening. Human factors. 58. 10.1177/0018720815616306.

Security officers typically rotate between certain roles such as being the X-ray screener, opening and searching bags, helping passengers to place their belongings in the bins correctly, and operating the body scanner. These rotations occur every 15–20 minutes by regulation at most airports around the world. However, at some airports a rush of passengers and shortage of officers can result in a screener staying on for 30, 45, or even 60 minutes at worst.

Security screeners will do several X-ray screening rotations during their 8–12 hour shift. End-of-day shifts are likely to have worse detection abilities as a screener becomes more tired during the course of the day.

Low frequency of dangerous items (sporadicity)

It’s as if someone handed you a Where’s Waldo book and told you that Waldo may or may not be on any given page. In fact he might not be in the book at all. You might give up rather quickly if you had the choice, especially if you had already spent several hours staring at similar images.

Fig. 6: Can you find Waldo? What if I told you that there was a 0.1% chance he was even in this image at all? (This Image is from a blog post on using computer vision to find Waldo.)

This prevalence effect has dangerous consequences — “miss” rates are higher for objects that appear less frequently. Also, if there is a relatively benign item like a liquid present in a bag, a screener may flag it and move on without detecting an explosive that might have also been concealed elsewhere in the bag. Relatively benign objects like liquids can appear rather frequently, so screeners are biased to look for prohibited items they know are likely to be present. Items are also likely to be less threatening and less consequential.

And if a screener misses a dangerous threat it’s possible that no one, including the screener, would ever know. Unless the passenger self-reports or the threat was part of a test. This means that we’ll never truly know how many firearms or other threats might have been missed at security checkpoints.

There are some existing methods to address the prevalence issue. A technology called Threat Image Projection (“TIP”) periodically projects a gun, explosive, or other threat onto a random bag. There’s a good chance that your bag at some point may have looked to a screener as if it had a gun inside! These images keep officers alert to these types of items. But TIP also has its own drawbacks — operators often report that TIP can be slow, obvious (e.g. a giant gun projected in the middle of a tiny bag), and stale (TIP libraries are not refreshed often). TIP is not a silver bullet for tackling the prevalence effect.

Motivation (or lack thereof)

In previous years, Bloomberg BNA reported that Homeland Security ranked in last place among federal agencies to work for.

These aspects of the job can have a demotivating effect on screeners, who can suffer from emotional exhaustion, low job satisfaction, and low motivation. Decreased motivation can have a detrimental impact on their screening ability; if screeners are not motivated they will miss threats.

They’ll also be more likely to quit their job. Some airports analyzed by BNA experienced a turnover of 30–80% across five years. In the TSA, annual attrition is about 10–20%, lower for full-time employees and higher for part-time employees. At some international airports attrition can be as high as 30%, and in non-airport environments (such as courthouses or office buildings) as high as 50%.

All of these constraints lead to a role that sets itself up, to some degree, for failure. Airport security administrators will often say that they have to be right 100% of the time while terrorists have to be right only once.

But 100% is an impossible figure to hit. Security screeners are only human, prone to making mistakes and getting tired among other shortcomings. Humans aren’t wired to stare at a screen for several hours per day, looking for items that may or may not be there. No matter how good new hardware might be, humans will always be limited in their ability to reliably interpret its output.

This is why at Synapse we’re using artificial intelligence to develop algorithms that can automatically detect dangerous threats like guns and knives with greater accuracy and speed than human operators. Algorithms don’t get tired and remain unbiased, allowing them to better catch adversarial examples that a non-alert screener might miss.

Adding an AI-based “second pair of eyes” increases detection rate by catching more threats but also eases the cognitive load on human screeners, allowing them to process bags more quickly and be more effective at their job.

Stay tuned for an upcoming post with more details about how we bolster the human decision-making process and alleviate some of these limitations with modern software.

Synapse Technology

Artificial Intelligence for a Threat-Free World