Artificial Intelligence Can Automate Security Checkpoints

Figure 1: This building utilizes an X-ray machine to scan for dangerous items. The checkpoint is manned by a single trained X-ray operator. What happens when a bag goes through the X-ray machine and he isn’t looking? An AI solution could stop a gun or knife from getting through.

Security checkpoint X-ray machines are pervasive throughout society. Whether it be at airports, schools, office buildings, or amusement parks, X-ray machines are becoming more and more part of daily life.

X-ray machines are used so security operators can understand the content of people’s bags as they enter a secure venue. Bags are placed on a conveyor belt, and the X-ray beams generate a projection of what’s inside the bag. These images are almost always interpreted by trained professionals, such as TSA (for airports) or specially-trained security guards (for other venues). For the purpose of this write-up, we refer to them as “trained operators”.

These trained operators work hard, but the human brain is fundamentally limited in its ability to interpret repetitive visual data. This phenomenon, known as Sporadic Visual Search, shows that as the items humans are looking for (guns, knives, etc) appear less often, they become harder to spot when they do finally appear. This leads to a lot of prohibited items making it through these checkpoints, despite the trained operator’s best efforts to prevent this from happening.

At Synapse Technology, we developed an Artificial Intelligence (AI) platform called Syntech ONE® that automatically identifies threats in these X-ray images. The AI helps operators catch threats that they might otherwise miss, while also allowing them to remain alert to what’s happening at the checkpoint by offloading some of the cognitive load from the operators. The system highlights threats it identifies, such as handguns and knives, to the operator by drawing a box around them. Humans can then confirm the AI’s detection and intervene to remove the threat.

Syntech ONE can be used in venues like courthouses, police stations, schools, office buildings, and amusement parks where the trained operators are primarily worried about catching threats like guns and knives. These operators don’t need to search for dozens of prohibited items in every bag. Instead, they’re looking for a more tightly-defined set of more dangerous items. This is the exact type of security checkpoint where Syntech ONE thrives.

Figure 2: Syntech ONE system installed on a Smiths 6040i. Syntech ONE’s AI interface is on the left screen, while the standard interface is on the right screen. In this case, Syntech ONE identified a hidden 9mm bullet.

We wanted to measure to what extent Syntech ONE improves operator performance at security screening checkpoints. Specifically, we decided to test whether humans who have never seen X-rays before but are aided by Syntech ONE are able to outperform trained X-ray operators. To test this, we designed an X-ray screening simulation experiment. Some experiment participants were given the AI tool while some were asked to find threats as they normally do.

The results of the study were compelling, as we demonstrated that untrained operators aided by Artificial Intelligence were able to outperform trained X-ray operators. Below, we talk more about our methodology and detail our findings*.

Figure 3: Example of a handgun lower receiver and slide. The X-ray on the left contains the slide of a firearm near the bottom. The X-ray on the right contains a lower receiver on the left side of the X-ray. These two threats are extremely difficult for humans to detect, but are able to be caught by AI.

In order to conduct the experiment as realistically as possible, we asked experiment subjects to identify threats in 500 different X-rays. We gave participants a maximum of 8 seconds per X-ray, simulating a real world environment. As per Sporadic Visual Search conditions, 95% of the X-rays did not contain threats. The remaining 5% X-rays contained either a sharp or an assembled, functional handgun.

Figure 4: Adversarial Sharp (left) and Adversarial Firearm (right). The box cutter is concealed under the laptop keyboard. The handgun is in the top right of the bag, angled away from the X-ray emitter.

Of the 25 X-rays containing threats, 10 of these contained what we consider an “adversarial” threat, a very difficult example where the bag owner is actively trying to sneak the gun or knife past the X-ray screener. 15 X-rays contained easy-to-find handguns or sharps (“Non-Adversarial X-rays”) that typically occur when someone accidentally leaves a gun or knife in their bag. The primary variable of interest was detection rate: What percentage of threats were successfully detected by the experiment subjects? Note, in the X-ray image on the left of Figure 4, one can see the fundamental limitations of the X-ray imaging hardware. Even with all of the time in the world, the best human screener and AI would have trouble finding the box cutter.

Figure 5: Non-Adversarial Sharp (left) and Non-Adversarial Handgun (right). The knife, in the top right of the bag, looks like it was accidentally left behind. The handgun is located to the left of the bag. This type of scan would occur when someone forgets their gun in a bag, similar to this news story.

We were able to gather study participants online. The experiment subjects without X-ray experience had backgrounds completely unrelated to security. Looking at the data, the difference in performance between the two types of experiment subjects was clear, as shown in Chart 1.

First, we performed some “sanity check” analyses. Based on common performance metrics, we see the trained operators without Syntech ONE performed as expected as shown in Chart 1*. Non-Adversarial detection rates ranged from 20% to 80% while adversarial detection rates ranged from 0% to 70%.

Chart 1: Experiment results. Untrained Humans with Syntech ONE are ale to detect more adversarial and non-adversarial threats than Trained X-ray Operators without Syntech ONE.

Reassured by our sanity check, we then summarized our results in Table 1. Test subjects who have never seen an X-ray before but were aided by AI were able to far outperform trained X-ray operators.

Table 1: Detection Rate Outcomes. The detection rates for Untrained Humans with Syntech ONE are superior to those of Trained Operators without Syntech ONE. The p-values show the data is statistically sound.

While untrained operators with AI clearly outperform trained operators in non-adversarial settings, it’s amazing to see just how much better the untrained operators perform when the threats are well-concealed within a bag. As part of the data, we also wanted to combine the adversarial and non-adversarial detections to understand the combined detection rate as shown in Table 2.

Understanding how the AI would perform on its own without a human is also of great interest. Interestingly, while the AI alone outperforms a trained operator, an untrained operator with AI outperforms AI alone, as shown in Table 2. This finding is critically important, as it shows how useful it is to have a human in the loop. That human, however, does not necessarily need to go through the timely and expensive certification processes to be effective.

Figure 6: “What’s this over here?” Similar to two humans collaborating, an AI and a human working together delivers superior results.

On its own, an AI isn’t perfect and will miss certain threats, even threats that might be obvious a human. Similarly, human operators have cognitive limitations and miss other prohibited items that the AI might catch. The AI operates like a second pair of eyes that is always consistent, never fatigues, and is always fully alert. The human can work with the AI like a partner to help make the right decisions.

Fundamentally, a human supported by AI makes better decisions. AI in combination with a human is greater than the sum of its parts.

Table 2: Detection Rates across Subject Types and AI. Beyond Table 1, we look at the combined detection rate and the performance of the AI alone.

As we dive into additional data collected from this experiment, we think about how to further improve the artificial intelligence platform. A common performance metric is the number of seconds an operator spends per X-ray image. Chart 2 compares the average time an experiment subject spends per threat versus threat-free X-ray image.

Chart 2: Average time the research subject spent analyzing each X-ray image. In cases where threats are present, both experiment subject types take similar times. In cases where there are no threats, Untrained Operators with Syntech ONE take more time than Trained X-ray Operators without Syntech ONE.

The data in Chart 2 aligns with expectations, but also shows where we can further tune the AI algorithm to increase operational efficiency. When X-ray images contain threats, the average review time per image is relatively similar between experiment subject types. We would expect the time spent per X-ray image when a threat is present and the AI is enabled to be very low, as the AI processes the image in less than a quarter of a second. The data shows that the experiment subjects still take some time to confirm the AI is correct, which is very important as it indicates that the human subjects do not blindly trust the AI. The goal of Syntech ONE is to work in conjunction with the human. In the future, when the AI is confident enough, we can remove the human from the loop and automatically divert bags to be opened up by a security guard.

When the X-ray images are threat-free, we see that trained operators are much faster at analyzing the image. Given that trained operators are better trained at scanning images quickly, it makes sense they are able to make a decision in less time when a threat is not present. Further updates to Syntech ONE can encourage untrained humans to move faster when the algorithm is confident that a threat is not present. We plan to explore this in future updates to the human-computer interaction principles between the algorithm and human.

Overall, our results demonstrate that the latest advancements in artificial intelligence are already powerful enough, in combination with an untrained human, to outperform trained X-ray operators. This finding holds particularly true on adversarial examples, which are the most relevant and realistic in the security industry, as bad actors will continuously innovate to conceal threats.

At Synapse Technology, as we further develop our platforms and roll out improved detection rates, we are excited to offload more X-ray review responsibilities in the future, enabling humans to focus where their intuition is most needed. We fundamentally believe in a future where AI is a force for good, one that will revolutionize the security industry and make the world a safer place.

Any venue that uses X-ray machines for security will be able to increase security and decrease costs with the help of Syntech ONE.

To deploy Syntech ONE at your checkpoint, reach us at info@syntechcorporation.com.

*All X-ray images, detection rates, reaction times, and performance metrics were generated by Synapse Technology utilizing internal resources and publicly available reference materials.