NIST joins “proceed with caution” chorus on AI

It’s 2024, the year after a year of nonstop wrestling with the question of whether artificial intelligence (AI) will help the world, damage the world, take over the world, save the world, destroy the world, or any number of other options. And if you listen to enough different expert predictions, you’d probably figure the answer is, “All of the above.”

Or, with a bit more humility and realism, “We really don’t know yet.” Just as we didn’t know exactly what the internet, the personal computer, the smartphone, social media, and other tech innovations were going to do when they came on the scene decades ago.

Which may be a large part of the reason for the tone of a policy paper published earlier this month by the National Institute of Standards and Technology (NIST) titled “Adversarial Machine Learning (AML): A Taxonomy and Terminology of Attacks and Mitigations.”

The paper is one of a series on what NIST calls “trustworthy and responsible AI,” which to some critics might sound like a laughable contradiction in terms. But while the paper goes into exhaustive technical detail on AML threats and possible mitigation of those threats, the overall message is to proceed with caution — that while AI offers obvious benefits, it also comes with obvious risks and threats. So don’t get reckless with it.

Because as is the case with every technology, AI is being used for both good and evil.

And on that front, you could also argue that the NIST guidance is a bit late to the party. Experts have been warning of the dangers of AI since well before the first version of Open AI’s ChatGPT was launched at the end of November 2022.

• Seven-plus years ago, a survey of AI researchers found that more than half believed that sometime relatively far in the future — 25 years or more — AI models could surpasses human intelligence and become an existential threat to humanity through the ability to control or manipulate human behavior, resources, and institutions.
• The Future of Life Institute famously published an open letter last March calling on AI labs to “immediately pause for at least six months the training of AI systems more powerful than GPT-4.” Among more than 33,700 who signed it were X (formerly Twitter) owner Elon Musk, Apple cofounder Steve Wozniak, and former presidential candidate Andrew Yang. The letter didn’t lead to a pause but has reportedly generated much debate and public unrest about the implications of the technology.
• Paul DelSignore, a writer for The Generator, warned just weeks ago in a post on Medium of “an unprecedented amount of identity theft, scams, and fake news” through AI-generated deepfakes. He created his own deepfake video with an AI service that needed only two minutes of real video of himself to create “an instant avatar that you can use to say anything you want, and in any language.”

Sure enough, a couple of days before the New Hampshire presidential primary, a number of voters complained that they had received a robocall with a voice that sounded like President Joe Biden urging them not to vote in the primary because, “Voting this Tuesday only enables the Republicans in their quest to elect Donald Trump again. Your vote makes a difference in November, not this Tuesday.”

The spoof was relatively easy to detect, but experts agree that such things are only going to get better, and therefore harder to detect.

The NIST paper doesn’t address every possible malicious use of AI but it declares up front that “AI systems can malfunction when exposed to untrustworthy data, and attackers are exploiting this issue,” and that “no foolproof method exists yet for protecting AI from misdirection”

As Apostol Vassilev, NIST computer scientist and one of the paper’s authors put it, “available defenses currently lack robust assurances that they fully mitigate the risks. We are encouraging the community to come up with better defenses.”

It would be good if AI experts heeded that call to action, because those risks are huge. This post will focus only on one of the two broad classes of AI systems — generative AI (GenAI) — because it is used in software development (the other class is predictive AI). And as we all ought to know by now, software is not only eating the world but running it. Meanwhile, most estimates are that more than 100 million people and organizations are now using or experimenting with GenAI tools.

A standard lexicon

To begin, NIST recommends “standardized terminology in AML to be used by the ML and cybersecurity communities. In other words, let’s use not only the same language, but the same jargon.

“This is useful — establishing the lexicon early allows for better communication which of course facilitates better outcomes from a functional and security perspective,” said Emile Monette, director of government contracts and value chain security with Synopsys, adding that it will “also aid with efficiency in the process.”

Then, as the title states, the NIST paper provides a taxonomy of AML attack techniques, which in the GenAI category include “evasion, [data] poisoning, privacy, and abuse attacks,” along with “attacks against available learning methods (e.g., supervised, unsupervised, semisupervised, federated learning, reinforcement learning) across multiple data modalities.”

Among the goals of attackers are to compromise the availability, integrity, and privacy of an AI system. Another is what NIST calls “abuse violations,” in which attackers gain control of GenAI models and then use them “to promote hate speech or discrimination, generate media that incites violence against specific groups, or scale offensive cybersecurity operations by creating images, text, or malicious code that enable a cyberattack.”

Beth Linker, director of product management with the Synopsys Software Integrity Group, said that “building out a taxonomy of these attacks is great. I like that they broke it out into predictive AI and GenAI,” adding that “understanding the generative taxonomy will help with safe adoption of GenAI platforms.”

Monette agrees, noting that “knowing the most common attack vectors can help developers ensure their code is addressing these kinds of threats.”

But he adds that whatever guidance the good guys get is going to be scrutinized by the bad guys too. “Whenever you publish a list of bad things you should mitigate, you’ve also given bad actors a roadmap for the things you’re not mitigating, or at least not prioritizing in mitigations,” he said.

Porous foundations

Another factor that may be making AML attacks easier is that many organizations are using so-called “foundation models” made by third parties to train the large language models (LLM) used to write software code.

That saves a lot of time and labor — instead of each organization doing its own “data collection, labeling, model training, model validation, and model deployment… in a single pipeline,” using one from a third party means developers just have to “fine-tune it for their specific application,” according to NIST.

But the Berryville Institute of Machine Learning (BIML), which announced the release of a research paper this past week titled “An Architectural Risk Analysis of Large Language Models: Applied Machine Learning Security,” warned of “serious consequences to relying on big tech’s black box LLM foundation models, which are quickly being adopted as critical corporate business tools by technology executives who have no understanding of how the tech actually operates and what the consequences are to their company.”

According to BIML cofounder Gary McGraw, the black box LLMs the group researched “had more than 80 inherent risks, including issues not previously revealed publicly.” In a press release, McGraw wrote that “to ensure the safe and secure use of the technology, the development of black box LLM foundational models should be regulated to require AI vendors to provide transparency into how they are constructed, the data they are constructed from, and how they work.”

“We are big fans of AI and machine learning,” he wrote, “but we are concerned about whether the current generation of LLMs are built with security in mind.”

Secure your software

Another proven way to mitigate the AML risk is to build security into software. It’s crucial to do so because GenAI code has become a fourth component of software projects, added to the three established components: the code you wrote, the code you bought, and open source code.

The obvious benefit of GenAI is that it can help software developers write code much faster than a human, or even a group of humans. But even if an organization doesn’t use a third-party foundation model, LLM code creates a new attack surface.

So it’s important that developers don’t fall for the mistaken perception that AI produces clean code. Like any junior developer, it doesn’t. AI code needs to be tested like any other code by multiple automated tools —at a minimum static analysis, dynamic analysis, and software composition analysis to find bugs, potential licensing conflicts and other defects that have plagued software since its beginning.

Indeed, the latest Open Source Security and Risk Analysis (OSSRA) report by Synopsys found that codebases still contain numerous vulnerabilities and licensing problems.

Of the 1,067 codebases scanned for the report, 96% contained open source. And of the 936 codebases that underwent a risk assessment, 84% had at least one vulnerability, and 74% had at least one high-risk vulnerability. There were 53% that had license conflicts and 91% contained components that were 10 versions or more behind the most current version of the component, with, with 49% that had no development activity within the past two years.

Obviously if GenAI tools are learning from existing codebases, it is a guarantee that they will bring these defects into generated code.

Jamie Boote, senior consultant with the Synopsys Software Integrity Group, said the NIST paper, “like most NIST guidance, is well thought out and actionable for certain audiences. For cybersecurity practitioners, it provides a valuable starting point to begin probing for weaknesses that are unique to AI/ML systems.”

But he also noted its limits — that when AI and ML functionality move into end user devices like smartphones, banking apps, and workplace software, “the guidance we security professionals provide to them will differ from what NIST provides to its audience.”

“Instead of telling end users how to thwart model poisoning attacks or membership inference attacks, the security industry will have to inform users how to properly vet the software they interact with or what data to withhold from untrusted AI/ML systems.”

Indeed, withholding data is one of the best security measures an end user can take. Data that’s never provided is data that can’t be compromised.