12 key risks associated with Generative AI (GAI)

Generative AI feels like magic. You give it a prompt, and it conjures up text, images, or even code that could pass for something a human made. But, as with any new tool, there are risks baked into its design and use. And unlike tools that perform predictable tasks, generative AI operates on probabilities — guessing what comes next based on patterns in its training data. This guessing game is where the risks begin.

NIST AI 600–1 is a profile of the Artificial Intelligence Risk Management Framework (AI RMF 1.0) specifically addressing Generative AI (GAI). The document identifies twelve key risks associated with GAI, including issues like data privacy, harmful bias, and the spread of misinformation. For each risk, it suggests actions organizations can take to mitigate them throughout the AI lifecycle. These actions are categorized according to the AI RMF’s functions (govern, map, measure, manage), providing a structured approach to risk management. The document also explores pre-deployment testing and other methods for assessing GAI risks, emphasizing the need for multi-stakeholder collaboration.

In the context of the AI RMF, risk refers to the composite measure of an event’s probability (or likelihood) of occurring and the magnitude or degree of the consequences of the corresponding event.

What Makes Generative AI Different?

Generative AI doesn’t just replicate patterns; it creates something new. That’s why it’s called “generative.” But this ability to generate comes with a downside: it can’t always distinguish fact from fiction. It’s designed to be convincing, not correct. This is what NIST refers to as “confabulation,” where the system produces something that looks right but isn’t.

Think about it like autocomplete on steroids. When autocomplete guesses the wrong word, you correct it. But with generative AI, the stakes are higher because the outputs can look so polished that they’re taken at face value. That’s a problem when the stakes involve anything from medical advice to news reporting.

Further Reading::

🤖ChatGPT for Vulnerability Detection by Tahir Balarabe

What are AI Agents?

Stable Diffusion Deepfakes: Creation and Detection

The Difference Between AI Assistants and AI Agents (And Why It Matters)

THE KEY RISKS

1.) CBRN Information or Capabilities

Eased access to or synthesis of materially nefarious information or design capabilities related to chemical, biological, radiological, or nuclear (CBRN)weapons or other dangerous materials or agents.

Trustworthy AI Characteristic: Safe, Explainable and Interpretable

2.) Confabulation

The production of confidently stated but erroneous or false content (known colloquially as “hallucinations” or “fabrications”) by which users may be misled or deceived.

Trustworthy AI Characteristics: Fair with Harmful Bias Managed, Safe, Valid and Reliable, Explainable and Interpretable

3.) Dangerous, Violent, or Hateful Content

Eased production of and access to violent, inciting, radicalizing, or threatening content as well as recommendations to carry out self-harm or conduct illegal activities. Includes difficulty controlling public exposure to hateful and disparaging or stereotyping content.

Trustworthy AI Characteristics: Safe, Secure and Resilient

4.) Data Privacy

Impacts due to leakage and unauthorized use, disclosure, or de-anonymization of biometric, health, location, or other personally identifiable information or sensitive data.

Trustworthy AI Characteristics: Accountable and Transparent, Privacy Enhanced, Safe, Secure and Resilient

5.) Environmental Impacts

Impacts due to high compute resource utilization in training or operating GAI models, and related outcomes that may adversely impact ecosystems.

Trustworthy AI Characteristics: Accountable and Transparent, Safe

6.) Harmful Bias or Homogenization

Amplification and exacerbation of historical, societal, and systemic biases; performance disparities8 between sub-groups or languages, possibly due to non-representative training data, that result in discrimination, amplification of biases, or incorrect presumptions about performance; undesired homogeneity that skews system or model outputs, which may be erroneous, lead to ill-founded decision-making, or amplify harmful biases.

Trustworthy AI Characteristics: Fair with Harmful Bias Managed, Valid and Reliable

7.) Human-AI Configuration

Arrangements of or interactions between a human and an AI system which can result in the human inappropriately anthropomorphizing GAI systems or experiencing algorithmic aversion, automation bias, over-reliance, or emotional entanglement with GAI systems.

Trustworthy AI Characteristics: Accountable and Transparent, Explainable and Interpretable, Fair with Harmful Bias Managed, Privacy Enhanced, Safe, Valid and Reliable

8.) Information Integrity

Lowered barrier to entry to generate and support the exchange and consumption of content which may not distinguish fact from opinion or fiction or acknowledge uncertainties, or could be leveraged for large-scale dis- and mis-information campaigns.

Trustworthy AI Characteristics: Accountable and Transparent, Safe, Valid and Reliable, Interpretable and Explainable

9.) Information Security

Lowered barriers for offensive cyber capabilities, including via automated discovery and exploitation of vulnerabilities to ease hacking, malware, phishing, offensive cyber operations, or other cyberattacks; increased attack surface for targeted cyberattacks, which may compromise a system’s availability or the confidentiality or integrity of training data, code, or model weights.

Trustworthy AI Characteristics: Privacy Enhanced, Safe, Secure and Resilient, Valid and Reliable

10.) Intellectual Property

Eased production or replication of alleged copyrighted, trademarked, or licensed content without authorization (possibly in situations which do not fall under fair use); eased exposure of trade secrets; or plagiarism or illegal replication.

Trustworthy AI Characteristics: Accountable and Transparent, Fair with Harmful Bias Managed, Privacy Enhanced

11.) Obscene, Degrading, and/or Abusive Content

Eased production of and access to obscene, degrading, and/or abusive imagery which can cause harm, including synthetic child sexual abuse material (CSAM), and nonconsensual intimate images (NCII) of adults.

Trustworthy AI Characteristics: Fair with Harmful Bias Managed, Safe, Privacy Enhanced

12.) Value Chain and Component Integration

Non-transparent or untraceable integration of upstream third-party components, including data that has been improperly obtained or not processed and cleaned due to increased automation from GAI; improper supplier vetting across the AI lifecycle; or other issues that diminish transparency or accountability for downstream users.

Trustworthy AI Characteristics: Accountable and Transparent, Explainable and Interpretable, Fair with Harmful Bias Managed, Privacy Enhanced, Safe, Secure and Resilient, Valid and Reliable

Why It Matters

Generative AI is powerful because it’s accessible. But that accessibility means its risks scale fast. If a system generates something harmful — whether it’s biased, untrue, or dangerous — the impact can multiply quickly. And because it’s often third parties building and deploying these tools, accountability gets fuzzy.

NIST’s framework is a start. It gives organizations a way to think systematically about generative AI risks. But the challenge isn’t just technical. It’s cultural. Are organizations willing to prioritize safety over speed? Are users skeptical enough to question what these systems produce? The answers to these questions will shape how generative AI evolves — and whether it becomes a tool that serves us or one that harms us.

Building Trustworthy AI

Five Principles of Trustworthy Generative AI (Artificial Intelligence) Models

What does it mean to make AI trustworthy? NIST offers some ideas:

• Safety: Systems should manage risks like harmful content or cybersecurity threats.
• Explainability: Users should understand how and why outputs are produced.
• Accountability: There should be transparency in how systems are built and operated.
• Fairness: Mitigate biases so outputs work equitably for different groups.
• Privacy: Protect sensitive information.
• Security: Defend against attacks and ensure reliability.
• Reliability: Ensure systems perform consistently over time.

Five Principles of Trustworthy Generative AI (Artificial Intelligence) Models

Managing Risks

Managing generative AI risks isn’t one-and-done. It’s a cycle — govern, map, measure, and manage.

• Govern: Organizations need policies to decide when to stop or adjust a system. They also need inventories of where and how these systems are used.
• Map: Understand what the system’s doing and in what contexts it’s being used. This includes documenting everything from data sources to downstream impacts.
• Measure: Choose metrics that matter, whether it’s detecting biases, measuring environmental impact, or evaluating reliability.
• Manage: Respond to risks by improving training data, tightening controls, or even pulling the plug when necessary.

Frequently Asked Questions About Generative AI Risks and Mitigation

1. What are some of the unique risks posed by Generative AI (GAI) compared to traditional AI?

GAI introduces or exacerbates several key risks:

• Confabulation: GAI systems can generate factually incorrect or inconsistent outputs, also referred to as “hallucinations” or “fabrications”, due to their nature of statistically predicting the next word/token in a sequence, rather than relying on established facts. This is especially prevalent in open-ended prompts and expert domains.
• Data Privacy: GAI models can leak, generate or infer sensitive information about individuals, including biometric, health, or location data. This occurs through “memorization” of training data or by combining disparate sources.
• Harmful Bias: GAI can amplify existing societal biases from training data, leading to discriminatory outputs or harmful homogenization, which skews the system or model outputs and can lead to incorrect or biased decision-making.
• Environmental Impact: The extensive computational resources needed to train and operate GAI models contribute to significant energy consumption and related environmental concerns.
• Information Integrity: GAI can lower the barrier to creating convincing misinformation and disinformation, making it harder to discern true vs. false information.
• Offensive Cyber Capabilities: GAI systems can be leveraged to enhance cyberattacks, including automated discovery of vulnerabilities, and generating malware and phishing content.
• CBRN risks: GAI may facilitate the misuse of Chemical, Biological, Radiological, and Nuclear (CBRN) related information through its ability to ideate and design novel harmful agents.

2. What is “confabulation” in the context of GAI, and why is it a significant risk?

“Confabulation,” often called “hallucination” or “fabrication,” refers to the tendency of GAI systems to produce and confidently present outputs that are erroneous, false, or inconsistent with the input or prior statements. This is a significant risk because it undermines the reliability of GAI, leading to potential misinformation and misguided decisions, particularly in critical contexts where accurate information is crucial. It arises from the way generative models are designed: they approximate the statistical distribution of their training data rather than adhering to strict factual correctness.

3. How can bias manifest in GAI systems, and what are its potential consequences?

Bias in GAI systems arises primarily from the data used for training these models. If the training data contains historical, societal, or systemic biases, the GAI model will likely reproduce and amplify these biases, leading to performance disparities across subgroups, perpetuation of stereotypes, and discriminatory outputs. This can result in significant harm to individuals, communities, and society, impacting areas like employment, healthcare, and access to information. Biased image generators can, for example, underrepresent certain demographic groups and create stereotyped output.

4. What are some strategies for mitigating the environmental impact of GAI models?

Mitigating the environmental impact of GAI involves several strategies:

• Model Compression: Using methods like model distillation to create smaller, more efficient versions of trained models can reduce computational resources at inference time.
• Energy Efficiency: Optimizing training processes and using hardware specifically designed for AI workloads can reduce energy consumption.
• Carbon Offsetting: Participating in carbon capture or offset programs can help neutralize the emissions from GAI training and usage.
• Careful Design and Development: Choosing architectures that are less computationally demanding can help reduce overall energy consumption.
• Lifecycle Assessment: Measuring and assessing the environmental impact during all stages of a GAI model’s lifecycle.

5. How can GAI be misused to spread misinformation, and what are the potential impacts?

GAI can facilitate the creation of highly realistic synthetic content, including “deepfakes” (audio-visual forgeries), and written disinformation. Malicious actors can use GAI to spread targeted disinformation, erode public trust in valid sources, and influence public opinion or behavior. The rapid spread of these AI-generated fabrications can cause chaos, as seen with a synthetic image of a Pentagon blast that caused a temporary stock market drop, highlighting the dangers of this technology in the wrong hands.

6. What are some of the security risks associated with GAI systems, and how can these be mitigated?

GAI systems pose multiple security risks including:

• Vulnerability exploitation: GAI can be used to automate the discovery and exploitation of system vulnerabilities, thereby enhancing the capabilities of malicious actors.
• Malware generation: GAI can generate malicious code, and phishing attacks.
• Data Breaches: Sensitive information can be revealed through data memorization or inference attacks.
• Prompt Injection: Malicious inputs can be introduced to force models to generate harmful outputs.

These risks can be mitigated through:

• Rigorous Security Assessments: Regular security testing and red-teaming to evaluate system vulnerabilities.
• Secure Development Practices: Including security considerations during all stages of the GAI lifecycle including supply chain.
• Monitoring and Incident Response: Having systems in place to monitor and respond to security incidents and attacks, and having a plan to halt deployment if negative risks are unacceptable.
• Content provenance: Establishing mechanisms to verify the origin of content (e.g., using watermarks or digital signatures) can assist with tracking and mitigating the spread of misinformation and for tracking intellectual property issues.
• Controlled access: Implementing robust access controls to protect training data and models from unauthorized access.

7. What is “content provenance,” and why is it important for GAI systems?

Content provenance refers to the documented history of digital content, including its origin, modifications, and ownership. It is vital for GAI because it allows for the verification of authenticity, tracking of alterations, and mitigation of misinformation. Provenance mechanisms include watermarks, digital signatures, and metadata. Implementing robust content provenance can help detect manipulated content, enable transparency, and assign accountability. This is important for protecting intellectual property and informing stakeholders of any risks associated with system output.

8. What is “red-teaming,” and why is it important for GAI safety?

Red-teaming involves simulating attacks on a GAI system to identify vulnerabilities and weaknesses. This is a vital method to assess and enhance the safety, security, and trustworthiness of GAI. This process can include techniques such as adversarial attacks or prompt injection and can identify risks like harmful bias, security vulnerabilities, and unexpected model behaviors. Red-teaming is also used to verify safety measures and to ensure that they have not been compromised. It is an important component of a comprehensive risk management strategy for GAI systems.