Domain 3: AI Risk Management

CRAGE Domain 3: AI Risk Management Study Guide

This study guide provides a comprehensive review of CRAGE Domain 3: AI Risk Management, which constitutes 16% of the examination. It covers internal risk assessments, threat taxonomy, third-party governance, supply chain integrity, regulatory requirements, and technical controls derived from NIST, ISO/IEC, OWASP, MITRE, and the EU AI Act.

Section 1: Detailed Core Concept Review

1.1 AI Risk Assessment and Threat Management

AI risk assessment establishes a system’s operating context, quantifies risks through repeatable testing, and dictates risk treatment. Post-deployment monitoring maintains visibility as models, data, and attacker techniques evolve.

NIST AI Risk Management Framework (AI RMF 1.0 / NIST AI 100-1)

Released on 26 January 2023, the NIST AI RMF defines four core functions: GOVERN, MAP, MEASURE, and MANAGE. While currently undergoing revision under the White House AI Action Plan, Version 1.0 remains the active standard.

+-----------------------------------------------------------------+
|                            GOVERN                               |
|   (Establishes culture, policies, roles, and risk tolerances)   |
+-----------------------------------------------------------------+
                                |
       +------------------------+------------------------+
       |                                                 |
       v                                                 v
+-----------------------------+       +-----------------------------+
|             MAP             | ----> |           MEASURE           |
| (Context, categorization,   |       |  (Testing, TEVV, metrics,   |
| capabilities & impacts)     |       |   independent assessment)   |
+-----------------------------+       +-----------------------------+
       |                                                 |
       +------------------------+------------------------+
                                |
                                v
+-----------------------------------------------------------------+
|                            MANAGE                               |
|   (Prioritizes, treats, and continuously monitors residual risk)|
+-----------------------------------------------------------------+

• MAP Function: The foundational context-setting phase before system testing or deployment.
  • MAP 1: Defines system context and intended purpose.
  • MAP 1.5: Establishes and documents organizational risk tolerances to serve as a baseline for evaluation.
  • MAP 2: Categorizes the AI system.
  • MAP 3: Evaluates capabilities, benefits, and costs.
  • MAP 4 / MAP 4.1: Identifies component and third-party risks, including legal, technology, and intellectual property (IP) infringement risks.
  • MAP 5 / MAP 5.1: Records the likelihood and magnitude of beneficial or harmful impacts using historical data, public incident reports, and external stakeholder feedback.
  • Key Milestone: Completion of MAP provides sufficient evidence for an initial go/no-go decision on system development or deployment.
• MEASURE Function: Converts MAP discoveries into empirical evidence, test results, and metrics before and during deployment.
  • MEASURE 1.1: Prioritizes significant risks and explicitly documents any unmeasurable risks or trustworthiness characteristics.
  • MEASURE 1.3: Mandates the inclusion of independent assessors or internal experts not involved in front-line development to minimize bias and conflicts of interest.
  • MEASURE 2 / 2.1 / 2.7: Evaluates trustworthy characteristics (e.g., MEASURE 2.7 for security and resilience) using documented tools and test sets.
  • TEVV (Test, Evaluation, Verification, and Validation): Conducted prior to deployment and on a regular schedule during operations, with formal documentation of results.
  • MEASURE 3.3: Establishes feedback mechanisms for end users and affected communities to report issues or appeal decisions, feeding data directly back into evaluation metrics.
• MANAGE Function: Prioritizes and treats risks using insights from MAP and MEASURE.
  • MANAGE 1.1: Re-evaluates whether the system fulfills its intended purpose and determines if deployment should proceed.
  • MANAGE 1.3: Defines four formal risk responses: Mitigating, Transferring, Avoiding, or Accepting.
  • MANAGE 1.4: Documents negative residual risk (the sum of all unmitigated risks) for downstream acquirers and end users.
  • MANAGE 3 / 4.1: Monitors operational risks, manages overrides and appeals, and oversees system decommissioning.

ISO/IEC AI Risk Frameworks

ISO/IEC standards separate high-level guidance, certifiable management systems, and specialized impact assessments.

Fundamental Rights Impact Assessment (FRIA)

Under Article 27 of the EU AI Act, deployers of high-risk AI systems must execute a FRIA prior to initial system deployment.

• Mandated Entities: Public bodies, private entities providing public services, and deployers of credit scoring or life/health insurance AI.
• Core Requirements: Detailed description of deployer processes, operational frequency/period, affected groups, specific harm risks, human oversight mechanisms, and corrective action workflows.
• Value Chain Integration: Completed by the deployer (not the provider). Results must be submitted to the market surveillance authority. Under Regulation (EU) 2026/1744 (Digital Omnibus on AI), FRIAs may cross-reference existing Data Protection Impact Assessments (DPIAs).
• Enforcement Timeline: Rules apply to Annex III high-risk systems starting 2 December 2027.

EC-Council ADG (Adopt, Defend, Govern) Framework Controls

• MC-2 (AI System Classification): Categorizes systems based on data sensitivity, autonomy, external exposure, harm potential, and business criticality. Maps directly to EU AI Act Articles 6 and 7, NIST MAP, and ISO 42001 clause 6.1.2.
• Governance Ownership: The AI Governance Council defines go/no-go thresholds and maintains the Governance Exception Register, which tracks justifications, risk acceptance, and mandatory expiration dates. Risk acceptance requires formal governance ownership and cannot be executed unilaterally by engineering teams.
• MC-8 (Production Monitoring): Continually monitors production for abuse, performance/data drift, data leakage, unsafe actions, bias emergence, and configuration drift.
• MC-10 (Governance Reviews & Reporting): Mandates periodic risk reviews, exception handling, and board-level reporting. Operates across three metric tiers: Control Effectiveness Metrics, Outcome Metrics (e.g., Mean Time to Detect - MTTD), and Board-Level Indicators (e.g., overall AI Risk Posture).

OWASP Risk Taxonomies (2026 Editions)

OWASP Top 10 for LLM Applications 2026 (Published August 2026)

Ranked using a weighted model (75% community vote, 25% empirical incident data):

1. LLM01: Prompt Injection
2. LLM02: Sensitive Information Disclosure
3. LLM03: Excessive Agency (Rose to 3rd place)
4. LLM04: Supply Chain
5. LLM05: Data and Model Poisoning
6. LLM06: Unbounded Consumption
7. LLM07: Misinformation
8. LLM08: Hidden Context Exposure (Broadened and renamed from System Prompt Leakage; explicitly forbids storing API keys or relying on system prompts as sole authorization boundaries)
9. LLM09: Vector and Embedding Weaknesses
10. LLM10: Improper Output Handling (Fell from 5th to 10th place)

OWASP Top 10 for Agentic Applications 2026 (Released 9 December 2025)

Focuses on autonomous agents utilizing memory, tools, and multi-agent workflows (Identifiers: ASI01–ASI10):

• ASI01: Agent Goal Hijack (The agentic manifestation of prompt injection where an agent's overarching objective is altered)
• ASI02: Tool Misuse and Exploitation
• ASI03: Identity and Privilege Abuse
• ASI04: Agentic Supply Chain Vulnerabilities (Focuses on compromised runtime assets like Model Context Protocol [MCP] servers and tool registries)
• ASI05: Unexpected Code Execution
• ASI06: Memory and Context Poisoning
• ASI07: Insecure Inter-Agent Communication
• ASI08: Cascading Failures
• ASI09: Human-Agent Trust Exploitation
• ASI10: Rogue Agents

Threat Intelligence and Adversarial ML Frameworks

MITRE ATLAS (Release 2026.10 – October 2026)

A structured knowledge base of real-world adversary behavior targeting AI systems, containing 16 tactics, 129 techniques, 92 sub-techniques, 44 mitigations, and 77 case studies.

• Unique Tactics: AI Model Access and AI Attack Adaptation (formerly AI Attack Staging prior to version 2026.08).
• Key Techniques: LLM Prompt Injection (AML.T0051), Training Data Poisoning (AML.T0020), Supply Chain Compromise (AML.T0010), and Replicate AI Model Behavior (AML.T0005).
• Key Mitigations: Verify AI Artifacts (AML.M0014) and Maintain an AI Bill of Materials (AML.M0023).

NIST Adversarial Machine Learning Taxonomy (NIST AI 100-2 E2025 / Draft SP 2400-102e2026)

Categorizes attacks across four adversary goals: availability breakdown, integrity violation, privacy compromise, and misuse violations.

+-----------------------------------------------------------------------------------+
|                        Adversarial ML Lifecycle Attack Stages                     |
+-----------------------------------------------------------------------------------+
                                          |
         +--------------------------------+--------------------------------+
         |                                                                 |
         v                                                                 v
+----------------------------------+                             +----------------------------------+
|          TRAINING STAGE          |                             |         INFERENCE STAGE          |
+----------------------------------+                             +----------------------------------+
| • Data Poisoning                 |                             | • Evasion Attacks                |
|   - Split-view (expired domains) |                             |   (inputs modified at runtime;   |
|   - Frontrunning (snapshot edits)|                             |    model parameters unaltered)   |
| (Effects persist after removal)  |                             | • Direct / Indirect Injection    |
+----------------------------------+                             +----------------------------------+
                                                                           |
                                                                           v
                                                                 +----------------------------------+
                                                                 |           QUERY ACCESS           |
                                                                 +----------------------------------+
                                                                 | • Model Extraction (replication) |
                                                                 | • Membership Inference           |
                                                                 | • Model Inversion (data reconstruction)|
                                                                 +----------------------------------+

Incident Reporting Deadlines (EU AI Act Article 73)

Deployers identifying a serious incident must notify the provider immediately (Article 26(5)). The provider/deployer must report serious incidents to the market surveillance authority according to strict statutory windows:

• Immediate Notification: Inform provider immediately upon identification.
• 2 Days: Incidents resulting in widespread infringement or critical infrastructure disruption.
• 10 Days: Incidents resulting in a person's death.
• 15 Days: Standard maximum deadline for all other serious incidents.

1.2 Third-Party AI Risk Management and Supply Chain Security

Organizations purchasing or integrating external AI components maintain ultimate legal and operational accountability for how those systems are deployed, even as direct operational control shifts to vendors.

ADG Shared Responsibility Matrix Across Deployment Classes

The ADG framework delineates owner responsibilities across nine core governance surfaces: Prompt, Context, Tools, Orchestration, Identity, Safety Layer, Telemetry, Model, and GOVERN Policy.

+---------------------------------------------------------------------------------------+
|                              ADG Deployment Classes                                   |
+---------------------------------------------------------------------------------------+
|  Governance Surface  | Self-Hosted / Homegrown | Foundation Model API | SaaS / Embedded AI  |
+----------------------+-------------------------+----------------------+--------------------+
| GOVERN Policy        | Customer                | Customer             | Customer           |
| Prompt               | Customer                | Customer             | Vendor             |
| Context              | Customer                | Customer             | Customer           |
| Tools                | Customer                | Customer             | Vendor             |
| Orchestration        | Customer                | Customer             | Vendor             |
| Identity             | Customer                | Customer             | Customer           |
| Safety Layer         | Customer                | Customer             | Shared             |
| Telemetry            | Customer                | Customer             | Customer           |
| Model                | Customer                | Vendor               | Vendor             |
+----------------------+-------------------------+----------------------+--------------------+
| Customer Ownership   | 9 of 9 Surfaces         | 7 of 9 Surfaces      | 4.5 of 9 Surfaces  |
+---------------------------------------------------------------------------------------+

• Core Governance Principle: Contractually transferring technical risk to a vendor does not transfer legal accountability for the application's outputs and impacts. Control MC-12 requires formal documentation of these responsibility boundaries.

Supplier Due Diligence and Artifact Verification

• Assurance Verification: Organizations must independently evaluate vendor security claims, data handling policies, alignment methodologies, and limitations rather than relying on vendor questionnaires or marketing attestations. Results are documented in the ADG Vendor AI Due Diligence Record (Step 8 of the ADG process backbone).
• Joint Cyber Agency Guidance (March 2026): Eight national cyber agencies recommend requiring vendors to supply and maintain an AI Bill of Materials (AI-BOM) and Software Bill of Materials (SBOM), while establishing mandatory incident and vulnerability reporting windows.
• Cryptographic Verification:
  • Hashes and Checksums: Validate file integrity during storage and transit. MITRE ATLAS AML.M0014 warns that checksums must never be accepted over the same channel used to deliver the AI artifact itself.
  • Digital Signatures & Lineage: Hashes only verify that a file matches a published digest; cryptographic signatures and append-only databases are required to verify origin, track edits prior to publication, and prevent serialization attacks.
  • OpenSSF Model Signing (OMS): Signs model parameters during training and validates signatures during upload, deployment selection, and downstream integration. Model storage should enforce non-executable, weight-only formats.

Software & Machine Learning Bills of Materials (BOMs)

+-----------------------------------------------------------------------------------+
|                        CycloneDX ML-BOM Schema (ECMA-424 v1.7)                    |
+-----------------------------------------------------------------------------------+
|  Component Types:                                                                 |
|  • machine-learning-model                                                         |
|  • data                                                                           |
+-----------------------------------------------------------------------------------+
|  modelCard Object Structure:                                                      |
|  ├── Model Parameters (Architectural details, tasks, hyperparameters)              |
|  ├── Quantitative Analysis (Benchmarking metrics, performance evaluations)       |
|  ├── Considerations                                                               |
|  │   ├── Model Limitations                                                        |
|  │   └── Fairness & Ethical Assessments                                          |
|  └── Datasets (Lineage, provenance hashes, training/validation sets)              |
+-----------------------------------------------------------------------------------+

• Standard BOM vs. AI-BOM: Traditional SBOMs list software dependencies and packages. An AI-BOM (supported by CycloneDX v1.5+ under standard ECMA-424, updated to v1.7 in October 2025) extends coverage to models, training datasets, runtime components, container images, system instructions, connected tools, and agent configurations.

EU AI Act Value-Chain Obligations

Rebranders and System Modifiers (Article 25)

• Role Reassignment (Art. 25(1)): A distributor, importer, or deployer that puts its brand on a high-risk system, makes a substantial modification, or changes its intended purpose into a high-risk category assumes full Article 16 Provider Obligations.
• Original Provider Duties: The original developer must supply technical documentation, capabilities, limitations, and targeted system access to support the new provider's compliance duties.
• Upstream Supplier Agreements (Art. 25(4)): High-risk AI providers must execute written agreements with third-party component suppliers. While free and open-source tools are excluded, General-Purpose AI (GPAI) models are explicitly included.

Downstream General-Purpose AI (GPAI) Obligations (Article 53)

Enforceable since 2 August 2025, providers of GPAI models must provide downstream integrators with compliance artifacts:

• Article 53(1)(a): Technical documentation (Annex XI) provided to the AI Office and national authorities.
• Article 53(1)(b): Integration information (Annex XII) supplied to downstream providers detailing model capabilities, limitations, and testing results.
• Article 53(1)(c) & (d): Text-and-data-mining copyright reservation policy and a public summary of training content based on the AI Office template.
• Exemptions: Open-source GPAI models with publicly available weights are exempt from 53(1)(a) and (b), unless the model presents systemic risk.

+-----------------------------------------------------------------------------------+
|                      EU AI Act Statutory Application Dates                        |
+-----------------------------------------------------------------------------------+
|  August 2, 2025     • GPAI Provider Obligations (Article 53) take effect.       |
|                     ------------------------------------------------------------- |
|  December 2, 2027   • High-Risk Annex III rules & Article 27 FRIAs take effect    |
|                       (Per Regulation EU 2026/1744 - Digital Omnibus).            |
|                     ------------------------------------------------------------- |
|  August 2, 2028     • High-Risk Annex I rules take effect.                        |
|                     ------------------------------------------------------------- |
|  August 2, 2030     • Compliance deadline for legacy public authority systems.    |
+-----------------------------------------------------------------------------------+

Procurement Governance and Ongoing Compliance

Procurement Standards & US OMB M-25-22 (April 2025)

• Cross-Functional Acquisition: Replaced M-24-18, mandating multi-disciplinary teams (procurement, legal, risk, technical) for AI acquisitions.
• Lock-In Protections: Solicitations must require knowledge transfer, data/model portability, licensing clarity, and pricing transparency.
• Data Use Restrictions: Contracts must permanently prohibit vendors from using non-public customer data, prompts, or generated outputs to train commercial models without explicit, affirmative consent.
• Standard Clauses (EU MCC-AI 2025): Updated model contract clauses (High-Risk and Light versions) enforce AI Act Chapter III rules (human oversight, accuracy, robustness, cybersecurity, data governance). IP, payment, and liability allocation terms must be attached via the primary service agreement.

Continuous Vendor Assurance Frameworks

• ISO/IEC 42001: Certifies that a vendor maintains an operational AI management system (Plan-Do-Check-Act cycle). It does not evaluate individual AI application outputs or guarantee error-free performance.
• Cloud Security Alliance (CSA) STAR for AI:
  • Controls Matrix v1.1 (June 2026): Contains 247 control objectives across 18 domains, mapped to ISO 42001 and ISO 27001.
  • Level 1: Self-assessment submitted via the AI Consensus Assessments Initiative Questionnaire (AI-CAIQ).
  • Level 2: Independent third-party audit and certification combined with ISO 27001 or ISO 42001.
• Ongoing Oversight: Assurance requires paired attestations, automated telemetry monitoring, contractually enforced sub-processor change notifications (GDPR Art. 28), and defined contingency/rollback procedures (NIST GOVERN 6.2).

Section 2: Short-Answer Practice Quiz

Questions

1. An enterprise risk manager is asked to evaluate where intended purpose, affected stakeholders, and organizational risk tolerance should be formally documented within the NIST AI Risk Management Framework. Which function handles this documentation, and how does it differ from the MEASURE function?
2. A commercial health insurance firm plans to deploy an automated AI model to evaluate policy claims. Under Article 27 of the EU AI Act, what assessment must be conducted prior to deployment, which entity bears responsibility for it, and to whom must the results be delivered?
3. A cybersecurity team is reviewing system prompt configurations for an enterprise generative AI tool. Under the OWASP Top 10 for LLM Applications 2026, which vulnerability addresses system prompt exposure, and what design constraint does it impose on application secrets?
4. An autonomous software agent running on a corporate network uses external tools and memory to fulfill tasks. An attacker embeds covert instructions in a document read by the agent, causing it to alter its primary instructions. Which OWASP Top 10 for Agentic Applications threat category describes this attack?
5. A security team is designing a red-team testing strategy to simulate realistic adversary behavior against an deployed predictive model. Should the team utilize OWASP or MITRE ATLAS as their reference framework, and why?
6. Contrast the mechanics and lifecycle timing of a data poisoning attack with an evasion attack against a machine learning model.
7. An organization integrates a foundation model via a third-party API. According to the EC-Council ADG shared responsibility model, which governance surfaces remain under the exclusive control of the acquiring organization, and can final risk accountability be transferred to the API provider?
8. When tracking third-party AI assets, how does an AI Bill of Materials (AI-BOM) created under the CycloneDX ML-BOM standard differ from a standard Software Bill of Materials (SBOM)?
9. A retail company acquires a white-label, high-risk AI system from a software vendor, places its own brand name on the application, and deploys it to consumers. Under Article 25 of the EU AI Act, what shift in legal obligations occurs, and what duty does the original vendor owe?
10. A vendor presents an ISO/IEC 42001 certificate during a third-party risk assessment as proof that its customer-facing AI model produces accurate and unbiased results. How should the acquiring risk team interpret the scope of this certification?

Section 3: Answer Key

Answers

1. Answer: Intended purpose, affected stakeholders, and organizational risk tolerance are documented under the MAP function (specifically MAP 1 and MAP 1.5). MAP establishes the qualitative context, system boundaries, and baseline yardsticks for the AI system. In contrast, MEASURE applies quantitative, qualitative, or mixed testing methods to evaluate and track the specific risks that were previously identified during the MAP phase.
2. Answer: The firm must complete a Fundamental Rights Impact Assessment (FRIA) prior to deploying the high-risk system. Primary responsibility for completing the FRIA sits with the deployer (the insurance firm), not the AI model provider. Upon completion, the deployer must formally notify and submit the assessment results to the relevant market surveillance authority.
3. Answer: The vulnerability is LLM08: Hidden Context Exposure (formerly named System Prompt Leakage in the 2025 edition). This vulnerability dictates that hidden context, such as system prompts, must always be assumed to be discoverable by end users. Consequently, system prompts must never contain API keys or credentials, nor can they serve as an application's sole authorization boundary.
4. Answer: This attack is classified as ASI01: Agent Goal Hijack. ASI01 represents the agentic counterpart to prompt injection, occurring when malicious inputs overwrite or manipulate an autonomous agent's context or system instructions, forcing it to stray from its intended objective.
5. Answer: The security team should utilize MITRE ATLAS because it categorizes real-world adversary tactics, techniques, and procedures (TTPs) specifically designed for designing red-team attack scenarios and detection signatures. OWASP frameworks focus primarily on categorizing application security vulnerability classes that software developers must remediate, rather than detailing step-by-step adversary execution paths.
6. Answer: Data poisoning occurs during the training or fine-tuning stage by corrupting datasets to introduce targeted errors, backdoors, or biases that persist even after the malicious data is removed. Evasion attacks occur at inference time by altering input data to cause a deployed model to misclassify specific instances without altering the underlying model parameters.
7. Answer: Under the ADG framework for a Foundation Model API, the customer retains exclusive control over 7 of the 9 surfaces: Prompt, Context, Tools, Orchestration, Identity, Safety Layer, and Telemetry. Legal and operational accountability for how the AI is used never transfers to the vendor, regardless of the deployment model or contractual arrangements.
8. Answer: A standard SBOM lists software packages and software dependencies, whereas an AI-BOM utilizing CycloneDX ML-BOM (ECMA-424) expands this scope to include machine learning models, training datasets, and a specialized modelCard object. The modelCard captures architectural parameters, quantitative benchmark evaluations, limitations, and fairness assessments.
9. Answer: Under Article 25(1) of the EU AI Act, the retail company is legally reclassified as the provider of the high-risk AI system and assumes all Article 16 provider obligations. The original vendor is legally required to cooperate with the new provider by furnishing technical documentation, operational limitation details, and targeted system access needed for validation.
10. Answer: An ISO/IEC 42001 certification confirms that the vendor maintains a compliant AI Management System (AIMS) following a structured Plan-Do-Check-Act cycle. It certifies the management processes and organizational controls governing AI development, but it does not certify, audit, or guarantee the specific accuracy, unbiased nature, or performance of an individual AI application output.

Section 4: Essay Practice Questions

The following essay prompts are provided for advanced self-study and conceptual synthesis. Answers are deliberately omitted.

1. Comparative Analysis of AI Risk Frameworks: Evaluate the alignment and structural differences between the NIST AI Risk Management Framework 1.0 (GOVERN, MAP, MEASURE, MANAGE) and the ISO/IEC AI standards suite (ISO/IEC 23894, ISO/IEC 42001, and ISO/IEC 42005). Explain how an enterprise can map NIST MAP and MEASURE outputs to achieve ISO/IEC 42001 compliance.
2. Threat Modeling Autonomous Systems: Contrast the security and governance challenges of static Large Language Model deployments (OWASP LLM Top 10 2026) with autonomous multi-agent architectures (OWASP Agentic Top 10 2026). Detail how failure modes such as Excessive Agency (LLM03) evolve into Agent Goal Hijack (ASI01), Tool Misuse (ASI02), and Cascading Failures (ASI08) when agents access runtime tools and shared memory.
3. Third-Party Risk and AI Supply Chain Architecture: Construct an end-to-end third-party vendor risk assessment program for integrating a external foundation model into a regulated business workflow. Incorporate the ADG Shared Responsibility Model, cryptographic artifact verification (OpenSSF Model Signing, hashes), CycloneDX ML-BOM tracking, and standard contractual protections (EU MCC-AI and OMB M-25-22 directives).
4. Regulatory Enforcement and Lifecycle Compliance under the EU AI Act: Trace the mandatory compliance obligations, assessment prerequisites, and operational reporting timelines required across the AI value chain under the EU AI Act (including amendments under Regulation (EU) 2026/1744). Analyze the specific duties triggered by Articles 25, 26, 27, 53, 72, and 73 for providers, deployers, and upstream GPAI vendors.
5. Adversarial Machine Learning Defenses and Provenance Management: Analyze the technical attack vectors associated with data poisoning (including split-view and frontrunning methods), model extraction, and indirect prompt injection as defined by NIST AI 100-2 E2025 and MITRE ATLAS. Propose a multi-layered defense architecture incorporating cryptographically signed append-only databases, model registries, non-executable weight formats, and post-market monitoring controls.

Section 5: Comprehensive Glossary

• ADG Framework (Adopt, Defend, Govern): An enterprise governance framework developed by EC-Council that defines control requirements (e.g., MC-2, MC-8, MC-12) and a 9-surface shared responsibility matrix across AI deployment models.
• Agent Goal Hijack (ASI01): A critical security threat in agentic AI where an adversary manipulates prompt contexts or external tool inputs to modify an autonomous agent's underlying objective.
• AI Bill of Materials (AI-BOM): An inventory manifest detailing the software components, machine learning models, training datasets, hyperparameters, runtime environments, and agent configurations of an AI system.
• AI Model Access: A tactic unique to the MITRE ATLAS framework describing techniques used by adversaries to gain physical, API, or query access to an AI model.
• AI Risk Tolerance: The pre-established boundary documented under NIST MAP 1.5 defining the level of risk an organization is willing to accept, serving as a yardstick for downstream risk treatment decisions.
• CycloneDX ML-BOM: An open standard (ECMA-424) managed by OWASP and Ecma International that extends Software Bills of Materials to include machine-learning-model assets, datasets, and structured modelCard objects.
• Data Poisoning: An attack executed during model training or fine-tuning where input data is maliciously altered to introduce backdoors, systemic bias, or targeted classification errors that persist post-training.
• Direct Prompt Injection: An attack vector where an end user inputs crafted instructions directly into a prompt interface to override system instructions or bypass safety filters.
• Evasion Attack: An inference-time attack where inputs are subtly modified to trick a trained model into misclassifying the input without modifying the underlying model weights.
• Excessive Agency (LLM03): A vulnerability where an LLM is granted disproportionate permissions, tools, or operational autonomy, enabling unapproved or destructive actions when manipulated.
• Frontrunning Poisoning: A training data attack where an adversary edits crowdsourced web pages or repositories immediately prior to a known, scheduled dataset scraping snapshot.
• Fundamental Rights Impact Assessment (FRIA): A mandatory assessment under EU AI Act Article 27 executed by deployers of high-risk AI systems to evaluate societal, individual, and rights-based risks prior to initial use.
• General-Purpose AI (GPAI) Model: An AI model trained on broad data at scale that displays significant generality and can perform a wide range of distinct tasks, subject to downstream transparency rules under EU AI Act Article 53.
• Governance Exception Register: A formal record maintained by an AI Governance Council that tracks approved risk acceptance decisions, underlying justifications, compensating controls, and explicit expiration dates.
• Hidden Context Exposure (LLM08): An OWASP 2026 risk classification (formerly System Prompt Leakage) highlighting that system prompts and hidden contexts are inherently discoverable and must not hold secrets or serve as access control boundaries.
• Indirect Prompt Injection: An attack where an LLM reads external third-party content (e.g., web pages, PDFs, tool responses) that contains embedded hidden text intended to hijack the model's behavior.
• ISO/IEC 23894:2023: Guidance standard providing AI-specific adaptations of the ISO 31000 risk management framework, covering risk identification, analysis, and evaluation across the AI life cycle.
• ISO/IEC 42001: A certifiable international standard specifying the requirements for establishing, implementing, maintaining, and continually improving an Artificial Intelligence Management System (AIMS).
• ISO/IEC 42005:2025: A guidance standard published in May 2025 defining methodologies for executing AI system impact assessments on individuals, groups, and society.
• Membership Inference: An adversarial privacy attack using model query outputs to determine whether a specific individual's data record was included in the model's training dataset.
• MITRE ATLAS: Adversarial Threat Landscape for AI Systems; a curated knowledge base of real-world adversary tactics, techniques, and procedures (TTPs) targeting AI-enabled applications.
• Model Card ( modelCard ): A standardized metadata object embedded in ML-BOMs detailing model parameters, quantitative evaluation metrics, operational limitations, and ethical considerations.
• Model Extraction: An attack where an adversary submits a series of crafted queries to a target model to reconstruct or clone a functionally equivalent model instance.
• Model Inversion: An adversarial attack that analyzes output probabilities from model queries to reconstruct representative samples of the original training data.
• Negative Residual Risk: Defined in NIST AI RMF MANAGE 1.4 as the sum total of all unmitigated risks remaining after risk treatment controls have been applied.
• OpenSSF Model Signing (OMS): A cryptographic protocol developed by the Open Source Security Foundation that signs model weights during training and validates signatures across deployment pipelines.
• Post-Market Monitoring: Ongoing requirements (e.g., EU AI Act Article 72 and NIST MEASURE 3) to continuously collect, analyze, and report operational telemetry and risk trends after system release.
• Split-View Poisoning: A training data corruption attack executed by purchasing expired web domain names that are linked to from trusted, curated dataset indices.
• TEVV (Test, Evaluation, Verification, and Validation): A systematic testing framework mandated under NIST MEASURE to establish metrics, benchmark performance, and validate trustworthiness before and during system operation.
• US OMB M-25-22: A US Office of Management and Budget memorandum (April 2025) mandating cross-functional acquisition teams, vendor lock-in protections, and strict prohibitions against training commercial models on non-public government data.