Domain 5: Operations and Audit Governance

CRAGE Domain 5: Operations and Audit Governance Study Guide

Domain Overview

CRAGE Domain 5 covers Operations and Audit Governance, which constitutes 16% of the CRAGE examination. This domain evaluates the knowledge required to operationalize, maintain, monitor, audit, and remediate artificial intelligence (AI) systems. The domain is divided into two major operational areas:

1. Sub-domain 5.1: Incident Response and Business Continuity Planning
2. Sub-domain 5.2: AI Assurance, Testing, Auditing, and Validation

Key Learning Modules

Sub-domain 5.1: Incident Response and Business Continuity Planning

AI Incident Response Framework & Governance Mappings

• NIST SP 800-61 Revision 3 (Published April 2025): Serves as a Cybersecurity Framework (CSF) 2.0 Community Profile for incident response, superseding Revision 2 (2012). It shifts from the classic four-phase lifecycle to spreading incident response across the six CSF 2.0 Functions:
  • Preparation Layer: Govern, Identify, Protect.
  • Incident Response Core: Detect, Respond, Recover.
  • Continuous Feedback: Lessons learned continuously feed the Improvement Category ( ID.IM ) of the Identify Function rather than waiting for a post-incident phase.
• Adopt, Defend, Govern (ADG) Framework Controls:
  • ADG Minimum Control MC-9: Demands AI-specific incident response procedures with replayable evidence capture, a documented playbook, and at least one tabletop exercise annually.
  • MC-9 Mappings: Maps to EU AI Act Article 73, NIST AI Risk Management Framework (AI RMF) MANAGE function, and ISO/IEC 42001 clause 10.2 (nonconformity and corrective action).

AI-Specific Incident Types & Harm Taxonomies

• OECD Definitions:
  • AI Incident: An event where an AI system's development, use, or malfunction directly or indirectly leads to harm (injury to health, disruption of critical infrastructure, human rights/legal violations, harm to property, communities, or environment).
  • AI Hazard: An event that could plausibly lead to an AI incident (potential harm, not actual harm).
• AI Failures & Vulnerabilities:
  • Model Drift: Operational harm in ADG taxonomy; triggers corrective maintenance in NIST AI RMF.
  • Prompt Injection (OWASP LLM01:2026): Inputs that alter LLM behavior unintendedly; indirect injection hides instructions in external content (e.g., websites, files).
  • Data Leakage (OWASP LLM02:2026 Sensitive Information Disclosure): Exposure of personal, confidential, or sensitive data.
  • Harmful Output: Misinformation (OWASP LLM07:2026) and unsafe content blocked by the ADG Safety Layer.
  • OWASP Top 10 for LLMs (2026): Includes LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM06 Unbounded Consumption (moved from prior ranking), and LLM10 Improper Output Handling.
  • ADG MC-8: Continuous production monitoring for abuse, drift, data leakage, unsafe actions, bias emergence, and configuration drift.

Statutory Incident Reporting (EU AI Act & Regulations)

• EU AI Act Article 73 Reporting Timelines:
  • General Serious Incident Rule: Report immediately upon establishing a causal link (or its reasonable likelihood) and no later than 15 days after becoming aware.
  • Death Trigger: Maximum 10 days if an incident results in a person's death.
  • Widespread Infringement / Critical Infrastructure Trigger: Maximum 2 days for widespread infringement or serious critical infrastructure disruption.
  • Reporting Destinations: High-risk AI reports go to the national market surveillance authority. General-purpose AI (GPAI) models with systemic risk report to the AI Office under Article 55(1)(c) (applicable since 2 August 2025).
  • Digital Omnibus on AI (Regulation EU 2026/1744): Moved high-risk compliance obligations to 2 December 2027 (Annex III) and 2 August 2028 (Annex I).
  • Deployer vs. Provider Obligations: Under Article 26(5), a deployer identifying a serious incident must inform the provider first, then the importer/distributor and market surveillance authority, and suspend system use.

Incident Execution, Evidence & Communication

• Incident Containment & Preservation: Containment levers include kill switches, human overrides, emergency stops, and agent-to-agent circuit breakers. Under EU AI Act Article 73(6), providers must not alter the system in a way that affects cause evaluation before informing competent authorities.
• ADG AI Incident Evidence Pack: Requires forensic capture of prompts, retrieved context, policy decisions, and tool calls for full replay capabilities.
• Parallel Communication Clocks:
  • GDPR Article 33: 72-hour window to notify supervisory authorities of personal data breaches (Digital Omnibus proposal suggests 96 hours).
  • EU AI Act Article 73: 15/10/2-day windows to market surveillance authorities.
  • Crisis Communication Plans (NIST SP 800-34): Designate exclusive authorized spokespersons for public messaging.

Business Continuity (BCP) & Disaster Recovery (DR)

• Core Metrics:
  • Maximum Tolerable Downtime (MTD): Total outage time accepted by the system owner for a business process.
  • Recovery Time Objective (RTO): Maximum time a system resource can remain unavailable; must be strictly less than MTD.
  • Recovery Point Objective (RPO): The point in time to which data can be restored (tolerable data loss).
• Fallback Mechanisms: NIST AI RMF MANAGE 2.1 and 2.4 mandate evaluating viable non-AI alternatives and implementing mechanisms to deactivate/supersede AI systems. A pre-approved manual or rules-based process is preferred when no trustworthy model state is available.
• Technical Recovery: Disaster recovery targets backup verification of approved model versions, prompts, configurations, and vector stores. NIST CSF 2.0 RC.RP-03 requires verifying backup integrity (checking model/data hashes against tampering or poisoning) prior to restoration. Substantial modifications post-recovery require a new conformity assessment under EU AI Act Article 43(4).

Sub-domain 5.2: AI Assurance, Testing, Auditing, and Validation

TEVV & AI Testing Frameworks

• Test, Evaluation, Verification, and Validation (TEVV): A continuous lifecycle process mandated by NIST AI RMF across design, development, deployment, and operational phases.
• ISO/IEC Standards:
  • ISO/IEC TR 29119-11:2020: Focuses on AI system testing and highlights the test oracle problem (the difficulty of defining the expected output for probabilistic/complex inputs) as the central testing challenge.
  • ISO/IEC TS 42119-2:2025: Adapts risk-based software testing standards to AI systems.
• ADG Minimum Control MC-4: Requires pre-deployment quality, safety, security, fairness, and failure-mode testing, documented via a signed evaluation report.

Security, Privacy, and Fairness Assurance

• Adversarial Security Testing:
  • NIST AI 100-2 E2025: Categorizes attack goals as availability breakdown, integrity violation, privacy compromise, and misuse enablement. Integrity threats include deployment-time evasion and training-time poisoning.
  • EU AI Act Requirements: Article 15(5) mandates high-risk AI resilience against poisoning, evasion, and model flaws. Article 55(1)(a) requires documented adversarial testing for GPAI with systemic risk.
  • ADG MC-3: Mandates separation of duties between development, security validation, and release approval authority.
• Privacy Assurance:
  • Attacks include membership inference, data reconstruction, property inference, and model extraction.
  • EU AI Act Article 4a (moved from ex-Article 10(5) by Digital Omnibus): Permits processing special-category personal data strictly for bias detection and correction only when synthetic or anonymized data is insufficient. Such data must be pseudonymized, access-controlled, unshared, and deleted upon bias remediation.
• Fairness & Bias Metrics:
  • NIST SP 1270: Identifies systemic, computational/statistical, and human-cognitive bias.
  • Demographic Parity: Evaluates equal selection rates across demographic groups.
  • Equalized Odds: Compares equal true positive and false positive rates across groups.
  • Four-Fifths Rule (29 CFR 1607.4(D)): US employment standard where a selection rate below 80% of the highest group's rate indicates potential adverse impact (not a universal proof of fairness).
  • ADG MC-11: Mandates fairness evaluations using representative test data with mandatory re-evaluation following model/data changes.

User Acceptance & Real-World Testing

• EU AI Act Article 60 Real-World Testing: Allows real-world testing of Annex III and Annex I Section A high-risk systems prior to market placement under specific constraints:An authority-approved testing plan.Informed consent from test participants.Maximum duration of 6 months (extendable once by an additional 6 months).System outputs must be fully reversible and capable of being disregarded.
  1. An authority-approved testing plan.
  2. Informed consent from test participants.
  3. Maximum duration of 6 months (extendable once by an additional 6 months).
  4. System outputs must be fully reversible and capable of being disregarded.
• EU AI Act Article 14(4): Requires human overseers to understand system limits, avoid automation bias, interpret outputs, and retain immediate override/stop capability.

Model Verification, Validation & Model Risk Management

• Verification vs. Validation:
  • Verification (ISO 9000): Objective evidence confirming specified requirements were fulfilled ("built right").
  • Validation (NIST AI RMF): Objective evidence confirming requirements for a specific intended use were fulfilled ("built the right thing").
• Regulatory Guidance Updates:
  • US Banking Supervision SR 26-2 (Issued April 2026): Superseded SR 11-7 for banking model risk management. It preserves expectations for "effective challenge" (independent, authoritative critical analysis) but explicitly excludes generative and agentic AI.

AI Audit, Documentation & Remediation Lifecycles

• Audit Standards:
  • ISO/IEC 42001 Clause 9.2: Mandates planned internal audits of the AI Management System (AIMS).
  • ISO/IEC 42006:2025: Establishes requirements for accredited bodies auditing and certifying ISO 42001 AIMS.
  • ISO 19011:2026: Fourth edition guidelines for auditing management systems.
  • GAO AI Accountability Framework (GAO-21-519SP): Built on Governance, Data, Performance, and Monitoring.
• Required ADG Control Artifacts:
  • MC-1: Published AI Inventory.
  • MC-3: Responsible, Accountable, Consulted, Informed (RACI) Matrix.
  • MC-4: Signed Evaluation Report.
  • MC-5: Approved Change Log.
  • MC-9: AI Incident Playbook & Annual Tabletop Record.
  • MC-10: Governance Review Records.
  • MC-11: Fairness Evaluation Report.
  • MC-12: Signed Responsibility Matrix.
• EU AI Act Retention Limits:
  • Article 18: Providers must retain technical documentation, quality management, and conformity documents for 10 years.
  • Article 19: Automatically generated event logs must be retained for at least 6 months.
• Remediation & Exceptions: Findings close only after corrective actions are verified as effective. Unmitigated risks require entry into the Governance Exception Register with justification, formal risk acceptance, and an explicit expiration date approved by the AI Governance Council.

Practice Quiz: Short-Answer Questions

Questions

1. How does NIST SP 800-61 Revision 3 restructure the classic incident response lifecycle compared to Revision 2?
2. What is the fundamental distinction between an AI incident and an AI hazard according to the OECD definition framework?
3. Under EU AI Act Article 73, what are the three distinct reporting timelines for serious incidents, and what triggers each?
4. Why is retraining a machine learning model prohibited during the immediate containment phase of an active AI incident investigation?
5. Differentiate between Maximum Tolerable Downtime (MTD), Recovery Time Objective (RTO), and Recovery Point Objective (RPO) within AI business continuity planning.
6. What is the "test oracle problem" identified in ISO/IEC TR 29119-11:2020, and why is it significant for AI assurance?
7. What four specific statutory conditions must be satisfied to conduct real-world testing of high-risk AI under EU AI Act Article 60?
8. How did the issuance of US banking supervisory guidance SR 26-2 in April 2026 alter the scope of model risk management rules?
9. Under what strict conditions does EU AI Act Article 4a permit the processing of special-category personal data during AI model development?
10. What are the mandatory retention periods specified under the EU AI Act for technical documentation versus automatically generated system logs?

Quiz Answer Key

1. NIST SP 800-61 Revision 3 abandons the rigid four-phase linear lifecycle of Revision 2 in favor of mapping incident response directly across the six functions of the NIST Cybersecurity Framework (CSF) 2.0. It establishes Govern, Identify, and Protect as the preparation layer, while Detect, Respond, and Recover constitute incident response execution. Additionally, it requires lessons learned to continuously feed into the CSF Identify Function (ID.IM) at any point during an incident rather than delaying until a post-incident phase.
2. According to the OECD, an AI incident is an event where an AI system's development, use, or malfunction directly or indirectly leads to actual harm, such as injury to health, critical infrastructure disruption, or human rights violations. In contrast, an AI hazard is an event that could plausibly lead to an AI incident, representing potential harm rather than actualized harm.
3. EU AI Act Article 73 requires providers to submit a serious incident report within 15 days of becoming aware for general cases, once a causal link is established. This timeline drops to no later than 10 days if the incident results in a person's death, and down to 2 days in the event of a widespread infringement or serious disruption to critical infrastructure.
4. Retraining a model during an active investigation alters the state and behavior of the system, violating EU AI Act Article 73(6), which forbids modifying an AI system in a manner that compromises forensic evaluation of the incident's root causes. Effective incident response requires containing the system (e.g., via kill switches or rolling back to a known-good backup) while preserving evidence such as prompts, model context, and logs for post-incident replay and analysis.
5. Maximum Tolerable Downtime (MTD) is the total outage duration an organization can accept for a business process before catastrophic impact occurs. Recovery Time Objective (RTO) is the targeted duration to restore the AI system service, which must be strictly shorter than the MTD. Recovery Point Objective (RPO) measures acceptable data loss expressed as the point in time to which data, model states, or vector stores must be restored.
6. The test oracle problem refers to the difficulty of determining whether an AI system's output is correct, given the probabilistic, open-ended, and complex nature of AI responses. ISO/IEC TR 29119-11:2020 identifies this as the primary testing challenge because traditional software testing relies on deterministic, pre-calculated pass/fail criteria that often do not exist for AI models.
7. Real-world testing under EU AI Act Article 60 requires an authority-approved testing plan, explicit informed consent from participants, a maximum testing duration of 6 months (extendable once by 6 months), and outputs that are fully reversible and capable of being disregarded. Furthermore, any serious incident occurring during the test must be reported under Article 73 and trigger mitigation, suspension, or termination of the test.
8. Issued in April 2026, SR 26-2 superseded SR 11-7 as the regulatory guidance for model risk management in US banking. While it retained core principles like effective challenge and conceptual soundness validation, it explicitly excluded generative and agentic AI from its operational scope.
9. EU AI Act Article 4a permits processing special-category data strictly for bias detection and mitigation when it is strictly necessary and cannot be accomplished using synthetic or anonymized data. Such data must be heavily pseudonymized, subject to strict access controls, forbidden from being shared with external parties, and permanently deleted immediately after the bias has been corrected or its retention period expires.
10. EU AI Act Article 18 mandates that providers maintain technical documentation, quality management documentation, and conformity records for at least 10 years after the AI system is placed on the market. Conversely, Article 19 requires that automatically generated event logs be retained for a minimum of 6 months, unless applicable national or Union law mandates a longer period.

Suggested Essay Questions

Note: Answers are intentionally omitted for self-study and discussion.

1. Integrated Framework Alignment in AI Incident Response: Evaluate how an organization can integrate NIST SP 800-61r3, ADG Minimum Control MC-9, and EU AI Act Article 73 into a unified operational Incident Response Plan. Analyze how governance roles, forensic evidence capture, and statutory reporting clocks interact during a severe production failure involving data leakage.
2. Comparative Evaluation of Fairness Metrics and Legal Standards: Compare Demographic Parity and Equalized Odds as mathematical frameworks for evaluating AI bias. Discuss how these technical metrics interact with regulatory standard benchmarks—such as the US Four-Fifths Rule (29 CFR 1607.4(D))—and explain why technical metric compliance alone does not guarantee legal non-discrimination.
3. Trade-off Analysis in Business Continuity and System Failover: When an operational high-risk AI system experiences severe drift or safety guardrail failures, organizations must choose between automated model rollback, rules-based alternative systems, or manual human fallbacks. Analyze the operational, risk, and compliance trade-offs of each fallback approach under NIST AI RMF MANAGE guidelines and EU AI Act Article 14/15 requirements.
4. Adversarial Assurance and Separation of Duties: Discuss the operational methodology for executing adversarial red-teaming against large language models using MITRE ATLAS and OWASP LLM risk frameworks. Detail why ADG Minimum Control MC-3 mandates independence between model development, security validation, and release approval, and describe the governance risks incurred when these roles overlap.
5. Auditing AI Governance: Evidence Collection and Exception Remediation: Explain the audit lifecycle for an AI Management System under ISO/IEC 42001 and ISO/IEC 42006:2025. Detail the specific artifacts required to substantiate ADG Minimum Controls MC-1 through MC-12, and trace the step-by-step lifecycle of an audit finding from discovery to risk acceptance in the Governance Exception Register.

Comprehensive Glossary