5.2AI Assurance, Testing, Auditing, and Validation
AI assurance proves, with evidence, that a system is accurate, secure, private, fair and fit for its users before release and for as long as it runs. Audits then check that the controls and the management system work, and remediation closes what the audits find.
Testing AI is harder than testing ordinary software because the right answer is often fuzzy and the system can behave differently tomorrow than today. Frameworks handle this by testing across the whole life cycle, against thresholds agreed in advance, and by keeping the results as evidence. Think of a driving test that is retaken every time the car or the road changes.
diagram
The NIST AI RMF spreads TEVV across the lifecycle with a release gate in the middle. Changes and drift send the system back for retesting.
Key points
The NIST AI Risk Management Framework (AI RMF) calls for test, evaluation, verification and validation (TEVV) throughout the AI lifecycle, ideally with verifiers distinct from testers.
TEVV runs from design (validating assumptions) through development (model validation) and deployment (system validation and integration) to operations (monitoring, incident tracking).
EU AI Act Article 9(8) requires testing high-risk AI before market placement against prior defined metrics and probabilistic thresholds suited to the intended purpose.
ISO/IEC TR 29119-11:2020 gives guidelines for testing AI-based systems and names the test oracle problem, knowing what the expected result should be, as the main challenge.
ISO/IEC TS 42119-2:2025 applies the ISO/IEC/IEEE 29119 software testing series to AI systems with a risk-based approach.
ADG (Adopt, Defend, Govern) MC-4 requires quality, safety, security, fairness and failure-mode testing before any production deployment, evidenced by a signed evaluation report.
NIST AI 600-1 names pre-deployment testing as one of four primary considerations for generative AI, alongside governance, content provenance and incident disclosure.
Exam tip
The best answer tests against criteria set before testing and keeps testing after release. A one-time benchmark score before launch is the classic distractor.
Security testing for AI means attacking your own system before someone else does. Red teams try to trick the model, poison its data, steal it or make it misbehave, using public catalogs of known attacks as a checklist. The results tell the people who approve releases whether the system is safe enough to ship.
diagram
NIST AI 100-2 E2025 groups adversarial machine learning attacks by attacker objective. A red team plan should cover each branch that applies to the system.
Key points
NIST AI 100-2 E2025 groups attacker objectives as availability breakdown, integrity violation and privacy compromise, adding misuse enablement for generative AI.
Integrity attacks include evasion with adversarial examples at deployment time and targeted, backdoor or model poisoning at training time.
EU AI Act Article 15(5) expects high-risk AI to resist data poisoning, model poisoning, adversarial examples or model evasion, confidentiality attacks and model flaws.
MITRE ATLAS is a living knowledge base of adversary tactics and techniques against AI-enabled systems, built from real-world attacks and red team demonstrations.
The OWASP Top 10 for large language model (LLM) applications 2026 runs from LLM01 Prompt Injection and LLM02 Sensitive Information Disclosure to LLM10 Improper Output Handling; Unbounded Consumption moved to LLM06.
Providers of general-purpose AI models with systemic risk must conduct and document adversarial testing under EU AI Act Article 55(1)(a).
ADG (Adopt, Defend, Govern) puts red teaming in the Defend pillar, and MC-3 separates deployment ownership, security validation and approval authority.
Exam tip
Red team findings go to the people who approve release, not only to the builders. If one team builds, tests and approves its own model, the governance gap is separation of duties (ADG MC-3).
A model can remember more than it should. Privacy testing checks whether someone could pull personal data back out of the model or its answers, or prove that a specific person's record was used in training. It also catches everyday leaks, like a chatbot repeating another customer's details.
Targets the training data
Targets the model
Attack
Membership inference
Data reconstruction
Model extraction
What leaks
Whether a record was used
Content or features of records
Information about the model
Privacy tests probe both the training data and the model itself. NIST AI 100-2 notes membership inference and data reconstruction can be mounted with only query access to a deployed model.
Key points
NIST AI 100-2 E2025 defines a privacy compromise as unintended leakage of restricted or proprietary information, such as training data, model weights or architecture.
Privacy attacks to test include data reconstruction, membership inference, property inference and model extraction, and some need only query access to a deployed model.
For generative AI, NIST AI 100-2 adds prompt extraction, data extraction and leaking information from user interactions.
OWASP LLM02 Sensitive Information Disclosure (same number in the 2026 list) covers personal data, financial and health records, credentials and confidential business data exposed by a large language model.
The NIST AI Risk Management Framework (AI RMF) MEASURE 2.10 requires the privacy risk of the AI system to be examined and documented.
EU AI Act Article 4a, moved from Article 10(5) by the Digital Omnibus, allows special-category data for bias detection only when strictly necessary and synthetic or anonymised data cannot do it.
Under Article 4a, such data must be pseudonymised, access-controlled, not shared with other parties, and deleted once the bias is corrected or its retention period ends.
Exam tip
To show a model is not leaking training data, choose privacy attack testing such as membership inference or extraction tests, not a policy statement. Synthetic or anonymised data comes before real sensitive data in tests.
Fairness testing asks whether the system treats groups of people differently in ways that matter. Teams pick a fairness measure that fits the harm, run representative test data through the model and compare results across groups. No single number proves a system is fair, so the context and the people affected decide which measure counts.
Demographic parity
Equalized odds
Compares across groups
Selection rates
Prediction independent of group
True positive rates
False positive rates
Watch out for
Four-fifths is a US hiring screen
Needs trusted outcome labels
Two common fairness metrics compare different things across groups. Choose the one that matches the harm, and do not treat a single threshold as proof of fairness.
Key points
NIST names three categories of AI bias: systemic, computational and statistical, and human-cognitive, and each can occur without discriminatory intent (NIST SP 1270).
Demographic parity compares selection rates across groups, while equalized odds compares both true positive and false positive rates across groups.
The US four-fifths rule in 29 Code of Federal Regulations (CFR) 1607.4(D) treats a selection rate under 80% of the highest group's rate as evidence of adverse impact.
The four-fifths rule comes from US federal employment selection guidelines and is not a general proof of fairness outside that context.
ADG (Adopt, Defend, Govern) MC-11 requires fairness evaluation with representative test data and established metrics, with re-evaluation after model or data changes.
EU AI Act Article 10(2)(f) and (g) require examining training, validation and testing data for possible biases and taking measures to detect, prevent and mitigate them.
NIST AI Risk Management Framework (AI RMF) MEASURE 2.11 requires fairness and bias to be evaluated and documented; ISO/IEC TR 24027:2021 covers bias measurement techniques.
Exam tip
Pick the metric from the harm: equal selection rates point to demographic parity, equal error rates point to equalized odds. A distractor treats passing the four-fifths rule as proof that the model is fair.
Lab results do not prove a system works for the people who will rely on it. User acceptance testing puts the system in front of real users and real workflows to confirm it meets their needs and that they can supervise it properly. Under the EU AI Act, testing a high-risk system in the real world is allowed, but only with a plan, approval and consent.
diagram
Acceptance moves from the lab to real users before the release gate. In the EU, a real-world pilot of a high-risk system needs an approved plan and informed consent.
Key points
NIST AI Risk Management Framework (AI RMF) MEASURE 2.3 requires performance shown for conditions like the deployment setting, and MEASURE 2.2 requires representative human-subject evaluations.
NIST AI 600-1 describes field testing of how people interact with and act on AI-generated information, including user experience (UX) and usability studies.
EU AI Act Article 14(4) requires overseers who understand capacities and limits, stay aware of automation bias, interpret outputs, and can override or stop the system.
Article 60, as amended by the Digital Omnibus, allows real-world testing of Annex III and Annex I Section A high-risk AI before market placement with an authority-approved testing plan.
Real-world tests last at most six months, extendable once by six more, need informed consent, and the AI's outputs must be effectively reversible and able to be disregarded.
A serious incident found during real-world testing is reported under Article 73, and the provider must mitigate it, suspend the test or terminate it.
The ADG (Adopt, Defend, Govern) AI Release Gate Checklist records pre-deployment verification of security, fairness, safety and governance approval before go-live.
Exam tip
For EU real-world testing, remember four anchors: an approved plan, informed consent, reversible outputs and a six-month limit extendable once. Lab benchmarks alone do not show fitness for the deployment setting.
Verification asks whether the model was built to its specification. Validation asks whether it actually does the job it is meant for, with the real data and people it will meet. A model can pass one and fail the other, and reviewers who did not build it give the most trustworthy answer.
Verification
Built right
Meets specified requirements
Checked against the spec
Both
Objective evidence
Documented results
Independent reviewers
Validation
Right thing built
Fit for intended use
Deployment-like conditions
Verification checks the model against its specification, and validation checks it against its intended use. Both rely on objective, documented evidence, ideally from people who did not build it.
Key points
Verification is confirmation, through objective evidence, that specified requirements have been fulfilled (ISO 9000:2015, 3.8.12).
Validation is confirmation, through objective evidence, that the requirements for a specific intended use or application have been fulfilled, as adopted by the NIST AI Risk Management Framework (AI RMF).
AI RMF MEASURE 2.5 requires showing the deployed system is valid and reliable and documenting the limits of generalizing beyond the conditions it was developed under.
EU AI Act Article 10 requires training, validation and testing data sets that are relevant, sufficiently representative and, as far as possible, free of errors and complete.
US bank supervisors replaced SR 11-7 with SR 26-2 in April 2026; it keeps effective challenge and validation of conceptual soundness and outcomes, but excludes generative and agentic AI.
Effective challenge is critical analysis by objective experts with the expertise, independence and organizational standing to force changes to a model.
ISO/IEC 42001 Annex A control A.6.2.4 requires defined and documented verification and validation measures for the AI system.
Exam tip
Verification means built right; validation means the right thing was built for its intended use. Older material cites SR 11-7 as current US model risk guidance, but SR 26-2 superseded it in April 2026.
An audit is an independent check that the AI governance program does what it says. The organization audits itself on a schedule, and an accredited certification body can audit it against ISO/IEC 42001 to issue a certificate. Public-sector auditors also have AI-specific question sets to work from.
Internal audit (first party)
Certification audit (third party)
Governing standards
ISO/IEC 42001 clause 9.2
ISO 19011 guidelines
ISO/IEC 42006
ISO/IEC 17021-1
Purpose
Conformity and effectiveness
Feeds management review
Independent certificate
Issued by an accredited body
The organization audits its own AI management system under clause 9.2, while an accredited certification body follows ISO/IEC 42006 on top of ISO/IEC 17021-1.
Key points
ISO/IEC 42001 clause 9.2 requires internal audits at planned intervals, run from an audit programme, to show the AI management system (AIMS) conforms and is effectively maintained.
Internal auditors are selected to ensure objectivity and impartiality, and results go to relevant management and feed the clause 9.3 management review by top management.
ISO/IEC 42006:2025 sets additional requirements for bodies that audit and certify AI management systems against ISO/IEC 42001, supplementing ISO/IEC 17021-1.
ISO 19011 gives guidelines for auditing management systems; its fourth edition, ISO 19011:2026, replaced the 2018 edition.
The US Government Accountability Office (GAO) AI Accountability Framework rests on four principles, governance, data, performance and monitoring, with questions and procedures for auditors.
Under EU AI Act Article 43, Annex III points 2 to 8 use internal control (Annex VI), while biometric systems may need a notified body assessment (Annex VII).
ADG (Adopt, Defend, Govern) MC-10 requires governance reviews with exception handling and board reporting for high-risk systems, evidenced by review records.
Exam tip
Clause 9.2 of ISO/IEC 42001 governs the internal audit of your own AIMS; ISO/IEC 42006 governs the certification body that audits you. Do not confuse the auditee standard with the auditor standard.
Auditors trust what they can see. Every control needs a named piece of evidence, such as a signed test report, a change log or minutes from a governance review, kept where it can be found and retained long enough. The ADG (Adopt, Defend, Govern) framework's rule is blunt: if it cannot be evidenced, it cannot be relied upon.
diagram
Each ADG Minimum Control names an evidence artifact an auditor can hold, and EU AI Act high-risk duties add technical documentation and logs.
Key points
ADG (Adopt, Defend, Govern) foundational principle 7 is that evidence is mandatory: if a control cannot be evidenced, it cannot be relied upon.
Evidence by control: MC-1 published AI inventory, MC-3 responsible, accountable, consulted and informed (RACI) matrix, MC-4 signed evaluation report, MC-5 approved change log.
More evidence: MC-9 incident playbook and yearly tabletop, MC-10 governance review records, MC-11 fairness evaluation report and MC-12 signed responsibility matrix.
ADG operating artifacts also include the AI System Profile, AI Incident Evidence Pack, Governance Exception Register and Vendor AI Due Diligence Record.
EU AI Act Article 11 requires technical documentation (Annex IV) before market placement, and Article 12 requires automatic event logging over the system's lifetime.
Providers keep technical, quality management and conformity documents for 10 years (Article 18) and automatically generated logs for at least six months (Article 19).
ISO/IEC 42001 requires documented information as evidence of audit programmes and results (clause 9.2) and of nonconformities and corrective actions (clause 10.2).
Exam tip
When asked what an auditor needs for a control, name the artifact, not the intention: a signed evaluation report for MC-4, a change log for MC-5, a tabletop record for MC-9.
Evidence register entry for one Minimum Control (placeholder values) · json
{
"control": "MC-9",
"artifact": "AI incident playbook and 2026 tabletop record",
"system": "contoso-claims-assistant",
"owner": "ir-lead@example.com",
"location": "https://grc.example.com/evidence/mc-9/2026",
"sha256": "<hash of the stored file>",
"collected": "2026-09-30",
"mapsTo": ["EU AI Act Art. 73", "ISO/IEC 42001 10.2"]
}
An audit finding is only useful if something changes. Findings go to the people who can act, owners agree a fix with a date, and someone checks later that the fix actually worked. Risks that will not be fixed are formally accepted, time-limited and visible to leadership.
diagram
Findings go to the owner for a root-cause fix, unresolved risk goes to the AI Governance Council as a time-limited exception, and closure waits for evidence that the fix works.
Key points
ISO/IEC 42001 clause 10.2: react to the nonconformity, evaluate and remove its causes, implement action, review corrective action effectiveness and update the management system if needed.
Clause 9.3 management review considers the status of actions from previous reviews, nonconformities and corrective actions, monitoring results and audit results.
ADG (Adopt, Defend, Govern) process step 7 is remediation with retest and governance review, and step 9 is a board-level reporting cycle for high-risk AI systems.
The ADG Governance Exception Register records each exception with justification, risk acceptance and expiration, and the AI Governance Council owns it.
ADG outcome metrics track AI-related compliance findings from internal or external audit and governance exceptions still open beyond their expiration date.
ADG board-level indicators include AI compliance coverage, exception backlog trend and AI incident trend.
The US Government Accountability Office (GAO) AI Accountability Framework gives auditors and third-party assessors questions and procedures for each key practice.
Exam tip
A finding closes only after the corrective action is verified as effective, not when the action is assigned. An accepted risk needs a justification, risk acceptance and an expiry date in the exception register.