Domain 4: Development and Implementation Governance

CRAGE Domain 4: Development and Implementation Governance Study Guide

1. Domain Overview and Detailed Review

Domain 4 of the CRAGE framework focuses on Development and Implementation Governance (representing 27% of the exam). This domain addresses how organizations inventory AI assets, implement secure architecture, establish data provenance, ensure privacy and safety, and continuously audit AI systems throughout their lifecycle.

4.1 AI Lifecycle and Asset Management

AI Asset Inventory and Ownership

• Core Principle: You cannot govern what you do not know about. An AI inventory acts as a register of every AI system, its accountable owner, risk rating, autonomy tier, and dependencies (data, models, tools, people).
• Key Framework Requirements:
  • ADG Minimum Control 1 (MC-1): Requires an inventory of all AI systems with an accountable owner, risk classification, and autonomy tier. The ADG Foundation phase begins with MC-1, MC-2, and MC-3.
  • ADG AI System Profile: Records system architecture, risk classification, autonomy tier, harm classes, and deployment patterns.
  • NIST AI RMF (GOVERN 1.6): Calls for mechanisms to inventory AI systems based on organizational risk priorities.
  • ISO/IEC 42001 Annex A.4: Mandates documentation of AI resources, including general resources (A.4.2), data (A.4.3), tooling (A.4.4), system/computing (A.4.5), and human resources (A.4.6).
  • EU AI Act (Article 49): Requires providers to register Annex III high-risk AI systems in the EU database (Article 71) prior to market placement.
  • NIST SP 800-218A: Notes that AI models and components must be added to asset inventories when a release is archived.
  • UK-US Guidelines: Assets to track and protect include models, data, user feedback, prompts, software, documentation, logs, and assessments.
• Exam Tip: The definitive first step of an AI governance program is building an AI inventory with named accountable owners.

Data Governance for Training Data

• Core Principle: Training data forms the foundation of model learning; poor or biased inputs result in compromised outputs. Data governance establishes rules for data sourcing, validation, cleaning, labeling, and access.
• Key Regulatory & Framework Controls:
  • EU AI Act Article 10: Governs training, validation, and testing data for high-risk systems. (Under Regulation (EU) 2026/1744 / Digital Omnibus, Article 10 applies from 2 December 2027 for Annex III systems).
  • EU AI Act Article 10(2) & 10(3): Requires examination and mitigation of biases impacting health, safety, or fundamental rights. Datasets must be relevant, representative, error-free, and complete to the best extent possible.
  • EU AI Act Article 4a: Created by the Digital Omnibus (deleting Article 10(5)) to house strictly necessary permissions for using special category data for bias detection.
  • GDPR Article 5(1)(c): Mandates data minimisation—personal data must be adequate, relevant, and limited to purpose.
  • ISO/IEC 42001 Annex A.7: Specifies controls for development data (A.7.2), acquisition (A.7.3), quality (A.7.4), provenance (A.7.5), and preparation (A.7.6).
  • ADG Warning: Storage-layer access control is insufficient for AI because retrieval mechanisms and tool connections collapse discrete sources into a single text stream.
• Exam Tip: Governance of training data requires representativeness, documented origin, and bias examination combined—encryption alone addresses security, not data governance.

Data Lineage and Provenance Tracking

• Definitions:
  • Provenance: The birth certificate of data (origin and authoritative sign-off).
  • Lineage: The travel diary of data (movement and transformations across pipelines).
• Implementation & Security Guidelines:
  • Multi-Agency AI Data Security Guidance (May 2025): Recommends authoritative data sourcing, provenance tracking, pipeline logging, checksums/cryptographic hashes, digital signatures on revisions, and append-only stores.
  • Split-View Poisoning Mitigation: Curators must publish raw data hashes; consumers must discard downloads failing hash validation to defend against expired domain abuse in URL lists.
  • ADG MC-6 (Context Policy): Defines provenance, retention, access restrictions, and trust ordering for context inputs. Covers output attribution to trace AI responses back to source documents.
  • NIST SP 800-218A (PS.3.2): Mandates tracking model/component provenance using artifacts such as a Software Bill of Materials (SBOM).

Restorable AI Assets and Known-Good State

• Core Principle: Restoring system integrity after compromise or failure requires archived, hashed, and traceable model weights, datasets, and prompts linked to approval records.
• Framework Guidance:
  • CISA / Joint Guidance (Deploying AI Systems Securely, 2024): Recommends encrypted, hashed copies of model releases stored in tamper-proof locations with keys in a Hardware Security Module (HSM), alongside automated rollback capabilities.
  • NIST SP 800-218A (PS.3.1): Archives releases with integrity verification, provenance, training logs, choice rationale, and stored fine-tuning/alignment data.
  • ADG Assets: Vector stores and embedding infrastructure are treated as governed data assets needing backup, recovery, encryption, and access control.
• Exam Tip: A valid recovery point requires an integrity-checked, previously approved version. Restoring an unverified backup risks redeploying a poisoned model.

Model Cards and Technical Documentation

• Documentation Types:
  • Model Cards: Short summaries detailing intended use, evaluation procedures, and demographic performance.
  • Datasheets: Data-side documentation covering motivation, composition, collection, and recommended use.
  • EU AI Act Article 11 / Annex IV Technical Documentation: Legal conformity files detailing system architecture, data provenance, design choices, and oversight, retained for 10 years after market placement (Article 18).
• Standards & Adaptations:
  • Digital Omnibus on AI: Extended simplified technical documentation privileges from SMEs to small mid-cap companies.
  • ISO/IEC 42001: Controls A.6.2.7 (technical documentation) and A.6.2.8 (event logging).

Model Registry and Version Governance

• Core Principles: Official cataloging of model versions coupled with independent release approval workflows.
• Regulatory & Control Standards:
  • Federal Reserve SR 26-2: US banking model risk guidance (replacing SR 11-7 in April 2026) requiring comprehensive inventories for individual and aggregate model risk analysis.
  • ADG MC-3: Enforces segregation of duties by separating deployment ownership, security validation, and approval authority (builders cannot approve their own releases).
  • ADG MC-5 (Model and Prompt Change Log): Logs all changes to models, prompts, configurations, and approval records.
  • NIST AI RMF (MANAGE 3.2): Third-party pre-trained base models must be tracked and monitored within the registry as part of regular maintenance.

4.2 AI Security Architecture and Design Controls

Secure-by-Design AI Architecture

• UK-US Guidelines Structure: Secure design, secure development, secure deployment, and secure operation/maintenance. Recommends secure-by-default settings and sandboxing/scanning imported model weights as untrusted code to prevent remote code execution (RCE).
• Framework Integration:
  • Deploying AI Systems Securely (2024): Advocates Zero Trust (ZT) architecture.
  • Google SAIF (Secure AI Framework): Maps 15 AI security risks to specific controls and mitigating parties.
  • EU AI Act Article 15: Mandates appropriate accuracy, robustness, and cybersecurity across the high-risk AI lifecycle.

ADG Governance Surfaces as Security Boundaries

ADG outlines nine governance surfaces applicable across in-house, API, or SaaS deployments:

Note: API/SaaS consumers delegate Model and Learning Loop surfaces to vendors but retain ownership of the remaining seven surfaces (e.g., Context, Identity, Telemetry).

Version Control for Code, Models, and Prompts

• Scope: Access-controlled versioning must cover code, models, parameters, configurations, data, prompts, and test suites.
• Standards:
  • NIST SP 800-218A: Tasks PS.1.1 (secure storage) and PS.2.1 (cryptographic signatures/hashes for model components).
  • ADG MC-5: Enforces change control and change logging for prompts and guardrails stored in GitOps repositories.
  • SLSA (Supply-chain Levels for Software Artifacts): Version 1.2 defines provenance attestations for build artifacts.

Access Control and Encryption for AI

• Privileged Protection:
  • Role-Based Access Control (RBAC) or Attribute-Based Access Control (ABAC).
  • Model weights restricted under two-person control and two-person integrity.
  • Phishing-resistant Multi-Factor Authentication (MFA) and Privileged Access Workstations (PAWs) for administrators.
• Encryption Standards:
  • Encryption at rest for weights, outputs, and logs using keys stored in Hardware Security Modules (HSMs).
  • Data in transit secured via modern Transport Layer Security (TLS).
  • Access to models trained on sensitive data limited to personnel authorized for the underlying data.

Securing AI APIs and Network Paths

• Threat Mitigations:
  • OWASP Unbounded Consumption (LLM06:2026 / formerly LLM10:2025): Prevents denial of wallet and model extraction via rate limits, quotas, timeouts, and input size restrictions.
  • Extraction/Inversion Hardening: Restrict response logits and log probabilities; return minimal required output.
  • Network Isolation: Maintain capability to sever inbound model connections during incidents.
  • Model Context Protocol (MCP): MCP clients must block requests to private/reserved IP ranges and use egress proxies to block Server-Side Request Forgery (SSRF).

Hardening Models Against Adversarial Machine Learning (AML)

NIST AI 100-2 E2025 details AML attacks and countermeasures:

[Adversarial Attacks]
 ├── Evasion Attacks      ──> Defended by: Adversarial Training, Randomized Smoothing, Formal Verification
 ├── Poisoning Attacks     ──> Defended by: Data Sanitization, Provenance, Hashes, Robust Training
 ├── Privacy Attacks      ──> Defended by: Differential Privacy (membership inference/reconstruction)
 └── Model Extraction     ──> Defended by: Query Limits, Pattern Detection, Side-Channel Hardening

Note: Most AML defenses are empirical and lack information-theoretic proofs.

Guardrails and the Safety Layer

• ADG Safety Layer: Comprises semantic filters, policy engines, circuit breakers, and harm detectors managed by DEFEND.
• OWASP Principles:
  • Improper Output Handling (LLM10:2026 / formerly LLM05:2025): Model outputs must be validated/sanitized before passing to downstream systems.
  • Complete Mediation: Enforce access authorization within downstream execution layers rather than delegating authority to model judgment.

Agent, Tool, and Model Context Protocol (MCP) Security

• ADG MC-7: Mandates a register of tools and MCP capabilities featuring trust tiering, invocation policy checks, and full audit logging.
• Agent Authority Statement: Formally bounds what an agent may access, decide, execute, and escalate.
• MCP Security Rules: Strictly forbids token passthrough (servers cannot accept unissued tokens); enforces progressive least-privilege scopes.
• OWASP Agentic Top 10 (2026): Highlights risks from ASI01 (Agent Goal Hijack) to ASI10 (Rogue Agents). Enforces the least-agency principle.

Vulnerability Management and Red Teaming

• ADG MC-4: Requires pre-production testing across quality, safety, security, and fairness, documented via a signed evaluation report.
• OWASP GenAI Red Teaming Guide (2025): Encompasses model evaluation, implementation testing, infrastructure assessment, and runtime behavior analysis.
• Vulnerability Remediation: Imported base models must be sandboxed and scanned. Software patches for GPUs, CPUs, and memory must be tracked via the Common Security Advisory Framework (CSAF).

AI Runtime Monitoring and Detection

• ADG MC-8: Sets detection rules built from MITRE ATT&CK and MITRE ATLAS.
• Drift vs. Attack Identification:
  • Gradual performance decline indicates data drift.
  • Abrupt performance shift indicates an attack or poisoning.
• Compliance & Privacy: EU AI Act Article 12 mandates automated event logging for high-risk systems. Log streams must be treated as sensitive data, requiring redaction, encryption, UEBA, and immutable backups.

4.3 Privacy, Trust, Safety, and Ethical Controls

Privacy by Design for AI Systems

• GDPR Requirements: Article 25 (Privacy by Design/Default), Article 5(1)(c) (Data Minimisation), and Article 35 (Data Protection Impact Assessments / DPIAs for high-risk processing).
• EDPB Opinion 28/2024: Models trained on personal data cannot be automatically assumed anonymous; anonymity must be proven case-by-case.

Differential Privacy and Federated Learning

• Differential Privacy (NIST SP 800-226): A mathematical framework quantifying individual privacy loss via parameter epsilon ( ϵ**)**.
  • Smaller Epsilon ( ϵ*)* = Stronger privacy, lower analytical accuracy.
  • Properties: Protection composes across releases and resists future arbitrary side-information attacks.
• Federated Learning: Distributes training across local nodes, sending model updates instead of raw data to an aggregator.
  • Required Safeguard: Federated learning alone is incomplete; requires secure aggregation/homomorphic encryption for input privacy and differential privacy for output privacy.

Anonymization vs. Pseudonymization

[Dataset Processing]
 ├── Pseudonymization (GDPR Art. 4(5)) ──> Identifiers replaced by codes; Key held separately.
 │                                          └── STILL Personal Data under GDPR (Recital 26 / EDPS v SRB).
 └── Anonymization                     ──> Irreversible removal of identifiability.
                                            └── Outside GDPR scope.
                                            └── EDPB 2026 Test: No isolation, no linkage, no inference.

Trustworthiness Trade-Offs in Practice

NIST AI RMF stresses that addressing trustworthiness characteristics independently is insufficient. Common operational tensions include:

1. Interpretability vs. Privacy
2. Predictive Accuracy vs. Interpretability
3. Privacy-Enhancing Techniques (PETs) vs. Accuracy (especially on sparse data)

Governance requires documenting and justifying these trade-offs transparently within the operational context rather than attempting to maximize one metric to the detriment of system safety.

Safety Controls and Human Oversight

• EU AI Act Article 14: High-risk oversight measures must allow humans to override, reverse outputs, or halt the system via a stop button. Overseers must explicitly mitigate automation bias (over-reliance on system recommendations).
• NIST AI RMF (MEASURE 2.6): Requires demonstrating that systems fail safely when operated beyond knowledge boundaries.

Transparency Duties to AI Users

EU AI Act Article 50 (effective 2 August 2026; Digital Omnibus grace period to 2 December 2026 for pre-existing generative systems):

• Providers: Must disclose when users interact with AI; must mark synthetic media (audio, image, video, text) in machine-readable formats.
• Deployers: Must label deepfakes, disclose unedited public-interest AI text, and notify individuals of emotion recognition or biometric categorization systems.
• Penalties: Non-compliance carries fines up to EUR 15 million or 3% of global annual turnover.

Explainability for Affected People

• NIST IR 8312 Principles: Explanation, Meaningful, Explanation Accuracy, Knowledge Limits.
• Methods:
  • Self-Interpretable: Decision trees, regression models.
  • Post-Hoc Explainability: LIME (Local Interpretable Model-Agnostic Explainer), SHAP (SHapley Additive exPlanations).
• Rights: EU AI Act Article 86 grants individuals affected by Annex III high-risk decisions the right to clear, meaningful explanations from deployers.

Ethics Review and AI Audits

• Standards: ISO/IEC 42005:2025 (AI impact assessment), ISO/IEC 42001 (management systems), and EU AI Act Article 27 (Fundamental Rights Impact Assessment / FRIA).
• ADG Governance Controls:
  • MC-10: Governance reviews, board reporting, and exception tracking via an expiring exception register.
  • MC-11: Fairness evaluations with representative test data.

Continuous Trust Monitoring

• Operational Metrics (ADG): Tracks risk posture scores, exception backlogs, quarterly fairness drift, and vendor exposure.
• Standards & Systems: ISO/IEC 42006:2025 sets requirements for ISO/IEC 42001 certification bodies. Managed feedback loops prevent self-reinforcing bias (EU AI Act Article 15(4)).

2. Short-Answer Quiz

1. What primary control requirement does ADG Minimum Control 1 (MC-1) mandate, and what is its position in the implementation sequence?
2. Under Article 10 of the EU AI Act, what quality standards must training, validation, and testing datasets meet for high-risk AI systems?
3. What is the operational distinction between data provenance and data lineage?
4. How does ADG Minimum Control 3 (MC-3) enforce segregation of duties in model release governance?
5. Why are imported third-party model weights treated as untrusted code under secure-by-design guidelines?
6. Which ADG surface is considered the highest-risk control plane, and why?
7. What is the technical trade-off involved when adjusting the privacy loss parameter epsilon (ϵ) in differential privacy?
8. Under GDPR Recital 26 and relevant case law (EDPS v SRB), why does pseudonymized data remain subject to data protection regulation?
9. What specific human oversight mechanism is mandated by Article 14(4) of the EU AI Act for high-risk AI systems during operation?
10. What disclosure obligations does Article 50 of the EU AI Act place on deployers regarding deepfakes and public-interest content?

3. Answer Key

1. Answer: ADG Minimum Control 1 (MC-1) mandates establishing a complete inventory of every AI system containing an accountable owner, risk classification, and autonomy tier. It represents the foundational first step of an AI governance program, executed alongside MC-2 and MC-3 during the Foundation phase.
2. Answer: Article 10 requires datasets for high-risk AI systems to be relevant, sufficiently representative, and, to the best extent possible, free of errors and complete for their intended purpose. Additionally, providers must implement data governance practices that examine and mitigate biases threatening health, safety, or fundamental rights.
3. Answer: Data provenance acts as a birth certificate that records where data originated and who authorized or vouched for it. Data lineage acts as a travel diary that traces how data moved, transformed, and was processed across systems from ingestion to final model output.
4. Answer: ADG MC-3 enforces segregation of duties by explicitly separating deployment ownership, security validation, and final approval authority. This structure ensures that the developers or engineers who build a model cannot independently approve its release into production environments.
5. Answer: Third-party models and serialized weights are treated as untrusted code because loading unverified binaries can allow arbitrary payload execution, leading to remote code execution (RCE). Consequently, secure-by-design principles require scanning, inspecting, and sandboxing imported weights before integration.
6. Answer: The Tools surface is the highest-risk control plane because risk escalates exponentially when an AI system transitions from generating text to taking real-world actions. Tools connect agents directly to external databases, APIs, and systems, expanding the potential impact of compromised execution.
7. Answer: The parameter epsilon (ϵ) balances privacy protection against data accuracy. A smaller epsilon value provides stronger privacy guarantees by adding more noise, but directly decreases the analytical accuracy and utility of the dataset or model outputs.
8. Answer: Pseudonymized data remains personal data under GDPR because the identity of individuals can still be attributed using separate, additional information. As long as a re-identification key or linking dataset exists within an organization, the data is not anonymous and remains subject to legal requirements.
9. Answer: Article 14(4) mandates that high-risk systems include human oversight controls that permit overseers to disregard, override, or reverse AI outputs. Furthermore, it requires an accessible stop button or equivalent mechanism to halt the system safely during anomalous behavior.
10. Answer: Deployers must explicitly disclose when content is a deepfake (manipulated image, audio, or video) and label AI-generated text published on matters of public interest that lacked human editorial review. They must also inform individuals when they are subjected to emotion recognition or biometric categorization systems.

4. Essay Questions

(Note: Answers are omitted as per directive; these questions are designed for self-study and deep synthesis).

1. Synthesizing AI Governance Frameworks: Analyze how ADG Minimum Controls (MC-1 through MC-11), NIST AI RMF GOVERN/MANAGE functions, and ISO/IEC 42001 Annex A controls overlap to form an integrated enterprise defense model.
2. Adversarial Machine Learning & Security Architecture: Compare the threat vectors of prompt injection, split-view data poisoning, and model extraction. Detail the multi-layered defense-in-depth controls required across the ADG Prompt, Context, Model, and API surfaces to mitigate these risks.
3. Legal vs. Technical Anonymity in AI Lifecycles: Evaluate the EDPB 2026 criteria for anonymization (no isolation, no linkage, no inference) alongside EDPB Opinion 28/2024 regarding trained AI models. Discuss the practical challenges of leveraging personal data for AI development while maintaining GDPR compliance.
4. Agentic Autonomy and Tool Security: Evaluate the security risks introduced by Model Context Protocol (MCP) integrations and agentic AI systems. Formulate a governance control framework incorporating the least-agency principle, Agent Authority Statements, and downstream complete mediation.
5. Managing Trustworthiness Trade-offs Under Legislative Pressure: Discuss the operational tensions between explainability (NIST IR 8312 / EU AI Act Art. 86), differential privacy (NIST SP 800-226), and model accuracy. How should an AI Governance Council structure trade-off analyses and exception handling for high-risk deployments?

5. Comprehensive Glossary

• ABAC (Attribute-Based Access Control): An authorization methodology that grants access rights based on attributes of the user, resource, and environment.
• ADG Framework (Adopt, Defend, Govern): An enterprise governance framework structuring AI lifecycle risk management across nine defined surfaces and eleven minimum controls.
• Adversarial Machine Learning (AML): A discipline studying attacks on ML systems (e.g., evasion, poisoning, extraction) and methods to harden systems against them (NIST AI 100-2 E2025).
• Agent Authority Statement: An operational artifact defining the explicit boundaries of what an AI agent is authorized to access, decide, execute, and escalate.
• Anonymization: The process of irreversibly altering data so that an individual cannot be identified directly or indirectly by any means reasonably likely to be used, placing it outside data protection laws.
• Automation Bias: The propensity for human operators to over-rely on automated system outputs and recommendations, explicitly targeted for mitigation under EU AI Act Article 14.
• Complete Mediation: A security architectural principle requiring that every access request to a downstream system be authorized by the downstream system itself, rather than trusting upstream model outputs.
• Context Policy (ADG MC-6): A defined governance policy establishing provenance, retention limits, access boundaries, and trust hierarchies for context inputs and vector stores.
• Datasheet (for Datasets): A standardized document accompanying a dataset that records its motivation, composition, collection methodology, pre-processing, and recommended usage boundaries.
• Differential Privacy: A mathematical framework (NIST SP 800-226) that quantifies and bounds the privacy risk to individuals in a dataset by injecting calibrated noise, governed by the privacy loss parameter epsilon (ϵ).
• Epsilon ( ϵ**)**: The privacy loss parameter in differential privacy; lower values signify stronger privacy guarantees but lower data utility.
• Federated Learning: A distributed machine learning technique where local nodes train models on decentralized data and transmit parameter updates, rather than raw data, to a central server.
• FRIA (Fundamental Rights Impact Assessment): An assessment required under EU AI Act Article 27 for certain high-risk deployers to evaluate societal and human rights impacts prior to deployment.
• GitOps: A software engineering practice where infrastructure, configurations, guardrails, and prompts are managed as version-controlled code in repositories.
• HSM (Hardware Security Module): A physical computing device that safeguards and manages digital keys, performing encryption and decryption operations within a secure environment.
• Improper Output Handling (OWASP LLM10:2026): A vulnerability where model output is passed to downstream components without validation, sanitization, or context-aware filtering.
• Least-Agency Principle: An extension of least privilege asserting that an AI agent should not be granted more autonomy, tools, or authority than the minimum required to execute its specific task.
• LIME (Local Interpretable Model-Agnostic Explainer): A post-hoc explainability technique that approximates complex model behavior locally around a specific prediction to explain feature contribution.
• Lineage: The tracked historical record detailing how data was moved, transformed, cleaned, and processed through an engineering pipeline.
• Model Card: A short, standardized document accompanying a trained model detailing its intended application, evaluation benchmarks, limitations, and demographic fairness metrics.
• Model Registry: A centralized, audited repository used to track model versions, lineage, artifacts, evaluation results, and release approval statuses.
• Model Context Protocol (MCP): An open standard enabling AI agents to connect securely to tools and context sources, governed by explicit token and network security rules.
• Post-Hoc Explainability: Methods used to interpret complex "black-box" model outputs from the outside by probing inputs and outputs without exposing internal model weights.
• Provenance: The audited record of data origin, ownership chain, and authoritative sign-off.
• Pseudonymization: The processing of personal data such that it can no longer be attributed to a specific data subject without separate, securely stored additional information (GDPR Article 4(5)).
• RBAC (Role-Based Access Control): An access control mechanism that restricts system access based on assigned organizational roles.
• Red Teaming: Controlled, adversarial testing of an AI system's security, safety, alignment, and jailbreak resistance prior to and during production release.
• SHAP (SHapley Additive exPlanations): A game-theoretic post-hoc explainability method that calculates individual feature contributions to a model output based on Shapley values.
• Split-View Poisoning: An attack leveraging domain expiration in curated URL lists to inject malicious content into dataset builds without altering the curator's published URL lists.
• Two-Person Integrity / Control: A security mechanism requiring the simultaneous presence and authorization of two qualified individuals to perform critical tasks, such as accessing raw model weights.
• Unbounded Consumption (OWASP LLM06:2026): A risk state where an AI API lacks rate limits or quotas, exposing the host organization to model extraction or denial-of-wallet costs.
• Zero Trust (ZT): A security architecture premised on the principle that no user, device, or system component inside or outside the network boundary is implicitly trusted.