AI security architecture applies familiar principles, such as secure by design, least privilege and defense in depth, to new assets like model weights, prompts, context and agent tools. This section covers how to design, build, protect, test and watch AI systems against AI-specific attacks.
Building security into an AI system at the blueprint stage is cheaper and safer than bolting it on later, just as wiring a house for alarms is easier before the walls go up. The leading government guidance splits the job into four stages and asks for safe defaults at each one. Frameworks such as Google's SAIF then map AI-specific risks to the controls that address them.
diagram
The UK-US guidelines organize AI security into four lifecycle areas, with lessons from operation feeding back into design.
Key points
The UK-US Guidelines for Secure AI System Development (27 November 2023) cover four areas: secure design, secure development, secure deployment, and secure operation and maintenance.
The guidelines take a secure-by-default approach: the most secure setting should be the default, and riskier capabilities need an explicit user opt-in.
Secure design starts with threat modelling AI-specific risks and confirming that AI is the most appropriate way to solve the task at all.
Imported third-party models and serialised weights should be treated as untrusted code, scanned and sandboxed, because loading them can enable remote code execution.
Deploying AI Systems Securely (2024) calls for secure-by-design principles and a Zero Trust (ZT) architecture to manage risks to and from the AI system.
Google's Secure AI Framework (SAIF) maps 15 AI security risks, from data poisoning to rogue actions, to controls and to the party able to mitigate each one.
EU AI Act Article 15 requires high-risk systems to achieve appropriate accuracy, robustness and cybersecurity and to perform consistently throughout their lifecycle.
Exam tip
When one option adds security testing just before launch and another threat-models at design and sets secure defaults, pick the design-stage answer. Secure by design is a lifecycle duty, not a final gate.
ADG breaks every AI system into nine places where engineers add controls and attackers try to get in, much like listing every door and window of a building before deciding where the locks go. The list works whether the AI is homegrown, called through an API or embedded in a SaaS product. What changes between those cases is who owns each surface.
diagram
The nine ADG surfaces grouped by the first pillar in each one's ADG alignment. Most are shared by two pillars, for example Context is GOVERN plus DEFEND.
Key points
ADG (Adopt, Defend, Govern) names nine governance surfaces: Model, Prompt, Context, Tools, Orchestration, Identity, Safety Layer, Telemetry and Learning Loop.
Each surface has a control objective; for Tools it is least privilege, strong validation, sandboxing and full audit logging.
ADG treats context as attack surface: retrieval sources, memory and session state need provenance and access control to prevent poisoning, leakage and cross-session contamination.
ADG holds that tool use is the highest-risk control plane, because risk escalates when a model moves from producing output to taking real-world action.
The Identity surface covers credentials, service accounts and delegated authority, with the objective of tracing every agent action back to human authority.
ADG consumes four threat catalogs: MITRE ATT&CK, MITRE ATLAS, the OWASP Top 10 for large language model (LLM) applications and the OWASP Top 10 for Agentic Applications.
A consumer of a foundation model application programming interface (API) still owns seven surfaces; model governance and the learning loop are shared with the vendor.
Exam tip
Buying AI as an API or SaaS shifts some surfaces to the vendor, but accountability for use never fully transfers. Context, Identity and Telemetry stay with the customer even for SaaS AI.
In an AI system, a one-line prompt edit or a new model file can change behavior as much as a code release. Version control keeps every one of those changes recorded, signed and reversible, like track changes for the whole system. Governance then decides which versions are allowed into production.
diagram
Prompt, model and configuration changes follow the same signed, reviewed path as code, and only an approved change becomes a signed release.
Key points
Deploying AI Systems Securely (2024) recommends version control with access controls for all code and artifacts, including models, parameters, configurations, data and tests.
NIST SP 800-218A, the Secure Software Development Framework (SSDF) profile for generative AI (July 2024), extends secure storage (PS.1.1) to model weights, pipelines and reward models.
SP 800-218A PS.2.1 asks providers to publish cryptographic hashes or digital signatures for a model, its components and documentation, and to sign model changes.
ADG (Adopt, Defend, Govern) Minimum Control 5 (MC-5) puts changes to prompts, tools, models and retrieval sources through governed change control, evidenced by a change log.
ADG governs system prompts and guardrail configurations as controlled text with versioning, approval workflows and rollback, kept in GitOps-style repositories.
The UK-US secure AI guidelines advise treating major changes to data, models or prompts like new versions, because they can change system behavior.
Supply-chain Levels for Software Artifacts (SLSA) defines provenance attestations for build artifacts; version 1.2 is the current specification.
Exam tip
Prompts are configuration, not chat history. An answer that lets engineers edit a production system prompt without change control violates ADG MC-5.
Model weights are the crown jewels of an AI system: whoever copies them owns the model. So access is limited to a few people under two-person rules, everything is encrypted, and the keys sit in special hardware. Training data and logs get similar treatment because they can reveal secrets too.
Limit who gets in
Protect the bits
Model weights
Least privilege
Two-person control
MFA for admins
Encrypt at rest
Keys held in an HSM
Hash each release
Training data and logs
RBAC or ABAC
Access matches data sensitivity
Encrypt at rest and in transit
Treat logs as sensitive data
Weights, training data and logs each need both access limits and cryptographic protection. The cells summarize the joint deployment guidance and NIST SP 800-218A.
Key points
Deploying AI Systems Securely recommends role-based access control (RBAC), or preferably attribute-based access control (ABAC), to limit model access to authorized personnel.
The same guidance limits access to model weights to privileged users under two-person control and two-person integrity.
Encrypt model weights, outputs and logs at rest with keys in a hardware security module (HSM), and protect data in transit with current Transport Layer Security (TLS).
Administrators need phishing-resistant multifactor authentication (MFA) and privileged access workstations, kept separate from ordinary users.
NIST SP 800-218A applies least privilege to model weights and suggests encryption, multi-party authorization or air-gapped environments where the risk warrants it.
SP 800-218A also advises limiting access to a model trained on sensitive data to people already allowed to see that data.
The UK-US secure AI guidelines say to treat logs as sensitive data and protect their confidentiality, integrity and availability.
Exam tip
To stop model weight theft, the strongest answer layers least privilege with two-person control, encryption with HSM-held keys and monitoring. A single control is the weaker option.
Most people reach an AI model through an API, so the API is the front door. A good front door checks who is calling, limits how much they can ask for, filters what goes in and out, and keeps a record. Without those checks, attackers can run up huge bills or slowly copy the model by asking it questions.
diagram
An API gateway in front of the model authenticates callers, enforces quotas and size limits, strips extraction aids such as logprobs and logs every call.
Key points
Deploying AI Systems Securely says exposed application programming interfaces (APIs) need authentication, authorization and secure protocols such as HTTPS.
Validate and sanitize all input at the API so malicious content, such as prompt injection, is less likely to reach the model.
OWASP's Unbounded Consumption risk covers denial of wallet and model extraction via the API; mitigations include rate limits, user quotas, timeouts and input size limits.
In the 2026 OWASP Top 10 for large language model (LLM) applications, Unbounded Consumption rose to LLM06:2026 from LLM10:2025.
Restricting logits and log probabilities in responses, and returning only the minimum data a task needs, makes model extraction and model inversion harder.
Protect the boundary between the IT environment and the AI system, and keep a way to cut all inbound connections to models during a major incident.
Model Context Protocol (MCP) clients should block requests to private and reserved IP ranges and consider egress proxies to prevent server-side request forgery (SSRF).
Exam tip
For denial of wallet or model extraction, rate limiting and quotas are the expected first answer, backed by monitoring for unusual query patterns.
Each kind of attack on a model has its own countermeasure, the way a building needs different defenses against fire, flood and burglary. Poisoning is fought with clean, traceable data; evasion with robust training; privacy attacks with differential privacy; and model theft with query limits. None of these defenses is perfect, so they are layered. Testing them is covered in 5.2.
diagram
Each attack family has its own main defenses in NIST AI 100-2 E2025. All of them are layered because none is complete.
Key points
NIST AI 100-2 E2025 (March 2025) is NIST's taxonomy of adversarial machine learning (AML) attacks and mitigations for predictive and generative AI.
Against evasion, three defense classes have proven resilient: adversarial training, randomized smoothing and formal verification, each trading some accuracy for robustness.
Against poisoning, NIST lists training data sanitization and robust training, backed by access control, data validation and dataset provenance and integrity attestation.
Differential privacy mitigates membership inference and data reconstruction, but it does not protect against model extraction, because it protects training data, not the model.
Model extraction is mitigated by limiting queries, detecting suspicious query patterns and hardening against side channels, though well-resourced attackers can circumvent these.
NIST warns that most AML mitigations are empirical, without information-theoretic security proofs, so they may fail against new attack techniques.
EU AI Act Article 15(5) expects high-risk systems to prevent, detect, respond to and control data poisoning, model poisoning, adversarial examples, confidentiality attacks and model flaws.
Exam tip
Match the control to the attack: provenance and sanitization for poisoning, adversarial training for evasion, differential privacy for membership inference, query limits for extraction.
Guardrails are the checks wrapped around a model, like the bumpers in a bowling lane: they filter what goes in, inspect what comes out and stop actions that break the rules. Because a model can always be talked into something, downstream systems must not simply trust its output. ADG calls this whole layer the Safety Layer.
diagram
Checks sit on both sides of the model, and the downstream system still enforces its own authorization instead of trusting the model's output.
Key points
ADG (Adopt, Defend, Govern) defines the Safety Layer as guardrails, policy engines, semantic filters, classifiers, circuit breakers and harm detectors, owned by DEFEND.
Its control objective is to block unsafe content, unfair outputs and unauthorized actions before they have impact.
ADG agentic hardening adds semantic firewalls before tool use and before response release, plus human override and emergency stop pathways.
OWASP notes large language models (LLMs) make no architectural distinction between instructions and data, so prompt injection defenses must be layered.
Improper Output Handling (LLM10:2026, formerly LLM05:2025) means model output must be validated and sanitized before any downstream system uses it.
OWASP's complete mediation advice is to enforce authorization in downstream systems rather than let the model decide whether an action is allowed.
ADG governs guardrail configurations as controlled text with versioning and change control, so a filter change is reviewed like a code change.
Exam tip
Guardrails reduce risk but never make a model trustworthy on their own. Prefer answers that add downstream authorization and human approval for high-impact actions.
An AI agent does not just answer; it acts, by calling tools such as email, databases or payment systems. Every tool is a new way for a tricked agent to cause real damage, so each one is registered, rated for risk, limited to the least access it needs and logged. The Model Context Protocol (MCP) is a common way agents connect to tools, and it has its own security rules.
diagram
Every tool call passes a policy check against the tool register, high-impact actions wait for a human, the server gets a narrowly scoped token, and the decision is logged for replay.
Key points
ADG (Adopt, Defend, Govern) Minimum Control 7 (MC-7) requires a register of all tools and Model Context Protocol (MCP) capabilities with trust tiering and invocation controls.
ADG's minimum MCP requirements: an approved server inventory, capability trust tiering, per-invocation policy checks, auditable traces and no implicit trust in tool-supplied instructions.
An Agent Authority Statement, an ADG operating artifact, defines what an agent may access, decide, execute and escalate.
The MCP security best practices forbid token passthrough: an MCP server must not accept tokens that were not explicitly issued to it.
MCP guidance recommends progressive least-privilege scopes: start with low-risk read operations and elevate only when a privileged operation is first attempted.
The OWASP Top 10 for Agentic Applications 2026 (December 2025) runs from ASI01 Agent Goal Hijack and ASI02 Tool Misuse and Exploitation to ASI10 Rogue Agents.
OWASP's least-agency principle extends least privilege: do not grant autonomy a task does not need, because unnecessary agency expands the attack surface.
Exam tip
For agent questions, the governance answer limits what the agent can do, with scoped tools and approval for irreversible actions, rather than trusting the model to behave.
Red teaming means hiring friendly attackers to break your AI before real ones do, by jailbreaking it, poisoning it or tricking its tools. Findings feed a fix-and-retest loop, and nothing ships until the release gate is signed. Vulnerability management carries on after launch, because new attacks and new flaws in AI libraries keep appearing.
diagram
Red-team findings flow into fixes and retests before the release gate, and the loop repeats after launch as new attacks appear.
Key points
ADG (Adopt, Defend, Govern) Minimum Control 4 (MC-4) requires quality, safety, security, fairness and failure-mode testing before production, evidenced by a signed evaluation report.
The UK-US secure AI guidelines say to release models only after security evaluation such as benchmarking and red teaming, and to tell users about known limitations.
The OWASP GenAI Red Teaming Guide (January 2025) covers four areas: model evaluation, implementation testing, infrastructure assessment and runtime behavior analysis.
Deploying AI Systems Securely advises inspecting imported pre-trained models in a secure zone and scanning them with approved AI-specific scanners before use.
The same guidance says to apply vendor patches for GPUs, CPUs and memory, preferably tracked through the Common Security Advisory Framework (CSAF).
The secure AI guidelines expect a route for security researchers to report vulnerabilities and published bulletins when issues are disclosed.
ADG's Maturity phase moves to continuous assurance, for example automated weekly red-team runs and persistent adversarial monitoring.
Exam tip
Red teaming is evidence for the release gate, not a substitute for it. The governance answer requires findings to be fixed and retested, with GOVERN signing off or recording an exception.
Once an AI system is live, you watch it the way a security operations center watches a network, but with new signals: prompts, outputs, tool calls and configuration. Slow changes usually mean the world has drifted; sudden changes can mean an attack. Good logs also let investigators replay exactly what happened.
diagram
The shape of a change guides the response: gradual degradation points to drift review, while an abrupt shift is treated as a possible attack.
Key points
Deploying AI Systems Securely calls for logs of inputs, outputs, intermediate states and errors, with automated alerts and triggers.
The same guidance says to monitor model architecture and configuration for unauthorized changes and to watch for attempts to elicit data or aggregate inference responses.
It also recommends user and entity behavior analytics (UEBA) and immutable backup storage so that log data cannot be altered.
The multi-agency AI Data Security guidance notes gradual performance change usually signals data drift, while abrupt change suggests an attack such as poisoning.
EU AI Act Article 12 requires high-risk systems to log events automatically over their lifetime, supporting traceability and post-market monitoring.
ADG (Adopt, Defend, Govern) builds Minimum Control 8 (MC-8) runtime monitoring detection rules from MITRE ATT&CK and MITRE ATLAS.
ADG treats logs and traces as sensitive data needing retention, redaction and access governance; forensic replay supports AI incident response (see 5.1).
Exam tip
Logging prompts and outputs is expected, but logs are sensitive data too. The best answer pairs monitoring with redaction, access control and retention rules.