Prompt Injection Defense

Model Context Protocol (MCP) tools and resources create a trust problem: a tool result can contain facts you need and instructions you must ignore.

Suppose an ML platform release assistant asks an MCP resource for a candidate model's evaluation summary. It receives this:

The first line is useful evidence. The second line is attacker-controlled text. If the model reads both and has release tools, one poisoned document can become an unauthorized deployment or privacy incident.

Prompt injection occurs when input content alters a model application's intended behavior. Security reviews focus on attacker-controlled content: a direct injection arrives in the user's message, while an indirect injection arrives inside content the application retrieved or a tool returned. Indirect injection is especially important for agents because the application fetches the payload for the attacker.^{[1]Reference 1Ignore Previous Prompt: Attack Techniques For Language Models.https://arxiv.org/abs/2211.09527[2]Reference 2Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection.https://arxiv.org/abs/2302.12173}

OWASP lists prompt injection as LLM01 in its 2025 LLM application risks and recommends constrained behavior, validated output formats, least privilege, human approval for high-risk actions, and adversarial testing.^{[3]Reference 3OWASP Top 10 for Large Language Model Applicationshttps://genai.owasp.org/llm-top-10/} Turn those principles into a small, testable defense boundary.

The implementation centers on this rule:

Core rule: Untrusted content may supply evidence. It never grants authority to perform an action.

Trace the attack path

Start by labeling where text came from and what authority it should carry. An evaluation summary returned by an MCP server may be operationally useful, but it remains untrusted if a benchmark author, issue commenter, web page, uploaded file, or compromised tool can influence it.

The small inventory below doesn't try to detect malicious wording. It identifies where privilege could cross: untrusted text connected to a sensitive effect.

Attack shapes and delivery paths differ, but none of them should receive authority:

• Direct: A user types "ignore the release gates."
• Indirect: An uploaded PDF, web page, email, retrieval chunk, or tool response contains the same sentence.
• Adversarial suffix: A crafted token tail changes behavior or evades filters. Test it as an attack probe; don't treat a screen as authorization.
• Multimodal: An image or audio transcription contributes hostile text. Log extracted text and apply the same trust label.
• Multi-turn: Several innocuous-looking turns accumulate into a request for a forbidden effect. Evaluate the complete trace, not one message.

A useful threat shortcut is the lethal trifecta: an agent can access private data, can read untrusted content, and can communicate externally or cause a consequential effect. Any one capability may be required by the product. Together, they turn an indirect injection into a plausible data-exfiltration or unauthorized-action path.^{[4]Reference 4The lethal trifecta for AI agents: private data, untrusted content, and external communicationhttps://simonwillison.net/2025/Jun/16/the-lethal-trifecta/}

An ML platform has all three ingredients if one model reads untrusted eval text, sees private red-team notes, and can promote candidates or send outbound links. Don't promise perfect instruction-following. Break a connection: keep private data out of the untrusted reader, remove write or outbound tools from it, or insert application authorization before any effect.

Prompt structure is a cue, not permission

A model API's message roles and content delimiters tell the model which text is intended as instruction and which text is supplied as data. Use them. They reduce accidental mixing and make tests easier to inspect.

They don't create a security boundary. Both trusted and untrusted text still influence model generation. A model that follows a malicious sentence inside <retrieved_eval> can still propose a dangerous tool call.

This prompt builder makes source and authority explicit. Notice that it preserves the suspicious text for summarization rather than claiming to sanitize the attack away.

A repeated reminder after an untrusted block, sometimes called a sandwich prompt, may further improve reliability. It still lives in the prompt. An allowlist, authorization lookup, spending cap, or approval record lives outside the prompt and can block an effect deterministically.

Quarantine raw content before tools

The riskiest design gives one model both raw external content and write-capable tools. A stronger design splits the work:

1. A reader sees raw content, has no tools, and emits typed evidence.
2. An orchestrator validates evidence and decides which requests are allowed.
3. An executor receives only approved arguments and narrow credentials.

In a live application, the reader may be an LLM constrained to an evidence schema. This deterministic stub demonstrates the contract: the result contains facts and provenance, not commands.

This split isn't a proof that the extracted fact is true. It's a containment pattern: raw attack tokens don't travel directly into the component that can cause a side effect.

Make model output a typed proposal

After reading evidence, a model may propose an answer or an action. Treat either as untrusted output. For tools, reject malformed output and unknown fields before business rules run.

Structured output features can constrain generation to a schema, reducing malformed payloads and unexpected keys.^{[5]Reference 5Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs} Schema conformance isn't authorization. A perfectly formed promotion request may still be forbidden.

Put authorization outside the model

The orchestrator decides whether a proposal may proceed. Its inputs come from trusted application state: authenticated user identity, project membership, frozen eval status, approval records, and tool permissions. The model doesn't get to invent any of them.

This gate turns a suspicious proposal into a blocked decision because promoting a model isn't automatically executable.

Approval alone isn't enough. An approver shouldn't be shown a promotion for a project they don't own or a stale eval suite. An approval identifier isn't authority either: validate a trusted record bound to the same user, candidate, eval suite, and target environment.

Use credentials that match this decision path. A reader needs no promotion credential. An executor should have a narrow promotion endpoint only, not arbitrary database write access. A browser or code tool belongs in a sandbox with tight filesystem and network access.

Block disclosure and exfiltration paths

Prompt injection isn't limited to tool calls. A hostile document can ask the model to leak internal red-team notes or send the user to an attacker-controlled "review report" URL. Validate outgoing effects and responses for the risks your workflow exposes.

This URL gate blocks a proposed outbound link unless it targets approved ML platform hosts over the expected HTTPS port. It also rejects credential-bearing URLs, which can make a link harder to review correctly.

Treat this helper as one application-layer check, not a complete network boundary. The HTTP client or egress proxy must also validate redirect targets and enforce DNS/IP rules so an approved-looking URL can't reach an unexpected destination after parsing.

Sensitive data needs an equally explicit rule. Don't place private red-team notes in the reader context unless that task needs them. Before displaying an answer, scan for protected fields and stop a response that includes them. Minimize accessible data first; leakage checks are a last guardrail.

Use detection as a signal

Pattern matching and classifiers can identify obvious attacks, route work for review, or provide telemetry. They shouldn't determine authorization. An adaptive attacker can phrase a request differently, and a legitimate document may discuss injections while teaching staff about security.

This cheap screen intentionally shows both outcomes: it flags malicious content and it also flags benign training content.

If you add a learned detector, calibrate it on your traffic and still retain policy gates. The detector estimates risk; it can't establish that a promotion is allowed.

Evaluate effects, not polite refusals

A secure-looking response isn't your real success condition. The question is whether an attack caused a forbidden effect: an unauthorized promotion, note disclosure, unsafe URL, tool call outside allowlist, or external request outside an approved destination.

Build trace fixtures that cover user text, retrieved documents, tool results, extracted media text, and multi-turn histories. This suite runs a miniature action gate against attack and benign traces.

That suite exposes a bug: the multi-turn trace reached an approved sensitive effect. Fix the approval workflow or policy gate, then run the suite again. Never count "model refused" as safety if a side effect still occurred.

For a release decision, pair attack success rate (ASR) with false rejection rate (FRR), uncertainty, delivery-path coverage, and minimum support. ASR alone rewards a system that blocks every legitimate request. A zero point estimate also isn't proof of zero risk: zero successes in a finite sample still has a nonzero upper confidence bound. Coverage says which paths ran, while per-path support says whether each path ran often enough to inform a release.

Frameworks such as PyRIT and Garak can help run and score adversarial probes, but your product-specific fixtures are still essential: only you know which model-release action, data field, or outbound destination is forbidden.^{[6]Reference 6PyRIT Documentationhttps://microsoft.github.io/PyRIT/[7]Reference 7garak Documentationhttps://docs.garak.ai/}

Production review checklist

Review an agent that reads outside content and can cause effects in this order:

NIST's Generative AI Profile identifies information integrity and information security as generative AI risks, then uses the AI RMF functions Govern, Map, Measure, and Manage to organize risk work across the system lifecycle.^{[8]Reference 8Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profilehttps://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence} That framing is why your injection defense needs logs and ownership, not prompt changes alone.

Practice: secure the MCP release assistant

You inherit an assistant that calls read_eval_summary, inserts returned text into a prompt, and exposes promote_model to the same model.

Design its fix before writing code:

1. Label the tool result as untrusted evidence.
2. Put raw text in a no-tool reader and permit only a typed eval fact to leave it.
3. Define the JSON proposal schema for any promotion request.
4. Implement project membership, eval freshness, target, and approval checks outside the model.
5. Add a trace where an injected MCP result attempts a promotion and assert that no effect executes.
6. Record enough state to explain a denial in an audit review.

A strong answer doesn't promise that the model will never follow injected text. It proves an important boundary clearly: following the text doesn't authorize the promotion.

Prompt-injection defense rules

• User text, retrieved documents, OCR text, and tool results are untrusted content, even when the application fetched them.
• Private data plus untrusted content plus outbound effects is the lethal-trifecta risk pattern; break at least one connection.
• Message roles, XML tags, and repeated reminders help the model follow intended authority; they don't enforce permissions.
• Keep raw content in a no-tool reader when privileged actions are possible.
• Parse model proposals into strict structures, then authorize using application state and narrow credentials.
• Evaluate unauthorized effects across attack paths, alongside uncertainty, false rejections, coverage, and minimum support.
• Preserve trace logs and control ownership so a failed attack, or a successful one, can be investigated.

Mastery check

Key concepts

• Direct and indirect prompt injection
• The lethal trifecta: private data, untrusted content, and outbound effects
• Untrusted context from retrieval, tools, media extraction, and history
• Soft prompt cues versus hard runtime authorization
• Quarantined reader, orchestrator, and privileged executor
• Strict tool proposal schemas and least-privilege execution
• Outcome-based attack evaluation with ASR uncertainty, FRR, coverage, and support

Evaluation rubric

• Foundational: Labels which content is untrusted and distinguishes direct from indirect injection.
• Intermediate: Explains why message roles and delimiters help reliability but can't authorize effects.
• Applied: Implements typed proposals plus policy checks for project membership, eval freshness, approvals, targets, and destinations.
• Advanced: Designs a quarantine boundary and an attack-trace release gate for a tool-using agent.

Common pitfalls

• Symptom: More prompt warnings, same unauthorized action risk. Cause: Text cues were mistaken for an authorization layer. Fix: Gate effects in application code.
• Symptom: Chat attacks fail, but retrieved pages or tool results trigger actions. Cause: Only direct injection was tested. Fix: Add indirect and multi-turn trace fixtures.
• Symptom: JSON is valid, so model promotion executes. Cause: Schema validation replaced business authorization. Fix: Check identity, project membership, eval freshness, target, and approval after parsing.
• Symptom: Security test blocks nearly all support requests. Cause: ASR was measured without FRR. Fix: Measure attacks and benign flows together.
• Symptom: A release reports 0% ASR after a small or uneven suite. Cause: The point estimate hides uncertainty and thin path coverage. Fix: Gate on an upper confidence bound, total sample support, and minimum support per required attack path.