Prompt Injection Defense

In the previous lesson, you connected an assistant to Model Context Protocol (MCP) tools and resources. That capability creates a trust problem: a tool result can contain facts you need and instructions you must ignore.

Suppose ShopFlow's returns assistant asks an MCP resource for a merchant policy. It receives this:

The first line is useful evidence. The second line is attacker-controlled text. If the model reads both and has refund tools, one poisoned document can become a money movement or privacy incident.

Prompt injection occurs when input content alters a model application's intended behavior. Security reviews focus on attacker-controlled content: a direct injection arrives in the user's message, while an indirect injection arrives inside content the application retrieved or a tool returned. Indirect injection is especially important for agents because the application fetches the payload for the attacker.^[1][2]

OWASP lists prompt injection as LLM01 in its 2025 LLM application risks and recommends constrained behavior, validated output formats, least privilege, human approval for high-risk actions, and adversarial testing.^[3] This lesson turns those principles into a small, testable defense boundary.

By the end, you will implement this rule:

Core rule: Untrusted content may supply evidence. It never grants authority to perform an action.

Trace the attack path

Start by labeling where text came from and what authority it should carry. A policy returned by an MCP server may be operationally useful, but it remains untrusted if an external merchant, customer, email sender, web page, or uploaded file can influence it.

The small inventory below doesn't try to detect malicious wording. It identifies where privilege could cross: untrusted text connected to a sensitive effect.

Attack shapes and delivery paths differ, but none of them should receive authority:

• Direct: A customer types "ignore the refund rules."
• Indirect: An uploaded PDF, web page, email, retrieval chunk, or tool response contains the same sentence.
• Adversarial suffix: A crafted token tail changes behavior or evades filters. Test it as an attack probe; don't treat a screen as authorization.
• Multimodal: An image or audio transcription contributes hostile text. Log extracted text and apply the same trust label.
• Multi-turn: Several innocuous-looking turns accumulate into a request for a forbidden effect. Evaluate the complete trace, not one message.

A useful threat shortcut is the lethal trifecta: an agent can access private data, can read untrusted content, and can communicate externally or cause a consequential effect. Any one capability may be required by the product. Together, they turn an indirect injection into a plausible data-exfiltration or unauthorized-action path.^[4]

ShopFlow has all three ingredients if one model reads merchant policy text, sees private account notes, and can create refunds or outbound links. The goal isn't to promise perfect instruction-following. Break a connection: keep private data out of the untrusted reader, remove write or outbound tools from it, or insert application authorization before any effect.

Prompt structure is a cue, not permission

A model API's message roles and content delimiters tell the model which text is intended as instruction and which text is supplied as data. Use them. They reduce accidental mixing and make tests easier to inspect.

They don't create a security boundary. Both trusted and untrusted text still influence model generation. A model that follows a malicious sentence inside <retrieved_policy> can still propose a dangerous tool call.

This prompt builder makes source and authority explicit. Notice that it preserves the suspicious text for summarization rather than claiming to sanitize the attack away.

A repeated reminder after an untrusted block, sometimes called a sandwich prompt, may further improve reliability. It still lives in the prompt. An allowlist, authorization lookup, spending cap, or approval record lives outside the prompt and can block an effect deterministically.

Quarantine raw content before tools

The riskiest design gives one model both raw external content and write-capable tools. A stronger design splits the work:

1. A reader sees raw content, has no tools, and emits typed evidence.
2. An orchestrator validates evidence and decides which requests are allowed.
3. An executor receives only approved arguments and narrow credentials.

In a live application, the reader may be an LLM constrained to an evidence schema. This deterministic stub demonstrates the contract: the result contains facts and provenance, not commands.

This split isn't a proof that the extracted fact is true. It is a containment pattern: raw attack tokens don't travel directly into the component that can cause a side effect.

Make model output a typed proposal

After reading evidence, a model may propose an answer or an action. Treat either as untrusted output. For tools, reject malformed output and unknown fields before business rules run.

Structured output features can constrain generation to a schema, reducing malformed payloads and unexpected keys.^[5] Schema conformance isn't authorization. A perfectly formed refund request may still be forbidden.

Put authorization outside the model

The orchestrator decides whether a proposal may proceed. Its inputs come from trusted application state: authenticated user identity, order owner, policy limits, approval records, and tool permissions. The model doesn't get to invent any of them.

This first gate turns a suspicious proposal into a blocked decision because issuing a refund isn't an automatically executable action.

Approval alone isn't enough. An approver shouldn't be shown a refund for somebody else's order or an amount beyond policy. An approval identifier isn't authority either: validate a trusted record bound to the same user, order, and amount.

Use credentials that match this decision path. A reader needs no refund credential. An executor should have a refund endpoint only, not arbitrary database write access. A browser or code tool belongs in a sandbox with tight filesystem and network access.

Block disclosure and exfiltration paths

Prompt injection isn't limited to tool calls. A hostile document can ask the model to leak internal notes or send the user to an attacker-controlled return-label URL. Validate outgoing effects and responses for the risks your workflow exposes.

This URL gate blocks a proposed outbound link unless it targets ShopFlow's approved support hosts.

Sensitive data needs an equally explicit rule. Don't place private account notes in the reader context unless that task needs them. Before displaying an answer, scan for protected fields and stop a response that includes them. Minimize accessible data first; leakage checks are a last guardrail.

Use detection as a signal

Pattern matching and classifiers can identify obvious attacks, route work for review, or provide telemetry. They shouldn't determine authorization. An adaptive attacker can phrase a request differently, and a legitimate document may discuss injections while teaching staff about security.

This cheap screen intentionally shows both outcomes: it flags malicious content and it also flags benign training content.

If you add a learned detector, calibrate it on your traffic and still retain policy gates. The detector estimates risk; it can't establish that a refund is allowed.

Evaluate effects, not polite refusals

A secure-looking response isn't your real success condition. The question is whether an attack caused a forbidden effect: an unauthorized refund, note disclosure, unsafe URL, tool call outside allowlist, or external request outside an approved destination.

Build trace fixtures that cover user text, retrieved documents, tool results, extracted media text, and multi-turn histories. The following suite runs a miniature action gate against attack and benign traces.

That suite exposes a bug: the multi-turn trace reached an approved sensitive effect. Fix the approval workflow or policy gate, then run the suite again. Never count "model refused" as safety if a side effect still occurred.

For a release decision, pair attack success rate (ASR) with false rejection rate (FRR) and delivery-path coverage. ASR alone rewards a system that blocks every legitimate request.

Frameworks such as PyRIT and Garak can help run and score adversarial probes, but your product-specific fixtures are still essential: only you know which ShopFlow action, data field, or outbound destination is forbidden.^[6][7]

Production review checklist

Review an agent that reads outside content and can cause effects in this order:

NIST's Generative AI Profile frames risks such as information integrity and information security as risks to map, measure, manage, and govern across the system lifecycle.^[8] That is why your injection defense needs logs and ownership, not only an improved prompt.

Practice: secure the MCP returns assistant

You inherit an assistant that calls read_merchant_policy, inserts returned text into a prompt, and exposes issue_refund to the same model.

Design its fix before writing code:

1. Label the tool result as untrusted evidence.
2. Put raw text in a no-tool reader and permit only a typed policy fact to leave it.
3. Define the JSON proposal schema for any refund request.
4. Implement ownership, amount, and approval checks outside the model.
5. Add a trace where an injected MCP result attempts a refund and assert that no effect executes.
6. Record enough state to explain a denial in an audit review.

A strong answer doesn't promise that the model will never follow injected text. It proves that following the text doesn't authorize the refund.

Key takeaways

• User text, retrieved documents, OCR text, and tool results are untrusted content, even when the application fetched them.
• Private data plus untrusted content plus outbound effects is the lethal-trifecta risk pattern; break at least one connection.
• Message roles, XML tags, and repeated reminders help the model follow intended authority; they don't enforce permissions.
• Keep raw content in a no-tool reader when privileged actions are possible.
• Parse model proposals into strict structures, then authorize using application state and narrow credentials.
• Evaluate unauthorized effects across attack paths, alongside false rejections and coverage.
• Preserve trace logs and control ownership so a failed attack, or a successful one, can be investigated.

Mastery check

Key concepts

• Direct and indirect prompt injection
• The lethal trifecta: private data, untrusted content, and outbound effects
• Untrusted context from retrieval, tools, media extraction, and history
• Soft prompt cues versus hard runtime authorization
• Quarantined reader, orchestrator, and privileged executor
• Strict tool proposal schemas and least-privilege execution
• Outcome-based attack evaluation with ASR, FRR, and coverage

Evaluation rubric

• Foundational: Labels which content is untrusted and distinguishes direct from indirect injection.
• Intermediate: Explains why message roles and delimiters help reliability but can't authorize effects.
• Applied: Implements typed proposals plus policy checks for ownership, limits, approvals, and destinations.
• Advanced: Designs a quarantine boundary and an attack-trace release gate for a tool-using agent.

Common pitfalls

• Symptom: More prompt warnings, same unauthorized action risk. Cause: Text cues were mistaken for an authorization layer. Fix: Gate effects in application code.
• Symptom: Chat attacks fail, but retrieved pages or tool results trigger actions. Cause: Only direct injection was tested. Fix: Add indirect and multi-turn trace fixtures.
• Symptom: JSON is valid, so refund executes. Cause: Schema validation replaced business authorization. Fix: Check identity, ownership, limits, and approval after parsing.
• Symptom: Security test blocks nearly all support requests. Cause: ASR was measured without FRR. Fix: Measure attacks and benign flows together.