Context Engineering

Long-context window management answers the mechanics of making a big context window work: KV-cache math, RoPE scaling, prefill-versus-decode bottlenecks, and where the lost-in-the-middle penalty bites. It answers "how do I fit more tokens into one model call and serve it cheaply?"

The engineering question changes: "given that I can fit a lot of tokens, what should go in the window, and what should stay out?" That discipline is called context engineering. Window management provides capacity; context engineering decides what evidence, tools, history, and state deserve that capacity.

The shift matters because more tokens don't guarantee better answers. A model may accept a very large prompt yet perform worse on a workload than it does with a smaller curated packet. The job is no longer only "make room." It's also to compare curation policies on answer quality, latency, and cost.

Motivation: the agent that got worse as it learned more

Picture an incident agent investigating why deploy RUN-842 failed its canary. It reads the deploy record, checks CI logs, searches rollback runbooks, and follows request traces. Every tool call dumps its raw output back into the conversation. After forty turns the context holds: the original alert, four full runbooks (most of which were a dead end), six multi-thousand-token trace exports, three issue-search results, and one early hallucinated guess that a database migration lock caused the outage.

The agent now performs worse than it did at turn five. It re-runs the dead-end issue search, cites a migration lock that doesn't exist, and picks a verbose, irrelevant rollback paragraph over the trace span that matters. The window is nowhere near full, yet the agent is failing.

This isn't a window-management problem. The KV cache fits, latency is fine, and RoPE isn't stretched. The context has accumulated low-signal and even wrong tokens, and the model is dutifully attending to all of them. Context engineering is the set of techniques that would have prevented this failure. The failed-canary agent will anchor the rest of the chapter.

Prompt engineering grew up into context engineering

For a few years the applied-AI conversation was dominated by prompt engineering: finding the right words and phrasing for a single instruction. Anthropic frames context engineering as the natural successor to that practice.^{[1]Reference 1Effective context engineering for AI agentshttps://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents} Prompt engineering is about writing one good instruction. Context engineering is the broader discipline of curating and maintaining the entire set of tokens present during inference: the system prompt, tool definitions, retrieved documents, conversation history, tool results, and any memory loaded back in.

Anthropic's framing is worth memorizing because it gives you a single guiding principle:

Find the smallest set of high-signal tokens that maximize the likelihood of your desired outcome.^{[1]Reference 1Effective context engineering for AI agentshttps://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents}

That's the whole job in one sentence. Every technique below is a way to push toward that minimal high-signal set. A 2025 survey of the field collects the same strategies under a formal taxonomy, confirming this is now a named subdiscipline rather than a collection of tips.^{[2]Reference 2A Survey of Context Engineering for Large Language Models.https://arxiv.org/abs/2507.13334}

Why curation needs evaluation: context rot

The reason you shouldn't assume "just add more" is empirical, not stylistic. Chroma's Context Rot report evaluated 18 models across increasing input lengths and reported non-uniform performance as input grew, including on simple retrieval and copying tasks.^{[3]Reference 3Context Rot: How Increasing Input Tokens Impacts LLM Performancehttps://research.trychroma.com/context-rot} The magnitude and shape depend on model, task, and distractors; additional tokens are a hypothesis to evaluate, not free signal.

Anthropic describes the same engineering concern with an "attention budget" mental model and recommends seeking the smallest high-signal token set that supports the desired outcome.^{[1]Reference 1Effective context engineering for AI agentshttps://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents} Padding a window with low-signal tokens always increases input cost and can lower workload quality; a paired evaluation should determine when.

Context rot is one reason behind the techniques below. Cost, latency, stale state, and contradictory evidence are others. Curation should be an explicit candidate policy with quality checks, not an article of faith.

Four failure modes of an overloaded context

Before fixing context, you need vocabulary for how it breaks. Drew Breunig describes four useful failure-mode labels for long contexts.^{[4]Reference 4How Long Contexts Failhttps://www.dbreunig.com/2025/06/22/how-contexts-fail-and-how-to-fix-them.html} They are diagnostic categories, not a formal completeness proof. Each one can show up in our failed-canary agent.

Don't turn reported examples into universal thresholds. Breunig cites an agent anecdote where long history encouraged repeated actions and a tool-use experiment where reducing tool count improved one model's benchmark result.^{[4]Reference 4How Long Contexts Failhttps://www.dbreunig.com/2025/06/22/how-contexts-fail-and-how-to-fix-them.html} Those observations justify testing history pruning and tool gating on your model, tools, and task distribution.

The write, select, compress, isolate taxonomy

Naming failure modes tells you what went wrong. LangChain organizes agent context strategies into four useful buckets: write, select, compress, and isolate.^{[5]Reference 5Context Engineering for Agentshttps://blog.langchain.com/context-engineering-for-agents/} A useful mental model from that work: the LLM is like a CPU and its context window is like RAM, a working set to manage deliberately. The buckets classify many common tactics without claiming they exhaust every design.

Write: keep state outside the window

The cheapest token is the one you never put in the window. Write means persisting information outside the context so it doesn't consume the attention budget until it's needed.^{[5]Reference 5Context Engineering for Agentshttps://blog.langchain.com/context-engineering-for-agents/} The classic pattern is a scratchpad: the agent writes notes, plans, or intermediate findings to a file or a state field, then reloads only the relevant note later. Agentic memory works the same way, persisting durable facts across sessions.^{[6]Reference 6Agent Memory: How to Build Agents that Learn and Rememberhttps://www.letta.com/blog/agent-memory}

For the failed-canary agent, a write strategy means: instead of leaving four full runbooks in the conversation, the agent records "auth callback errors confirmed; migration lock ruled out" as a one-line note and drops the raw tool results. The finding survives; the tokens don't.

Select: pull in only what this step needs

Select means retrieving only the tokens relevant to the current step.^{[5]Reference 5Context Engineering for Agentshttps://blog.langchain.com/context-engineering-for-agents/} Retrieval-augmented generation (RAG) is the canonical example, and you already studied it: a high-recall retriever surfaces a handful of relevant chunks instead of dumping the whole corpus. But selection applies to more than documents. Tool selection is a select problem too: the confusion failure mode above is what happens when you fail to gate tools and load all 50 instead of the 5 this task needs.

Prompt caching, which you met earlier, is an orthogonal efficiency tactic: if a stable prefix is reused, a provider may avoid repeating part of the prefill computation.^{[7]Reference 7Prompt caching.https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching} Caching does not select better evidence or reduce the number of tokens the model reasons over. It improves eligible repeated-prefix economics only when the cache policy and provider support it.

Compress: shrink what must stay

When information has to stay in the window, compress reduces it to the required tokens.^{[5]Reference 5Context Engineering for Agentshttps://blog.langchain.com/context-engineering-for-agents/} Two common candidate patterns follow.

The first is compaction: when a conversation approaches the budget, summarize it and start a fresh window seeded with that summary.^{[1]Reference 1Effective context engineering for AI agentshttps://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents} The agent keeps its working knowledge but sheds the verbose transcript that produced it.

The second is tool-result pruning, a low-risk candidate fix for our failed-canary agent. Anthropic describes clearing old tool results as a light-touch form of compaction: once the relevant finding has been captured, old raw results can often leave the active context.^{[1]Reference 1Effective context engineering for AI agentshttps://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents} Preserve evidence that the next decision still needs, and compare quality before adopting an aggressive pruning policy.

Isolate: split work across focused windows

Isolate means splitting context across focused workers so no single window has to hold every exploratory trace.^{[5]Reference 5Context Engineering for Agentshttps://blog.langchain.com/context-engineering-for-agents/} A lead agent delegates a focused subtask, such as "check linked issues and rollback notes for every failed-canary exception", to a worker with its own clean context window. The worker does noisy exploration in isolation and returns a bounded evidence summary to the lead.^{[1]Reference 1Effective context engineering for AI agentshttps://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents}

Isolation can reduce distraction and confusion because the lead doesn't need every intermediate search result. It also adds coordination overhead and creates clash risk when worker outputs disagree, so isolate is a candidate for separable exploration, not a default.

Putting it together: rebuilding the failed-canary agent's context

With the taxonomy in hand, the running example fixes itself. Instead of letting the window grow monotonically, the agent runs a curation step before each model call:

1. Select only the tools relevant to the current phase (an investigation needs deploy lookup and traces, not cluster administration), then evaluate whether tool-call accuracy improves.
2. Write durable findings to a scratchpad ("auth callback errors confirmed; migration lock ruled out") and drop the raw tool results.
3. Compress by pruning trace exports older than a few turns and compacting the transcript once it grows large.
4. Isolate the noisy "check every linked issue" exploration into a sub-agent that returns a one-paragraph summary.
5. Detect poisoning: when the early migration-lock guess is identified as wrong, remove it from history so it stops being cited.

The window now holds the alert, current evidence, a short notes block, and a clean tool set. In the concrete script below, it shrinks an illustrative 12.4K-token raw set to a 3.2K-token active window plus short external notes. Lower input cost is immediate; better task performance still requires evaluation.

The logic is simple enough to simulate with a tiny script. This version turns the failed-canary agent's messy transcript into a compact working set by keeping the current alert and evidence, writing durable findings to notes, and dropping poisoned or irrelevant tokens.

Why this matters as windows grow

Large-window model offerings make it tempting to treat curation as obsolete.^{[8]Reference 81M context is now generally available for Opus 4.6 and Sonnet 4.6https://claude.com/blog/1m-context-ga} Bigger windows raise the capacity ceiling; they don't establish quality for an overloaded agent trace. Frameworks such as LangGraph expose short- and long-term memory plus summarization or deletion patterns because state management remains an application responsibility.^{[9]Reference 9LangGraph Memory Overviewhttps://docs.langchain.com/oss/python/concepts/memory}

The practical upshot: when an agent underperforms, inspect the transmitted context alongside model and window choices. Name a suspected failure mode, apply a bounded candidate change, and measure whether it improves the task.

Common pitfalls

Use this checklist before shipping an agent that handles long sessions:

• Does the context shrink or stay bounded across turns, or does it only grow? Unbounded growth invites distraction and rot.
• Are tool results pruned or compacted once they are no longer needed for the next step?
• Does a phase-gated active tool set outperform loading every available tool on your evaluation cases?
• When a sub-agent is used, does it return a distilled summary rather than its full transcript?
• Do you have an eval that catches poisoning, that's, a wrong fact persisting and being re-cited across turns?

Diagnostic playbook

When a long-running agent underperforms, use this sequence:

1. Inspect the actual context, not the window size or cost alone.
2. Name the failure mode: poisoning, distraction, confusion, or clash.
3. Choose the matching move: write, select, compress, or isolate.
4. Rebuild the next call around the smallest high-signal working set.
5. Compare baseline and curated calls on accuracy, latency, token cost, and failure recurrence.
6. Only after that ask whether you still need a different model, window, or architecture.

Long-context window management gave you the machinery to hold many tokens. Context engineering adds the judgment to decide which tokens earn their place. The next chapter shifts to model architecture, where sparse routing changes the compute-per-token economics that all of this sits on top of.

Mastery check

Before moving on, make sure you can do all of these without hand-waving:

• Explain how context engineering differs from prompt engineering and from window management, and why all three are distinct skills.
• Define context rot and use it to justify evaluating a curated alternative even when a prompt fits.
• Diagnose an agent failure as poisoning, distraction, confusion, or clash from its symptoms, and name a fix for each.
• Classify RAG, summarization, scratchpads, and focused workers into write, select, compress, or isolate, and explain why prompt caching instead changes repeated-prefix cost.
• Rebuild the failed-canary agent's next call around a small high-signal working set and explain what stays in the window, what moves to notes, and what gets dropped.
• Explain why a larger supported window doesn't remove the need for curation evaluation.

Evaluation rubric

• Strong: You name the failure mode, pick the matching move, and specify what stays in the active window, what moves outside it, and what should be deleted.
• Partial: You notice bloat and suggest summarization or a bigger window, but you don't diagnose poisoning, distraction, confusion, or clash with enough specificity.
• Weak: You treat "it still fits" as proof the context is healthy, or you leave poisoned and stale tokens in place because capacity remains.

Follow-up questions

Our agent has a 1M-token window. Why bother curating context at all?

Because window size sets capacity, not quality. Context-rot evaluations report that reliability can decline as input grows, so a technically valid prompt still deserves comparison with a curated candidate. The relevant measurements are task quality, cost, latency, and whether the context contains the evidence needed for the answer.

How do you decide between compressing the context and isolating it into a sub-agent?

Compress when the information must remain available to the same reasoning thread but is too verbose, for example a long transcript that you summarize or stale tool results you prune. Isolate when a subtask is noisy and self-contained, such as a broad search that will generate dozens of intermediate results the main agent never needs to see. Isolation buys the cleanest separation but adds coordination overhead and a risk of clash between sub-agent outputs, so prefer compression for in-thread bloat and reserve isolation for genuinely separable, exploration-heavy subtasks.

An agent keeps repeating a wrong fact about an incident. Which failure mode is this and how do you fix it?

That's context poisoning: an error entered the context and is now being re-referenced on later turns. Appending a correction may not help, because both the wrong fact and the correction stay in view. A strong candidate fix is to remove disproven notes from the rebuilt context (or compact the transcript without them), then run an evaluation that checks whether the wrong fact is still re-cited.

How many tools is too many to load into one agent's context?

There's no universal limit. Treat overlapping tool descriptions and irrelevant tool calls as symptoms to measure. Test a phase-gated tool set against the full registry on representative tasks, then retain the smallest set that preserves coverage and improves selection accuracy. That's a selection fix for the confusion failure mode.

You're curating the failed-canary agent's next call. What belongs in the window, what should move to notes, and what should be dropped?

Keep only the current alert, latest trace span, applicable rollback runbook, short scratchpad, and small tool set needed for this phase. Move durable findings such as "auth callback errors confirmed" or "migration lock ruled out" into notes so the knowledge survives without bloating the active window. Drop stale logs, irrelevant tools, and any poisoned guesses that the agent has already proven false. That answer uses the full taxonomy at once: write out durable facts, select current evidence, compress or prune old history, and isolate anything noisy enough to deserve its own clean sub-window.