Login breaks after an account moves to an enterprise plan. Tests for the validator pass. Tests for the gateway pass. An assistant reads both files and suggests checking the account settings.
It hasn't seen the middleware between them. A larger context window can keep that missing hop beside its callers, but only if you include it. The interesting question is which relationships survive when you assemble the prompt.
The bug between two passing tests
Consider this small example. The request starts with enterprise_plan="team". The validator accepts it, a middleware allowlist removes the field, and the gateway rejects the request because the field is missing. Each component's isolated test can pass while their composition fails.
| Component | What it does | Why its isolated test can pass |
|---|---|---|
validator.py | Accepts enterprise_plan="team" | Its input includes a valid plan |
plan_transform.py | Copies only allowlisted fields | Its test doesn't require the new plan field |
gateway.py | Requires enterprise_plan | Its test supplies the field directly |
Retrieving only the validator and gateway leaves the transformation out of view. A prompt with all three files can support a specific diagnosis: the allowlist drops a required field. An integration test that sends the request through all three components would check that diagnosis.
These three small files don't need a million tokens. The same problem becomes harder when the path spans a substantial codebase, generated schemas, configuration, and tests. Long context gives you room to preserve those relationships without summarizing every file first. Iterative search can also find the missing hop; the choice is how to gather sufficient evidence, not whether search or long context wins in general.
What a million-token window buys
Start with token count, not pages. A token is a model-specific unit, so it isn't a fixed number of words or pages.
Code, tables, logs, OCR output, and different languages tokenize differently. Count the real corpus with the target model's tokenizer before choosing a route. BPE, WordPiece, and SentencePiece explains why counts move across models.
Even a model migration can move that boundary. Anthropic reports that Claude Sonnet 5 produces about 30% more tokens than Sonnet 4.6 for the same text. The same corpus can consume more of a 1M-token budget and change request cost under the new tokenizer.[1]
Budget the complete request. Source files share space with instructions, tool definitions, prior messages, and tool results. A shared context window also needs room for generated tokens, including reasoning tokens where the API counts them. A maximum output length isn't extra space beyond that window.[2]
For example, a 1,050,000-token window with 128,000 tokens reserved for output leaves 922,000 for input. If instructions and history already use 22,000, the corpus budget is 900,000. These are planning numbers, not a promise that a provider will admit every request at the boundary.
The selected routes below were checked on September 2, 2026. They show why you must read both capacity and billing rules. Dollar pairs are standard USD input/output prices per million tokens, excluding cache discounts and other fees. This isn't a complete model catalog.
| Provider route | Documented capacity | Operational constraint |
|---|---|---|
| Claude Sonnet 5 | 1M context; 128K maximum output[1] | 1M is billed at the same per-token rate as a short prompt ($2 / $10). Recount with the new tokenizer; use prompt caching for stable prefixes.[3][4] |
| GPT-5.6 Sol | 1,050,000 context; 128K maximum output[5] | Prompts above 272K input bill 2x input and 1.5x output for the full request. Cache writes are 1.25x uncached input.[5][6] |
| Gemini 3.1 Pro Preview | 1,048,576 input; 65,536 output[7] | Still preview. Prompts above 200K: $4 / $18 versus $2 / $12. Cache hits and storage are billed separately.[8][9] |
| Grok 4.6 | 500K context[10] | Prompts at or above 200K double input, cached-input, and output rates for the whole request. Matching prefixes may get automatic cache discounts; hits aren't guaranteed.[11][12] |
Google also lists Gemini 3.1 Flash-Lite with a 1,048,576-token input limit and standard text input/output prices of $0.25 / $1.50, without Pro's 200K price switch.[13][8] xAI lists 1M for Grok 4.3 despite Grok 4.6's 500K limit.[11] Newer and larger-context aren't synonyms.
Read these rows as admission and billing rules. They say when a request is accepted and which price band applies. They don't say where retrieval or reasoning stays accurate.
💡 Key insight: A request can fit within the context limit and still fail its task. Capacity and accuracy need separate checks.
🎯 Production tip: Price the whole request on each side of a threshold. On GPT-5.6 Sol, crossing 272K input tokens changes the rates for the full request, not only the extra tokens. Claude Sonnet 5 instead keeps the same per-token rate across its window.[5][3]
Where capacity becomes unreliable
Fitting is only the first test. A prompt can be accepted and still fail because the model misses evidence, prefill takes too long, or repeated turns repay the same input.
Evidence can fit and still be missed
Lost in the Middle evaluated multi-document question answering and key-value retrieval. Models in those experiments often used evidence near the beginning or end better than evidence in the middle.[14] The pattern depends on the task and model, so it isn't a universal attention curve.
Separate two measurements:
- Advertised context: maximum token count accepted by the route.
- Effective context: the range of lengths, evidence positions, and distractor loads where your task meets its quality target.
Suppose plan_transform.py lands in the middle of a 400K dump. The file fits, but the model can still miss it. Test several file orders rather than turning a historical position effect into a universal instruction to put everything at the edges. File names and boundaries should remain explicit in every version.
Prefill latency grows
Before the model generates its first answer token, it processes the whole prompt. That pass is called prefill.
Standard full attention compares token pairs, so its compute grows quadratically with sequence length.[15] FlashAttention reduces memory traffic for exact attention, but it doesn't remove that pair count.[16]
Cold prefill can dominate the wait for a long prompt, but the user's time to first visible token also includes queueing and any hidden reasoning before the answer. Measure that complete delay under realistic load. A fast attention kernel doesn't guarantee a responsive chat interface.
Repeated input compounds cost
The first request isn't the only request. Retries and follow-up turns can resend the same large working set.
Prompt caching can reduce repeated prefix processing, but provider rules differ and hits depend on prompt stability.[4][6][9][12]
More input can also change answer quality. This failure mode is often called context rot or retrieval decay.
Chroma's technical report tested 18 models across controlled tasks and found nonuniform degradation as input length increased.[17] Treat that report as evidence of a risk, not a universal score for every model or workload.
Evaluate effective context
Now replace the route maximum with a measured boundary. Ask: at what length, evidence position, and distractor load does the actual task stop clearing its quality target?
Needle in a Haystack inserts one fact at different prompt depths and checks exact recovery.[18] It measures a retrieval floor, not multi-document synthesis. A pass still doesn't prove that the model can join evidence across documents.
RULER adds retrieval, multi-hop tracing, aggregation, and question answering across long inputs.[19] Those tasks expose failures that a single hidden fact can't.
Build a workload sweep around the failure you care about. For the login bug, the unit isn't "can the model recite a buried string." It's "can it join the validator, the transform, and the gateway when two of those files are far apart and a pile of unrelated services is in the way?"
| Test | Vary | Failure it catches |
|---|---|---|
| Single-fact retrieval | Total length and needle depth | Position-sensitive recall |
| Multi-evidence answer | Evidence count, order, and distance | Missed joins and contradictions |
| Distractor stress | Relevant-to-irrelevant ratio | Context dilution |
| End-to-end task | Route, cache state, and concurrency | Quality, TTFT, and cost regression |
For the login example, score three things separately: did the response name the missing field, identify the transform that drops it, and propose a test that actually exercises that path? A response that only says "check middleware" hasn't completed the diagnosis.
Keep the bug and expected answer fixed while adding unrelated files. Sweep evidence positions and repeat each configuration across cases and runs. Record success counts as well as rates, latency percentiles, and cost. One lucky correct answer at 1M tokens isn't an effective-context measurement.
What the technique stack changes
Evaluation gives you a failure boundary. To understand what moves that boundary, separate model-position methods from attention kernels, KV memory, and sequence distribution.
Long-context systems combine methods that solve different problems:
| Layer | Examples | Actual job |
|---|---|---|
| Position and training | RoPE, position interpolation, YaRN, LongRoPE, ALiBi | Train or adapt a model to use farther token positions[20][21][22][23][24] |
| Attention kernel | FlashAttention | Reduce reads and writes between accelerator memory levels while computing exact attention[16] |
| KV memory and serving | GQA/MQA, PagedAttention | Reduce KV heads or allocator fragmentation; these are separate levers[25][26] |
| Sequence distribution | Ring Attention | Distribute long sequences across devices when you own the training or serving stack[27] |
These techniques aren't interchangeable. RoPE supplies positional information; methods such as position interpolation, YaRN, and LongRoPE adapt how those positions are represented when extending a trained model. They don't remove the need to store attention state. PagedAttention reduces allocation waste and supports sharing cached blocks; it doesn't shrink each stored key or value.[26]
The KV cache makes the serving cost concrete. During autoregressive generation, each new token reuses keys and values for previous tokens instead of recomputing them. Those vectors stay live for the sequence, so token count, KV-head count, and precision show up directly in memory:
Here, is layer count, is cached tokens, is KV-head count, is head dimension, and is bytes per value. The leading 2 counts keys and values.
One token in a 32-layer decoder with eight KV heads, head dimension 128, and two-byte values needs bytes of cached keys and values. Multiply by one million tokens and you get 131.1 GB, or about 122.1 GiB, for one sequence.
This estimate assumes every layer keeps full-history keys and values with the same dimensions and precision. Sliding-window layers, compressed attention state, and shared prefixes need architecture-specific accounting. It excludes model weights, temporary activations, and allocator overhead, so it isn't a total GPU-memory estimate.
The script below turns that estimate into a small comparison. It varies sequence length and KV-head count, then prints both decimal gigabytes and binary gibibytes:
1def estimate_kv_cache(
2 layers: int,
3 seq_len: int,
4 kv_heads: int,
5 head_dim: int,
6 bytes_per_elem: int = 2,
7) -> dict[str, float | int]:
8 total_bytes = 2 * layers * seq_len * kv_heads * head_dim * bytes_per_elem
9 return {
10 "bytes": total_bytes,
11 "gb": total_bytes / 1e9,
12 "gib": total_bytes / (1024**3),
13 }
14
15scenarios = [
16 ("128K GQA (8 heads)", 32, 128_000, 8, 128),
17 ("500K GQA (8 heads)", 32, 500_000, 8, 128),
18 ("1M GQA (8 heads)", 32, 1_000_000, 8, 128),
19 ("1M MHA (32 heads)", 32, 1_000_000, 32, 128),
20]
21
22print(f"{'Scenario':<22} | {'Tokens':>10} | {'KV Heads':>8} | {'GB (10^9)':>10} | {'GiB (2^30)':>10}")
23print("-" * 70)
24for name, layers, seq_len, kv_heads, head_dim in scenarios:
25 res = estimate_kv_cache(layers, seq_len, kv_heads, head_dim)
26 gb_str = f"{res['gb']:.1f} GB"
27 gib_str = f"{res['gib']:.1f} GiB"
28 print(f"{name:<22} | {seq_len:>10,d} | {kv_heads:>8d} | {gb_str:>10} | {gib_str:>10}")1Scenario | Tokens | KV Heads | GB (10^9) | GiB (2^30)
2----------------------------------------------------------------------
3128K GQA (8 heads) | 128,000 | 8 | 16.8 GB | 15.6 GiB
4500K GQA (8 heads) | 500,000 | 8 | 65.5 GB | 61.0 GiB
51M GQA (8 heads) | 1,000,000 | 8 | 131.1 GB | 122.1 GiB
61M MHA (32 heads) | 1,000,000 | 32 | 524.3 GB | 488.3 GiBA one-million-token prompt fits the model's context. Does that prove one-million tokens of KV cache fit for a live sequence?
Answer
No. Cache size also scales with layers, KV heads, head dimension, bytes per value, and active sequences. Apply the estimate with serving headroom, then measure admission and latency on the target hardware.
When to load the corpus and when to retrieve
The cache estimate answers a hardware question. It doesn't answer whether you should load the whole corpus. Long context and RAG solve different parts of that decision.
| Strategy | Use it when | Main risk |
|---|---|---|
| Full context | Corpus is bounded and cross-document relationships drive the answer | Prefill latency, repeated-token cost, and missed evidence |
| Retrieval first | Corpus is large and most artifacts are irrelevant to each request | Retriever can omit evidence needed for a join |
| Hybrid | Retrieval can narrow the corpus without breaking relationships | More evaluation and pipeline complexity |
A hybrid path can retrieve likely files, expand their dependencies, and keep the resulting slice together. For the login bug, following the call from validator to middleware is more useful than increasing the number of keyword matches. Production RAG Pipelines covers retrieval and reranking; Long Context Window Management covers placement and compaction.
Retrieve a high-recall set, count it with the target tokenizer, pack a stable prefix, then score the real task:
For repeated reviews of the same source revision, put stable files early in the prompt and the changing question after them. When a file changes, invalidate the affected content. A high cache-hit rate on an old revision would make the wrong review cheaper.
OpenAI's GPT-5.6 family supports implicit caching plus explicit breakpoints, bills writes at 1.25 times uncached input, and lets you set a cache key for a shared prefix.[6]
Google exposes context caching, xAI caches matching prefixes automatically, and Anthropic supports explicit prompt caching. For Sonnet 5, Anthropic's 5-minute writes cost 1.25x base input and cache hits cost 0.1x.[9][12][4][3] Prefix Caching and Prompt Caching covers the serving-side reuse mechanics.
⚠️ Common mistake: Reusing one giant cold prompt on every turn. Cache a stable prefix when reuse is real. Retrieve or compact when evidence changes.
For the broken login, success is a correct explanation backed by a failing-then-passing integration test. Whether that takes three carefully retrieved files or a much larger dependency slice is something you can measure. Choose the smallest evidence set that reliably supports the diagnosis, then test how it behaves under traffic. Scaling LLM Inference follows that final step into batching and KV-cache pressure.