On this page
Login breaks after an account moves to an enterprise plan. Tests for the validator pass. Tests for the gateway pass. An assistant reads both files and suggests checking the account settings.
It hasn't seen the middleware between them. A larger context window can keep that missing hop beside its callers, but only if you include it. The interesting question is which relationships survive when you assemble the prompt.
The bug between two passing tests
Consider this small example. The request starts with enterprise_plan="team". The validator accepts it, a middleware allowlist removes the field, and the gateway rejects the request because the field is missing. Each component's isolated test can pass while their composition fails.
| Component | What it does | Why its isolated test can pass |
|---|---|---|
validator.py | Accepts enterprise_plan="team" | Its input includes a valid plan |
plan_transform.py | Copies only allowlisted fields | Its test doesn't require the new plan field |
gateway.py | Requires enterprise_plan | Its test supplies the field directly |
Retrieving only the validator and gateway leaves the transformation out of view. A prompt with all three files can support a specific diagnosis: the allowlist drops a required field. An integration test that sends the request through all three components would check that diagnosis.
Before reading the output, predict which check fails. This runnable Python example keeps the three components together so you can inspect the exact field loss. It first checks them separately, then sends the same request through the complete path:
def validate(request):
if request.get("enterprise_plan") != "team":
raise ValueError("invalid enterprise_plan")
return dict(request)
def transform(request, allowed_fields):
return {key: value for key, value in request.items() if key in allowed_fields}
def gateway(request):
if "enterprise_plan" not in request:
raise ValueError("enterprise_plan missing at gateway")
return "session created"
request = {"account_id": "a17", "enterprise_plan": "team"}
old_allowlist = {"account_id"}
assert validate(request) == request
assert transform(request, old_allowlist) == {"account_id": "a17"}
assert gateway(request) == "session created"
print("Three isolated checks: PASS")
try:
gateway(transform(validate(request), old_allowlist))
except ValueError as error:
assert str(error) == "enterprise_plan missing at gateway"
print(f"Composed path: FAIL ({error})")
else:
raise AssertionError("The old allowlist should break the composed path")
new_allowlist = old_allowlist | {"enterprise_plan"}
assert gateway(transform(validate(request), new_allowlist)) == "session created"
print("Composed path after allowlist correction: PASS")Three isolated checks: PASS
Composed path: FAIL (enterprise_plan missing at gateway)
Composed path after allowlist correction: PASSThe transform's isolated check passes because it verifies the old allowlist behavior. The integration check exposes a contract mismatch between that behavior and the gateway's requirement. More tokens help only when they bring this missing relationship into view.
These three small files don't need a million tokens. The same problem becomes harder when the path spans a substantial codebase, generated schemas, configuration, and tests. Long context gives you room to preserve those relationships without summarizing every file first. Iterative search can also find the missing hop. Choose the method that gathers sufficient evidence for the task; search and long context can each help. If you do keep the whole path in the prompt, what does a million-token limit actually include, and where does the price change?
What a million-token window buys
Start with token count. A token is model-specific, and its word or page equivalent varies by content and tokenizer. That count is also the unit on the price card. On some routes the rate changes once the prompt crosses a threshold, while other routes keep one rate.
Code, tables, logs, output from optical character recognition (OCR), and different languages tokenize differently. Count the real corpus with the target model's tokenizer before choosing a route. BPE, WordPiece, and SentencePiece explains why counts move across models.
Even a model migration can move that boundary. Anthropic reports that the newer tokenizer used by Sonnet 5 produces about 30% more tokens than the tokenizer in Sonnet 4.6 and earlier models, depending on the content. The same corpus can consume more of a 1M-token budget and change request cost under the new tokenizer.[1]
Budget the complete request. Source files share space with instructions, tool definitions, prior messages, and tool results. A shared context window also needs room for generated tokens, including reasoning tokens where the API counts them. A maximum output length isn't extra space beyond that window.[2]
For example, a 1,050,000-token window with 128,000 tokens reserved for output leaves 922,000 for input. If instructions and history already use 22,000, the corpus budget is 900,000. Treat this as a planning budget; confirm provider admission limits before sending a request near the boundary.
The selected routes below were checked on October 15, 2026. They show why you must read both capacity and billing rules. Dollar pairs are standard USD input/output prices per million tokens on the direct provider API, excluding cache discounts and other fees. This dated sample illustrates several available routes; check each provider's current catalog for newer models.
| Provider route | Documented capacity | Operational constraint |
|---|---|---|
Claude Sonnet 5 (claude-sonnet-5) | 1M context; 128K maximum output on the regular Messages API[3] | Active legacy model. 1M is billed at the same per-token rate as a short prompt ($2 / $10). Recount with the new tokenizer; use prompt caching for stable prefixes.[1][4] |
GPT-5.6 Sol (gpt-5.6-sol) | 1,050,000 context; 128K maximum output. Reserving the full output budget leaves 922K for input.[5] | Standard $4 / $20; above 272K input, $8 / $30 for the full request. Cache writes are 1.25x uncached input. These promotional rates are guaranteed at least through November 21, 2026.[5][6] |
Gemini 3.1 Pro Preview (gemini-3.1-pro-preview) | 1,048,576 input; 65,536 output[7] | Still preview. Prompts above 200K: $4 / $18 versus $2 / $12. Cache hits and storage are billed separately.[8][9] |
Grok 4.7 (grok-4.7) | 500K context[10] | Standard $2 / $6; at or above 200K input, $4 / $12 for the whole request. Cached input also doubles from $0.50 to $1. Measure cache-hit rates with the exact prefixes in your workload.[11][12] |
Google also lists Gemini 3.1 Flash-Lite with a 1,048,576-token input limit and standard text input/output prices of $0.25 / $1.50, without Pro's 200K price switch.[13][8] xAI lists 1M for Grok 4.3 despite Grok 4.7's 500K limit.[11] Newer and larger-context aren't synonyms.
Read these rows as admission and billing rules. They say when a request is accepted and which price band applies. They don't say where retrieval or reasoning stays accurate.
Key insight: A request can fit within the context limit and still fail its task. Capacity and accuracy need separate checks.
For a concrete threshold check, suppose GPT-5.6 Sol produces 1,000 billable output tokens with caching disabled. At exactly 272,000 input tokens, the cost is $1.108. At 272,001, it becomes $2.206008. One extra input token almost doubles this request's bill because the whole request changes bands. This applies the dated rates above; use a provider usage receipt for the measured API charge.[5]
Production tip: Recount before and after packing tool results or history. Crossing a price threshold can matter more than the number of newly added tokens.
Where capacity becomes unreliable
Fitting is only the first test. A prompt can be accepted and still fail because the model misses evidence, prefill takes too long, or repeated turns repay the same input.
Evidence can fit and still be missed
Lost in the Middle, published in TACL in 2024, evaluated multi-document question answering and key-value retrieval. Models in those experiments often used evidence near the beginning or end better than evidence in the middle.[14] The position effect varies by task and model; measure it on your workload.
Separate two measurements:
- Advertised context: maximum token count accepted by the route.
- Effective context: the range of lengths, evidence positions, and distractor loads where your task meets its quality target.
Suppose plan_transform.py lands in the middle of a 400K dump. The file fits, but the model can still miss it. Test several file orders rather than turning a historical position effect into a universal instruction to put everything at the edges. File names and boundaries should remain explicit in every version.
Prefill latency grows
Before the model generates its first answer token, it processes the whole prompt. That pass is called prefill.
Standard full attention compares token pairs, so its compute grows quadratically with sequence length.[15] FlashAttention reduces memory traffic for exact attention, but it doesn't remove that pair count.[16]
Cold prefill can dominate the wait for a long prompt, but the user's time to first visible token also includes queueing and any hidden reasoning before the answer. Measure that complete delay under realistic load; responsive chat depends on the full request path, including attention-kernel speed.
Repeated input compounds cost
Retries and follow-up turns can resend the same large working set after the first request.
Prompt caching can reduce repeated prefix processing, but provider rules differ and hits depend on prompt stability.[4][6][9][12]
More input can also change answer quality. This failure mode is often called context rot or retrieval decay.
Chroma's July 2025 technical report tested 18 models across controlled tasks and found nonuniform degradation as input length increased.[17] Treat the historical report as risk evidence; evaluate the 2026 routes above on your own tasks.
Evaluate effective context
Now replace the route maximum with a measured boundary. Ask: at what length, evidence position, and distractor load does the actual task stop clearing its quality target?
The single-needle form of Needle in a Haystack inserts one fact at different prompt depths and checks exact recovery.[18] It measures a retrieval floor; multi-document synthesis needs a separate evaluation. The current repository also includes multi-fact recall and UUID-chain tasks, so record which task and scorer you ran. Use multi-document tasks to test whether the model joins evidence across documents.
RULER, introduced in 2024, adds retrieval, multi-hop tracing, aggregation, and question answering across long inputs.[19] Those tasks expose failures that a single hidden fact can't.
Build a workload sweep around the failure you care about. For the login bug, the unit isn't "can the model recite a buried string." It's "can it join the validator, the transform, and the gateway when two of those files are far apart and a pile of unrelated services is in the way?"
| Test | Vary | Failure it catches |
|---|---|---|
| Single-fact retrieval | Total length and needle depth | Position-sensitive recall |
| Multi-evidence answer | Evidence count, order, and distance | Missed joins and contradictions |
| Distractor stress | Relevant-to-irrelevant ratio | Context dilution |
| End-to-end task | Route, cache state, and concurrency | Quality, TTFT, and cost regression |
For the login example, score three things separately: did the response name the missing field, identify the transform that drops it, and propose a test that actually exercises that path? A response that only says "check middleware" hasn't completed the diagnosis.
Add a missing-evidence control: omit the transform entirely. The validator and gateway alone don't establish why the field disappears. A grounded answer should request that missing code or mark the diagnosis as a hypothesis. Also include a corrected-allowlist case where login succeeds. Otherwise, an assistant that always guesses "the middleware drops the plan" could pass every example without reading the evidence.
Keep the bug and expected answer fixed while adding unrelated files. Sweep evidence positions and repeat each configuration across cases and runs. Record success counts as well as rates, latency percentiles, and cost. Measure effective context across repeated cases, positions, and runs.
What the technique stack changes
Evaluation gives you a failure boundary. To understand what moves that boundary, separate model-position methods from attention kernels, key/value memory, and sequence distribution.
Long-context systems combine methods that solve different problems:
| Layer | Examples | Actual job |
|---|---|---|
| Position and training | Rotary position embeddings (RoPE), position interpolation, YaRN, LongRoPE, attention with linear biases (ALiBi) | Train or adapt a model to use farther token positions[20][21][22][23][24] |
| Attention kernel | FlashAttention | Reduce reads and writes between accelerator memory levels while computing exact attention[16] |
| Key/value memory and serving | Grouped-query attention (GQA), multi-query attention (MQA), PagedAttention | Share key/value heads or reduce allocator fragmentation; these are separate levers[25][26][27] |
| Sequence distribution | Ring Attention | Distribute long sequences across devices when you own the training or serving stack[28] |
These techniques act at different layers. RoPE supplies positional information. Methods such as position interpolation, YaRN, and LongRoPE adapt how those positions are represented when extending a trained model; the serving system still stores attention state. PagedAttention reduces allocation waste and supports sharing cached blocks, while each stored key and value retains its dimensions.[27]
The key/value (KV) cache makes the serving cost concrete. During autoregressive generation, each new token reuses keys and values for previous tokens instead of recomputing them. Those vectors stay live for the sequence, so token count, KV-head count, and precision show up directly in memory:
Here, is layer count, is cached tokens, is KV-head count, is head dimension, and is bytes per value. The leading 2 counts keys and values.
One token in a 32-layer decoder with eight KV heads, head dimension 128, and two-byte values needs bytes of cached keys and values. Multiply by one million tokens and you get 131.1 GB, or about 122.1 GiB, for one sequence.
This estimate assumes every layer keeps full-history keys and values with the same dimensions and precision. Sliding-window layers, compressed attention state, and shared prefixes need architecture-specific accounting. The estimate covers KV storage; add model weights, temporary activations, and allocator overhead to budget total GPU memory.
The script below turns that estimate into a small comparison. It varies sequence length and KV-head count, then prints both decimal gigabytes and binary gibibytes. The last row uses 32 KV heads, as in a 32-head multi-head attention (MHA) layer, instead of GQA's eight shared KV heads:
def estimate_kv_cache(
layers: int,
seq_len: int,
kv_heads: int,
head_dim: int,
bytes_per_elem: int = 2,
) -> dict[str, float | int]:
total_bytes = 2 * layers * seq_len * kv_heads * head_dim * bytes_per_elem
return {
"bytes": total_bytes,
"gb": total_bytes / 1e9,
"gib": total_bytes / (1024**3),
}
scenarios = [
("128K GQA (8 heads)", 32, 128_000, 8, 128),
("500K GQA (8 heads)", 32, 500_000, 8, 128),
("1M GQA (8 heads)", 32, 1_000_000, 8, 128),
("1M MHA (32 heads)", 32, 1_000_000, 32, 128),
]
print(f"{'Scenario':<22} | {'Tokens':>10} | {'KV Heads':>8} | {'GB (10^9)':>10} | {'GiB (2^30)':>10}")
print("-" * 70)
for name, layers, seq_len, kv_heads, head_dim in scenarios:
res = estimate_kv_cache(layers, seq_len, kv_heads, head_dim)
gb_str = f"{res['gb']:.1f} GB"
gib_str = f"{res['gib']:.1f} GiB"
print(f"{name:<22} | {seq_len:>10,d} | {kv_heads:>8d} | {gb_str:>10} | {gib_str:>10}")Scenario | Tokens | KV Heads | GB (10^9) | GiB (2^30)
----------------------------------------------------------------------
128K GQA (8 heads) | 128,000 | 8 | 16.8 GB | 15.6 GiB
500K GQA (8 heads) | 500,000 | 8 | 65.5 GB | 61.0 GiB
1M GQA (8 heads) | 1,000,000 | 8 | 131.1 GB | 122.1 GiB
1M MHA (32 heads) | 1,000,000 | 32 | 524.3 GB | 488.3 GiBA one-million-token prompt fits the model's context. Does that prove one-million tokens of KV cache fit for a live sequence?
Answer
No. Cache size also scales with layers, KV heads, head dimension, bytes per value, and active sequences. Apply the estimate with serving headroom, then measure admission and latency on the target hardware.
When to load the corpus and when to retrieve
The cache estimate answers a hardware question. It doesn't answer whether you should load the whole corpus. Long context and retrieval-augmented generation (RAG), which retrieves evidence before generating an answer, solve different parts of that decision.
| Strategy | Use it when | Main risk |
|---|---|---|
| Full context | Corpus is bounded and cross-document relationships drive the answer | Prefill latency, repeated-token cost, and missed evidence |
| Retrieval first | Corpus is large and most artifacts are irrelevant to each request | Retriever can omit evidence needed for a join |
| Hybrid | Retrieval can narrow the corpus without breaking relationships | More evaluation and pipeline complexity |
A hybrid path can retrieve likely files, expand their dependencies, and keep the resulting slice together. For the login bug, following the call from validator to middleware is more useful than increasing the number of keyword matches. Production RAG Pipelines covers retrieval and reranking. Long Context Window Management covers placement and compaction.
Retrieve a high-recall set, count it with the target tokenizer, pack a stable prefix, then score the real task:
For repeated reviews of the same source revision, put stable files early in the prompt and the changing question after them. When a file changes, invalidate the affected content. A high cache-hit rate on an old revision would make the wrong review cheaper.
OpenAI's GPT-5.6 family supports implicit caching plus explicit breakpoints and bills writes at 1.25 times uncached input. For those tokens, the write rate replaces the ordinary input rate, so each token receives one applicable rate. Put an explicit breakpoint after reusable source content in each request when the changing question would otherwise move the cached endpoint. A matching prefix needs an eligible lookup boundary as well as unchanged text. Routing is automatic on GPT-5.6 and later; an optional cache key separates accounting rather than improving routing.[6]
Google exposes context caching, xAI caches matching prefixes automatically, and Anthropic supports explicit breakpoints or request-level automatic caching. Google's Interactions API supports implicit caching only. Use generateContent when you need a managed explicit cache. For Sonnet 5, Anthropic's 5-minute writes cost 1.25x base input and cache hits cost 0.1x. Those multipliers are model-specific. Newer Claude routes can have lower read rates.[9][12][4][1]
For ten reviews sharing a 900K prefix on GPT-5.6 Sol's long-context rate, ten uncached prefix reads cost $72. One write plus nine full cache hits costs $15.48. This comparison covers only the shared prefix. It assumes every later request hits the entire prefix before expiry and excludes changing input, output, and tool fees. If you write that prefix once and never reuse it, its $9 write costs more than an uncached $7.20 read.[29]
Prefix Caching and Prompt Caching covers the serving-side reuse mechanics. Cache savings don't enlarge the context window or remove the stored attention state of the active sequence.
Common mistake: Reusing one giant cold prompt on every turn. Cache a stable prefix when reuse is real. Retrieve or compact when evidence changes.
For the broken login, success is a correct explanation backed by a failing-then-passing integration test. Whether that takes three carefully retrieved files or a much larger dependency slice is something you can measure. Choose the smallest evidence set that reliably supports the diagnosis, then test how it behaves under traffic. Scaling LLM Inference follows that final step into batching and KV-cache pressure.