Long Context Window Management

Speculative decoding can amortize weight traffic across useful proposed tokens. Long context adds another pressure: prefill must ingest more input, the model must maintain the state its attention architecture needs, and the answer still has to find the right evidence. Full-attention layers retain growing KV caches; local or recurrent layers have different state costs.

Keep one incident in mind. A canary rollback is approved somewhere inside a year's service traces, and someone asks whether it was approved and what caused it. Sending the whole archive may fit a model's advertised limit, but that alone doesn't make the decisive note reachable or the request affordable.

Follow that request through a sequence of decisions. First count the prompt and output budget, then price prefill and cached decode. Next pack evidence so position and retrieval behavior are testable. Finally choose full context, a local window, RAG, or a hybrid and attach latency, memory, and recall gates. Advertised capacity is how many tokens a context window accepts; effective utilization is whether the model can find and reason over the right evidence under those operational limits.

The attention mechanism in a standard transformer still compares tokens to tokens during prompt ingestion, so compute grows quickly with sequence length. FlashAttention^{[1]Reference 1FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.https://arxiv.org/abs/2205.14135} and paged KV-cache serving^{[2]Reference 2Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} reduce memory traffic and allocation waste. They improve the bill without changing what full attention must compare.

Read the figure as three answers to the same incident. Full attention keeps every position visible, at the steepest prefill and memory cost. A sliding window keeps a local band and may block a distant note. Retrieval makes a small prompt first, so the retriever, not the attention pattern alone, decides whether the note gets in.

First gate: can the prompt fit, and what does it cost?

Treat a context window as a shared budget, not as an input-only bucket. System instructions, conversation history, retrieved evidence, the user request, and the output you reserve all compete for the model's limit. A prompt that fits with a ten-token answer may not fit with a useful generation allowance.

For a decoder-only model, start with serialized_input_tokens + reserved_generation_tokens <= context_limit. Count the actual chat template, tool schemas, separators, and any multimodal input using the deployment's tokenizer/accounting rules. Also check separate input/output caps and the server's configured maximum: a checkpoint's supported window, an API request cap, and your tested quality budget needn't match. Reasoning-token accounting is provider-specific. The 128K example below is a dated architecture fixture, not a claim about today's largest window.

The quadratic bottleneck

Ask what happens when a 4K trace grows to 128K before reaching for an optimization. Standard full attention forms $O(n^2)$ token-pair scores in Big-O notation for sequence length $n$. Causal attention has $n(n+1)/2$ valid pairs, still quadratic. Doubling the sequence roughly quadruples those checks; a 32x increase creates about 32² = 1,024x as many valid pairs during prefill. That isn't a 1,024x prediction for total latency: projections, feed-forward layers, and hardware utilization also matter.

That's the first bill. Optimized kernels reduce memory traffic and wall time, but they don't remove the full-attention scaling law. The second bill appears during full-attention decoding: each generated token attends to the cached prefix, so a long prompt becomes both a compute problem and a GPU-memory scheduling problem.

Prefill pays for the pairwise comparison: $O(n^2)$ score computations per layer. Decode pays for keeping prior key/value vectors in the KV cache and reading them again for each new token. Extending the usable range adds a third question about training and careful RoPE scaling. These are separate costs, so one successful benchmark won't answer all three.

Memory in concrete numbers

Put a number on the second bill. Llama 3 70B is an 80-layer decoder with 64 query heads, 8 KV heads, hidden size 8192, and head dimension 128.^{[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783} Llama 3.1 keeps that shape and advertises a 128K window. Use this as a dated fixture: BF16 KV tensors, 2 bytes per cached element, and a 128K prompt (131,072 tokens).

With GQA, the cache holds 8 KV heads, so the calculation gives 40 GiB per active sequence. Replace those 8 heads with the 64 heads of full multi-head attention (MHA) and the cache becomes 320 GiB. The BF16 weight file is about 130 GiB ($70 \times 10^9 \times 2$ bytes). One GQA request fits below the weight file; four concurrent requests need 160 GiB of KV before activations or runtime buffers.

For uniform layers with separate, equal-width K/V arrays, the formula that produced those numbers is:

Read the result as logical, unshared KV per active sequence, summed across all layers. Multiply by concurrent sequences, include generated tokens as the cache grows, then account for paging, prefix sharing, quantization scales, and runtime buffers. A sharded model distributes this state across devices, sometimes with replicated KV heads; don't charge the entire 40 GiB to every GPU or assume perfectly even sharding. The code below isolates the raw tensor term. Its one-byte row models the payload only, not scale metadata or validated FP8 support.

def kv_cache_gib(
    layers: int,
    kv_heads: int,
    head_dim: int,
    sequence_tokens: int,
    bytes_per_element: int,
) -> float:
    if any(type(x) is not int or x <= 0 for x in
           (layers, kv_heads, head_dim, bytes_per_element)):
        raise ValueError("architecture dimensions and element size must be positive integers")
    if type(sequence_tokens) is not int or sequence_tokens < 0:
        raise ValueError("sequence_tokens must be a nonnegative integer")
    bytes_used = (
        2 * layers * kv_heads * head_dim * sequence_tokens * bytes_per_element
    )
    return bytes_used / (1024**3)

WEIGHTS_GIB = (70 * 10**9 * 2) / (1024**3)

for label, heads, dtype_bytes, n_seq in [
    ("1x GQA BF16", 8, 2, 1),
    ("1x GQA FP8", 8, 1, 1),
    ("4x GQA BF16", 8, 2, 4),
    ("1x MHA BF16", 64, 2, 1),
]:
    cache = n_seq * kv_cache_gib(80, heads, 128, 131_072, dtype_bytes)
    print(
        f"{label}: {cache:.0f} GiB KV vs {WEIGHTS_GIB:.0f} GiB weights "
        f"({'KV wins' if cache > WEIGHTS_GIB else 'weights still larger'})"
    )

1x GQA BF16: 40 GiB KV vs 130 GiB weights (weights still larger)
1x GQA FP8: 20 GiB KV vs 130 GiB weights (weights still larger)
4x GQA BF16: 160 GiB KV vs 130 GiB weights (KV wins)
1x MHA BF16: 320 GiB KV vs 130 GiB weights (KV wins)

Sum mixed layers separately

Before using the 40 GiB result for another checkpoint, inspect its layers. For separate K/V arrays, sum

$\text{logical KV bytes}=\sum_\ell n_\ell\big(H_{K,\ell}d_{K,\ell}b_{K,\ell}+H_{V,\ell}d_{V,\ell}b_{V,\ell}\big).$

Here $n_\ell$ is the retained token count in layer $\ell$. With actual local-cache eviction, it is at most the layer's window; full layers grow with context. MLA uses a latent-state formula instead, and recurrent layers need their recurrent state added separately. A local attention mask alone doesn't prove the runtime has freed old cache storage.

Architecture check, October 10, 2026: the pinned Gemma 4 12B configuration has 40 local layers (8 KV heads, width 256, window 1,024) and eight full layers (1 KV head, width 512).^{[4]Reference 4Gemma 4 12B Configurationhttps://huggingface.co/google/gemma-4-12B/blob/023679ed352de9bb66cc873c9009ce3482585c08/config.json} Predict what happens if you mistakenly apply its local head shape to all 48 layers at 128K:

n, b = 131_072, 2  # assumed BF16 cache; not a runtime memory measurement
local = 2 * 40 * 8 * 256 * min(n, 1_024) * b
full = 2 * 8 * 1 * 512 * n * b
wrong_uniform = 2 * 48 * 8 * 256 * n * b
print(f"local layers: {local / 2**30:.4f} GiB")
print(f"full layers: {full / 2**30:.4f} GiB")
print(f"layer-specific sum: {(local + full) / 2**30:.4f} GiB")
print(f"wrong uniform estimate: {wrong_uniform / 2**30:.4f} GiB")

local layers: 0.3125 GiB
full layers: 2.0000 GiB
layer-specific sum: 2.3125 GiB
wrong uniform estimate: 48.0000 GiB

The sum assumes two separate cache arrays, BF16 payloads, no cross-layer sharing, and eviction to the local window. Gemma's shared global K/V projection isn't sufficient to halve it: the pinned implementation applies different key/value transformations and sends separate states to cache.update.^{[5]Reference 5Transformers Gemma 4 Unified Attention and Cache Updatehttps://github.com/huggingface/transformers/blob/2c4914fb939fe9de0d8e7a798af4684d552f18b4/src/transformers/models/gemma4_unified/modeling_gemma4_unified.py} Model-weight dtype also doesn't establish cache dtype. Inspect the actual backend's retained buffers and peak memory before admitting requests.

Prefill vs. decode: two different bottlenecks

The same 128K request can hurt twice, for different reasons.^{[2]Reference 2Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} During prefill, the model ingests the prompt and full attention pays for its long pairwise score matrix. During decode, the model emits tokens one at a time; the KV cache avoids recomputing old projections, but each new token still reads the cached prefix.

For a single dense-attention decode step, a useful logical KV read-volume estimate (ignoring writes of the new step and weight traffic) is:

where $L$ is layers, $H_{kv}$ is KV heads, $d_h$ is head dim, $n_{\text{ctx}}$ is cached tokens, and $b$ is bytes per element. It resembles the residency formula because each cached K/V element is needed. This isn't a guaranteed HBM-traffic lower bound: on-chip reuse can reduce traffic, while repeated loads and kernel layout can increase it. Measure the memory hierarchy your bottleneck actually uses.

Compare that with weight-stream bytes using an idealized decode batch that loads the dense model's weights once for the batch. One 128K sequence needs about 40 GiB of logical KV reads versus about 130 GiB of BF16 weights; four independent sequences raise the KV side to 160 GiB while reusing the same weights. This explains why batching and context length can shift a weight-bandwidth bottleneck toward KV bandwidth. It's not a measured crossover: kernels, reuse, sharding, and compute utilization determine the real one.

Keep the phase attached to each intervention. FlashAttention reduces attention-kernel IO, which matters most during large prefills and time to first token (TTFT). Chunked prefill lets the scheduler interleave a long ingestion with in-flight decodes.^{[6]Reference 6Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html} PagedAttention, GQA, and KV-cache quantization reduce allocation waste or bytes per cached token. Prefix reuse skips repeated prefill for matching prefixes. The right fix is whichever moves your measured bottleneck.

Attention variants that cut long-context cost

Instead of keeping every position visible, change the visibility pattern. Mistral 7B pairs GQA with sliding window attention (SWA): each token attends to a fixed local window rather than the entire prefix.^{[7]Reference 7Mistral 7B.https://arxiv.org/abs/2310.06825} If that window has width $w$, attention cost moves from $O(n^2)$ toward $O(n \times w)$. Gemma 2 alternates local layers with global-attention layers, using a 4096-token local window and an 8192-token global span in its report.^{[8]Reference 8Gemma 2: Improving Open Language Models at a Practical Sizehttps://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf} Local layers pay less; global layers provide occasional long-range paths.

Test the dependency before choosing that trade. A width-$w$ window limits direct attention in one layer, not necessarily the whole network's receptive field: information can propagate through stacked layers. Mistral's report describes this larger receptive field.^{[7]Reference 7Mistral 7B.https://arxiv.org/abs/2310.06825} Still, an evicted raw fact isn't directly addressable later, and an indirect path doesn't guarantee exact recall. Sliding-window attention is a cheaper visibility pattern, not full-prefix access at a discount.

Here's a second failure to predict when the window rolls forward. Once early tokens leave the cache, quality can collapse in some models. Xiao et al. linked this behavior in their evaluated models to attention sinks, initial tokens that attract disproportionate attention; their StreamingLLM experiments retained a few sink KV entries alongside the recent window for stable long generation without fine-tuning.^{[9]Reference 9Efficient Streaming Language Models with Attention Sinkshttps://arxiv.org/abs/2309.17453} Sink count and quality need model-specific tests. Retaining sinks can stabilize generation, but it doesn't restore lookup of every evicted fact.

When the input doesn't fit: truncation and compaction

Visibility is only useful after the prompt fits. In a multi-turn chat or agent loop, history grows on every turn, so the manager must decide which state survives the next request.

Truncation is the hard cut: protect the system prompt and latest user turn, then drop old turns until the token budget closes. It's cheap and predictable, but an omitted fact is gone. Count tokens rather than characters or message count because code and prose consume the budget at different rates.

Summarization replaces old turns with a model-written state note. Context compaction applies the same idea to an agent transcript: fold completed steps into a smaller state before continuing. It preserves more meaning per token, at the cost of another model call and the risk that the summarizer drops a detail that becomes important later.

Either route starts with the same token-budgeting rule: reserve system instructions and expected output, then fill remaining space from newest to oldest. Classify each older turn as disposable or as state that must be compacted.

Reserve non-negotiable space before touching history. If it doesn't fit, fail explicitly. Drop complete conversational units, not individual messages that leave an answer without its question or a tool result without its call. Compaction is lossy: preserve source IDs and unresolved state outside the prose summary, and recount the resulting request. Never promote summarized external text into system instructions.

This example handles plain user/assistant chat. Completed exchanges are atomic; tool-call transcripts require a richer grouping rule. count_request counts a whole serialized request, so a real adapter can include template overhead instead of adding separate content counts.

def fit_history(
    system: dict,
    exchanges: list[list[dict]],
    latest_user: dict,
    context_limit: int,
    reserved_output: int,
    count_request,
) -> list[dict]:
    """Preserve mandatory messages and a contiguous suffix of complete exchanges."""
    if (type(context_limit) is not int or context_limit <= 0
            or type(reserved_output) is not int or reserved_output < 0):
        raise ValueError("invalid context or output budget")
    if system["role"] != "system" or latest_user["role"] != "user":
        raise ValueError("expected a system message and latest user message")
    if any([m["role"] for m in turn] != ["user", "assistant"] for turn in exchanges):
        raise ValueError("this example accepts complete plain-chat exchanges only")
    budget = context_limit - reserved_output
    mandatory = [system, latest_user]
    if count_request(mandatory) > budget:
        raise ValueError("mandatory input and reserved output exceed context limit")
    kept = []
    for turn in reversed(exchanges):
        candidate = [system, *turn, *kept, latest_user]
        if count_request(candidate) > budget:
            break
        kept = [*turn, *kept]
    return [system, *kept, latest_user]

# Deterministic stand-in, NOT a real tokenizer: 1 framing unit per message.
def count_request(messages: list[dict]) -> int:
    return sum(1 + len(m["content"].split()) for m in messages)

system = {"role": "system", "content": "You are an incident assistant."}
exchanges = [[
    {"role": "user", "content": "incident one with a fairly long trace summary"},
    {"role": "assistant", "content": "triaged incident one"},
]]
latest = {"role": "user", "content": "what failed in canary"}
kept = fit_history(system, exchanges, latest, 16, 4, count_request)
print([m["role"] for m in kept])
print(f"input={count_request(kept)}, output reserved=4, limit=16")
try:
    fit_history(system, exchanges, latest, 14, 4, count_request)
except ValueError as error:
    print(error)

['system', 'user']
input=11, output reserved=4, limit=16
mandatory input and reserved output exceed context limit

In the example, the oldest turns fall away while the system prompt and newest user turn survive. That's the right result only if those old turns are disposable. If one contains the canary's root cause, compact it into a state note before it crosses the budget boundary.

Second gate: do stretched positions still mean what you think?

The positional encoding article explains why attention needs a position signal. Here the operational question is narrower: what happens when a model trained on one range receives positions far beyond it? Rotary Position Embeddings (RoPE)^{[10]Reference 10RoFormer: Enhanced Transformer with Rotary Position Embedding.https://arxiv.org/abs/2104.09864} rotates those position encodings into attention, and interpolation or rescaling tries to extend that learned coordinate system. Llama 3 also raised its RoPE base $\theta$ to 500,000, stretching the useful range before interpolation.^{[3]Reference 3The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

RoPE basics

Picture each pair of embedding dimensions as a dial. Dials rotate at different speeds, so a token's position is a combination of angles rather than one scalar. Comparing two rotated vectors exposes their relative offset, which gives attention distance-aware information without adding a separate lookup row for every position.

RoPE rotates pairs of query and key head dimensions, not the residual token embedding itself. Write each pair as a complex number $x_j$ with frequency $\omega_j$:

$\text{RoPE}(x_m, m)_j = e^{i m \omega_j} x_j, \qquad \omega_j = \theta^{-2j/d_{\text{rot}}}$

Reading the formula

Here $d_{\text{rot}}$ is the even number of rotated head dimensions and $j$ indexes pairs. Position $m$ rotates pair $j$ by $m\omega_j$ radians. In a query-key dot product, the two rotations yield a relative-position angle. Fast frequencies distinguish nearby positions; slow frequencies change more gradually. These are periodic signals, not unique position IDs.

Position interpolation and NTK-aware scaling

Suppose training stopped at 8,192 tokens and deployment targets 32,768. Naive extrapolation introduces relative offsets and joint frequency patterns outside the trained range, even though individual angles are periodic. Position interpolation instead compresses the position coordinate:

$m' = m \times \frac{L_{\text{train}}}{L_{\text{target}}}$

Uniform interpolation protects the range by compressing every frequency band equally. NTK-aware (Neural Tangent Kernel) scaling changes that compromise: it stretches low-frequency dimensions more while keeping high-frequency dimensions closer to their original behavior. Try it to protect local precision, not because every checkpoint tolerates the same factor.

def interpolate_position(position: int, trained_window: int, target_window: int) -> float:
    """Map an extended position into the original coordinate range."""
    if any(type(x) is not int for x in (position, trained_window, target_window)):
        raise ValueError("positions and windows must be integers")
    if trained_window <= 0 or target_window < trained_window or not 0 <= position < target_window:
        raise ValueError("require positive windows, extension, and an in-range position")
    return position * trained_window / target_window

trained_window = 8_192
target_window = 32_768
for position in [0, 8_192, 16_384, 32_767]:
    mapped = interpolate_position(position, trained_window, target_window)
    print(f"extended position {position:>5} -> trained coordinate {mapped:7.2f}")

extended position     0 -> trained coordinate    0.00
extended position  8192 -> trained coordinate 2048.00
extended position 16384 -> trained coordinate 4096.00
extended position 32767 -> trained coordinate 8191.75

The arithmetic explains the idea; serving code selects a checkpoint-compatible variant. Hugging Face Transformers 5.19.0 documentation checked October 10, 2026 uses rope_parameters: linear is uniform interpolation, dynamic rescales the RoPE base, yarn is YaRN, longrope follows LongRoPE, and llama3 is the Llama 3.1 frequency split.^{[11]Reference 11Utilities for Rotary Embeddinghttps://huggingface.co/docs/transformers/main/internal/rope_utils[12]Reference 12LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenshttps://arxiv.org/abs/2402.13753} These aren't interchangeable settings for every architecture. Some models configure RoPE separately for local and full layers. Older installed versions may use a different schema.

from transformers import LlamaConfig

config = LlamaConfig(
    max_position_embeddings=8_192,
    rope_parameters={
        "rope_type": "dynamic",
        "rope_theta": 10000.0,
        "factor": 4.0,
    },
)
print(f"RoPE type={config.rope_parameters['rope_type']} factor={config.rope_parameters['factor']}")
print(f"configured max positions={config.max_position_embeddings}")
assert config.max_position_embeddings == 8_192

RoPE type=dynamic factor=4.0
configured max positions=8192

This constructs and checks a configuration with Transformers 5.19.0 on CPU. No weights are loaded. The scale factor hasn't changed the configured maximum to 32,768. It doesn't train extended behavior or raise the serving engine's limits either. Use the checkpoint's published settings and evaluate any extension. A "yarn" setup carries fields such as original_max_position_embeddings and, optionally, attention_factor; Llama 3 checkpoints use a larger base than this illustrative 10,000.^{[11]Reference 11Utilities for Rotary Embeddinghttps://huggingface.co/docs/transformers/main/internal/rope_utils}

Dynamic scaling introduces a cache-consistency question too. If a frequency basis changes with length and the intended calculation uses that new basis for every position, previously rotated keys must agree with it. The YaRN paper calls out this implementation issue.^{[13]Reference 13YaRN: Efficient Context Window Extension of Large Language Models.https://arxiv.org/abs/2309.00071} Consider one unit-vector pair, with a key at position 1 and a query at position 2:

from math import cos, sin

def rotated(position, frequency):
    angle = position * frequency
    return (cos(angle), sin(angle))

def dot(a, b):
    return sum(x * y for x, y in zip(a, b))

query = rotated(2, 0.5)
old_key = rotated(1, 1.0)
consistent_key = rotated(1, 0.5)
print(f"new query / stale key: {dot(query, old_key):.6f}")
print(f"new query / consistent key: {dot(query, consistent_key):.6f}")
assert abs(dot(query, consistent_key) - cos(0.5)) < 1e-12

new query / stale key: 1.000000
new query / consistent key: 0.877583

This is a two-dimensional arithmetic example, not a defect test for Transformers. Implementations can freeze a basis, retain unrotated keys, or otherwise manage transitions. Test the checkpoint's intended cached behavior against an equivalent uncached path across length boundaries. Restart or invalidate incompatible cached state when changing position configuration.

YaRN (Yet another RoPE extensioN)

YaRN makes the selective compromise explicit. It combines NTK-by-parts interpolation, rather than simply the NTK-aware base change, with a temperature factor on attention logits.^{[13]Reference 13YaRN: Efficient Context Window Extension of Large Language Models.https://arxiv.org/abs/2309.00071} High-frequency dimensions receive no interpolation to protect local order, low-frequency dimensions receive full interpolation to gain range, and middle frequencies use a linear ramp. Attention scaling is distinct from the output sampler's temperature.

That treatment improved long-context perplexity over plain interpolation at aggressive extension ratios in the YaRN evaluation.^{[13]Reference 13YaRN: Efficient Context Window Extension of Large Language Models.https://arxiv.org/abs/2309.00071} It's evidence for testing the strategy, not a portability guarantee. Each new model family needs its own recall and loss checks.

Third gate: can the model reach the evidence?

Now return to the canary note. Even with a valid budget and an extended position range, retrieval can depend on where that note lands. Liu et al. found that accuracy varied with evidence position in their evaluated tasks and models, with stronger performance near the beginning or end than in the middle.^{[14]Reference 14Lost in the Middle: How Language Models Use Long Contextshttps://aclanthology.org/2024.tacl-1.9/}

That makes prompt layout part of system design. A model can ingest every trace and still miss the rollback approval if it sits in the weak middle. Treat the context as a map of reachable evidence, not as a bag whose tokens all receive equal attention.

What the curve looks like

Liu et al.'s evaluated tasks often show this shape; it isn't a law for every current model or prompt.^{[14]Reference 14Lost in the Middle: How Language Models Use Long Contextshttps://aclanthology.org/2024.tacl-1.9/} The later Chroma report found no notable position variation in one of its NIAH experiments even while longer inputs hurt performance.^{[15]Reference 15Context Rot: How Increasing Input Tokens Impacts LLM Performancehttps://www.trychroma.com/research/context-rot} Length sensitivity and middle sensitivity are separate observations.

Mitigation strategies

Strategic information placement

When a depth sweep exposes a middle miss, change layout before changing the model. Suppose five retrieved chunks describe the failed deployment: a rollback approval, a canary error, a deploy note, a noisy log, and a generic runbook clause. Put the two facts that answer the incident at opposite edges, then leave weaker support in the middle.

The function below turns that hypothesis into a candidate prompt. It doesn't prove the layout works; it gives you a controlled prompt to compare with the unchanged baseline:

from dataclasses import dataclass

@dataclass
class Document:
    text: str
    relevance: float

def arrange_context(
    system_prompt: str,
    retrieved_docs: list[Document],
    user_query: str,
) -> str:
    """Put the two strongest chunks at the edges; leave the rest in the middle."""
    ranked_docs = sorted(retrieved_docs, key=lambda d: d.relevance, reverse=True)
    head_docs = ranked_docs[:1]
    tail_docs = ranked_docs[1:2]
    middle_docs = ranked_docs[2:]

    context = [system_prompt]
    context.extend(d.text for d in head_docs)
    context.extend(d.text for d in middle_docs)
    context.extend(d.text for d in tail_docs)
    context.append(user_query)

    return "\n\n".join(context)

docs = [
    Document("Rollback approved on 2024-03-15 by incident lead #42.", 0.95),
    Document("Original canary error: auth callback returned 500.", 0.92),
    Document("Deploy owner requested a staged rollback.", 0.88),
    Document("Noisy log excerpt: cache warmed successfully.", 0.45),
    Document("Generic rollback runbook clause 7B.", 0.30),
]

prompt = arrange_context(
    system_prompt="You are an incident assistant. Answer using only the evidence below.",
    retrieved_docs=docs,
    user_query="Was the rollback approved?",
)
print(prompt)

You are an incident assistant. Answer using only the evidence below.

Rollback approved on 2024-03-15 by incident lead #42.

Deploy owner requested a staged rollback.

Noisy log excerpt: cache warmed successfully.

Generic rollback runbook clause 7B.

Original canary error: auth callback returned 500.

Was the rollback approved?

The output puts rollback approval at the head and the canary error at the tail, with weaker support between them. This string illustrates ordering only: a real chat adapter must preserve roles, delimit and label sources, and recount the final request. Relevance scores aren't probabilities. Compare this candidate with the unchanged baseline on the same depth sweep before adopting it.

Repeated key information

If a fact is essential and cheap to restate, put a source-labeled excerpt near the evidence head and near the query. Keep retrieved facts as evidence, not system-level instructions. Duplication consumes tokens and can amplify stale or conflicting facts, so compare with the same unique evidence and evaluate whether the extra copy helps.

Chunked processing

If one prompt still has too much distractor text, process the document in chunks and aggregate the partial results. Chunking changes the question from “did one pass find every fact?” to “did the aggregation preserve the facts needed for the final answer?”

The packing policy places the two strongest facts at the sequence edges and lower-priority support in the middle. That's a hypothesis about reachability, not a rule to memorize. If position-sliced evaluation still misses evidence, change the packing order or retrieve a smaller set and rerun the same cases.

Fourth gate: should you read everything or retrieve first?

The serving path starts with the question, not the model's maximum window. For the rollback incident, ask whether the evidence fits with output headroom, whether it must be fresh or cited, whether the same corpus will be queried repeatedly, and whether the answer needs a joint scan or one targeted fact.

The product choice is often a large context window versus retrieval-augmented generation (RAG). Long context scans a selected packet together, which helps when relationships across that packet matter but makes prefill larger. RAG finds candidate sections first, which shrinks the generation prompt but adds retrieval latency and a new failure: the needed section may never be selected.

For one question over a short, stable policy, full context is a sensible baseline. For repeated questions, fresh data, or a targeted lookup across a large archive, retrieval is usually the first comparison to measure. The table keeps those costs visible before you commit to a route.

Repeated queries over the same large prefix deserve a separate look. Without a usable cache hit, resending it repeats prefill work; a retained prefix also occupies memory. A fresh full-context snapshot can be current, and a stale retrieval index can be wrong. Neither route provides citations automatically: preserve document IDs and versions, and verify the answer's support.

A concrete decision example

Suppose the archive holds 200,000 tokens of service traces and the model limit is 128K. The question is, “Which service emitted the most timeout errors in March?” A top-k retriever might omit records needed for a count, while the full archive can't fit. Use an exact database filter and aggregation over all qualifying records. The model can explain the resulting counts with source IDs; a bounded packet of likely matches isn't a complete dataset.

Predict this counterexample: the two highest-ranked rows are both from service A, but service B has three matching rows. Ranking scores measure relevance, not completeness.

from collections import Counter

records = [
    ("r1", "2026-03", "A", "timeout", 0.99),
    ("r2", "2026-03", "A", "timeout", 0.98),
    ("r3", "2026-03", "B", "timeout", 0.80),
    ("r4", "2026-03", "B", "timeout", 0.79),
    ("r5", "2026-03", "B", "timeout", 0.78),
    ("r6", "2026-02", "A", "timeout", 0.97),
    ("r7", "2026-03", "A", "success", 0.96),
]
march = [r for r in records if r[1] == "2026-03" and r[3] == "timeout"]
top_two = sorted(march, key=lambda r: r[4], reverse=True)[:2]
partial = Counter(r[2] for r in top_two)
complete = Counter(r[2] for r in march)
print(f"top-2 counts: {dict(partial)}")
print(f"complete counts: {dict(complete)}")
assert partial.most_common(1)[0][0] != complete.most_common(1)[0][0]

top-2 counts: {'A': 2}
complete counts: {'A': 2, 'B': 3}

Top-2 selects the wrong winner. Filtering every qualifying record and aggregating in code fixes this miniature case. At scale, define event identity, deduplication, timezone boundaries, and a data watermark before running the equivalent database query. For semantic synthesis rather than exact counts, retrieve-then-pack can still be useful, but assess coverage of the required evidence separately.

Fifth gate: can the chosen path run on real GPUs?

Once the evidence path is chosen, capacity becomes the release constraint. Long-context serving often runs out of room in the KV cache before it runs out of model weights, especially when several long requests decode together.

Grouped query attention (GQA)

GQA (Grouped-Query Attention)^{[16]Reference 16GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.https://arxiv.org/abs/2305.13245} lowers KV-cache bytes by sharing key/value heads across query groups. MHA gives every query head its own K/V heads, GQA shares K/V heads across groups, and MQA shares one K/V set across all query heads. In an otherwise comparable architecture, moving from 64 query/KV heads to 8 KV heads cuts the cache term by 8x; moving to one cuts it by 64x. Those are memory ratios, not quality or throughput guarantees. Use a trained compatible architecture or an evaluated conversion/uptraining procedure; changing a head-count field doesn't preserve pretrained behavior.

See our MQA/GQA deep-dive for the full architecture details.

Head sharing isn't the only architectural KV cut. Multi-head latent attention (MLA) caches a compressed latent plus positional-key state rather than full K/V heads. DeepSeek-V2 reports a 93.3% KV-memory reduction versus DeepSeek 67B.^{[17]Reference 17DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434} That's a comparison of those architectures, not a generic compression ratio. MLA is a trained architecture choice, not a serving flag for a GQA checkpoint.

def relative_kv_bytes(query_heads: int, kv_heads: int) -> float:
    if (type(query_heads) is not int or type(kv_heads) is not int
            or query_heads <= 0 or kv_heads <= 0 or query_heads % kv_heads):
        raise ValueError("positive KV heads must divide query heads")
    return kv_heads / query_heads

query_heads = 64
for label, kv_heads in [("MHA", 64), ("GQA", 8), ("MQA", 1)]:
    fraction = relative_kv_bytes(query_heads, kv_heads)
    print(f"{label}: {fraction:.3f}x MHA KV bytes ({1 / fraction:.0f}x smaller)")

MHA: 1.000x MHA KV bytes (1x smaller)
GQA: 0.125x MHA KV bytes (8x smaller)
MQA: 0.016x MHA KV bytes (64x smaller)

Quantized KV-cache

If the cache still crowds out concurrency, reduce bytes per element. BF16 or FP16 stores each cached element in 2 bytes; an FP8 KV-cache candidate uses 1 byte, so the 40 GiB fixture falls to about 20 GiB. That buys capacity, not free quality. Calibrated scales and long-depth tests decide whether the trade survives your workload.

The snippet follows vLLM documentation checked October 10, 2026 for an uncalibrated FP8 cache, where scales default to 1.0. The docs recommend dataset calibration through llm-compressor and expose kv_cache_dtype_skip_layers for selected layers.^{[18]Reference 18Quantized KV Cachehttps://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache/} A "sliding_window" exclusion only matters when matching local layers exist; skipping nonexistent layers doesn't protect a full-attention checkpoint. This GPU-dependent configuration wasn't executed here. Check your installed runtime, checkpoint, accelerator, and attention backend. With FlashAttention 3's FP8 path, the documented attention computation quantizes queries too, so this isn't always a storage-only change. Mixed cache dtypes won't achieve the uniform 2x raw-byte reduction above.

from vllm import LLM

llm = LLM(
    model="your-org/your-model",
    kv_cache_dtype="fp8",
    kv_cache_dtype_skip_layers=["sliding_window"],
)

def admitted_sequences(memory_budget_bytes: int, kv_per_request_bytes: int) -> int:
    if (type(memory_budget_bytes) is not int or memory_budget_bytes < 0
            or type(kv_per_request_bytes) is not int or kv_per_request_bytes <= 0):
        raise ValueError("require nonnegative integer budget and positive integer cache bytes")
    return memory_budget_bytes // kv_per_request_bytes

cache_budget = 64 * 2**30  # hypothetical budget after weights/runtime reservation
for dtype, kv_gib in [("BF16", 40), ("FP8 candidate", 20)]:
    slots = admitted_sequences(cache_budget, kv_gib * 2**30)
    print(f"{dtype}: at most {slots} full-length request(s) in cache budget")
assert admitted_sequences(0, 1) == 0
assert admitted_sequences(10, 1) == 10
for args in ((True, 1), (10, 0), (-1, 1), (10, 0.1)):
    try:
        admitted_sequences(*args)
    except ValueError:
        pass
    else:
        raise AssertionError("invalid byte counts accepted")

BF16: at most 1 full-length request(s) in cache budget
FP8 candidate: at most 3 full-length request(s) in cache budget

Use exact integer byte counts for this division. Binary floating-point floor division can surprise you: 1.0 // 0.1 is 9, even though the decimal quantities suggest ten slots. The worksheet gives an upper bound on raw full-length cache payloads, not a safe scheduler admission policy. Include block rounding, scale metadata, future output growth, temporary buffers, and safety headroom before reserving real capacity.

PagedAttention (vLLM)

Bytes per token are only half of capacity. A contiguous reservation can strand free memory as requests grow and finish. PagedAttention manages KV in non-contiguous blocks or “pages,” like virtual memory, so fixed-size blocks are allocated on demand (see our KV cache and PagedAttention deep-dive). The vLLM paper describes near-zero waste and sharing for common prefixes in its evaluated workloads.^{[2]Reference 2Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}

That layout reduces reservation and fragmentation waste, but it does not change the intrinsic KV bytes for a model and sequence length. It can make more of the GPU usable; it can't make one 128K sequence weigh less. The paper reported higher throughput in its evaluated serving workloads.^{[2]Reference 2Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}

Prefix reuse and prompt caching

The incident archive may also have a 48K prefix shared by every question: instructions, schemas, and a stable runbook. Prefix sharing reuses materialized prompt blocks for common prefixes instead of recomputing them from scratch.^{[2]Reference 2Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180[19]Reference 19Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/} In current vLLM, the switch is enable_prefix_caching=True.^{[19]Reference 19Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

Prefix reuse cuts repeated prefill; it doesn't raise the context limit or remove cached tokens from attention during decode. vLLM reuses full blocks with matching prefix tokens and cache identity, including relevant adapter/multimodal inputs and isolation salts.^{[19]Reference 19Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/} A changed early token invalidates downstream matches; eviction or a cold worker can remove reuse. Persist the conversation separately: a KV cache is reusable execution state, not durable semantic memory.

The example assumes three sequential requests, one exact shared prefix, and no eviction or partial-block loss. It counts newly prefilling tokens, not billed tokens, FLOPs, or a latency speedup. Suffix tokens still attend to the cached prefix.

def prefill_tokens_without_reuse(shared_prefix: int, unique_suffixes: list[int]) -> int:
    if any(type(x) is not int or x < 0 for x in [shared_prefix, *unique_suffixes]):
        raise ValueError("token counts must be nonnegative integers")
    return sum(shared_prefix + suffix for suffix in unique_suffixes)

def prefill_tokens_with_reuse(shared_prefix: int, unique_suffixes: list[int]) -> int:
    prefill_tokens_without_reuse(shared_prefix, unique_suffixes)  # validate counts
    return shared_prefix + sum(unique_suffixes) if unique_suffixes else 0

shared_prefix = 48_000
questions = [800, 1_200, 600]
uncached = prefill_tokens_without_reuse(shared_prefix, questions)
reused = prefill_tokens_with_reuse(shared_prefix, questions)
print(f"uncached input tokens processed: {uncached:,}")
print(f"with reusable prefix candidate: {reused:,}")
print(f"avoided repeated prefix tokens: {uncached - reused:,}")

uncached input tokens processed: 146,600
with reusable prefix candidate: 50,600
avoided repeated prefix tokens: 96,000

Ring attention across multiple GPUs

PagedAttention improves allocation on each device. It doesn't by itself distribute one sequence's attention work across devices. Ring Attention partitions blockwise attention and overlaps KV-block communication with computation; the paper evaluates long-context distributed Transformer training.^{[20]Reference 20Ring Attention with Blockwise Transformers for Near-Infinite Context.https://arxiv.org/abs/2310.01889} This motivates context parallelism, not a drop-in vLLM decode flag. Serving requires a compatible implementation and separate prefill/decode measurements; communication overhead remains.

Serving snapshot, October 10, 2026: vLLM separately documents decode context parallelism for supported GQA/MLA paths. It shards token history within the existing tensor-parallel ranks to reduce KV duplication and does not itself add GPUs. Its prefill guide describes full-KV gathering versus partial-KV ring exchange as approaches under active development.^{[21]Reference 21vLLM Context Parallel Deploymenthttps://github.com/vllm-project/vllm/blob/main/docs/serving/context_parallel_deployment.md} Check the installed method/backend and group constraints; don't infer production readiness from the research paper or multiply every speedup by GPU count.

The same guide distinguishes physical block size from the initialized full-attention grid's effective_attention_block_size. With physical size 16 and DCP=4, one logical block spans 64 tokens. A 1,025-token history therefore needs 17 of those logical blocks, covering 1,088 positions. Rounding by 16 instead would predict 1,040 positions, using the wrong grid. Per-device bytes still depend on head sharding, dtype, and layout. The reported effective size can be None, and gRPC omits it when replicas disagree; don't substitute a guessed size. Partial cache events carry their own actual size.^{[21]Reference 21vLLM Context Parallel Deploymenthttps://github.com/vllm-project/vllm/blob/main/docs/serving/context_parallel_deployment.md}

Final gate: does it work at the lengths and depths you will serve?

Capacity numbers and a successful short prompt aren't a release test. NIAH (Needle-in-a-Haystack) hides a known fact, the “needle,” at controlled positions or “depths” inside filler, the “haystack,” then asks the model to retrieve it.^{[22]Reference 22Needle In A Haystack: Pressure Testing LLMshttps://github.com/gkamradt/needle-in-a-haystack} For the incident, the needle can be the rollback approval while irrelevant traces fill the rest.

Sweep context lengths from 4K to 128K and depths from 0% to 100% to build a heatmap. A model that retrieves every tested needle stays uniform; a position-sensitive model develops weak middle cells as length grows. The visual below is an illustrative failure surface, not a score for any named model.

NIAH tests one fact. RULER^{[23]Reference 23RULER: What's the Real Context Size of Your Long-Context Language Models?https://arxiv.org/abs/2404.06654} broadens the probe to multiple needles, multi-hop tracing across distant records, and aggregation before answering. Those tasks expose whether the model can combine evidence rather than merely spot a highlighted sentence.

Pair retrieval scores with perplexity or next-token loss by sequence length. A healthy extension should not show a sudden loss spike immediately beyond the original training window. If RoPE or cache changes cause a sharp jump, investigate configuration or distribution shift before blaming a harder task.

A 2025 Chroma report tested 18 models and described reliability degradation with longer inputs, distractors, and less explicit query-answer links as context rot.^{[15]Reference 15Context Rot: How Increasing Input Tokens Impacts LLM Performancehttps://www.trychroma.com/research/context-rot} Treat that report as a prompt to test your model and workload, not as a universal accuracy curve. A larger window permits more input; it doesn't prove every added token helps.

A real runner must control depth without spending the output budget. The next function uses integer token ids so you can inspect the insertion index, actual depth, and generation headroom before wiring in a model.

import math

def place_needle(
    haystack: list[int],
    needle: list[int],
    instruction: list[int],
    context_limit: int,
    requested_depth: float,
    max_new_tokens: int,
) -> tuple[list[int], float]:
    """Insert needle ids at a requested depth and keep input-plus-output inside the limit."""
    if (type(context_limit) is not int or context_limit <= 0
            or type(max_new_tokens) is not int or max_new_tokens < 0):
        raise ValueError("invalid context or generation budget")
    if (type(requested_depth) not in (int, float) or not 0 <= requested_depth <= 1
            or not math.isfinite(requested_depth)):
        raise ValueError("depth must be finite and between 0 and 1")
    if not needle:
        raise ValueError("needle must be nonempty")
    document_budget = context_limit - max_new_tokens - len(instruction)
    if document_budget < len(needle):
        raise ValueError("context limit is too small for instruction, needle, and output")

    filler_budget = document_budget - len(needle)
    clipped = haystack[:filler_budget]
    insert_idx = int(len(clipped) * requested_depth)
    document = clipped[:insert_idx] + needle + clipped[insert_idx:]
    packed = instruction + document
    if len(packed) + max_new_tokens > context_limit:
        raise AssertionError("packed prompt plus generation exceeds the context limit")
    actual_depth = insert_idx / max(len(clipped), 1)
    return packed, actual_depth

packed, depth = place_needle(
    haystack=list(range(20)),
    needle=[99, 99],
    instruction=[1, 2],
    context_limit=24,
    requested_depth=0.5,
    max_new_tokens=4,
)
print(packed)
print(f"tokens={len(packed)} actual_depth={depth:.2f}")
for invalid in (True, -0.1, 1.1, float('nan'), float('inf'), 10**1000):
    try:
        place_needle([0] * 20, [99], [1], 24, invalid, 4)
    except ValueError:
        pass
    else:
        raise AssertionError("invalid depth accepted")
short, short_depth = place_needle([], [99], [1], 8, 1.0, 2)
assert short == [1, 99] and short_depth == 0.0

[1, 2, 0, 1, 2, 3, 4, 5, 6, 7, 99, 99, 8, 9, 10, 11, 12, 13, 14, 15]
tokens=20 actual_depth=0.50

The packed sequence keeps 2 instruction tokens, inserts the needle at 50% of the clipped haystack, and leaves 4 generation tokens under the 24-token limit. Depth here means insertion index divided by filler length; it excludes instructions and the needle itself. Empty filler has depth zero. Short haystacks don't fill the window, so record the actual packed length. This token-ID exercise assumes instruction already includes all fixed request framing; a model runner must use the real serialized prompt, including the final question.

Now test the harness against a deliberately limited reader that only scans the first and last four document tokens. The middle failure below comes from executing that reader, not from manually assigning pass/fail flags. It checks the test mechanism, not an LLM's quality. Run this after place_needle.py.

def contains(sequence: list[int], target: list[int]) -> bool:
    return any(sequence[i:i + len(target)] == target
               for i in range(len(sequence) - len(target) + 1))

needle = [99, 98]
for requested_depth in (0.0, 0.5, 1.0):
    packed, actual_depth = place_needle([0] * 30, needle, [1, 2], 24,
                                       requested_depth, 4)
    document = packed[2:]
    full_scan = contains(document, needle)
    edge_scan = contains(document[:4], needle) or contains(document[-4:], needle)
    assert full_scan
    print(f"depth={actual_depth:.1f}: full_scan={full_scan}, edge_scan={edge_scan}")

depth=0.0: full_scan=True, edge_scan=True
depth=0.5: full_scan=True, edge_scan=False
depth=1.0: full_scan=True, edge_scan=True

For a model evaluation, save per-case prompt length, actual depth, source/needle ID, expected answer, generated answer, and scorer result. Include absent-needle controls, multiple distractor seeds, and multi-hop tasks. Compare the same cases before and after the change; report per-slice sample counts and uncertainty rather than approving a release from three booleans. Measure TTFT, decode latency, peak KV residency, and concurrency on the actual serving stack. The CPU exercises here validate arithmetic and harness control flow; they don't measure recall, GPU memory, or speed.

Don't collapse the heatmap to one average before looking at its cells. These hypothetical runs each have 100 cases at each of three depths and the same 90% overall score:

for name, correct in [('baseline', (90, 90, 90)), ('candidate', (100, 70, 100))]:
    rates = [count / 100 for count in correct]
    overall = sum(correct) / 300
    gate = min(rates) >= 0.85  # chosen fixture requirement, not a universal threshold
    print(f"{name}: overall={overall:.0%}, worst depth={min(rates):.0%}, gate={gate}")

baseline: overall=90%, worst depth=90%, gate=True
candidate: overall=90%, worst depth=70%, gate=False

The candidate improves both edges while breaking the chosen middle-depth requirement. This check evaluates synthetic counts, not a model or statistical confidence. In a real release, choose requirements before measuring, retain paired case outcomes, audit scorer errors, report uncertainty, and test enough cases in the slices that matter to users. Current versions of the original NIAH repository include multi-needle and UUID-chain tasks too; the classic one-fact experiment doesn't describe every test in that project.^{[22]Reference 22Needle In A Haystack: Pressure Testing LLMshttps://github.com/gkamradt/needle-in-a-haystack}

Evaluation rubric

• Compute the serialized input/output budget and reject a mandatory-input overflow without silently dropping the latest request.
• Derive raw KV bytes with KV heads, dtype, sequence length, and concurrency; distinguish logical state from per-device residency and measured traffic.
• Show why top-k can reverse an exact count, and choose complete aggregation or coverage-tested retrieval according to the question.
• Design a length/depth evaluation with actual outputs, negative controls, and serving measurements; don't infer effective recall from acceptance alone.

Follow-up questions

What to carry forward

Return to the rollback approval. Did the request retain the correct document version? Did the packer reserve a usable answer budget? Can this checkpoint use the note at its actual position? Does the serving path meet the latency requirement while other users are decoding? A maximum-token acceptance test answers only part of the first question.

Keep provenance outside lossy summaries, calculate cache state layer by layer, and evaluate the selected evidence path on paired cases. Position scaling, packing, retrieval, cache precision, and distributed execution change different parts of the request. Measure the failure each change is meant to fix, including regressions hidden by an aggregate score.