Prefix Caching and Prompt Caching

Three developers ask a coding assistant different questions about the same repository. Each request carries 6,000 shared tokens: 600 of instructions, a 4,200-token architecture guide, an 800-token tool schema, and 400 tokens of examples. The first question adds another 48 tokens. Should the model process that entire guide three times?

An available prefix cache can reuse the guide's previously computed state. Put a changing request ID before the guide, though, and the guide no longer has the same preceding context. The unchanged text is insufficient. Reuse also needs compatible model settings, an eligible matching boundary, and the right sharing scope.

For ordinary full causal attention, prefix caching retains Key-Value (KV) state for an exact starting token sequence. The state can live in a GPU pool or another cache tier. Our idealized warm request reuses 6,000 tokens and freshly processes its 48-token question during prefill. Standard KV caching retains state inside a request during decoding; prefix caching makes compatible state reusable across requests. A cache hit saves input work, not a previously generated answer.

The previous KV-cache lesson explained how a request retains attention state while generating. Here, we tackle how independent requests safely borrow and share that state across queries, tenants, and GPU clusters.

Worked example: tracking tokens across cache hits

Suppose our coding assistant's shared prefix breaks down into four parts:

Assume the first request has made its prefix available before the second arrives. Predict what request two can skip: it can avoid recomputing the shared guide, but it still processes its own question and generates its own response. A simultaneous cold burst can require multiple warmups; matching text doesn't mean a cache write has finished.

With the eligible entry available, the runtime reuses those 6,000 prefix tokens for requests two and three. New question tokens still attend to the cached guide; reuse does not remove the guide from the context.

Now place a changing request_id before the guide. Tokens before the changed ID may still match, including message wrappers. The guide after it cannot reuse state computed with a different ID. Moving the ID after the eligible guide boundary preserves more reusable work.

For questions of 48, 37, and 51 tokens, three cold requests freshly process 18,136 input tokens. One cold request followed by two full-prefix hits freshly processes 6,136, avoiding 12,000 repeated prefix-token computations. This authored ledger assumes compatible settings, a qualifying 6,000-token boundary, and an available entry. It doesn't predict FLOPs, billing, or milliseconds.

The prefix tokens saved count represents a work ledger, not a billing outcome. Provider write and read fees, cache retention lifetimes, and arrival rates determine whether the initial cache write pays for itself. Those savings also depend on how the inference engine indexes and stores those 6,000 tokens. What does a "match" actually hash under the hood?

Two indexes, one causal dependency

During generation, a decoder-only transformer stores keys and values for tokens it has already processed. When predicting token 501, it doesn't recompute attention keys and values for tokens 1 through 500. That's the normal KV cache.

Retaining that state for generation doesn't by itself make it discoverable by the next incoming request.

With ordinary full causal attention, position $t$ attends only to positions $0 \dots t$. Appending a question doesn't change the prefix's ideal mathematical computation. Changing earlier tokens can change deeper-layer state for a later tools section even if the tools text stays identical. Compatibility includes weights, adapters, positions, attention configuration, and any nontext conditioning. Different execution paths can still introduce floating-point variation.

A single whole-prompt hash would miss a partial overlap. A hash table indexed by successive prefix blocks can discover it. A compressed token trie offers another approach:

SGLang's paper describes RadixAttention retaining prompt and generation state through a radix tree and evicting eligible leaves with LRU.^{[1]Reference 1SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104} A trie split can expose a shared initial segment. Its current cache granularity depends on the backend; the original paper isn't a guarantee that every deployment caches arbitrary single-token boundaries.

vLLM uses a hash-based index, not that radix-tree implementation.^{[2]Reference 2Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/} With sixteen-token blocks, 6,005 shared positions contain 375 complete blocks (6,000 tokens). Eligibility and the logits boundary can leave additional work, as the previous lesson demonstrated. PagedAttention supplies block-oriented attention storage; it isn't itself the prefix-matching algorithm.^{[3]Reference 3Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}

The vLLM key includes its parent hash, block token IDs, and relevant extras such as adapter IDs, multimodal hashes, and a cache salt. A teaching notation is:

$H_k = \text{hash}(\text{serialize}(H_{k-1},\ \text{token\_ids}_k,\ \text{extras}_k))$

The engine's cache namespace must also be compatible with its model revision; this formula doesn't assert that every vLLM block embeds a model ID. vLLM documents sha256 as the default and sha256_cbor for reproducible cross-language serialization. It also offers noncryptographic options with a collision-risk warning. Collision resistance is different from authenticating who selected a sharing scope.^{[2]Reference 2Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

Before looking at the diagram, predict the downstream key: if the repository guide changes in block 1 while the tools schema in block 2 stays identical, can block 2 reuse its old cache key?

The strings above stand in for token blocks. This fixture tests history dependence, not vLLM's serialization or cache availability: a matching key still needs a stored usable entry.

Prompt architecture: the static-to-dynamic contract

Cache hits require strictly stable prefixes. Put shared, immutable content first and variable, dynamic content last. Before comparing these two layouts, predict which pair shares more initial tokens.

Good prompt shape:

Poor prompt shape:

In the second layout, changing the question changes the context preceding the guide. A common wrapper might still match, but the guide after the divergence can't reuse its old state. These layouts illustrate serialized content order, not instructions to violate an API's role rules.

That's why agent architectures keep dynamic tool outputs and user-specific state after stable instructions whenever possible. Anthropic also renders tools, then system, then messages internally, so reshuffling a tool schema invalidates a cache even when the visible user text didn't move at all.^{[4]Reference 4Prompt caching.https://platform.claude.com/docs/en/build-with-claude/prompt-caching}

These lists contain named sections for clarity; in production, compare actual tokenizer token IDs when diagnosing cache behavior. Always preserve proper message roles; these sketches aren't instructions to stuff user queries into a system message.

Exact-match invalidation traps in production

Semantic equivalence isn't enough. Prefix reuse depends on the exact resulting token sequence, so formatting and serialization differences break the prefix earlier than you might expect.

• Whitespace and line endings: Extra spaces or \r\n instead of \n can change tokenization. The result depends on preprocessing and the tokenizer.
• JSON key order: Reordering keys in JSON rendered as prompt text can alter its token sequence. A provider may separately canonicalize structured API fields.
• Non-deterministic renderers: Timestamps, generated IDs, iteration over sets, or unstable external tool ordering can change the rendered prefix. Python dictionaries themselves preserve insertion order.

Make prompt templates strictly deterministic at the serialization boundary. Serialize JSON with sorted keys where key order doesn't carry meaning, and preserve ordered arrays. Don't blindly strip whitespace from code or structured markdown, since whitespace often defines syntax. Log a prefix version hash so a cache miss can be traced directly to rendered prompt tokens rather than guessed from template files.

Exact stability still doesn't guarantee memory residency. Self-hosted engines silently evict unreferenced blocks when VRAM is tight, using least recently used (LRU) policies.^{[1]Reference 1SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104} A later request takes the correct cold path and recomputes the prefix. Treat eviction as a latency and cost event, not a correctness failure: capacity-plan for misses and never let application correctness depend on a cache hit.

Hosted provider contracts and pricing economics

Hosted providers don't expose the same raw block-hash controls as vLLM. They still match exact prefixes, but the knobs, field names, and break-even economics differ by vendor and model family. Before reading either contract, predict what happens when a stable guide is followed by a changing question: which provider setting decides where reusable work ends?

OpenAI documents prompt caching in usage fields.^{[5]Reference 5Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching} Anthropic documents cache_control at the top level for automatic caching and on content blocks for explicit breakpoints.^{[4]Reference 4Prompt caching.https://platform.claude.com/docs/en/build-with-claude/prompt-caching} Both APIs expose the same underlying mechanics through different controls.

Provider contracts checked September 22, 2026. Model, platform, settings, and cache availability matter. The examples below are request structures and authored usage fixtures; they aren't live API results.

Don't assume a hosted cache behaves like your local runtime. Verify:

OpenAI's guide distinguishes GPT-5.6 and later from earlier models. The newer family has a 1,024-visible-token minimum, exact boundary reporting, and prompt_cache_options.ttl: "30m" (minimum lifetime after the latest write or reuse). Earlier-model minimums vary with request settings; reported cached counts round down to multiples of 128, and retention uses prompt_cache_retention.^{[5]Reference 5Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching}

The current guide's matching section says implicit lookup can also check earlier eligible message endings and the initial developer-message boundary. A later gotcha still describes a latest-message-only miss. Don't turn that inconsistency into a claim that every changing question necessarily misses. When you want only a particular stable prefix written, use an explicit marker and explicit-only mode; confirm actual reuse from usage and diagnostics.^{[5]Reference 5Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching[6]Reference 6Prompt cache diagnosticshttps://developers.openai.com/api/docs/guides/prompt-caching/diagnostics}

For our coding assistant, mark the end of the repository guide explicitly and place the user question after it. This Responses API request structure illustrates the controls. Replace its placeholder with the complete guide; the placeholder alone won't meet the minimum:

Explicit-only mode leaves the suffix outside the selected write. For GPT-5.6 and later, prompt_cache_key is optional separate accounting, not a routing requirement. Diagnostics even distinguish a key-related reported miss from a physical cache miss. Earlier models use keys to help routing. Choose any customer key in the trusted gateway; a caller-supplied label isn't authenticated isolation.^{[6]Reference 6Prompt cache diagnosticshttps://developers.openai.com/api/docs/guides/prompt-caching/diagnostics}

Current OpenAI pricing lists 1.25-times write and 0.10-times read rates for GPT-6 Astra and the GPT-5.6 family, relative to their corresponding uncached input rates.^{[7]Reference 7OpenAI API Pricinghttps://developers.openai.com/api/docs/pricing} Anthropic's pricing lists 1.25-times five-minute writes and 2-times one-hour writes. Most Claude reads cost 0.10 times base input; Fable 5.1 and Mythos 5.1 use 0.025 times. Availability and other pricing dimensions remain model-specific.^{[8]Reference 8Anthropic Model Pricinghttps://platform.claude.com/docs/en/about-claude/pricing}

At a 0.10-times read rate, the 5-minute write surcharge is 0.25 units and each subsequent full read saves 0.90 units. The break-even requirement is:

$\text{Break-Even Reads} = \left\lceil \frac{\text{Write Rate} - 1.0}{1.0 - \text{Read Rate}} \right\rceil$

For the 5-minute tier, $\lceil 0.25 / 0.90 \rceil = 1$ read repays that premium. For the 1-hour tier with a 1.0-unit surcharge, $\lceil 1.0 / 0.90 \rceil = 2$ reads are required. If a cached prefix is written and never read before it expires, you paid a 25% to 100% financial penalty over uncached input!

Claude minimums currently range from 512 to 4,096 tokens: Opus 5 uses 512, Sonnet 5 uses 1,024, and Haiku 4.5 uses 4,096. The guide directs Bedrock users to AWS's contract. A breakpoint has a twenty-block lookback for earlier entries, not a twenty-block prefix-length limit. Requests have four breakpoint slots; top-level automatic cache_control uses one. Concurrent reuse becomes available after the first response begins.^{[4]Reference 4Prompt caching.https://platform.claude.com/docs/en/build-with-claude/prompt-caching}

For the same stable-guide pattern, Claude's Messages API places the explicit marker on a system content block. Replace the placeholder with your complete guide; the literal placeholder is below the model's minimum:

Both examples select a reusable boundary. Neither marker creates a hit unless a compatible entry is available. For supported newer OpenAI models, prompt_cache_options.prewarm: true can write the selected prefix without generating a response; cache-write charges still apply.^{[5]Reference 5Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching}

Now compute the exact reuse threshold that repays the write premium across cache lifetimes:

This fixture uses Responses API field names; Chat Completions uses a different usage schema. For an authored Claude usage breakdown of the same 2,600-token total, use 2,000 cache_read_input_tokens, 400 cache_creation_input_tokens, and 200 input_tokens. Claude's input_tokens excludes the read/write portions. Sum all three before calculating its reused-token percentage.^{[4]Reference 4Prompt caching.https://platform.claude.com/docs/en/build-with-claude/prompt-caching}

Distributed serving: cache-aware routing versus round-robin

With independent worker-local caches, round-robin routing can require one cold warmup per worker. After those workers warm, repeated requests can hit. A deployment with shared tiers or cache transfers has different locality and transfer costs; not every cache entry is confined permanently to its original GPU.

Cache-aware routing can prefer a compatible warm worker. SGLang's paper describes cache-aware scheduling inside its engine; that isn't itself a distributed router.^{[1]Reference 1SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104} Routing affinity also doesn't enforce cache ownership. Apply the authenticated sharing scope at lookup independently of the destination.

Affinity needs health and load handling. A hot prefix can overload its preferred worker; spilling to another worker may be worth a cold warmup. Bounded-load consistent hashing is one policy, not a universal router implementation or a fixed requests-per-second threshold.

The fixture uses a fixed set of three initially cold workers, sequential completed warmups, one prefix, and no eviction. It hashes modulo the worker count, so changing that count can remap many keys. This is static affinity, not consistent hashing or a health-aware router.

Prefill versus decode: what prefix caching doesn't change

In our full-attention example, new input and generated tokens still attend to the retained prefix. GPU-resident entries still occupy the pool; another cache tier can add transfer costs. Reuse doesn't extend the model's context limit. Long output generation can dominate total response time even when input processing improves.

Separate time to first token (TTFT) from inter-token latency (ITL). A hit can reduce input processing; client TTFT also includes queueing and transport. It doesn't directly skip output-token generation. Decode performance can still change indirectly with batching, kernels, or load.

Prefix caching doesn't match by meaning. These questions express similar intent but don't share the same complete question prefix; any common message wrapper is a separate overlap:

Reusing an answer for such questions belongs to semantic caching. Semantic caching needs an answer-validity policy. Correct prefix reuse preserves the supplied context; it doesn't make the model's answer accurate or eliminate numerical differences between execution paths.

Here are authored phase times, with decode deliberately held constant. They are assumptions, not measurements of the running example or a prediction for your model:

Before looking at dashboards, predict this counterexample: if input work falls by 1,050 ms but queueing grows by 1,180 ms, does the client see the first token sooner?

The warm request is slower from the client's perspective. Compare both server work and client timestamps under representative load; a hit alone doesn't determine latency.

Telemetry: token-level yield versus binary hit counters

Usage fields report credited reuse for a request. Operations teams also need to know whether that reuse helps the route over time. Provider accounting and physical cache behavior need not be identical, as OpenAI's key-change diagnostic illustrates.^{[6]Reference 6Prompt cache diagnosticshttps://developers.openai.com/api/docs/guides/prompt-caching/diagnostics}

For self-hosted runtimes, log prefix cache hit rates, reused token counts, prefill latency, decode latency, and GPU memory allocation. For hosted APIs, parse and record returned cache fields. Always slice metrics by route, model revision, prefix version, and traffic window. Long-lived multi-turn chat sessions reuse substantial prefixes too; route names alone don't explain savings.

Synthetic application log schema:

Version the static prefix in telemetry. If the repository guide changes, blocks containing and following the first changed token can't reuse the old cache entry, but earlier unchanged blocks can still hit. A version label correlates that partial warmup with software rollouts.

Beware the binary hit counter trap. A request reusing 6,000 of 6,048 tokens has 99.2% token reuse; one reusing 600 has 9.9%. Both increment a binary hit counter. These percentages measure input-token volume, not prefill FLOPs or elapsed time. Compute a volume-weighted token ratio, then measure time and price separately:

$\text{Token Hit Ratio} = \frac{\sum \text{Cached Input Tokens}}{\sum \text{Total Prompt Tokens}}$

Memory management: cached can also mean available

Physical GPU VRAM is strictly bounded. What happens when hundreds of concurrent requests arrive and memory fills up?

vLLM V1's block pool distinguishes references, cached content, and allocation availability. On release, an unreferenced block can retain its cache key while joining the free queue. A matching request can touch it; allocation for different content can evict it. Resident pool storage need not shrink when ownership changes.^{[2]Reference 2Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

Don't recycle a block while owners or pending GPU operations still need it. Proper scheduler preemption can instead release ownership and arrange recomputation or supported offload. When available blocks run out, the engine must defer or preempt work according to its policy. The previous lesson's failed-append lab demonstrated that allocation failure must preserve existing state.

No universal 85%/95% VRAM watermarks follow from this design. Measure referenced blocks, cached unreferenced candidates, available blocks, eviction churn, and preemption. GPU allocation alone can stay constant while the logical pool fills.

In the authored snapshot below, how many blocks can be allocated: 150, 550, or 950?

The allocation order could replace more cached entries; this is a lower bound, not a simulation of the engine's free-queue order. A hit-rate drop can accompany successful allocation without any change in resident pool bytes.

Treat LRU eviction as a capacity signal. If your token hit ratio suddenly drops during peak traffic hours, check eviction churn: high concurrency may be evicting unpinned prefixes before subsequent requests arrive to reuse them.

Multi-tenant security: timing side-channels and cache isolation

Prefix caching introduces subtle security vulnerabilities in multi-tenant architectures if trust boundaries aren't strictly isolated.

Consider a probe containing suspected confidential text. A repeatedly faster response could reveal shared cache availability. One fast response doesn't prove which tenant submitted a document: queueing, routing, and other warm prefixes are confounders. Returned cache counters can also be a probing signal.

Incorrect history keys or collisions can cause reuse of the wrong state; that isn't automatically an out-of-bounds memory-corruption bug. Exact token matching and SHA-256 also don't authenticate the caller or authorize cross-tenant sharing.

vLLM supports an optional request cache_salt included in the first block's key. The gateway should select and enforce the scope from verified identity, overriding or rejecting unauthorized caller-selected values.^{[2]Reference 2Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/} A salt known to an attacker and accepted without authorization isn't an isolation boundary. Distinct structured keys and collision resistance protect this lookup path under the configured policy; they don't prove all side channels impossible.

Provider boundaries are separate contracts. OpenAI documents organization and regional processing isolation; its newer accounting keys should not be described as proof of physically separate KV storage. Claude documents workspace isolation on its API, Claude Platform on AWS, and Microsoft Foundry, and organization isolation on Bedrock and Google Cloud. A tenant label in application logs doesn't create an additional provider isolation boundary.^{[5]Reference 5Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching[4]Reference 4Prompt caching.https://platform.claude.com/docs/en/build-with-claude/prompt-caching}

This CPU fixture verifies distinct keys for supplied scopes. It doesn't authenticate those strings or test a provider's isolation:

Now try a forged label. The sessions below are trusted test fixtures standing in for an authentication result; they aren't a real login service.

Failure modes: debugging stale outputs and silent eviction

A common engineering trap is blaming exact-prefix caching when a model outputs outdated rules:

• Symptom: The coding bot continues enforcing last week's 90-day key rotation rule even after the repository guide changed to 30 days.
• Input check: Did the request actually contain the new guide? Inspect retrieval and rollout versions, plus a protected digest of the actual rendered stable guide. Comparing a full prompt digest to a guide-only digest is meaningless; changing user questions also changes the full digest.
• Cache check: Were preceding history, weights, adapters, and other conditioning included in the computation key? Correct exact keys prevent the edited guide from matching its old entry; custom keys can be wrong.
• Answer check: If input and cache behavior are correct, evaluate instruction following and answer quality. Fresh computation can still produce stale advice.

For supported OpenAI Responses models, set prompt_cache_options.comparison_response_id to a recent completed baseline response's ID and inspect prompt_cache_diagnostics. This requests a comparison; it doesn't load the old conversation or change caching behavior. Diagnostics are best effort. Use current usage counts for credited reuse.^{[6]Reference 6Prompt cache diagnosticshttps://developers.openai.com/api/docs/guides/prompt-caching/diagnostics}

Before promoting the guide, test both cold and warm requests against the updated rule. The cache should change the work performed, not which guide the model receives.