Prefix Caching and Prompt Caching

PagedAttention packs live KV blocks inside a serving engine. Prefix caching asks the next question: what if the same prompt prefix appears again in later requests?

The key-value (KV) cache inside one request saves work while the model generates the next token. Prefix caching saves work across requests that begin with the same tokens.

That distinction matters for long-context products. Suppose a coding assistant receives the same repository guidelines, architecture notes, tool schema, and safety instructions on every request. Without prefix caching, the model pays the prefill cost again and again. With compatible prefix caching enabled, the runtime can reuse computed KV state for the shared prefix, then process only the new user question.

Per-request KV cache

During generation, a decoder-only transformer stores keys and values for tokens it has already processed. When it predicts token 501, it doesn't recompute attention keys and values for tokens 1 through 500. That's the normal KV cache.

This cache belongs to the active sequence. It helps decode speed, but it doesn't automatically help the next request.

Prefix caching is different. It asks: if request B starts with the exact same token prefix as request A, can the runtime reuse the already-computed KV blocks for that prefix?

vLLM calls this Automatic Prefix Caching. Its current documentation enables the feature with enable_prefix_caching=True; it's a serving configuration choice, not a behavior to assume from PagedAttention alone.^[1] SGLang's RadixAttention uses a radix tree to reuse KV cache state across structured generation calls, with an LRU eviction policy and a cache-aware scheduler that groups requests with shared prefixes to raise the hit rate.^[2] For a self-hosted engine, verify that reuse is enabled, scoped correctly, and measurable before designing prompt layouts around hits.

Many engines cache full KV blocks, not arbitrary partial tokens. vLLM's design docs explain that each full block is hashed with its parent hash, the block's token IDs, and extra fields such as LoRA IDs or multimodal input hashes.^[1] That's why a tiny change early in the prompt can invalidate everything after it: the parent hash changes, so later blocks no longer match. vLLM also lets a request supply a cache_salt that's mixed into the first block hash, so only requests carrying the same salt can reuse those blocks.^[1] This is one mechanism for tenant-aware reuse boundaries.

Prompt shape

Cache hits require stable prefixes. Put shared content first and variable content last.

Good shape:

Poor shape:

In the poor shape, the first tokens change with every user question. The shared policy appears later, after the mismatch. Many caching systems match from the prefix, so the cache opportunity disappears.

That's one reason agent prompts should keep dynamic tool results and user-specific state after stable instructions when possible.

Hosted API prompt caching

Hosted providers expose prompt caching differently. OpenAI documents automatic prompt caching for repeated prompt prefixes and reports cached tokens in usage fields.^[3] Anthropic documents cache_control at the top level for automatically selected prefix caching and on content blocks for explicit breakpoints.^[4]

Don't assume provider caches behave like your local runtime. Check:

OpenAI's current docs say caching is available for prompts of 1,024 tokens or more and report cached_tokens inside usage.prompt_tokens_details, including zero for shorter prompts.^[3] OpenAI's caching is automatic, has no separate cache-write fee, and lets you pass an optional prompt_cache_key to improve routing affinity for shared prefixes. Supported models can also offer prompt_cache_retention choices such as in_memory or 24h, so check the selected model before depending on cache lifetime.^[3] Anthropic exposes cache_creation_input_tokens, cache_read_input_tokens, and input_tokens so you can separate cache writes, reads, and uncached suffix tokens.^[4]

The two pricing models differ, which changes the break-even math. OpenAI cache hits are billed at a reduced input rate with no separate write charge. Anthropic documents a write premium and then a read discount: a 5-minute cache write costs 1.25 times the base input price, a 1-hour write costs 2 times, and a cache read costs 0.1 times.^[4] Under those published ratios, a cached prefix repays the extra write cost after one read at the 5-minute tier or two reads at the 1-hour tier. Anthropic also uses model-specific minimum cacheable prefix lengths; its current active-model table spans 1,024 to 4,096 tokens.^[4] Below the selected model's minimum, cache fields can stay zero and you pay normal input cost. Model and pricing behavior change over time, so production code should inspect usage fields rather than assume a fixed discount.

Anthropic's explicit breakpoints add one more boundary to remember: each breakpoint checks at most 20 preceding content blocks for reusable content.^[4] If a prompt contains more than 20 content blocks before a breakpoint, add another cache_control breakpoint before that lookback window so older reusable content stays discoverable. Top-level automatic caching avoids manual breakpoint placement for many common conversation shapes.

Cache-aware routing

In a sharded self-hosted deployment, a prefix-cache entry generally resides on the worker that computed it. If a load balancer spreads requests with the same prefix across many replicas using round-robin, each replica may build its own copy and hit rate falls. Cache-aware routing sends matching prefixes to the same worker while it remains healthy, so the cache can stay warm where it was built. SGLang's scheduler groups shared-prefix requests,^[2] and OpenAI documents prompt_cache_key as a routing-affinity control.^[3] Treat affinity as a hint, not an unconditional pin: OpenAI's docs recommend adding more prompt_cache_key values when one shared prefix-key combination exceeds roughly 15 requests per minute. Pin too hard and a popular worker can overload.

What prefix caching doesn't do

Prefix caching doesn't make generation free. It reduces repeated prefill work for shared input tokens. Output tokens still require decoding. If your cost is dominated by long generated answers, prefix caching helps less.

That's why serving teams watch time to first token (TTFT) and inter-token latency (ITL) separately. Prefix hits should lower TTFT because repeated prefill gets shorter. ITL often stays about the same because the model still has to decode each new output token.

It also doesn't understand semantic similarity. These two prompts may mean the same thing, but they don't share an exact token prefix:

That's semantic caching territory. Semantic caching reuses previous final answers for similar requests, which changes product behavior and needs safety checks. Prefix caching reuses computation, not answers.

Instrumentation

Track cache hits like any other serving metric.

For local runtimes, log prefix-cache hit rate, reused token count, prefill latency, decode latency, and GPU memory. For hosted APIs, log cached token fields from the response. In both cases, slice by route. A code-assistant route with a long stable repository guide should have a higher cache hit rate than a free-form chat route.

Example log:

Version the static prefix in telemetry and, where your cache-key design permits, in the prefix itself. If the policy text changes, exact-prefix matching already forces a miss after the first changed token. A version label makes that expected warmup miss visible and can intentionally invalidate reuse when a behavior-changing input is outside the hashed prompt.

Isolation boundary

Cache reuse is safe only inside a declared scope. Public tool schemas or public policy text may be reusable broadly; tenant-specific instructions, private documents, adapter state, and authorization context must not become cross-tenant reusable state by accident. A local engine can incorporate a tenant salt or equivalent scope into its cache key, while a hosted API requires the provider's documented isolation guarantees and your own request design.

Design rule

Prefix caching rewards prompt discipline: stable instructions, policy text, schemas, and examples should come first, while user-specific facts, retrieved snippets, and tool results belong later. Measure hit rate, but treat cached-token savings as an optimization rather than permission to send uncontrolled context forever.

Work a cache hit by hand

Suppose the shared prefix for a repository assistant is 6,000 tokens:

Three developers then ask different questions. If the prefix is identical, the runtime can reuse those 6,000 prefix tokens for the second and third requests. It still has to process each user's question and decode each answer, but it avoids repeating most of the prefill work.

Now change one thing: put request_id: R-18492 near the top of the prompt before the shared guide. The first tokens now differ per request. A prefix matcher may miss the whole shared guide even though the guide text itself is unchanged. That's why prompt shape isn't cosmetic. It controls whether the runtime can see the reusable prefix.

This accounting script makes the savings concrete:

Common mistake: blaming exact-prefix caching for stale guidance

• Symptom: The code assistant keeps answering with last week's lint rule after the repository guide changed.

• Cause to investigate first: The rendered prompt, retrieval source, rollout version, or custom cache key still supplied old guide state. In an exact-prefix cache, changed guide tokens don't match old cached KV blocks.

• Fix: Log and inspect the rendered prompt plus its policy version. When the policy text changes, confirm cached-token counts drop for the changed prefix and warm again only for requests containing the new policy. Treat any custom key that ignores changed behavior inputs as a correctness bug.

Mastery check

Evaluation rubric

• Foundational: Explains the difference between the normal per-request KV cache and prefix caching across repeated requests.
• Foundational: Rearranges a prompt so stable instructions, policy text, schemas, and examples stay before user-specific state.
• Intermediate: Explains why one early token change invalidates later reusable blocks through cumulative parent-hash matching.
• Intermediate: Distinguishes prefix caching from semantic caching and names why semantic reuse has higher product risk.
• Advanced: Uses usage fields, route slices, and prefill latency to prove that a production route is getting real cache hits.
• Advanced: Defines versioning, routing, and tenant-isolation rules that expose prompt rollouts and prevent cross-tenant leakage.

Follow-up questions

Common pitfalls

• Symptom: Two questions mean nearly the same thing, but cached-token counts stay at zero. Cause: Prefix caching matches exact token prefixes, not semantic similarity. Fix: Treat semantic reuse as a different system and keep the stable text byte- and token-identical at the start of the prompt.
• Symptom: Moving request_id or retrieved snippets near the top tanks hit rate. Cause: The first mismatch happens before the reusable instructions and guide. Fix: Put shared instructions, guide, schemas, and examples first, then push request-specific state later.
• Symptom: Shared caches improve latency, but privacy review can't prove one tenant never reused another tenant's prefix. Cause: Reusable prefixes were keyed only by tokens, not by tenant scope. Fix: Add tenant-scoped cache keys or salts, don't rely on routing alone for isolation, and log enough metadata to prove cache reads stayed inside the right boundary.
• Symptom: Cached-token share is high, yet user-perceived latency barely improves. Cause: Decode dominates the route, so saved prefill is a small part of total time. Fix: Measure prefill and decode separately and use prefix caching where repeated input is large and answers aren't the main bottleneck.
• Symptom: Teams page on every miss spike after a guide or prompt update. Cause: They expect hit rate to stay flat even when the cache key should change. Fix: Expect temporary warmup misses after a new prefix version goes live and verify recovery with the new version label in logs.
• Symptom: Cost projections or alerts drift away from reality after a provider update. Cause: The system assumed old minimum lengths, lifetimes, or discounts instead of reading usage fields. Fix: Keep thresholds and pricing out of hardcoded mental models and inspect the fields the provider returns on each request.