Prefix Caching and Prompt Caching

Three developers ask a coding assistant different questions about the same repository. Each request carries 6,000 shared tokens: 600 of instructions, a 4,200-token architecture guide, an 800-token tool schema, and 400 tokens of examples. The first question adds another 48 tokens. Should the model process that entire guide three times?

An available prefix cache can reuse the guide's previously computed state. Put a changing request ID before the guide, though, and the guide no longer has the same preceding context. The unchanged text is insufficient. Reuse also needs compatible model settings, an eligible matching boundary, and the right sharing scope.

For ordinary full causal attention, prefix caching retains Key-Value (KV) state for an exact starting token sequence. The state can live in a GPU pool or another cache tier. Our idealized warm request reuses 6,000 tokens and freshly processes its 48-token question during prefill. Standard KV caching retains state inside a request during decoding; prefix caching makes compatible state reusable across requests. A cache hit saves input work, not a previously generated answer.

The previous KV-cache lesson kept that state inside one generating request. The ledger below asks how many of those tokens a second request can skip, and what it still has to compute.

View larger

Worked example: tracking tokens across cache hits

Suppose our coding assistant's shared prefix breaks down into four parts:

system instructions: 600 tokens
repo guide:          4,200 tokens
tool schema:           800 tokens
few-shot examples:     400 tokens

Assume the first request has made its prefix available before the second arrives. Predict what request two can skip: it can avoid recomputing the shared guide, but it still processes its own question and generates its own response. During a simultaneous cold burst, each request may need a warmup because a cache hit becomes available only after the write completes.

With the eligible entry available, the runtime reuses those 6,000 prefix tokens for requests two and three. New question tokens still attend to the cached guide; reuse does not remove the guide from the context.

Now place a changing request_id before the guide. Tokens before the changed ID may still match, including message wrappers. The guide after it cannot reuse state computed with a different ID. Moving the ID after the eligible guide boundary preserves more reusable work.

For questions of 48, 37, and 51 tokens, three cold requests freshly process 18,136 input tokens. One cold request followed by two full-prefix hits freshly processes 6,136, avoiding 12,000 repeated prefix-token computations. This authored ledger assumes compatible settings, a qualifying 6,000-token boundary, and an available entry. It doesn't predict FLOPs, billing, or milliseconds.

shared_prefix_tokens = 6000
question_tails = [48, 37, 51]

total_fresh_without_cache = 0
total_fresh_with_cache = 0

for request_index, tail_tokens in enumerate(question_tails, start=1):
    fresh_without_cache = shared_prefix_tokens + tail_tokens
    fresh_with_cache = fresh_without_cache if request_index == 1 else tail_tokens
    reused_prefix = 0 if request_index == 1 else shared_prefix_tokens

    total_fresh_without_cache += fresh_without_cache
    total_fresh_with_cache += fresh_with_cache

    print(
        f"request {request_index}: reused_prefix={reused_prefix:4d} "
        f"fresh_tokens={fresh_with_cache:4d}"
    )

saved_tokens = total_fresh_without_cache - total_fresh_with_cache
print(f"fresh tokens without cache: {total_fresh_without_cache}")
print(f"fresh tokens with cache:    {total_fresh_with_cache}")
print(f"prefix tokens saved:        {saved_tokens}")

request 1: reused_prefix=   0 fresh_tokens=6048
request 2: reused_prefix=6000 fresh_tokens=  37
request 3: reused_prefix=6000 fresh_tokens=  51
fresh tokens without cache: 18136
fresh tokens with cache:    6136
prefix tokens saved:        12000

The prefix tokens saved count represents a work ledger, not a billing outcome. Provider write and read fees, cache retention lifetimes, and arrival rates determine whether the initial cache write pays for itself. Those savings also depend on how the inference engine indexes and stores those 6,000 tokens. What does a "match" actually hash under the hood?

Two indexes, one causal dependency

During generation, a decoder-only transformer stores keys and values for tokens it has already processed. When predicting token 501, it doesn't recompute attention keys and values for tokens 1 through 500. That's the normal KV cache.

Retaining that state for generation doesn't by itself make it discoverable by the next incoming request.

With ordinary full causal attention, position $t$ attends only to positions $0 \dots t$. Appending a question doesn't change the prefix's ideal mathematical computation. Changing earlier tokens can change deeper-layer state for a later tools section even if the tools text stays identical. Compatibility includes weights, adapters, positions, attention configuration, and any nontext conditioning. Different execution paths can still introduce floating-point variation.

A single whole-prompt hash would miss a partial overlap. A hash table indexed by successive prefix blocks can discover it. A compressed token trie offers another approach:

SGLang's paper describes RadixAttention retaining prompt and generation state through a radix tree and evicting eligible leaves with LRU.^{[1]Reference 1SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104} A trie split can expose a shared initial segment. The original paper describes the algorithm; each backend's current cache granularity determines whether single-token boundaries are available, so verify that behavior in the deployed implementation.

vLLM uses a hash-based index, not that radix-tree implementation.^{[2]Reference 2Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/} With sixteen-token blocks, 6,005 shared positions contain 375 complete blocks (6,000 tokens). Eligibility and the logits boundary can leave additional work, as the previous lesson demonstrated. PagedAttention supplies block-oriented attention storage; it isn't itself the prefix-matching algorithm.^{[3]Reference 3Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}

The vLLM key includes its parent hash, block token IDs, and relevant extras such as adapter IDs, multimodal hashes, and a cache salt. A teaching notation is:

The engine's cache namespace must also be compatible with its model revision; this formula doesn't assert that every vLLM block embeds a model ID. vLLM documents sha256 as the default and sha256_cbor for reproducible cross-language serialization. It also offers noncryptographic options with a collision-risk warning. Collision resistance is different from authenticating who selected a sharing scope.^{[2]Reference 2Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

Before looking at the diagram, predict the downstream key: if the repository guide changes in block 1 while the tools schema in block 2 stays identical, can block 2 reuse its old cache key?

View larger

from hashlib import sha256
import json

def chain(blocks: list[str]) -> list[str]:
    parent = "root"
    hashes = []
    for block in blocks:
        material = json.dumps(["teaching-block-v1", parent, block], separators=(",", ":"))
        parent = sha256(material.encode()).hexdigest()
        hashes.append(parent)
    return hashes

original = chain(["system", "repo guide v1", "tools"])
edited = chain(["system", "repo guide v2", "tools"])

for index, (left, right) in enumerate(zip(original, edited)):
    print(f"block {index}: {'same key' if left == right else 'different key'}")
assert original[0] == edited[0]
assert all(a != b for a, b in zip(original[1:], edited[1:]))
assert all(len(digest) == 64 for digest in original + edited)

block 0: same key
block 1: different key
block 2: different key

The strings above stand in for token blocks. This fixture tests history dependence, not vLLM's serialization or cache availability: a matching key still needs a stored usable entry. That entry also has to be reachable from the front of the prompt. A question placed before the guide changes the history the guide's blocks inherit.

Prompt architecture: the static-to-dynamic contract

Cache hits require strictly stable prefixes. Put shared, immutable content first and variable, dynamic content last. Before comparing these two layouts, predict which pair shares more initial tokens.

Good prompt shape:

system: You are the repository maintenance assistant.
repo_guide: <shared repository guide>
tools: <shared tool schema>
examples: <shared examples>
user: Why is this test failing?

Poor prompt shape:

user: Why is this test failing?
system: You are the repository maintenance assistant.
repo_guide: <shared repository guide>
tools: <shared tool schema>

In the second layout, changing the question changes the context preceding the guide. A common wrapper might still match, but the guide after the divergence can't reuse its old state. These layouts illustrate serialized content order, not instructions to violate an API's role rules.

That's why agent architectures keep dynamic tool outputs and user-specific state after stable instructions whenever possible. Anthropic also renders tools, then system, then messages internally, so reshuffling a tool schema invalidates a cache even when the visible user text didn't move at all.^{[4]Reference 4Prompt caching (Claude API documentation)https://platform.claude.com/docs/en/build-with-claude/prompt-caching}

def shared_prefix_units(left: list[str], right: list[str]) -> int:
    count = 0
    for a, b in zip(left, right):
        if a != b:
            break
        count += 1
    return count

stable_first_a = ["system", "repo guide", "tools", "test failure question"]
stable_first_b = ["system", "repo guide", "tools", "lint failure question"]
variable_first_a = ["test failure question", "system", "repo guide", "tools"]
variable_first_b = ["lint failure question", "system", "repo guide", "tools"]

print("stable-first reusable units:", shared_prefix_units(stable_first_a, stable_first_b))
print("variable-first reusable units:", shared_prefix_units(variable_first_a, variable_first_b))

stable-first reusable units: 3
variable-first reusable units: 0

These lists contain named sections for clarity; in production, compare actual tokenizer token IDs when diagnosing cache behavior. Always preserve proper message roles; these sketches aren't instructions to stuff user queries into a system message.

Exact-match invalidation traps in production

Semantic equivalence isn't enough. Prefix reuse depends on the exact resulting token sequence, so formatting and serialization differences break the prefix earlier than you might expect.

• Whitespace and line endings: Extra spaces or \r\n instead of \n can change tokenization. The result depends on preprocessing and the tokenizer.
• JSON key order: Reordering keys in JSON rendered as prompt text can alter its token sequence. A provider may separately canonicalize structured API fields.
• Non-deterministic renderers: Timestamps, generated IDs, iteration over sets, or unstable external tool ordering can change the rendered prefix. Python dictionaries themselves preserve insertion order.

Make prompt templates strictly deterministic at the serialization boundary. Serialize JSON with sorted keys where key order doesn't carry meaning, and preserve ordered arrays. Don't blindly strip whitespace from code or structured markdown, since whitespace often defines syntax. Log a prefix version hash so a cache miss can be traced directly to rendered prompt tokens rather than guessed from template files.

Exact token stability enables a match, while memory pressure can still evict unreferenced blocks. Self-hosted engines use least recently used (LRU) policies when VRAM is tight.^{[1]Reference 1SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104} A later request takes the correct cold path and recomputes the prefix. Treat eviction as a latency and cost event; capacity-plan for misses and keep application correctness independent of cache hits.

Hosted provider contracts and pricing economics

Hosted providers don't expose the same raw block-hash controls as vLLM. They still match exact prefixes, but the knobs, field names, and break-even economics differ by vendor and model family. Before reading either contract, predict what happens when a stable guide is followed by a changing question: which provider setting decides where reusable work ends?

OpenAI documents prompt caching in usage fields.^{[5]Reference 5Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching} Anthropic documents cache_control at the top level for automatic caching and on content blocks for explicit breakpoints.^{[4]Reference 4Prompt caching (Claude API documentation)https://platform.claude.com/docs/en/build-with-claude/prompt-caching} Both APIs expose the same underlying mechanics through different controls.

Provider contracts checked October 16, 2026. Model, platform, settings, and cache availability matter. The examples below show request structures and authored usage fixtures; use returned provider fields for live API behavior.

Don't assume a hosted cache behaves like your local runtime. Verify:

OpenAI's guide distinguishes GPT-5.6 and later from earlier models. The newer family has a 1,024-visible-token minimum, exact boundary reporting, and prompt_cache_options.ttl: "30m" (minimum lifetime after the latest write or reuse). Earlier-model minimums vary with request settings; reported cached counts round down to multiples of 128, and retention uses prompt_cache_retention.^{[5]Reference 5Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching}

Implicit lookup can check earlier eligible message endings and the initial consecutive developer-message boundary. A lookup candidate becomes a stored entry only after a matching prefix has been written; writing through a changing question doesn't create a separate entry ending at the stable guide. Mark that guide in both the warmup and later requests. Explicit-only mode also keeps the changing suffix outside the write; confirm actual reuse from usage and diagnostics.^{[5]Reference 5Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching[6]Reference 6Prompt cache diagnosticshttps://developers.openai.com/api/docs/guides/prompt-caching/diagnostics}

Predict the difference between a candidate boundary and an available entry. These tuple elements stand in for token IDs; the fixture models the documented request-boundary dependency. OpenAI's private cache implementation remains internal:

guide = (7,) * 6000
first = guide + (11,) * 48
second = guide + (12,) * 37
entries = {first}  # Only the first complete request was written.

def longest_available_prefix(tokens, candidates):
    return max((end for end in candidates if tokens[:end] in entries), default=0)

candidates = [len(second), len(guide)]
before = longest_available_prefix(second, candidates)
entries.add(guide)  # A warmup with an explicit marker writes this boundary.
after = longest_available_prefix(second, candidates)
assert (before, after) == (0, 6000)
print("reused tokens without / with the guide entry:", before, "/", after)

reused tokens without / with the guide entry: 0 / 6000

View larger

For our coding assistant, mark the end of the repository guide explicitly and place the user question after it. This Responses API request structure illustrates the controls. Replace its placeholder with the complete guide; the placeholder alone won't meet the minimum:

{
  "model": "gpt-6-astra",
  "prompt_cache_options": { "mode": "explicit", "ttl": "30m" },
  "prompt_cache_key": "workspace-alpha:repo-guide-v7",
  "input": [
    {
      "role": "developer",
      "content": [{
        "type": "input_text",
        "text": "<stable instructions and the complete repository guide>",
        "prompt_cache_breakpoint": { "mode": "explicit" }
      }]
    },
    { "role": "user", "content": "Why is this test failing?" }
  ]
}

Explicit-only mode leaves the suffix outside the selected write. For GPT-5.6 and later, prompt_cache_key is optional separate accounting, not a routing requirement. Diagnostics even distinguish a key-related reported miss from a physical cache miss. Earlier models use keys to help routing. Choose any customer key in the trusted gateway; a caller-supplied label isn't authenticated isolation.^{[6]Reference 6Prompt cache diagnosticshttps://developers.openai.com/api/docs/guides/prompt-caching/diagnostics}

Current OpenAI pricing lists 1.25-times write and 0.10-times read rates for GPT-6 Astra and the GPT-5.6 family, relative to their corresponding uncached input rates. GPT-6.1 Sol's read rate is 0.05 times.^{[7]Reference 7OpenAI API Pricinghttps://developers.openai.com/api/docs/pricing} Anthropic's pricing lists 1.25-times five-minute writes and 2-times one-hour writes. Most Claude reads cost 0.10 times base input; Fable 5.1 and Mythos 5.1 use 0.025 times, while Opus 5.5 and Sonnet 5.5 use 0.05 times. Availability and other pricing dimensions remain model-specific.^{[8]Reference 8Anthropic Model Pricinghttps://platform.claude.com/docs/en/about-claude/pricing}

At a 0.10-times read rate, the 5-minute write surcharge is 0.25 units and each subsequent full read saves 0.90 units. The break-even requirement is:

For the 5-minute tier, $\lceil 0.25 / 0.90 \rceil = 1$ read repays that premium. For the 1-hour tier with a 1.0-unit surcharge, $\lceil 1.0 / 0.90 \rceil = 2$ reads are required. If a cached prefix is written and never read before it expires, you paid a 25% to 100% financial penalty over uncached input!

Claude cache minimums are model-specific. As checked October 16, 2026, the guide lists thresholds of 512, 1,024, 2,048, or 4,096 tokens. Confirm the exact model and platform before placing a breakpoint. The guide directs Bedrock users to AWS's contract. A breakpoint checks twenty positions, including itself, for earlier entries. An entry twenty new positions behind the breakpoint is just outside that lookup window, which measures eligible positions rather than prefix length. Consecutive tool-use blocks or consecutive tool-result blocks count as one position on the Claude API. Requests have four breakpoint slots; top-level automatic cache_control uses one. Concurrent reuse becomes available after the first response begins.^{[4]Reference 4Prompt caching (Claude API documentation)https://platform.claude.com/docs/en/build-with-claude/prompt-caching}

For the same stable-guide pattern, Claude's Messages API places the explicit marker on a system content block. Replace the placeholder with your complete guide; the literal placeholder is below the model's minimum:

{
  "model": "claude-opus-5",
  "max_tokens": 256,
  "system": [{
    "type": "text",
    "text": "<stable instructions and the complete repository guide>",
    "cache_control": { "type": "ephemeral" }
  }],
  "messages": [{ "role": "user", "content": "Why is this test failing?" }]
}

Both examples select a reusable boundary. Neither marker creates a hit unless a compatible entry is available. For supported newer OpenAI models, prompt_cache_options.prewarm: true can write the selected prefix without generating a response; cache-write charges still apply.^{[5]Reference 5Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching}

response_usage = {
    "input_tokens": 2600,
    "input_tokens_details": {
        "cached_tokens": 2000,
        "cache_write_tokens": 400,
    },
}

details = response_usage["input_tokens_details"]
cache_reads = details.get("cached_tokens", 0)
cache_writes = details.get("cache_write_tokens", 0)
uncached = response_usage["input_tokens"] - cache_reads - cache_writes
print(f"cache-read prompt tokens: {cache_reads}")
print(f"cache-write prompt tokens: {cache_writes}")
print(f"uncached prompt tokens: {uncached}")
assert cache_reads + cache_writes + uncached == response_usage["input_tokens"]
assert min(cache_reads, cache_writes, uncached) >= 0

cache-read prompt tokens: 2000
cache-write prompt tokens: 400
uncached prompt tokens: 200

Now compute the exact reuse threshold that repays the write premium across cache lifetimes:

from math import ceil

read_rate = 0.1
for ttl, write_rate in [("5m", 1.25), ("1h", 2.0)]:
    extra_write_cost = write_rate - 1.0
    savings_per_read = 1.0 - read_rate
    reads_needed = ceil(extra_write_cost / savings_per_read)
    print(f"{ttl} cache: {reads_needed} reuse request(s) to repay write premium")

5m cache: 1 reuse request(s) to repay write premium
1h cache: 2 reuse request(s) to repay write premium

The earlier usage fixture uses Responses API field names; Chat Completions uses a different usage schema. For an authored Claude usage breakdown of the same 2,600-token total, use 2,000 cache_read_input_tokens, 400 cache_creation_input_tokens, and 200 input_tokens. Claude's input_tokens excludes the read/write portions. Sum all three before calculating its reused-token percentage.^{[4]Reference 4Prompt caching (Claude API documentation)https://platform.claude.com/docs/en/build-with-claude/prompt-caching}

The break-even calculation assumes full reads of the written prefix. Can three reported hits still cost more than four cold requests? Try a 6,000-token guide where only the first 1,024 tokens stay stable and the other 4,976 change each time. The authored usage rows assume all changing tokens are written at the 1.25-times rate, and reads use the 0.10-times rate. Output and other charges are held aside.^{[7]Reference 7OpenAI API Pricinghttps://developers.openai.com/api/docs/pricing[8]Reference 8Anthropic Model Pricinghttps://platform.claude.com/docs/en/about-claude/pricing}

usage = [(0, 6000)] + [(1024, 4976)] * 3  # (read tokens, write tokens)
base_cost = sum(read + write for read, write in usage)
cached_cost = sum(0.10 * read + 1.25 * write for read, write in usage)
hits = sum(read > 0 for read, _ in usage)
assert hits == 3 and cached_cost > base_cost
print(f"requests with reads: {hits}/{len(usage)}")
print(f"input cost units, uncached / cached: {base_cost:.1f} / {cached_cost:.1f}")
print(f"cost increase despite hits: {cached_cost / base_cost - 1:.1%}")

requests with reads: 3/4
input cost units, uncached / cached: 24000.0 / 26467.2
cost increase despite hits: 10.3%

View larger

Price each returned read, write, and uncached portion. A high binary hit rate can hide repeated write premiums. If only the first 1,024 tokens are worth retaining, place the write boundary there and process the changing suffix uncached; verify that boundary qualifies for the selected model and settings.

Distributed serving: cache-aware routing versus round-robin

With independent worker-local caches, round-robin routing can require one cold warmup per worker. After those workers warm, repeated requests can hit. A deployment with shared tiers or cache transfers has different locality and transfer costs; not every cache entry is confined permanently to its original GPU.

Cache-aware routing can prefer a compatible warm worker. SGLang's paper describes cache-aware scheduling inside its engine; a distributed router needs its own cross-worker routing logic.^{[1]Reference 1SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104} Routing affinity also doesn't enforce cache ownership. Apply the authenticated sharing scope at lookup independently of the destination.

Affinity needs health and load handling. A hot prefix can overload its preferred worker; spilling to another worker may be worth a cold warmup. Bounded-load consistent hashing is one policy; configure its load boundary for the router and workload.

The fixture uses a fixed set of three initially cold workers, sequential completed warmups, one prefix, and no eviction. Its modulo hash models fixed worker affinity, so changing the worker count can remap many keys. Health-aware routing would need additional signals and logic.

from hashlib import sha256

workers = ["gpu-a", "gpu-b", "gpu-c"]
prefix = "repo-guide-v7|tool-schema-v2"

def affinity_worker(stable_prefix: str) -> str:
    digest = int(sha256(stable_prefix.encode()).hexdigest(), 16)
    return workers[digest % len(workers)]

round_robin = [workers[i % len(workers)] for i in range(4)]
affinity = [affinity_worker(prefix) for _ in range(4)]
print("round-robin workers:", round_robin)
print("affinity workers:", affinity)
print("cold workers with round-robin:", len(set(round_robin)))
print("cold workers with affinity:", len(set(affinity)))

round-robin workers: ['gpu-a', 'gpu-b', 'gpu-c', 'gpu-a']
affinity workers: ['gpu-b', 'gpu-b', 'gpu-b', 'gpu-b']
cold workers with round-robin: 3
cold workers with affinity: 1

View larger

Sending every repeat to one warm worker cuts cold prefills. It does not remove the work that still happens after the prefix hits.

Prefill versus decode: what prefix caching doesn't change

In our full-attention example, new input and generated tokens still attend to the retained prefix. GPU-resident entries still occupy the pool; another cache tier can add transfer costs. Reuse doesn't extend the model's context limit. Long output generation can dominate total response time even when input processing improves.

Separate time to first token (TTFT) from inter-token latency (ITL). A hit can reduce input processing; client TTFT also includes queueing and transport. It doesn't directly skip output-token generation. Decode performance can still change indirectly with batching, kernels, or load.

Prefix caching doesn't match by meaning. These questions express similar intent but don't share the same complete question prefix; any common message wrapper is a separate overlap:

Why did this test fail after the refactor?
What broke in the failing spec after the code change?

Reusing an answer for such questions belongs to semantic caching. Semantic caching needs an answer-validity policy. Correct prefix reuse preserves the supplied context; it doesn't make the model's answer accurate or eliminate numerical differences between execution paths.

Here are authored phase-time assumptions, with decode deliberately held constant to isolate prefill savings. Use a trace from the running engine to predict its latency:

uncached = {"prefill_ms": 1220, "decode_ms": 880}
cached = {"prefill_ms": 170, "decode_ms": 880}

prefill_saved = uncached["prefill_ms"] - cached["prefill_ms"]
decode_change = uncached["decode_ms"] - cached["decode_ms"]
print(f"prefill saved: {prefill_saved} ms")
print(f"decode change: {decode_change} ms")
assert prefill_saved > decode_change

prefill saved: 1050 ms
decode change: 0 ms

Before looking at dashboards, predict this counterexample: if input work falls by 1,050 ms but queueing grows by 1,180 ms, does the client see the first token sooner?

# Authored times; first-token work includes computation through the first output.
cold = {"queue_ms": 20, "first_token_work_ms": 1220}
warm = {"queue_ms": 1200, "first_token_work_ms": 170}
cold_ttft = sum(cold.values())
warm_ttft = sum(warm.values())
print("first-token work saved:", cold["first_token_work_ms"] - warm["first_token_work_ms"], "ms")
print("cold / warm TTFT:", cold_ttft, "/", warm_ttft, "ms")
print("client TTFT change:", warm_ttft - cold_ttft, "ms")
assert warm["first_token_work_ms"] < cold["first_token_work_ms"]
assert warm_ttft > cold_ttft

first-token work saved: 1050 ms
cold / warm TTFT: 1240 / 1370 ms
client TTFT change: 130 ms

View larger

The warm request is slower from the client's perspective. Compare both server work and client timestamps under representative load; a hit alone doesn't determine latency.

Telemetry: token-level yield versus binary hit counters

Usage fields report credited reuse for a request. Operations teams also need to know whether that reuse helps the route over time. Provider accounting and physical cache behavior need not be identical, as OpenAI's key-change diagnostic illustrates.^{[6]Reference 6Prompt cache diagnosticshttps://developers.openai.com/api/docs/guides/prompt-caching/diagnostics}

For self-hosted runtimes, log prefix cache hit rates, reused token counts, prefill latency, decode latency, and GPU memory allocation. For hosted APIs, parse and record returned cache fields. Always slice metrics by route, model revision, prefix version, and traffic window. Long-lived multi-turn chat sessions reuse substantial prefixes too; route names alone don't explain savings.

Synthetic application log schema:

{
  "route": "repo-guide-rag",
  "prompt_tokens": 6048,
  "cached_tokens": 6000,
  "cache_write_tokens": 0,
  "prefill_ms": 170,
  "decode_ms": 880,
  "cache_version_id": "repo_guide_v7"
}

Version the static prefix in telemetry. If the repository guide changes, blocks containing and following the first changed token can't reuse the old cache entry, but earlier unchanged blocks can still hit. A version label correlates that partial warmup with software rollouts.

Beware the binary hit counter trap. A request reusing 6,000 of 6,048 tokens has 99.2% token reuse; one reusing 600 has 9.9%. Both increment a binary hit counter. These percentages measure input-token volume, not prefill FLOPs or elapsed time. Compute a volume-weighted token ratio, then measure time and price separately:

View larger

Memory management: cached can also mean available

Physical GPU VRAM is strictly bounded. What happens when hundreds of concurrent requests arrive and memory fills up?

vLLM V1's block pool distinguishes references, cached content, and allocation availability. On release, an unreferenced block can retain its cache key while joining the free queue. A matching request can touch it; allocation for different content can evict it. Resident pool storage need not shrink when ownership changes.^{[2]Reference 2Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

Don't recycle a block while owners or pending GPU operations still need it. Proper scheduler preemption can instead release ownership and arrange recomputation or supported offload. When available blocks run out, the engine must defer or preempt work according to its policy. The previous lesson's failed-append lab demonstrated that allocation failure must preserve existing state.

No universal 85%/95% VRAM watermarks follow from this design. Measure referenced blocks, cached unreferenced candidates, available blocks, eviction churn, and preemption. GPU allocation alone can stay constant while the logical pool fills.

In the authored snapshot below, how many blocks can be allocated: 150, 550, or 950?

pool = {"referenced": 450, "cached_unreferenced": 400, "uncached_free": 150}
assert sum(pool.values()) == 1000
available = pool["cached_unreferenced"] + pool["uncached_free"]
new_blocks = 200
print("available blocks:", available)
print("200 new blocks fit:", new_blocks <= available)
print("available after allocating 200:", available - new_blocks)
print("must replace at least this many cached blocks:", max(0, new_blocks - pool["uncached_free"]))
assert available == 550

available blocks: 550
200 new blocks fit: True
available after allocating 200: 350
must replace at least this many cached blocks: 50

View larger

The allocation order could replace more cached entries; this lower bound leaves the engine's free-queue order unspecified. A hit-rate drop can accompany successful allocation without any change in resident pool bytes.

Treat LRU eviction as a capacity signal. If your token hit ratio suddenly drops during peak traffic hours, check eviction churn: high concurrency may be evicting unpinned prefixes before subsequent requests arrive to reuse them.

events = [
    {"version": "guide-v6", "cached_tokens": 6000, "input_tokens": 6048},
    {"version": "guide-v7", "cached_tokens": 600, "input_tokens": 6048},
    {"version": "guide-v7", "cached_tokens": 6000, "input_tokens": 6048},
]

for version in sorted({event["version"] for event in events}):
    rows = [event for event in events if event["version"] == version]
    hits = sum(event["cached_tokens"] > 0 for event in rows)
    reused = sum(event["cached_tokens"] for event in rows)
    total = sum(event["input_tokens"] for event in rows)
    print(f"{version}: {hits}/{len(rows)} requests hit; {reused / total:.1%} tokens reused")

guide-v6: 1/1 requests hit; 99.2% tokens reused
guide-v7: 2/2 requests hit; 54.6% tokens reused

Multi-tenant security: timing side-channels and cache isolation

A cache hit is faster, and that speed can leak. A probe that contains suspected confidential text, if it comes back repeatedly faster, may indicate that another tenant already warmed that prefix. Queueing, routing, and other warm prefixes also affect one response's timing, so identify the source with repeated controlled probes and provider-side evidence. Returned cache counters can also be a probing signal.

Incorrect history keys or collisions can cause reuse of the wrong state; memory corruption requires a separate out-of-bounds condition. Exact token matching and SHA-256 also don't authenticate the caller or authorize cross-tenant sharing.

vLLM supports an optional request cache_salt included in the first block's key. The gateway should select and enforce the scope from verified identity, overriding or rejecting unauthorized caller-selected values.^{[2]Reference 2Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/} A salt known to an attacker and accepted without authorization isn't an isolation boundary. Distinct structured keys and collision resistance protect this lookup path under the configured policy; they don't prove all side channels impossible.

Provider boundaries are separate contracts. OpenAI documents organization and regional processing isolation; its newer accounting keys describe cache accounting, while physical KV storage separation requires separate provider documentation. Claude documents workspace isolation on its API, Claude Platform on AWS, and Microsoft Foundry, and organization isolation on Bedrock and Google Cloud. A tenant label in application logs cannot create an additional provider isolation boundary.^{[5]Reference 5Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching[4]Reference 4Prompt caching (Claude API documentation)https://platform.claude.com/docs/en/build-with-claude/prompt-caching}

This CPU fixture verifies distinct keys for supplied scopes. Authentication of those strings and provider isolation require separate controls:

from hashlib import sha256
import json

def cache_key(tenant: str, prefix_version: str, tokens: str) -> str:
    material = json.dumps(
        ["model-revision-42", tenant, prefix_version, tokens],
        ensure_ascii=False, separators=(",", ":"),
    )
    return sha256(material.encode()).hexdigest()

prefix = "system|private-guide|tools"
alpha = cache_key("tenant-alpha", "v7", prefix)
beta = cache_key("tenant-beta", "v7", prefix)
print("same tokens, scoped keys differ:", alpha != beta)
print("stored key length:", len(alpha))
print("alpha key for display:", alpha[:12])
assert alpha != beta
assert len(alpha) == 64
assert cache_key("a|b", "c", "d") != cache_key("a", "b|c", "d")

same tokens, scoped keys differ: True
stored key length: 64
alpha key for display: 5f867552dca0

Now try a forged label. The sessions below stand in for authentication results; a real login service must verify the session before supplying its tenant scope.

from hashlib import sha256
import json

verified_sessions = {"session-a": "tenant-alpha", "session-b": "tenant-beta"}

def scoped_key(scope: str, prefix: str) -> str:
    return sha256(json.dumps([scope, prefix], separators=(",", ":")).encode()).hexdigest()

def gateway_key(session: str, client_label: str, prefix: str) -> str:
    # The untrusted client_label never selects the sharing scope.
    principal = verified_sessions[session]
    return scoped_key(principal, prefix)

prefix = "private-guide-v7"
victim = gateway_key("session-a", "tenant-alpha", prefix)
unsafe_probe = scoped_key("tenant-alpha", prefix)
safe_probe = gateway_key("session-b", "tenant-alpha", prefix)
print("caller-selected probe matches victim:", unsafe_probe == victim)
print("server-selected probe matches victim:", safe_probe == victim)
assert unsafe_probe == victim and safe_probe != victim
try:
    gateway_key("unknown-session", "tenant-alpha", prefix)
except KeyError:
    print("unknown session rejected before lookup")
else:
    raise AssertionError("unauthenticated session accepted")

caller-selected probe matches victim: True
server-selected probe matches victim: False
unknown session rejected before lookup

View larger

Failure modes: debugging stale outputs and silent eviction

A common engineering trap is blaming exact-prefix caching when a model outputs outdated rules:

• Symptom: The coding bot continues enforcing last week's 90-day key rotation rule even after the repository guide changed to 30 days.
• Input check: Did the request actually contain the new guide? Inspect retrieval and rollout versions, plus a protected digest of the actual rendered stable guide. Comparing a full prompt digest to a guide-only digest is meaningless; changing user questions also changes the full digest.
• Cache check: Were preceding history, weights, adapters, and other conditioning included in the computation key? Correct exact keys prevent the edited guide from matching its old entry; custom keys can be wrong.
• Answer check: If input and cache behavior are correct, evaluate instruction following and answer quality. Fresh computation can still produce stale advice.

For supported OpenAI Responses models, set prompt_cache_options.comparison_response_id to a recent completed baseline response's ID and inspect prompt_cache_diagnostics. This requests a comparison; it doesn't load the old conversation or change caching behavior. Diagnostics are best effort. Use current usage counts for credited reuse.^{[6]Reference 6Prompt cache diagnosticshttps://developers.openai.com/api/docs/guides/prompt-caching/diagnostics}

from hashlib import sha256

def exact_prefix_key(guide: str) -> str:
    return sha256(f"system|{guide}|tools".encode()).hexdigest()

before = exact_prefix_key("keys rotate after 90 days")
after = exact_prefix_key("keys rotate after 30 days")
print("changed guide invalidates exact-prefix key:", before != after)
assert before != after

changed guide invalidates exact-prefix key: True

Before promoting the guide, test both cold and warm requests against the updated rule. The cache should change the work performed, not which guide the model receives. A reused prefix and a reused answer are different events. Point to the token boundary, the usage field, or the tenant scope that made the reuse allowed.