Semantic Caching & Cost Optimization

A cached answer is behavior produced by one particular release, prompt policy, corpus, and access scope. Semantic caching only works when that contract is explicit.

Suppose the deployed developer-docs assistant repeatedly answers the public question, "How long are revoked API keys retained?" Developers phrase that question many ways. Reusing a verified answer could save generation work and respond faster. Reusing the answer after an API-key retention policy update, across a tenant boundary, or for a live incident-status question could be plainly wrong.

The safe version starts with one semantic cache for public policy answers. It will serve nothing until a shadow replay shows safe hits and worthwhile savings.

An answer is reusable only inside its contract

A response cache isn't a memory of generally true sentences. It's a store of outputs generated under a particular contract. The release bundle from the previous chapter gives us most of that contract already.

For this system, a reusable answer must match all of these fields:

Semantic lookup has a second version boundary. The embedding model, vector preprocessing, and index build determine the score distribution. Record them as embedding_model_id and embedding_index_version in the cache policy and hard-gate semantic hits on those fields. If either changes, rebuild the vectors and recalibrate the threshold in shadow mode before serving semantic hits. An old τ under a new index is not a safe score, even when the admitted answer text still looks right.

Start the lab with the exact release scope and one response generated by the promoted release.

For requests already eligible for answer reuse, an ordinary key-value cache can safely reuse normalized exact repeats as long as its key contains the full scope. It can't see through paraphrasing.

The scope fields must come from application-owned route policy and authenticated context, not from raw user text or a model's guess. The same rule applies to eligibility flags: requires_live_data and writes_state must come from a route or intent-class policy map (or a deterministic classifier behind an admission gate), never from free-form model labels on user text alone. Unknown routes and unknown response classes should bypass answer reuse. The lab also checks that the caller-selected scope agrees with the request before deriving a key.

The exact cache does the correct thing: it refuses a paraphrase and refuses an old answer under a new policy release. Semantic caching adds only the first capability. It must not weaken the second.

Similarity retrieves a candidate, not a truth

A semantic cache embeds a new question, searches stored question embeddings, and proposes a nearby saved answer. In a larger store, that retrieval step is commonly an approximate nearest-neighbor (ANN) search. Systems such as GPTCache apply candidate retrieval before deciding whether to call the LLM at all.^{[1]Reference 1GPTCache: An Open-Source Semantic Cache for LLM Applications Enabling Faster Answers and Cost Savings.https://aclanthology.org/2023.nlposs-1.24/} Sentence-BERT showed why this shape works: sentence embeddings can be compared efficiently with cosine similarity for semantic matching tasks.^{[2]Reference 2Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.https://arxiv.org/abs/1908.10084}

For two vectors $a$ and $b$, cosine similarity is:

$$\operatorname{cosine}(a, b) = \frac{a \cdot b}{\lVert a \rVert_2 \lVert b \rVert_2}$$

The numerator measures their aligned components. Dividing by both lengths makes the result compare direction rather than vector magnitude. A high score says two encoded questions are close under this embedding model. It does not say their answers are interchangeable.

The tiny vectors below are an instructional fixture, not scores from a commercial embedding model. They let us see the failure mode without downloading a model: a restore-key exception can sit near a general retention question while still needing a different answer.

Eligibility rules run before the score threshold

The assistant shouldn't response-cache live incident state or actions at any threshold. Even for a public-policy question, a cached answer must be from the same release scope, and the serving index must match the embedding identity that calibrated the threshold.

This decision procedure checks the non-negotiable rules first. Route policy owns live/write classification. Only an eligible, same-scope request with a matching embedding policy reaches the similarity threshold.

A live incident-status request has a similarity score of 1.00 against a saved response. Can it be a semantic answer-cache hit?

Answer

No. Its answer depends on live operational state. Eligibility is checked before similarity, so the request bypasses response reuse regardless of score.

Version changes invalidate answers without guessing

A time-to-live (TTL) can expire old entries after a period. It can't know that an API-key retention policy changed five minutes after an answer was stored. The release bundle provides a stronger invalidation hook: if policy evidence or answer behavior changes, the release or corpus version changes and old entries are no longer in scope.

Cache identity should inherit the release identity from deployment. Eviction can clean up storage later; correctness shouldn't depend on eviction finishing first. Keep a bounded TTL as a cleanup policy and backstop, but don't treat it as the authoritative invalidation signal.

Why should changing the model, prompt, evidence snapshot, or authorization policy invalidate cached answers even when query similarity stays high?

Answer

Similarity says the request resembles an old request; it doesn't prove the old answer was produced under the current behavior and access contract. Version those dependencies in cache eligibility.

Choose a threshold in shadow mode

Serving a semantic hit immediately turns a retrieval mistake into a user-visible wrong answer. Shadow mode runs the lookup decision but still serves the normal fresh path. Reviewers then label whether each proposed reuse would have been acceptable.

A good cache metric separates two questions:

• Proposal rate: how often would the cache return something?
• Hit precision: among proposed hits, how often is answer reuse acceptable?

High proposal rate without high precision is a cheaper system that's wrong more often. A raw cache hit rate can't distinguish those outcomes.

The labeled fixture below contains public-policy paraphrases, a subtle opened-item exception, and ineligible requests. Thresholds are specific to this fixture and embedding setup; don't copy them into production. Seven probes are enough to explain the tradeoff, but not enough to authorize a 99% production precision claim.

The larger fixture keeps the same 0.980 policy but evaluates 5,714 proposed hits from 10,000 requests. All are accepted, so observed precision is 100%. More importantly, the 95% Wilson lower bound is 99.93%, above the 99% gate. A confidence bound prevents a tiny perfect sample from looking stronger than it's. The traffic still needs to represent the production mix; statistical confidence can't repair a biased replay.

Why doesn't a high hit rate prove that semantic caching is helping?

Answer

A hit is only beneficial when the reused answer is acceptable under the active release and scope. A loose threshold can raise hit rate by serving nearby but incorrect answers.

A safe cache still has to pay for itself

Every semantic lookup incurs work, even on a miss: embedding the request, searching an index, and recording metrics. Evaluate cost only after the precision gate passes.

Let:

• $N$ be requests in a measured period.
• $C_g$ be average fresh-generation cost per request.
• $C_l$ be semantic-lookup cost per request.
• $h$ be the observed safe-hit fraction.

If a hit skips fresh generation, expected period savings are:

$$\text{savings} = N \left(h C_g - C_l\right)$$

These quantities must come from the workload and model you plan to operate. The next example uses labeled measurement fixtures, not provider prices.

Don't guess from list price. Measure generation and lookup cost for the real release and traffic mix, then rerun the gate when either changes.

Authorize writes before they become reusable

The read path now rejects records without approved admission evidence. The write path must enforce the same rule before adding a record to the servable index. A freshly generated answer isn't automatically safe to repeat across paraphrases.

Treat admission as write authorization. Keep unreviewed answers in a quarantine store, attach the evaluation artifact that approved a response class, and admit only records inside the tested release scope.

The admitted record still isn't a universal truth. Reads must match its release and access scope, then pass the calibrated semantic threshold. Admission prevents one bad fresh generation from silently becoming a high-fanout cached answer.

Promote only the narrow policy you tested

Don't turn on semantic caching for all support. Promote only the public-policy scope that passed shadow evidence. Account state and write actions still bypass.

Record why each request hit or bypassed

Once the cache is serving, traces must answer: which release generated the cached response, which cache policy reused it, and why a request bypassed reuse? Without those fields, wrong-hit incidents become difficult to reconstruct.

Watch production for accepted-hit review failures, user retries after cache hits, scope bypass volume, p95 latency, and realized saved generation. An incident should be able to disable this cache policy pointer without changing the production release that generates fresh responses.

Semantic response caching isn't prompt-prefix caching

The cache in this lab can return a stored answer for a paraphrase and skip generation. Provider prompt caching operates at a different layer. OpenAI's documented prompt caching applies automatically to prompts of at least 1,024 tokens, requires an exact prefix match, and reports how many input tokens were cached. The model still computes a new output.^{[3]Reference 3Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching}

The distinction determines the evaluation: semantic answer caching needs labeled reuse precision; prompt-prefix caching needs token and latency accounting. The next chapter expands that economics.