Semantic Caching & Cost Optimization

A cached answer is behavior produced by one particular release, prompt policy, corpus, and access scope. Semantic caching only works when that contract is explicit.

Suppose the deployed developer-docs assistant repeatedly answers the public question, "How long are revoked API keys retained?" Developers phrase that question many ways. Reusing a verified answer could save generation work and respond faster. Reusing the answer after an API-key retention policy update, across a tenant boundary, or for a live incident-status question could be plainly wrong.

The safe version starts with one semantic cache for public policy answers. It will serve nothing until a shadow replay shows safe hits and worthwhile savings.

An answer is reusable only inside its contract

A response cache isn't a memory of generally true sentences. It's a store of outputs generated under a particular contract. The release bundle from the previous chapter gives us most of that contract already.

For this system, a reusable answer must match all of these fields:

Semantic lookup has a second version boundary. The embedding model, vector preprocessing, and index build determine the score distribution. Record them as embedding_model_id and embedding_index_version in the cache policy. If either changes, rebuild the vectors and recalibrate the threshold in shadow mode before serving semantic hits. The admitted answer may still be valid, but an old score threshold no longer has evidence behind it.

Start the lab with the exact release scope and one response generated by the promoted release.

For requests already eligible for answer reuse, an ordinary key-value cache can safely reuse normalized exact repeats as long as its key contains the full scope. It can't see through paraphrasing.

The scope fields must come from application-owned route policy and authenticated context, not from raw user text or a model's guess. Unknown response classes should bypass answer reuse. The lab also checks that the caller-selected scope agrees with the request before deriving a key.

The exact cache does the correct thing: it refuses a paraphrase and refuses an old answer under a new policy release. Semantic caching adds only the first capability. It must not weaken the second.

Similarity retrieves a candidate, not a truth

A semantic cache embeds a new question, searches stored question embeddings, and proposes a nearby saved answer. In a larger store, that retrieval step is commonly an approximate nearest-neighbor (ANN) search. Systems such as GPTCache apply candidate retrieval before deciding whether to call the LLM at all.^{[1]Reference 1GPTCache: An Open-Source Semantic Cache for LLM Applications Enabling Faster Answers and Cost Savings.https://aclanthology.org/2023.nlposs-1.24/} Sentence-BERT showed why this shape works: sentence embeddings can be compared efficiently with cosine similarity for semantic matching tasks.^{[2]Reference 2Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.https://arxiv.org/abs/1908.10084}

For two vectors $a$ and $b$, cosine similarity is:

$$\operatorname{cosine}(a, b) = \frac{a \cdot b}{\lVert a \rVert_2 \lVert b \rVert_2}$$

The numerator measures their aligned components. Dividing by both lengths makes the result compare direction rather than vector magnitude. A high score says two encoded questions are close under this embedding model. It does not say their answers are interchangeable.

The tiny vectors below are an instructional fixture, not scores from a commercial embedding model. They let us see the failure mode without downloading a model: a restore-key exception can sit near a general retention question while still needing a different answer.

Eligibility rules run before the score threshold

The assistant shouldn't response-cache live incident state or actions at any threshold. Even for a public-policy question, a cached answer must be from the same release scope.

This decision procedure checks the non-negotiable rules first. Only an eligible, same-scope request reaches the similarity threshold.

Version changes invalidate answers without guessing

A time-to-live (TTL) can expire old entries after a period. It can't know that an API-key retention policy changed five minutes after an answer was stored. The release bundle provides a stronger invalidation hook: if policy evidence or answer behavior changes, the release or corpus version changes and old entries are no longer in scope.

Cache identity should inherit the release identity from deployment. Eviction can clean up storage later; correctness shouldn't depend on eviction finishing first. Keep a bounded TTL as a cleanup policy and backstop, but don't treat it as the authoritative invalidation signal.

Choose a threshold in shadow mode

Serving a semantic hit immediately turns a retrieval mistake into a user-visible wrong answer. Shadow mode runs the lookup decision but still serves the normal fresh path. Reviewers then label whether each proposed reuse would have been acceptable.

A good cache metric separates two questions:

• Proposal rate: how often would the cache return something?
• Hit precision: among proposed hits, how often is answer reuse acceptable?

High proposal rate without high precision is a cheaper system that's wrong more often. A raw cache hit rate can't distinguish those outcomes.

The labeled fixture below contains public-policy paraphrases, a subtle opened-item exception, and ineligible requests. Thresholds are specific to this fixture and embedding setup; don't copy them into production. Seven probes are enough to explain the tradeoff, but not enough to authorize a 99% production precision claim.

The larger fixture keeps the same 0.980 policy but evaluates 5,714 proposed hits from 10,000 requests. All are accepted, so observed precision is 100%. More importantly, the 95% Wilson lower bound is 99.93%, above the 99% gate. A confidence bound prevents a tiny perfect sample from looking stronger than it's. The traffic still needs to represent the production mix; statistical confidence can't repair a biased replay.

A safe cache still has to pay for itself

Every semantic lookup incurs work, even on a miss: embedding the request, searching an index, and recording metrics. Evaluate cost only after the precision gate passes.

Let:

• $N$ be requests in a measured period.
• $C_g$ be average fresh-generation cost per request.
• $C_l$ be semantic-lookup cost per request.
• $h$ be the observed safe-hit fraction.

If a hit skips fresh generation, expected period savings are:

$$\text{savings} = N \left(h C_g - C_l\right)$$

These quantities must come from the workload and model you plan to operate. The next example uses labeled measurement fixtures, not provider prices.

Don't guess from list price. Measure generation and lookup cost for the real release and traffic mix, then rerun the gate when either changes.

Authorize writes before they become reusable

The read path now rejects records without approved admission evidence. The write path must enforce the same rule before adding a record to the servable index. A freshly generated answer isn't automatically safe to repeat across paraphrases.

Treat admission as write authorization. Keep unreviewed answers in a quarantine store, attach the evaluation artifact that approved a response class, and admit only records inside the tested release scope.

The admitted record still isn't a universal truth. Reads must match its release and access scope, then pass the calibrated semantic threshold. Admission prevents one bad fresh generation from silently becoming a high-fanout cached answer.

Promote only the narrow policy you tested

Don't turn on semantic caching for all support. Promote only the public-policy scope that passed shadow evidence. Account state and write actions still bypass.

Record why each request hit or bypassed

Once the cache is serving, traces must answer: which release generated the cached response, which cache policy reused it, and why a request bypassed reuse? Without those fields, wrong-hit incidents become difficult to reconstruct.

Watch production for accepted-hit review failures, user retries after cache hits, scope bypass volume, p95 latency, and realized saved generation. An incident should be able to disable this cache policy pointer without changing the production release that generates fresh responses.

Semantic response caching isn't prompt-prefix caching

The cache in this lab can return a stored answer for a paraphrase and skip generation. Provider prompt caching operates at a different layer. OpenAI's documented prompt caching applies automatically to prompts of at least 1,024 tokens, requires an exact prefix match, and reports how many input tokens were cached. The model still computes a new output.^{[3]Reference 3Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching}

The distinction determines the evaluation: semantic answer caching needs labeled reuse precision; prompt-prefix caching needs token and latency accounting. The next chapter expands that economics.

Mastery check

Key concepts

• A cached answer belongs to an immutable release scope, not question text alone.
• Exact response caches catch identical scoped requests; semantic caches retrieve answer candidates across paraphrases.
• Similarity is evidence for candidate retrieval, never permission to ignore access, freshness, or side-effect boundaries.
• New corpus or release identity naturally invalidates old answer reuse.
• Embedding-model or index changes require re-embedding and threshold recalibration.
• Shadow-mode precision confidence and measured break-even savings are promotion gates.
• Servable cache writes need a validated admission path; don't let unreviewed answers become reusable records.
• Provider prompt caching reuses repeated input processing, while semantic response caching can skip generation.

Practice tasks

1. Add response_schema="json-citations-v3" to a new scope and prove an old natural-language answer can't hit it.
2. Add one public-policy question that's almost similar but requires a different answer. Re-run the threshold sweep and explain the new selected threshold.
3. Reduce the full shadow window to four accepted proposals. Compute the Wilson lower bound and explain why observed 100% precision no longer passes the gate.
4. Replace the fixture costs with measured numbers for a workload you control. Compute the hit fraction required to break even.
5. Add a cache-policy rollback event that turns semantic hits back into misses while keeping fresh generation on the same release.
6. Change embedding_model_id, rebuild the index under a new version, and keep it shadow-only until its threshold passes calibration.
7. Add a new generated public-policy answer without an approved evidence ID. Prove that it remains quarantined until admission evidence is attached.

Evaluation rubric

• Foundational: Explains why a paraphrase misses an exact cache and why similarity can propose reuse.
• Foundational: Identifies live data and write actions as ineligible for response reuse before considering score.
• Intermediate: Builds a cache key or metadata filter that includes release, corpus, tenant, access, locale, and schema scope.
• Intermediate: Separates generated answers from admitted cache records and quarantines writes without approved evidence.
• Intermediate: Calibrates a threshold in shadow mode using accepted-hit precision and a sample-size-aware lower bound rather than raw hit rate.
• Advanced: Computes measured break-even savings and promotes only the response class proved safe and worthwhile.
• Advanced: Distinguishes semantic stored-answer reuse from provider prefix-computation reuse and chooses evidence for each.

Self-check questions

Common pitfalls

Optimizing hit count instead of correct reuse

• Symptom: Hit rate rises while users report answers for a nearby but different policy case.
• Cause: Threshold was loosened without accepted-reuse labels.
• Fix: Run shadow replay, gate on precision, and exclude classes where a near match is unsafe.

Caching dynamic or write requests

• Symptom: User sees stale incident data or a workflow appears completed when no action ran.
• Cause: Cache eligibility was treated as a similarity decision.
• Fix: Bypass live-state and side-effect requests before vector lookup can produce a servable hit.

Keeping entries across a policy release

• Symptom: A new API-key retention rule is live, but responses still quote the previous rule.
• Cause: Cache records weren't scoped to release and evidence version.
• Fix: Include release and corpus identifiers in lookup filters; treat a version change as an immediate miss.

Reusing a threshold after an embedding change

• Symptom: A previously safe threshold suddenly admits unrelated questions or misses valid paraphrases.
• Cause: The embedding model, preprocessing, or index build changed, so the old score distribution no longer applies.
• Fix: Version the embedding model and index, rebuild vectors, and rerun labeled shadow calibration before serving hits.

Proving safety but not value

• Symptom: Quality remains stable, yet overall request cost or latency gets worse.
• Cause: Embedding and index lookup costs exceed saved generations.
• Fix: Measure lookup overhead and safe-hit fraction for the target traffic before promoting.

Unreviewed answers enter the reusable store

• Symptom: One incorrect fresh answer gets repeated across several paraphrased questions.
• Cause: The write path admitted every generated answer directly into the servable cache.
• Fix: Treat cache admission as write authorization. Admit only validated response classes with recorded evidence; quarantine or review new records before reuse.