Semantic Caching & Cost Optimization

When the incident-evidence assistant answers "How long are revoked API keys retained?", developers will ask it a dozen ways. Reusing one verified answer would be cheaper and faster. Reusing it for "What is incident INC-48192's current status?", after a policy update, or across a tenant boundary would be a confidently wrong cached answer.

The last chapter pinned every answer to one immutable release bundle: model, prompt, policy, corpus, and serving image together. That identity is what makes reuse possible, and also what makes it unsafe when any field moves. The PolicyOps team will cache public-policy answers only, and only after a shadow replay shows safe hits that still save money.

An answer is reusable only inside its contract

A response cache isn't a memory of generally true sentences. It's a store of outputs generated under a particular contract. The last lab's stable release was deploy-answerer@sha256:32b8ed409b8e, pinned to its ReleaseOps evidence corpus. Public API-key policy belongs to a separate policy-answerer release with a different corpus and release identity. A cached sentence from one bundle isn't a cached sentence from the other, even when the question text matches.

Hold two boundaries apart before continuing. If the live incident question lands near the retention question in vector space, it still bypasses. If the policy corpus moves, an exact repeat still misses under the new scope. Similarity can nominate a candidate; the contract decides whether it may serve.

For this system, a reusable answer must match all of these fields:

Semantic lookup has a second version boundary. The embedding model, vector preprocessing, and index build determine the score distribution. Record them as embedding_model_id and embedding_index_version in the cache policy and hard-gate semantic hits on those fields. If either changes, rebuild the vectors and recalibrate the threshold in shadow mode before serving semantic hits. An old $\tau$ under a new index isn't a safe score, even when the admitted answer text still looks right.

The lab starts from one promoted public-policy release of that same service, plus one admitted answer generated under it.

The first cell only establishes identity. Predict its three labels before running it: the release ID and corpus version should be visible, and the seed answer should have a stable ID that later traces can carry forward.

For requests already eligible for answer reuse, an ordinary key-value cache can safely reuse normalized exact repeats as long as its key contains the full scope. It can't see through paraphrasing.

The scope fields must come from application-owned route policy and authenticated context, not from raw user text or a model's guess. The same rule applies to eligibility flags: requires_live_data and writes_state must come from a route or intent-class policy map (or a deterministic classifier behind an admission gate), never from free-form model labels on user text alone. Unknown routes and unknown response classes should bypass answer reuse. The lab also checks that the caller-selected scope agrees with the request before deriving a key.

Predict the next output before running it: the normalized exact repeat should hit, the paraphrase should miss, the updated policy should miss, and the cross-tenant request should raise a scope error. None of those outcomes needs a vector search.

The exact cache does the correct thing: it refuses a paraphrase and refuses an old answer under a new policy release. Semantic caching adds only the first capability. It must not weaken the second.

Similarity retrieves a candidate, not a truth

A semantic cache embeds a new question, searches stored question embeddings, and proposes a nearby saved answer. Systems such as GPTCache put that candidate lookup before the decision to call an LLM at all.^{[1]Reference 1GPTCache: An Open-Source Semantic Cache for LLM Applications Enabling Faster Answers and Cost Savings.https://aclanthology.org/2023.nlposs-1.24/}

In a larger store, the lookup is commonly an approximate nearest-neighbor (ANN) search. Sentence-BERT showed why this shape works: sentence embeddings can be compared efficiently with cosine similarity for semantic matching tasks.^{[2]Reference 2Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.https://arxiv.org/abs/1908.10084}

For two vectors $a$ and $b$, cosine similarity is:

$$\operatorname{cosine}(a, b) = \frac{a \cdot b}{\lVert a \rVert_2 \lVert b \rVert_2}$$

The numerator measures their aligned components. Dividing by both lengths makes the result compare direction rather than vector magnitude. A high score says two encoded questions are close under this embedding model. It does not say their answers are interchangeable.

The tiny vectors below are an instructional fixture, not scores from a commercial embedding model. They let us see the failure mode without downloading a model: a restore-key exception can sit near a general retention question while still needing a different answer. Restore-key is still a public-policy read, so a live/write gate won't save you.

Predict the scores before running the cell. The paraphrase should be nearest, the restore-key exception should look dangerously close, and the live incident should be far away. Which of those predictions would be enough to serve an answer? None by itself.

💡 Key insight: Cosine similarity ranks nearby questions. It doesn't certify that two answers are interchangeable under the current release, corpus, and access scope.

Live incident state is far from the policy axis here, so a score threshold would reject it. Restore-key wouldn't. Eligibility can catch live state and writes. It can't catch a nearby policy exception.

Eligibility rules run before the score threshold

The assistant shouldn't cache live incident state or write actions at any threshold. Restore-key is different: it's still a public-policy read, so eligibility won't catch it. Even for an eligible question, a cached answer must be from the same release scope, and the serving index must match the embedding identity that calibrated the threshold.

This decision procedure checks the non-negotiable rules first. Route policy owns live/write classification. Only an eligible, same-scope request with a matching embedding policy reaches the similarity threshold. Similarity is last, not first.

Predict the four decisions before running the next cell: the paraphrase should hit; the incident and revoke request should bypass before scoring; the stale index should miss even with a strong score. The order is the safety property.

The function returns as soon as a gate fails. Live incidents and key-revocation writes never reach the score. A stale embedding index misses even when the question is a perfect paraphrase of an admitted answer.

A live incident-status request has a similarity score of 1.00 against a saved response. Can it be a semantic answer-cache hit?

Answer

No. Its answer depends on live operational state. Eligibility is checked before similarity, so the request bypasses response reuse regardless of score.

Version changes invalidate answers without guessing

A time-to-live (TTL) can expire old entries after a period. It can't know that an API-key retention policy changed five minutes after an answer was stored. The release bundle provides a stronger invalidation hook: if policy evidence or answer behavior changes, the release or corpus version changes and old entries are no longer in scope.

Before running the invalidation example, predict both decisions: the old scope should hit, while the same question under the new release should miss immediately. Storage eviction can lag behind correctness.

Cache identity should inherit the release identity from deployment. Eviction can clean up storage later; correctness shouldn't depend on eviction finishing first. Keep a bounded TTL as a cleanup policy and backstop, but don't treat it as the authoritative invalidation signal.

Why should changing the model, prompt, evidence snapshot, or authorization policy invalidate cached answers even when query similarity stays high?

Answer

Similarity says the request resembles an old request; it doesn't prove the old answer was produced under the current behavior and access contract. Version those dependencies in cache eligibility.

Choose a threshold in shadow mode

Serving a semantic hit immediately turns a retrieval mistake into a user-visible wrong answer. Shadow mode runs the lookup decision but still serves the normal fresh path. Reviewers then label whether each proposed reuse would have been acceptable.

A good cache metric separates two questions:

• Proposal rate: how often would the cache return something?
• Hit precision: among proposed hits, how often is answer reuse acceptable?

High proposal rate without high precision is a cheaper system that's wrong more often. A raw cache hit rate can't distinguish those outcomes.

The labeled fixture below contains public-policy paraphrases, the restore-key exception, and ineligible requests. These scores are a separate measurement from the 3-d plot. That plot put restore-key at 0.994 so you could see a wrong neighbor inside the 0.98 cone. Here restore-key sits at 0.965, which is how a 0.980 threshold can reject it after you have labels. Don't mix the two score columns, and don't copy either threshold into production. Seven probes are enough to explain the tradeoff, but not enough to authorize a 99% production precision claim.

Pause at the three candidate thresholds. The lowest threshold should admit the most proposals and the wrong exception; the strictest should reject valid paraphrases as well as the exception. The selected value needs to clear precision while preserving useful proposal volume, then earn confidence on a larger replay.

The larger fixture keeps the same 0.980 policy but evaluates 5,714 proposed hits from 10,000 requests. All are accepted, so observed precision is 100%. The 95% Wilson lower bound is 99.93%, above the 99% gate. A confidence bound prevents a tiny perfect sample from looking more reliable than the sample supports. The traffic still needs to represent the production mix; statistical confidence can't repair a biased replay.

Why doesn't a high hit rate prove that semantic caching is helping?

Answer

A hit is only beneficial when the reused answer is acceptable under the active release and scope. A loose threshold can raise hit rate by serving nearby but incorrect answers.

A safe cache still has to pay for itself

Every semantic lookup incurs work, even on a miss: embedding the request, searching an index, and recording metrics. Evaluate cost only after the precision gate passes.

Let:

• $N$ be requests in a measured period.
• $C_g$ be average fresh-generation cost per request.
• $C_l$ be semantic-lookup cost per request.
• $h$ be the observed safe-hit fraction.

If a hit skips fresh generation, expected period savings are:

$$\text{savings} = N \left(h C_g - C_l\right)$$

These quantities must come from the workload and model you plan to operate. The next example uses labeled measurement fixtures, not provider prices.

Compute the break-even point before running the cell. With $C_l = 0.00008$ and $C_g = 0.0040$, $h^* = C_l / C_g = 2\%$. The fixture's safe-hit fraction is 57.1%, so its savings should be positive, but that conclusion follows only after safe-hit precision has passed.

Don't guess from list price. Measure generation and lookup cost for the real release and traffic mix, then rerun the gate when either changes.

Those savings only count if the reused answers were allowed into the index. A cheap wrong answer isn't a win.

Authorize writes before they become reusable

The read path now rejects records without approved admission evidence. The write path must enforce the same rule before adding a record to the servable index. A freshly generated answer isn't automatically safe to repeat across paraphrases.

Treat admission as write authorization. Keep unreviewed answers in a quarantine store, attach the evaluation artifact that approved a response class, and admit only records inside the tested release scope.

Predict the write decisions: missing evidence should quarantine, approved public-policy evidence should admit, and a live-incident response class should quarantine. A fresh answer earns reuse only after this gate.

The admitted record still isn't a universal truth. Reads must match its release and access scope, then pass the calibrated semantic threshold. Admission prevents one bad fresh generation from silently becoming a high-fanout cached answer.

Promote only the narrow policy you tested

Don't turn on semantic caching for every route PolicyOps owns. Promote only the public-policy scope that passed shadow evidence. Account state, live incidents, and write actions still bypass.

Read the promotion gate as three independent questions. Predict the output before running it: quality, economics, and scope should all pass for the tested public-policy class, while untested dynamic and write routes remain outside promotion.

That promotion is the artifact the next chapter's cost ledger consumes: public-policy hits skip generation, and everything else still has to be priced.

Record why each request hit or bypassed

Once the cache is serving, a trace has to reconstruct a hit: which release generated the stored answer, which cache policy reused it, and why a neighboring request bypassed. Without those fields, a wrong-hit incident is a ghost.

Watch production for accepted-hit review failures, user retries after cache hits, scope bypass volume, p95 latency, and realized saved generation. An incident has to disable this cache policy pointer without changing the production release that generates fresh responses.

Those signals also point to different repairs. A wrong answer after a valid hit sends you to admission, scope, or threshold evidence; rising bypass volume sends you to route policy or version churn; savings below expectation sends you to lookup cost, miss rate, or traffic mix. Keep the decision trace that lets an operator tell those cases apart.

Semantic response caching isn't prompt-prefix caching

The cache in this lab can return a stored answer for a paraphrase and skip generation. Provider prompt caching operates at a different layer: it reuses work for an identical rendered prompt prefix, then generates a new answer. OpenAI says prompt caching is enabled by default for supported models. Current docs list a minimum of 1,024 visible input tokens for GPT-5.6 and later and 2,048 for earlier models, with occasional shorter hits. GPT-5.6 and later support explicit or implicit cache breakpoints, so a shared shorter prefix isn't automatically reusable unless it ends at an eligible breakpoint.^{[3]Reference 3Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching}

The distinction determines the evaluation: semantic answer caching needs labeled reuse precision; prompt-prefix caching needs token and latency accounting. The next chapter expands that economics.

Predict the final two routes before running the last cell. A paraphrased public-policy question should skip answer generation but pay for semantic lookup; a live incident with the same long prefix may reuse input work while still generating a fresh response. Same word "cache," different work and different correctness boundary.