Personalize this lesson
Adapt explanations and teaching visuals to your background and preferred voice.
When the incident-evidence assistant answers "How long are revoked API keys retained?", developers will ask it a dozen ways. Reusing one verified answer can be cheaper and faster. Reusing it for "What is incident INC-48192's current status?", after a policy update, or across a tenant boundary could serve a wrong or unauthorized answer.
The previous chapter recorded the intended combination of model, prompt, policy, corpus, and serving image. Reuse must preserve the approved contract and verify its dependencies. We'll work with a public API-key policy assistant and simulate shadow decisions, where a proposed hit doesn't replace the fresh answer.
This is a CPU-only lab. The policy, approval records, vectors, acceptance labels, costs, and durations are supplied fixtures. No model, Redis instance, vector index, review service, or deployment runs here. A local dictionary stands in for a protected, revocable approval registry; the code doesn't authenticate reviewers or establish that their labels are correct.
An answer is reusable only inside its contract
A response cache stores complete outputs. Its scope records the conditions under which an output may be reused. Our fictional policy says revoked API keys remain visible in audit logs for 30 days. That answer belongs to the policy-answerer release and its policy corpus, not to every service that happens to mention API keys.
| Request | Can an answer be reused? | Why |
|---|---|---|
| "How long are revoked API keys retained?" | Candidate | Public docs-policy answer can remain stable within one policy release. |
| "How long do revoked keys stay in audit logs?" | Candidate | Paraphrase of the same public docs-policy question, after evaluation. |
| "What is incident INC-48192's current status?" | No | Answer depends on live incident state. |
| "Revoke API key KEY-48192." | No | The request asks for a side effect, not a reusable answer. |
Hold two boundaries apart before continuing. If the live incident question lands near the retention question in vector space, it still bypasses. If the policy corpus moves, an exact repeat still misses under the new scope. Similarity can nominate a candidate; the contract decides whether it may serve.
Multi-tenant security boundaries can't rely on prompt strings alone. If tenant A asks about database retention rules and receives an answer reflecting tenant A's confidential configuration, tenant B asking the exact same question must never receive tenant A's cached response. An unfiltered shared index can leak that response. Compound scope is one part of preventing reuse across authorization contexts:
| Field | Why it matters |
|---|---|
release_id | Identifies the intended model, prompt, policy, and runtime combination; verify the loaded dependencies too. |
corpus_version | Rejects records declared under another corpus version once readers use the new scope. |
tenant_id and access_scope | Restricts lookup and reuse to the declared context; current resource permissions still need enforcement. |
locale and response_schema | Rejects records declared for another language or output contract; admission must validate the actual content. |
Scope equality isn't authorization. Authenticate and check current permissions on every request, including exact-cache hits. A role name alone can't represent document-specific access or a permission revoked moments ago. This lab restricts reuse to one public policy class. For private answers, also enforce the current resource permissions and version their dependencies; if those checks aren't available, don't serve cached content.[1]
Conversation history, account state, tool results, and generation settings can also change the answer. Include any permitted dependencies in identity, or exclude those requests from reuse. Here, a follow-up such as "What about mine?" bypasses because it depends on conversation context.
Semantic lookup has a second version boundary. The embedding model, vector preprocessing, and search/index policy determine which candidates appear and how they score. Record them in the cache policy. If the effective encoder or preprocessing changes, re-embed stored questions; if any retrieval component changes, revalidate before serving hits. A threshold is the minimum accepted similarity score, not a universal constant. Our compact index-version label stands in for preprocessing and search settings; a production manifest must resolve those effective settings too.
The cells form one Python session. Start with an answer, an expiry time, and a supplied approval digest bound to the exact answer content and scope. The fixed clock uses seconds so expiry checks are reproducible. Hashes detect a mismatch; a real registry needs access control, review provenance, and revocation. These fictional 30-day and later rate-limit answers describe our fixture, not a vendor's policy.
from dataclasses import asdict, dataclass, fields, replace
from hashlib import sha256
import json
import math
def nonempty_text(value: object) -> bool:
return type(value) is str and bool(value.strip())
def finite_number(value: object) -> bool:
if type(value) not in (int, float):
return False
try:
return math.isfinite(value)
except OverflowError:
return False
@dataclass(frozen=True)
class ReleaseScope:
release_id: str
corpus_version: str
tenant_id: str
access_scope: str
locale: str
response_schema: str
def __post_init__(self) -> None:
if any(not nonempty_text(getattr(self, field.name)) for field in fields(self)):
raise ValueError("scope fields must be nonempty strings")
@dataclass(frozen=True)
class Request:
text: str
tenant_id: str = "docs-public"
access_scope: str = "public-policy"
locale: str = "en-US"
response_schema: str = "cited-answer-v2"
requires_live_data: bool = False
writes_state: bool = False
route_id: str = "public-policy"
authorized_now: bool = True
context_dependent: bool = False
def __post_init__(self) -> None:
if any(not nonempty_text(getattr(self, name)) for name in (
"text", "tenant_id", "access_scope", "locale", "response_schema", "route_id"
)):
raise ValueError("request fields must be nonempty strings")
if any(type(getattr(self, name)) is not bool for name in (
"requires_live_data", "writes_state", "authorized_now", "context_dependent"
)):
raise ValueError("request flags must be booleans")
@dataclass(frozen=True)
class CachedAnswer:
answer_id: str
source_query: str
response: str
scope: ReleaseScope
response_class: str
admission_evidence_id: str
expires_at: int
def __post_init__(self) -> None:
if any(not nonempty_text(getattr(self, name)) for name in (
"answer_id", "source_query", "response", "response_class"
)) or type(self.scope) is not ReleaseScope or type(self.admission_evidence_id) is not str:
raise ValueError("invalid cached answer fields")
if type(self.expires_at) is not int or self.expires_at < 0:
raise ValueError("expiry must be a nonnegative integer")
stable_scope = ReleaseScope(
release_id="policy-answerer@sha256:df2d4fe7b0c5",
corpus_version="api-key-policy-2026-04",
tenant_id="docs-public",
access_scope="public-policy",
locale="en-US",
response_schema="cited-answer-v2",
)
seed_answer = CachedAnswer(
answer_id="ans_api_key_retention_30d",
source_query="How long are revoked API keys retained?",
response="Revoked API keys remain visible in audit logs for 30 days.",
scope=stable_scope,
response_class="public-policy",
admission_evidence_id="eval-public-policy-api-keys-v4",
expires_at=4_600,
)
NOW = 1_000 # deterministic clock, seconds
def answer_digest(answer: CachedAnswer) -> str:
payload = asdict(answer)
payload.pop("admission_evidence_id")
return sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()
# Supplied approval fixture. Production writes belong to a protected review process.
approved_answer_digests = {
seed_answer.admission_evidence_id: answer_digest(seed_answer),
}
print(f"release_id={stable_scope.release_id}")
print(f"corpus_version={stable_scope.corpus_version}")
print(f"seed_answer={seed_answer.answer_id}")release_id=policy-answerer@sha256:df2d4fe7b0c5
corpus_version=api-key-policy-2026-04
seed_answer=ans_api_key_retention_30dAn ordinary key-value response cache can retrieve an exact repeat by key. This isn't a Transformer's attention KV cache. Preserve the request bytes unless a specific normalization has been proved safe: lowercasing code identifiers or collapsing whitespace in code can merge requests with different answers. We'll hash the unchanged text plus the full scope.
The scope fields must come from application-owned route policy and authenticated context, not from raw user text or a model's guess. The same rule applies to eligibility flags: requires_live_data and writes_state must come from a route or intent-class policy map, never from free-form model labels alone. Unknown routes and unknown response classes should bypass answer reuse. The lab checks that the selected scope agrees with the request before deriving a key.
These dataclasses represent middleware outputs, not untrusted request JSON. Their constructors reject unknown or malformed fields, but don't authenticate them. Defaults keep the fixture short; a real wrapper must obtain authenticated context and the current authorization result explicitly. For example, authorized_now=1 or requires_live_data=None must never act as a valid flag. A public-policy route must reject or bypass messages outside its evaluated class rather than trusting a user-supplied route label. The response string is simplified text; declaring cited-answer-v2 doesn't validate a real structured response or its citations. Those checks belong in admission.
The next cell checks identity only. A real exact-cache read must also pass the eligibility, current-permission, admission, and expiry gates we'll add below. Expect an exact repeat to share its key, a paraphrase and new release to have different keys, and a cross-tenant request to raise a scope error.
def request_matches_scope(request: Request, scope: ReleaseScope) -> bool:
return (
request.tenant_id == scope.tenant_id
and request.access_scope == scope.access_scope
and request.locale == scope.locale
and request.response_schema == scope.response_schema
)
def exact_key(scope: ReleaseScope, request: Request) -> str:
if not request_matches_scope(request, scope):
raise ValueError("request does not match cache scope")
payload = {
"scope": asdict(scope),
"text": request.text,
"route_id": request.route_id,
}
encoded = json.dumps(payload, sort_keys=True).encode("utf-8")
return sha256(encoded).hexdigest()
same_words = Request("How long are revoked API keys retained?")
paraphrase = Request("How long do revoked keys stay in audit logs?")
cross_tenant = replace(same_words, tenant_id="internal-admin")
updated_scope = replace(
stable_scope,
release_id="policy-answerer@sha256:new-policy",
corpus_version="api-key-policy-2026-05",
)
seed_key = exact_key(stable_scope, Request(seed_answer.source_query))
print(f"exact_repeat_hit={exact_key(stable_scope, same_words) == seed_key}")
print(f"paraphrase_exact_hit={exact_key(stable_scope, paraphrase) == seed_key}")
print(f"updated_policy_hit={exact_key(updated_scope, same_words) == seed_key}")
try:
exact_key(stable_scope, cross_tenant)
except ValueError as error:
print(f"cross_tenant_rejected={error}")exact_repeat_hit=True
paraphrase_exact_hit=False
updated_policy_hit=False
cross_tenant_rejected=request does not match cache scopeThe key demonstration shows different identities for a paraphrase and a changed policy scope; it hasn't read a cache yet. Exact matching avoids merging different request text, but an identical repeat can still retrieve a wrong, stale, or unauthorized answer if its gates are incomplete. On a miss, the system might use semantic lookup or the authorized fresh path. That path could be a model, retrieval plus a model, or a deterministic policy lookup.
Semantic caching bridges that syntactic gap, but it introduces geometric risk. It must resolve reworded questions without weakening release or correctness boundaries.
Similarity retrieves a candidate, not a truth
A semantic cache embeds an incoming question, searches stored question embeddings, and proposes a nearby saved answer. Lookup can use exact vector search or approximate nearest-neighbor (ANN) search. GPTCache describes candidate lookup before invoking an LLM.[2] Sentence-BERT demonstrated efficient sentence-embedding comparison with cosine similarity on semantic matching tasks; that result doesn't certify cached-answer equivalence.[3]
Take a stored question vector (1, 0) and a candidate (0.99, 0.04). Their dot product is 0.99; their lengths are 1 and about 0.9908. Dividing gives 0.9992, a very close direction. A different question (0.94, 0.10) still scores about 0.9944.
For two nonzero vectors and , cosine similarity is:
The numerator measures their aligned components. Dividing by both lengths makes the result compare direction rather than vector magnitude. A high score says two encoded questions are close under this embedding model. It does not say their answers are interchangeable.
Check the API's units before copying a threshold. This lab uses cosine similarity, where larger is closer. RedisVL uses Redis COSINE distance from 0 to 2, where smaller is closer. For distance defined as , similarity at least 0.995 corresponds to distance at most 0.005, not distance 0.995.[4]
The tiny vectors below are an instructional fixture, not scores from a commercial embedding model. They let us see the failure mode without downloading a model: a restore-key exception can sit near a general retention question while still needing a different answer. Restore-key is still a public-policy read, so a live/write gate won't save you.
Predict the scores before running the cell. The paraphrase should be nearest, the restore-key exception should look dangerously close, and the live incident should be far away. Which of those predictions would be enough to serve an answer? None by itself.
fixture_vectors = {
seed_answer.source_query: (1.00, 0.00, 0.00),
"How long do revoked keys stay in audit logs?": (0.99, 0.04, 0.00),
"Can I restore a revoked API key?": (0.94, 0.10, 0.00),
"What is incident INC-48192's current status?": (0.00, 0.05, 1.00),
}
def cosine(left: tuple[float, ...], right: tuple[float, ...]) -> float:
if not left or len(left) != len(right):
raise ValueError("vectors must have the same nonzero dimension")
if not all(finite_number(value) for value in (*left, *right)):
raise ValueError("vector coordinates must be finite numbers")
left_scale, right_scale = max(map(abs, left)), max(map(abs, right))
if left_scale == 0 or right_scale == 0:
raise ValueError("cosine is undefined for a zero vector")
# Scale before normalization so large finite coordinates don't overflow.
left_scaled = tuple(value / left_scale for value in left)
right_scaled = tuple(value / right_scale for value in right)
left_norm, right_norm = math.hypot(*left_scaled), math.hypot(*right_scaled)
score = math.fsum((a / left_norm) * (b / right_norm)
for a, b in zip(left_scaled, right_scaled))
return max(-1.0, min(1.0, score))
seed_vector = fixture_vectors[seed_answer.source_query]
for question in [
"How long do revoked keys stay in audit logs?",
"Can I restore a revoked API key?",
"What is incident INC-48192's current status?",
]:
score = cosine(seed_vector, fixture_vectors[question])
print(f"{question} | score={score:.3f}")How long do revoked keys stay in audit logs? | score=0.999
Can I restore a revoked API key? | score=0.994
What is incident INC-48192's current status? | score=0.000
💡 Key insight: Cosine similarity ranks nearby questions. It doesn't certify that two answers are interchangeable under the current release, corpus, and access scope.
At the initial 0.990 cutoff, the live incident vector is too far away while the restore-key exception clears it. Our current route map excludes live state and writes, but doesn't distinguish restoration from retention within public-policy reads. A narrower evaluated class could make that distinction before lookup.
Eligibility rules run before the score threshold
The assistant shouldn't cache live incident state or write actions at any threshold. Restore-key is different: it's still a public-policy read, so eligibility won't catch it. Even for an eligible question, a cached answer must be from the same release scope, and the serving index must match the embedding identity that calibrated the threshold.
The request gate runs before either exact lookup or embedding. Tenant and permission filters must restrict vector search itself, not merely filter a globally retrieved answer afterward. Recheck the returned record before serving it. A cache failure or timeout should fall back to the normal authorized fresh path, not to an unfiltered lookup.
For example, RedisVL exposes filter_expression for cache reads and filters for stored entries. Its default read filter is None, which searches the whole cache. A scope dictionary used only in logs doesn't isolate retrieval.[4]
The procedure below takes one retrieved candidate so we can test its gates without a database. The initial 0.990 threshold is deliberately unsafe and remains shadow-only. Predict which eligible question it wrongly accepts. Then check that a revoked permission, expired record, unknown route, or NaN score still fails closed.
@dataclass(frozen=True)
class CachePolicy:
embedding_model_id: str
embedding_index_version: str
selected_threshold: float
def __post_init__(self) -> None:
if not all(nonempty_text(value) for value in (
self.embedding_model_id, self.embedding_index_version
)) or not finite_number(self.selected_threshold) or not -1 <= self.selected_threshold <= 1:
raise ValueError("need declared retrieval versions and a finite similarity threshold")
# Initial shadow-only candidate. Calibration below will replace it.
ACTIVE_CACHE_POLICY = CachePolicy(
embedding_model_id="docs-query-embedder-v3",
embedding_index_version="api-key-policy-index-2026-04-v2",
selected_threshold=0.990,
)
# Route or intent-class map owns eligibility. Do not trust free-form model labels alone.
ROUTE_ELIGIBILITY = {
"public-policy": {"requires_live_data": False, "writes_state": False},
"incident-state": {"requires_live_data": True, "writes_state": False},
"key-admin": {"requires_live_data": False, "writes_state": True},
}
def request_from_route(text: str, route_id: str) -> Request:
route = ROUTE_ELIGIBILITY.get(route_id)
if route is None:
return Request(text, route_id=route_id)
return Request(text, route_id=route_id, **route)
def request_gate(request: Request, active: ReleaseScope) -> str | None:
if type(request) is not Request or type(active) is not ReleaseScope:
return "DENY_INVALID_REQUEST"
if not request.authorized_now:
return "DENY_PERMISSION"
route = ROUTE_ELIGIBILITY.get(request.route_id)
if route is None:
return "BYPASS_UNKNOWN_ROUTE"
if route["requires_live_data"] or route["writes_state"]:
return "BYPASS_DYNAMIC_OR_WRITE"
if request.requires_live_data or request.writes_state or request.context_dependent:
return "BYPASS_CONTEXT_DEPENDENT"
if not request_matches_scope(request, active):
return "MISS_SCOPE_CHANGED"
return None
def record_is_admitted(record: CachedAnswer) -> bool:
return (
type(record) is CachedAnswer
and record.response_class == "public-policy"
and approved_answer_digests.get(record.admission_evidence_id)
== answer_digest(record)
)
def record_gate(record: CachedAnswer, active: ReleaseScope, now: int) -> str | None:
if type(record) is not CachedAnswer:
return "BYPASS_UNVALIDATED_RECORD"
if type(now) is not int or now < 0:
return "MISS_INVALID_CLOCK"
if record.scope != active:
return "MISS_SCOPE_CHANGED"
if now >= record.expires_at:
return "MISS_EXPIRED"
if not record_is_admitted(record):
return "BYPASS_UNVALIDATED_RECORD"
return None
def decide_candidate(
request: Request,
record: CachedAnswer,
active: ReleaseScope,
score: float,
cache_policy: CachePolicy | None = None,
now: int = NOW,
) -> str:
rejection = request_gate(request, active)
if rejection:
return rejection
cache_policy = ACTIVE_CACHE_POLICY if cache_policy is None else cache_policy
if type(cache_policy) is not CachePolicy or cache_policy != ACTIVE_CACHE_POLICY:
return "MISS_EMBEDDING_VERSION"
rejection = record_gate(record, active, now)
if rejection:
return rejection
if not finite_number(score) or not -1 <= score <= 1:
return "MISS_INVALID_SCORE"
if score < cache_policy.selected_threshold:
return "MISS_BELOW_THRESHOLD"
return "SEMANTIC_HIT"
policy_paraphrase = request_from_route(
"How long do revoked keys stay in audit logs?",
"public-policy",
)
live_incident = request_from_route(
"What is incident INC-48192's current status?",
"incident-state",
)
revoke_action = request_from_route(
"Revoke API key KEY-48192.",
"key-admin",
)
stale_index_policy = replace(
ACTIVE_CACHE_POLICY,
embedding_index_version="api-key-policy-index-2026-03-v1",
)
policy_score = cosine(seed_vector, fixture_vectors[policy_paraphrase.text])
restore_request = request_from_route("Can I restore a revoked API key?", "public-policy")
restore_score = cosine(seed_vector, fixture_vectors[restore_request.text])
for name, request, score, kwargs in [
("paraphrase", policy_paraphrase, policy_score, {}),
("restore_wrong_hit", restore_request, restore_score, {}),
("live", live_incident, 1.0, {}),
("write", revoke_action, 1.0, {}),
("revoked_permission", replace(policy_paraphrase, authorized_now=False), 1.0, {}),
("unknown_route", request_from_route("unknown", "unregistered"), 1.0, {}),
("stale_index", policy_paraphrase, policy_score, {"cache_policy": stale_index_policy}),
("expired", policy_paraphrase, policy_score, {"now": seed_answer.expires_at}),
("invalid_score", policy_paraphrase, float("nan"), {}),
]:
print(f"{name}={decide_candidate(request, seed_answer, stable_scope, score, **kwargs)}")paraphrase=SEMANTIC_HIT
restore_wrong_hit=SEMANTIC_HIT
live=BYPASS_DYNAMIC_OR_WRITE
write=BYPASS_DYNAMIC_OR_WRITE
revoked_permission=DENY_PERMISSION
unknown_route=BYPASS_UNKNOWN_ROUTE
stale_index=MISS_EMBEDDING_VERSION
expired=MISS_EXPIRED
invalid_score=MISS_INVALID_SCOREThe restore_wrong_hit line is the remaining defect: this threshold proposes the wrong answer even though the structural gates pass. We'll use supplied pair labels to choose a different threshold. They don't become an online oracle: a real new exception could still pass the chosen cutoff. DENY_PERMISSION and DENY_INVALID_REQUEST end the request; bypasses and misses may use the normal fresh path only if that path is authorized. Declared retrieval labels also don't prove which embedder or index actually ran.
A live incident-status request has a similarity score of 1.00 against a saved response. Can it be a semantic answer-cache hit?
Answer
No. Its answer depends on live operational state. Eligibility is checked before similarity, so the request bypasses response reuse regardless of score.
Make changed dependencies ineligible before relying on eviction
A time-to-live (TTL) bounds entry age under its clock and expiration semantics. It doesn't discover a policy change five minutes after admission. A 24-hour TTL alone could allow stale reuse for the remaining lifetime; it doesn't guarantee that every hit will be wrong or non-compliant. Some products explicitly tolerate bounded staleness. This policy assistant requires readers to use the approved policy version.
Our policy uses versioned dependencies together with expiration:
- Publish an approved corpus version: A webhook can initiate rebuilding and evaluation; it shouldn't merely increment a name while leaving old evidence loaded. Activate the approved corpus and cache scope together. Readers using the new scope reject old entries without waiting for deletion. Delayed events, old router caches, and in-flight requests still need coordination or an explicit cutover policy.
- Publish evaluated release changes: Model, prompt, or effective configuration changes get a new release contract. Use that contract in cache reads and writes. Deprecation notices also require checking fresh-path availability; they don't update application scope automatically.
- Enforce expiry and revoke bad approvals: Check time on reads even if storage cleanup runs later. TTL limits age and storage lifetime here, while version checks handle published dependency changes and revocation removes known-bad answers.
Before running the invalidation example, predict both decisions: a reader using the old scope still hits, while one using the new scope misses. The local comparison is immediate; a distributed rollout needs every reader to observe the intended scope. Storage eviction can happen later.
policy_update = replace(
stable_scope,
release_id="policy-answerer@sha256:7a12policy",
corpus_version="api-key-policy-2026-05",
)
same_question = Request(seed_answer.source_query)
old_release_decision = decide_candidate(
same_question, seed_answer, stable_scope, 1.00
)
new_release_decision = decide_candidate(
same_question, seed_answer, policy_update, 1.00
)
print(f"old_release={old_release_decision}")
print(f"new_policy_release={new_release_decision}")
print(f"old_record_ineligible={new_release_decision == 'MISS_SCOPE_CHANGED'}")
exact_cache = {seed_key: seed_answer}
def read_exact(request: Request, active: ReleaseScope, now: int = NOW) -> str:
rejection = request_gate(request, active)
if rejection:
return rejection
record = exact_cache.get(exact_key(active, request))
if record is None:
return "EXACT_MISS"
return record_gate(record, active, now) or "EXACT_HIT"
print(f"exact_current={read_exact(same_question, stable_scope)}")
print(f"exact_expired={read_exact(same_question, stable_scope, seed_answer.expires_at)}")
print(f"exact_revoked={read_exact(replace(same_question, authorized_now=False), stable_scope)}")old_release=SEMANTIC_HIT
new_policy_release=MISS_SCOPE_CHANGED
old_record_ineligible=True
exact_current=EXACT_HIT
exact_expired=MISS_EXPIRED
exact_revoked=DENY_PERMISSIONCache identity should inherit the approved dependencies from serving. Version checks avoid depending on storage deletion, but don't propagate a cutover themselves. Enforce expiration on reads, coordinate scope changes, and retain a revocation path. These solve different freshness and admission problems.
Why should changing the model, prompt, evidence snapshot, or authorization policy invalidate cached answers even when query similarity stays high?
Answer
Similarity says the request resembles an old request; it doesn't prove the old answer was produced under the current behavior and access contract. Version those dependencies in cache eligibility.
Choose a threshold in shadow mode
Serving a semantic hit immediately turns a retrieval mistake into a user-visible wrong answer. Shadow mode runs the lookup decision in parallel with the live system but continues serving the normal fresh path. Reviewers or offline evaluation pipelines then label whether each proposed reuse would have been acceptable.
A threshold changes proposal volume and, for these labeled candidate pairs, precision and recall. Keep their denominators separate:
- Proposal rate: proposed hits divided by all requests in the stated window. An eligible-only denominator gives a different rate; name it explicitly.
- Hit precision: acceptable proposed reuses divided by all proposed reuses.
- Recall over labeled pairs: acceptable proposed reuses divided by all acceptable eligible pairs evaluated, including ones below threshold. This isn't recall over every possible answer in the database; our fixture supplies one candidate pair per request.
On the seven supplied probes:
0.990proposes five reuses, including the restore-key exception. Precision is 4/5 = 80%; recall is 4/4 = 100%. This remains a shadow proposal, so the wrong retention answer isn't sent to a user.0.995proposes all four valid paraphrases and rejects the exception. Observed precision and recall are both 100%; proposal rate is 4/7 = 57.1%, or 4/5 = 80% among eligible probes.0.999proposes one valid pair. Observed precision stays 100%, but recall is 1/4 = 25% and proposal rate is 1/7 = 14.3%.
Raising the cutoff reduces proposal volume for a fixed candidate set. Precision needn't improve monotonically: unacceptable pairs can score higher than acceptable ones.
Seven probes explain the mechanics; they can't establish 99% production precision on their own. Labels belong to each request-and-cached-answer pair, not just the request in isolation. The fresh model output isn't automatically the correct label either; compare against policy evidence.
Pause at the three candidate thresholds. The lowest threshold should admit the most proposals and the wrong exception; the strictest should reject valid paraphrases as well as the exception. The selected value needs to clear precision while preserving useful proposal volume, then earn confidence on a larger replay.
@dataclass(frozen=True)
class ShadowProbe:
name: str
score: float
eligible: bool
acceptable_reuse: bool
def __post_init__(self) -> None:
if not nonempty_text(self.name) or not finite_number(self.score) or not -1 <= self.score <= 1:
raise ValueError("invalid probe name or score")
if type(self.eligible) is not bool or type(self.acceptable_reuse) is not bool:
raise ValueError("probe labels must be booleans")
if self.acceptable_reuse and not self.eligible:
raise ValueError("acceptable pair must be eligible")
shadow_probes = [
ShadowProbe("key retention paraphrase", policy_score, True, True),
ShadowProbe("audit log retention wording", 0.998, True, True),
ShadowProbe("revoked key wording", 0.996, True, True),
ShadowProbe("policy FAQ reworded", 0.995, True, True),
ShadowProbe("restore-key exception", restore_score, True, False),
ShadowProbe("live incident state", 0.000, False, False),
ShadowProbe("revoke key action", 0.997, False, False),
]
def replay_at(threshold: float, probes: tuple[ShadowProbe, ...] = tuple(shadow_probes)) -> dict[str, float | int | None]:
if not finite_number(threshold) or not -1 <= threshold <= 1:
raise ValueError("threshold must be a finite similarity")
if any(type(probe) is not ShadowProbe for probe in probes) or len({probe.name for probe in probes}) != len(probes):
raise ValueError("need valid probes with unique IDs")
proposed = [
probe for probe in probes
if probe.eligible and probe.score >= threshold
]
accepted = [probe for probe in proposed if probe.acceptable_reuse]
eligible = sum(probe.eligible for probe in probes)
acceptable = sum(probe.acceptable_reuse for probe in probes)
precision = len(accepted) / len(proposed) if proposed else None
return {
"proposed": len(proposed),
"accepted": len(accepted),
"precision": precision,
"proposal_rate": len(proposed) / len(probes) if probes else None,
"eligible_proposal_rate": len(proposed) / eligible if eligible else None,
"recall": len(accepted) / acceptable if acceptable else None,
}
for threshold in [0.990, 0.995, 0.999]:
metrics = replay_at(threshold)
precision_label = f"{metrics['precision']:.1%}" if metrics["precision"] is not None else "undefined"
print(
f"threshold={threshold:.3f} "
f"proposed={metrics['proposed']} "
f"precision={precision_label} "
f"proposal_rate={metrics['proposal_rate']:.1%} "
f"recall={metrics['recall']:.1%}"
)
selected_threshold = 0.995
ACTIVE_CACHE_POLICY = replace(ACTIVE_CACHE_POLICY, selected_threshold=selected_threshold)
print(f"restore_after_calibration={decide_candidate(restore_request, seed_answer, stable_scope, restore_score)}")
def wilson_lower_bound(successes: int, trials: int, z: float = 1.96) -> float:
if type(successes) is not int or type(trials) is not int or not 0 <= successes <= trials:
raise ValueError("counts must satisfy 0 <= successes <= trials")
if not finite_number(z) or not 0 < z <= 10:
raise ValueError("z must be finite, positive, and within this lab's bound")
if trials == 0:
return 0.0
observed = successes / trials
inverse_n = 1 / trials
denominator = 1 + z**2 * inverse_n
center = observed + z**2 * inverse_n / 2
margin = z * math.sqrt(
observed * (1 - observed) * inverse_n + z**2 * inverse_n**2 / 4
)
return (center - margin) / denominator
# Synthetic full-window fixture at the selected threshold. In production,
# populate these counts from representative labeled shadow traffic.
full_shadow_requests = 10_000
full_shadow_proposed = 5_714
full_shadow_accepted = 5_714
full_shadow_precision = full_shadow_accepted / full_shadow_proposed
precision_lower_bound = wilson_lower_bound(
full_shadow_accepted,
full_shadow_proposed,
)
safe_hit_fraction = full_shadow_accepted / full_shadow_requests
print(
f"full_shadow proposed={full_shadow_proposed} "
f"precision={full_shadow_precision:.1%} "
f"precision_lower_bound={precision_lower_bound:.2%}"
)threshold=0.990 proposed=5 precision=80.0% proposal_rate=71.4% recall=100.0%
threshold=0.995 proposed=4 precision=100.0% proposal_rate=57.1% recall=100.0%
threshold=0.999 proposed=1 precision=100.0% proposal_rate=14.3% recall=25.0%
restore_after_calibration=MISS_BELOW_THRESHOLD
full_shadow proposed=5714 precision=100.0% precision_lower_bound=99.93%
The larger, synthetic fixture evaluates the frozen 0.995 policy on a separate window of 10,000 eligible requests that missed the exact cache. All 5,714 proposals are acceptable, so observed precision is 100%. Using z=1.96, the lower endpoint of the two-sided 95% Wilson interval is 99.93%, above the 99% gate. Four successes out of four would give only about 51.0%. The bound accounts for sample size under a binomial model, not every source of uncertainty.[5]
Use a held-out replay after threshold selection. Repeating four paraphrases thousands of times doesn't create thousands of independent quality checks. Sample across policy topics, tenants, time windows, and hard negatives; account for correlated requests when estimating uncertainty. An empty proposal set has undefined precision and a zero lower bound, not evidence of perfect quality.
Revalidate whenever the retrieval policy changes. A threshold belongs to its embedding model, preprocessing, index, candidate selection, and traffic distribution. Changes can alter score distributions; they needn't change every score. If the effective encoder or preprocessing changes, re-embed stored source questions. If an index/search change preserves the vectors, rebuilding every embedding may be unnecessary, but evaluate its returned pairs and cutoff before cutover. The admitted answer text can remain valid when only retrieval geometry changes.
Try changing the restore-key vector to (0.9999, 0.01, 0). It then scores about 0.99995, above every valid paraphrase. No similarity threshold can retain those paraphrases while rejecting that exception. Narrow the reusable question class, add a separately evaluated equivalence check, or disable semantic reuse for that class. Threshold tuning can't always solve the problem.
Why doesn't a high hit rate prove that semantic caching is helping?
Answer
A hit is only beneficial when the reused answer is acceptable under the active release and scope. A loose threshold can raise hit rate by serving nearby but incorrect answers.
A safe cache still has to pay for itself
Consider 10,000 requests that actually reach semantic lookup after eligibility and exact-cache checks. Generating every answer at USD 0.0040 would cost USD 40.00. If 5,714 safe hits skip generation, only 4,286 requests generate, costing USD 17.144. Lookup at USD 0.00008 on all 10,000 requests adds USD 0.80. Net savings are USD 22.056, or USD 22.06 rounded.
These are synthetic accounting inputs, not measured provider prices. The arithmetic assumes constant generation cost and no wrong hits in this window. Define:
- be requests that reach semantic lookup in the period.
- be fresh-generation cost per request in this simplified model.
- be semantic-lookup cost per request.
- be safe hits divided by those lookup requests.
If a hit skips fresh generation, expected period savings are:
For a real workload, sum the generation cost avoided on each actual hit, then subtract lookup, cache serving, writes, storage, and other incremental costs. Hits and misses can have different output lengths. Exclude bypasses from this denominator or include their zero lookup cost explicitly. False hits may avoid spending too, but they aren't useful savings; measure their rate and harm separately.
Use the actual alternative path for . If it already benefits from provider prefix caching, comparing answer reuse against fully uncached input charges overstates savings. In shadow mode, fresh generation still runs, so these savings are a counterfactual forecast; add shadow lookup to actual experiment spending. Measure realized avoidance after controlled activation.
Compute the break-even point before running the cell. With and , . The fixture's safe-hit fraction is 57.1%, so its savings should be positive, but that conclusion follows only after safe-hit precision has passed.
requests_per_day = full_shadow_requests
fresh_generation_usd = 0.0040 # synthetic fixed full-answer cost
semantic_lookup_usd = 0.00008 # synthetic embed + index lookup cost
without_cache = requests_per_day * fresh_generation_usd
with_cache = requests_per_day * (
semantic_lookup_usd + (1 - safe_hit_fraction) * fresh_generation_usd
)
savings = without_cache - with_cache
break_even_hit_fraction = semantic_lookup_usd / fresh_generation_usd
print(f"safe_hit_fraction={safe_hit_fraction:.1%}")
print(f"break_even_hit_fraction={break_even_hit_fraction:.1%}")
print(f"daily_savings_fixture_usd={savings:.2f}")
print(f"savings_positive={savings > 0}")safe_hit_fraction=57.1%
break_even_hit_fraction=2.0%
daily_savings_fixture_usd=22.06
savings_positive=TrueDon't infer latency from the cost ratio either. Suppose the fresh path takes 1,000 ms and semantic lookup adds 20 ms. In a fixed-duration simulation, 5,714 hits take 20 ms and 4,286 misses take 1,020 ms. Mean latency falls, but p95 rises from 1,000 to 1,020 ms. Never average route percentiles to estimate the combined p95.
The next cell computes the mean and nearest-rank p95 from all simulated requests. No network, model, or database is being benchmarked.
from statistics import mean
lookup_ms, generation_ms = 20, 1_000
latencies_ms = (
[lookup_ms] * full_shadow_accepted
+ [lookup_ms + generation_ms] * (full_shadow_requests - full_shadow_accepted)
)
p95_ms = sorted(latencies_ms)[math.ceil(0.95 * len(latencies_ms)) - 1]
print(f"fresh_mean_ms={generation_ms}")
print(f"cache_mean_ms={mean(latencies_ms):.1f}")
print(f"fresh_p95_ms={generation_ms}")
print(f"cache_p95_ms={p95_ms}")fresh_mean_ms=1000
cache_mean_ms=448.6
fresh_p95_ms=1000
cache_p95_ms=1020
Those savings only count if the reused answers were allowed into the index. A cheap wrong answer isn't a win.
Authorize writes before they become reusable
The read path now rejects records without approved admission evidence. The write path must enforce the same rule before adding a record to the servable index. A freshly generated answer isn't automatically safe to repeat across paraphrases.
Treat admission as write authorization. Keep unreviewed answers in a quarantine store, attach the evaluation artifact that approved a response class, and admit only records inside the tested release scope.
Predict the write decisions: missing evidence should quarantine, approved public-policy evidence should admit, and a live-incident response class should quarantine. A fresh answer earns reuse only after this gate.
def admission_decision(answer: CachedAnswer) -> str:
if type(answer) is not CachedAnswer:
return "QUARANTINE_INVALID_RECORD"
if answer.response_class != "public-policy":
return "QUARANTINE_RESPONSE_CLASS"
if answer.scope != stable_scope:
return "QUARANTINE_SCOPE_CHANGED"
if not record_is_admitted(answer):
return "QUARANTINE_EVIDENCE_MISMATCH"
if NOW >= answer.expires_at:
return "QUARANTINE_EXPIRED"
return "ADMIT_SERVABLE"
unreviewed_answer = replace(
seed_answer,
answer_id="ans_restore_key_review",
admission_evidence_id="",
)
validated_policy_answer = replace(
seed_answer,
answer_id="ans_rate_limit_public",
source_query="What is the default API rate limit?",
response="Default API keys allow 600 requests per minute unless the account policy says otherwise.",
admission_evidence_id="eval-public-policy-rate-limits-v1",
)
dynamic_incident_answer = replace(
seed_answer,
answer_id="ans_live_incident_status",
response_class="live-incident-status",
)
# Represents review of this exact record, not permission for arbitrary answers.
approved_answer_digests[validated_policy_answer.admission_evidence_id] = answer_digest(validated_policy_answer)
tampered_answer = replace(seed_answer, response="Revoked keys remain usable forever.")
for label, answer in [
("unreviewed", unreviewed_answer),
("validated_policy", validated_policy_answer),
("dynamic_incident", dynamic_incident_answer),
("copied_evidence", tampered_answer),
]:
print(f"{label}={admission_decision(answer)}")
# Local revocation demonstration. Restore the fixture approval for later cells.
approved_digest = approved_answer_digests.pop(seed_answer.admission_evidence_id)
print(f"revoked_exact_read={read_exact(same_question, stable_scope)}")
print(f"revoked_semantic_read={decide_candidate(policy_paraphrase, seed_answer, stable_scope, policy_score)}")
approved_answer_digests[seed_answer.admission_evidence_id] = approved_digestunreviewed=QUARANTINE_EVIDENCE_MISMATCH
validated_policy=ADMIT_SERVABLE
dynamic_incident=QUARANTINE_RESPONSE_CLASS
copied_evidence=QUARANTINE_EVIDENCE_MISMATCH
revoked_exact_read=BYPASS_UNVALIDATED_RECORD
revoked_semantic_read=BYPASS_UNVALIDATED_RECORDCopying an approved evidence ID onto changed text fails: approval binds the answer, source question, class, scope, and expiry. Removing its approval blocks both local read paths even though its TTL remains valid. Review can still be wrong, so retain a revocation path and audit reuse. Distributed readers must observe that revocation. Exact and semantic reads need permission, scope, admission, and expiry checks; semantic reads additionally need the calibrated threshold.
Promote only the narrow policy you tested
Don't turn on semantic caching for every route PolicyOps owns. Promote only the public-policy scope that passed shadow evidence. Account state, live incidents, and write actions still bypass.
The decision below checks quality, economics, scope, and a declared tail-latency budget. It derives the Wilson endpoint from integer review counts rather than accepting a caller's claimed bound, and checks the report's retrieval identity and age. Compare the supplied passing fixture with low precision, negative savings, and stale scope. Each failure stays shadow-only.
The function returns a symbolic verdict, not a deployment command. It assumes trusted ingestion verified held-out case membership, review coverage, evidence artifacts, costs, and durations. A timestamp or report ID doesn't prove those checks ran, and this synthetic report provides no production-quality evidence. A live rollout also needs a tested disable switch and ongoing review.
@dataclass(frozen=True)
class CachePromotionGate:
minimum_precision_lower_bound: float
minimum_daily_savings_usd: float
required_scope: ReleaseScope
maximum_p95_latency_ms: int
gate = CachePromotionGate(
minimum_precision_lower_bound=0.99,
minimum_daily_savings_usd=5.00,
required_scope=stable_scope,
maximum_p95_latency_ms=1_050,
)
@dataclass(frozen=True)
class CacheEvidence:
report_id: str
scope: ReleaseScope
retrieval_policy: CachePolicy
lookup_requests: int
proposed: int
reviewed_proposed: int
acceptable_proposed: int
daily_savings_usd: float
p95_latency_ms: int
completed_at: int
def promotion_decision(evidence: CacheEvidence, now: int = NOW) -> str:
hold = "KEEP_SHADOW_ONLY"
if type(evidence) is not CacheEvidence or not nonempty_text(evidence.report_id):
return hold
if evidence.scope != gate.required_scope or evidence.retrieval_policy != ACTIVE_CACHE_POLICY:
return hold
counts = (evidence.lookup_requests, evidence.proposed,
evidence.reviewed_proposed, evidence.acceptable_proposed)
if any(type(count) is not int or count < 0 for count in counts):
return hold
if not (0 < evidence.proposed <= evidence.lookup_requests
and evidence.reviewed_proposed == evidence.proposed
and evidence.acceptable_proposed <= evidence.reviewed_proposed):
return hold
if type(now) is not int or type(evidence.completed_at) is not int or not 0 <= evidence.completed_at <= now <= evidence.completed_at + 3_600:
return hold
if not finite_number(evidence.daily_savings_usd) or not finite_number(evidence.p95_latency_ms):
return hold
lower = wilson_lower_bound(evidence.acceptable_proposed, evidence.proposed)
passed = (lower >= gate.minimum_precision_lower_bound
and evidence.daily_savings_usd >= gate.minimum_daily_savings_usd
and 0 <= evidence.p95_latency_ms <= gate.maximum_p95_latency_ms)
return "PROMOTE_PUBLIC_POLICY_SEMANTIC_CACHE" if passed else hold
supplied_evidence = CacheEvidence(
"synthetic-cache-window-v1", stable_scope, ACTIVE_CACHE_POLICY,
full_shadow_requests, full_shadow_proposed, full_shadow_proposed,
full_shadow_accepted, savings, p95_ms, NOW,
)
for name, evidence in [
("evaluated", supplied_evidence),
("low_precision", replace(supplied_evidence, acceptable_proposed=5_600)),
("cost_loss", replace(supplied_evidence, daily_savings_usd=-0.40)),
("stale_scope", replace(supplied_evidence, scope=policy_update)),
]:
print(f"{name}={promotion_decision(evidence)}")evaluated=PROMOTE_PUBLIC_POLICY_SEMANTIC_CACHE
low_precision=KEEP_SHADOW_ONLY
cost_loss=KEEP_SHADOW_ONLY
stale_scope=KEEP_SHADOW_ONLYThe next chapter can price this hypothetical plan: acceptable public-policy hits skip generation, while lookup and every remaining fresh response still incur work. This exercise hasn't activated a cache or measured realized savings.
Record why each request hit or bypassed
If the cache is activated, a trace should explain which release generated the stored answer, which retrieval policy proposed reuse, and why a neighboring request bypassed. Log request IDs rather than raw prompts by default, and protect logs with access and retention controls. The following local dictionaries don't export spans or establish actual serving identity. A bypass before lookup has no candidate score or admission evidence to report.
def trace_decision(request_id: str, request: Request, score: float) -> dict:
if not nonempty_text(request_id) or type(request) is not Request:
raise ValueError("need an owned request ID and validated request metadata")
prelookup = request_gate(request, stable_scope)
cache_decision = prelookup or decide_candidate(request, seed_answer, stable_scope, score)
return {
"request_id": request_id,
"release_id": stable_scope.release_id,
"corpus_version": stable_scope.corpus_version,
"tenant_id": request.tenant_id,
"access_scope": request.access_scope,
"cache_policy": "public-policy-semantic-v1",
"embedding_model_id": ACTIVE_CACHE_POLICY.embedding_model_id,
"embedding_index_version": ACTIVE_CACHE_POLICY.embedding_index_version,
"answer_id": seed_answer.answer_id if cache_decision == "SEMANTIC_HIT" else "",
"admission_evidence_id": None if prelookup else seed_answer.admission_evidence_id,
"decision": cache_decision,
"score": None if prelookup or not finite_number(score) else score,
"threshold": ACTIVE_CACHE_POLICY.selected_threshold,
}
hit_trace = trace_decision("req-policy-1", policy_paraphrase, policy_score)
bypass_trace = trace_decision("req-live-1", live_incident, 1.00)
print(f"hit_decision={hit_trace['decision']} answer_id={hit_trace['answer_id']}")
print(f"bypass_decision={bypass_trace['decision']}")
print(f"traced_release={hit_trace['release_id'] == stable_scope.release_id}")
print(f"traced_scope={hit_trace['corpus_version'] == stable_scope.corpus_version and hit_trace['access_scope'] == stable_scope.access_scope}")
print(f"traced_index={hit_trace['embedding_index_version'] == ACTIVE_CACHE_POLICY.embedding_index_version}")
print(f"traced_admission={hit_trace['admission_evidence_id'] == seed_answer.admission_evidence_id}")
print(f"traced_model={hit_trace['embedding_model_id'] == ACTIVE_CACHE_POLICY.embedding_model_id}")
print(f"bypass_has_score={bypass_trace['score'] is not None}")hit_decision=SEMANTIC_HIT answer_id=ans_api_key_retention_30d
bypass_decision=BYPASS_DYNAMIC_OR_WRITE
traced_release=True
traced_scope=True
traced_index=True
traced_admission=True
traced_model=True
bypass_has_score=FalseWatch production for accepted-hit review failures, user retries after cache hits, scope bypass volume, p95 latency, and realized saved generation. An incident has to disable this cache policy pointer without changing the production release that generates fresh responses.
Those signals also point to different repairs. A wrong answer after a valid hit sends you to admission, scope, or threshold evidence; rising bypass volume sends you to route policy or version churn; savings below expectation sends you to lookup cost, miss rate, or traffic mix. Keep the decision trace that lets an operator tell those cases apart.
Tiered caching hierarchy and provider prompt caching
One possible architecture checks exact answers first, semantic candidates second, and reusable model-prefix work on the fresh path. These are application choices, not four mandatory cache levels or a guaranteed latency ordering.
- Exact response lookup: After authorization and eligibility, hash the unchanged request text and compound scope. A valid admitted hit skips embedding lookup and answer generation. Network, permission, and record checks still cost time; this lesson provides no measured sub-millisecond claim.
- Semantic response lookup: On an eligible exact miss, embed and search within the permitted scope. Serve a candidate only if record, version, expiry, and calibrated reuse gates pass. Lookup latency depends on the encoder, index, load, and infrastructure; an answer hit still pays that lookup cost.
- Prefix reuse during fresh inference: A serving engine such as vLLM can reuse KV blocks for a matching prefix with compatible identities and inputs.[6] OpenAI's current guide requires a matching rendered prefix and model-specific cache eligibility, settings, and breakpoints. New input still needs processing and a new response is generated. Cached-input and cache-write billing vary by provider/model; self-hosted reuse doesn't automatically change an API rate card.[7]
- Uncached inference work: Process any input not covered by prefix reuse and generate the response. A prefix miss doesn't create another model layer; it's more work in the same inference path. Other optimizations can still affect cost and latency.
Chunked prefill is a scheduling technique: vLLM divides large prefills into chunks and batches them with decode work. It isn't prefix caching, though the techniques can coexist.[8] A valid prefix hit reuses computation, not an old answer or a fresh-state check. Fetch current incident state when the task requires it; cached or uncached inference over stale supplied evidence can both be wrong. Prefix-cache isolation also needs attention in shared systems; vLLM documents per-request cache salts to restrict reuse to a trust group.[6]
| Tier | Layer | Matching mechanism | Result on hit | Main operational risk |
|---|---|---|---|---|
| L1 | Exact response lookup | Unchanged text plus full scope | Returns an admitted unexpired answer, skips generation | Wrong record, stale dependencies, or missing authorization |
| L2 | Semantic response lookup | Scoped candidate search plus reuse gates | Returns a stored answer, skips generation | Wrong reuse on a nearby exception |
| L3 | Prefix reuse in inference | Eligible matching prefix under model/cache contract | Reuses KV work, still generates output | Missing isolation or relying on stale supplied context |
| Fresh work | Inference | Input not reused plus output generation | Produces a new completion | Generation isn't evidence of correctness or freshness |
The distinction determines how you evaluate each tier: semantic answer caching requires labeled precision gates; prompt-prefix caching requires token accounting and latency benchmarking. The next chapter expands that token economics.
Predict the final routes before running the last cell. A paraphrased policy question is a hypothetical L2 hit; a live incident bypasses answer reuse but can still receive a provider prefix hit and generate new output. The last cell doesn't run L1, call a provider, or fetch live state. Its prefix flag is a supplied result, not a prediction that a shared prefix guarantees a hit.
def simulate_route(request: Request, score: float, provider_prefix_hit: bool) -> str:
if type(provider_prefix_hit) is not bool:
raise ValueError("prefix result must be a boolean fixture")
decision = decide_candidate(request, seed_answer, stable_scope, score)
if decision.startswith("DENY_"):
return "DENIED; generation_calls=0"
if decision == "SEMANTIC_HIT":
return "STORED_ANSWER; generation_calls=0"
# The prefix flag is a fixture for a provider-reported result, not a lookup.
return f"FRESH_ANSWER; generation_calls=1; prefix_hit={provider_prefix_hit}"
print(simulate_route(policy_paraphrase, policy_score, False))
print(simulate_route(live_incident, 1.0, True))
print(simulate_route(replace(policy_paraphrase, authorized_now=False), 1.0, True))STORED_ANSWER; generation_calls=0
FRESH_ANSWER; generation_calls=1; prefix_hit=True
DENIED; generation_calls=0