LLM-Powered Search Engine

CodeAtlas has a search failure that looks like a win. A developer asks which deprecation policy applies to parser-v2 SDK migration for service accounts and the large language model (LLM) returns a fluent answer based on an old policy or workspace-b's private roadmap. Search was fast; it still failed the user.

Multi-tenant serving made tenant identity, cost, and visibility part of every request. Search carries those boundaries into evidence: a response can be fast and fluent yet still be wrong if it uses an old policy, cites a private roadmap, or merges documents that apply to different workspaces.

You're the Staff AI Engineer at CodeAtlas. Developers ask, "can service accounts rotate OAuth tokens every 90 days?" They also ask, "what happens if parser-v2 deprecation affects our SDK migration?" For each query, the product must find allowed API docs, release notes, and workspace policies, then write a cited answer without letting private roadmaps cross a tenant boundary.

Before looking at components, predict the hard case. If a private roadmap is semantically closer than the public deprecation policy, which candidate is allowed to influence ranking? If two allowed documents disagree, what should the user see while the system checks their citations? Keep those answers in mind; each later stage owns one part of that contract.

If embeddings or chunking feel rusty: embeddings turn sentences into dense vectors so similar meanings sit close together. Chunking splits long documents into short passages so retrieval can select a useful span. You'll reuse both, plus the hybrid retriever from Hybrid Search, inside an interactive latency budget.

What makes LLM-powered search different from a chatbot over documents?

Answer

It must retrieve fresh, permission-safe evidence, rank it, synthesize only supported claims, attach citations, and stream an answer within an interactive search latency budget.

The parser-v2 query isn't a list of links

The old CodeAtlas product returned ranked hits for "parser-v2 deprecation." A developer still had to open the policy page, the service-account exception, and the SDK migration note, then decide which sentence governed their case.

The new product must retrieve those documents, remove private-roadmap from workspace-a's evidence, and show how each answer sentence is supported. That's Retrieval-Augmented Generation (RAG)^{[1]Reference 1Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.https://arxiv.org/abs/2005.11401} in product terms: generation reads retrieved evidence instead of treating model memory as the source of truth.

Once synthesis enters the product, citation binding, abstention, and authorization failures enter with it. The pipeline therefore retrieves with keyword and embedding signals, reranks candidates, applies visibility and freshness policy, and only then writes a short answer whose claims can be checked.

What CodeAtlas used to return vs what it returns now

---

What does the LLM add after traditional retrieval?

Answer

The LLM performs synthesis: it reads retrieved evidence, resolves conflicts, explains the answer in user language, and binds claims to citations. It should not replace retrieval or source verification.

What the parser-v2 answer has to hit

These are example SLOs for CodeAtlas, not universal constants. They exist so the later stages can argue about the same budget.

---

Why must "workspace isolation" be a first-class requirement in this design?

Answer

Search evidence can include private roadmaps, incident notes, contracts, and policy exceptions. Retrieval, caching, ranking, and citations must enforce tenant visibility before the LLM ever sees context.

How the pipeline works end to end

The architecture has four query-time layers: plan the query, retrieve candidate evidence, pack the best evidence into context, and generate an answer while verifying citations. Each layer narrows what the next one is allowed to see.

Separate from this online path, a background ingestion system crawls or receives source updates, cleans them, chunks them, and keeps sparse and dense indexes fresh. Without that path, an "AI search engine" becomes a prompt wrapper around stale data. With it, query-time code can spend its budget on choosing and checking evidence instead of parsing documents.

Crawl and index before query time

Start with one update: parser-v2.md changes its migration window and arrives with an ACL tag for workspace-a. Before looking at query code, predict what must survive ingestion. The crawler needs the new revision, the parser needs stable text and source offsets, and the indexer needs chunks that carry tenant, document version, and freshness metadata alongside sparse and dense representations.

That metadata is part of searchable evidence, not a side table to consult after generation. If the crawler drops the ACL, an unauthorized chunk can enter candidate generation. If the index keeps the old revision, a fast answer can still be stale. Store enough lineage to trace each chunk back to its source and version, then publish the new index view only when those fields are present.

Tracing a concrete query

Use one request as the debugging case: "Which deprecation policy applies to parser-v2 SDK migration for service accounts?" Before reading the trace, predict two things: which sources should survive authorization, and which single passage you'd keep if the answer had room for only one. This request needs both policy and exception, so it exposes every shortcut that loses one.

Keep this trace as a debugging breadcrumb. When a final answer fails, we can ask whether the planner chose the wrong route, retrieval missed a source, reranking buried it, packing dropped it, or verification rejected the claim.

---

Why separate offline ingest from the online query path?

Answer

Ingest normalizes, chunks, indexes, and refreshes source material before users ask questions. The online path then stays focused: plan, retrieve, rerank, pack evidence, synthesize, and verify. That boundary also gives each freshness decision a home: source updates belong to ingest, while query urgency belongs to planning.

Plan the parser-v2 question before you retrieve

Query understanding

The first hop answers an operational question: what kind of truth does this query need? Classify intent, choose a route, and decide whether retrieval needs rewriting. A large synthesis model can do that work, but using it for every CodeAtlas query adds measured latency and cost that "find the parser-v2 deprecation policy" doesn't need.

Predict the route before naming the machinery. "What is build-1042 doing right now?" should bypass document search. "Compare service-account exceptions" needs several sources. "What changed in the latest guide?" needs freshness. A small router or encoder-class classifier can make those choices, then attach hints such as "prefer exact matches" or "fresh data required." Benchmark it on your hardware and traffic shape because routing errors become retrieval errors downstream.

If a query is vague, HyDE (Hypothetical Document Embeddings) is one recall lever: generate a hypothetical relevant document, embed that document, and use its vector as an alternate dense-search query.^{[3]Reference 3Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496} That generated text may contain false details. Treat HyDE as a way to find candidates, never as answer evidence, and validate whether it improves recall for your query mix.

Routing targets

Not every query should hit the same retriever. Exact, structured questions often route to APIs, SQL, or an entity store first, then optionally use the LLM for verbalization. Treating every request like unstructured RAG adds latency and can make numeric answers less reliable because prose is standing in for live state.

The planner chooses the branch, not the answer. Live numeric state goes to an API, exact titles go to BM25, and comparisons pay for hybrid retrieval. A "latest" query starts with the hot tier, where freshness can be measured instead of guessed from wording.

The planner below is a fast stand-in for a small classifier. Give it a raw query and inspect the contract it returns: intent, route, sub-queries, filters, and whether live data is required. Those fields aren't decoration; downstream retrieval uses them to enforce the route we just chose.

When should a query planner route to a structured API instead of generic RAG?

Answer

Use a structured API for exact, live, or numeric facts such as deploy status, quota remaining, entitlement, or current feature-flag state. Generic semantic retrieval is weaker for those values and can return stale text.

Multi-stage retrieval

Now predict what broad recall should return for our parser-v2 question. BM25 should find the exact policy title and service-account phrase. Dense retrieval should add the FAQ that describes the same workflow in different words. Neither list is the answer; each is a candidate set with different blind spots.

We use hybrid retrieval to broaden that candidate recall before refining for precision. Keyword search can miss synonyms. Dense vector embeddings, which place queries and documents near one another when their meanings are related, can miss exact names, product IDs, or acronyms. Hybrid Search built this lane; this capstone adds freshness routing, workspace isolation, evidence packing, and a search latency budget.

At query time, run sparse retrieval and dense retrieval in parallel, then fuse their ranked lists with Reciprocal Rank Fusion (RRF).^{[4]Reference 4Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.https://dl.acm.org/doi/10.1145/1571941.1572114} A second stage applies a more expensive reranker, often a cross-encoder or late-interaction model,^{[5]Reference 5ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.https://arxiv.org/abs/2004.12832} to 50 passages. Packing then keeps 8 chunks with source IDs. The order matters: cheap methods widen the funnel, while expensive methods spend compute only on candidates that survived the first pass.

Why hybrid retrieval?

BM25 is strong at exact keyword matching; dense retrieval captures semantic similarity.^{[2]Reference 2The Probabilistic Relevance Framework: BM25 and Beyond.https://doi.org/10.1561/1500000019} Dense retrieval typically uses bi-encoders, which encode queries and documents separately so document vectors can be indexed ahead of time. Some systems instead use ColBERT (Contextualized Late Interaction over BERT): query and document tokens interact after each side has been encoded.^{[5]Reference 5ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.https://arxiv.org/abs/2004.12832} On mixed traffic, the combination can surface candidates that either retriever would miss alone.

In practice, hybrid search is more than running two retrievers. Chunking decides what can be found, rank fusion decides how lists meet, and evidence assembly decides what the model can quote. A useful pattern embeds short child chunks for recall, then fetches their larger parent passage for synthesis so surrounding context survives.

Fuse on rank instead of raw score because BM25 and dense-similarity scores use different scales. Adding them without calibration is brittle. Reciprocal Rank Fusion scores documents from rank positions: score(d) = sum over lists of 1 / (k + rank(d)). Cormack et al. evaluated RRF with k = 60; use that as a starting point to validate, not a universal optimum.^{[4]Reference 4Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.https://dl.acm.org/doi/10.1145/1571941.1572114} Apply authorization filtering before fusion. Otherwise, a high-ranked forbidden document can change ranks even if you remove it later.

Freshness and index tiers

Freshness isn't one global ranking switch. Real search products rarely use one monolithic index, because a changelog and a stable API reference need different update paths. A useful pattern is a hot-warm split:

Freshness should influence ranking, not replace relevance. Combine retrieval score with time decay, then let the reranker decide whether the newer document is better. If you rank by recency alone, a low-quality post from five minutes ago can outrank the canonical source. For "latest migration guide," the hot tier narrows the time window; it doesn't get to skip authorization or evidence checks.

Why does hybrid retrieval beat dense-only retrieval for developer-doc search?

Answer

Dense retrieval captures semantic paraphrases, but BM25 preserves exact identifiers such as API versions, issue IDs, feature flags, and policy titles. Rank fusion gives the reranker both kinds of candidates.

Latency budget

TTFT is a critical constraint of this system. The values below are an illustrative budget, not a promise. Before reading the total, predict the critical path: if planning overlaps retrieval, which stage contributes its full duration, and which stage is hidden under another? Validate the answer against product objectives and measured stage timings.

Note: If you support speculative retrieval, cheap lexical or dense search can begin before rewrite finishes. If you don't, planning and retrieval stack sequentially. Make that overlap decision explicit in the TTFT budget; otherwise a mean stage time can hide the real critical path.

What is the main latency tradeoff in query planning?

Answer

A better rewrite can improve recall, but waiting for it serially delays retrieval. Speculative retrieval starts cheap searches from the raw query while the planner finishes, then merges or cancels results.

Grounded synthesis with citations

The synthesis prompt can require citations and abstention, but prompting can't prove grounding. The product must keep source IDs attached to evidence, check generated claims, and decide what may be shown while support is pending or missing.

Before packing, predict the failure. If the correct exception lands in the middle of 40 passages, will the model use it? If a citation points to a document title but not to the sentence that supports the claim, can a reviewer verify it? These questions make context layout and citation binding part of the system, not prompt decoration.

Grounding has a context problem too. Put the strongest chunks at the start and end of the prompt, leave room for the answer, and avoid stuffing mediocre passages into the middle of a huge context. Long-context models still show "lost in the middle" behavior: Liu et al. measured a U-shaped use of context, where evidence in the center of the context window is used less reliably than evidence at the edges.^{[6]Reference 6Lost in the Middle: How Language Models Use Long Contextshttps://aclanthology.org/2024.tacl-1.9/} Reranking and tight top-K therefore protect both quality and latency.

Visible citations can be a subset of retrieved evidence, but each one must bind to a specific claim and source span. A source list is provenance metadata, not proof that every generated sentence is supported.

Before generation, pack allowed passages into a bounded context while preserving source IDs:

Compression after selection, not instead of it

Sometimes authorized, reranked evidence is still too large for the prefill budget. Query-aware prompt compression is a packing lever after access filtering, deduplication, and passage selection. It isn't permission filtering, retrieval, or a license to keep every weak result.

LLMLingua uses a smaller language model's cross-perplexity signal to estimate which prompt tokens carry less information for the target model, then applies coarse-to-fine pruning.^{[7]Reference 7LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models.https://arxiv.org/abs/2310.05736} The paper reports compression ratios as high as 20× on GSM8K, BBH, ShareGPT, and Arxiv-March23 with limited performance loss. That result doesn't guarantee safe compression for cited search evidence. Those tasks don't test claim-to-span verification, so evaluate support and answer quality at each target ratio.

Protect exact spans before compression: source IDs, document titles, code symbols, version numbers, dates, quantities, negation, and quoted policy language. Keep a mapping from compressed text back to original source offsets. The synthesizer may read compressed evidence, but citation verification should resolve claims against the original passage.

Track raw_tokens, selected_tokens, compressed_tokens, protected-span survival, retrieval recall, citation support, and TTFT together. If the correct exception or negation disappears, a smaller prompt is a regression.

Where does LLMLingua-class compression belong in evidence packing?

Answer

After authorization, reranking, deduplication, and passage selection. Protect exact evidence spans, compress only the remaining text, retain source-offset mappings, and verify generated claims against original passages.

Keeping temperature low reduces answer variance, but it doesn't create factuality. Predict what should happen when one of the two required passages is absent: the synthesizer may draft, but publication must wait for evidence checks. The next deterministic example turns that support gate into code.

Why is low temperature not enough to make generated search factual?

Answer

Low temperature reduces variance, but factuality comes from retrieved evidence, source IDs, evidence packing, and claim-level verification. A deterministic unsupported answer is still unsupported.

Streaming with pending support

Once evidence is selected, the user still has to wait for synthesis and verification. For one-way progressive delivery, Server-Sent Events (SSE) work through the browser EventSource API over HTTP and are easy to render.^{[8]Reference 8Using server-sent eventshttps://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events} The transport is simple; the policy isn't. Hold factual claims until verification completes, or label streamed text as provisional and update or retract unsupported claims.

Streaming endpoint

Predict the safe event sequence for the parser-v2 answer: plan and sources can appear early, but a policy claim should remain visibly pending until its cited span passes. This event contract fits a low-risk path that permits provisional rendering. It emits the plan and sources, declares pending support, streams text, then attaches claim-check results. A higher-risk workflow can omit text frames until support becomes supported.

A live CodeAtlas stream also needs periodic heartbeat events and upstream proxies that don't buffer. Log support-state transitions so a provisional parser-v2 sentence that later gets blocked can be audited. The audit trail should show which claim, source span, and verifier decision changed the UI state.

---

Why use SSE for search-style answer streaming?

Answer

SSE fits one-way progressive token delivery over HTTP when the browser needs plan, source, text, and support-state events from the server. It doesn't decide whether unverified facts may be displayed.

A fluent parser-v2 sentence can still be unsupported

Suppose CodeAtlas writes, "Start the migration within 30 days," while its citation only says that migration planning has begun. The sentence sounds plausible, and its citation exists, but the cited span doesn't support its deadline. RAG reduces this risk; it doesn't remove it.

FActScore makes the remaining error visible by splitting long-form text into atomic facts and scoring the fraction supported by a knowledge source.^{[9]Reference 9FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation.https://arxiv.org/abs/2305.14251} On biography generation, Min et al. reported FActScores of 42% for InstructGPT, 58% for ChatGPT, and 71% for search-augmented PerplexityAI. Retrieval helped, yet nearly three in ten atomic facts remained unsupported on that task.

CodeAtlas can reuse that decomposition at request time: extract claims, bind each one to a cited span, then verify entailment before the UI treats the text as final. Predict the failure response now: a weak claim should be blocked, marked provisional, or regenerated, not hidden behind a correct-looking source list.

Strong systems stack several checks rather than betting on one verifier. Cheap checks can reject obvious misses; more expensive checks decide whether meaning, numbers, and negation survive.

NLI-based verification estimates whether a cited passage entails a claim rather than merely sharing words. TRUE^{[10]Reference 10TRUE: Re-evaluating Factual Consistency Evaluation.https://arxiv.org/abs/2204.04991} found large-scale NLI methods and question-generation-plus-answering methods to be strong automatic factual-consistency checks on its benchmarks. NLI is therefore a useful measured layer, not proof of correctness.

The function below is deliberately only a lexical precheck. It can cheaply reject a missing phrase; it can't label entailment. Claims that pass still need a calibrated verifier or a policy that keeps them provisional.

Why can lexical matching not replace NLI or another calibrated verifier?

Answer

Lexical matching checks whether words appear in a source. Entailment evaluation estimates whether the passage supports the meaning of the claim, including paraphrases or contradictions.

Verification is another billable stage. At search QPS, even a one-cent synthesizer plus a claim checker becomes a budget problem. That's why quality gates and cost gates have to be designed together.

What running this costs

At scale, every extra candidate, token, and claim becomes a cost decision. Exact dollar figures drift with vendor pricing, model choice, and answer length, so reason from units of work rather than one fixed price table. Before looking at the scenario, predict which route should be cheapest: a live structured lookup, a cache hit, or full synthesis with verification.

Use scenario inputs rather than presenting provider pricing as fixed:

• Web retrieval: ~$0.002
• Reranking 50 query-document pairs: ~$0.001
• 1,500 prompt tokens + 400 output tokens at $0.003 per 1K tokens: ~$0.006
• Verifying 5 claims: ~$0.001

For these inputs, total cost is roughly $0.01 per query. At 1,000 QPS, that scenario approaches $600 per minute. Routing and caching are therefore not polish; they keep avoidable requests off the synthesis path.

$$\begin{aligned} \text{cost/query} \approx\;& C_{\text{search}} + N_{\text{pairs}} \cdot C_{\text{rerank}} \\ &+ (T_{\text{prompt}} + T_{\text{output}}) \cdot C_{\text{token}} \\ &+ N_{\text{claims}} \cdot C_{\text{verify}} \end{aligned}$$

Which stage dominates variable cost depends on provider rates, fan-out, prompt length, and verification policy. Measure the breakdown first. Then route cacheable or structured queries away from unnecessary synthesis and trim top-K only where retrieval evaluation allows it.

In the worked 1,000 QPS cost scenario, why does routing matter?

Answer

The scenario costs hundreds of dollars per minute. Cache hits, structured routes, smaller models, and evaluated top-K reduction can change that envelope before traffic reaches synthesis.

Serving-side latency levers

TTFT can be dominated by retrieval, prompt assembly, or prefill; long answers can shift pressure toward decode. Before choosing an optimization, identify which bar is actually long in a trace. A serving fix aimed at decode can't shorten a slow reranker.

PagedAttention improves KV-cache memory management and can increase concurrency when KV allocation is the binding constraint.^{[11]Reference 11Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} If decode is the bottleneck, speculative decoding can reduce latency when a draft model is accepted often enough; benchmark it for answer shapes and the target model in this service.^{[12]Reference 12Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192}

These serving optimizations matter after retrieval depth and prompt length are under control. If 40 mediocre passages still reach the model, no serving trick can rescue TTFT. First reduce work that should never have entered the context; then optimize the work that remains.

GPU capacity planning

Split concurrency by which latency you mean. Product SLOs here are often TTFT (about 0.8 to 1.5 seconds to first token). End-to-end generation is longer because it includes first token plus the rest of the answer. A dashboard that reports only TTFT can hide contexts that remain resident during decode.

For a 1K QPS design point, the low end of the 1K to 10K QPS per-region range, use the overlapped 800 ms TTFT as about 1.0 s in round numbers and budget remaining decode separately. A 120-token answer at roughly 80 tokens/s after the first token needs about 1.5 s more, so end-to-end is about 2.5 s:

$$\begin{aligned} L_{\text{TTFT}} &\approx 1.0\,\text{s} \\ \text{streaming sessions at first token} &\approx 1000 \times 1.0 = 1000 \end{aligned}$$ $$\begin{aligned} L_{\text{e2e}} &\approx 2.5\,\text{s} \\ \text{full in-flight answers and KV held} &\approx 1000 \times 2.5 = 2500 \end{aligned}$$

Little's Law still holds ($L = \lambda W$). Use $W = L_{\text{TTFT}}$ for admission of streaming sessions and $W = L_{\text{e2e}}$ for decode concurrency and KV residency. The 2500 figure is not first-token-only concurrency: it's the number of answers still in flight across their full 2.5-second lifetime.

Decode throughput alone is a lower bound. Twenty GPUs at 6k tok/s can emit the tokens, but 2500 concurrent contexts may first exhaust high-bandwidth memory (HBM) reserved for KV state. That's a different constraint from tokens per second.

Reuse the multi-tenant lesson's roughly 1.22 GiB per 4K-token request: 2K tokens is about 0.61 GiB each. If packing leaves 60% of each 80 GiB GPU usable for KV, then $2500 \times 0.61 / (0.6 \times 80) \approx 32$ GPUs before weight residency and headroom. Formula sketch:

$$\begin{aligned} B_{\text{KV}} &= \text{concurrent} \times \text{average tokens} \\ &\quad \times \text{KV bytes/token} \\ N_{\text{GPU, KV}} &\approx \left\lceil \frac{B_{\text{KV}}}{\text{usable KV HBM/GPU}} \right\rceil \end{aligned}$$

PagedAttention improves packing; it doesn't remove byte accounting. Don't assume a universal "requests per GPU" number. Measure your model, context length, batching, and quantization. The decode line and KV line can disagree, as they do here: 20 GPUs can emit the tokens while 32 GPUs are needed to hold 2500 contexts.

$$\begin{aligned} R_{\text{decode}} &= \text{QPS} \times \text{average output tokens} \\ N_{\text{GPU, decode}} &\approx \left\lceil \frac{R_{\text{decode}}}{\text{effective decode tok/s/GPU}} \right\rceil \end{aligned}$$

For long prompts, budget prefill separately too:

$$\text{Prefill capacity} \approx \frac{\text{QPS} \times \text{avg prompt tokens}}{\text{effective prefill tok/s/GPU}}$$

At the same 1K QPS design point, split pools explicitly (illustrative shares; measure yours):

Peak 10K QPS is an order of magnitude above this worked example. Shard regions, raise cache hit rates, or replicate pools only after remeasuring stage hit rates. The 1K QPS GPU count doesn't cover the top of the requirements range.

Size the system against whichever pool saturates first: retriever GPUs, reranker GPUs, prefill workers, or decode workers. The answer model may be the expensive tier, but measured retrieval or reranking can still be the first bottleneck. When a request misses its TTFT SLO, locate that pool before adding model capacity.

Why can reranking or retrieval be the bottleneck even when the LLM is expensive?

Answer

Large fan-out multiplies query-document pairs, web/API calls, and cleanup work before synthesis starts. If retrieval depth or reranker batches saturate first, faster decoding won't fix TTFT.

---

How to know it's working

A fluent answer can cite two passages and still omit the service-account exception. Which metric should fail first? RAGAS^{[13]Reference 13RAGAS: Automated Evaluation of Retrieval Augmented Generation.https://arxiv.org/abs/2309.15217} introduced reference-free evaluation around faithfulness, answer relevance, and context relevance. That's one offline layer when gold answers are scarce, not a replacement for labeled retrieval metrics, citation checks, or online product metrics. A response can be consistent with mediocre evidence and still fail the user.

This example wires a scorecard together with simple token-overlap proxies so it stays runnable. Read the output as a diagnosis prompt, not a quality badge: citation resolution only checks whether a visible citation ID maps to an available passage; it doesn't prove claim support. These proxies aren't RAGAS or calibrated faithfulness judges. Replace them in an evaluation service while keeping labeled retrieval metrics separate.

Why keep reference-free RAGAS-style metrics separate from labeled retrieval metrics?

Answer

They answer different questions. Reference-free metrics judge answer/context consistency; labeled metrics reveal whether the retriever found and ranked known relevant passages.

Use the scorecard to locate the failing handoff. Low Recall@k points upstream to crawling, chunking, query planning, or candidate retrieval. Good recall with poor context relevance points to selection or packing. Good context with unsupported claims points to synthesis or citation verification. This turns an evaluation result into an owner and a next experiment.

Release only when the relevant gates agree: authorized retrieval is recalled, citations resolve to source spans, claims pass support checks, and TTFT and cost stay inside their budgets. A green answer-quality score can't waive an authorization failure, and a fast stream can't waive a missing source.

---

Scaling without blowing the budget

As traffic grows, evaluate each latency or cost lever against quality and authorization tests:

---

Why is semantic response caching risky without freshness and permission checks?

Answer

Similar queries can have different tenant visibility, quota state, policy version, or time sensitivity. Cache reuse must satisfy similarity, freshness, and authorization thresholds.

Failure patterns worth replaying

Replay the same parser-v2 query with one fault at a time: remove the fresh revision, admit private-roadmap, drop the exception during packing, or flip a citation's negation. Observe which metric and trace span change. This isolates the broken handoff instead of blaming "the LLM" for every bad answer.