LLM-Powered Search Engine

Multi-tenant serving made tenant identity, cost, and visibility part of every request. Large language model (LLM)-powered search applies that discipline to evidence: retrieval, ranking, caches, citations, and generated claims must all respect freshness and authorization before the model writes an answer.

You're the Staff AI Engineer at CodeAtlas. Developers type questions such as "can service accounts rotate OAuth tokens every 90 days?" and "what happens if parser-v2 deprecation affects our SDK migration?" A generative search experience must retrieve allowed API docs, release notes, and workspace policies, synthesize a cited answer, and keep private roadmaps and incident notes outside unauthorized contexts.

An LLM-powered search engine combines retrieval, ranking, synthesis, and citation into a user-facing answer system. The core engineering challenge is balancing retrieval and generation latency with factual accuracy, freshness, and workspace isolation.

Building a search engine that generates answers rather than just lists of links requires orchestrating retrieval, reranking, and LLM synthesis inside an interactive latency budget. This CodeAtlas case study designs that system around evidence and measurable service objectives.

If the ideas of embeddings or chunking feel rusty, here's a quick reminder: embeddings turn sentences into dense vectors so similar meanings sit close together in math space, and chunking splits long documents into short, self-contained passages so retrieval can grab the right snippet. Both are prerequisites for everything that follows.

What makes LLM-powered search different from a chatbot over documents?

Answer

It must retrieve fresh, permission-safe evidence, rank it, synthesize only supported claims, attach citations, and stream an answer within an interactive search latency budget.

From keywords to reasoning

Traditional search points users to candidate documents. LLM-powered search retrieves API docs, policy pages, release notes, and incident records, then returns a cited answer with a support state. This relies on Retrieval-Augmented Generation (RAG)^{[1]Reference 1Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.https://arxiv.org/abs/2005.11401}, which conditions generation on retrieved evidence rather than treating model memory as the source of truth.

Traditional search leaves synthesis to the user. Generative search performs that synthesis, which means it also owns citation binding, abstention, and authorization failures.

This capstone uses a practical answer-engine architecture: retrieve with keyword and embedding signals, rerank candidates, apply visibility and freshness policy, then synthesize a short cited answer whose claims can be checked.

Traditional vs LLM search

---

What does the LLM add after traditional retrieval?

Answer

The LLM performs synthesis: it reads retrieved evidence, resolves conflicts, explains the answer in user language, and binds claims to citations. It should not replace retrieval or source verification.

What the system must deliver

Before selecting components, define performance and functional constraints. These are example SLOs for this interactive developer-product case study, not universal constants.

---

Why must "workspace isolation" be a first-class requirement in this design?

Answer

Search evidence can include private roadmaps, incident notes, contracts, and policy exceptions. Retrieval, caching, ranking, and citations must enforce tenant visibility before the LLM ever sees context.

How the pipeline works end to end

The architecture has four query-time layers: plan the query, retrieve candidate evidence, pack the best evidence into context, and generate an answer while verifying citations.

Separate from this online path, a background ingestion system crawls or receives source updates, cleans them, chunks them, and keeps sparse and dense indexes fresh. Without that ingest path, an "AI search engine" quickly becomes a prompt wrapper around stale data.

Tracing a concrete query

Follow the same question through every stage of the pipeline: "Which deprecation policy applies to parser-v2 SDK migration for service accounts?"

This trace will reappear as we look at each component in detail.

---

Why separate offline ingest from the online query path?

Answer

Ingest normalizes, chunks, indexes, and refreshes source material before users ask questions. The online path must stay fast: plan, retrieve, rerank, pack evidence, synthesize, and verify.

Inside the pipeline

1. Query understanding

The first step is classifying intent and deciding whether retrieval needs rewriting. A large synthesis model can perform this task, but using it on every query adds measured latency and cost that many simple routes don't need.

A common approach uses specialized, smaller models or encoder-class classifiers for routing, rewriting, and decomposition. Benchmark the chosen planner on your hardware and traffic shape; its job is to distinguish a simple fact lookup from an exploratory or freshness-sensitive request and attach retrieval hints such as "prefer exact matches" or "fresh data required."

One useful trick is HyDE (Hypothetical Document Embeddings): generate a hypothetical relevant document, embed that document, and use its vector as an alternate dense-search query.^{[2]Reference 2Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496} The generated text may contain false details, so HyDE is a retrieval technique, not answer evidence. Validate whether it improves recall for your query mix.

Routing targets

Not every query should hit the same retriever. Exact, structured questions often route to APIs, SQL, or an entity store first, then optionally use the LLM only for verbalization. Treating every request like unstructured RAG adds latency and often makes numeric answers less reliable.

This Python code structures a low-latency query planner. It takes the raw user query as input and returns a structured execution plan, including classified intent and decomposed sub-queries. In production, the decision can come from a small classifier, rules plus embeddings, or a compact model rather than a frontier synthesis model.

When should a query planner route to a structured API instead of generic RAG?

Answer

Use a structured API for exact, live, or numeric facts such as deploy status, quota remaining, entitlement, or current feature-flag state. Generic semantic retrieval is weaker for those values and can return stale text.

2. Multi-stage retrieval

We use a hybrid retrieval strategy to broaden candidate recall before refining for precision. A single retrieval method is rarely sufficient for complex queries. Relying only on keyword search might miss documents that use synonyms, while relying entirely on dense vector embeddings (encoding data as points in a high-dimensional space so related concepts are near each other) can overlook exact matches for specific names, product IDs, or acronyms.

By designing a multi-stage pipeline, we capture the best of both approaches. The first stage runs sparse retrieval and dense retrieval in parallel, then fuses the ranked lists with Reciprocal Rank Fusion (RRF)^{[3]Reference 3Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.https://dl.acm.org/doi/10.1145/1571941.1572114}. The second stage applies a more expensive reranker (often a cross-encoder or late-interaction model^{[4]Reference 4ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.https://arxiv.org/abs/2004.12832}) to a much smaller candidate set, passing only the most relevant chunks or parent passages to the LLM.

This visual outlines the flow of this multi-stage retrieval process.

Why hybrid retrieval?

BM25 (Best Match 25, a reliable keyword search algorithm) excels at exact keyword matching; dense retrieval captures semantic similarity. Dense retrieval typically uses bi-encoders (which encode queries and documents separately for fast lookup), though more advanced systems may use architectures like ColBERT (Contextualized Late Interaction over BERT) for late interaction between query and document tokens.^{[4]Reference 4ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.https://arxiv.org/abs/2004.12832} On a mixed workload, combining retrievers can surface candidates that either one would miss alone.

In practice, hybrid search isn't just "run two retrievers." You also need rank fusion, chunking, and evidence assembly. A common pattern is to embed short child chunks for recall, then retrieve the larger parent passage for synthesis so the model sees enough surrounding context to answer cleanly.

Why fuse on rank instead of raw score? BM25 and dense-similarity scores have different scales, so adding them without calibration is brittle. Reciprocal Rank Fusion scores documents from rank positions: score(d) = sum over lists of 1 / (k + rank(d)). Cormack et al. evaluated RRF with k = 60; treat that value as a starting point to validate, not a universal optimum.^{[3]Reference 3Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.https://dl.acm.org/doi/10.1145/1571941.1572114} Apply authorization filtering before fusion so a high-ranked forbidden document never becomes generation context.

Freshness and index tiers

Real search products rarely use one monolithic index. A useful pattern is a hot-warm split:

Freshness should influence ranking, not replace relevance. A common pattern is to combine retrieval score with time decay, then let the reranker decide whether the newer document is better. If you rank purely by recency, a low-quality post from five minutes ago can outrank the canonical source.

Why does hybrid retrieval beat dense-only retrieval for developer-doc search?

Answer

Dense retrieval captures semantic paraphrases, but BM25 preserves exact identifiers such as API versions, issue IDs, feature flags, and policy titles. Rank fusion gives the reranker both kinds of candidates.

Latency budget

TTFT is a critical constraint of this system. The values below are an illustrative budget; validate them against product objectives and measured stage timings.

Note: If you support speculative retrieval, cheap lexical or dense search can begin before rewrite finishes. If you don't, planning and retrieval stack sequentially. Make that decision explicit when you budget TTFT.

What is the main latency tradeoff in query planning?

Answer

A better rewrite can improve recall, but waiting for it serially delays retrieval. Speculative retrieval starts cheap searches from the raw query while the planner finishes, then merges or cancels results.

3. LLM synthesis with grounded citations

The synthesis prompt can require citations and abstention, but prompting can't prove grounding. The product must keep source IDs attached to evidence, check generated claims, and decide what may be shown while support is pending or missing.

Grounding isn't just a prompt problem. Evidence packing matters too. Put the strongest chunks first, leave room for the model to answer, and avoid stuffing dozens of mediocre passages into the middle of a huge prompt. Long-context models still show "lost in the middle" behavior, where evidence in the center of the context window gets used less reliably.^{[5]Reference 5Lost in the Middle: How Language Models Use Long Contextshttps://arxiv.org/abs/2307.03172} That's one more reason reranking and tight top-K beat dumping everything into a large context window.

The citations a user sees can be a subset of retrieved evidence, but each visible citation must bind to a specific claim and its source span. Don't treat a source list as proof that every generated sentence is supported.

Before generation, pack allowed passages into a bounded context while preserving source IDs:

Compression after selection, not instead of it

Sometimes the authorized, reranked evidence is still too large for the prefill budget. Query-aware prompt compression is another packing lever after access filtering, deduplication, and passage selection. It isn't permission filtering, retrieval, or a license to keep every weak result.

LLMLingua uses a smaller language model's cross-perplexity signal to estimate which prompt tokens carry less information for the target model, then applies coarse-to-fine pruning.^{[6]Reference 6LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models.https://arxiv.org/abs/2310.05736} Its paper reports compression ratios as high as 20× on evaluated tasks with limited performance loss, but that number isn't a service guarantee. Search evidence needs its own support and answer-quality evaluation at each target ratio.

Protect exact spans before compression: source IDs, document titles, code symbols, version numbers, dates, quantities, negation, and quoted policy language. Keep a mapping from compressed text back to original source offsets. The synthesizer may read compressed evidence, but citation verification should resolve claims against the original uncompressed passage.

Track raw_tokens, selected_tokens, compressed_tokens, protected-span survival, retrieval recall, citation support, and TTFT together. A smaller prompt isn't a win if the correct exception or negation disappears.

Where does LLMLingua-class compression belong in evidence packing?

Answer

After authorization, reranking, deduplication, and passage selection. Protect exact evidence spans, compress only the remaining text, retain source-offset mappings, and verify generated claims against original passages.

Keeping temperature low reduces answer variance, but it doesn't create factuality. The next deterministic example represents a support gate around generated claims: a real synthesizer can draft text, while publication depends on evidence checks.

Why is low temperature not enough to make generated search factual?

Answer

Low temperature reduces variance, but factuality comes from retrieved evidence, source IDs, evidence packing, and claim-level verification. A deterministic unsupported answer is still unsupported.

4. Streaming architecture

For one-way progressive delivery, Server-Sent Events (SSE) work through the browser EventSource API over HTTP and are straightforward to render^{[7]Reference 7Using server-sent eventshttps://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events}. Streaming creates a product-policy choice: either hold factual claims until verification completes, or label streamed text as provisional and update or retract unsupported claims.

Streaming endpoint

This event contract fits a low-risk answer path that permits provisional rendering. It emits the plan and sources, declares pending support, streams text, then attaches the result of claim checks. A higher-risk workflow can omit text frames until support becomes supported.

In production, add periodic heartbeat events and configure upstream proxies not to buffer the stream. Also log state transitions so provisional output that becomes blocked can be audited.

---

Why use SSE for search-style answer streaming?

Answer

SSE fits one-way progressive token delivery over HTTP when the browser needs plan, source, text, and support-state events from the server. It doesn't decide whether unverified facts may be displayed.

Why answers still hallucinate and how to catch them

One of the most critical challenges in search is hallucination, when the LLM generates plausible but incorrect facts. Even with RAG, models can misinterpret complex details. Strong systems usually stack multiple checks rather than betting on one verifier:

NLI-based verification estimates whether a cited passage entails a claim rather than merely sharing words. TRUE^{[8]Reference 8TRUE: Re-evaluating Factual Consistency Evaluation.https://arxiv.org/abs/2204.04991} found that large-scale NLI methods and question-generation-plus-answering methods were strong automatic factual-consistency checks on its benchmarks, so NLI is a useful measured layer, not proof of correctness.

This function is deliberately only a lexical precheck. It can cheaply reject a missing phrase; it can't label entailment. Claims that pass it still need a calibrated verifier or a policy that keeps them provisional.

Why can lexical matching not replace NLI or another calibrated verifier?

Answer

Lexical matching checks whether words appear in a source. Entailment evaluation estimates whether the passage supports the meaning of the claim, including paraphrases or contradictions.

---

Skills to defend in review

Production questions to answer

How do you handle queries that need real-time data? Real-time data needs its own freshness pipeline. Use a hot index for newly crawled content plus direct routes to live APIs or structured stores for exact numeric fields. When the planner marks a query as freshness-sensitive, search the hot tier first, apply recency-aware scoring, then merge with the warmer corpus so the LLM sees both new evidence and canonical background context.

How would you detect and handle hallucinated citations? Decompose the generated answer into atomic claims, attach each claim to its cited source, and verify entailment with NLI or a calibrated lighter check. If support is weak or contradictory, flag the statement, remove it, or regenerate with stricter grounding and a smaller evidence set.

What's your latency budget for each pipeline stage? For a consumer-facing experience, a reasonable example SLO is roughly 0.8-1.5 seconds to first token. Query planning often fits in 10-50ms, retrieval fan-out in 150-500ms, reranking in 50-150ms, evidence packing in 10-30ms, and synthesis in roughly 300-800ms to first token. Split synthesis into prefill and decode because prompt length drives prefill cost while answer length drives decode cost.

How do you evaluate answer quality at scale? Start with reference-free metrics such as faithfulness, answer relevance, and context relevance. Add labeled retrieval metrics such as Recall@k or nDCG on benchmark sets, plus citation precision and unsupported-claim rate. In production, track citation click-through, reformulation rate, dwell time, abandonment, and explicit user feedback.

Common design failures

• Designing everything around a single LLM call instead of specialized routers, retrievers, rerankers, and verifiers.
• Treating structured or freshness-critical queries like generic vector search.
• Ignoring latency requirements for interactive search experiences.
• Skipping rigorous citation verification and trusting prompt instructions alone.
• Treating long context windows as a replacement for reranking and evidence packing.
• Overlooking inference cost per query when scaling to millions of users.

Common mistakes when building generative search

---

What running this costs

Building a search engine at scale requires careful cost management. Exact dollar figures drift with vendor pricing, model choice, and answer length, so reason from units of work rather than one fixed price table.

Before writing the formula, use scenario inputs rather than presenting provider pricing as fixed:

• Web retrieval: ~$0.002
• Reranking 200 query-document pairs: ~$0.001
• 1,500 prompt tokens + 400 output tokens at $0.003 per 1K tokens: ~$0.006
• Verifying 5 claims: ~$0.001

For these inputs, total cost is roughly $0.01 per query. At 1,000 QPS, that scenario approaches $600 per minute, which is why routing and caching need their own budgets.

$$\begin{aligned} \text{cost/query} \approx\;& C_{\text{search}} + N_{\text{pairs}} \cdot C_{\text{rerank}} \\ &+ (T_{\text{prompt}} + T_{\text{output}}) \cdot C_{\text{token}} \\ &+ N_{\text{claims}} \cdot C_{\text{verify}} \end{aligned}$$

Which stage dominates variable cost depends on provider rates, fan-out, prompt length, and verification policy. Measure the breakdown, then route cacheable or structured queries away from unnecessary synthesis and trim top-K where retrieval evaluation allows it.

In the worked 1,000 QPS cost scenario, why does routing matter?

Answer

The scenario costs hundreds of dollars per minute. Cache hits, structured routes, smaller models, and evaluated top-K reduction can change that envelope before traffic reaches synthesis.

Serving-side latency levers

TTFT can be dominated by retrieval, prompt assembly, or prefill; long answers can shift pressure toward decode. Profile the critical path before choosing an optimization.

PagedAttention improves KV-cache memory management and can increase concurrency when KV allocation is the binding constraint.^{[9]Reference 9Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} If decode is the bottleneck, speculative decoding can reduce latency when a draft model is accepted often enough; benchmark it for the answer shapes and target model in this service.^{[10]Reference 10Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192}

These serving optimizations matter only after retrieval depth and prompt length are under control. If you keep sending 40 mediocre passages to the model, no amount of serving cleverness will rescue TTFT.

GPU capacity planning

Split concurrency by which latency you mean. Product SLOs here are often TTFT (about 0.8 to 1.5 seconds to first token), while end-to-end generation (full answer) and KV hold time are longer.

For a 1K QPS design point (the low end of the 1K to 10K QPS per-region range; size peak regions separately):

$$\begin{aligned} L_{\text{TTFT}} &\approx 1.0\,\text{s} \\ \text{streaming sessions at first token} &\approx 1000 \times 1.0 = 1000 \end{aligned}$$ $$\begin{aligned} L_{\text{e2e}} &\approx 1.5\,\text{s} \\ \text{full in-flight answers and KV held} &\approx 1000 \times 1.5 = 1500 \end{aligned}$$

Little's Law still holds ($L = \lambda W$); use $W = L_{\text{TTFT}}$ for admission of streaming sessions and $W = L_{\text{e2e}}$ for decode concurrency and KV residency. The 1500 figure is not first-token-only concurrency.

Decode throughput alone is a lower bound. Twenty GPUs at 6k tok/s can emit the tokens, but 1500 concurrent contexts may bind KV HBM first. Using the multi-tenant lesson's roughly 1.22 GiB per 4K-token request as a sketch: if average prompt plus generated context is about 2K tokens (roughly 0.61 GiB each) and packing leaves 60% of each 80 GiB GPU usable for KV, then $1500 \times 0.61 / 0.6 \approx 1525$ GiB of raw HBM capacity, or about 20 80-GiB GPUs before weight residency and headroom. Formula sketch:

$$\begin{aligned} B_{\text{KV}} &= \text{concurrent} \times \text{average tokens} \\ &\quad \times \text{KV bytes/token} \\ N_{\text{GPU, KV}} &\approx \left\lceil \frac{B_{\text{KV}}}{\text{usable KV HBM/GPU}} \right\rceil \end{aligned}$$

PagedAttention improves packing; it doesn't remove the byte accounting.

The next step is to translate concurrency into throughput and memory requirements for your actual serving stack. Don't assume a universal "requests per GPU" number. Measure your own model, context length, batching strategy, and quantization settings. PagedAttention improves packing efficiency and reduces KV-cache fragmentation under concurrent load^{[9]Reference 9Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}, but capacity still depends on empirical tokens-per-second benchmarks.

$$\begin{aligned} R_{\text{decode}} &= \text{QPS} \times \text{average output tokens} \\ N_{\text{GPU, decode}} &\approx \left\lceil \frac{R_{\text{decode}}}{\text{effective decode tok/s/GPU}} \right\rceil \end{aligned}$$

For long prompts, also budget prefill separately:

$$\text{Prefill capacity} \approx \frac{\text{QPS} \times \text{avg prompt tokens}}{\text{effective prefill tok/s/GPU}}$$

At the same 1K QPS design point, split pools explicitly (illustrative shares; measure yours):

Peak 10K QPS is an order of magnitude above this worked example: either shard regions, raise cache hit rates, or replicate every pool ~10× after you remeasure stage hit rates. Don't pretend the 1K QPS GPU count covers the top of the requirements range.

In practice, size the system against whichever pool saturates first: retriever GPUs, reranker GPUs, prefill workers, or decode workers. The answer model may be an expensive tier, but measured retrieval or reranking can still be the first bottleneck.

Why can reranking or retrieval be the bottleneck even when the LLM is expensive?

Answer

Large fan-out multiplies query-document pairs, web/API calls, and cleanup work before synthesis starts. If retrieval depth or reranker batches saturate first, faster decoding won't fix TTFT.

---

How to know it's working

RAGAS^{[11]Reference 11RAGAS: Automated Evaluation of Retrieval Augmented Generation.https://arxiv.org/abs/2309.15217} introduced reference-free evaluation around faithfulness, answer relevance, and context relevance. It's one offline layer when gold answers are scarce, not a replacement for labeled retrieval metrics, citation checks, or online product metrics: a response can be consistent with mediocre evidence and still fail the user.

This example wires a scorecard together with simple token-overlap proxies so it stays runnable. Its citation-resolution metric only checks whether a visible citation ID maps to an available passage; it doesn't prove claim support. These proxies aren't RAGAS or calibrated faithfulness judges. Replace them in an evaluation service while keeping labeled retrieval metrics separate.

Why keep reference-free RAGAS-style metrics separate from labeled retrieval metrics?

Answer

They answer different questions. Reference-free metrics judge answer/context consistency; labeled metrics reveal whether the retriever found and ranked known relevant passages.

---

Scaling without blowing the budget

As traffic grows, evaluate each latency or cost lever against quality and authorization tests:

---

Why is semantic response caching risky without freshness and permission checks?

Answer

Similar queries can have different tenant visibility, quota state, policy version, or time sensitivity. Cache reuse must satisfy similarity, freshness, and authorization thresholds.