LLM-Powered Search Engine

Multi-tenant serving made tenant identity, cost, and visibility part of every request. Large language model (LLM)-powered search applies that discipline to evidence: retrieval, ranking, caches, citations, and generated claims must all respect freshness and authorization before the model writes an answer.

You're the Staff AI Engineer at CodeAtlas. Developers type questions such as "can service accounts rotate OAuth tokens every 90 days?" and "what happens if parser-v2 deprecation affects our SDK migration?" A generative search experience must retrieve allowed API docs, release notes, and workspace policies, synthesize a cited answer, and keep private roadmaps and incident notes outside unauthorized contexts.

An LLM-powered search engine combines retrieval, ranking, synthesis, and citation into a user-facing answer system. The core engineering challenge is balancing retrieval and generation latency with factual accuracy, freshness, and workspace isolation.

Building a search engine that generates answers rather than just lists of links requires orchestrating retrieval, reranking, and LLM synthesis inside an interactive latency budget. This CodeAtlas case study designs that system around evidence and measurable service objectives.

If the ideas of embeddings or chunking feel rusty, here's a quick reminder: embeddings turn sentences into dense vectors so similar meanings sit close together in math space, and chunking splits long documents into short, self-contained passages so retrieval can grab the right snippet. Both are prerequisites for everything that follows.

From keywords to reasoning

Traditional search points users to candidate documents. LLM-powered search retrieves API docs, policy pages, release notes, and incident records, then returns a cited answer with a support state. This relies on Retrieval-Augmented Generation (RAG)^{[1]Reference 1Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.https://arxiv.org/abs/2005.11401}, which conditions generation on retrieved evidence rather than treating model memory as the source of truth.

Traditional search leaves synthesis to the user. Generative search performs that synthesis, which means it also owns citation binding, abstention, and authorization failures.

This capstone uses a practical answer-engine architecture: retrieve with keyword and embedding signals, rerank candidates, apply visibility and freshness policy, then synthesize a short cited answer whose claims can be checked.

Traditional vs LLM search

---

What the system must deliver

Before selecting components, define performance and functional constraints. These are example SLOs for this interactive developer-product case study, not universal constants.

---

How the pipeline works end to end

The architecture has four query-time layers: plan the query, retrieve candidate evidence, pack the best evidence into context, and generate an answer while verifying citations.

Separate from this online path, a background ingestion system crawls or receives source updates, cleans them, chunks them, and keeps sparse and dense indexes fresh. Without that ingest path, an "AI search engine" quickly becomes a prompt wrapper around stale data.

Tracing a concrete query

Follow the same question through every stage of the pipeline: "Which deprecation policy applies to parser-v2 SDK migration for service accounts?"

This trace will reappear as we look at each component in detail.

---

Inside the pipeline

1. Query understanding

The first step is classifying intent and deciding whether retrieval needs rewriting. A large synthesis model can perform this task, but using it on every query adds measured latency and cost that many simple routes don't need.

A common approach uses specialized, smaller models or encoder-class classifiers for routing, rewriting, and decomposition. Benchmark the chosen planner on your hardware and traffic shape; its job is to distinguish a simple fact lookup from an exploratory or freshness-sensitive request and attach retrieval hints such as "prefer exact matches" or "fresh data required."

One useful trick is HyDE (Hypothetical Document Embeddings): generate a hypothetical relevant document, embed that document, and use its vector as an alternate dense-search query.^{[2]Reference 2Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496} The generated text may contain false details, so HyDE is a retrieval technique, not answer evidence. Validate whether it improves recall for your query mix.

Routing targets

Not every query should hit the same retriever. Exact, structured questions often route to APIs, SQL, or an entity store first, then optionally use the LLM only for verbalization. Treating every request like unstructured RAG adds latency and often makes numeric answers less reliable.

This Python code structures a low-latency query planner. It takes the raw user query as input and returns a structured execution plan, including classified intent and decomposed sub-queries. In production, the decision can come from a small classifier, rules plus embeddings, or a compact model rather than a frontier synthesis model.

2. Multi-stage retrieval

We use a hybrid retrieval strategy to broaden candidate recall before refining for precision. A single retrieval method is rarely sufficient for complex queries. Relying only on keyword search might miss documents that use synonyms, while relying entirely on dense vector embeddings (encoding data as points in a high-dimensional space so related concepts are near each other) can overlook exact matches for specific names, product IDs, or acronyms.

By designing a multi-stage pipeline, we capture the best of both approaches. The first stage runs sparse retrieval and dense retrieval in parallel, then fuses the ranked lists with Reciprocal Rank Fusion (RRF)^{[3]Reference 3Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.https://dl.acm.org/doi/10.1145/1571941.1572114}. The second stage applies a more expensive reranker (often a cross-encoder or late-interaction model^{[4]Reference 4ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.https://arxiv.org/abs/2004.12832}) to a much smaller candidate set, passing only the most relevant chunks or parent passages to the LLM.

This visual outlines the flow of this multi-stage retrieval process.

Why hybrid retrieval?

BM25 (Best Match 25, a reliable keyword search algorithm) excels at exact keyword matching; dense retrieval captures semantic similarity. Dense retrieval typically uses bi-encoders (which encode queries and documents separately for fast lookup), though more advanced systems may use architectures like ColBERT (Contextualized Late Interaction over BERT) for late interaction between query and document tokens.^{[4]Reference 4ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.https://arxiv.org/abs/2004.12832} On a mixed workload, combining retrievers can surface candidates that either one would miss alone.

In practice, hybrid search isn't just "run two retrievers." You also need rank fusion, chunking, and evidence assembly. A common pattern is to embed short child chunks for recall, then retrieve the larger parent passage for synthesis so the model sees enough surrounding context to answer cleanly.

Why fuse on rank instead of raw score? BM25 and dense-similarity scores have different scales, so adding them without calibration is brittle. Reciprocal Rank Fusion scores documents from rank positions: score(d) = sum over lists of 1 / (k + rank(d)). Cormack et al. evaluated RRF with k = 60; treat that value as a starting point to validate, not a universal optimum.^{[3]Reference 3Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.https://dl.acm.org/doi/10.1145/1571941.1572114} Apply authorization filtering before fusion so a high-ranked forbidden document never becomes generation context.

Freshness and index tiers

Real search products rarely use one monolithic index. A useful pattern is a hot-warm split:

Freshness should influence ranking, not replace relevance. A common pattern is to combine retrieval score with time decay, then let the reranker decide whether the newer document is better. If you rank purely by recency, a low-quality post from five minutes ago can outrank the canonical source.

Latency budget

TTFT is a critical constraint of this system. The values below are an illustrative budget; validate them against product objectives and measured stage timings.

Note: If you support speculative retrieval, cheap lexical or dense search can begin before rewrite finishes. If you don't, planning and retrieval stack sequentially. Make that decision explicit when you budget TTFT.

3. LLM synthesis with grounded citations

The synthesis prompt can require citations and abstention, but prompting can't prove grounding. The product must keep source IDs attached to evidence, check generated claims, and decide what may be shown while support is pending or missing.

Grounding isn't just a prompt problem. Evidence packing matters too. Put the strongest chunks first, leave room for the model to answer, and avoid stuffing dozens of mediocre passages into the middle of a huge prompt. Long-context models still show "lost in the middle" behavior, where evidence in the center of the context window gets used less reliably.^{[5]Reference 5Lost in the Middle: How Language Models Use Long Contextshttps://arxiv.org/abs/2307.03172} That's one more reason reranking and tight top-K beat dumping everything into a large context window.

The citations a user sees can be a subset of retrieved evidence, but each visible citation must bind to a specific claim and its source span. Don't treat a source list as proof that every generated sentence is supported.

Before generation, pack allowed passages into a bounded context while preserving source IDs:

Keeping temperature low reduces answer variance, but it doesn't create factuality. The next deterministic example represents a support gate around generated claims: a real synthesizer can draft text, while publication depends on evidence checks.

4. Streaming architecture

For one-way progressive delivery, Server-Sent Events (SSE) work through the browser EventSource API over HTTP and are straightforward to render^{[6]Reference 6Using server-sent eventshttps://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events}. Streaming creates a product-policy choice: either hold factual claims until verification completes, or label streamed text as provisional and update or retract unsupported claims.

Streaming endpoint

This event contract fits a low-risk answer path that permits provisional rendering. It emits the plan and sources, declares pending support, streams text, then attaches the result of claim checks. A higher-risk workflow can omit text frames until support becomes supported.

In production, add periodic heartbeat events and configure upstream proxies not to buffer the stream. Also log state transitions so provisional output that becomes blocked can be audited.

---

Why answers still hallucinate and how to catch them

One of the most critical challenges in search is hallucination, when the LLM generates plausible but incorrect facts. Even with RAG, models can misinterpret complex details. Strong systems usually stack multiple checks rather than betting on one verifier:

NLI-based verification estimates whether a cited passage entails a claim rather than merely sharing words. TRUE^{[7]Reference 7TRUE: Re-evaluating Factual Consistency Evaluation.https://arxiv.org/abs/2204.04991} found that large-scale NLI methods and question-generation-plus-answering methods were strong automatic factual-consistency checks on its benchmarks, so NLI is a useful measured layer, not proof of correctness.

This function is deliberately only a lexical precheck. It can cheaply reject a missing phrase; it can't label entailment. Claims that pass it still need a calibrated verifier or a policy that keeps them provisional.

---

Skills to defend in review

Production questions to answer

How do you handle queries that need real-time data? Real-time data needs its own freshness pipeline. Use a hot index for newly crawled content plus direct routes to live APIs or structured stores for exact numeric fields. When the planner marks a query as freshness-sensitive, search the hot tier first, apply recency-aware scoring, then merge with the warmer corpus so the LLM sees both new evidence and canonical background context.

How would you detect and handle hallucinated citations? Decompose the generated answer into atomic claims, attach each claim to its cited source, and verify entailment with NLI or a calibrated lighter check. If support is weak or contradictory, flag the statement, remove it, or regenerate with stricter grounding and a smaller evidence set.

What's your latency budget for each pipeline stage? For a consumer-facing experience, a reasonable example SLO is roughly 0.8-1.5 seconds to first token. Query planning often fits in 10-50ms, retrieval fan-out in 150-500ms, reranking in 50-150ms, evidence packing in 10-30ms, and synthesis in roughly 300-800ms to first token. Split synthesis into prefill and decode because prompt length drives prefill cost while answer length drives decode cost.

How do you evaluate answer quality at scale? Start with reference-free metrics such as faithfulness, answer relevance, and context relevance. Add labeled retrieval metrics such as Recall@k or nDCG on benchmark sets, plus citation precision and unsupported-claim rate. In production, track citation click-through, reformulation rate, dwell time, abandonment, and explicit user feedback.

Common design failures

• Designing everything around a single LLM call instead of specialized routers, retrievers, rerankers, and verifiers.
• Treating structured or freshness-critical queries like generic vector search.
• Ignoring latency requirements for interactive search experiences.
• Skipping rigorous citation verification and trusting prompt instructions alone.
• Treating long context windows as a replacement for reranking and evidence packing.
• Overlooking inference cost per query when scaling to millions of users.

Common mistakes when building generative search

---

What running this costs

Building a search engine at scale requires careful cost management. Exact dollar figures drift with vendor pricing, model choice, and answer length, so reason from units of work rather than one fixed price table.

Before writing the formula, use scenario inputs rather than presenting provider pricing as fixed:

• Web retrieval: ~$0.002
• Reranking 200 query-document pairs: ~$0.001
• 1,500 prompt tokens + 400 output tokens at $0.003 per 1K tokens: ~$0.006
• Verifying 5 claims: ~$0.001

For these inputs, total cost is roughly $0.01 per query. At 1,000 QPS, that scenario approaches $600 per minute, which is why routing and caching need their own budgets.

$$\begin{aligned} \text{cost/query} \approx\;& C_{\text{search}} + N_{\text{pairs}} \cdot C_{\text{rerank}} \\ &+ (T_{\text{prompt}} + T_{\text{output}}) \cdot C_{\text{token}} \\ &+ N_{\text{claims}} \cdot C_{\text{verify}} \end{aligned}$$

Which stage dominates variable cost depends on provider rates, fan-out, prompt length, and verification policy. Measure the breakdown, then route cacheable or structured queries away from unnecessary synthesis and trim top-K where retrieval evaluation allows it.

Serving-side latency levers

TTFT can be dominated by retrieval, prompt assembly, or prefill; long answers can shift pressure toward decode. Profile the critical path before choosing an optimization.

PagedAttention improves KV-cache memory management and can increase concurrency when KV allocation is the binding constraint.^{[8]Reference 8Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} If decode is the bottleneck, speculative decoding can reduce latency when a draft model is accepted often enough; benchmark it for the answer shapes and target model in this service.^{[9]Reference 9Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192}

These serving optimizations matter only after retrieval depth and prompt length are under control. If you keep sending 40 mediocre passages to the model, no amount of serving cleverness will rescue TTFT.

GPU capacity planning

For a scenario with 1K QPS sustained and 1.5 seconds average latency per request:

$$\text{Concurrent requests} = \text{QPS} \times \text{avg latency} = 1000 \times 1.5 = 1500$$

This is Little's Law (average items in system = arrival rate × average time in system) in practice: at 1000 requests/second with 1.5-second average service time, around 1500 requests are in-flight simultaneously.

The next step is to translate concurrency into throughput and memory requirements for your actual serving stack. Don't assume a universal "requests per GPU" number. Measure your own model, context length, batching strategy, and quantization settings. PagedAttention improves packing efficiency and reduces KV-cache fragmentation under concurrent load^{[8]Reference 8Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}, but capacity still depends on empirical tokens-per-second benchmarks.

$$\text{GPUs needed} \approx \left\lceil \frac{\text{QPS} \times \text{avg output tokens}}{\text{effective decode tok/s/GPU}} \right\rceil$$

For long prompts, also budget prefill separately:

$$\text{Prefill capacity} \approx \frac{\text{QPS} \times \text{avg prompt tokens}}{\text{effective prefill tok/s/GPU}}$$

In practice, size the system against whichever pool saturates first: retriever GPUs, reranker GPUs, prefill workers, or decode workers. The answer model may be an expensive tier, but measured retrieval or reranking can still be the first bottleneck.

---

How to know it's working

RAGAS^{[10]Reference 10RAGAS: Automated Evaluation of Retrieval Augmented Generation.https://arxiv.org/abs/2309.15217} introduced reference-free evaluation around faithfulness, answer relevance, and context relevance. It's one offline layer when gold answers are scarce, not a replacement for labeled retrieval metrics, citation checks, or online product metrics: a response can be consistent with mediocre evidence and still fail the user.

This example wires a scorecard together with simple token-overlap proxies so it stays runnable. Its citation-resolution metric only checks whether a visible citation ID maps to an available passage; it doesn't prove claim support. These proxies aren't RAGAS or calibrated faithfulness judges. Replace them in an evaluation service while keeping labeled retrieval metrics separate.

---

Scaling without blowing the budget

As traffic grows, evaluate each latency or cost lever against quality and authorization tests:

---

Mastery check

Key concepts

• Offline ingest vs online query path
• Query planning, routing, and freshness detection
• Hybrid retrieval with BM25, dense retrieval, and rank fusion
• Reranking, evidence packing, and source-ID preservation
• Grounded synthesis with citations
• Claim-level verification with span checks and NLI
• Streaming answer delivery with SSE
• Latency budgeting, routing, and cost control
• Offline and online evaluation for search quality

What strong answers show

• Foundational: Explains why generative search needs both offline indexing and an online retrieval plus synthesis path
• Intermediate: Chooses when to route a query to structured APIs, hybrid retrieval, or fresh-web lookup
• Intermediate: Explains why hybrid sparse plus dense retrieval beats either retriever alone for developer-doc search
• Intermediate: Designs a TTFT budget that includes planning, retrieval, reranking, packing, and synthesis rather than treating latency as one LLM number
• Advanced: Designs claim-level verification that distinguishes quote matching, NLI, and selective fallback regeneration
• Advanced: Defends routing and caching rules that keep cost and latency under control at high QPS
• Advanced: Separates retrieval metrics, citation metrics, and online product metrics in one search evaluation loop

Follow-up questions

Common pitfalls

"One LLM call can replace search architecture"

• Symptom: Answers sound polished but use stale, missing, or unauthorized evidence.
• Cause: Retrieval, routing, and verification were collapsed into one generation step.
• Fix: Split the system into planner, retriever, reranker, evidence packer, synthesizer, and verifier. Keep trust boundaries outside the model.

"Dense retrieval is enough"

• Symptom: Search misses exact API versions, build IDs, or policy titles even though semantically related passages appear.
• Cause: Dense retrieval is good at paraphrases, not exact lexical anchors.
• Fix: Use hybrid sparse plus dense retrieval, then fuse ranks before reranking.

"Long context makes reranking optional"

• Symptom: The right document was retrieved, but the final answer still ignores or misstates it.
• Cause: Dumping many mediocre passages into context creates prompt bloat and lost-in-the-middle effects.
• Fix: Rerank aggressively, keep source IDs attached, and pack only the strongest evidence the answer model needs.

"Citations alone prove factuality"

• Symptom: Answers show footnotes but still contain unsupported or misleading claims.
• Cause: The system attached sources without checking whether each claim is entailed by the cited text.
• Fix: Add claim extraction plus span checks, NLI, and selective fallback regeneration for weak support.

"Cost tuning starts at the model endpoint"

• Symptom: Query cost spikes even after switching to a cheaper model.
• Cause: Retrieval fan-out, rerank pairs, long prompts, and verification work were left uncontrolled.
• Fix: Route simple queries earlier, trim top-K harder, cache safely, and reserve frontier synthesis for genuinely multi-hop requests.

Practice drill

Create a search-answer trace for one developer-doc query that mixes live facts and cited policy text:

1. Split the query into structured API calls, sparse retrieval, dense retrieval, reranking, evidence packing, synthesis, and claim verification.
2. Log which documents were removed by ACL filtering before fusion or reranking.
3. Add a claim-support table with atomic claim, cited span, entailment result, and final support state.
4. Mark latency budget by stage and choose one optimization that doesn't weaken authorization or citation support.

The trace should prove the answer was authorized, fresh, cited, and verifiable, not fluent alone.