Advanced RAG: HyDE & Self-RAG

A developer opens an internal API-docs assistant and types, "What about the other one?" They could mean the backup API key from the last turn, the second failed batch job, or the alternative embedding endpoint. Searching for "quota" can pull billing limits instead of the rate-limit policy. The index is healthy; the search object is not. Before changing ANN settings, ask what is missing: a referent, corpus vocabulary, or a dependency between facts.

The previous chapter showed how HNSW and IVF find nearby chunks quickly. Retrieval-augmented generation (RAG) grounds an answer in those chunks, while self-reflective RAG (Self-RAG) decides when retrieval and evidence checks are needed. That speed doesn't help when the search object is "the other one." A fast index returns the wrong neighborhood faster. Stay with this request and this docs index. Measure whether rewriting, Hypothetical Document Embeddings (HyDE), Self-RAG, or Corrective RAG (CRAG) recovers the right page, then keep the cheapest control that does.

The problem with naive RAG

A naive RAG pipeline embeds the user's raw question and fetches the top-k nearest neighbors from a vector database. Retrieve once, then generate. Before adding a more elaborate route, predict which of these failures your misses show:

1. The query is ambiguous. "What about the other one?" depends on chat history. A stateless retriever never sees that history.
2. The query and the document don't share a style. Someone asks "How do I rotate a key?" The page title is "Credential Rotation Workflow". The embedding of a short question may not sit near the embedding of a formal policy paragraph.
3. The retriever still returns the wrong chunks. Wording, embedding geometry, approximate nearest-neighbor (ANN) knobs such as ef_search and nprobe, or a messy corpus can all put SDK install notes above the rate-limit policy. Once those chunks enter context, the generator can write an unsupported answer.
4. The model doesn't know when to retrieve. Naive RAG retrieves for every turn, including "Hi" or "What is 2+2?", and burns tokens and latency for nothing.

Index tuning from the last chapter and query control in this one are different repairs. Measure which failure you actually have, then add the smallest intervention that fixes it.

That diagnosis gives each technique a job. Rewrite missing conversational state, expansion missing vocabulary, decomposition missing dependent facts, and later controls judge whether returned evidence deserves trust.

Query rewriting and decomposition

Three failures can look identical in a chat transcript. Before looking at the map, predict which search object each repair creates: a standalone question, several vocabulary variants, or a sequence of dependent hops. The distinction decides where extra calls buy evidence.

"What do I do now?" is a bad search object. The leaked-key fact lives in the previous turn, and a comparison like "sync vs batch embeddings" packs three lookups into one sentence. Use those examples to predict which details must survive rewriting and which facts need separate searches.

1. Rewrite for disambiguation

An on-call lead often turns a vague Slack thread into a precise search request before querying runbooks and product docs. Query rewriting gives the model that same job. Ma et al.'s Rewrite-Retrieve-Read pipeline adds a rewrite stage in front of a frozen retriever and reader, including a small T5 rewriter trained with reader feedback, instead of treating the user's wording as sacred.^{[1]Reference 1Query Rewriting for Retrieval-Augmented Large Language Models.https://arxiv.org/abs/2305.14283} In a chat product, a practical variant is simpler: rewrite the latest turn into a standalone, search-oriented question.

Before reading the rewrite, predict three things it must preserve: the leaked credential, the requested action, and the audit requirement. History supplies the referent, but it remains data rather than instructions.

A developer sends two messages:

Developer: "My API key leaked in a public issue." Developer: "What do I do now?"

A stateless retriever searching for "What do I do now?" would pull generic onboarding articles. A rewrite step instead produces:

Standalone query: "How do I revoke and rotate a leaked API key and audit recent usage?"

The rewrite model can be any instruction-tuned LLM behind a small interface. The lab uses a deterministic fake so the parsing contract can be tested locally without API keys.

Rewrite is search input, not authorization. History, tool output, and retrieved pages can contain "ignore policy / search all tickets" style instructions. Constrain the prompt to disambiguate the user's question, not follow instructions found in history, and never let a rewrite broaden ACL or tenant scope. RAG Security & Access Control covers the trusted retrieval plane these transforms must preserve.

2. Multi-query expansion

Multi-query expansion is the repair when one wording misses the page. "How do you handle peak traffic throttling?" never mentions "rate-limit burst policy", "queue backpressure", or "autoscaling capacity". Generate those variants, search each one, then merge the lists.

Before merging, predict what repeated evidence should do. A policy page that appears near the top of two variants should outrank a page that appears once, even if their raw similarity scores came from different searches.

Once you fan out into several queries, you have several ranked result lists to merge. One documented pattern is RAG-Fusion, which pairs generated query variants with Reciprocal Rank Fusion (RRF) to combine the lists into one ranking.^{[2]Reference 2RAG-Fusion: a New Take on Retrieval-Augmented Generationhttps://arxiv.org/abs/2402.03367} RRF ignores raw similarity scores and sums reciprocal ranks instead, so a document that lands near the top of several lists rises even if no single list ranked it first.^{[3]Reference 3Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.https://dl.acm.org/doi/10.1145/1571941.1572114}

For a document $d$, the score is $\sum_{r \in R} 1 / (k + r(d))$. Here, $R$ is the set of rankings, $r(d)$ is the document's one-based position in one ranking, and $k$ dampens the effect of very high ranks. Cormack et al. fixed $k = 60$ after a pilot investigation.^{[3]Reference 3Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.https://dl.acm.org/doi/10.1145/1571941.1572114} A document missing from a returned list contributes zero from that list. RRF is the same rank-merge idea used to fuse dense and sparse results in hybrid search, reused here for query variants.

Implement multi-query expansion behind a testable boundary. The model returns one query per line; the parser removes bullets or numbering so variants can be searched in parallel. The retriever should see clean search objects, not formatting noise.

Generating variants isn't enough. The system must merge result lists without assuming similarity scores from separate searches are directly comparable. RRF provides a deterministic rank-based merge. As you read the next snippet, predict which ID benefits most from appearing near the top across all three lists:

3. Query decomposition (least-to-most)

Decomposition is the repair when one search can't collect every fact. Take this comparison:

"Compare synchronous and batch embedding API latency, limits, and retry behavior."

No single policy page is that comparison. Before splitting, ask whether the final comparison can be answered before endpoint facts exist. That dependency blocks it, so split the question into hops the index can hit:

1. "What latency does the synchronous embedding endpoint target?"
2. "What throughput and completion limits does the batch embedding endpoint have?"
3. "What retry behavior applies to each endpoint?"
4. "Compare synchronous and batch embeddings given the latency, limit, and retry facts."

This pattern is closely related to least-to-most prompting, which breaks hard problems into simpler steps.^{[4]Reference 4Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelshttps://arxiv.org/abs/2205.10625} In RAG, teams reuse that idea for retrieval coverage rather than chain-of-thought supervision. The first three hops gather explicit evidence; the final hop synthesizes it. That dependency improves coverage, but adds queries and a chance that an early miss poisons later hops. Measure supported-answer accuracy and latency before releasing it.

A real model would produce the sub-questions. The rest of the system should only depend on the line-oriented contract.

4. Step-back prompting (abstraction)

When a query is overloaded with specific runtime parameters, error codes, or customer-specific configurations, direct dense search often fails if the knowledge corpus documents high-level architecture rules, rate limits, or protocols instead of the exact numerical combination.

Step-back prompting abstracts the specific inquiry into a broader, foundational question.^{[5]Reference 5Take a Step Back: Evoking Reasoning via Step-Back Prompting in Large Language Models.https://arxiv.org/abs/2310.06117} Zheng et al. prompt the model to "step back" and identify the overarching concept or policy behind the question.

Predict which evidence each query can recover. The specific query should find incident details; the broader query should find governing policy. Neither query alone can reliably provide both.

For instance, consider an operational question:

Specific query: "Why did our client receive an HTTP 429 when sending 120 requests/minute on the tier 2 embedding endpoint?" Step-back query: "What are the rate limits, burst thresholds, and quota policies for tier 2 embedding endpoints?"

In RAG, the retriever queries the index for both the original specific question and the abstracted step-back question. The step-back search retrieves foundational rules and architecture guides, while the specific search retrieves endpoint parameters. The generator conditions on both evidence sets, keeping broad policy from replacing exact diagnosis.

Implement step-back generation behind a clear interface that pairs the original question with its conceptual counterpart. The pair is the boundary: later code can choose retrieval behavior without losing the original wording:

When should you rewrite a query instead of searching the raw user message?

Answer

Rewrite when the latest message depends on chat history, uses vague language, or combines multiple intents. The retriever needs a standalone search request even if the generator later sees the original wording.

These transforms change the search object before the same hybrid index from earlier chapters runs. They don't grade the chunks that come back. HyDE goes one step further: it changes the kind of vector used for search. That makes proxy quality and evidence boundaries matter.

HyDE (Hypothetical Document Embeddings)

Standard dense retrieval matches a query embedding to document embeddings. Queries are often short and interrogative, while indexed passages are longer and declarative. HyDE changes how the query vector is built when that mismatch hurts retrieval.

HyDE (Hypothetical Document Embeddings) generates one or more hypothetical documents, embeds those document-style proxies, and retrieves real corpus passages near the resulting vector.^{[6]Reference 6Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496} Gao et al. prompt the model to "write a document that answers the question." If you need a throughput-limit clause but can't remember the wording, don't search "Can we handle more traffic?" Write a one-paragraph summary of the policy you expect and search for documents that look like that paragraph.

Hold on to the launch-week question. Predict what changes: the source of the vector, not the truth of the answer. A short question may sit near generic API prose; a document-style proxy may sit near the quota and batch policies.

A developer asks:

Query: "Can we increase embedding throughput during a launch week?"

A standard dense retriever might embed the short question and pull generic "embedding API overview" articles that don't mention launch traffic. HyDE instead prompts the model to write a hypothetical policy paragraph:

Hypothetical document: "Embedding throughput increases: teams can request a temporary tokens-per-minute quota increase, use the batch embedding endpoint for offline jobs, shard requests across approved projects, and apply exponential backoff when rate limits are returned..."

That generated paragraph is longer, declarative, and uses vocabulary like "tokens-per-minute quota", "batch embedding endpoint", and "exponential backoff". It's an illustrative search proxy, not an answer. The generated limit may be wrong, so only retrieved source text may support the final response.

Cache HyDE proxies and rewrite strings with tenant- and principal-scoped keys (plus policy version), not raw query text alone. A shared string key lets Tenant A's proxy or rewrite serve Tenant B's retrieval path.

How HyDE works

The original HyDE pipeline has three phases:

1. Generate: Prompt an instruction-tuned language model to sample one or more hypothetical passages for the query. The passages may contain fabricated details, but they can still capture relevance patterns that look like real documents.^{[6]Reference 6Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496}
2. Embed: Encode each sampled passage with a document encoder such as Contriever. Average the vectors to estimate the expected hypothetical-document embedding:^{[6]Reference 6Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496}

$\hat{\mathbf{v}}_q = \frac{1}{N}\sum_{k=1}^{N} f(\hat{d}_k)$

Here $f$ is the document encoder, $\hat{d}_k$ is the $k$-th sampled proxy, and $N$ is the number of samples. Gao et al. also try averaging in the original query vector as one extra hypothesis. 3. Retrieve: Search the corpus with that averaged vector. The paper's key intuition is that the encoder's dense bottleneck filters much of the fabricated detail while preserving the semantic neighborhood of relevant documents.^{[6]Reference 6Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496}

The production version uses a real generator and dense encoder. This small runnable version deliberately generates one proxy and uses keyword sets so the retrieval contract is visible: the query first becomes a document-like paragraph, then the retriever searches with that paragraph rather than the original question. As the snippet runs, predict why the two policy chunks should outrank the generic overview.

The keyword sketch shows the contract. In the paper, the actual query vector is an average of document embeddings. With two 3-d proxies below, calculate each coordinate before running it: add the two values, then divide by two:

HyDE changes the search object first, then retrieves. In the sketch below, the short question sits away from the policy cluster, while the proxy sits next to the quota and batch chunks. Ask what still remains unproven after the proxy moves closer.

On TREC DL19, Gao et al. report Contriever nDCG@10 of 44.5 and HyDE nDCG@10 of 61.3. DL20 nDCG@10 moves from 42.1 to 57.9, and they also report gains on BEIR and multilingual Mr. TyDi.^{[6]Reference 6Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496} A proxy can still invent a high-impact identifier or policy detail, so transfer to your corpus is a hypothesis to test.

When to use HyDE

HyDE was designed for zero-shot retrieval without relevance labels. Gao et al. evaluate it on web search, BEIR, and multilingual Mr. TyDi tasks.^{[6]Reference 6Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496} Use it when users phrase questions differently from stored documents. Skip it when the query already is the identifier you must keep.

Gate HyDE away from exact-match lookups such as IDs, dates, prices, or error codes. That's an engineering inference from the mechanism, not a claim from the paper: if a proxy invents a precise fact, retrieval can drift toward text that echoes the invention instead of the source chunk.

Before the router runs, classify each example: preserve the incident ID; reshape the conceptual throughput question. That boundary keeps a useful proxy from rewriting facts that must match exactly.

Why should HyDE be gated away from incident IDs, error codes, dates, and prices?

Answer

The hypothetical document can invent precise details. For exact lookups, invented values can pull retrieval toward the wrong chunks, while ordinary hybrid search preserves the user's exact identifier.

Self-RAG (Self-reflective RAG)

Many retrieve-then-generate pipelines fetch context once without a model-generated critique step. Self-RAG instead fine-tunes a generator to emit special reflection tokens that control retrieval and score candidate generation segments.^{[7]Reference 7Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.https://arxiv.org/abs/2310.11511} A relevant passage can still fail to support a particular claim, so ask two questions separately: should the model retrieve, and does the returned evidence support its next claim?

Reflection tokens

Self-RAG uses one retrieval token family and three critique token families:^{[7]Reference 7Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.https://arxiv.org/abs/2310.11511}

1. Retrieve with values Yes, No, or Continue. This decides whether the model should fetch evidence before generating the next segment. Continue keeps retrieval going for later segments.
2. ISREL with labels such as Relevant or Irrelevant. This scores whether a retrieved passage is helpful for the current query or segment.
3. ISSUP with labels Fully supported, Partially supported, and No support. This checks whether the generated claim is grounded in retrieved evidence.
4. ISUSE with utility scores from 1 to 5. This measures how useful the final response is for the user.

Paper examples serialize these as inline control tags such as [Retrieve=Yes], [ISREL=Relevant], [ISSUP=Fully Supported], and [ISUSE=4]. The exact bracket syntax is less important than the four decision families, but each critique tag keeps its family name visible.

Walk the same launch-throughput query through those tokens. Before reading the sequence, predict the first decision and final support label: the model needs policy evidence, and the generated claim must remain narrower than any invented detail. A Self-RAG model might emit:

1. [Retrieve=Yes], the model decides it needs evidence before answering.
2. It retrieves a passage: "Temporary tokens-per-minute increases require approval and backoff-aware clients..."
3. [ISREL=Relevant], the passage is useful for the current segment.
4. The model generates: "For launch traffic, request a temporary TPM increase and keep exponential backoff enabled."
5. [ISSUP=Fully Supported], the claim is grounded in the retrieved text.
6. [ISUSE=4], the response is helpful but could be more detailed.

Without reflection tokens, a standard RAG pipeline might have retrieved the same passage without explicitly scoring passage relevance or claim support. Reflection tokens expose the model's predicted judgments for scoring and control; they aren't proof that a claim is true.

Architecture

Self-RAG is more than "retrieve once, then critique at the end." At inference time it can emit a retrieval decision, retrieve top-k passages on demand, generate candidate segments conditioned on different passages in parallel, and score those branches with reflection-token probabilities.^{[7]Reference 7Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.https://arxiv.org/abs/2310.11511}

Asai et al. run a segment-level beam search (default beam width 2) and rank each continuation with a weighted sum of the probabilities of the most desirable critique tokens. Their reported default weights are 1.0 for ISREL, 1.0 for ISSUP, and 0.5 for ISUSE; they raise the support weight when they want citation precision.^{[7]Reference 7Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.https://arxiv.org/abs/2310.11511} Before looking at branches, predict why a support-heavy product policy should reject a fluent claim that says launch traffic is unlimited. That decoding loop is what makes Self-RAG distinct from a simple prompted guardrail.

Implementation note

Deploying a true Self-RAG system requires a generator specifically fine-tuned to emit these reflection tokens during generation. Asai et al. train Llama 2 7B and 13B generators that way, with a separate Llama 2 7B critic that inserts reflection tokens into supervised examples offline before generator training.^{[7]Reference 7Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.https://arxiv.org/abs/2310.11511} At inference the generator emits the tokens itself; you don't host the critic.

A prompted frontier model can imitate parts of this control loop, but that isn't the same system. Without reflection-token fine-tuning, you're building a Self-RAG-inspired agentic pipeline: separate routing, retrieval grading, and answer validation calls stitched together in application code.

Self-RAG cost comes from on-demand retrieval plus branching over multiple passages and scoring those branches, not from a few extra control tokens. Evaluate it where support-aware generation is worth that additional serving path.

The scoring sketch below doesn't implement Self-RAG training. It ranks two branches once a trained model has supplied relevance, support, and utility probabilities. Use the 0.2 / 0.6 / 0.2 policy to predict which branch survives before reading code. It puts more weight on support than Asai et al.'s 1.0 / 1.0 / 0.5 default, but either score still needs source evidence.

CRAG (Corrective RAG)

Corrective Retrieval-Augmented Generation (CRAG) focuses on correction after imperfect retrieval. It adds a lightweight Retrieval Evaluator that scores retrieved question-document pairs and routes the request before generation.^{[8]Reference 8Corrective Retrieval Augmented Generation.https://arxiv.org/abs/2401.15884}

The evaluator

In the paper, the evaluator is a lightweight T5-large model fine-tuned to score each retrieved question-document pair on a $[-1, 1]$ range, then threshold those scores into one of three actions:^{[8]Reference 8Corrective Retrieval Augmented Generation.https://arxiv.org/abs/2401.15884}

With the teaching thresholds below, predict the route for best scores 0.81, 0.17, and 0.45. Numbers are only a check; the decision boundary between usable, unusable, and uncertain evidence is what matters.

1. Correct: At least one retrieved document clears the upper threshold. Refine internal knowledge and answer from it.
2. Incorrect: All retrieved documents fall below the lower threshold. Discard them and fall back to web search.
3. Ambiguous: Scores land between those two cases. Combine refined internal evidence with web results.

Yan et al. set those thresholds empirically per dataset, for example $0.59$ and $-0.99$ on PopQA.^{[8]Reference 8Corrective Retrieval Augmented Generation.https://arxiv.org/abs/2401.15884} The sketch below uses a 0-1 teaching scale (upper=0.7, lower=0.2) so you can check the same three-way rule by hand. Don't copy those teaching numbers into production.

A developer asks about "current embedding TPM increase process." The internal knowledge base has weak coverage, so the evaluator might score the retrieved internal documents as Incorrect, triggering a web-search fallback. A production system still needs source allowlists and citation checks before trusting those web results.

If the internal documents are somewhat relevant but incomplete, the evaluator returns Ambiguous, and CRAG combines refined internal strips with web results. That extra branch is correction, not proof: external text remains untrusted until source and support checks pass.

The distinction from Self-RAG is where correction happens. CRAG doesn't train the generator to emit reflection tokens. Instead, it inserts a separate evaluator between retrieval and generation, then uses that evaluator to trigger correction paths.

Before running the classifier, map each best score to a path: 0.81 should stay internal, 0.17 should leave it, and 0.45 should combine sources.

What separates true Self-RAG from a prompted critique loop?

Answer

True Self-RAG uses a generator trained to emit retrieval and critique tokens during generation. A prompted critique loop can be useful, but it's application orchestration around a standard model.

Knowledge refinement

Even relevant documents contain noise. CRAG includes a decompose-then-recompose step. Yan et al. split a retrieved document into strips of a few sentences, score each strip with the same evaluator, drop strips below a filter threshold (they use $-0.5$), and concatenate the rest in order, keeping a small top-$k$.^{[8]Reference 8Corrective Retrieval Augmented Generation.https://arxiv.org/abs/2401.15884}

The paper scores each retrieved document, then applies thresholds. The sketch below collapses that into a classify helper and uses keyword checks for strip filtering so you can run it without a T5 evaluator. Watch the boundary: a correction route can discard noisy sentences, but a missing fact stays missing.

If you instantiate this class with real components and run crag.run("current embedding TPM increase process"), the evaluator might return "incorrect" because internal docs lack coverage. The pipeline would then call web_search.search(...) and pass the web results through refine_knowledge before generating the answer.

Agentic and iterative retrieval

Self-RAG and CRAG both introduce feedback, but at different boundaries: Self-RAG can make retrieval and critique decisions while generating segments, while CRAG routes after initial retrieval. An application can generalize that feedback into agentic retrieval: instead of one fixed retrieve-then-generate pass, a model with tool access searches, reads results, tests whether evidence is sufficient, and either answers or searches again with a refined query.

The mechanism is older than the agent framing. IRCoT studies one version of this pattern: retrieve, reason a step, use that step to drive the next retrieval, and repeat until the chain of evidence is complete.^{[9]Reference 9Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionshttps://arxiv.org/abs/2212.10509} Multi-hop questions like "Which embedding endpoint has the highest p95 latency, and what retry budget applies to that endpoint?" need exactly this, because the answer to the second part depends on resolving the first.

Before choosing a rung, predict the dependency in the request above. A single-fact FAQ should stop early; endpoint selection followed by a retry lookup must carry evidence from one hop into the next. Each escalation costs latency and tokens:

1. Naive RAG for single-fact lookups answerable by one retrieval.
2. Query rewriting plus hybrid search and reranking for conversational ambiguity and vocabulary mismatch. This is a practical baseline to evaluate before more expensive routes.
3. HyDE behind a router for abstract or vocabulary-mismatched queries where a document-style proxy helps.
4. A correction gate (CRAG-style evaluator or prompted critique) when retrieval quality is inconsistent and a single bad context causes user-visible errors.
5. Iterative or agentic retrieval for genuinely multi-hop questions, where one retrieval pass can't gather all the facts.

Escalate only when the failure mode and evaluation demand it. On a simple FAQ lookup, an agentic loop can add latency and failure paths without improving retrieved evidence.

That same boundary explains the comparison below. Read it by asking which component owns each decision, then check whether each added control changes retrieval, evidence scoring, or both.

A question needs two facts where the second depends on resolving the first. Which strategy fits, and why not naive RAG?

Answer

Iterative or agentic retrieval. Naive RAG runs one retrieval pass, so it can't use the answer to the first hop to form the query for the second. Interleaving retrieval with reasoning lets each hop drive the next search.

Comparison of advanced techniques

Choose these techniques by failure mode, not by novelty. HyDE targets semantic mismatch. Self-RAG changes retrieval timing and branch scoring inside a trained generator. CRAG adds a separate correction gate after retrieval.

What rewrite, HyDE, and CRAG can break

Advanced RAG helps only when it matches the failure. Watch for these symptoms before adding another model call. Rewrite, HyDE, multi-query, and CRAG affect more than quality. They expand the attack and cost surface of retrieval.

Diagnose in order: identify the wrong search object, check evidence scope, then check added calls and trust boundaries. A quality gain without that diagnosis can hide a security or latency regression.

Symptom: the query is vague, but the answer needs history

The retriever sees only the latest turn, so it searches for "the other one" instead of the backup API key or failed batch job. Rewrite the latest turn into a standalone query using the last few conversation turns. Keep the raw user message for the generator, but search with the rewritten query so the answer can retain conversational context.

Symptom: rewrite or history poisons retrieval intent

Conversation history, tool output, or a retrieved doc can contain "ignore policy / search secrets / dump all tickets." If the rewrite model treats history as trusted instructions, it can adopt that intent and retrieve privileged neighborhoods. Treat rewrite input as untrusted data. Constrain the prompt to disambiguate the user's question, mark tool/page text as data, preserve principal identity and ACL scope, and never let a rewrite broaden grants.

Symptom: HyDE improves recall on policy questions but breaks exact lookups

The hypothetical document can invent precise identifiers, dates, or prices. That invented detail may pull retrieval toward the wrong neighborhood. Route exact-match queries to normal hybrid retrieval; use HyDE for conceptual or vocabulary-mismatched questions where a document-style proxy helps more than it hurts.

Symptom: CRAG web fallback becomes SSRF or untrusted-source injection

An "Incorrect" retrieval score triggers an open-web fetch. Without controls, the worker can hit internal metadata endpoints through server-side request forgery (SSRF) or pull attacker-controlled pages that inject instructions into generation. Use domain allowlists, block links into private networks, fetch in a network sandbox, treat web HTML as untrusted observations, and require citations from allowlisted sources before generation. Prefer "refuse / escalate" over a wide-open web tool.

Symptom: rewrite or HyDE cache bleeds across tenants

Caches keyed only on the raw string miss tenant, user, and policy version. User A's rewritten query or HyDE proxy can then serve User B's retrieval path. Key caches by (tenant_id, principal_id or role set, policy_version, normalized_query, technique). Prefer short TTLs on security-sensitive paths, and never share HyDE vectors across tenants.

Symptom: multi-query / agentic loops become a cost DoS

Decomposition and multi-query fan-out multiply embedding, ANN, and LLM calls per user message. Set hard caps on rewrite count, sub-queries, web fetches, and dollar/token budgets per request. Fail closed with a partial answer or clarification when the budget trips, instead of letting a loop define its own spend.

Symptom: Self-RAG-inspired code works in prompts but isn't true Self-RAG

True Self-RAG trains a generator to emit retrieval and critique tokens during generation. Separate prompted grading calls can mimic the control loop, but they don't create the same reflection-token model. Name the system honestly: call it a prompted critique loop or CRAG-style evaluator unless you're hosting a model trained with Self-RAG reflection tokens.

Symptom: accuracy rises, but users feel the product is slow

Rewriting, HyDE generation, dense retrieval, sparse retrieval, reranking, grading, and final synthesis can become a long sequential path. Parallelize independent retrievals, cap reranker candidates, cache repeated rewrites or hypothetical documents with tenant-safe keys, and measure retrieval and generation latency separately. If supported-answer quality misses its gate, lower latency alone isn't a release win.

Add one control at a time

Don't stack rewrite, HyDE, CRAG, and an agent loop on the first launch-throughput query. Treat each control as an experiment: name the failure, add one intervention, and compare against the same labeled cases. Add a control only when a measured failure needs it:

Stable baseline

Start with query rewriting and hybrid search (dense retrieval + sparse keyword retrieval). This fixes basic vocabulary mismatch and conversational ambiguity for a low cost: one extra LLM call for rewriting. That baseline lets every later route earn its complexity.

Add reranking

Add a cross-encoder reranker after retrieval when initial retrieval has decent recall but final candidates are noisy. It can improve precision, at moderate cost: cross-encoders are slower than initial retrieval, so keep candidate count small.

Specialized handling (HyDE / router)

Use a router to classify queries. For a conceptual request such as "Explain launch-week embedding throughput planning," choose HyDE because vocabulary mismatch is likely. For a precise request such as "What happened in incident INC-48291?", choose standard retrieval because HyDE risks inventing the exact identifier. The router pays HyDE's extra generation only where document-style proxy helps.

Add correction loops

If accuracy is still insufficient, add a prompted critique loop or a CRAG-style evaluator before investing in a true Self-RAG model. Prompt the model or a lightweight evaluator to grade retrieved documents before answer generation; when evidence is weak, trigger a rewrite, a second retrieval pass, or web fallback. This captures much of the reliability gain without requiring reflection-token fine-tuning.

Latency is the production constraint in advanced RAG. A pipeline with rewriting, HyDE, retrieval, reranking, grading, and generation turns one answer into several sequential model and retrieval steps. Stream final generation, fetch dense and sparse results in parallel, and cache reusable artifacts when traffic is repetitive. Track each stage so a quality gain can be tied to its added work.

Don't release a more elaborate route because it improves a few anecdotes. Compare supported-answer quality and latency on a labeled set, then release only paths that meet both requirements. For each route, retain the corpus snapshot, model and retriever versions, candidate count, latency percentiles, token or dollar cost, and citation-support results. Without that evidence, route comparisons are not reproducible.

Before running the selector below, predict which route clears both gates: supported accuracy at least 0.93 and p95 no more than 350 ms.

Try it yourself

Stay with five API-docs messages and measure top-3 evidence hits. This exercise needn't reproduce paper scores. It should connect each retrieval miss to one intervention and one release decision.

Exercise: Build a query rewriter for a developer-support assistant

Collect five developer-support messages from an API docs assistant (or write realistic ones). Include one ambiguous message that needs conversation history, one complex comparison, and one vague keyword so each control has a case to test.

Step 1, Rewrite: Write a Python function that takes a developer message plus the last two turns of chat history and outputs a standalone query. Run it on the five messages and inspect the results. Check whether the rewritten query contains full intent while leaving authorization and tenant scope unchanged.

Step 2, Measure: For each original message, manually decide which internal policy documents should be retrieved. Then run the rewritten query through a simple dense-retrieval setup (even a small embedding model like all-MiniLM-L6-v2 against a dozen policy chunks). Count how many of the top-3 results match the manual gold set. The rewrite should improve hit rate for ambiguous and vague cases.

Step 3, Diagnose: Pick one message where retrieval still fails. Before changing code, predict the failure: vocabulary mismatch (try multi-query expansion), semantic asymmetry (try HyDE), or weak evidence (try a CRAG-style evaluator). Implement only that fix and measure again.

Expected outcome: Record which interventions improve top-3 evidence hits or catch weak retrieval on the examples, and which add latency without a gain. A small exercise may not reproduce paper results; its value is exposing the measurement loop.

Read the measurements before choosing a route

The exercise should leave you with evidence about retrieval quality, latency, and the shape of each failure. Turn those observations into a route decision before adding a more expensive path. Three deployment questions provide the boundary.

When does HyDE justify its latency cost?

HyDE adds at least one extra generation call before retrieval. That cost is worth paying only when measured retrieval quality improves enough to justify it. A smaller instruction model may be adequate for the hypothetical document, but evaluate it rather than assuming equivalence. Cache repeated proxy documents only when generation configuration and source policy make reuse valid. For exact-match lookups like incident IDs, error codes, or dates, skip HyDE.

When does Self-RAG degrade instead of help?

Self-RAG degrades when retrieval itself is weak or when the model's learned critique tokens stop correlating with real answer quality. It also raises inference cost because the model may retrieve on demand, branch over multiple passages, and spend extra decoding steps on critique tokens before choosing a continuation. For a simple FAQ bot, first measure a cheaper baseline such as rewrite plus reranking or a correction gate before committing to a specialized Self-RAG serving path.

Can you combine HyDE and Self-RAG?

As a system design, you can use HyDE to propose initial candidate passages, then let a true Self-RAG model score branches with ISREL, ISSUP, and ISUSE. That composition needs its own evaluation; neither mechanism guarantees the other improves it. If you don't have a reflection-token model, describe the composition as HyDE plus a CRAG-style evaluator or prompted support checks rather than Self-RAG.