Advanced RAG: HyDE & Self-RAG

A developer opens an internal AI docs assistant and types, "What about the other one?" They could mean the backup API key discussed earlier, the second failed batch job, or the alternative embedding endpoint. An on-call engineer can hit a similar problem by searching for "quota" and getting billing limits instead of the rate-limit policy. Search fails when the user's words don't line up with the evidence they need.

Vector database internals showed how an index finds nearby chunks quickly. Now move one layer above the index: use retrieval-augmented generation (RAG) controls to rewrite messy requests, search with HyDE (Hypothetical Document Embeddings), critique generation with Self-RAG (Self-reflective RAG), or correct weak retrieval with Corrective Retrieval-Augmented Generation (CRAG). Add the smallest intervention that repairs a measured failure.

The problem with naive RAG

A standard RAG implementation (often called "Naive RAG") typically embeds the user's raw query and fetches the top-k nearest neighbors from a vector database. This simple "retrieve-then-generate" pipeline suffers from systematic failure modes:

1. The query is ambiguous: "What about the other one?" relies entirely on conversation history about a failed batch job, which a stateless retriever lacks.
2. The query and document are semantically misaligned: A user asks "How do I rotate a key?" but the documentation is titled "Credential Rotation Workflow". The embedding of the conversational question may not match the embedding of the formal security document.
3. The retriever returns irrelevant documents: Embedding geometry, query wording, approximate nearest-neighbor (ANN) settings, or corpus quality can all put SDK install notes above the rate-limit policy. If irrelevant chunks enter context, the generator may produce an unsupported answer.
4. The model doesn't know when to retrieve: Naive RAG retrieves for every query, even "Hi" or "What is 2+2?", wasting tokens and latency.

These failure modes are common in real applications. A useful production pipeline measures them separately, then adds the smallest retrieval intervention that repairs the observed failure rather than making every question pay for an elaborate loop.

Query rewriting and decomposition

User queries are rarely optimal for vector search. They're often short, lack context, or contain multiple distinct questions about endpoints, limits, errors, and credentials all at once.

1. Rewrite for disambiguation

An on-call lead often turns a vague Slack thread into a precise search request before querying runbooks and product docs. Query rewriting gives the model that same job. Conversational queries often rely on implicit context. Ma et al.'s Rewrite-Retrieve-Read pipeline adds a dedicated rewrite stage before retrieval instead of treating the user's wording as sacred.^{[1]Reference 1Query Rewriting for Retrieval-Augmented Large Language Models.https://arxiv.org/abs/2305.14283} In a chat product, a practical variant is to rewrite the latest turn into a standalone, search-oriented question.

Here's a concrete example. A developer sends two messages:

Developer: "My API key leaked in a public issue." Developer: "What do I do now?"

A stateless retriever searching for "What do I do now?" would pull generic onboarding articles. A rewrite step instead produces:

Standalone query: "How do I revoke and rotate a leaked API key and audit recent usage?"

In production, the rewrite model can be any instruction-tuned LLM behind a small interface. Start with a copy-runnable sketch that uses a deterministic fake model so the parsing contract can be tested locally without API keys.

Rewrite input is untrusted text. History, tool output, and retrieved pages can contain "ignore policy / search all tickets" style instructions. Constrain the rewrite prompt to disambiguate the user's question, not to follow instructions found in history, and never let a rewrite broaden ACL or tenant scope. RAG Security & Access Control covers the trusted retrieval plane these transforms must preserve.

2. Multi-query expansion

A single query might miss relevant documents due to vocabulary mismatch. Multi-Query Expansion generates synonymous queries to improve recall.

For example, if a user asks "How do you handle peak traffic throttling?", relevant internal documents might use terms like "rate-limit burst policy", "queue backpressure", or "autoscaling capacity". To improve recall across different vocabularies, the retriever can generate query variations automatically.

Once you fan out into several queries, you have several ranked result lists to merge. One documented pattern is RAG-Fusion, which pairs generated query variants with Reciprocal Rank Fusion (RRF) to combine the lists into one ranking.^{[2]Reference 2RAG-Fusion: a New Take on Retrieval-Augmented Generationhttps://arxiv.org/abs/2402.03367} RRF ignores raw similarity scores and sums reciprocal ranks instead, so a document that lands near the top of several lists rises even if no single list ranked it first.^{[3]Reference 3Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.https://dl.acm.org/doi/10.1145/1571941.1572114}

For a document $d$, the score is $\sum_{r \in R} 1 / (k + r(d))$. Here, $R$ is the set of rankings, $r(d)$ is the document's one-based position in one ranking, and $k$ dampens the effect of very high ranks. Cormack et al. fixed $k = 60$ after a pilot investigation.^{[3]Reference 3Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.https://dl.acm.org/doi/10.1145/1571941.1572114} A document missing from a returned list contributes zero from that list. RRF is the same rank-merge idea used to fuse dense and sparse results in hybrid search, reused here for query variants.

Implement multi-query expansion behind a testable boundary. The model returns one query per line, and the parser removes bullets or numbering so all variants can be searched in parallel.

Generating variants isn't enough; the system must merge their result lists without assuming similarity scores from separate searches are directly comparable. RRF provides a deterministic rank-based merge:

3. Query decomposition (least-to-most)

Complex questions often require multiple retrieval steps, as a single search may fail to gather all the necessary facts. Decomposition breaks a complex query into a series of simpler sub-queries that can be executed sequentially or in parallel.

For instance, consider the following analytical query:

"Compare synchronous and batch embedding API latency, limits, and retry behavior."

A standard retriever might struggle to find a single document containing this exact comparison. Instead, we decompose it into sub-questions:

1. "What latency does the synchronous embedding endpoint target?"
2. "What throughput and completion limits does the batch embedding endpoint have?"
3. "What retry behavior applies to each endpoint?"
4. "Compare synchronous and batch embeddings given the latency, limit, and retry facts."

This decomposition pattern is closely related to least-to-most prompting, which decomposes hard problems into simpler steps.^{[4]Reference 4Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelshttps://arxiv.org/abs/2205.10625} In RAG, teams reuse that idea for retrieval coverage rather than chain-of-thought supervision. Answering simpler questions first can gather explicit evidence for each sub-fact before synthesis, but it also adds queries and possible error propagation. Measure supported-answer accuracy and latency before releasing it.

Here's a simple implementation. A real model would produce the sub-questions, but the rest of the system should only depend on the line-oriented contract.

When should you rewrite a query instead of searching the raw user message?

Answer

Rewrite when the latest message depends on chat history, uses vague language, or combines multiple intents. The retriever needs a standalone search request even if the generator later sees the original wording.

HyDE (Hypothetical Document Embeddings)

Standard dense retrieval matches a query embedding to document embeddings. Queries are often short and interrogative, while indexed passages are longer and declarative. HyDE changes how the query vector is built when that mismatch hurts retrieval.

HyDE (Hypothetical Document Embeddings) bridges this gap by generating one or more hypothetical documents, embedding those document-style proxies, and retrieving real corpus passages near the resulting vector.^{[5]Reference 5Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496} In Gao et al., the model is prompted to "write a document that answers the question." Suppose you need a specific throughput-limit clause but can't remember the exact wording. Instead of asking the knowledge base "Can we handle more traffic?", you write a one-paragraph summary of what you expect the policy to say and ask, "Where are documents that look like this paragraph?"

Here's a concrete API-doc example. A developer asks:

Query: "Can we increase embedding throughput during a launch week?"

A standard dense retriever might embed the short question and pull generic "embedding API overview" articles that don't mention launch traffic. HyDE instead prompts the model to write a hypothetical policy paragraph:

Hypothetical document: "Embedding throughput increases: teams can request a temporary tokens-per-minute quota increase, use the batch embedding endpoint for offline jobs, shard requests across approved projects, and apply exponential backoff when rate limits are returned..."

That generated paragraph is longer, declarative, and uses vocabulary like "tokens-per-minute quota", "batch embedding endpoint", and "exponential backoff". It's an illustrative search proxy, not an answer: the generated limit may be wrong, and only retrieved source text may support the final response.

Cache HyDE proxies and rewrite strings with tenant- and principal-scoped keys (plus policy version), not raw query text alone. A shared string key lets Tenant A's proxy or rewrite serve Tenant B's retrieval path.

How HyDE works

The original HyDE pipeline has three phases:

1. Generate: Prompt an instruction-tuned language model to sample one or more hypothetical passages for the query. The passages may contain fabricated details, but they can still capture relevance patterns that look like real documents.^{[5]Reference 5Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496}
2. Embed: Encode each sampled passage with a document encoder such as Contriever, then average the vectors to estimate the expected hypothetical-document embedding.^{[5]Reference 5Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496}
3. Retrieve: Search the corpus with that averaged vector. The paper's key intuition is that the encoder's dense bottleneck filters much of the fabricated detail while preserving the semantic neighborhood of relevant documents.^{[5]Reference 5Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496}

The production version uses a real generator and dense encoder. This small runnable version deliberately generates one proxy and uses keyword sets so the retrieval contract is visible: the query first becomes a document-like paragraph, then the retriever searches with that paragraph rather than the original question.

HyDE flow is simple but important: change the search object first, then retrieve.

Across the paper's zero-shot experiments, HyDE improved retrieval over the underlying Contriever or mContriever baseline.^{[5]Reference 5Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496} The mechanism can still fail on a new corpus, especially when a proxy invents a high-impact identifier or policy detail.

When to use HyDE

HyDE was designed for zero-shot retrieval without relevance labels. Gao et al. evaluate it on web search, BEIR, and multilingual Mr. TyDi tasks.^{[5]Reference 5Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496} Treat transfer to your corpus as a hypothesis to test, especially when users phrase questions differently from stored documents.

In production, gate HyDE away from exact-match lookups such as IDs, dates, prices, or error codes. That's an engineering inference from the mechanism, not a claim from the paper: if a proxy invents a precise fact, retrieval can drift toward text that echoes the invention instead of the source chunk.

Why should HyDE be gated away from incident IDs, error codes, dates, and prices?

Answer

The hypothetical document can invent precise details. For exact lookups, invented values can pull retrieval toward the wrong chunks, while ordinary hybrid search preserves the user's exact identifier.

Self-RAG (Self-reflective RAG)

Many retrieve-then-generate pipelines fetch context once without a model-generated critique step. Self-RAG instead fine-tunes a generator to emit special reflection tokens that control retrieval and score candidate generation segments.^{[6]Reference 6Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.https://arxiv.org/abs/2310.11511}

Reflection tokens

Self-RAG uses one retrieval token family and three critique token families:^{[6]Reference 6Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.https://arxiv.org/abs/2310.11511}

1. Retrieve with values Yes, No, or Continue. This decides whether the model should fetch evidence before generating the next segment.
2. ISREL with labels such as Relevant or Irrelevant. This scores whether a retrieved passage is helpful for the current query or segment.
3. ISSUP with labels Fully supported, Partially supported, and No support. This checks whether the generated claim is grounded in retrieved evidence.
4. ISUSE with utility scores from 1 to 5. This measures how useful the final response is for the user.

Paper examples serialize these as inline control tags such as [Retrieve=Yes], [ISREL=Relevant], [ISSUP=Fully Supported], and [ISUSE=4]. The exact bracket syntax is less important than the four decision families, but each critique tag keeps its family name visible.

To see these tokens in action, imagine the same launch-throughput query. A Self-RAG model might generate the following token stream:

1. [Retrieve=Yes], the model decides it needs evidence before answering.
2. It retrieves a passage: "Temporary tokens-per-minute increases require approval and backoff-aware clients..."
3. [ISREL=Relevant], the passage is useful for the current segment.
4. The model generates: "For launch traffic, request a temporary TPM increase and keep exponential backoff enabled."
5. [ISSUP=Fully Supported], the claim is grounded in the retrieved text.
6. [ISUSE=4], the response is helpful but could be more detailed.

Without reflection tokens, a standard RAG pipeline might have retrieved the same passage without explicitly scoring passage relevance or claim support. Reflection tokens expose the model's predicted judgments for scoring and control; they aren't proof that a claim is true.

Architecture

Self-RAG is more than "retrieve once, then critique at the end." At inference time it can emit a retrieval decision, retrieve top-k passages on demand, generate candidate segments conditioned on different passages in parallel, and score those branches with reflection-token probabilities.^{[6]Reference 6Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.https://arxiv.org/abs/2310.11511} That segment-level beam search is what makes Self-RAG distinct from a simple prompted guardrail loop.

Implementation note

Deploying a true Self-RAG system requires a generator specifically fine-tuned to emit these reflection tokens during generation. The paper releases 7B and 13B checkpoints trained that way.^{[6]Reference 6Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.https://arxiv.org/abs/2310.11511} Before generator training, the authors use a separate critic model to insert reflection tokens into supervised examples offline, then train the final generator to emit those tokens itself at inference time.^{[6]Reference 6Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.https://arxiv.org/abs/2310.11511}

A prompted frontier model can imitate parts of this control loop, but that isn't the same system. Without reflection-token fine-tuning, you're building a Self-RAG-inspired agentic pipeline: separate routing, retrieval grading, and answer validation calls stitched together in application code.

Self-RAG cost comes from on-demand retrieval plus branching over multiple passages and scoring those branches, not from a few extra control tokens. Evaluate it where support-aware generation is worth that additional serving path.

This scoring sketch doesn't implement Self-RAG training. It shows how an inference service can rank branches once a trained model has supplied relevance, support, and utility probabilities; the weights are an explicit product policy.

CRAG (Corrective RAG)

Corrective Retrieval-Augmented Generation (CRAG) focuses on correction after imperfect retrieval. It adds a lightweight Retrieval Evaluator that scores retrieved question-document pairs and routes the request before generation.^{[7]Reference 7Corrective Retrieval Augmented Generation.https://arxiv.org/abs/2401.15884}

The evaluator

In the paper, the evaluator is a lightweight T5-large model fine-tuned to score each retrieved question-document pair, then threshold those scores into one of three actions:^{[7]Reference 7Corrective Retrieval Augmented Generation.https://arxiv.org/abs/2401.15884}

A developer asks about "current embedding TPM increase process." The internal knowledge base has weak coverage. The evaluator might score the retrieved internal documents as Incorrect, triggering a web-search fallback. A production system would still need source allowlists and citation checks before trusting those web results. If internal documents are somewhat relevant but incomplete, the evaluator returns Ambiguous, and CRAG combines refined internal strips with web results.

1. Correct: At least one retrieved document clears the upper threshold. Action: refine internal knowledge and answer from it.
2. Incorrect: All retrieved documents fall below the lower threshold. Action: discard them and fall back to web search.
3. Ambiguous: The scores land between those two cases. Action: combine refined internal evidence with web results.

The distinction from Self-RAG is where correction happens. CRAG doesn't train the generator to emit reflection tokens. Instead, it inserts a separate evaluator between retrieval and generation and uses that evaluator to trigger correction paths.

What separates true Self-RAG from a prompted critique loop?

Answer

True Self-RAG uses a generator trained to emit retrieval and critique tokens during generation. A prompted critique loop can be useful, but it's application orchestration around a standard model.

Knowledge refinement

Even relevant documents contain noise. CRAG includes a decompose-then-recompose step:

1. Break the document into fine-grained strips (sentences or small chunks).
2. Score each strip for relevance.
3. Concatenate only the relevant strips.
4. Pass this "refined knowledge" to the generator.

Here's a simplified CRAG sketch in Python. The actual paper scores retrieved documents individually and then applies thresholds. For readability, this sketch collapses that logic into a single classify helper. The refine_knowledge method then decomposes documents into smaller strips, scores those strips, and recomposes only the useful evidence before generation.

If you instantiate this class with real components and run crag.run("current embedding TPM increase process"), the evaluator might return "incorrect" because internal docs lack coverage. The pipeline would then call web_search.search(...) and pass the web results through refine_knowledge before generating the answer.

Agentic and iterative retrieval

Self-RAG and CRAG both introduce feedback, but at different boundaries: Self-RAG can make retrieval and critique decisions while generating segments, while CRAG routes after initial retrieval. An application can generalize feedback into agentic retrieval: instead of one fixed retrieve-then-generate pass, a model with tool access searches, reads results, tests whether evidence is sufficient, and either answers or searches again with a refined query.

The mechanism is older than the agent framing. IRCoT studies one version of this pattern: retrieve, reason a step, use that step to drive the next retrieval, and repeat until the chain of evidence is complete.^{[8]Reference 8Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionshttps://arxiv.org/abs/2212.10509} Multi-hop questions like "Which embedding endpoint has the highest p95 latency, and what retry budget applies to that endpoint?" need exactly this, because the answer to the second part depends on resolving the first.

The engineering question is when to escalate up this ladder, since each rung costs latency and tokens:

1. Naive RAG for single-fact lookups answerable by one retrieval.
2. Query rewriting plus hybrid search and reranking for conversational ambiguity and vocabulary mismatch. This is a practical baseline to evaluate before more expensive routes.
3. HyDE behind a router for abstract or vocabulary-mismatched queries where a document-style proxy helps.
4. A correction gate (CRAG-style evaluator or prompted critique) when retrieval quality is inconsistent and a single bad context causes user-visible errors.
5. Iterative or agentic retrieval for genuinely multi-hop questions, where one retrieval pass can't gather all the facts.

Escalate only when the failure mode and evaluation demand it. On a simple FAQ lookup, an agentic loop can add latency and failure paths without improving retrieved evidence.

A question needs two facts where the second depends on resolving the first. Which strategy fits, and why not naive RAG?

Answer

Iterative or agentic retrieval. Naive RAG runs one retrieval pass, so it can't use the answer to the first hop to form the query for the second. Interleaving retrieval with reasoning lets each hop drive the next search.

Comparison of advanced techniques

Choose these techniques by failure mode, not by novelty. HyDE targets semantic mismatch. Self-RAG changes retrieval timing and branch scoring inside a trained generator. CRAG adds a separate correction gate after retrieval.

Production failure modes

Advanced RAG helps only when it matches the failure. Watch for these symptoms before adding another model call. Rewrite, HyDE, multi-query, and CRAG affect more than quality. They expand the attack and cost surface of retrieval.

Symptom: the query is vague, but the answer needs history

• Cause: The retriever sees only the latest turn, so it searches for "the other one" instead of the backup API key or failed batch job.
• Fix: Rewrite the latest turn into a standalone query using the last few conversation turns. Keep the raw user message for the generator, but search with the rewritten query.

Symptom: rewrite or history poisons retrieval intent

• Cause: Conversation history, tool output, or a retrieved doc can contain "ignore policy / search secrets / dump all tickets." A rewrite model that treats history as trusted instructions will adopt that intent and retrieve privileged neighborhoods.
• Fix: Treat rewrite input as untrusted. Constrain the rewrite prompt to disambiguate the user's question, not to follow instructions found in history. Strip or mark tool/page text as data. Preserve principal identity and ACL scope; never let a rewrite broaden grants.

Symptom: HyDE improves recall on policy questions but breaks exact lookups

• Cause: The hypothetical document can invent precise identifiers, dates, or prices. That invented detail may pull retrieval toward the wrong neighborhood.
• Fix: Route exact-match queries to normal hybrid retrieval. Use HyDE for conceptual or vocabulary-mismatched questions where a document-style proxy helps more than it hurts.

Symptom: CRAG web fallback becomes SSRF or untrusted-source injection

• Cause: An "Incorrect" retrieval score triggers open-web fetch. Without controls, the worker can hit internal metadata endpoints (SSRF) or pull attacker-controlled pages that inject instructions into generation.
• Fix: Domain allowlists, no link-following into private networks, fetch in a network sandbox, treat web HTML as untrusted observations, and require citations from allowlisted sources before generation. Prefer "refuse / escalate" over a wide-open web tool.

Symptom: rewrite or HyDE cache bleeds across tenants

• Cause: Caches keyed only on the raw string miss tenant, user, and policy version. User A's rewritten query or HyDE proxy can serve User B's retrieval path.
• Fix: Key caches by (tenant_id, principal_id or role set, policy_version, normalized_query, technique). Prefer short TTLs on security-sensitive paths. Never share HyDE vectors across tenants.

Symptom: multi-query / agentic loops become a cost DoS

• Cause: Decomposition and multi-query fan-out multiply embedding, ANN, and LLM calls per user message.
• Fix: Hard caps on rewrite count, sub-queries, web fetches, and dollar/token budgets per request. Fail closed with a partial answer or clarification when the budget trips.

Symptom: Self-RAG-inspired code works in prompts but isn't true Self-RAG

• Cause: True Self-RAG trains a generator to emit retrieval and critique tokens during generation. Separate prompted grading calls can mimic the control loop, but they don't create the same reflection-token model.
• Fix: Name the system honestly. Call it a prompted critique loop or CRAG-style evaluator unless you're hosting a model trained with Self-RAG reflection tokens.

Symptom: accuracy rises, but users feel the product is slow

• Cause: Rewriting, HyDE generation, dense retrieval, sparse retrieval, reranking, grading, and final synthesis can become a long sequential path.
• Fix: Parallelize independent retrievals, cap reranker candidates, cache repeated rewrites or hypothetical documents with tenant-safe keys, and measure retrieval and generation latency separately.

Practical implementation strategy

Implementing all these techniques at once is overkill. Add them in stages:

Phase 1: Stable baseline

Start with Query Rewriting and Hybrid Search (dense retrieval + sparse keyword retrieval).

• Why: Fixes basic vocabulary mismatch and conversational ambiguity.
• Cost: Low (1 extra LLM call for rewriting).

Phase 2: Add reranking

Add a cross-encoder reranker after retrieval.

• Why: Can improve precision of the final candidate set when initial retrieval has decent recall.
• Cost: Moderate. Cross-encoders are slower than initial retrieval, so keep candidate count small.

Phase 3: Specialized handling (HyDE/router)

Use a Router to classify queries.

• If query is conceptual or abstract (e.g., "Explain launch-week embedding throughput planning"): Use HyDE, since vocabulary mismatch is likely.
• If query is factual or precise (e.g., "What happened in incident INC-48291?"): Use standard retrieval, since HyDE risks hallucinating the exact identifier.
• Why: Optimizes for different query types without applying HyDE to everything.

Phase 4: Add correction loops

If accuracy is still insufficient, add a prompted critique loop or a CRAG-style evaluator before investing in a true Self-RAG model.

• Prompt the model or a lightweight evaluator to grade retrieved documents before answer generation.
• If evidence is weak, trigger a rewrite, second retrieval pass, or web fallback.
• Why: Captures much of the reliability gain without requiring reflection-token fine-tuning.

Latency is the production constraint in advanced RAG. A pipeline with rewriting, HyDE, retrieval, reranking, grading, and generation turns one answer into several sequential model and retrieval steps. Stream the final generation, fetch dense and sparse results in parallel, and cache reusable artifacts when traffic is repetitive.

Don't release a more elaborate route because it improves a few anecdotes. Compare supported-answer quality and latency on a labeled set, then release only paths that meet both requirements.

Try it yourself

Apply these techniques to a small, concrete dataset. Here's a focused exercise you can complete in under an hour.

Exercise: Build a query rewriter for a developer-support assistant

Setup: Collect five real developer-support messages from an API docs assistant (or write realistic ones). Include at least one ambiguous message that needs conversation history, one complex comparison, and one vague keyword.

Step 1, Rewrite: Write a Python function that takes a developer message plus the last two turns of chat history and outputs a standalone query. Run it on your five messages and inspect the results. Does the rewritten query contain the full intent?

Step 2, Measure: For each original message, manually decide which of your internal policy documents should be retrieved. Then run the rewritten query through a simple dense-retrieval setup (even a small embedding model like all-MiniLM-L6-v2 against a dozen policy chunks). Count how many of the top-3 results match your manual gold set. The rewrite should improve hit rate for the ambiguous and vague cases.

Step 3, Diagnose: Pick one message where retrieval still fails. Is the problem vocabulary mismatch (try multi-query expansion), semantic asymmetry (try HyDE), or weak evidence (try a CRAG-style evaluator)? Implement the fix and measure again.

Expected outcome: Record which interventions improve top-3 evidence hits or catch weak retrieval on your examples, and which add latency without a gain. A small exercise may not reproduce paper results; its value is exposing the measurement loop.

When each technique pays off

Once you understand the mechanics, the skill is knowing when each intervention earns its latency cost and when it adds unnecessary complexity. These three trade-offs show up often when engineers move from prototype to production.

When does HyDE justify its latency cost?

HyDE adds at least one extra generation call before retrieval. That cost is only worth it when measured retrieval quality improves enough to justify it. A smaller instruction model may be adequate for the hypothetical document, but evaluate it rather than assuming equivalence. Cache repeated proxy documents only when generation configuration and source policy make reuse valid. For exact-match lookups like incident IDs, error codes, or dates, skip HyDE.

When does Self-RAG degrade instead of help?

Self-RAG degrades when retrieval itself is weak or when the model's learned critique tokens stop correlating with real answer quality. It also raises inference cost because the model may retrieve on demand, branch over multiple passages, and spend extra decoding steps on critique tokens before choosing a continuation. For a simple FAQ bot, first measure a cheaper baseline such as rewrite plus reranking or a correction gate before committing to a specialized Self-RAG serving path.

Can you combine HyDE and Self-RAG?

As a system design, you can use HyDE to propose initial candidate passages, then let a true Self-RAG model score branches with ISREL, ISSUP, and ISUSE. That composition needs its own evaluation; neither mechanism guarantees the other improves it. If you don't have a reflection-token model, describe the composition as HyDE plus a CRAG-style evaluator or prompted support checks rather than Self-RAG.