Advanced RAG: HyDE & Self-RAG

A developer reports a leaked API key, then asks an internal docs assistant, "What do I do now?" Searching that latest turn alone misses the revocation procedure. The index can be healthy while the search request is missing its subject. Before changing ANN settings, recover the missing context.

The previous chapter showed how HNSW and IVF find nearby chunks quickly. Retrieval-augmented generation (RAG) grounds an answer in those chunks, while self-reflective RAG (Self-RAG) decides when retrieval and evidence checks are needed. That speed doesn't help when the search object is "the other one." A fast index returns the wrong neighborhood faster. Keep the API-docs assistant as the running example, moving from leaked-key recovery to throughput-policy questions. Measure whether rewriting, Hypothetical Document Embeddings (HyDE), Self-RAG, or Corrective RAG (CRAG) recovers the right page, then compare supported answers against the added latency and cost.

The problem with naive RAG

A naive RAG pipeline embeds the user's raw question and fetches the top-k nearest neighbors from a vector database. Retrieve once, then generate. Before adding a more elaborate route, predict which of these failures your misses show:

1. The query is ambiguous. "What about the other one?" depends on chat history. A stateless retriever never sees that history.
2. The query and the document don't share a style. Someone asks "How do I rotate a key?" The page title is "Credential Rotation Workflow". The embedding of a short question may not sit near the embedding of a formal policy paragraph.
3. The retriever still returns the wrong chunks. Wording, embedding geometry, approximate nearest-neighbor (ANN) knobs such as ef_search and nprobe, or a messy corpus can all put SDK install notes above the rate-limit policy. Once those chunks enter context, the generator can write an unsupported answer.
4. The model doesn't know when to retrieve. Naive RAG retrieves for every turn, including "Hi" or "What is 2+2?", and burns tokens and latency for nothing.

Index tuning from the last chapter and query control in this one are different repairs. Measure which failure you actually have, then add the smallest intervention that fixes it.

Each technique changes a different part of the request. Rewriting restores conversational context when the rewrite is correct. Expansion tries alternative vocabulary. Decomposition separates required facts, which may be independent or dependent. Later controls estimate relevance and support, but none establishes source truth by itself.

Query rewriting and decomposition

Three failures can look identical in a chat transcript. A rewrite creates a standalone question; expansion creates vocabulary variants; decomposition creates a plan for required facts. Before looking at the figure, ask whether each lookup can be formulated now or needs a result from an earlier lookup.

"What do I do now?" is a bad search object. The leaked-key fact lives in the previous turn, and a comparison like "sync vs batch embeddings" packs three lookups into one sentence. Use those examples to predict which details must survive rewriting and which facts need separate searches.

1. Rewrite for disambiguation

An on-call lead often turns a vague Slack thread into a precise search request before querying runbooks and product docs. Query rewriting gives the model that same job. Ma et al.'s Rewrite-Retrieve-Read pipeline adds a rewrite stage in front of a frozen retriever and reader, including a small T5 rewriter trained with reader feedback, instead of treating the user's wording as sacred.^{[1]Reference 1Query Rewriting for Retrieval-Augmented Large Language Models.https://arxiv.org/abs/2305.14283} In a chat product, a practical variant is simpler: rewrite the latest turn into a standalone, search-oriented question.

Before reading the rewrite, predict what must survive: the leaked-key context and the request for recovery steps. Never copy the secret key itself into a search query. History supplies the referent, but it remains data rather than instructions.

A developer sends two messages:

Developer: "My API key leaked in a public issue." Developer: "What do I do now?"

A stateless retriever searching for "What do I do now?" might return generic onboarding articles. A rewrite step instead produces:

Standalone query: "How do I revoke and rotate a leaked API key and audit recent usage?"

The rewrite model sits behind a small interface. The deterministic fake below checks plumbing without API keys; it neither understands arbitrary conversation nor tests resistance to injected instructions. Revoke and rotate follow from the reported leak. Auditing is an added retrieval subtopic, not evidence that the user requested a tool action.

from typing import Protocol, TypedDict

class ChatMessage(TypedDict):
    role: str
    content: str

class RewriteModel(Protocol):
    def rewrite(self, latest_query: str, history_text: str) -> str: ...

class FakeRewriteModel:
    def rewrite(self, latest_query: str, history_text: str) -> str:
        if "api key leaked" in history_text.lower() and "what do i do" in latest_query.lower():
            return "How do I revoke and rotate a leaked API key and audit recent usage?"
        return latest_query

def rewrite_query_with_history(
    query: str, chat_history: list[ChatMessage], model: RewriteModel
) -> str:
    history_text = "\n".join(
        f"{message['role']}: {message['content']}" for message in chat_history
    )
    rewritten = model.rewrite(query, history_text).strip()
    return rewritten or query  # An empty rewrite mustn't erase the request.

history = [
    {"role": "developer", "content": "My API key leaked in a public issue."},
]

rewritten = rewrite_query_with_history("What do I do now?", history, FakeRewriteModel())
print(rewritten)

How do I revoke and rotate a leaked API key and audit recent usage?

Rewrite is search input, not authorization. History, tool output, and retrieved pages can contain "ignore policy / search all tickets" style instructions. Constrain the prompt to disambiguate the user's question, not follow instructions found in history, and never let a rewrite broaden ACL or tenant scope. RAG Security & Access Control covers the trusted retrieval plane these transforms must preserve.

2. Multi-query expansion

Multi-query expansion is the repair when one wording misses the page. "How do you handle peak traffic throttling?" never mentions "rate-limit burst policy", "queue backpressure", or "autoscaling capacity". Generate those variants, search each one, then merge the lists.

Before merging, predict how repeated appearances affect the score. Raw scores are not automatically calibrated across queries or retrievers. Cosine similarity is bounded in $[-1,1]$, but its relevance meaning can differ across queries; BM25 has a different scale. Score fusion is possible with suitable calibration and evaluation. Here we use ranks to avoid assuming those scales are comparable.

Once you fan out into several queries, you have several ranked result lists to merge. One documented pattern is RAG-Fusion, which pairs generated query variants with Reciprocal Rank Fusion (RRF) to combine the lists into one ranking.^{[2]Reference 2RAG-Fusion: a New Take on Retrieval-Augmented Generationhttps://arxiv.org/abs/2402.03367} RRF ignores raw similarity scores and sums reciprocal ranks instead, so a document that lands near the top of several lists rises even if no single list ranked it first.^{[3]Reference 3Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.https://dl.acm.org/doi/10.1145/1571941.1572114}

For a document $d$, the score across rankings $R$ is:

$\text{RRF\_Score}(d) = \sum_{r \in R} \frac{1}{k + r(d)}$

Here, $R$ is the set of variant rankings, $r(d)$ is the document's one-based position in one ranking, and $k$ is a rank damping constant. Cormack et al. fixed $k = 60$ after empirical tuning.^{[3]Reference 3Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.https://dl.acm.org/doi/10.1145/1571941.1572114}

Why $k = 60$? With $k = 0$, ranks 1 and 2 contribute 1.0 and 0.5. With $k = 60$, they contribute about 0.01639 and 0.01613, a 1.6% drop. Increasing $k$ reduces the advantage of a single high position relative to appearing in several lists. It doesn't guarantee protection from noisy variants. One first-place appearance ties two second-place appearances when $k=0$; it doesn't beat them. A document missing from a truncated list contributes zero. Repeated or near-identical variants can overweight the same result, so cap and evaluate expansion rather than treating every appearance as independent evidence.

Implement multi-query expansion behind a testable boundary. The model returns one query per line; the parser removes list markers, deduplicates variants, and caps their count so they can be searched in parallel. Keep the original query as a separate retrieval candidate when exact wording matters. The retriever should see clean search objects, not formatting noise.

import re
from typing import Protocol

class QueryExpansionModel(Protocol):
    def expand(self, query: str, n: int) -> str: ...

class FakeExpansionModel:
    def expand(self, query: str, n: int) -> str:
        return "\n".join(
            [
                "Rate-limit burst policy during launch traffic",
                "Queue backpressure controls for high-volume periods",
                "Autoscaling capacity for traffic spikes",
            ][:n]
        )

def clean_query_line(line: str) -> str:
    # Remove a list marker only when followed by whitespace. Preserve "429".
    return re.sub(r"^(?:[-*]|\d+[.)])\s+", "", line.strip()).strip()

def generate_multi_queries(
    query: str, model: QueryExpansionModel, n: int = 3
) -> list[str]:
    if not 1 <= n <= 5:
        raise ValueError("n must be between 1 and 5")
    content = model.expand(query, n)
    queries = [clean_query_line(line) for line in content.splitlines()]
    return list(dict.fromkeys(item for item in queries if item))[:n] or [query]

queries = generate_multi_queries(
    "How do you handle peak traffic throttling?", FakeExpansionModel(), n=3
)

for query in queries:
    print(f"- {query}")

- Rate-limit burst policy during launch traffic
- Queue backpressure controls for high-volume periods
- Autoscaling capacity for traffic spikes

Generating variants isn't enough. The system must merge result lists without assuming similarity scores from separate searches are directly comparable. RRF provides a deterministic rank-based merge. As you read the next snippet, predict which ID benefits most from appearing near the top across all three lists:

def reciprocal_rank_fusion(rankings: list[list[str]], rank_constant: int = 60) -> list[tuple[str, float]]:
    if rank_constant < 0:
        raise ValueError("rank_constant must be nonnegative")
    scores: dict[str, float] = {}
    for ranking in rankings:
        # A result list ranks distinct documents, not repeated copies of a hit.
        for rank, document_id in enumerate(dict.fromkeys(ranking), start=1):
            scores[document_id] = scores.get(document_id, 0.0) + 1 / (rank_constant + rank)
    return sorted(scores.items(), key=lambda pair: (-pair[1], pair[0]))

ranked_lists = [
    ["rate-limit-policy", "quota-increase", "autoscaling"],
    ["autoscaling", "rate-limit-policy", "backpressure"],
    ["rate-limit-policy", "batch-api", "autoscaling"],
]

for document_id, score in reciprocal_rank_fusion(ranked_lists)[:3]:
    print(f"{document_id}: {score:.4f}")

rate-limit-policy: 0.0489
autoscaling: 0.0481
batch-api: 0.0161

3. Query decomposition (least-to-most)

Decomposition is the repair when one search can't collect every fact. Take this comparison:

"Compare synchronous and batch embedding API latency, limits, and retry behavior."

No single document in the corpus covers all three dimensions across both architectures. A naive dense retriever pulls chunks that mention "embedding API", but misses either the batch limits or the retry backoff policy. The repair breaks the prompt into focused sub-queries:

1. "What latency does the synchronous embedding endpoint target?"
2. "What throughput and completion limits does the batch embedding endpoint have?"
3. "What retry behavior applies to each endpoint?"
4. "Compare synchronous and batch embeddings given the latency, limit, and retry facts."

There are two distinct execution topologies for decomposed queries:

1. Parallel scatter-gather decomposition: When searches have no dependencies (as in the comparison above, where the endpoint names are already known), they can run concurrently. If $T_i$ measures each search's completion time from the common launch, the wait for all results is $\max(T_1,T_2,T_3)$. Query planning, scheduling, fusion, and final synthesis add work outside that wait; limited concurrency can serialize searches.
2. Sequential least-to-most chains: When sub-problems depend on each other, resolve the dependency before constructing the next query. Zhou et al.'s least-to-most prompting studies this decomposition-and-sequential-solving pattern for reasoning; using it with a retriever is an application of that idea.^{[4]Reference 4Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelshttps://arxiv.org/abs/2205.10625} For example, "Find the endpoint with the highest error rate during the last incident, then retrieve its circuit-breaker configuration." Hop 2 needs the endpoint from Hop 1.

A frequent failure mode in decomposed retrieval is partial-evidence hallucination: if sub-query 3 returns empty context, the synthesizer can fabricate a plausible retry policy to complete the comparison table. A reliable pipeline explicitly flags missing evidence dimensions instead of guessing.

A real model produces the sub-questions. The rest of the system depends only on the clean line-oriented contract:

import re
from typing import Protocol

class DecompositionModel(Protocol):
    def decompose(self, query: str) -> str: ...

class FakeDecompositionModel:
    def decompose(self, query: str) -> str:
        return "\n".join(
            [
                "What latency does the synchronous embedding endpoint target?",
                "What throughput and completion limits does the batch embedding endpoint have?",
                "What retry behavior applies to each endpoint?",
                "How do synchronous and batch embeddings compare given those facts?",
            ]
        )

def decompose_query(query: str, model: DecompositionModel) -> list[str]:
    lines = [
        re.sub(r"^(?:[-*]|\d+[.)])\s+", "", line.strip()).strip()
        for line in model.decompose(query).splitlines()
    ]
    return list(dict.fromkeys(line for line in lines if line))[:4]

sub_questions = decompose_query(
    "Compare synchronous and batch embedding API latency, limits, and retry behavior.",
    FakeDecompositionModel(),
)

for index, question in enumerate(sub_questions, start=1):
    print(f"{index}. {question}")

1. What latency does the synchronous embedding endpoint target?
2. What throughput and completion limits does the batch embedding endpoint have?
3. What retry behavior applies to each endpoint?
4. How do synchronous and batch embeddings compare given those facts?

4. Step-back prompting (abstraction)

When a query is overloaded with specific runtime parameters, error codes, or customer-specific configurations, direct dense search often fails if the knowledge corpus documents high-level architecture rules, rate limits, or protocols instead of the exact numerical combination.

Step-back prompting abstracts the specific inquiry into a broader, foundational question.^{[5]Reference 5Take a Step Back: Evoking Reasoning via Step-Back Prompting in Large Language Models.https://arxiv.org/abs/2310.06117} Zheng et al. prompt the model to "step back" and identify the overarching concept or policy behind the question.

Predict which evidence each query can recover. The specific query should find incident details; the broader query should find governing policy. Neither query alone can reliably provide both.

For instance, consider an operational question:

Specific query: "Why did our client receive an HTTP 429 when sending 120 requests/minute on the tier 2 embedding endpoint?" Step-back query: "What are the rate limits, burst thresholds, and quota policies for tier 2 embedding endpoints?"

In RAG, the retriever queries the index for both the original specific question and the abstracted step-back question. The step-back search retrieves foundational rules and architecture guides, while the specific search retrieves endpoint parameters. The generator conditions on both evidence sets, keeping broad policy from replacing exact diagnosis.

Implement step-back generation behind a clear interface that pairs the original question with its conceptual counterpart. The pair is the boundary: later code can choose retrieval behavior without losing the original wording:

from typing import Protocol

class StepBackModel(Protocol):
    def step_back(self, query: str) -> str: ...

class FakeStepBackModel:
    def step_back(self, query: str) -> str:
        if "429" in query and "tier 2" in query.lower():
            return "What are the rate limits, burst thresholds, and quota policies for tier 2 embedding endpoints?"
        return query

def generate_step_back_query(query: str, model: StepBackModel) -> str:
    return model.step_back(query).strip()

def plan_step_back_queries(
    query: str, model: StepBackModel
) -> dict[str, str]:
    abstract_query = generate_step_back_query(query, model)
    return {
        "original_query": query,
        "step_back_query": abstract_query,
    }

plan = plan_step_back_queries(
    "Why did our client receive an HTTP 429 when sending 120 requests/minute on tier 2?",
    FakeStepBackModel(),
)

print(f"original:  {plan['original_query']}")
print(f"step-back: {plan['step_back_query']}")

original:  Why did our client receive an HTTP 429 when sending 120 requests/minute on tier 2?
step-back: What are the rate limits, burst thresholds, and quota policies for tier 2 embedding endpoints?

These transforms change the search object before the same hybrid index from earlier chapters runs. They don't grade the chunks that come back. HyDE goes one step further: it changes the kind of vector used for search. That makes proxy quality and evidence boundaries matter.

HyDE (Hypothetical Document Embeddings)

Standard dense retrieval matches a query embedding to document embeddings. A question can be short and vague while the relevant passage names a specific policy. That mismatch can hurt retrieval, especially without relevance supervision. It doesn't imply that query and document vectors occupy separate subspaces: retrievers can be trained to align them. HyDE instead uses a generated passage as an intermediate search representation.^{[6]Reference 6Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496}

HyDE (Hypothetical Document Embeddings) prompts an instruction-tuned LLM zero-shot to write a hypothetical document answering the prompt, embeds that document-style proxy with the corpus passage encoder, and retrieves real corpus passages near the resulting vector.^{[6]Reference 6Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496} Gao et al. prompt the model to "write a document that answers the question." If you need a throughput-limit clause but can't remember the exact internal phrasing, don't search "Can we handle more traffic?" Have the model generate a hypothetical policy paragraph, and search for documents that match that paragraph's dense representation.

Hold on to the launch-week question. Predict what changes: the source of the vector, not the truth of the answer. A document-style proxy may supply useful policy vocabulary, or invent a cause that moves search in the wrong direction. Its location must be evaluated with the actual encoder and corpus.

A developer asks:

Query: "Can we increase embedding throughput during a launch week?"

A standard dense retriever might embed that short question and pull generic "embedding API overview" articles because they share high-frequency terms like "embedding" and "throughput". HyDE instead prompts the model to generate a hypothetical policy paragraph:

Hypothetical document: "Embedding throughput increases: teams can request a temporary tokens-per-minute quota increase, use the batch embedding endpoint for offline jobs, shard requests across approved projects, and apply exponential backoff when rate limits are returned..."

That generated paragraph is longer, declarative, and packed with domain vocabulary like "tokens-per-minute quota", "batch embedding endpoint", and "exponential backoff". It's an illustrative search proxy, not an answer. The generated limit might be completely wrong, so only retrieved source text may support the final response.

Cache HyDE proxies and rewrite strings with tenant- and principal-scoped keys (plus policy version), not raw query text alone. A shared string key lets Tenant A's proxy or rewrite serve Tenant B's retrieval path.

How HyDE works

The original HyDE pipeline has three phases:

1. Generate: Prompt an instruction-tuned language model to sample one or more hypothetical passages for the query. The passages may contain fabricated details, but they capture the topical density, syntax, and relevance patterns of real corpus documents.^{[6]Reference 6Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496}
2. Embed: Encode each sampled passage with the corpus document encoder $f$. Average the vectors to estimate the expected hypothetical-document embedding:^{[6]Reference 6Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496}

$\hat{\mathbf{v}}_q = \frac{1}{N}\sum_{k=1}^{N} f(\hat{d}_k)$

Here $f$ is the document encoder, $\hat{d}_k$ is the $k$-th sampled proxy, and $N$ is the number of samples. Gao et al. also try averaging in the original query vector as one extra hypothesis. 3. Retrieve: Search the corpus with that averaged vector. Gao et al. hypothesize that the encoder's dense bottleneck can discard some fabricated detail while retaining useful relevance patterns.^{[6]Reference 6Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496} Compression doesn't identify which details are false; a fabricated name or policy can still change the neighbors. Use the same compatible encoder and preprocessing as the corpus. For cosine search, normalize a nonzero mean vector if the backend requires unit vectors; don't silently change the metric.

The production version uses a real generator and dense encoder. This small runnable version deliberately generates one proxy and uses keyword sets so the retrieval contract is visible: the query first becomes a document-like paragraph, then the retriever searches with that paragraph rather than the original question. As the snippet runs, predict why the two policy chunks should outrank the generic overview.

from dataclasses import dataclass
from typing import Protocol

class HypotheticalDocGenerator(Protocol):
    def generate(self, query: str) -> str: ...

@dataclass(frozen=True)
class Chunk:
    id: str
    text: str

class FakeHyDEGenerator:
    def generate(self, query: str) -> str:
        return (
            "Embedding throughput increases require a temporary tokens-per-minute "
            "quota request, batch embedding jobs, and exponential backoff for rate limits."
        )

class KeywordRetriever:
    def __init__(self, chunks: list[Chunk]) -> None:
        self.chunks = chunks

    def search(self, search_text: str, k: int = 2) -> list[Chunk]:
        query_terms = set(
            search_text.lower().replace(",", " ").replace(".", " ").split()
        )

        def score(chunk: Chunk) -> int:
            chunk_terms = set(
                chunk.text.lower().replace(",", " ").replace(".", " ").split()
            )
            return len(query_terms & chunk_terms)

        return sorted(self.chunks, key=score, reverse=True)[:k]

def hyde_retrieve(
    query: str, generator: HypotheticalDocGenerator, retriever: KeywordRetriever
) -> list[Chunk]:
    hypothetical_doc = generator.generate(query)
    return retriever.search(hypothetical_doc, k=2)

chunks = [
    Chunk("generic-embeddings", "The embedding API converts text into vectors for search."),
    Chunk(
        "throughput-quota",
        "Launch traffic needs a temporary tokens-per-minute quota request and approval.",
    ),
    Chunk("batch-embeddings", "Batch embedding jobs support offline workloads with retry backoff."),
]

query = "Can we increase embedding throughput during a launch week?"
retriever = KeywordRetriever(chunks)
required_ids = {"throughput-quota", "batch-embeddings"}
routes = {
    "raw question": retriever.search(query),
    "useful proxy": hyde_retrieve(query, FakeHyDEGenerator(), retriever),
    "drifting proxy": retriever.search("The embedding API converts text into vectors for search."),
}
for route, matches in routes.items():
    match_ids = [chunk.id for chunk in matches]
    recall = len(required_ids & set(match_ids)) / len(required_ids)
    print(f"{route}: {match_ids}; required Recall@2={recall:.1f}")
assert required_ids <= {chunk.id for chunk in routes["useful proxy"]}
assert "throughput-quota" not in {chunk.id for chunk in routes["drifting proxy"]}

raw question: ['throughput-quota', 'generic-embeddings']; required Recall@2=0.5
useful proxy: ['throughput-quota', 'batch-embeddings']; required Recall@2=1.0
drifting proxy: ['generic-embeddings', 'batch-embeddings']; required Recall@2=0.5

The raw question misses the batch policy, the useful proxy recovers both required chunks, and the drifting proxy loses the quota policy. The corpus, top-k, and gold IDs stayed fixed; only the search text changed. These are executed keyword traces, not measured dense-HyDE performance. The useful proxy contains the exact words needed by the authored corpus, while ties retain corpus order. Neither proxy may become answer evidence.

In the paper, the actual query vector is an average of document embeddings. With two 3-d proxies below, calculate each coordinate before running it: add the two values, then divide by two:

def mean_vector(vectors: list[list[float]]) -> list[float]:
    if not vectors or not vectors[0]:
        raise ValueError("at least one nonempty vector is required")
    count = len(vectors)
    dims = len(vectors[0])
    if any(len(vector) != dims for vector in vectors):
        raise ValueError("all vectors must have the same dimension")
    return [sum(vector[i] for vector in vectors) / count for i in range(dims)]

proxy_a = [0.2, 0.8, 0.1]
proxy_b = [0.4, 0.6, 0.3]
query_vector = mean_vector([proxy_a, proxy_b])
print([round(value, 2) for value in query_vector])

[0.3, 0.7, 0.2]

HyDE changes the search representation first, then retrieves. The figure below uses invented coordinates to show a useful proxy near policy chunks and a drifting proxy near the generic overview. These are neither measured embeddings nor a projection of a trained model; the drawing is schematic rather than a distance-preserving map. Only the three corpus chunks are retrieval candidates, not the query or either proxy. Ask what still remains unproven when a proxy sits beside a relevant chunk.

In the paper's InstructGPT-based configuration on TREC DL19, Gao et al. report Contriever nDCG@10 of 44.5 and HyDE nDCG@10 of 61.3. DL20 nDCG@10 moves from 42.1 to 57.9, and they also report gains on BEIR and multilingual Mr. TyDi.^{[6]Reference 6Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496} These are the paper's 0-100 ranking scores, not answer-accuracy percentages or a product comparison. A proxy can still invent a high-impact identifier or policy detail, so transfer to your corpus is a hypothesis to test.

When to use HyDE

HyDE was designed for zero-shot retrieval without relevance labels. Gao et al. evaluate it on web search, BEIR, and multilingual Mr. TyDi tasks.^{[6]Reference 6Precise Zero-Shot Dense Retrieval without Relevance Labels.https://arxiv.org/abs/2212.10496} Use it when users phrase questions differently from stored documents. Skip it when the query already is the identifier you must keep.

Preserve exact identifiers and constraints such as incident IDs, error codes, CVEs, commit hashes, dates, and prices. For this assistant, bypassing HyDE on exact lookups is a conservative application policy, not a requirement in the HyDE algorithm. A mixed conceptual query can keep an exact-term search alongside a proxy. If a proxy invents a cause for an incident, it may favor documents matching that cause rather than the relevant incident. We'll call that hallucination drift.

Before the router runs, classify each example: preserve the incident ID; reshape the conceptual throughput question. That boundary keeps a useful proxy from rewriting facts that must match exactly.

import re

EXACT_LOOKUP = re.compile(
    r"\b(?:[A-Z]{2,}-\d+|(?:HTTP|error)\s+[A-Z0-9-]+|\d{4}-\d{2}-\d{2})\b|[$€£]\s*\d",
    re.IGNORECASE,
)

def retrieval_route(query: str) -> str:
    if EXACT_LOOKUP.search(query):
        return "hybrid_exact_preserving"
    return "hyde_candidate"

queries = [
    "What happened in incident INC-48291?",
    "Why did HTTP 429 occur on 2026-08-31?",
    "How should we plan embedding throughput for launch traffic?",
]

for query in queries:
    print(f"{retrieval_route(query)}: {query}")

hybrid_exact_preserving: What happened in incident INC-48291?
hybrid_exact_preserving: Why did HTTP 429 occur on 2026-08-31?
hyde_candidate: How should we plan embedding throughput for launch traffic?

The regex is a deliberately incomplete guard for known forms, not a security classifier. Unmatched identifiers, version strings, and locale-specific dates need corpus-specific handling. Keep the original query and exact terms available even on a proxy route.

Self-RAG (Self-reflective RAG)

Many retrieve-then-generate pipelines fetch context once without a model-generated critique step. Self-RAG instead fine-tunes a generator to emit special reflection tokens that control retrieval and score candidate generation segments during autoregressive decoding.^{[7]Reference 7Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.https://arxiv.org/abs/2310.11511} A relevant passage can still fail to support a particular claim, so ask two questions separately: should the model retrieve, and does the returned evidence support its next claim?

Reflection tokens

Self-RAG integrates one retrieval token family and three critique token families directly into the generator's vocabulary:^{[7]Reference 7Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.https://arxiv.org/abs/2310.11511}

1. Retrieve with values Yes, No, or Continue. This predicts whether to fetch evidence before the next segment (typically a sentence). No can avoid unnecessary retrieval. Continue reuses prior evidence. Actual latency savings depend on the serving path; a no-retrieval prediction can also be wrong.
2. ISREL, Relevant or Irrelevant. The generator predicts passage relevance before its passage-conditioned continuation. Decoding can reward relevant branches or apply hard filtering; the token doesn't automatically establish that every noisy passage was removed.
3. ISSUP, Fully supported, Partially supported, or No support. After generating a candidate segment, the model predicts support in the passage. A wrong support judgment can still select an unsupported claim.
4. ISUSE, ratings 1 to 5. This predicts overall response utility. The decoder maps the rating distribution into a weighted score in $[-1,1]$; that score is not a calibrated probability of helpfulness.

We'll use readable paper-style labels such as [Retrieve=Yes] and [ISSUP=Fully Supported]. These describe token families, not literal strings to paste into an arbitrary model tokenizer. A deployed checkpoint has its own registered token spellings and token IDs.

Walk the same launch-throughput query through those tokens. Before reading the sequence, predict the first decision and final support label: the model needs policy evidence, and the generated claim must remain narrower than any invented detail. A Self-RAG model might emit:

1. [Retrieve=Yes]: the model decides it needs external policy evidence before answering.
2. It retrieves an illustrative passage: "Temporary tokens-per-minute (TPM) increases require approval. Clients must use exponential backoff after rate-limit errors."
3. [ISREL=Relevant]: the retrieved passage contains actionable quota rules.
4. The model generates: "For launch traffic, request a temporary TPM increase and keep exponential backoff enabled."
5. [ISSUP=Fully Supported]: the claim is factually supported by the passage.
6. [ISUSE=4]: the response is helpful and directly answers the user's intent.

Without reflection tokens, a standard RAG pipeline might have retrieved the same passage without explicitly scoring passage relevance or claim support. Reflection tokens expose the model's predicted judgments for scoring and control; they aren't proof that a claim is true.

Architecture and segment-level beam search

Self-RAG is more than "retrieve once, then critique at the end." At inference time it can emit a retrieval decision, retrieve top-$k$ passages on demand, generate candidate segments conditioned on different passages in parallel, and score those branches with reflection-token probabilities.^{[7]Reference 7Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.https://arxiv.org/abs/2310.11511}

During decoding, Self-RAG runs a segment-level beam search (default beam width 2). For each candidate segment $y_t$ conditioned on passage $d$, the beam score balances generation likelihood with a weighted critique score:

$\text{Score}(y_t,d)=L(y_t,d)+w_{\text{rel}}s_{\text{rel}}+w_{\text{sup}}s_{\text{sup}}+w_{\text{use}}s_{\text{use}}$

Each $s$ normalizes token probabilities within its own family. Thus $s_{\text{rel}}=p_R/(p_R+p_I)$ and $s_{\text{sup}}=(p_F+0.5p_P)/(p_F+p_P+p_N)$. Here $R/I$ mean relevant/irrelevant and $F/P/N$ mean fully/partially/no support. Each $p$ is a token probability at its reflection step; family probabilities needn't sum to one over the full vocabulary. Utility is $s_{\text{use}}=\sum_{i=1}^{5}a_i p_i/\sum_{i=1}^{5}p_i$ with $a=[-1,-0.5,0,0.5,1]$. The critique reward is linear. Utility can be zero or negative, so taking its logarithm would be undefined.^{[7]Reference 7Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.https://arxiv.org/abs/2310.11511}

The paper combines a generation term $L$ with that reward. The released inference code optionally adds the exponential of mean generated-token log probability, and also supports critique-only scoring.^{[8]Reference 8Self-RAG released short-form inference scorer (pinned source)https://github.com/AkariAsai/self-rag/blob/1fcdc420e48f50a7d7ab1ece5494221b93252e99/retrieval_lm/run_short_form.py#L134-L157} Check the exact decoder configuration before reproducing results. The example below isolates the critique reward and doesn't implement sequence scoring or beam search.

Asai et al.'s reported default critique weights are 1.0 for ISREL, 1.0 for ISSUP, and 0.5 for ISUSE; they raise the support weight $w_{\text{sup}}$ when citation precision is critical. In the example below, with equal generation likelihood assumed and weights 0.2 / 0.6 / 0.2, Branch A (grounded policy claim) defeats Branch B (an unsupported claim of unlimited traffic), even though Branch B scored slightly higher on user utility:

Implementation note

Deploying a true Self-RAG system requires a generator specifically fine-tuned to emit these reflection tokens during generation. Asai et al. train Llama 2 7B and 13B generators that way, with a separate Llama 2 7B critic that inserts reflection tokens into supervised examples offline before generator training.^{[7]Reference 7Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.https://arxiv.org/abs/2310.11511} At inference the generator emits the tokens itself; you don't host the critic.

A prompted model can imitate parts of this control loop. Without reflection-token fine-tuning, you're building a prompted critique loop: routing, retrieval grading, and answer validation orchestrated in application code. Separate hosted calls can add round-trips; local, batched, or combined checks have different costs. Self-RAG predicts reflection tokens during decoding, but retrieval, passage-conditioned continuations, and critique tokens still add serving work.

Self-RAG cost comes from on-demand retrieval plus branching over multiple passages and scoring those branches, not from a few extra control tokens. Evaluate it where support-aware generation is worth that additional serving path.

The scoring sketch below executes the family normalization with authored token probabilities. It doesn't call a trained model or implement beam search. Branch A's support distribution is [0.94, 0.04, 0.02], so its support score is 0.94 + 0.5 * 0.04 = 0.96. Its utility score is 0.81. Check how these become a weighted score of 0.922.

from dataclasses import dataclass
from math import isfinite

def family_score(probabilities: list[float], weights: list[float]) -> float:
    if len(probabilities) != len(weights) or not probabilities:
        raise ValueError("one probability per family weight is required")
    if any(not isfinite(p) or not 0 <= p <= 1 for p in probabilities):
        raise ValueError("probabilities must be finite and in [0, 1]")
    total = sum(probabilities)
    if total <= 0:
        raise ValueError("token family must have positive probability mass")
    return sum(p * weight for p, weight in zip(probabilities, weights)) / total

@dataclass(frozen=True)
class CandidateSegment:
    text: str
    relevance: list[float]  # relevant, irrelevant
    support: list[float]  # fully, partially, unsupported
    utility: list[float]  # ratings 1 through 5

def branch_score(candidate: CandidateSegment) -> float:
    return (
        0.2 * family_score(candidate.relevance, [1, 0])
        + 0.6 * family_score(candidate.support, [1, 0.5, 0])
        + 0.2 * family_score(candidate.utility, [-1, -0.5, 0, 0.5, 1])
    )

candidates = [
    CandidateSegment("temporary TPM increase requires approval",
                     [0.92, 0.08], [0.94, 0.04, 0.02], [0.01, 0.02, 0.04, 0.20, 0.73]),
    CandidateSegment("all launch traffic is unlimited",
                     [0.95, 0.05], [0.20, 0.16, 0.64], [0, 0, 0.04, 0.16, 0.80]),
]
winner = max(candidates, key=branch_score)

print(f"chosen segment: {winner.text}")
print(f"score: {branch_score(winner):.3f}")
assert abs(branch_score(candidates[1]) - 0.534) < 1e-12
assert family_score([1, 1, 1, 1, 1], [-1, -0.5, 0, 0.5, 1]) == 0
assert family_score([1, 0, 0, 0, 0], [-1, -0.5, 0, 0.5, 1]) == -1

chosen segment: temporary TPM increase requires approval
score: 0.922

CRAG (Corrective RAG)

Corrective Retrieval-Augmented Generation (CRAG) focuses on evaluation and correction after initial retrieval. It doesn't require retraining the answer generator to emit reflection tokens. A separately trained retrieval evaluator scores retrieved documents and selects correction routes; the paper combines it with both ordinary and Self-RAG generators.^{[9]Reference 9Corrective Retrieval Augmented Generation.https://arxiv.org/abs/2401.15884}

The evaluator and 3-way confidence routing

In the original paper, the evaluator is a lightweight T5-large model fine-tuned to score each retrieved question-document pair on a $[-1, 1]$ range, then threshold those scores into one of three distinct paths:^{[9]Reference 9Corrective Retrieval Augmented Generation.https://arxiv.org/abs/2401.15884}

1. Correct ($\max \text{score} > \delta_U$): At least one document clears the upper threshold. CRAG treats the set as usable and refines its internal evidence. A high relevance score doesn't establish source truth or coverage of every required fact.
2. Incorrect ($\max \text{score} < \delta_L$): Every document falls below the lower threshold. CRAG discards the internal evidence and searches the web in its paper configuration. This is a predicted failure, not proof that all documents were irrelevant.
3. Ambiguous (otherwise): Neither condition holds, including threshold equality. CRAG combines refined internal and external evidence. The score alone doesn't determine which fact is missing.

Yan et al. set those thresholds empirically per dataset, for example $0.59$ and $-0.99$ on PopQA.^{[9]Reference 9Corrective Retrieval Augmented Generation.https://arxiv.org/abs/2401.15884} The sketch below uses a 0-1 teaching scale (upper=0.7, lower=0.2) so you can check the three-way rule by hand. Don't copy those teaching numbers into production without calibrating on your own labeled evals.

Fallback security and data governance

A request for a provider's public API rate limits may justify public web fallback after Incorrect retrieval. But a query about your organization's internal infrastructure, customer data, or security runbooks never should: public search engines can't answer internal questions, and sending private queries to third-party search APIs risks catastrophic data leakage.

Guarded web fallback requires production controls:

• Tenant and data policy gating: Never export queries containing customer names, proprietary code, or internal hostnames.
• Query review: Use an approved public query. Removing names alone can leave confidential intent or other identifying details; sanitization is not authorization.
• Source and network restrictions: Allow only approved public sources. Revalidate redirects and resolved addresses; block private, loopback, link-local, and metadata destinations, including IPv6 equivalents.
• Injection handling: Treat fetched text as untrusted evidence. Source allowlisting and text sanitization don't prevent every injection; keep tool permissions and policy decisions outside retrieved instructions.

If an internal question can't be answered from internal evidence, the safest action is to refuse and escalate, not search the public web. Retrieval relevance isn't permission to export confidential context.

Before running the classifier, map each best score to a path: 0.81 should stay internal, 0.17 should leave it, and 0.45 should combine sources.

from typing import Literal
from math import isfinite

Decision = Literal["correct", "incorrect", "ambiguous"]

def crag_action(scores: list[float], lower: float = 0.2, upper: float = 0.7) -> Decision:
    # Teaching 0-1 scale. The paper scores [-1, 1] with dataset-specific thresholds.
    if not 0 <= lower < upper <= 1:
        raise ValueError("thresholds must satisfy 0 <= lower < upper <= 1")
    if any(not isfinite(score) or not 0 <= score <= 1 for score in scores):
        raise ValueError("scores must be finite and in [0, 1]")
    if not scores:
        return "incorrect"  # No evidence: correction still needs policy approval.
    best_score = max(scores)
    if best_score > upper:
        return "correct"
    if best_score < lower:
        return "incorrect"
    return "ambiguous"

print(f"strong retrieval: {crag_action([0.81, 0.15])}")
print(f"weak retrieval: {crag_action([0.10, 0.17])}")
print(f"uncertain retrieval: {crag_action([0.45, 0.09])}")
assert crag_action([0.2]) == crag_action([0.7]) == "ambiguous"
assert crag_action([]) == "incorrect"

strong retrieval: correct
weak retrieval: incorrect
uncertain retrieval: ambiguous

Knowledge refinement (decompose-then-recompose)

Even relevant documents contain distracting noise. Long retrieved passages often contain navigation menus, copyright notices, or tangential details that dilute the generator's attention. CRAG introduces a decompose-then-recompose algorithm to isolate key facts:

1. Decompose: Segment each document into smaller knowledge strips. The paper keeps a one- or two-sentence document as one strip and splits longer passages into units of a few sentences.
2. Filter: Score each strip with the retrieval evaluator. Discard strips whose confidence score falls below a filtering threshold $\beta$ (Yan et al. use $\beta = -0.5$).
3. Recompose: Concatenate the surviving, high-confidence strips in their original document order to form an information-dense context payload.

The runnable controller below demonstrates the routing contract. It preserves source IDs, caps the number of strips, and denies external web fallback by default. It uses keyword relevance and a deliberately crude sentence splitter; it doesn't reproduce the trained evaluator or bound token length. An application must explicitly supply an approved public query before external search can fire:

Predict the source IDs for each route: correct keeps internal evidence, incorrect keeps only public evidence, and ambiguous combines both. Then check the failure case: an empty internal result with no approved public query makes zero web calls. The returned object is candidate context, not a verified answer. A real adapter must enforce ACLs, source policy, network controls, timeouts, and claim-level support checks separately.

from dataclasses import dataclass
from typing import Literal, Protocol

Decision = Literal["correct", "incorrect", "ambiguous"]

@dataclass(frozen=True)
class Evidence:
    source_id: str
    text: str

@dataclass(frozen=True)
class RequestPolicy:
    # Trusted application input, never populated by the rewrite model.
    approved_public_query: str | None = None

class SearchBackend(Protocol):
    def search(self, query: str, k: int = 5) -> list[Evidence]: ...

class RetrievalEvaluator(Protocol):
    def classify(self, query: str, docs: list[Evidence]) -> Decision: ...
    def is_relevant_strip(self, query: str, strip: str) -> bool: ...

class CorrectiveRetrieval:
    def __init__(self, internal: SearchBackend, evaluator: RetrievalEvaluator,
                 public: SearchBackend) -> None:
        self.internal, self.evaluator, self.public = internal, evaluator, public

    def run(self, query: str, policy: RequestPolicy) -> tuple[str, list[Evidence]]:
        # Internal backend must enforce the caller's ACL before returning hits.
        docs = self.internal.search(query, k=5)
        decision = self.evaluator.classify(query, docs) if docs else "incorrect"
        if decision not in {"correct", "incorrect", "ambiguous"}:
            raise ValueError("unknown retrieval decision")
        context = self.refine(query, docs) if decision != "incorrect" else []
        if decision != "correct":
            if not policy.approved_public_query or not policy.approved_public_query.strip():
                return "needs_authorized_evidence", context
            # A separate reviewed query avoids leaking the private request.
            public_query = policy.approved_public_query
            external = self.refine(public_query, self.public.search(public_query, k=5))
            # Reserve room for both sources when ambiguity triggers a mixed route.
            context = context[:3] + external[:3] if context and external else context + external
        # No answer generation or semantic support verification in this fixture.
        return ("candidate_context" if context else "no_evidence"), context[:6]

    def refine(self, query: str, docs: list[Evidence]) -> list[Evidence]:
        kept: list[Evidence] = []
        seen: set[tuple[str, str]] = set()
        for doc in docs:
            # Simplified splitter: real prose needs sentence-aware segmentation.
            for strip in (part.strip() for part in doc.text.split(".")):
                key = (doc.source_id, strip)
                if strip and key not in seen and self.evaluator.is_relevant_strip(query, strip):
                    kept.append(Evidence(doc.source_id, strip))
                    seen.add(key)
        return kept[:6]

class FakeSearch:
    def __init__(self, docs: list[Evidence]) -> None:
        self.docs, self.calls = docs, []

    def search(self, query: str, k: int = 5) -> list[Evidence]:
        self.calls.append(query)
        return self.docs[:k]

class FakeEvaluator:
    def __init__(self, decision: Decision) -> None:
        self.decision = decision

    def classify(self, query: str, docs: list[Evidence]) -> Decision:
        return self.decision  # Authored branch fixture, not a relevance prediction.

    def is_relevant_strip(self, query: str, strip: str) -> bool:
        return bool({"quota", "tpm", "backoff"} & set(strip.lower().split()))

internal_docs = [Evidence("internal:quota-v3", "TPM increases need internal approval. Old SDK notes.")]
public_docs = [Evidence("public:retry-v2", "Use exponential backoff after rate limits.")]
for decision in ("correct", "incorrect", "ambiguous"):
    internal, public = FakeSearch(internal_docs), FakeSearch(public_docs)
    pipeline = CorrectiveRetrieval(internal, FakeEvaluator(decision), public)
    status, context = pipeline.run("our private TPM process", RequestPolicy("public retry policy"))
    print(decision, status, [item.source_id for item in context])
    assert len(public.calls) == (decision != "correct")
    assert all(call == "public retry policy" for call in public.calls)

public = FakeSearch(public_docs)
blocked = CorrectiveRetrieval(FakeSearch([]), FakeEvaluator("incorrect"), public)
status, context = blocked.run("private project secret", RequestPolicy())
assert status == "needs_authorized_evidence" and not context and not public.calls
print("private fallback:", status, "web_calls=0")
assert blocked.run("private project secret", RequestPolicy("   "))[0] == "needs_authorized_evidence"
assert not public.calls

# Relevant-looking words can survive even when the strip negates the claim.
evaluator = FakeEvaluator("correct")
assert evaluator.is_relevant_strip("TPM increase", "No TPM increase is permitted")
print("keyword overlap is not claim support")

correct candidate_context ['internal:quota-v3']
incorrect candidate_context ['public:retry-v2']
ambiguous candidate_context ['internal:quota-v3', 'public:retry-v2']
private fallback: needs_authorized_evidence web_calls=0
keyword overlap is not claim support

Agentic and iterative retrieval

Self-RAG and CRAG both introduce feedback, but at different boundaries: Self-RAG can make retrieval and critique decisions while generating segments, while CRAG routes after initial retrieval. An application can generalize that feedback into agentic retrieval: instead of one fixed retrieve-then-generate pass, a model with tool access searches, reads results, tests whether evidence is sufficient, and either answers or searches again with a refined query.

The mechanism is older than the agent framing. IRCoT studies one version of this pattern: retrieve, reason a step, use that step to drive the next retrieval, and repeat until the chain of evidence is complete.^{[10]Reference 10Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionshttps://arxiv.org/abs/2212.10509} For "Which embedding endpoint has the highest p95 latency, and what retry budget applies to that endpoint?", the second search can use the endpoint identified by the first. Compare latency over the same workload and time window. If the first hop is uncertain, preserve that uncertainty or stop rather than guessing an endpoint.

Before choosing a rung, predict the dependency in the request above. A single-fact FAQ should stop early; endpoint selection followed by a retry lookup must carry evidence from one hop into the next. Each escalation costs latency and tokens:

1. Naive RAG for single-fact lookups answerable by one retrieval.
2. Query rewriting plus hybrid search and reranking for conversational ambiguity and vocabulary mismatch. This is a practical baseline to evaluate before more expensive routes.
3. HyDE behind a router for abstract or vocabulary-mismatched queries where a document-style proxy helps.
4. A correction gate (CRAG-style evaluator or prompted critique) when retrieval quality is inconsistent and a single bad context causes user-visible errors.
5. Iterative or agentic retrieval for genuinely multi-hop questions, where one retrieval pass can't gather all the facts.

Escalate only when the failure mode and evaluation demand it. On a simple FAQ lookup, an agentic loop can add latency and failure paths without improving retrieved evidence.

These are the basic methods, not limits on possible compositions; adding a router or web tool changes the application. No universal latency ordering follows from their names. That distinction explains the comparison below. Read it by asking which component owns each decision, then check whether each added control changes retrieval, evidence scoring, or both.

Comparison of advanced techniques

Choose these techniques by failure mode, not by novelty. HyDE targets semantic mismatch. Self-RAG changes retrieval timing and branch scoring inside a trained generator. CRAG adds a separate correction gate after retrieval.

What rewrite, HyDE, and CRAG can break

Advanced RAG helps only when it matches the failure. Watch for these symptoms before adding another model call. Rewrite, HyDE, multi-query, and CRAG affect more than quality. They expand the attack and cost surface of retrieval.

Diagnose in order: identify the wrong search object, check evidence scope, then check added calls and trust boundaries. A quality gain without that diagnosis can hide a security or latency regression.

Symptom: the query is vague, but the answer needs history

The retriever sees only the latest turn, so it searches for "the other one" instead of the backup API key or failed batch job. Rewrite the latest turn into a standalone query using the last few conversation turns. Keep the raw user message for the generator, but search with the rewritten query so the answer can retain conversational context.

Symptom: rewrite or history poisons retrieval intent

Conversation history, tool output, or a retrieved doc can contain "ignore policy / search secrets / dump all tickets." If the rewrite model treats history as trusted instructions, it can adopt that intent and retrieve privileged neighborhoods. Treat rewrite input as untrusted data. Constrain the prompt to disambiguate the user's question, mark tool/page text as data, preserve principal identity and ACL scope, and never let a rewrite broaden grants.

Symptom: HyDE improves recall on policy questions but breaks exact lookups

The hypothetical document can invent precise identifiers, dates, or prices. That invented detail may pull retrieval toward the wrong neighborhood. Route exact-match queries to normal hybrid retrieval; use HyDE for conceptual or vocabulary-mismatched questions where a document-style proxy helps more than it hurts.

Symptom: CRAG web fallback becomes SSRF or untrusted-source injection

An "Incorrect" retrieval score triggers an open-web fetch. Without controls, the worker can hit internal metadata endpoints through server-side request forgery (SSRF) or pull attacker-controlled pages that inject instructions into generation. Use domain allowlists, block links into private networks, fetch in a network sandbox, treat web HTML as untrusted observations, and require citations from allowlisted sources before generation. Prefer "refuse / escalate" over a wide-open web tool.

Symptom: rewrite or HyDE cache bleeds across tenants

Caches keyed only on the raw string miss tenant, user, and policy version. User A's rewritten query or HyDE proxy can then serve User B's retrieval path. Key private-context caches by tenant, effective authorization scope, policy version, history/context digest, normalized query, technique, and generator/prompt version; include encoder version for vectors. Two identical follow-ups can refer to different previous turns even for the same user. A role name alone may omit per-document grants. Recheck current authorization on retrieval, and invalidate revoked scope rather than relying on TTL alone. Share only artifacts explicitly classified as public and context-independent.

Symptom: multi-query / agentic loops become a cost DoS

Decomposition and multi-query fan-out multiply embedding, ANN, and LLM calls per user message. Set hard caps on rewrite count, sub-queries, web fetches, and dollar/token budgets per request. Fail closed with a partial answer or clarification when the budget trips, instead of letting a loop define its own spend.

Symptom: Self-RAG-inspired code works in prompts but isn't true Self-RAG

True Self-RAG trains a generator to emit retrieval and critique tokens during generation. Separate prompted grading calls can mimic the control loop, but they don't create the same reflection-token model. Name the system honestly: call it a prompted critique loop or CRAG-style evaluator unless you're hosting a model trained with Self-RAG reflection tokens.

Symptom: accuracy rises, but users feel the product is slow

Rewriting, HyDE generation, dense retrieval, sparse retrieval, reranking, grading, and final synthesis can become a long sequential path. Parallelize independent retrievals, cap reranker candidates, cache repeated rewrites or hypothetical documents with tenant-safe keys, and measure retrieval and generation latency separately. If supported-answer quality misses its gate, lower latency alone isn't a release win.

Latency waterfall economics and deployment gates

Don't stack rewrite, HyDE, CRAG, and an agent loop on the first launch-throughput query. Treat each control as an experiment: name the failure, add one intervention, and compare against the same labeled cases.

Advanced RAG pipelines turn one user message into several sequential model and retrieval calls. To set realistic service level objectives (SLOs), map out where milliseconds go in the latency waterfall.

Trace the request before estimating a percentile

There is no portable latency tier for "HyDE" or "CRAG." Model size, hardware, prompt length, queueing, network path, retrieved candidate count, and cache state all matter. Record a request trace under a declared workload. For time to first token (TTFT), stop the end-to-end clock when the first answer token reaches the user; full-answer completion is a different measurement.

This is one invented uncached rewrite-plus-rerank request, with serial stages and no unlisted overhead:

For this trace, TTFT is 395 ms. It is not a p95, a typical service measurement, or a forecast for an 8B or 70B model. An uncached HyDE route adds proxy generation and encoding before its search. A corrective route adds evaluation and possibly external retrieval. Work can overlap or be skipped, so adding controls needn't produce a fixed latency ordering.

Can you add component p95s to get end-to-end p95? No. Calculate each request's total from its trace, then take the percentile of those totals. The slow requests at two stages may be different requests. Predict the result below: each stage is slow once, on different requests, so two requests are slow overall.

from math import ceil

def nearest_rank(values: list[int], percentile: float) -> int:
    if not values or not 0 < percentile <= 1:
        raise ValueError("need observations and a percentile in (0, 1]")
    return sorted(values)[ceil(percentile * len(values)) - 1]

# Invented serial durations: (stage A, stage B), in milliseconds.
traces = [(10, 10)] * 18 + [(100, 10), (10, 100)]
stage_a_p95 = nearest_rank([a for a, b in traces], 0.95)
stage_b_p95 = nearest_rank([b for a, b in traces], 0.95)
request_p95 = nearest_rank([a + b for a, b in traces], 0.95)
print(f"sum of stage p95s: {stage_a_p95 + stage_b_p95} ms")
print(f"p95 of request totals: {request_p95} ms")
assert (stage_a_p95, stage_b_p95, request_p95) == (10, 10, 110)

sum of stage p95s: 20 ms
p95 of request totals: 110 ms

This tiny sample exposes a calculation error; it doesn't estimate a stable production tail. Declare your percentile convention, sample size, workload, and treatment of failures and timeouts.

Tail latency hazards and fan-out variance

When three searches launch together and all results are required, their wait is $\max(T_1,T_2,T_3)$. Suppose each search finishes within 25 ms with probability 0.95. If those completion times are independent, all three finish by 25 ms with probability $0.95^3\approx0.857$, so 14.3% of fan-outs exceed that threshold. With identical independent latency distributions, the fan-out p95 corresponds to each search's $0.95^{1/3}\approx0.983$ quantile. Shared load and query difficulty can correlate the times; measure the actual fan-out instead of assuming independence. Planning and fusion remain outside this wait.

To protect user-facing SLOs:

• Speculative retrieval: Evaluate searching the original query while a rewrite runs. Reuse its evidence only if it covers the actual request. A high similarity score on "What do I do now?" cannot recover the missing referent. Cancellation may not stop billed work.
• Query routing: Evaluate a cheap router on labeled failure types. There is no universal 85/15 split or latency target; report route frequency, misrouting, and quality on your traffic.
• Tenant-scoped semantic caching: Cache common rewrites, HyDE vectors, and reranked contexts with tenant- and permission-aware cache keys.

Adding controls incrementally

Add a control only when a measured failure needs it:

Stable baseline

Keep raw-query hybrid search (dense retrieval + sparse keyword retrieval) as a control. Add rewriting for history-dependent turns, not automatically for every request. Compare retrieved evidence and answer support; a rewrite adds generation work and can also damage an already precise query.

Add reranking

Add a cross-encoder reranker after retrieval when initial retrieval has decent recall but final candidates are noisy. It can improve precision, at moderate cost: cross-encoders are slower than initial retrieval, so keep candidate count small.

Specialized handling (HyDE / router)

Use a router to classify queries. For a conceptual request such as "Explain launch-week embedding throughput planning," evaluate a HyDE route if your labeled cases show vocabulary mismatch. For a precise request such as "What happened in incident INC-48291?", choose standard retrieval because HyDE risks inventing the exact identifier. The router pays HyDE's extra generation only where a document-style proxy helps.

Add correction loops

If accuracy is still insufficient, add a prompted critique loop or a CRAG-style evaluator before investing in a true Self-RAG model. Prompt the model or a lightweight evaluator to grade retrieved documents before answer generation; when evidence is weak, trigger a rewrite, a second retrieval pass, or web fallback. This tests whether correction helps your failures without requiring reflection-token fine-tuning; it doesn't establish parity with Self-RAG.

Don't release a more elaborate route because it improves a few anecdotes. Compare supported-answer quality and latency on a labeled set, then release only paths that meet both requirements. For each route, retain the corpus snapshot, model and retriever versions, candidate count, latency percentiles, token or dollar cost, and citation-support results. Without that evidence, route comparisons aren't reproducible.

The next table is synthetic decision data, not measured model performance. Define supported accuracy as the fraction of all answerable evaluation questions answered correctly with source-supported claims; abstentions on those questions count as failures. Separately score correct abstention on unanswerable cases, citation precision/completeness, and access violations.

For a real release, record the sample size, paired baseline changes, important slices, and uncertainty before choosing a threshold. Measure end-to-end p95 at a declared workload, not just retriever latency. Here, the authored gate is supported accuracy at least 0.93 and p95 no more than 350 ms. Predict the eligible route, and what should happen if none qualifies.

evaluations = [
    {"route": "rewrite+hybrid", "supported_accuracy": 0.91, "p95_ms": 180},
    {"route": "hyde+rerank", "supported_accuracy": 0.94, "p95_ms": 260},
    {"route": "agentic-loop", "supported_accuracy": 0.95, "p95_ms": 710},
]
minimum_supported_accuracy = 0.93
maximum_p95_ms = 350

eligible = [
    row for row in evaluations
    if row["supported_accuracy"] >= minimum_supported_accuracy
    and row["p95_ms"] <= maximum_p95_ms
]
released = max(eligible, key=lambda row: row["supported_accuracy"], default=None)
if released is None:
    print("hold: no route clears both gates")
else:
    print(f"eligible synthetic route: {released['route']}")
    print(f"supported_accuracy={released['supported_accuracy']:.2f} p95_ms={released['p95_ms']}")
assert max([], key=lambda row: row["supported_accuracy"], default=None) is None

eligible synthetic route: hyde+rerank
supported_accuracy=0.94 p95_ms=260

Hands-on diagnostics: evaluating query transforms on live misses

Stay with five API-docs messages and measure top-3 evidence hits. This exercise needn't reproduce paper scores. It should connect each retrieval miss to one intervention and one release decision.

Exercise: Build a query rewriter for a developer-support assistant

Collect five developer-support messages from an API docs assistant (or write realistic ones). Include one ambiguous message that needs conversation history, one complex comparison, and one vague keyword so each control has a case to test.

Step 1, Rewrite: Write a Python function that takes a developer message plus the last two turns of chat history and outputs a standalone query. Run it on the five messages and inspect the results. Check whether the rewritten query contains full intent while leaving authorization and tenant scope unchanged.

Step 2, Measure: For each original message, manually decide which internal policy documents should be retrieved. Run both original and rewritten queries against the same dozen chunks and retrieval configuration. The keyword retriever above is enough to debug the loop; a later dense-encoder run is a separate experiment. Label a gold set of required chunk IDs before inspecting results. For each question, report Hit@3 (any required chunk found), Recall@3 (fraction of required chunks found), and all-required-found (complete evidence coverage). A comparison needing three facts can score Hit@3=1 while missing two of them. Don't assume rewriting wins.

Keep these five examples as development cases, not a release benchmark. Add held-out questions before tuning gates.

Step 3, Diagnose: Pick one message where retrieval still fails. Before changing code, predict the failure: vocabulary mismatch (try multi-query expansion), semantic asymmetry (try HyDE), or weak evidence (try a CRAG-style evaluator). Implement only that fix and measure again.

Expected outcome: Record which interventions improve top-3 evidence hits or catch weak retrieval on the examples, and which add latency without a gain. A small exercise may not reproduce paper results; its value is exposing the measurement loop.

Architecture decisions: matching retrieval rungs to failure modes

The exercise should leave you with evidence about retrieval quality, latency, and the shape of each failure. Turn those observations into a route decision before adding a more expensive path. Three deployment questions provide the boundary.

When does HyDE justify its latency cost?

HyDE adds at least one extra generation call before retrieval. That cost is worth paying only when measured retrieval quality improves enough to justify it. A smaller instruction model may be adequate for the hypothetical document, but evaluate it rather than assuming equivalence. Cache repeated proxy documents only when generation configuration and source policy make reuse valid. For exact-match lookups like incident IDs, error codes, or dates, skip HyDE.

When does Self-RAG degrade instead of help?

Self-RAG degrades when retrieval itself is weak or when the model's learned critique tokens stop correlating with real answer quality. It also raises inference cost because the model may retrieve on demand, branch over multiple passages, and spend extra decoding steps on critique tokens before choosing a continuation. For a simple FAQ bot, first measure a cheaper baseline such as rewrite plus reranking or a correction gate before committing to a specialized Self-RAG serving path.

Can you combine HyDE and Self-RAG?

As a system design, you can use HyDE to propose initial candidate passages, then let a true Self-RAG model score branches with ISREL, ISSUP, and ISUSE. That composition needs its own evaluation; neither mechanism guarantees the other improves it. If you don't have a reflection-token model, describe the composition as HyDE plus a CRAG-style evaluator or prompted support checks rather than Self-RAG.