LearnApplied LLM EngineeringHybrid Search: Dense + Sparse

🔍MediumRAG & Retrieval

Hybrid Search: Dense + Sparse

Upgrade a permission-safe RAG retriever with BM25, semantic scores, rank fusion, and recall gates for exact codes and paraphrased policy questions.

32 min read

Learning path

Step 72 of 197 in the full curriculum

Production RAG Pipelines Reranking and Cross-Encoders for RAG

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

Production RAG Pipelines gave policy-answerer-v1 a hard RAG rule: only permitted, current evidence may reach an answer. Its retrieve() function filters first, then counts overlapping tokens. That recovered a question from Luna, an EU support specialist, about stale service-account keys. It missed the paraphrase that chapter already named: "refresh an expired machine credential" when the policy says "stale service-account key rotation."

Hybrid search combines sparse lexical retrieval with dense vector retrieval. We'll keep the existing evidence boundary and add diagnostic code KROT-14 to the fictional policy. BM25, cosine similarity, and Reciprocal Rank Fusion (RRF) from Core Retrieval Algorithms become the retrieval path in policy-answerer-v2. The lab executes ranking and filtering, but uses hand-authored vectors rather than evaluating a learned encoder.

Two query styles, two possible misses

Luna needs the same policy for two distinct searches:

Query	Useful signal	Required evidence
`KROT-14`	Exact policy code	`eu-key-rotation-v2-rule`
`refresh expired machine credential`	Meaning close to "stale service-account key rotation"	`eu-key-rotation-v2-rule`

A word-matching index has a decisive clue for the first query and no shared vocabulary for the second. That's the vocabulary mismatch problem (or lexical gap): without stemming, synonym rules, or learned expansion, our analyzer cannot connect these two phrases. Sparse retrieval can bridge some lexical gaps when its representation includes expanded terms.

A trained dense encoder may connect the paraphrase to the policy even without shared words. It may also struggle with an unfamiliar internal identifier such as KROT-14. Its tokenization and ranking depend on the checkpoint; splitting a code into subwords does not establish that the encoder will miss it. We'll construct complementary failures to test the hybrid path, then distinguish that regression test from an experiment with a real encoder.

Both lanes receive the same permitted corpus in this service. Fusion can improve recall, but can also demote a useful hit. Authorization decides which candidates may cross the retrieval boundary; fusion only orders those candidates.

Diagram showing Luna query, Filter permitted, BM25 sparse, and Dense cosine. — Luna query, Filter permitted, BM25 sparse, and Dense cosine.

The filter is the evidence boundary from the previous lesson. Sparse and dense lanes only see IDs that already passed it.

Recreate the permitted candidate universe

The lab reuses the policy shape from the previous lesson. It adds diagnostic policy code KROT-14 so an exact-identifier query has an unambiguous expected result. Two tempting records remain in storage but must not be searchable for Luna: a superseded revision and a restricted admin-only rule.

Before running the filter, predict the searchable set. The current EU support rule should survive; the old revision should fail freshness, and the admin rule should fail authorization. That set becomes input to both lanes.

permitted-candidates.py

from __future__ import annotations

from dataclasses import dataclass
from datetime import date
from math import fsum, hypot, isfinite, log
import re

@dataclass(frozen=True)
class PolicyChunk:
    chunk_id: str
    document_id: str
    parent_id: str
    version: str
    region: str
    acl_tag: str
    effective_from: date
    effective_to: date | None
    text: str

@dataclass(frozen=True)
class Caller:
    actor_id: str
    region: str
    acl_tags: frozenset[str]

EVAL_DATE = date(2026, 5, 27)
LUNA = Caller("luna-48291", "EU", frozenset({"support:eu"}))
CHUNKS = [
    PolicyChunk(
        "eu-key-rotation-v2-rule",
        "eu-access",
        "eu-access-v2",
        "eu-access/2026-04-01",
        "EU",
        "support:eu",
        date(2026, 4, 1),
        None,
        (
            "Rule KROT-14. Stale service-account keys qualify for automated "
            "rotation up to 14 days after becoming stale, provided the risk "
            "signal is at most 48 hours old."
        ),
    ),
    PolicyChunk(
        "eu-key-rotation-v1-rule",
        "eu-access",
        "eu-access-v1",
        "eu-access/2025-02-01",
        "EU",
        "support:eu",
        date(2025, 2, 1),
        date(2026, 3, 31),
        "Rule KROT-14. Stale service-account keys require manual rotation within 30 days.",
    ),
    PolicyChunk(
        "restricted-admin-key-rotation",
        "admin-override-terms",
        "admin-override-terms",
        "admin-override/2026-05-01",
        "EU",
        "security:admins",
        date(2026, 5, 1),
        None,
        "ADMIN-KROT-1. Security admins may run emergency key rotation.",
    ),
    PolicyChunk(
        "eu-session-timeout-v1-rule",
        "eu-session",
        "eu-session-v1",
        "eu-session/2026-01-03",
        "EU",
        "support:eu",
        date(2026, 1, 3),
        None,
        "Idle browser sessions expire after 30 days of inactivity.",
    ),
    PolicyChunk(
        "eu-audit-rebuild-v1",
        "eu-audit",
        "eu-audit-rebuild-v1",
        "eu-audit/2026-02-10",
        "EU",
        "support:eu",
        date(2026, 2, 10),
        None,
        "Rule AUD-7. Missing audit-log shards after ingestion failure qualify for replay rebuild.",
    ),
]

def is_current(chunk: PolicyChunk, on_date: date) -> bool:
    return chunk.effective_from <= on_date and (
        chunk.effective_to is None or on_date <= chunk.effective_to
    )

def permitted_chunks(
    caller: Caller,
    chunks: list[PolicyChunk],
    on_date: date,
) -> list[PolicyChunk]:
    return [
        chunk
        for chunk in chunks
        if chunk.region == caller.region
        and chunk.acl_tag in caller.acl_tags
        and is_current(chunk, on_date)
    ]

permitted = permitted_chunks(LUNA, CHUNKS, EVAL_DATE)
permitted_ids = [chunk.chunk_id for chunk in permitted]
print("Permitted current ids:", permitted_ids)
assert "eu-key-rotation-v2-rule" in permitted_ids
assert "eu-key-rotation-v1-rule" not in permitted_ids
assert "restricted-admin-key-rotation" not in permitted_ids

Output

Permitted current ids: ['eu-key-rotation-v2-rule', 'eu-session-timeout-v1-rule', 'eu-audit-rebuild-v1']

Every normal retrieval path below passes permitted, not CHUNKS, to its rankers. This is a code convention, not a sandbox: the ranker functions still accept arbitrary lists. A serving endpoint must derive Caller from trusted authentication and authorization state, not accept client-supplied ACL tags. Recheck permission when fetching text for context, and scope caches and traces to the same caller and policy version.

The fixed EVAL_DATE keeps replay behavior stable; a live service must supply its actual effective date. The chunk shape preserves document_id and parent_id. The previous lesson's permission checks and citation-packing contract still apply, but this lab does not implement a parent fetcher or answer generator.

The next check makes the negative cases visible. Look for two IDs that remain stored but never become searchable.

evidence-boundary-regression.py

blocked_ids = sorted(
    chunk.chunk_id
    for chunk in CHUNKS
    if chunk.chunk_id not in permitted_ids
)
print("Searchable by Luna:", permitted_ids)
print("Stored but blocked:", blocked_ids)
assert blocked_ids == ["eu-key-rotation-v1-rule", "restricted-admin-key-rotation"]
old_rule = CHUNKS[1]
assert is_current(old_rule, date(2026, 3, 31))  # end dates are inclusive here
assert not is_current(old_rule, date(2026, 4, 1))
assert not is_current(CHUNKS[0], date(2026, 3, 31))
assert permitted_chunks(Caller("other-region", "US", LUNA.acl_tags), CHUNKS, EVAL_DATE) == []
assert permitted_chunks(Caller("no-tags", "EU", frozenset()), CHUNKS, EVAL_DATE) == []

Output

Searchable by Luna: ['eu-key-rotation-v2-rule', 'eu-session-timeout-v1-rule', 'eu-audit-rebuild-v1']
Stored but blocked: ['eu-key-rotation-v1-rule', 'restricted-admin-key-rotation']

This test keeps a deliberately attractive hidden policy in storage. Later ranking changes fail loudly if they accidentally widen the searchable set.

Hybrid request path where Luna's query hits an ACL and freshness filter first. Three permitted chunks fork into a BM25 lane for KROT-14 and a dense lane for the paraphrase, then Reciprocal Rank Fusion merges those permitted IDs into a top-k candidate pool. The superseded v1 rule and the admin-only rule leave the filter as a dead end and never enter either lane. — Luna's query hits the ACL and freshness filter first. The three permitted IDs fork into BM25 and dense search; the superseded v1 rule and the admin-only rule leave as a dead end and never reach fusion.

Build the sparse lane with BM25

Sparse retrieval represents a document by vocabulary terms. Most coordinates are zero because a short policy chunk uses only a small part of the vocabulary. You already scored that representation with BM25 in Core Retrieval Algorithms: rare matching terms contribute more, term frequency saturates, and length normalization adjusts their contribution.^{[2]Reference 2The Probabilistic Relevance Framework: BM25 and Beyond.https://doi.org/10.1561/1500000019}

The analyzer below keeps KROT-14 as the single token krot-14. Splitting or dropping its digits can weaken the exact-match signal, so index and query analysis must agree. The scorer uses Lucene's documented positive IDF form, with a leading 1 +; this doesn't reproduce Lucene's entire index and scoring implementation.^{[3]Reference 3BM25Similarity: Apache Lucene 10.3.1https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/search/similarities/BM25Similarity.html}

For a first calculation, imagine three equal-length chunks and a code appearing once in only one of them. Its inverse document frequency is ln(1 + (3 - 1 + 0.5)/(1 + 0.5)) = ln(8/3), about 0.981. With one occurrence in an average-length chunk, the frequency factor is 2.2 / 2.2 = 1. The matching chunk therefore scores 0.981; nonmatching chunks score zero and aren't lexical candidates. The actual chunks below have different lengths, so their length factors differ.

For a query term $t$ and document $d$ , the lab computes:

\operatorname{BM25}(q,d)=\sum_{t \in q}\operatorname{IDF}(t) \frac{f(t,d)(k_1+1)} {f(t,d)+k_1(1-b+b\lvert d\rvert/\operatorname{avgdl})}

Here, $q$ is the set of distinct analyzed query terms, $f(t,d)$ is the term count in chunk $d$ , $\lvert d\rvert$ is its length in tokens, and avgdl is the corpus average. Two parameters control the curve:

$k_1$ controls term-frequency saturation. Holding the length-normalization factor fixed, the frequency factor approaches $k_1+1=2.2$ when $k_1=1.2$ . Repeated mentions have diminishing returns; this does not make the scorer immune to keyword stuffing.
$b$ controls document length normalization. For the same matching term and occurrence count in one corpus, $b=0.75$ gives a shorter chunk a larger contribution than a longer one. It does not establish which document is more relevant. The lab's $k_1=1.2$ and $b=0.75$ match the defaults documented for Lucene 10.3.1.^{[3]Reference 3BM25Similarity: Apache Lucene 10.3.1https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/search/similarities/BM25Similarity.html}

The exact identifier krot-14 occurs only in the relevant current chunk, so it receives positive lexical weight. This toy recomputes statistics on permitted chunks. A deployed index may use broader collection statistics; filtering its hits doesn't automatically recompute IDF per caller.

The analyzer also drops common function words. Without that stopword rule, a query containing only a shared word such as "a" could appear to retrieve an unrelated policy.

Before running the lane, predict the two outputs. KROT-14 should produce a strong hit because its token is rare; the paraphrase should return nothing because none of its words overlap the policy.

bm25-lane.py

TOKEN_RE = re.compile(r"[a-z0-9]+(?:-[a-z0-9]+)*")
STOPWORDS = {"a", "an", "the", "after", "for", "of", "is", "within", "when"}

def tokens(text: str) -> list[str]:
    return [
        token
        for token in TOKEN_RE.findall(text.lower())
        if token not in STOPWORDS
    ]

def finite_number(value: object) -> bool:
    if type(value) not in (int, float):
        return False
    try:
        return isfinite(value)
    except OverflowError:
        return False

def bm25_rank(
    query: str,
    chunks: list[PolicyChunk],
    top_k: int = 2,
    k1: float = 1.2,
    b: float = 0.75,
) -> list[tuple[PolicyChunk, float]]:
    if type(top_k) is not int or top_k <= 0:
        raise ValueError("top_k must be a positive integer")
    if not finite_number(k1) or k1 < 0 or not finite_number(b) or not 0 <= b <= 1:
        raise ValueError("require finite k1 >= 0 and 0 <= b <= 1")
    if len({chunk.chunk_id for chunk in chunks}) != len(chunks):
        raise ValueError("chunk IDs must be unique within the corpus")
    if not chunks:
        return []

    doc_tokens = {chunk.chunk_id: tokens(chunk.text) for chunk in chunks}
    avgdl = sum(len(value) for value in doc_tokens.values()) / len(chunks)
    query_terms = sorted(set(tokens(query)))
    if avgdl == 0 or not query_terms:
        return []
    ranked: list[tuple[PolicyChunk, float]] = []

    for chunk in chunks:
        document = doc_tokens[chunk.chunk_id]
        score = 0.0
        for term in query_terms:
            term_count = document.count(term)
            if term_count == 0:
                continue
            containing_docs = sum(term in value for value in doc_tokens.values())
            idf = log(1 + (len(chunks) - containing_docs + 0.5) / (containing_docs + 0.5))
            numerator = term_count * (k1 + 1)
            denominator = term_count + k1 * (1 - b + b * len(document) / avgdl)
            if not finite_number(numerator) or not finite_number(denominator):
                raise ValueError("BM25 arithmetic exceeded the finite numeric range")
            score += idf * (numerator / denominator)
            if not finite_number(score):
                raise ValueError("BM25 arithmetic exceeded the finite numeric range")
        if score > 0:
            ranked.append((chunk, score))

    return sorted(ranked, key=lambda item: (-item[1], item[0].chunk_id))[:top_k]

EXACT = "KROT-14"
PARAPHRASE = "refresh expired machine credential"
bm25_exact = bm25_rank(EXACT, permitted)
bm25_paraphrase = bm25_rank(PARAPHRASE, permitted)

print("BM25 exact:", [chunk.chunk_id for chunk, _ in bm25_exact])
print("BM25 paraphrase:", [chunk.chunk_id for chunk, _ in bm25_paraphrase])
assert bm25_exact[0][0].chunk_id == "eu-key-rotation-v2-rule"
assert bm25_paraphrase == []
assert bm25_rank("the", permitted) == []
assert bm25_rank(EXACT, []) == []
assert bm25_rank("KROT-14 KROT-14", permitted) == bm25_exact

Output

BM25 exact: ['eu-key-rotation-v2-rule']
BM25 paraphrase: []

BM25 did its job. It recovered the policy from its code and openly failed when the user used no policy vocabulary. A real evaluation set needs both query types; otherwise the lexical lane can look perfect while customers miss evidence.

That failure points to representation, not permissions. The next sparse option changes which terms are indexed, while keeping the same evidence boundary.

BM25 isn't the only sparse option. SPLADE (SParse Lexical AnD Expansion) learns vocabulary weights using a Transformer's masked language modeling (MLM) head. Let $z_{ij}$ be that head's logit for vocabulary item $j$ at token position $i$ . The original SPLADE used sum pooling:^{[4]Reference 4SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking.https://arxiv.org/abs/2107.05720}

w_j=\sum_i \log(1+\operatorname{ReLU}(z_{ij}))

SPLADE v2 introduced max pooling, keeping the strongest activation across positions:^{[5]Reference 5SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalhttps://arxiv.org/abs/2109.10086}

w_j=\max_i \log(1+\operatorname{ReLU}(z_{ij}))

ReLU zeros negative activations, and regularization encourages a sparse representation. Learned positive weights can include vocabulary items absent from the input, giving queries and documents additional ways to overlap. Which expansions help this policy corpus is an empirical question; no SPLADE checkpoint is run here.

These nonnegative weights can support dot-product retrieval through an inverted index that accepts learned term weights and the required scoring function. A BM25-only endpoint does not automatically implement SPLADE scoring. Evaluate its query cost, index size, candidate recall, and permission behavior before replacing the lexical lane.

diagnose-sparse-miss.py

required_text = next(
    chunk.text for chunk in permitted if chunk.chunk_id == "eu-key-rotation-v2-rule"
)
exact_overlap = sorted(set(tokens(EXACT)) & set(tokens(required_text)))
paraphrase_overlap = sorted(set(tokens(PARAPHRASE)) & set(tokens(required_text)))
print("Exact overlap:", exact_overlap)
print("Paraphrase overlap:", paraphrase_overlap)
assert exact_overlap == ["krot-14"]
assert paraphrase_overlap == []

Output

Exact overlap: ['krot-14']
Paraphrase overlap: []

Add a dense semantic lane

A dense retriever encodes queries and chunks as compact vectors, then ranks their matches. The original Dense Passage Retrieval (DPR) used separate BERT-base query and passage encoders, 768-dimensional CLS representations, and dot-product scoring.^{[6]Reference 6Dense Passage Retrieval for Open-Domain Question Answering.https://arxiv.org/abs/2004.04906} Passage vectors were computed offline; online retrieval encoded the query and searched the passage index. Other checkpoints use different dimensions, preprocessing, and scoring conventions. The lab uses cosine similarity as in Core Retrieval Algorithms, over permitted current chunks.

Bi-encoder training aims to make relevant query-passage pairs score above irrelevant pairs. That can recover paraphrases, but it does not guarantee good handling of domain codes or new vocabulary. Nor do all bi-encoders place vectors on a unit sphere: that description applies when vectors are normalized. To diagnose a real KROT-14 miss, inspect the actual checkpoint's tokens and retrieved passages, then compare against the lexical lane on held-out code queries.

The next cell uses hand-authored three-dimensional vectors as test fixtures. No embedding model produced them. Their named coordinates make the geometry easy to inspect; learned embedding coordinates generally don't have these clean human-assigned meanings:

Dimension	Meaning in this fixture
1	Key-rotation intent
2	Session timeout intent
3	Audit-log rebuild intent

The lookup intentionally has no vector for KROT-14. That simulates an unavailable semantic result, not how a real encoder necessarily handles an unfamiliar code. A real encoder normally emits a nonzero vector even for a poorly represented identifier; it may then rank the wrong passages.

The lab ranks those vectors with cosine similarity:

\operatorname{Sim}(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\|_2 \|\mathbf{v}\|_2}

Vectors pointing in a similar direction score closer to 1; their raw length doesn't decide the result.

For the paraphrase vector (0.98, 0.05, 0), the norm is about 0.9813. Its cosine with the key-rotation axis is 0.98 / 0.9813 ≈ 0.9987, while the session axis scores about 0.0510. Both are positive, but only the first is the intended policy. Positivity alone isn't a relevance judgment.

dense-lane.py

Vector = tuple[float, float, float]

DOCUMENT_VECTORS: dict[str, Vector] = {
    "eu-key-rotation-v2-rule": (1.00, 0.00, 0.00),
    "eu-session-timeout-v1-rule": (0.00, 1.00, 0.00),
    "eu-audit-rebuild-v1": (0.00, 0.00, 1.00),
}
QUERY_VECTORS: dict[str, Vector] = {
    PARAPHRASE: (0.98, 0.05, 0.00),
    "stale service-account key rotation within 14 days": (0.96, 0.15, 0.02),
}

def cosine(left: Vector, right: Vector) -> float:
    if not left or len(left) != len(right) or not all(
        finite_number(x) for x in (*left, *right)
    ):
        raise ValueError("vectors must have matching nonempty dimensions and finite numbers")
    left_scale = max(abs(value) for value in left)
    right_scale = max(abs(value) for value in right)
    if left_scale == 0 or right_scale == 0:
        raise ValueError("cosine is undefined for a zero vector")
    # Scaling first avoids overflow from squaring large finite coordinates.
    left_scaled = tuple(value / left_scale for value in left)
    right_scaled = tuple(value / right_scale for value in right)
    left_norm, right_norm = hypot(*left_scaled), hypot(*right_scaled)
    score = fsum(
        (a / left_norm) * (b / right_norm)
        for a, b in zip(left_scaled, right_scaled)
    )
    return max(-1.0, min(1.0, score))  # contain floating-point roundoff

def dense_rank(
    query: str,
    chunks: list[PolicyChunk],
    top_k: int = 2,
) -> list[tuple[PolicyChunk, float]]:
    if type(top_k) is not int or top_k <= 0:
        raise ValueError("top_k must be a positive integer")
    query_vector = QUERY_VECTORS.get(query)
    if query_vector is None:
        return []  # explicit fixture miss, not an encoder inference result
    ranked = [
        (chunk, cosine(query_vector, DOCUMENT_VECTORS[chunk.chunk_id]))
        for chunk in chunks
    ]
    return sorted(
        [(chunk, score) for chunk, score in ranked if score > 0],
        key=lambda item: (-item[1], item[0].chunk_id),
    )[:top_k]

dense_exact = dense_rank(EXACT, permitted)
dense_paraphrase = dense_rank(PARAPHRASE, permitted)
print("Dense exact:", [chunk.chunk_id for chunk, _ in dense_exact])
print("Dense paraphrase:", [chunk.chunk_id for chunk, _ in dense_paraphrase])
assert dense_exact == []
assert dense_paraphrase[0][0].chunk_id == "eu-key-rotation-v2-rule"
assert abs(cosine((0.98, 0.05, 0.0), (1.0, 0.0, 0.0)) - 0.9987009944894547) < 1e-9
assert cosine((1.0, 0.0, 0.0), (-1.0, 0.0, 0.0)) == -1.0
assert dense_rank(PARAPHRASE, []) == []

Output

Dense exact: []
Dense paraphrase: ['eu-key-rotation-v2-rule', 'eu-session-timeout-v1-rule']

This exact cosine scan validates ranking over supplied vectors, not encoder quality or approximate-neighbor recall. The score > 0 cutoff is a toy candidate rule, not a calibrated relevance threshold. A learned-encoder experiment must replace both vector tables, use the checkpoint's query/document preprocessing and scoring convention, and evaluate its actual ranks. The engineered miss here only checks that the sparse fallback survives fusion.

Fuse candidates without mixing score scales

You already fused rank lists with Reciprocal Rank Fusion in Core Retrieval Algorithms. The same reason applies here: BM25 scores and cosine similarities don't share units. A BM25 value reflects term statistics and scorer settings; cosine measures vector alignment between -1 and 1. Adding raw scores can give a lane more influence because its score differences are larger, rather than because its evidence is better.

Why use reciprocals? A rank-sum method gives each adjacent rank change the same cost; it also needs a convention for candidates absent from a list. Reciprocal ranks emphasize early positions: rank 1 contributes 1.0, rank 2 contributes 0.5, and rank 10 contributes 0.1. This is a scoring choice, not a model of the probability that a document is relevant.

With $k=0$ , the contribution drops 50% from rank 1 to rank 2. But one rank-1 vote ties, rather than beats, two rank-2 votes: $1/1 = 1/2+1/2 = 1$ . Smoothing changes how strongly a top position competes with agreement across lanes.

Cormack, Clarke, and Buettcher (2009) used a constant $k$ to reduce the influence of high-ranking outliers:^{[7]Reference 7Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.https://dl.acm.org/doi/10.1145/1571941.1572114}

\operatorname{RRF}(d)=\sum_{l:\,d\in L_l}\frac{1}{k+\operatorname{rank}_l(d)}

Here $L_l$ is lane $l$ 's returned list, ranks start at 1, and $k$ controls the rank penalty slope. We use $k=60$ , which the original paper selected in its pilot experiments. That is a starting setting to evaluate, not a universally optimal value:

Rank 1 contributes $1 / (60 + 1) = 1/61 \approx 0.01639$ .
Rank 2 contributes $1 / (60 + 2) = 1/62 \approx 0.01613$ .

The step from rank 1 to rank 2 shrinks from a 50% cliff down to just a 1.6% step:

\frac{1/61 - 1/62}{1/61} = \frac{1}{62} \approx 1.61\%

This gentle slope allows multi-lane consensus to accumulate. A candidate that ranks consistently near the top of both lanes can outscore a candidate that topped only one lane.

For the shared-language query, eu-key-rotation-v2-rule ranks first in both lanes, earning $1 / 61 + 1 / 61 = 2/61 \approx 0.03279$ . The session-timeout distractor ranks second in both lanes, earning $1 / 62 + 1 / 62 = 2/62 \approx 0.03226$ . These values aren't probabilities. They're rank-based scores used to order the fused candidate set.

rrf-fusion.py

RRF_K = 60

def reciprocal_rank_fusion(
    result_lists: list[list[tuple[PolicyChunk, float]]],
    k: int = RRF_K,
) -> list[tuple[PolicyChunk, float]]:
    if type(k) is not int or not finite_number(k) or k < 0:
        raise ValueError("k must be a finite nonnegative integer")
    by_id: dict[str, PolicyChunk] = {}
    scores: dict[str, float] = {}
    for results in result_lists:
        seen: set[str] = set()
        for rank, (chunk, _raw_score) in enumerate(results, start=1):
            if chunk.chunk_id in seen:
                raise ValueError("a lane must return each chunk ID at most once")
            seen.add(chunk.chunk_id)
            if chunk.chunk_id in by_id and by_id[chunk.chunk_id] != chunk:
                raise ValueError("same chunk ID has conflicting records across lanes")
            by_id[chunk.chunk_id] = chunk
            scores[chunk.chunk_id] = scores.get(chunk.chunk_id, 0.0) + 1 / (k + rank)
    return sorted(
        [(by_id[chunk_id], score) for chunk_id, score in scores.items()],
        key=lambda item: (-item[1], item[0].chunk_id),
    )

def hybrid_rank(
    query: str,
    caller: Caller,
    chunks: list[PolicyChunk],
    top_k: int = 2,
    candidate_pool: int | None = None,
) -> list[tuple[PolicyChunk, float]]:
    if type(top_k) is not int or top_k <= 0 or (
        candidate_pool is not None
        and (type(candidate_pool) is not int or candidate_pool <= 0)
    ):
        raise ValueError("result and candidate depths must be positive integers")
    searchable = permitted_chunks(caller, chunks, EVAL_DATE)
    # Separate the lane budget from final selection; this corpus has only 3 IDs.
    pool = candidate_pool if candidate_pool is not None else max(50, top_k)
    pool = min(pool, len(searchable)) if searchable else top_k
    fused = reciprocal_rank_fusion(
        [bm25_rank(query, searchable, pool), dense_rank(query, searchable, pool)]
    )
    return fused[:top_k]

SHARED_WORDS = "stale service-account key rotation within 14 days"
for query in [EXACT, PARAPHRASE, SHARED_WORDS]:
    hits = hybrid_rank(query, LUNA, CHUNKS)
    print(query, "->", [chunk.chunk_id for chunk, _ in hits])

shared_fused = hybrid_rank(SHARED_WORDS, LUNA, CHUNKS)
assert shared_fused[0][0].chunk_id == "eu-key-rotation-v2-rule"
assert shared_fused[0][1] == 2 / 61
assert reciprocal_rank_fusion([[], []]) == []
assert hybrid_rank("the", LUNA, CHUNKS) == []

Output

KROT-14 -> ['eu-key-rotation-v2-rule']
refresh expired machine credential -> ['eu-key-rotation-v2-rule', 'eu-session-timeout-v1-rule']
stale service-account key rotation within 14 days -> ['eu-key-rotation-v2-rule', 'eu-session-timeout-v1-rule']

RRF doesn't manufacture relevance. It makes the two candidate sources interoperable. If both lanes miss the right chunk, a fused list will still be wrong.

Treat lane depth and final context size as separate controls. If top_k is 2 and each lane only returns 2 hits, deeper evidence never meets in fusion. A larger lane pool can preserve it, at additional search and fusion cost. It can also introduce competing dual-list hits, so increasing depth doesn't guarantee better final Recall@k.

A dual-listed distractor can still outrank a correct one-lane hit after fusion. Before reading the table, compare 2/62 with 1/61 and predict where the correct one-lane hit lands. Gate on Recall@k after fusion alongside per-lane diagnostics:

Chunk	Sparse rank	Dense rank	RRF
key-rotation (correct)	1	absent	$1/61$
session-timeout (distractor)	2	2	$2/62$
audit-rebuild (noise)	absent	1	$1/61$

$2/62 > 1/61$ , so the distractor wins. The correct chunk ties the dense-only noise at $1/61$ and loses the identifier tie-break, falling to third.

fusion-depth-pathology.py

# Toy ranks (not the live BM25/dense indexes): correct evidence is sparse-only
# rank 1; a dual-listed distractor is rank 2 in both lanes.
correct = next(c for c in permitted if c.chunk_id == "eu-key-rotation-v2-rule")
distractor = next(c for c in permitted if c.chunk_id == "eu-session-timeout-v1-rule")
noise = next(c for c in permitted if c.chunk_id == "eu-audit-rebuild-v1")

sparse_only_lists = [
    [(correct, 8.0), (distractor, 3.0)],  # correct r1, distractor r2
    [(noise, 0.9), (distractor, 0.85)],  # dense: noise r1, distractor r2; correct absent
]
fused_full = reciprocal_rank_fusion(sparse_only_lists)
fused_ids = [chunk.chunk_id for chunk, _ in fused_full]
top2_ids = fused_ids[:2]

# Dual-list distractor: 1/(60+2) + 1/(60+2) = 2/62
# Sparse-only correct: 1/(60+1) = 1/61
# Dense-only noise: 1/(60+1) = 1/61
# 2/62 > 1/61, so the distractor wins fusion; correct ties noise and loses the id break.
assert fused_ids[0] == "eu-session-timeout-v1-rule"
assert abs(fused_full[0][1] - 2 / 62) < 1e-12
assert abs(next(s for c, s in fused_full if c.chunk_id == correct.chunk_id) - 1 / 61) < 1e-12

recall_after_fusion_at_1 = int(correct.chunk_id in fused_ids[:1])
recall_after_fusion_at_2 = int(correct.chunk_id in top2_ids)
print("Fused order:", fused_ids)
print("Recall@1 after fusion:", recall_after_fusion_at_1)
print("Recall@2 after fusion:", recall_after_fusion_at_2)
assert recall_after_fusion_at_1 == 0
assert recall_after_fusion_at_2 == 0  # buried past dual-list distractor + dense-only noise
assert correct.chunk_id in fused_ids  # still present deeper in the fused pool

Output

Fused order: ['eu-session-timeout-v1-rule', 'eu-audit-rebuild-v1', 'eu-key-rotation-v2-rule']
Recall@1 after fusion: 0
Recall@2 after fusion: 0

If the generator only packs top_k=2, the correct policy is present in a sparse lane but absent from context. That is a fusion-depth failure, not a permission failure. In this example, both input lists are already complete: increasing their depth cannot change the three supplied votes. Raising the fused budget retains the target; a reranker must receive that fuller pool to have a chance to promote it. Neither choice guarantees the final answer will use it.

One chunk must receive at most one vote per lane. Repeated chunk IDs would otherwise let a retriever manufacture agreement with itself. Check that invariant alongside empty inputs and undefined cosine values:

ranking-boundaries.py

from dataclasses import replace

def rejects(call):
    try:
        call()
    except ValueError:
        return True
    return False

assert rejects(lambda: reciprocal_rank_fusion([[(correct, 2.0), (correct, 1.0)]]))
assert rejects(lambda: reciprocal_rank_fusion([
    [(correct, 2.0)], [(replace(correct, text="conflicting version"), 1.0)],
]))
assert rejects(lambda: cosine((0.0, 0.0, 0.0), (1.0, 0.0, 0.0)))
assert rejects(lambda: cosine((float("nan"), 0.0, 0.0), (1.0, 0.0, 0.0)))
assert rejects(lambda: bm25_rank(EXACT, permitted, k1=-1))
assert rejects(lambda: dense_rank(PARAPHRASE, permitted, top_k=0))
assert rejects(lambda: hybrid_rank(EXACT, LUNA, CHUNKS, candidate_pool=0))
assert rejects(lambda: bm25_rank(EXACT, [correct, correct]))
assert rejects(lambda: dense_rank(PARAPHRASE, permitted, top_k=True))
assert rejects(lambda: hybrid_rank(EXACT, LUNA, CHUNKS, candidate_pool=1.5))
assert rejects(lambda: reciprocal_rank_fusion([[]], k=True))
assert rejects(lambda: bm25_rank(EXACT, permitted, b="0.75"))
assert cosine((1e308, 1e308, 0.0), (1e308, 1e308, 0.0)) > 0.999999999
assert cosine((1e-308, 0.0, 0.0), (1.0, 0.0, 0.0)) == 1.0
unsmoothed = reciprocal_rank_fusion(sparse_only_lists, k=0)
assert all(score == 1.0 for _, score in unsmoothed)
assert bm25_rank(EXACT, [replace(correct, text="")]) == []
print("Duplicate votes, conflicting records, invalid vectors and depths: rejected")
print("All-empty lexical corpus: no candidates")

Output

Duplicate votes, conflicting records, invalid vectors and depths: rejected
All-empty lexical corpus: no candidates

RRF with smoothing constant 60: session-timeout has rank 2 in both lanes and scores 0.03226. Audit-rebuild and the correct key-rotation rule each have one rank-1 vote and score 0.01639. A common zero-based bar axis shows the dual-list distractor nearly twice as high; the identifier tie-break places the correct rule third, eliminating it when top-k is 2. — Session-timeout appears at rank 2 in both lists, so it receives 2/62. The correct key-rotation rule is rank 1 in only one list, so it receives 1/61 and falls behind.

Should you tune weights instead?

RRF is a useful baseline because it doesn't require fitting unrelated score scales. It isn't an automatic winner. Bruch, Gai, and Ingber found a tuned convex combination of normalized scores outperformed RRF in their experiments, including out-of-domain tests.^{[8]Reference 8An Analysis of Fusion Functions for Hybrid Retrieval.https://arxiv.org/abs/2210.11934} That result motivates a comparison on your data, not a universal replacement rule.

Suppose a candidate has normalized dense score 0.8 and sparse score 0.2. Giving dense a weight of 0.7 yields 0.7 × 0.8 + 0.3 × 0.2 = 0.62. In general:

\operatorname{score}_{\text{hybrid}}(d) =\alpha\widetilde{\operatorname{score}}_{\text{dense}}(d) +(1-\alpha)\widetilde{\operatorname{score}}_{\text{sparse}}(d)

The tildes denote scores normalized within their lanes before fusion. Raw linear combination ( $0.7 \times \text{cosine} + 0.3 \times \text{BM25}$ ) is mathematically valid, but $0.7$ doesn't mean 70% of the ranking influence when score scales differ. Influence on ordering depends on weighted score differences between candidates. At $\alpha=1$ , BM25 contributes nothing; at other weights either lane can decide a comparison. There is no fixed percentage of ordering controlled by BM25.

Min-max scaling maps observed scores into $[0, 1]$ :

\widetilde{s}_i = \frac{s_i - s_{\min}}{s_{\max} - s_{\min}}

While simple, min-max scaling introduces three subtle production challenges:

Outlier distortion: A very large maximum relative to the other scores can compress their normalized differences.
Pool-depth instability: A deeper pool may change $s_{\min}$ or $s_{\max}$ , changing that lane's influence on the mix. Depth changes do not necessarily change either endpoint or the final order.
Zero spread: If every score is equal, the formula divides by zero. Our code explicitly assigns zeros to a constant-score lane; that is a chosen fallback, not the result of the formula.

A sigmoid transform ( $\sigma((s-\mu)/\tau)$ , $\tau>0$ ) bounds scores; z-score standardization does not. Both introduce choices about location and scale, whose estimation and stability need evaluation. RRF avoids fitting raw score scales and its formula needs no relevance labels, but candidate depth, $k$ , and final selection still need testing.

The next cell shows BM25 contributing more despite its smaller coefficient, then min-max normalizes each lane before the same mix. Normalization rescales observed values; it doesn't calibrate them into relevance probabilities.

Predict the raw outcome first: even a 0.3 BM25 weight can dominate a 0.7 cosine weight when BM25 is several times larger. Normalization changes the units, not the evidence boundary.

lane-normalized-weighted-fusion.py

def min_max(scores: dict[str, float]) -> dict[str, float]:
    if not scores:
        return {}
    values = list(scores.values())
    if not all(finite_number(value) for value in values):
        raise ValueError("scores must be finite numbers")
    low, high = min(values), max(values)
    if high == low:
        return {key: 0.0 for key in scores}  # no within-lane discrimination
    # Avoid overflow when high - low spans large opposite-signed scores.
    scale = max(abs(value) for value in values)
    low_scaled, high_scaled = low / scale, high / scale
    spread = high_scaled - low_scaled
    return {key: (value / scale - low_scaled) / spread for key, value in scores.items()}

def score_map(
    ranked: list[tuple[PolicyChunk, float]],
    chunk_ids: list[str],
) -> dict[str, float]:
    by_id = {chunk.chunk_id: score for chunk, score in ranked}
    return {chunk_id: by_id.get(chunk_id, 0.0) for chunk_id in chunk_ids}

lane_ids = [chunk.chunk_id for chunk in permitted]
bm25_raw = score_map(bm25_rank(SHARED_WORDS, permitted, top_k=3), lane_ids)
dense_raw = score_map(dense_rank(SHARED_WORDS, permitted, top_k=3), lane_ids)
alpha = 0.7
krot = "eu-key-rotation-v2-rule"

raw_weighted = {
    chunk_id: alpha * dense_raw[chunk_id] + (1 - alpha) * bm25_raw[chunk_id]
    for chunk_id in bm25_raw
}
# Before normalization, the BM25 term is still larger than the dense term.
assert (1 - alpha) * bm25_raw[krot] > alpha * dense_raw[krot]

bm25_n = min_max(bm25_raw)
dense_n = min_max(dense_raw)
normalized_weighted = {
    chunk_id: alpha * dense_n[chunk_id] + (1 - alpha) * bm25_n[chunk_id]
    for chunk_id in bm25_raw
}
weighted_order = sorted(
    normalized_weighted,
    key=lambda chunk_id: (-normalized_weighted[chunk_id], chunk_id),
)
rrf_order = [
    chunk.chunk_id
    for chunk, _ in reciprocal_rank_fusion(
        [
            bm25_rank(SHARED_WORDS, permitted, top_k=3),
            dense_rank(SHARED_WORDS, permitted, top_k=3),
        ]
    )
]
print("Raw BM25:", {k: round(v, 3) for k, v in bm25_raw.items()})
print("Raw dense:", {k: round(v, 3) for k, v in dense_raw.items()})
print("Raw weighted (unadjusted scales):", {k: round(v, 3) for k, v in raw_weighted.items()})
print("Lane-normalized weighted order:", weighted_order)
print("RRF order:", rrf_order)
assert weighted_order[0] == krot
assert rrf_order[0] == krot
assert min_max({}) == {}
assert min_max({"a": 0.0, "b": 0.0}) == {"a": 0.0, "b": 0.0}
assert min_max({"a": 7.0, "b": 7.0}) == {"a": 0.0, "b": 0.0}
assert min_max({"a": -2.0, "b": 2.0}) == {"a": 0.0, "b": 1.0}
assert min_max({"a": -1e308, "b": 0.0, "c": 1e308}) == {
    "a": 0.0, "b": 0.5, "c": 1.0,
}

Output

Raw BM25: {'eu-key-rotation-v2-rule': 3.788, 'eu-session-timeout-v1-rule': 0.579, 'eu-audit-rebuild-v1': 0.0}
Raw dense: {'eu-key-rotation-v2-rule': 0.988, 'eu-session-timeout-v1-rule': 0.154, 'eu-audit-rebuild-v1': 0.021}
Raw weighted (unadjusted scales): {'eu-key-rotation-v2-rule': 1.828, 'eu-session-timeout-v1-rule': 0.282, 'eu-audit-rebuild-v1': 0.014}
Lane-normalized weighted order: ['eu-key-rotation-v2-rule', 'eu-session-timeout-v1-rule', 'eu-audit-rebuild-v1']
RRF order: ['eu-key-rotation-v2-rule', 'eu-session-timeout-v1-rule', 'eu-audit-rebuild-v1']

Both methods agree on this toy query, so it demonstrates no ranking improvement from normalization. Zero-filling missing scores is also a convention, not a fact about an unseen candidate. Here every permitted chunk is scored and omitted sparse hits truly have zero overlap. With truncated lists, an absent candidate's score is unknown; rescore the union or explicitly evaluate your missing-score rule. Min-max values can change when candidate depth changes, and a constant-score lane contributes zero in this implementation.

Weighted fusion still only ranks IDs that already passed the filter. The next test makes that boundary fail loudly.

Attack the evidence boundary through fusion

The stored admin-only policy contains a unique code. If the permission boundary moved after retrieval, BM25 would have an easy hidden hit to surface. The mandatory invariant is that Luna never receives the hidden record. This fixture also returns an empty list because no permitted text matches the code and no query vector is supplied. A real dense encoder could return permitted but irrelevant candidates; an empty result isn't the only safe outcome.

Predict the safe output before the attack test runs: an empty result for Luna is success, not a retrieval miss. The hidden code must never enter either lane.

hidden-source-attack.py

NO_ACCESS = Caller("visitor-9000", "APAC", frozenset())
attack_hits = hybrid_rank("ADMIN-KROT-1", LUNA, CHUNKS)
attack_ids = [chunk.chunk_id for chunk, _ in attack_hits]
no_access_hits = hybrid_rank(EXACT, NO_ACCESS, CHUNKS)
print("Visible candidates for hidden code:", attack_ids)
print("Visible candidates without corpus access:", no_access_hits)
assert "restricted-admin-key-rotation" not in attack_ids
assert attack_ids == []
assert no_access_hits == []

Output

Visible candidates for hidden code: []
Visible candidates without corpus access: []

Gate the retriever on recall and safety

In the previous lesson, answer quality depended on retrieving current permitted evidence. That means the retrieval upgrade needs its own release cases before you measure generated text.

Each query here has exactly one labeled relevant chunk. Recovering it among the first two candidates scores 1; missing it scores 0. Averaging those values is both HitRate@2 and mean Recall@2 for this special case. With several relevant chunks, per-query recall is the fraction recovered, while hit rate only asks whether any was found. Neither metric proves the final answer is faithful.

Before reading the matrix, predict its shape. BM25 should miss the paraphrase, dense should miss KROT-14, and hybrid should cover all three frozen positives while keeping the two excluded IDs out.

Three engineered query regressions using hand-authored vectors: BM25 and dense each recover the required chunk twice; hybrid recovers it three times. Separate checks exclude the restricted admin rule and superseded revision. These counts are not an encoder benchmark or a corpus-wide leakage measurement. — The vectors and queries were constructed to expose complementary failures. Hybrid gets three hits in this fixture, not a measured production advantage. The code checks unauthorized and superseded IDs separately.

retrieval-release-gate.py

@dataclass(frozen=True)
class RetrievalCase:
    name: str
    query: str
    expected_chunk_id: str

CASES = [
    RetrievalCase("exact-code", EXACT, "eu-key-rotation-v2-rule"),
    RetrievalCase("paraphrase", PARAPHRASE, "eu-key-rotation-v2-rule"),
    RetrievalCase("shared-language", SHARED_WORDS, "eu-key-rotation-v2-rule"),
]

def recall_at_2(rank_fn) -> float:
    recovered = 0
    for case in CASES:
        ids = [chunk.chunk_id for chunk, _ in rank_fn(case.query)]
        recovered += case.expected_chunk_id in ids[:2]
    return recovered / len(CASES)

bm25_recall = recall_at_2(lambda query: bm25_rank(query, permitted))
dense_recall = recall_at_2(lambda query: dense_rank(query, permitted))
hybrid_recall = recall_at_2(lambda query: hybrid_rank(query, LUNA, CHUNKS))

restricted_attack = hybrid_rank("ADMIN-KROT-1", LUNA, CHUNKS)
superseded_attack = hybrid_rank(
    "KROT-14 key rotation manual 30 days",
    LUNA,
    CHUNKS,
)
restricted_ids = [chunk.chunk_id for chunk, _ in restricted_attack]
superseded_ids = [chunk.chunk_id for chunk, _ in superseded_attack]
safety_pass = (
    "restricted-admin-key-rotation" not in restricted_ids
    and "eu-key-rotation-v1-rule" not in superseded_ids
)

print(f"BM25 Recall@2: {bm25_recall:.2f}")
print(f"Dense Recall@2: {dense_recall:.2f}")
print(f"Hybrid RRF Recall@2: {hybrid_recall:.2f}")
print("Restricted attack ids:", restricted_ids)
print("Superseded attack ids:", superseded_ids)
print("Safety gate:", safety_pass)
assert hybrid_recall == 1.0
assert hybrid_recall > bm25_recall
assert hybrid_recall > dense_recall
assert restricted_ids == []
assert "eu-key-rotation-v1-rule" not in superseded_ids
assert safety_pass

Output

BM25 Recall@2: 0.67
Dense Recall@2: 0.67
Hybrid RRF Recall@2: 1.00
Restricted attack ids: []
Superseded attack ids: ['eu-key-rotation-v2-rule', 'eu-session-timeout-v1-rule']
Safety gate: True

These three fixtures demonstrate complementary failures; they don't prove an offline lift for a production corpus. A release decision needs a held-out set drawn from real support requests, including exact codes, paraphrases, unsupported questions, languages served by the product, and attempts to request hidden policies.

The same evaluation must retain workload and system context: corpus and index versions, encoder and tokenizer, lane depths, fusion settings, latency percentiles, and the correctness or support check used for answers. Recall numbers without those details are hard to reproduce or compare.

Gate	What to freeze	Failure meaning
Permitted Recall@k	Query and required current chunk ID	Correct evidence never reaches context selection
Restricted-source exclusion	Queries that strongly match hidden chunks	Retriever boundary is unsafe
Superseded-source exclusion	Queries matching old policy wording	Freshness filter regressed
Abstention cases	Questions with no permitted supporting evidence	Retrieval or answer layer overreaches

Trace each lane before adding reranking

When a final answer is wrong, you need to tell apart three failures:

Failure	Trace evidence	Next repair
Retrieval miss	Expected chunk absent from sparse, dense, and fused candidates	Improve indexing, encoder, query handling, or lane depth
Fusion ordering issue	Expected chunk exists in a lane but falls below context budget	Tune fusion on held-out labels
Later precision issue	Correct chunk is in fused candidates but distractors rank above it	Add and evaluate the reranking stage in the next lesson

Store IDs, ranks, model and index versions, fusion settings, effective date, grant revision, and timing in an access-controlled trace. IDs and caller metadata can themselves reveal sensitive information; omitting full policy text isn't an anonymization guarantee. The trace must use the same lane depths and fusion settings as the retrieval path being diagnosed. Record the fuller fused pool separately from the final shortlist to expose selection losses.

hybrid-retrieval-trace.py

def trace_hybrid_request(
    query: str,
    query_kind: str,
    caller: Caller,
) -> dict[str, object]:
    searchable = permitted_chunks(caller, CHUNKS, EVAL_DATE)
    pool = max(1, min(50, len(searchable)))
    sparse = bm25_rank(query, searchable, pool)
    dense = dense_rank(query, searchable, pool)
    fused = reciprocal_rank_fusion([sparse, dense])
    return {
        "versions": {
            "retriever": "policy-retriever-v2",
            "index": "policy-index/2026-05-27",
            "sparse": "bm25-tokenizer-v1",
            "dense": "fixture-embeddings-v1",
            "fusion": f"rrf-k{RRF_K}",
        },
        "caller_fixture": {
            "actor_id": caller.actor_id,
            "region": caller.region,
            "acl_tags": sorted(caller.acl_tags),
            "grant_revision": "fixture-grants-v1",
        },
        "effective_date": EVAL_DATE.isoformat(),
        "query_kind": query_kind,
        "candidate_pool": pool,
        "final_top_k": 2,
        "sparse_ids": [chunk.chunk_id for chunk, _ in sparse],
        "dense_ids": [chunk.chunk_id for chunk, _ in dense],
        "fused_pool_ids": [chunk.chunk_id for chunk, _ in fused],
        "fused_ids": [chunk.chunk_id for chunk, _ in fused[:2]],
        # Simulated timings exercise the checker below; they aren't measurements.
        "timings_ms_fixture": {
            "authorize": 2, "bm25": 4, "dense": 11, "fusion": 1, "retrieval_total": 18,
        },
    }

trace = trace_hybrid_request(PARAPHRASE, "paraphrase-regression", LUNA)
stores_raw_policy_text = any(
    chunk.text in str(trace)
    for chunk in CHUNKS
)
print("Versions:", trace["versions"])
print("Sparse ids:", trace["sparse_ids"])
print("Dense ids:", trace["dense_ids"])
print("Full fused pool ids:", trace["fused_pool_ids"])
print("Fused ids:", trace["fused_ids"])
print("Trace stores raw policy text:", stores_raw_policy_text)
assert trace["fused_ids"][0] == "eu-key-rotation-v2-rule"
assert "eu-session-timeout-v1-rule" in trace["fused_ids"]
assert "restricted-admin-key-rotation" not in str(trace)
assert not stores_raw_policy_text

Output

Versions: {'retriever': 'policy-retriever-v2', 'index': 'policy-index/2026-05-27', 'sparse': 'bm25-tokenizer-v1', 'dense': 'fixture-embeddings-v1', 'fusion': 'rrf-k60'}
Sparse ids: []
Dense ids: ['eu-key-rotation-v2-rule', 'eu-session-timeout-v1-rule']
Full fused pool ids: ['eu-key-rotation-v2-rule', 'eu-session-timeout-v1-rule']
Fused ids: ['eu-key-rotation-v2-rule', 'eu-session-timeout-v1-rule']
Trace stores raw policy text: False

The correct evidence is present, but the semantic lane also kept a session-timeout distractor. Retrieval satisfies this case's candidate recall; a reranker can help order or reject distractors, subject to its own evaluation. The trace's ordered ID lists encode ranks. Its grant revision and durations are fixture labels, not records from an identity provider or timers. The full-text substring check misses fragments and other sensitive values; it is not a privacy audit. Use controlled query-kind values, rather than putting arbitrary user text in that field.

Diversify candidates with maximal marginal relevance

Hybrid fusion can pack near-duplicate chunks into the shortlist. For a question requiring several policies, redundant excerpts may crowd out other useful evidence. For Luna's narrow key-rotation question, session-timeout evidence is irrelevant: filling a slot with a different topic is not automatically an improvement.

Maximal Marginal Relevance (MMR) explicitly balances candidate relevance against intra-result redundancy:^{[9]Reference 9The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries.https://dl.acm.org/doi/10.1145/290941.291025}

\operatorname{MMR}(q, D, S) = \operatorname{argmax}_{d_i \in D \setminus S} \left[ \lambda \operatorname{Sim}_1(d_i, q) - (1-\lambda) \max_{d_j \in S} \operatorname{Sim}_2(d_i, d_j) \right]

Here:

$D$ is the candidate pool from fusion.
$S$ is the set of chunks already selected into context.
$D \setminus S$ represents the remaining unselected candidates.
$\operatorname{Sim}_1(d_i, q)$ measures query-document relevance.
$\operatorname{Sim}_2(d_i, d_j)$ measures document-document redundancy (such as vector cosine similarity between candidate $d_i$ and an already selected chunk $d_j$ ).
$\lambda \in [0, 1]$ controls the diversity trade-off. At $\lambda=1$ , the redundancy penalty disappears. At $\lambda=0$ , only redundancy with selected evidence affects later choices. With no selected evidence yet, this implementation sets that penalty to zero, so $\lambda=0$ starts with an ID tie-break. Choose $\lambda$ against the task's required evidence; there is no universally good interval.

Predict the second selection below. The synthetic extra chunk is nearly collinear with the key-rotation vector, while the session vector is orthogonal. This geometry tests MMR's objective; all candidate IDs are assumed admissible, and no real chunks or encoder produce this extended vector table.

maximal-marginal-relevance.py

def maximal_marginal_relevance(
    query_vector: Vector,
    candidate_vectors: dict[str, Vector],
    top_k: int = 2,
    lambda_param: float = 0.5,
) -> list[str]:
    if type(top_k) is not int or top_k <= 0:
        raise ValueError("top_k must be a positive integer")
    if not finite_number(lambda_param) or not 0.0 <= lambda_param <= 1.0:
        raise ValueError("lambda_param must be a finite number between 0.0 and 1.0")
    if not candidate_vectors:
        return []

    selected: list[str] = []
    unselected = set(candidate_vectors.keys())

    while unselected and len(selected) < top_k:
        best_candidate: str | None = None
        best_score = float("-inf")

        for candidate_id in sorted(unselected):
            cand_vec = candidate_vectors[candidate_id]
            relevance = cosine(cand_vec, query_vector)
            redundancy = (
                max(cosine(cand_vec, candidate_vectors[sel_id]) for sel_id in selected)
                if selected
                else 0.0
            )
            mmr_score = lambda_param * relevance - (1.0 - lambda_param) * redundancy
            if mmr_score > best_score:
                best_score = mmr_score
                best_candidate = candidate_id

        if best_candidate is None:
            break
        selected.append(best_candidate)
        unselected.remove(best_candidate)

    return selected

# Add a synthetic near-duplicate vector (cosine with the rule is about 0.9998).
MMR_VECTORS: dict[str, Vector] = {
    "eu-key-rotation-v2-rule": (1.00, 0.00, 0.00),
    "eu-key-rotation-v2-duplicate": (0.99, 0.02, 0.00),
    "eu-session-timeout-v1-rule": (0.00, 1.00, 0.00),
}
query_vec = QUERY_VECTORS[PARAPHRASE]

greedy_top2 = maximal_marginal_relevance(
    query_vec, MMR_VECTORS, top_k=2, lambda_param=1.0
)
diverse_top2 = maximal_marginal_relevance(
    query_vec, MMR_VECTORS, top_k=2, lambda_param=0.5
)

print("Greedy top-2 (lambda=1.0):", greedy_top2)
print("Diverse MMR top-2 (lambda=0.5):", diverse_top2)
assert greedy_top2 == ["eu-key-rotation-v2-duplicate", "eu-key-rotation-v2-rule"]
assert diverse_top2 == ["eu-key-rotation-v2-duplicate", "eu-session-timeout-v1-rule"]
assert maximal_marginal_relevance(query_vec, {}, top_k=2) == []

Output

Greedy top-2 (lambda=1.0): ['eu-key-rotation-v2-duplicate', 'eu-key-rotation-v2-rule']
Diverse MMR top-2 (lambda=0.5): ['eu-key-rotation-v2-duplicate', 'eu-session-timeout-v1-rule']

Greedy selection keeps two near-duplicates. MMR instead chooses the session-timeout rule, which is less redundant and irrelevant to this question. The example demonstrates a diversity trade-off, not better answer quality. For this narrow request, deduplicate by source or verified equivalent content and leave a slot unused if no other evidence is needed. For a multi-policy request, evaluate whether MMR preserves every required rule and condition.

MMR handles diversity inside an existing permitted candidate set. It can't retrieve missing evidence that was omitted during initial retrieval, can't authorize a hidden source, and doesn't replace a cross-encoder that evaluates subtle query-document relevance.

Enforce retrieval latency budgets

Two retrieval lanes add work, so evaluate latency alongside recall. This code calls BM25 and dense ranking serially; a deployed service might run them concurrently if its infrastructure supports that. Measure wall-clock latency over representative queries, concurrency levels, cache states, and filtering selectivity. Parallel lane time is not their simple sum, and adding per-stage p95 values does not produce end-to-end p95.

The next check uses simulated durations and locally chosen stage budgets, plus a 50 ms overall retrieval budget. Stage limits alone allow an overall miss: 10 ms authorization, parallel lanes of 12 and 40 ms, and 8 ms fusion already take 58 ms before any additional overhead. The checker validates supplied telemetry retrospectively. A live endpoint also needs installed deadlines and cancellation behavior; this function does not stop slow work.

retrieval-latency-gate.py

RETRIEVAL_BUDGET_MS = {
    "authorize": 10, "bm25": 12, "dense": 40, "fusion": 8, "retrieval_total": 50,
}

def exceeded_retrieval_budgets(timings: dict[str, float]) -> list[str]:
    return [
        stage
        for stage, budget in RETRIEVAL_BUDGET_MS.items()
        if stage not in timings or not finite_number(timings[stage])
        or timings[stage] < 0 or timings[stage] > budget
    ]

healthy = trace["timings_ms_fixture"]
missing_dense = {
    stage: elapsed
    for stage, elapsed in healthy.items()
    if stage != "dense"
}
print("Healthy exceeded:", exceeded_retrieval_budgets(healthy))
print("Missing timing exceeded:", exceeded_retrieval_budgets(missing_dense))
all_stages_pass = {
    "authorize": 10, "bm25": 12, "dense": 40, "fusion": 8, "retrieval_total": 58,
}
print("Stage limits met, total missed:", exceeded_retrieval_budgets(all_stages_pass))
assert exceeded_retrieval_budgets(healthy) == []
assert exceeded_retrieval_budgets(missing_dense) == ["dense"]
assert exceeded_retrieval_budgets({**healthy, "dense": 40}) == []
assert exceeded_retrieval_budgets({**healthy, "dense": 41}) == ["dense"]
assert exceeded_retrieval_budgets({**healthy, "dense": float("nan")}) == ["dense"]
assert exceeded_retrieval_budgets({**healthy, "dense": -1}) == ["dense"]
assert exceeded_retrieval_budgets({**healthy, "dense": True}) == ["dense"]
assert exceeded_retrieval_budgets({**healthy, "dense": "11"}) == ["dense"]
assert exceeded_retrieval_budgets({**healthy, "retrieval_total": 50}) == []
assert exceeded_retrieval_budgets(all_stages_pass) == ["retrieval_total"]

Output

Healthy exceeded: []
Missing timing exceeded: ['dense']
Stage limits met, total missed: ['retrieval_total']

Next Step

Continue to Reranking and Cross-Encoders for RAG

Hybrid retrieval recovers `KROT-14` and the credential paraphrase in this fixture, but the trace still keeps `eu-session-timeout-v1-rule` beside the right policy. Next you'll evaluate a precision stage that reorders the shortlist before context construction.

PreviousProduction RAG Pipelines

Share this article