Hybrid Search: Dense + Sparse

Production RAG Pipelines gave policy-answerer-v1 a hard RAG rule: only permitted, current evidence may reach an answer. Its retrieve() function filters first, then counts overlapping tokens. That recovered a question from Luna, an EU support specialist, about stale service-account keys. It missed the paraphrase that chapter already named: "refresh an expired machine credential" when the policy says "stale service-account key rotation."

Pause on that miss. The exact code carries a rare lexical signal; the paraphrase carries meaning without the policy's words. Before naming a search method, predict which signal each lane can recover and which lane will fail.

You'll upgrade only that retrieval lane. Hybrid search runs sparse BM25 and dense semantic search over the same permitted corpus, then Reciprocal Rank Fusion (RRF) merges candidate IDs. You already ranked passages that way in Core Retrieval Algorithms. Here those two lanes sit behind the evidence boundary, with diagnostic code KROT-14 so an exact identifier has an unambiguous expected hit. That retrieval path is policy-answerer-v2. Authorization, freshness, citation, and abstention don't change.

One retriever can't cover both queries

Luna needs the same policy for two different searches:

A word-matching index has a decisive clue for the first query and no shared vocabulary for the second. A semantic encoder can represent the second query near the policy, but an unfamiliar internal identifier may carry little useful semantic signal. Neither failure says one method is bad. They solve different recall problems.

Both lanes have to search the same permitted corpus. Fusion improves recall; it doesn't widen access. Keep that ordering in view as the diagram and code add ranking detail: authorization decides what can be searched, while fusion decides how permitted candidates are ordered.

The filter is the evidence boundary from the previous lesson. Sparse and dense only see IDs that already passed it.

Why must the permission filter run before both retrieval lanes rather than after fusion?

Answer

A hidden chunk can be highly ranked by either lane. If text or identifiers from that chunk reach fusion, model context, caches, or request logs before filtering, the service has already crossed its evidence boundary.

Recreate the permitted candidate universe

The lab reuses the policy shape from the previous lesson. It adds diagnostic policy code KROT-14 so an exact-identifier query has an unambiguous expected result. Two tempting records remain in storage but must not be searchable for Luna: a superseded revision and a restricted admin-only rule.

Before running the filter, predict the searchable set. The current EU support rule should survive; the old revision should fail freshness, and the admin rule should fail authorization. That set becomes input to both lanes.

Every ranker below receives permitted, not CHUNKS. That API boundary is what stops a new ranking algorithm from widening Luna's access.

The fixed EVAL_DATE keeps replay behavior stable. The chunk shape also preserves document_id and parent_id from the previous lesson, even though ranking changes while citation packing stays the same.

The next check makes the negative cases visible. Look for two IDs that remain stored but never become searchable.

This test keeps a deliberately attractive hidden policy in storage. Later ranking changes fail loudly if they accidentally widen the searchable set.

Build the sparse lane with BM25

Sparse retrieval represents a document by vocabulary terms. Most coordinates are zero because a short policy chunk uses only a small part of the vocabulary. You already scored that representation with BM25 in Core Retrieval Algorithms: a chunk ranks higher when it shares rare query terms, with a cap on repeated terms and a penalty for unusually long documents.^{[1]Reference 1The Probabilistic Relevance Framework: BM25 and Beyond.https://doi.org/10.1561/1500000019}

The lab below uses the common Lucene-style IDF form with a leading 1 +, which is the practical formula in many search engines rather than the classic Okapi IDF alone. The previous RAG tokenizer split on non-letters, so KROT-14 became krot and 14. This analyzer keeps hyphenated rule codes as one token.

For a query term $t$ and document $d$, the lab computes:

$$\operatorname{BM25}(q,d)=\sum_{t \in q}\operatorname{IDF}(t) \frac{f(t,d)(k_1+1)} {f(t,d)+k_1(1-b+b\lvert d\rvert/\operatorname{avgdl})}$$

Here, $f(t,d)$ is the term count in the chunk, $\lvert d\rvert$ is chunk length in tokens, and avgdl is the corpus average. k1 controls term-frequency saturation; b controls length normalization. The exact identifier krot-14 occurs only in the relevant current chunk, so it receives strong lexical weight.

The analyzer also drops common function words. Without that stopword rule, a query containing only a shared word such as "a" could appear to retrieve an unrelated policy.

Before running the lane, predict the two outputs. KROT-14 should produce a strong hit because its token is rare; the paraphrase should return nothing because none of its words overlap the policy.

BM25 did its job. It recovered the policy from its code and openly failed when the user used no policy vocabulary. A real evaluation set needs both query types; otherwise the lexical lane can look perfect while customers miss evidence.

That failure points to representation, not permissions. The next sparse option changes which terms are indexed, while keeping the same evidence boundary.

BM25 isn't the only sparse option. SPLADE learns sparse expansion weights, so a chunk can gain indexable related terms while preserving sparse retrieval infrastructure.^{[2]Reference 2SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking.https://arxiv.org/abs/2107.05720} That can improve vocabulary mismatch cases, but it doesn't turn sparse retrieval into an authorization layer or guarantee better recall on security-policy queries. Evaluate a SPLADE candidate against the same permitted corpus, held-out required-evidence IDs, latency budget, and hidden-source exclusions before replacing BM25.

If BM25 returns no result for "refresh expired machine credential," should you lower an authorization or freshness filter?

Answer

No. The evidence boundary is not a recall knob. Add a semantic retrieval lane or improve the searchable representation, then test the change inside the same permission and version constraints.

Add a dense semantic lane

A dense retriever encodes queries and chunks as compact vectors, then retrieves chunks with high similarity. Dense Passage Retrieval (DPR), for example, uses separate encoders for questions and passages so passage representations can be indexed before requests arrive.^{[3]Reference 3Dense Passage Retrieval for Open-Domain Question Answering.https://arxiv.org/abs/2004.04906} You used the same cosine-ranking idea in Core Retrieval Algorithms. The new constraint is that those vectors are only allowed for permitted, current chunks.

Downloading and training an encoder would hide the retrieval mechanics in this lab. Instead, the next cell uses frozen three-dimensional vectors as test fixtures. Read them as outputs already produced by an embedding model:

This fixture is deliberately honest about one failure: the internal code KROT-14 has no semantic vector by itself. The paraphrase does.

The lab ranks those vectors with cosine similarity. Vectors pointing in a similar direction score closer to 1; their raw length doesn't decide the result.

Predict the dense lane before inspecting its output: the zero vector for KROT-14 should produce no hit, while the paraphrase should point most strongly at the key-rotation vector.

The fixture doesn't claim that every production encoder misses every identifier. It establishes a regression case: this chosen encoder representation doesn't recover the code-only query, so deleting the sparse lane would fail a known requirement.

Fuse candidates without mixing score scales

You already fused independent rank lists with Reciprocal Rank Fusion in Core Retrieval Algorithms. The same reason applies here: BM25 scores and cosine similarities don't share units. A BM25 value reflects term statistics in this index; a cosine value reflects vector alignment. Adding raw values can let whichever scale is numerically larger control the order.

RRF avoids that comparison. It contributes $1 / (k + r)$ for each rank $r$ at which a chunk appears:

$$\operatorname{RRF}(d)=\sum_{\text{lane } l}\frac{1}{k+\operatorname{rank}_l(d)}$$

Cormack, Clarke, and Buettcher fixed k=60 in a pilot and found it near-optimal, though the choice wasn't critical in that region.^{[4]Reference 4Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.https://dl.acm.org/doi/10.1145/1571941.1572114} Treat it as a starting value, not a universal optimum. A chunk found by both lanes gains two contributions; a strong result found by one lane remains eligible.

For the shared-language query, eu-key-rotation-v2-rule ranks first in both lanes, so its fused score is 1 / 61 + 1 / 61 = 0.0328. The session-timeout distractor ranks second in both lanes, so its score is 1 / 62 + 1 / 62 = 0.0323. These values aren't probabilities. They're rank-based scores used to order the fused candidate set.

Now predict the failure case before running fusion code. A document that appears second in both lanes earns 2/62; a document that appears first in only one lane earns 1/61. The repeated mid-ranked hit can win, so the final context budget must be tested after fusion.

RRF doesn't manufacture relevance. It makes the two candidate sources interoperable. If both lanes miss the right chunk, a fused list will still be wrong.

Don't set each lane's depth equal to the final context budget. If top_k is 2 and each lane only returns 2 hits, mid-ranked evidence never meets in fusion. Retrieve a deeper candidate pool per lane, fuse, then keep top_k.

A dual-listed distractor can still outrank a correct one-lane hit after fusion. Before reading the table, compare 2/62 with 1/61 and predict where the correct one-lane hit lands. Gate on Recall@k after fusion alongside per-lane diagnostics:

$2/62 > 1/61$, so the distractor wins. The correct chunk ties the dense-only noise at $1/61$ and loses the identifier tie-break, falling to third.

If the generator only packs top_k=2, the correct policy is present in a sparse lane but absent from context. That is a fusion-depth failure, not a permission failure. Gate on Recall@k after fusion, raise fused top_k, deepen the weak lane, or pass the fuller fused candidate set to a later precision stage before claiming retrieval succeeded.

Why shouldn't you add a BM25 score directly to a cosine-similarity score?

Answer

They don't share units or a common numeric scale. A large BM25 value can dominate a cosine value even when that isn't the ranking behavior you want. RRF combines rank positions instead; normalized weighted fusion is another option when you tune it on labeled queries.

Should you tune weights instead?

RRF is a good first implementation because it doesn't require calibrating unrelated score scales. It isn't an automatic winner. Bruch, Gai, and Ingber found RRF sensitive to its parameters, and that a tuned convex combination of normalized lane scores outperformed it in their in-domain and out-of-domain tests. That comparison was sample-efficient: a small labeled query set was enough to tune the mix.^{[5]Reference 5An Analysis of Fusion Functions for Hybrid Retrieval.https://arxiv.org/abs/2210.11934}

$$\operatorname{score}_{\text{hybrid}}(d) =\alpha\widetilde{\operatorname{score}}_{\text{dense}}(d) +(1-\alpha)\widetilde{\operatorname{score}}_{\text{sparse}}(d)$$

The tildes denote scores normalized within their lanes before fusion. That comparison is an evaluation task, not a reason to guess an alpha in production. Keep a fixed held-out split, version the encoder and index, report Recall@k and latency for every candidate, and retain RRF if a tuned method doesn't hold up out of sample.

Raw 0.7 * cosine + 0.3 * BM25 is invalid without lane normalization. The next cell shows why magnitudes dominate weights, then min-max normalizes each lane before the same convex mix:

Predict the raw outcome first: even a 0.3 BM25 weight can dominate a 0.7 cosine weight when BM25 is several times larger. Normalization changes the units, not the evidence boundary.

Both calibrated methods agree on this toy query. Agreement on one toy query doesn't license an untuned alpha; it only demonstrates that lane scales must be made comparable before weighted fusion becomes a candidate against RRF.

Weighted fusion still only ranks IDs that already passed the filter. The next test makes that boundary fail loudly.

Attack the evidence boundary through fusion

The stored admin-only policy contains a unique code. If the permission boundary moved after retrieval, BM25 would have an easy hidden hit to surface. A hybrid implementation must return nothing for Luna's request for that code.

Predict the safe output before the attack test runs: an empty result for Luna is success, not a retrieval miss. The hidden code must never enter either lane.

Gate the retriever on recall and safety

In the previous lesson, answer quality depended on retrieving current permitted evidence. That means the retrieval upgrade needs its own release cases before you measure generated text.

Recall@2 answers a narrow question: for each supported query, did the correct permitted chunk appear in the first two candidates? It doesn't say whether the evidence order is perfect or whether the final answer is faithful. Those are later checks. Here, recall exposes whether the generator even gets a chance to see the right policy.

Before reading the matrix, predict its shape. BM25 should miss the paraphrase, dense should miss KROT-14, and hybrid should cover all three frozen positives while keeping the two excluded IDs out.

These three fixtures demonstrate complementary failures; they don't prove an offline lift for a production corpus. A release decision needs a held-out set drawn from real support requests, including exact codes, paraphrases, unsupported questions, languages served by the product, and attempts to request hidden policies.

The same evaluation must retain workload and system context: corpus and index versions, encoder and tokenizer, lane depths, fusion settings, latency percentiles, and the correctness or support check used for answers. Recall numbers without those details are hard to reproduce or compare.

Hybrid reaches Recall@2 of 1.00 on these three labeled queries. Can you say hybrid search is better for all security-policy queries?

Answer

No. You can say it passes the three frozen cases, including two complementary single-lane failures, while preserving the safety boundary. Broader claims require a representative held-out query set and latency measurements.

Trace each lane before adding reranking

When a final answer is wrong, you need to tell apart three failures:

Store IDs, ranks, model and index versions, fusion settings, and timing in a trace. Don't log policy text in a broad diagnostic event.

When the trace below appears, predict the repair before inspecting the final prose: an empty sparse list with the right dense hit is a lane-specific miss, while an extra permitted distractor points toward later precision work.

The correct evidence is present, but the semantic lane also kept a session-timeout distractor. That's the boundary between retrieval and reranking: retrieval satisfies candidate recall; a reranker decides whether a distractor should remain near context. The trace can also preserve the latency budget created in the production RAG lesson. Two retrieval lanes add work, so the release check should report that cost explicitly.

The next check treats missing timing as a failure of observability, even when retrieval is correct. A route can't pass a latency gate when one stage never reports its elapsed time.

If a context budget is being wasted by several near-duplicate candidates, Maximal Marginal Relevance (MMR) is one selection strategy: choose a relevant result while penalizing candidates too similar to what has already been selected.^{[6]Reference 6The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries.https://dl.acm.org/doi/10.1145/290941.291025} MMR handles diversity in an existing permitted candidate set. It doesn't retrieve a missing policy and doesn't replace a cross-encoder that must compare query relevance precisely.

The paraphrase trace still carries eu-session-timeout-v1-rule next to the right policy. Recall succeeded; precision is the next job.