Hybrid Search: Dense + Sparse

The production RAG lesson built policy-answerer-v1 around a hard rule: only permitted, current evidence may reach an answer. Its simple term-overlap retriever was easy to audit, but it misses a support specialist who asks to "refresh an expired machine credential" when the policy says "stale service-account key rotation."

Upgrade only that retrieval lane. You'll build policy-answerer-v2. The code KROT-14 gives sparse retrieval an exact signal; dense retrieval catches paraphrased meaning; and Reciprocal Rank Fusion (RRF) merges candidate lists. The authorization, freshness, citation, and abstention contract doesn't change.

One retriever can't cover both queries

Luna, an EU support specialist, needs the same policy for two different searches:

A word-matching index has a decisive clue for the first query and no shared vocabulary for the second. A semantic encoder can represent the second query near the policy, but an unfamiliar internal identifier may carry little useful semantic signal. Neither failure says one method is bad. They solve different recall problems.

The safe online order is visible in the figure: define the current permitted corpus first, run both retrieval lanes only over that corpus, then fuse candidate IDs. Fusion improves recall; it doesn't widen access.

Why must the permission filter run before both retrieval lanes rather than after fusion?

Answer

A hidden chunk can be highly ranked by either lane. If text or identifiers from that chunk reach fusion, model context, caches, or request logs before filtering, the service has already crossed its evidence boundary.

Recreate the permitted candidate universe

The lab reuses the policy shape from the previous lesson. It adds diagnostic policy code KROT-14 so an exact-identifier query has an unambiguous expected result. Two tempting records remain in storage but must not be searchable for Luna: a superseded revision and a restricted admin-only rule.

Every ranker below receives permitted, not CHUNKS. This isn't a presentation detail: it's the API boundary that prevents a new ranking algorithm from weakening the service contract.

The fixed EVAL_DATE keeps replay behavior stable. The chunk shape also preserves document_id and parent_id from the previous lesson, even though ranking changes while citation packing stays the same.

This test keeps a deliberately attractive hidden policy in storage. Later ranking changes fail loudly if they accidentally widen the searchable set.

Build the sparse lane with BM25

Sparse retrieval represents a document by vocabulary terms. Most coordinates are zero because a short policy chunk uses only a small part of the vocabulary. BM25 ranks a document higher when it shares rare query terms, while limiting the reward for repeated terms and compensating for unusually long documents.^{[1]Reference 1The Probabilistic Relevance Framework: BM25 and Beyond.https://doi.org/10.1561/1500000019} The lab below uses the common Lucene-style IDF form with a leading 1 +, which is the practical formula in many search engines rather than the classic Okapi IDF alone.

For a query term $t$ and document $d$, the lab computes:

$$\operatorname{BM25}(q,d)=\sum_{t \in q}\operatorname{IDF}(t) \frac{f(t,d)(k_1+1)} {f(t,d)+k_1(1-b+b\lvert d\rvert/\operatorname{avgdl})}$$

Here, $f(t,d)$ is the term count in the chunk, $\lvert d\rvert$ is chunk length in tokens, and avgdl is the corpus average. k1 controls term-frequency saturation; b controls length normalization. The exact identifier krot-14 occurs only in the relevant current chunk, so it receives strong lexical weight.

The small analyzer below keeps hyphenated rule codes intact and removes common function words. Without that stopword rule, a query containing only a shared word such as "a" could appear to retrieve an unrelated policy.

BM25 did its job. It recovered the policy from its code and openly failed when the user used no policy vocabulary. A real evaluation set needs both query types; otherwise the lexical lane can look perfect while customers miss evidence.

BM25 isn't the only sparse option. SPLADE learns sparse expansion weights, so a chunk can gain indexable related terms while preserving sparse retrieval infrastructure.^{[2]Reference 2SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking.https://arxiv.org/abs/2107.05720} That can improve vocabulary mismatch cases, but it doesn't turn sparse retrieval into an authorization layer or guarantee better recall on security-policy queries. Evaluate a SPLADE candidate against the same permitted corpus, held-out required-evidence IDs, latency budget, and hidden-source exclusions before replacing BM25.

If BM25 returns no result for "refresh expired machine credential," should you lower an authorization or freshness filter?

Answer

No. The evidence boundary is not a recall knob. Add a semantic retrieval lane or improve the searchable representation, then test the change inside the same permission and version constraints.

Add a dense semantic lane

A dense retriever encodes queries and chunks as compact vectors, then retrieves chunks with high similarity. Dense Passage Retrieval (DPR), for example, uses separate encoders for questions and passages so passage representations can be indexed before requests arrive.^{[3]Reference 3Dense Passage Retrieval for Open-Domain Question Answering.https://arxiv.org/abs/2004.04906}

Downloading and training an encoder would hide the retrieval mechanics in this lab. Instead, the next cell uses frozen three-dimensional vectors as test fixtures. Read them as outputs already produced by an embedding model:

This fixture is deliberately honest about one failure: the internal code KROT-14 has no semantic vector by itself. The paraphrase does.

The lab ranks those vectors with cosine similarity. Vectors pointing in a similar direction score closer to 1; their raw length doesn't decide the result.

The fixture doesn't claim that every production encoder misses every identifier. It establishes a regression case: this chosen encoder representation doesn't recover the code-only query, so deleting the sparse lane would fail a known requirement.

Fuse candidates without mixing score scales

BM25 scores and cosine similarities don't share units. A BM25 value reflects term statistics in this index; a cosine value reflects vector alignment. Adding raw values can let whichever scale is numerically larger control the order.

Reciprocal Rank Fusion avoids that comparison. It contributes $1 / (k + r)$ for each rank $r$ at which a chunk appears:

$$\operatorname{RRF}(d)=\sum_{\text{lane } l}\frac{1}{k+\operatorname{rank}_l(d)}$$

We use k=60, the setting reported in the original RRF experiments, as a starting value rather than a universal optimum.^{[4]Reference 4Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.https://dl.acm.org/doi/10.1145/1571941.1572114} A chunk found by both lanes gains two contributions; a strong result found by one lane remains eligible.

For the shared-language query, eu-key-rotation-v2-rule ranks first in both lanes, so its fused score is 1 / 61 + 1 / 61 = 0.0328. The session-timeout distractor ranks second in both lanes, so its score is 1 / 62 + 1 / 62 = 0.0323. These values aren't probabilities. They are rank-based scores used to order the fused candidate set.

RRF doesn't manufacture relevance. It makes the two candidate sources interoperable. If both lanes miss the right chunk, a fused list will still be wrong.

Don't set each lane's depth equal to the final context budget. If top_k is 2 and each lane only returns 2 hits, mid-ranked evidence never meets in fusion. Retrieve a deeper candidate pool per lane, fuse, then keep top_k.

A dual-listed distractor can still outrank a correct one-lane hit after fusion. Gate on Recall@k after fusion alongside per-lane diagnostics:

If the generator only packs top_k=2, the correct policy is present in a sparse lane but absent from context. That is a fusion-depth failure, not a permission failure. Gate on Recall@k after fusion, raise fused top_k, deepen the weak lane, or pass the fuller fused candidate set to a later precision stage before claiming retrieval succeeded.

Why shouldn't you add a BM25 score directly to a cosine-similarity score?

Answer

They don't share units or a common numeric scale. A large BM25 value can dominate a cosine value even when that isn't the ranking behavior you want. RRF combines rank positions instead; normalized weighted fusion is another option when you tune it on labeled queries.

Attack the evidence boundary through fusion

The stored admin-only policy contains a unique code. If the permission boundary moved after retrieval, BM25 would have an easy hidden hit to surface. A hybrid implementation must return nothing for Luna's request for that code.

Gate the retriever on recall and safety

In the previous lesson, answer quality depended on retrieving current permitted evidence. That means the retrieval upgrade needs its own release cases before you measure generated text.

Recall@2 answers a narrow question: for each supported query, did the correct permitted chunk appear in the first two candidates? It doesn't say whether the evidence order is perfect or whether the final answer is faithful. Those are later checks. Here, recall exposes whether the generator even gets a chance to see the right policy.

These three fixtures demonstrate complementary failures; they don't prove an offline lift for a production corpus. A release decision needs a held-out set drawn from real support requests, including exact codes, paraphrases, unsupported questions, languages served by the product, and attempts to request hidden policies.

Hybrid reaches Recall@2 of 1.00 on this tiny set. Can you say hybrid search is better for all security-policy queries?

Answer

No. You can say it repairs the two frozen failure cases while preserving the safety boundary. Broader claims require a representative held-out query set and latency measurements.

Trace each lane before adding reranking

When a final answer is wrong, you need to tell apart three failures:

Store IDs, ranks, model and index versions, fusion settings, and timing in a trace. Don't log policy text in a broad diagnostic event.

The correct evidence is present, but the semantic lane also kept a session-timeout distractor. That's the boundary between retrieval and reranking: retrieval satisfies candidate recall; a reranker decides whether a distractor should remain near context. The trace can also preserve the latency budget created in the production RAG lesson. Two retrieval lanes add work, so the release check should report that cost explicitly.

If a context budget is being wasted by several near-duplicate candidates, Maximal Marginal Relevance (MMR) is one selection strategy: choose a relevant result while penalizing candidates too similar to what has already been selected.^{[5]Reference 5The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries.https://dl.acm.org/doi/10.1145/290941.291025} MMR handles diversity in an existing permitted candidate set. It doesn't retrieve a missing policy and doesn't replace a cross-encoder that must compare query relevance precisely.

Should you tune weights instead?

RRF is a good first implementation because it doesn't require calibrating unrelated score scales. It isn't an automatic winner. Bruch et al. found that RRF can be sensitive to its parameter and that a tuned convex combination can outperform it in their tested settings.^{[6]Reference 6An Analysis of Fusion Functions for Hybrid Retrieval.https://arxiv.org/abs/2210.11934} If you have enough labeled queries, compare it against normalized weighted fusion:

$$\operatorname{score}_{\text{hybrid}}(d) =\alpha\widetilde{\operatorname{score}}_{\text{dense}}(d) +(1-\alpha)\widetilde{\operatorname{score}}_{\text{sparse}}(d)$$

The tildes denote scores normalized within their lanes before fusion. That comparison is an evaluation task, not a reason to guess an alpha in production. Keep a fixed held-out split, version the encoder and index, report Recall@k and latency for every candidate, and retain RRF if a tuned method doesn't hold up out of sample.

Raw 0.7 * cosine + 0.3 * BM25 is invalid without lane normalization. The next cell shows why magnitudes dominate weights, then min-max normalizes each lane before the same convex mix:

Both calibrated methods agree on this toy query. Agreement on one toy query doesn't license an untuned alpha; it only demonstrates that lane scales must be made comparable before weighted fusion becomes a candidate against RRF.