Hybrid Search: Dense + Sparse

The production RAG lesson built policy-answerer-v1 around a hard rule: only permitted, current evidence may reach an answer. Its simple term-overlap retriever was easy to audit, but it misses a support specialist who asks to "refresh an expired machine credential" when the policy says "stale service-account key rotation."

Upgrade only that retrieval lane. You'll build policy-answerer-v2. The code KROT-14 gives sparse retrieval an exact signal; dense retrieval catches paraphrased meaning; and Reciprocal Rank Fusion (RRF) merges candidate lists. The authorization, freshness, citation, and abstention contract doesn't change.

One retriever can't cover both queries

Luna, an EU support specialist, needs the same policy for two different searches:

A word-matching index has a decisive clue for the first query and no shared vocabulary for the second. A semantic encoder can represent the second query near the policy, but an unfamiliar internal identifier may carry little useful semantic signal. Neither failure says one method is bad. They solve different recall problems.

The safe online order is visible in the figure: define the current permitted corpus first, run both retrieval lanes only over that corpus, then fuse candidate IDs. Fusion improves recall; it doesn't widen access.

Recreate the permitted candidate universe

The lab reuses the policy shape from the previous lesson. It adds diagnostic policy code KROT-14 so an exact-identifier query has an unambiguous expected result. Two tempting records remain in storage but must not be searchable for Luna: a superseded revision and a restricted admin-only rule.

Every ranker below receives permitted, not CHUNKS. This isn't a presentation detail: it's the API boundary that prevents a new ranking algorithm from weakening the service contract.

The fixed EVAL_DATE keeps replay behavior stable. The chunk shape also preserves document_id and parent_id from the previous lesson, even though ranking changes while citation packing stays the same.

This test keeps a deliberately attractive hidden policy in storage. Later ranking changes fail loudly if they accidentally widen the searchable set.

Build the sparse lane with BM25

Sparse retrieval represents a document by vocabulary terms. Most coordinates are zero because a short policy chunk uses only a small part of the vocabulary. BM25 ranks a document higher when it shares rare query terms, while limiting the reward for repeated terms and compensating for unusually long documents.^{[1]Reference 1The Probabilistic Relevance Framework: BM25 and Beyond.https://doi.org/10.1561/1500000019} The lab below uses the common Lucene-style IDF form with a leading 1 +, which is the practical formula in many search engines rather than the classic Okapi IDF alone.

For a query term $t$ and document $d$, the lab computes:

$$\operatorname{BM25}(q,d)=\sum_{t \in q}\operatorname{IDF}(t) \frac{f(t,d)(k_1+1)} {f(t,d)+k_1(1-b+b\lvert d\rvert/\operatorname{avgdl})}$$

Here, $f(t,d)$ is the term count in the chunk, $\lvert d\rvert$ is chunk length in tokens, and avgdl is the corpus average. k1 controls term-frequency saturation; b controls length normalization. The exact identifier krot-14 occurs only in the relevant current chunk, so it receives strong lexical weight.

The small analyzer below keeps hyphenated rule codes intact and removes common function words. Without that stopword rule, a query containing only a shared word such as "a" could appear to retrieve an unrelated policy.

BM25 did its job. It recovered the policy from its code and openly failed when the user used no policy vocabulary. A real evaluation set needs both query types; otherwise the lexical lane can look perfect while customers miss evidence.

BM25 isn't the only sparse option. SPLADE learns sparse expansion weights, so a chunk can gain indexable related terms while preserving sparse retrieval infrastructure.^{[2]Reference 2SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking.https://arxiv.org/abs/2107.05720} That can improve vocabulary mismatch cases, but it doesn't turn sparse retrieval into an authorization layer or guarantee better recall on security-policy queries. Evaluate a SPLADE candidate against the same permitted corpus, held-out required-evidence IDs, latency budget, and hidden-source exclusions before replacing BM25.

Add a dense semantic lane

A dense retriever encodes queries and chunks as compact vectors, then retrieves chunks with high similarity. Dense Passage Retrieval (DPR), for example, uses separate encoders for questions and passages so passage representations can be indexed before requests arrive.^{[3]Reference 3Dense Passage Retrieval for Open-Domain Question Answering.https://arxiv.org/abs/2004.04906}

Downloading and training an encoder would hide the retrieval mechanics in this lab. Instead, the next cell uses frozen three-dimensional vectors as test fixtures. Read them as outputs already produced by an embedding model:

This fixture is deliberately honest about one failure: the internal code KROT-14 has no semantic vector by itself. The paraphrase does.

The lab ranks those vectors with cosine similarity. Vectors pointing in a similar direction score closer to 1; their raw length doesn't decide the result.

The fixture doesn't claim that every production encoder misses every identifier. It establishes a regression case: this chosen encoder representation doesn't recover the code-only query, so deleting the sparse lane would fail a known requirement.

Fuse candidates without mixing score scales

BM25 scores and cosine similarities don't share units. A BM25 value reflects term statistics in this index; a cosine value reflects vector alignment. Adding raw values can let whichever scale is numerically larger control the order.

Reciprocal Rank Fusion avoids that comparison. It contributes $1 / (k + r)$ for each rank $r$ at which a chunk appears:

$$\operatorname{RRF}(d)=\sum_{\text{lane } l}\frac{1}{k+\operatorname{rank}_l(d)}$$

We use k=60, the setting reported in the original RRF experiments, as a starting value rather than a universal optimum.^{[4]Reference 4Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.https://dl.acm.org/doi/10.1145/1571941.1572114} A chunk found by both lanes gains two contributions; a strong result found by one lane remains eligible.

For the shared-language query, eu-key-rotation-v2-rule ranks first in both lanes, so its fused score is 1 / 61 + 1 / 61 = 0.0328. The session-timeout distractor ranks second in both lanes, so its score is 1 / 62 + 1 / 62 = 0.0323. These values aren't probabilities. They are rank-based scores used to order the fused candidate set.

RRF doesn't manufacture relevance. It makes the two candidate sources interoperable. If both lanes miss the right chunk, a fused list will still be wrong.

Don't set each lane's depth equal to the final context budget. If top_k is 2 and each lane only returns 2 hits, mid-ranked evidence never meets in fusion. Retrieve a deeper candidate pool per lane, fuse, then keep top_k.

Attack the evidence boundary through fusion

The stored admin-only policy contains a unique code. If the permission boundary moved after retrieval, BM25 would have an easy hidden hit to surface. A hybrid implementation must return nothing for Luna's request for that code.

Gate the retriever on recall and safety

In the previous lesson, answer quality depended on retrieving current permitted evidence. That means the retrieval upgrade needs its own release cases before you measure generated text.

Recall@2 answers a narrow question: for each supported query, did the correct permitted chunk appear in the first two candidates? It doesn't say whether the evidence order is perfect or whether the final answer is faithful. Those are later checks. Here, recall exposes whether the generator even gets a chance to see the right policy.

These three fixtures demonstrate complementary failures; they don't prove an offline lift for a production corpus. A release decision needs a held-out set drawn from real support requests, including exact codes, paraphrases, unsupported questions, languages served by the product, and attempts to request hidden policies.

Trace each lane before adding reranking

When a final answer is wrong, you need to tell apart three failures:

Store IDs, ranks, model and index versions, fusion settings, and timing in a trace. Don't log policy text in a broad diagnostic event.

The correct evidence is present, but the semantic lane also kept a session-timeout distractor. That's the boundary between retrieval and reranking: retrieval satisfies candidate recall; a reranker decides whether a distractor should remain near context. The trace can also preserve the latency budget created in the production RAG lesson. Two retrieval lanes add work, so the release check should report that cost explicitly.

If a context budget is being wasted by several near-duplicate candidates, Maximal Marginal Relevance (MMR) is one selection strategy: choose a relevant result while penalizing candidates too similar to what has already been selected.^{[5]Reference 5The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries.https://dl.acm.org/doi/10.1145/290941.291025} MMR handles diversity in an existing permitted candidate set. It doesn't retrieve a missing policy and doesn't replace a cross-encoder that must compare query relevance precisely.

Should you tune weights instead?

RRF is a good first implementation because it doesn't require calibrating unrelated score scales. It isn't an automatic winner. Bruch et al. found that RRF can be sensitive to its parameter and that a tuned convex combination can outperform it in their tested settings.^{[6]Reference 6An Analysis of Fusion Functions for Hybrid Retrieval.https://arxiv.org/abs/2210.11934} If you have enough labeled queries, compare it against normalized weighted fusion:

$$\operatorname{score}_{\text{hybrid}}(d) =\alpha\widetilde{\operatorname{score}}_{\text{dense}}(d) +(1-\alpha)\widetilde{\operatorname{score}}_{\text{sparse}}(d)$$

The tildes denote scores normalized within their lanes before fusion. That comparison is an evaluation task, not a reason to guess an alpha in production. Keep a fixed held-out split, version the encoder and index, report Recall@k and latency for every candidate, and retain RRF if a tuned method doesn't hold up out of sample.

Build it yourself

Extend policy-answerer-v2 without weakening its contract:

1. Add at least eight permitted positive queries: exact codes, natural-language paraphrases, and mixed queries.
2. Add at least four negative or adversarial queries: restricted admin-only rules, superseded policy wording, wrong region, and absent evidence.
3. Replace the frozen dense vectors with embeddings produced by your chosen bi-encoder, recording the model version.
4. Compare BM25, dense, and RRF using permitted Recall@k; record p50 and p95 retrieval latency.
5. Save a compact trace for each failed case containing candidate IDs, lane ranks, versions, and timings, but not restricted content.
6. Keep the fused top candidates as the input artifact for the next lesson's reranker.

The important artifact isn't a search demo. It's a retrieval report showing which evidence questions are recovered, which must abstain, and which source boundaries remain enforced after the upgrade.

Mastery check

You're ready to use hybrid retrieval in a RAG system when you can:

• Explain why BM25 recovers rare identifiers and why dense retrieval can recover paraphrases.
• Explain when learned sparse expansion such as SPLADE is a candidate alternative to BM25.
• Implement BM25 scoring and RRF fusion over permitted candidate records.
• Explain why MMR diversifies selected evidence but doesn't repair a retrieval miss.
• Explain why BM25 scores and cosine similarities must not be added without calibration.
• Treat frozen semantic vectors as a test fixture, not proof that one production embedding model behaves the same way.
• Evaluate retrieval with expected evidence IDs before evaluating generated answers.
• Preserve authorization and freshness filtering ahead of both retrieval lanes.
• Produce a lane-by-lane trace that makes a missed chunk diagnosable.

Evaluation rubric

Common pitfalls