Hybrid Search: Dense + Sparse Retrieval

The production RAG lesson built policy-answerer-v1 around a hard rule: only permitted, current evidence may reach an answer. Its simple term-overlap retriever was easy to audit, but it misses a customer who asks for a "swap for a broken reconditioned notebook" when the policy says "replacement for a damaged refurbished laptop."

This lesson upgrades only that retrieval lane. You'll build policy-answerer-v2. The code RPL-14 gives sparse retrieval an exact signal; dense retrieval catches paraphrased meaning; and Reciprocal Rank Fusion (RRF) merges candidate lists. The authorization, freshness, citation, and abstention contract doesn't change.

One retriever can't cover both queries

Luna, an EU support specialist, needs the same policy for two different searches:

A word-matching index has a decisive clue for the first query and no shared vocabulary for the second. A semantic encoder can represent the second query near the policy, but an unfamiliar internal identifier may carry little useful semantic signal. Neither failure says one method is bad. They solve different recall problems.

The safe online order is:

Recreate the permitted candidate universe

The lab reuses the policy shape from the previous lesson. It adds diagnostic policy code RPL-14 so an exact-identifier query has an unambiguous expected result. Two tempting records remain in storage but must not be searchable for Luna: a superseded revision and a restricted merchant rule.

Every ranker below receives permitted, not CHUNKS. This isn't a presentation detail: it is the API boundary that prevents a new ranking algorithm from weakening the service contract.

The fixed EVAL_DATE keeps replay behavior stable. The chunk shape also preserves document_id and parent_id from the previous lesson, even though this chapter changes ranking rather than citation packing.

This test keeps a deliberately attractive hidden policy in storage. Later ranking changes fail loudly if they accidentally widen the searchable set.

Build the sparse lane with BM25

Sparse retrieval represents a document by vocabulary terms. Most coordinates are zero because a short policy chunk uses only a small part of the vocabulary. BM25 ranks a document higher when it shares rare query terms, while limiting the reward for repeated terms and compensating for unusually long documents.^[1]

For a query term $t$ and document $d$, the lab computes:

$$\operatorname{BM25}(q,d)=\sum_{t \in q}\operatorname{IDF}(t) \frac{f(t,d)(k_1+1)} {f(t,d)+k_1(1-b+b\lvert d\rvert/\operatorname{avgdl})}$$

Here, $f(t,d)$ is the term count in the chunk, $\lvert d\rvert$ is chunk length in tokens, and avgdl is the corpus average. k1 controls term-frequency saturation; b controls length normalization. The exact identifier rpl-14 occurs only in the relevant current chunk, so it receives strong lexical weight.

The small analyzer below keeps hyphenated rule codes intact and removes common function words. Without that stopword rule, a query containing only a shared word such as "a" could appear to retrieve an unrelated policy.

BM25 did its job. It recovered the policy from its code and openly failed when the user used no policy vocabulary. A real evaluation set needs both query types; otherwise the lexical lane can look perfect while customers miss evidence.

BM25 is not the only sparse option. SPLADE learns sparse expansion weights, so a chunk can gain indexable related terms while preserving sparse retrieval infrastructure.^[2] That can improve vocabulary mismatch cases, but it doesn't turn sparse retrieval into an authorization layer or guarantee better recall on ShopFlow queries. Evaluate a SPLADE candidate against the same permitted corpus, held-out required-evidence IDs, latency budget, and hidden-source exclusions before replacing BM25.

Add a dense semantic lane

A dense retriever encodes queries and chunks as compact vectors, then retrieves chunks with high similarity. Dense Passage Retrieval (DPR), for example, uses separate encoders for questions and passages so passage representations can be indexed before requests arrive.^[3]

Downloading and training an encoder would hide the retrieval mechanics in this lab. Instead, the next cell uses frozen three-dimensional vectors as test fixtures. Read them as outputs already produced by an embedding model:

This fixture is deliberately honest about one failure: the internal code RPL-14 has no semantic vector by itself. The paraphrase does.

The fixture doesn't claim that every production encoder misses every identifier. It establishes a regression case: this chosen encoder representation doesn't recover the code-only query, so deleting the sparse lane would fail a known requirement.

Fuse candidates without mixing score scales

BM25 scores and cosine similarities don't share units. A BM25 value reflects term statistics in this index; a cosine value reflects vector alignment. Adding raw values can let whichever scale is numerically larger control the order.

Reciprocal Rank Fusion avoids that comparison. It contributes $1 / (k + r)$ for each rank $r$ at which a chunk appears:

$$\operatorname{RRF}(d)=\sum_{\text{lane } l}\frac{1}{k+\operatorname{rank}_l(d)}$$

We use k=60, the setting reported in the original RRF experiments, as a starting value rather than a universal optimum.^[4] A chunk found by both lanes gains two contributions; a strong result found by one lane remains eligible.

RRF doesn't manufacture relevance. It makes the two candidate sources interoperable. If both lanes miss the right chunk, a fused list will still be wrong.

Attack the evidence boundary through fusion

The stored VIP policy contains a unique code. If the permission boundary moved after retrieval, BM25 would have an easy hidden hit to surface. A hybrid implementation must return nothing for Luna's request for that code.

Gate the retriever on recall and safety

In the previous lesson, answer quality depended on retrieving current permitted evidence. That means the retrieval upgrade needs its own release cases before you measure generated prose.

Recall@2 answers a narrow question: for each supported query, did the correct permitted chunk appear in the first two candidates? It does not say whether the evidence order is perfect or whether the final answer is faithful. Those are later checks. Here, recall exposes whether the generator even gets a chance to see the right policy.

These three fixtures demonstrate complementary failures; they don't prove an offline lift for a production corpus. A release decision needs a held-out set drawn from real support requests, including exact codes, paraphrases, unsupported questions, languages served by the product, and attempts to request hidden policies.

Trace each lane before adding reranking

When a final answer is wrong, you need to tell apart three failures:

Store IDs, ranks, model and index versions, fusion settings, and timing. Don't log policy text in a broad diagnostic event.

The correct evidence is present, but the semantic lane also kept a footwear distractor. That is exactly the boundary between this lesson and the next: retrieval satisfies candidate recall; reranking decides whether a distractor should remain near context. The trace can also preserve the latency budget created in the production RAG lesson. Two retrieval lanes add work, so the release check should report that cost explicitly.

If a context budget is being wasted by several near-duplicate candidates, Maximum Marginal Relevance (MMR) is one selection strategy: choose a relevant result while penalizing candidates too similar to what has already been selected.^[5] MMR handles diversity in an existing permitted candidate set. It doesn't retrieve a missing policy and doesn't replace a cross-encoder that must compare query relevance precisely.

Should you tune weights instead?

RRF is a good first implementation because it doesn't require calibrating unrelated score scales. It isn't an automatic winner. Bruch et al. found that RRF can be sensitive to its parameter and that a tuned convex combination can outperform it in their tested settings.^[6] If you have enough labeled queries, compare it against normalized weighted fusion:

$$\operatorname{score}_{\text{hybrid}}(d) =\alpha\operatorname{score}_{\text{dense}}(d) +(1-\alpha)\operatorname{score}_{\text{sparse}}(d)$$

That comparison is an evaluation task, not a reason to guess an alpha in production. Keep a fixed held-out split, version the encoder and index, report Recall@k and latency for every candidate, and retain RRF if a tuned method doesn't hold up out of sample.

Build it yourself

Extend policy-answerer-v2 without weakening its contract:

1. Add at least eight permitted positive queries: exact codes, natural-language paraphrases, and mixed queries.
2. Add at least four negative or adversarial queries: restricted merchant rules, superseded policy wording, wrong region, and absent evidence.
3. Replace the frozen dense vectors with embeddings produced by your chosen bi-encoder, recording the model version.
4. Compare BM25, dense, and RRF using permitted Recall@k; record p50 and p95 retrieval latency.
5. Save a compact trace for each failed case containing candidate IDs, lane ranks, versions, and timings, but not restricted content.
6. Keep the fused top candidates as the input artifact for the next lesson's reranker.

The important artifact is not a search demo. It is a retrieval report showing which evidence questions are recovered, which must abstain, and which source boundaries remain enforced after the upgrade.

Mastery check

You are ready to use hybrid retrieval in a RAG system when you can:

• Explain why BM25 recovers rare identifiers and why dense retrieval can recover paraphrases.
• Explain when learned sparse expansion such as SPLADE is a candidate alternative to BM25.
• Implement BM25 scoring and RRF fusion over permitted candidate records.
• Explain why MMR diversifies selected evidence but doesn't repair a retrieval miss.
• Explain why BM25 scores and cosine similarities must not be added without calibration.
• Treat frozen semantic vectors as a test fixture, not proof that one production embedding model behaves the same way.
• Evaluate retrieval with expected evidence IDs before evaluating generated answers.
• Preserve authorization and freshness filtering ahead of both retrieval lanes.
• Produce a lane-by-lane trace that makes a missed chunk diagnosable.

Evaluation rubric

Common pitfalls