Reranking and Cross-Encoders for RAG

The hybrid-search lesson built policy-answerer-v2 for retrieval-augmented generation (RAG): only current, permitted policy chunks enter retrieval, then sparse and dense ranks are fused. That fixed recall. It didn't guarantee that the best chunk fits into a small generator context window.

policy-answerer-v3 adds the missing reranking step. Luna is answering a developer's question:

Can a service account use the legacy token endpoint for 10 more days if audit logging is enabled?

Hybrid retrieval has already found the current rule api-token-legacy-v2-rule, but a nearby troubleshooting note appears above it. You'll add a reranking stage that reorders only the permitted candidates, admits supported evidence into a two-chunk maximum context budget, and emits a release trace for evaluation.

The boundary: retrieve candidates, then improve their order

A two-stage retriever separates two problems:

The reranker can't restore a missing document. It must stay inside the same permission boundary: never search a restricted or superseded document as a shortcut.

The runtime contract is compact: accept the permitted hybrid candidates, score each query-chunk pair, admit only score-gated top context, and write an evidence-level trace.

The lab starts from a small fixture representing the hybrid output from the previous lesson. api-token-legacy-v2-rule is present, so recall succeeded. It's at rank 3, so a two-chunk context window would still omit the answer.

The candidate record keeps document_id and parent_id from the earlier RAG pipeline. Reranking changes evidence order, not the source identity that citation packing needs later.

Keep the evidence boundary executable

The store also contains records Luna may not use: an admin-only rule and an expired policy revision. A regression test should prove that neither enters reranking, even though either might be attractive for the query.

The allowlist is the first-stage chunk-ID set, not a fresh corpus scan. Stable IDs keep candidate membership explicit even if records are loaded from another store layer.

With no reranker, the generator receives the first two hybrid candidates. That context contains a troubleshooting note that explicitly says doesn't authorize temporary access, plus a password-reset policy. The correct legacy-token rule is present in retrieval results but absent from context.

Why a cross-encoder helps

A bi-encoder encodes a query and each chunk separately, then compares stored document vectors to a query vector. Separately encoded representations make large-scale retrieval practical because document embeddings can be indexed before a request arrives.^[1]

A cross-encoder receives the query and one candidate chunk together and returns a relevance score for that pair. Passage reranking with BERT applies this joint scoring only after an initial retrieval stage, because request-time scoring for each pair is more expensive than searching a prebuilt index.^[2]

That distinction matters here. A troubleshooting note shares legacy token endpoint, migration, and 14 days with Luna's question, but it also says doesn't authorize temporary access. The legacy-token rule must match the requested endpoint, temporary-access remedy, migration window, and audit condition together.

The next fixture is deliberately transparent. It isn't a trained transformer, and it doesn't claim real model accuracy. It makes the pairwise requirements inspectable before you plug in a model endpoint.

The score below rewards a chunk only when its policy can support the developer's constraints. It also applies an explicit contradiction penalty. A learned cross-encoder would learn a scoring function from labeled query-passage pairs; the fixture gives the lab a stable expected result.

Now context selection changes for the right reason: the candidate set stays fixed while the supported rule moves to rank 1. A context budget is a maximum, not a quota, so low-scoring near matches aren't admitted merely to fill space.

Measure ordering, not vibes

For this stage, keep evaluation narrow:

1. Recall@candidate_k checks whether first-stage retrieval gave the reranker a chance.
2. MRR (Mean Reciprocal Rank) averages how early the first relevant chunk appears across a release suite.
3. NDCG@context_k (Normalized Discounted Cumulative Gain) checks whether relevant chunks fit near the top of the context budget.^[3]

For this one request, reciprocal rank (RR) is easy to read: rank 3 gives 1 / 3; rank 1 gives 1. MRR is the mean of those per-request values. NDCG supports graded relevance; this lab uses binary relevance, where a relevant chunk has rel_i = 1 and an irrelevant chunk has rel_i = 0:

$$\operatorname{DCG}_k = \sum_{i=1}^{k} \frac{2^{rel_i}-1}{\log_2(i+1)} \qquad \operatorname{NDCG}_k = \frac{\operatorname{DCG}_k}{\operatorname{IDCG}_k}$$

Don't confuse improved evidence ordering with a correct generated answer. The next lesson will evaluate faithfulness and citation agreement after context reaches the generator.

Candidate count sets the ceiling and the bill

If the reranker receives only the first two candidates, it can't select api-token-legacy-v2-rule: the rule was cut before pair scoring. Use a candidate-recall gate before comparing reranker models.

A cross-encoder scores every query-candidate pair online. This fixture uses a deliberately sequential cost model so the candidate-count tradeoff stays visible. It isn't a p95 estimator or a hardware benchmark: production serving may batch pairs, and percentiles must be measured end to end.

The lab result is intentionally specific: retrieve at least three candidates, rerank three, and pass two chunks. A real service chooses these numbers from held-out traces and measured end-to-end p95 latency by candidate count, input length, and batch strategy, not from a generic default.

Choose an interaction design deliberately

Three ranking designs differ in when the query and chunk can interact:

ColBERT keeps document-side representations indexable while applying token-level late interaction at request time.^[4] For each query token, MaxSim keeps its highest similarity to any document token, then sums those per-token maxima into a relevance score. It isn't an automatic upgrade. Compare it against a cross-encoder on the same held-out candidate lists, ordering metric, memory cost, and p95 latency.

Emit the trace that evaluation needs

Shipping a reranker without evidence-level traces makes failures ambiguous. Record first-stage IDs, actual reranker input, candidate source identity, policy versions, pair scores, model versions, selected context IDs, and release gates. A document version is part of correctness: a well-ranked stale rule is still unsafe context.

This trace proves that reranking improved the evidence presented to generation. It doesn't yet prove that an answer states the rule faithfully or cites it correctly. That's exactly the boundary for the next lesson.

Production checks

Before releasing a real reranker:

One attractive mistake is caching a reranker result without its source version. If api-token-legacy-v2-rule changes, a cached score for an older chunk can keep stale evidence at rank 1. Key caches and traces by chunk checksum or policy version, then rerun golden traces after source updates.

Mastery check

You're ready to add a reranker to a production RAG pipeline when you can:

• Preserve authorization and freshness filtering before any pair scoring.
• Explain why first-stage recall and final evidence order need separate gates.
• Compare bi-encoder retrieval, cross-encoder reranking, and ColBERT-style late interaction.
• Measure ordering improvement with MRR or NDCG inside the context budget.
• Choose candidate count using candidate recall and measured end-to-end latency.
• Emit a trace suitable for downstream faithfulness and citation evaluation.

Evaluation rubric

Follow-up questions

Common pitfalls

Reranking is used to hide recall failure

• Symptom: Teams swap reranker models, but a gold policy chunk never appears in final context.
• Cause: First-stage retrieval did not include the chunk in its candidate set.
• Fix: Gate on Recall@candidate_k first, then evaluate ordering only on candidate sets that contain the answer.

Candidate count grows without a serving measurement

• Symptom: Ordering metrics improve while request latency exceeds the product budget.
• Cause: Each additional cross-encoder pair consumes request-time inference.
• Fix: Benchmark p95 latency by candidate count and input length, then choose the smallest candidate set that clears recall and ordering gates.

Evidence order improves but grounding remains untested

• Symptom: NDCG rises, but users still receive unsupported answers.
• Cause: Reranking evaluates selected context, not whether generation follows it.
• Fix: Carry the reranking trace into answer faithfulness and citation checks in the next evaluation stage.