Reranking and Cross-Encoders for RAG

The hybrid-search lesson built policy-answerer-v2 for retrieval-augmented generation (RAG): only current, permitted policy chunks enter retrieval, then sparse and dense ranks are fused. That fixed recall. It didn't guarantee that the best chunk fits into a small generator context window.

policy-answerer-v3 adds the missing reranking step. Luna is answering a developer's question:

Can a service account use the legacy token endpoint for 10 more days if audit logging is enabled?

Hybrid retrieval has already found the current rule api-token-legacy-v2-rule, but a nearby troubleshooting note appears above it. You'll add a reranking stage that reorders only the permitted candidates, admits supported evidence into a two-chunk maximum context budget, and emits a release trace for evaluation.

The boundary: retrieve candidates, then improve their order

A two-stage retriever separates two problems:

The reranker can't restore a missing document. It must stay inside the same permission boundary: never search a restricted or superseded document as a shortcut.

The runtime contract is compact: accept the permitted hybrid candidates, score each query-chunk pair, admit only score-gated top context, and write an evidence-level trace.

Why does the reranker receive the permitted hybrid candidate list instead of every policy chunk?

Answer

Reranking is an ordering stage, not an authorization stage. Scoring restricted or stale chunks can leak their content through scores, logs, caches, or final context even if they are filtered later.

The lab starts from a small fixture representing the hybrid output from the previous lesson. api-token-legacy-v2-rule is present, so recall succeeded. It's at rank 3, so a two-chunk context window would still omit the answer.

The candidate record keeps document_id and parent_id from the earlier RAG pipeline. Reranking changes evidence order, not the source identity that citation packing needs later.

Keep the evidence boundary executable

The store also contains records Luna may not use: an admin-only rule and an expired policy revision. A regression test should prove that neither enters reranking, even though either might be attractive for the query.

The allowlist is the first-stage chunk-ID set, not a fresh corpus scan. Stable IDs keep candidate membership explicit even if records are loaded from another store layer.

With no reranker, the generator receives the first two hybrid candidates. That context contains a troubleshooting note that explicitly says doesn't authorize temporary access, plus a password-reset policy. The correct legacy-token rule is present in retrieval results but absent from context.

Why a cross-encoder helps

A bi-encoder encodes a query and each chunk separately, then compares stored document vectors to a query vector. Separately encoded representations make large-scale retrieval practical because document embeddings can be indexed before a request arrives.^{[1]Reference 1Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.https://arxiv.org/abs/1908.10084}

A cross-encoder receives the query and one candidate chunk together and returns a relevance score for that pair. Passage reranking with BERT applies this joint scoring only after an initial retrieval stage, because request-time scoring for each pair is more expensive than searching a prebuilt index.^{[2]Reference 2Passage Re-ranking with BERT.https://arxiv.org/abs/1901.04085}

Because the query and chunk share one input sequence, a cross-encoder has a maximum input length (often 512 tokens for classic BERT-style rerankers). If a wrapper silently truncates an overlong pair, the model may rank only a prefix while the generator later injects the full chunk. That is a grounding bug, not a latency tweak: the score no longer describes the text that will be packed into context. Log the tokenized pair length, define an explicit truncation policy, and treat unexpected truncation as a release failure.

Many Hugging Face wrappers default to truncation=True. That is convenient and dangerous: the authorizing clause can fall off the right edge while the generator still sees it.

The fixture fails on purpose. A production scorer must keep the scored text identical to the packed context or record the kept span and refuse to treat a prefix score as a full-chunk rank. Silent truncation=True would have ranked background text while the prompt still injected the authorizing rule.

That distinction matters here. A troubleshooting note shares legacy token endpoint, migration, and 14 days with Luna's question, but it also says doesn't authorize temporary access. The legacy-token rule must match the requested endpoint, temporary-access remedy, migration window, and audit condition together.

The next fixture is deliberately transparent. It isn't a trained transformer, and it doesn't claim real model accuracy. It makes the pairwise requirements inspectable before you plug in a model endpoint.

The score below rewards a chunk only when its policy can support the developer's constraints. It also applies an explicit contradiction penalty. A learned cross-encoder would learn a scoring function from labeled query-passage pairs; the fixture gives the lab a stable expected result.

The target policy is retrieved at rank 3 but the prompt accepts two chunks. Is this a recall failure or an ordering failure?

Answer

It's an ordering failure. Recall succeeded because the target is in the candidate set. Reranking must move it into the context budget.

Now context selection changes for the right reason: the candidate set stays fixed while the supported rule moves to rank 1. A context budget is a maximum, not a quota, so low-scoring near matches aren't admitted merely to fill space.

MIN_CONTEXT_SCORE = 5 works because this fixture has a deterministic integer scorer with known range. Learned cross-encoder scores are usually uncalibrated, and their scale can shift across model versions and queries. Prefer rank order plus context budget. Learned CE logits are not portable thresholds across queries. If product needs a score gate, calibrate it for the exact model on labeled data and re-tune after reranker changes.

Measure ordering, not vibes

For this stage, keep evaluation narrow:

1. Recall@candidate_k checks whether first-stage retrieval gave the reranker a chance.
2. MRR (Mean Reciprocal Rank) averages how early the first relevant chunk appears across a release suite.
3. NDCG@context_k (Normalized Discounted Cumulative Gain) checks whether relevant chunks fit near the top of the context budget.^{[3]Reference 3Introduction to Information Retrieval.https://nlp.stanford.edu/IR-book/}

For this one request, reciprocal rank (RR) is easy to read: rank 3 gives 1 / 3; rank 1 gives 1. MRR is the mean of those per-request values. NDCG supports graded relevance; this lab uses binary relevance, where a relevant chunk has rel_i = 1 and an irrelevant chunk has rel_i = 0:

$$\operatorname{DCG}_k = \sum_{i=1}^{k} \frac{2^{rel_i}-1}{\log_2(i+1)} \qquad \operatorname{NDCG}_k = \frac{\operatorname{DCG}_k}{\operatorname{IDCG}_k}$$

Don't confuse improved evidence ordering with a correct generated answer. The next lesson will evaluate faithfulness and citation agreement after context reaches the generator.

Candidate count sets the ceiling and the bill

If the reranker receives only the first two candidates, it can't select api-token-legacy-v2-rule: the rule was cut before pair scoring. Use a candidate-recall gate before comparing reranker models.

A cross-encoder scores every query-candidate pair online. This fixture uses a deliberately sequential cost model so the candidate-count tradeoff stays visible. It isn't a p95 estimator or a hardware benchmark: production serving may batch pairs, and percentiles must be measured end to end.

The lab result is intentionally specific: retrieve at least three candidates, rerank three, and pass two chunks. A real service chooses these numbers from held-out traces and measured end-to-end p95 latency by candidate count, input length, and batch strategy, not from a generic default.

Can this sequential fixture prove deployed p95 latency for a batched cross-encoder endpoint?

Answer

No. It makes the candidate-count tradeoff visible. Measure end-to-end p95 latency with the real model, input lengths, batching strategy, and hardware before setting a production budget.

Choose an interaction design deliberately

Three ranking designs differ in when the query and chunk can interact:

ColBERT keeps document-side representations indexable while applying token-level late interaction at request time.^{[4]Reference 4ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.https://arxiv.org/abs/2004.12832} For each query token, MaxSim keeps its highest similarity to any document token, then sums those per-token maxima into a relevance score. It isn't an automatic upgrade. Compare it against a cross-encoder on the same held-out candidate lists, ordering metric, memory cost, and p95 latency.

Why can a document embedding be indexed before a request, while a cross-encoder relevance score can't?

Answer

A document embedding is computed without knowing the query. A cross-encoder score depends on a specific query and a specific chunk processed together, so it exists only at request time.

Emit the trace that evaluation needs

Shipping a reranker without evidence-level traces makes failures ambiguous. Record first-stage IDs, actual reranker input, candidate source identity, policy versions, pair scores, model versions, selected context IDs, and release gates. A document version is part of correctness: a well-ranked stale rule is still unsafe context.

This trace proves that reranking improved the evidence presented to generation. It doesn't yet prove that an answer states the rule faithfully or cites it correctly. That's exactly the boundary for the next lesson.

Production checks

Before releasing a real reranker:

One attractive mistake is caching a reranker result without its source version. If api-token-legacy-v2-rule changes, a cached score for an older chunk can keep stale evidence at rank 1. Key caches and traces by chunk checksum or policy version, then rerun golden traces after source updates.