Reranking and Cross-Encoders for RAG

Hybrid retrieval can find the necessary evidence inside a retrieval-augmented generation (RAG) system and still leave it outside the generator's context. Consider this fictional enterprise policy lab:

Today is day 10 after deprecation. Can a service account use the legacy token endpoint with audit logging enabled?

BM25 matches an operational note (api-token-troubleshooting-v1) on exact keyword overlap. The note mentions legacy token errors and explicitly warns that it doesn't authorize access. Dense search matches a password-reset document (api-password-reset-v1) by semantic proximity. The governing rule is api-token-legacy-v2-rule (Rule AUTH-14). It allows service accounts to use the endpoint within 14 days of deprecation if audit logging is active. Hybrid retrieval still places that rule at rank 3.

The generator prompt budget accepts only two passages. The troubleshooting note and the password-reset policy occupy the prompt while Rule AUTH-14 stays outside it. An assistant might then produce an unsupported refusal: "No, legacy token access is strictly forbidden." We haven't run a generator here. That response illustrates the downstream risk.

For this question's one labeled relevant chunk, candidate recall is 1: the rule is in the candidate set. The four candidates have trusted, current permission decisions in the fixture. This demonstrates an ordering failure at cutoff two; measure corpus-wide retrieval and privacy behavior separately.

The previous hybrid-search pipeline, policy-answerer-v2, filters for current, permitted chunks, then fuses sparse and dense ranks. policy-answerer-v3 adds a reranker to score each query-chunk pair before context selection. The executable labs use a hand-written scorer to test ordering and trace contracts. They don't run a learned model or establish its accuracy.

View larger

The boundary: retrieve candidates, then improve their order

A two-stage retriever separates two problems. Retrieval decides whether evidence gets a chance to compete; reranking decides which admitted evidence reaches the top. Generation should support its claims with selected evidence. Merely restricting the prompt doesn't prevent a model from inventing claims or using its parametric memory.

The target is in the candidate set but below a two-chunk cutoff. Reranking may change its rank and selected context, not its source identity or access permissions.

The reranker can't restore a missing document. It must stay inside the same permission boundary: never search a restricted or superseded document as a shortcut. Permitted hybrid candidates enter scoring; admin-only and superseded records never do. That skip is the boundary, not a later filter.

This pipeline accepts permitted candidates, scores their query-chunk pairs, selects context, and records the decision. A freshness or permission revocation may remove a candidate between stages; it must never add a forbidden one. The fixed-membership examples assume an unchanged authorized snapshot.

Start with the first-stage receipt, before any model score. The small fixture shows api-token-legacy-v2-rule at rank 3. Its presence confirms recall; its position predicts a miss at a two-chunk cutoff. Predict the next output: the answer stays out until some stage changes the order.

from __future__ import annotations

from dataclasses import dataclass
from math import isfinite, log2

def finite_number(value: object) -> bool:
    if type(value) not in (int, float):
        return False
    try:
        return isfinite(value)
    except OverflowError:
        return False

def nonnegative_integer(value: object, name: str) -> None:
    if type(value) is not int or value < 0:
        raise ValueError(f"{name} must be a nonnegative integer")

@dataclass(frozen=True)
class Candidate:
    chunk_id: str
    document_id: str
    parent_id: str
    version: str
    permitted: bool
    current: bool
    first_stage_rank: int
    text: str

QUERY = (
    "legacy token endpoint for service account on day 10 after deprecation "
    "with audit logging enabled"
)
TARGET_ID = "api-token-legacy-v2-rule"
CANDIDATES = [
    Candidate(
        "api-token-troubleshooting-v1",
        "api-token-troubleshooting",
        "api-token-troubleshooting-v1",
        "api-token-troubleshooting/2026-04-20",
        True,
        True,
        1,
        (
            "Legacy token endpoint errors can be inspected during migration. "
            "This note does not authorize temporary access."
        ),
    ),
    Candidate(
        "api-password-reset-v1",
        "api-password-reset",
        "api-password-reset-v1",
        "api-password-reset/2026-01-03",
        True,
        True,
        2,
        "Password reset tokens expire after 30 minutes.",
    ),
    Candidate(
        "api-token-legacy-v2-rule",
        "api-auth",
        "api-auth-v2",
        "api-auth/2026-04-01",
        True,
        True,
        3,
        (
            "Rule AUTH-14. Service accounts may use the legacy token endpoint "
            "within 14 days of deprecation when audit logging is enabled."
        ),
    ),
    Candidate(
        "api-audit-export-v1",
        "api-audit",
        "api-audit-v1",
        "api-audit/2026-03-12",
        True,
        True,
        4,
        "Audit logs can be exported within 14 days.",
    ),
]

first_stage = sorted(CANDIDATES, key=lambda candidate: candidate.first_stage_rank)
first_stage_ids = [candidate.chunk_id for candidate in first_stage]
print("Hybrid order:", first_stage_ids)
assert TARGET_ID in first_stage_ids
assert first_stage_ids.index(TARGET_ID) + 1 == 3

Hybrid order: ['api-token-troubleshooting-v1', 'api-password-reset-v1', 'api-token-legacy-v2-rule', 'api-audit-export-v1']

The candidate record carries document_id and parent_id from the earlier RAG pipeline. Reranking may move evidence, but citation packing still points to the same source identity. A higher rank doesn't change who may read the chunk. What keeps an admin-only rule out of the scorer?

Keep the evidence boundary executable

The source store still holds an admin-only rule and an expired revision. Neither should enter reranking, even if the wording looks useful. Here, permitted and current are trusted fixture decisions, not fields supplied by the question or source text.

SOURCE_STORE = CANDIDATES + [
    Candidate(
        "admin-token-legacy",
        "admin-token-terms",
        "admin-token-terms",
        "admin-token/2026-05-01",
        False,
        True,
        0,
        "Admin service accounts receive immediate legacy token access.",
    ),
    Candidate(
        "api-token-legacy-v1-rule",
        "api-auth",
        "api-auth-v1",
        "api-auth/2025-02-01",
        True,
        False,
        0,
        "Service accounts may use the legacy token endpoint within 30 days.",
    ),
]
FIRST_STAGE_RECORDS = {
    (candidate.chunk_id, candidate.version): candidate
    for candidate in first_stage
}

def in_snapshot(candidate: Candidate) -> bool:
    return (
        isinstance(candidate, Candidate)
        and candidate.permitted is True
        and candidate.current is True
        and FIRST_STAGE_RECORDS.get((candidate.chunk_id, candidate.version)) == candidate
    )

def rerankable_candidates(store: list[Candidate]) -> list[Candidate]:
    return [
        candidate
        for candidate in store
        if in_snapshot(candidate)
    ]

rerankable = rerankable_candidates(SOURCE_STORE)
blocked_ids = sorted(
    candidate.chunk_id
    for candidate in SOURCE_STORE
    if candidate not in rerankable
)
print("Rerankable:", [candidate.chunk_id for candidate in rerankable])
print("Blocked:", blocked_ids)
assert rerankable == CANDIDATES
assert blocked_ids == ["admin-token-legacy", "api-token-legacy-v1-rule"]

Rerankable: ['api-token-troubleshooting-v1', 'api-password-reset-v1', 'api-token-legacy-v2-rule', 'api-audit-export-v1']
Blocked: ['admin-token-legacy', 'api-token-legacy-v1-rule']

The snapshot compares complete records, including text and parent identity. Matching an ID and version alone would admit changed content under an old receipt. In a service, fetch the recorded revisions and recheck the caller's current permissions before remote scoring and context packing. A changed record needs a fresh retrieval decision. Sending text to a hosted reranker is itself a disclosure: user access doesn't automatically authorize an external processor. Scope caches, logs, and provider use to the same data-handling policy.

Now replay context packing without reranking. The generator receives a troubleshooting note that grants no temporary access and a password-reset policy. The legacy-token rule is retrieved but absent from context.

CONTEXT_BUDGET = 2

def pack_context(candidates: list[Candidate], budget: int) -> list[str]:
    nonnegative_integer(budget, "context budget")
    if len({candidate.chunk_id for candidate in candidates}) != len(candidates):
        raise ValueError("duplicate context candidate IDs")
    if any(not in_snapshot(candidate) for candidate in candidates):
        raise ValueError("context candidate outside authorized snapshot")
    return [candidate.chunk_id for candidate in candidates[:budget]]

context_before = pack_context(first_stage, CONTEXT_BUDGET)
print("Context before rerank:", context_before)
print("Target reaches generation:", TARGET_ID in context_before)
assert TARGET_ID not in context_before

Context before rerank: ['api-token-troubleshooting-v1', 'api-password-reset-v1']
Target reaches generation: False

The current hybrid search ordering leaves out the right chunk. Reranking is one possible repair; improving first-stage ranking or context selection is another. What additional signal comes from reading the query and chunk together?

Read the scoring contract before comparing model families

Which tensor becomes the score? Does it depend on one passage or the whole list? Those questions explain both serving behavior and cache validity. A decoder backbone doesn't imply that a reranker generates a ranked list or even uses next-token probabilities.

BERT-style encoder classification heads

BERT-style cross-encoders concatenate the query and candidate chunk: [CLS] Query [SEP] Chunk [SEP]. Bidirectional self-attention lets the two sides interact before pooling.^{[1]Reference 1Passage Re-ranking with BERT.https://arxiv.org/abs/1901.04085[2]Reference 2BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.https://arxiv.org/abs/1810.04805} This is one architecture, not a description of every checkpoint sold under the BGE or Jina name.

Let $h_{\mathrm{CLS}}(q, d) \in \mathbb{R}^d$ be the final hidden state of the [CLS] token. A linear projection head maps this pooled vector to a scalar logit:

Applying a sigmoid function $\sigma(z) = 1 / (1 + e^{-z})$ bounds the logit into $(0, 1)$. During pointwise training, each query-passage example carries a binary relevance label $y \in \{0, 1\}$ and contributes binary cross-entropy:

At inference time, there are no gold labels or gradient updates. The model computes $z$ (or $\sigma(z)$) for each candidate and sorts descending. Mathematically, sigmoid is strictly increasing, so it preserves order. Finite-precision saturation can introduce ties near 0 or 1. A sigmoid score isn't automatically a calibrated relevance probability. Sentence Transformers documents checkpoints that return raw logits and an optional sigmoid transformation.^{[3]Reference 3Cross Encoder Usagehttps://github.com/huggingface/sentence-transformers/blob/main/docs/cross_encoder/usage/usage.rst}

Pointwise scoring evaluates each pair in isolation, but training may use a pairwise preference loss. Given a relevant passage $d^+$ with logit $z^+$ and an irrelevant passage $d^-$ with logit $z^-$, the pairwise logistic preference loss penalizes inverted score differences:^{[4]Reference 4Learning to Rank using Gradient Descent.https://www.microsoft.com/en-us/research/publication/learning-to-rank-using-gradient-descent/}

A model trained with pairwise preference losses still scores individual query-passage pairs independently at serving time. It doesn't need to evaluate all $O(k^2)$ passage pairs during inference.

Token-based scoring and decoder classification heads

monoT5 is an encoder-decoder model trained to predict true or false; its published scoring method renormalizes probabilities over those two target tokens.^{[5]Reference 5Document Ranking with a Pretrained Sequence-to-Sequence Model.https://arxiv.org/abs/2003.06713} Qwen3-Reranker uses a causal decoder and lowercase yes/no logits at the final prompt position.^{[6]Reference 6Qwen3-Reranker-0.6B model card and scoring examplehttps://huggingface.co/Qwen/Qwen3-Reranker-0.6B} An illustrative prompt might be:

Instruction: Decide whether the document satisfies the query conditions.
Query: legacy token endpoint for service account on day 10 with audit logging
Document: Rule AUTH-14. Service accounts may use the legacy token endpoint...
Relevant (Yes/No):

Use the checkpoint's actual template and tokenizer, not this sketch. For two single-token labels at the same prediction position, log-odds equal the difference between their logits:

Renormalizing over only these two labels gives:

This is not the full-vocabulary probability $P(\text{"Yes"})$: other possible tokens were removed from the denominator. Binary-label methods need the relevant forward computation, not a long generated explanation. An encoder-decoder also runs its encoder; don't equate its execution path with a causal decoder.

Other decoder-based scorers have different contracts. BGE-reranker-v2-gemma's example takes the single Yes logit, without subtracting No. RankLLaMA's published model example loads a sequence-classification model with one scalar output.^{[7]Reference 7BGE-reranker-v2-gemma model card and scoring examplehttps://huggingface.co/BAAI/bge-reranker-v2-gemma[8]Reference 8RankLLaMA v1 passage reranker model cardhttps://huggingface.co/castorini/rankllama-v1-7b-lora-passage} Neither should be described as the binary formula above.

As of September 21, 2026, Qwen3-Reranker provides 0.6B, 4B, and 8B checkpoints under Apache 2.0. Its card documents 32K context, while the example wrapper chooses max_length=8192; a wrapper can impose a smaller effective limit.^{[6]Reference 6Qwen3-Reranker-0.6B model card and scoring examplehttps://huggingface.co/Qwen/Qwen3-Reranker-0.6B} Jina Reranker v3.5 is a 0.6B listwise model with causal attention and cosine-based scoring, not the BERT-style head above. Its published weights use CC BY-NC 4.0; check deployment licensing before commercial use.^{[9]Reference 9Jina Reranker v3.5 model cardhttps://huggingface.co/jinaai/jina-reranker-v3.5}

Hosted Cohere Rerank accepts a query and candidate list and returns ranked items with relevance scores.^{[10]Reference 10Rerank API.https://docs.cohere.com/v2/reference/rerank} Test whether each score changes with the other candidates. Benchmark the actual checkpoint or service on your evidence and workload; measure latency, VRAM, and instruction following for the deployed family and settings.

Here are the losses for invented logits $z^+=1.2$ and $z^-=0.2$. This computes the objective, not a trained model's performance.

from math import exp, log1p

def softplus(x: float) -> float:
    return max(x, 0.0) + log1p(exp(-abs(x)))

positive, negative = 1.2, 0.2
pointwise_mean = (softplus(-positive) + softplus(negative)) / 2
pairwise_loss = softplus(-(positive - negative))
print(f"Pointwise mean loss: {pointwise_mean:.3f}")
print(f"Pairwise preference loss: {pairwise_loss:.3f}")
print(f"Preference loss after reversing scores: {softplus(positive - negative):.3f}")

Pointwise mean loss: 0.531
Pairwise preference loss: 0.313
Preference loss after reversing scores: 1.313

Training data must represent the ranking task. Hard negatives from the same first-stage retriever expose plausible near matches; random unrelated negatives can make the task too easy. Keep query and document leakage out of held-out evaluation, and check negation, dates, and no-answer questions separately from an overall average.

Preserve the text that was actually scored

Joint interaction also introduces a pair-length ceiling. Original BERT uses a maximum sequence length of 512 tokens, including special tokens.^{[2]Reference 2BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.https://arxiv.org/abs/1810.04805} Other checkpoints expose different limits, so read the tokenizer and model configuration instead of assuming 512.

If a wrapper silently truncates an overlong pair, the model may score only a prefix while the generator later receives the full chunk. An omitted clause could determine relevance or qualify a rule. Log tokenized pair length, define a truncation policy, and reject unexpected truncation so the score's scope is clear.

Truncation isn't only a local-model concern. Check a local wrapper's tokenizer configuration and truncation strategy. Cohere's Rerank API truncates documents to max_tokens_per_doc, default 4096.^{[10]Reference 10Rerank API.https://docs.cohere.com/v2/reference/rerank} Either way, ranked text can disagree with packed text. A sliding-window strategy can retain later clauses, but max-over-window aggregation can favor long documents and omit qualifications from other windows. Record the aggregation policy and keep necessary surrounding context.

The accounting below uses whitespace tokens only. It reserves three illustrative BERT special tokens. A real implementation must use the checkpoint's tokenizer and exact special-token count.

from hashlib import sha256
import json

@dataclass(frozen=True)
class PairBudget:
    max_pair_tokens: int
    query_tokens: list[str]
    chunk_tokens: list[str]
    special_tokens: int = 3

def tokenize(text: str) -> list[str]:
    return text.lower().split()

def truncate_document_tail(budget: PairBudget) -> tuple[list[str], bool]:
    """Keep the full query, then fill the remaining budget from the chunk head."""
    query = budget.query_tokens
    nonnegative_integer(budget.max_pair_tokens, "max pair tokens")
    nonnegative_integer(budget.special_tokens, "special tokens")
    if budget.max_pair_tokens == 0:
        raise ValueError("max pair tokens must be positive")
    remaining = budget.max_pair_tokens - len(query) - budget.special_tokens
    if remaining <= 0:
        raise ValueError("no document room after query and special tokens")
    kept_chunk = budget.chunk_tokens[:remaining]
    truncated = len(kept_chunk) < len(budget.chunk_tokens)
    return query + kept_chunk, truncated

def token_checksum(tokens: list[str]) -> str:
    serialized = json.dumps(tokens, ensure_ascii=False, separators=(",", ":"))
    return sha256(serialized.encode("utf-8")).hexdigest()

query = "May service accounts use the legacy token endpoint on day 10?"
# The authorizing clause is at the end; retaining the head drops it.
chunk = (
    "Background: migration tooling and temporary endpoints are under review. "
    "Operational notes discuss log shipping and dashboard ownership. "
    "Rule AUTH-14. Service accounts may use the legacy token endpoint "
    "within 14 days of deprecation when audit logging is enabled."
)
budget = PairBudget(
    max_pair_tokens=24,
    query_tokens=tokenize(query),
    chunk_tokens=tokenize(chunk),
)
scored_tokens, was_truncated = truncate_document_tail(budget)
packed_tokens = budget.query_tokens + budget.chunk_tokens  # generator sees full text
scored_checksum = token_checksum(scored_tokens)
packed_checksum = token_checksum(packed_tokens)
authorizing_clause = "within 14 days of deprecation when audit logging is enabled"
scored_text = " ".join(scored_tokens)
print("Truncated:", was_truncated)
print("Scored pair tokens including specials:", len(scored_tokens) + budget.special_tokens, "/", budget.max_pair_tokens)
print("Authorizing clause in scored text:", authorizing_clause in scored_text)
print("Checksum match (scored == packed):", scored_checksum == packed_checksum)
assert was_truncated
assert authorizing_clause not in scored_text
assert scored_checksum != packed_checksum
# This fixture's policy refuses any scored/packed mismatch.
release_ok = scored_checksum == packed_checksum
assert release_ok is False
print("Release gate (scored/packed mismatch):", release_ok)

Truncated: True
Scored pair tokens including specials: 24 / 24
Authorizing clause in scored text: False
Checksum match (scored == packed): False
Release gate (scored/packed mismatch): False

That failure is intentional: this lab refuses any scored/packed token mismatch. The checksum compares normalized toy token lists, not exact source bytes; lowercasing and whitespace splitting erase some differences. Real rerankers and generators can use different tokenizers, so record the scored source span and exact text separately from each model's token accounting. A parent expansion or explicit window score can be valid if its scope and aggregation are recorded.

With the text boundary explicit, test ranking independently of a model endpoint. The fixture below uses transparent phrase rules, not a trained transformer. Its integer scores exercise ordering and packing code. They aren't semantic entailment or authorization decisions.

@dataclass(frozen=True)
class Request:
    endpoint: str
    remedy: str
    days_since_deprecation: int
    audit_enabled: bool

def validate_request(request: Request) -> None:
    if any(
        not isinstance(value, str) or not value.strip()
        for value in (request.endpoint, request.remedy)
    ):
        raise ValueError("endpoint and remedy must be nonempty strings")
    nonnegative_integer(request.days_since_deprecation, "days since deprecation")
    if type(request.audit_enabled) is not bool:
        raise ValueError("audit enabled must be a Boolean")

REQUEST = Request(
    endpoint="legacy token endpoint",
    remedy="temporary access",
    days_since_deprecation=10,
    audit_enabled=True,
)
validate_request(REQUEST)
requirements = [
    REQUEST.endpoint,
    REQUEST.remedy,
    "day 10 is within the 14-day window",
    "audit logging is enabled",
]
print("Query conditions:", requirements)
assert 0 <= REQUEST.days_since_deprecation <= 14
assert REQUEST.audit_enabled

Query conditions: ['legacy token endpoint', 'temporary access', 'day 10 is within the 14-day window', 'audit logging is enabled']

The rule should rank first for this question. The scorer awards phrase-match points and subtracts six for the troubleshooting note's disclaimer. This deliberately narrow heuristic can't distinguish arbitrary paraphrases or resolve conflicting policies. The audit-export chunk gets timing points even though its 14-day window applies to a different action, a useful reminder of the heuristic's limits.

@dataclass(frozen=True)
class PairScore:
    candidate: Candidate
    score: float
    reasons: tuple[str, ...]

class ConstraintAwarePairScorer:
    def score(self, request: Request, candidate: Candidate) -> PairScore:
        text = candidate.text.lower()
        score = 0
        reasons: list[str] = []

        if request.endpoint.lower() in text:
            score += 2
            reasons.append("endpoint")
        if request.remedy.lower() in text:
            score += 3
            reasons.append("remedy")
        if "service accounts" in text:
            score += 1
            reasons.append("principal")
        if "within 14 days" in text and 0 <= request.days_since_deprecation <= 14:
            score += 2
            reasons.append("migration-window")
        if "audit logging is enabled" in text and request.audit_enabled:
            score += 2
            reasons.append("audit-condition")
        if "does not authorize temporary access" in text:
            score -= 6
            reasons.append("non-authorizing-note")
        return PairScore(candidate, score, tuple(reasons))

def rerank(
    request: Request,
    candidates: list[Candidate],
    scorer: ConstraintAwarePairScorer,
) -> list[PairScore]:
    validate_request(request)
    if len({candidate.chunk_id for candidate in candidates}) != len(candidates):
        raise ValueError("duplicate candidate IDs")
    if any(not in_snapshot(candidate) for candidate in candidates):
        raise ValueError("candidate outside authorized snapshot")
    scored: list[PairScore] = []
    for candidate in candidates:
        result = scorer.score(request, candidate)
        if (
            not isinstance(result, PairScore)
            or result.candidate != candidate
            or not in_snapshot(result.candidate)
            or not finite_number(result.score)
        ):
            raise ValueError("invalid score or changed candidate in scorer response")
        scored.append(result)
    return sorted(
        scored,
        key=lambda result: (-result.score, result.candidate.first_stage_rank, result.candidate.chunk_id),
    )

reranked = rerank(REQUEST, rerankable, ConstraintAwarePairScorer())
reranked_ids = [result.candidate.chunk_id for result in reranked]
for result in reranked:
    print(result.candidate.chunk_id, result.score, result.reasons)
assert reranked_ids[0] == TARGET_ID
assert rerank(REQUEST, [], ConstraintAwarePairScorer()) == []

api-token-legacy-v2-rule 7 ('endpoint', 'principal', 'migration-window', 'audit-condition')
api-audit-export-v1 2 ('migration-window',)
api-password-reset-v1 0 ()
api-token-troubleshooting-v1 -1 ('endpoint', 'remedy', 'non-authorizing-note')

Context selection now changes for the right reason. The candidate set stays fixed while the supported rule moves to rank 1. A context budget is a maximum, not a quota, so low-scoring near matches are not admitted merely to fill space.

Two chunks is only this lab's simplified budget. Production packing counts generator tokens after reserving instructions and output space, limits redundant passages from one parent document, and preserves clauses needed to interpret a rule. If packing expands a scored passage into a larger parent section, record that expansion explicitly rather than claiming every added sentence received the passage's score.

MIN_CONTEXT_SCORE = 5
selected_after = [
    result.candidate
    for result in reranked
    if result.score >= MIN_CONTEXT_SCORE
][:CONTEXT_BUDGET]
context_after = pack_context(selected_after, CONTEXT_BUDGET)
print("Context before:", context_before)
print("Context after:", context_after)
assert TARGET_ID in context_after
assert "api-token-troubleshooting-v1" not in context_after
assert set(reranked_ids) == set(first_stage_ids)

Context before: ['api-token-troubleshooting-v1', 'api-password-reset-v1']
Context after: ['api-token-legacy-v2-rule']

The fixture's threshold of 5 separates this rule from these three distractors. It isn't a permission rule. With audit disabled, the rule still scores 5 and remains relevant because it explains why access isn't allowed. A generator must apply the actual policy conditions, not interpret a high relevance score as an automatic affirmative. Scores that rank a denial highly can be completely correct.

A score of 7 beat these distractors. Would the same cutoff admit the right evidence on a query that has no good answer?

Score calibration and safe admission thresholds

Learned scores need calibration before interpreting them as probabilities. Define the event a probability represents, then test calibration on held-out labels; raw sigmoid values alone have no 85% answerability or 2:1 relevance interpretation.

Scores can vary with query type, candidate distribution, and checkpoint. Imagine a hypothetical no-answer query whose best distractor scores 0.90 and an answerable query whose best evidence scores 0.60. The shared threshold alone can't separate them. These numbers illustrate the problem; they aren't measured score ranges for broad or technical queries.

An unvalidated static threshold (such as score >= 0.70) can produce two failure modes:

1. Evidence starvation: Useful evidence below the threshold never reaches generation.
2. Noise admission: Irrelevant candidates above the threshold consume context.

Compare these admission strategies on labeled development cases, including no-answer cases:

• Rank-first with context budget: Take the highest-ranked items that fit. Strictly increasing score transformations preserve mathematical order. This avoids a fixed score scale but still selects distractors when every candidate is irrelevant; it isn't an abstention rule.
• Score margin falloff gating: Rather than filtering on an absolute score, filter candidates whose score falls significantly below the top candidate's score: $$\text{Keep chunk } i \iff s(q, d_i) \ge s(q, d_1) - \Delta_{\text{margin}}$$ A common additive shift leaves this rule unchanged; changing score scale generally doesn't. Nearby scores don't prove that sources agree or that any is relevant. Tune $\Delta_{\text{margin}}$ for the checkpoint and inspect admitted evidence.
• Calibrated admission or abstention: Platt scaling fits $P(y=1\mid z)=1/(1+e^{Az+B})$; isotonic regression fits a monotone mapping.^{[11]Reference 11Probability calibrationhttps://scikit-learn.org/stable/modules/calibration.html} Define $y$ carefully: passage relevance isn't the same event as having enough evidence for a complete answer. Fit calibration and choose thresholds on development data independent of scorer training, then evaluate reliability and abstention on an untouched test set. Revalidate when the checkpoint, tokenizer, scoring policy, or workload changes.

None of those rules tells you whether AUTH-14 moved into the two-passage budget. How do you measure that move on this request?

Measure candidate recall and top-of-list relevance

With pair scoring in place, ask how to prove it improves the stage that owns the failure. Keep evaluation narrow, and let each metric answer a different question:

1. Recall@candidate_k checks whether first-stage retrieval gave the reranker a chance.
2. Mean reciprocal rank (MRR) averages how early the first relevant chunk appears across a release suite.
3. Normalized discounted cumulative gain (NDCG) at the context cutoff checks whether relevant chunks fit near the top of the context budget.^{[12]Reference 12Introduction to Information Retrieval.https://nlp.stanford.edu/IR-book/}

For this one request, reciprocal rank (RR) is easy to read: rank 3 gives $1/3$; rank 1 gives $1$. MRR is the mean of those per-request values. NDCG supports graded relevance.

This lab uses binary relevance, where a relevant chunk has $rel_i = 1$ and an irrelevant chunk has $rel_i = 0$. With binary labels, linear-gain and exponential-gain DCG formulas agree, so the numbers below don't depend on which variant you pick:

Predict the fixture result before running it: moving the only relevant chunk from rank 3 to rank 1 should change RR from $1/3$ to $1$, and put the chunk inside NDCG@2.

RELEVANT_IDS = {TARGET_ID}

def reciprocal_rank(ids: list[str], relevant_ids: set[str]) -> float:
    if len(ids) != len(set(ids)):
        raise ValueError("duplicate ranked IDs")
    for rank, chunk_id in enumerate(ids, start=1):
        if chunk_id in relevant_ids:
            return 1.0 / rank
    return 0.0

def ndcg_at_k(ids: list[str], relevant_ids: set[str], k: int) -> float:
    nonnegative_integer(k, "NDCG cutoff")
    if len(ids) != len(set(ids)):
        raise ValueError("unique ranked IDs required")
    dcg = sum(
        (2 ** int(chunk_id in relevant_ids) - 1) / log2(rank + 1)
        for rank, chunk_id in enumerate(ids[:k], start=1)
    )
    ideal_hits = min(len(relevant_ids), k)
    idcg = sum((2**1 - 1) / log2(rank + 1) for rank in range(1, ideal_hits + 1))
    return dcg / idcg if idcg else 0.0

before_rr = reciprocal_rank(first_stage_ids, RELEVANT_IDS)
after_rr = reciprocal_rank(reranked_ids, RELEVANT_IDS)
before_ndcg = ndcg_at_k(first_stage_ids, RELEVANT_IDS, CONTEXT_BUDGET)
after_ndcg = ndcg_at_k(reranked_ids, RELEVANT_IDS, CONTEXT_BUDGET)
print(f"RR: {before_rr:.2f} -> {after_rr:.2f}")
print(f"NDCG@{CONTEXT_BUDGET}: {before_ndcg:.2f} -> {after_ndcg:.2f}")
assert after_rr > before_rr
assert before_ndcg == 0.0 and after_ndcg == 1.0

RR: 0.33 -> 1.00
NDCG@2: 0.00 -> 1.00

These numbers check this hand-labeled fixture, not model quality or generated-answer correctness. Deduplicate IDs before scoring: repeating a relevant chunk mustn't earn gain twice. This implementation defines NDCG as zero when there are no relevant labels; report that convention and evaluate no-answer behavior separately. Treat unjudged documents carefully rather than assuming every missing label means irrelevance.

For a held-out query suite, compare per-query scores on identical authorized candidates, then aggregate MRR/NDCG with uncertainty and query-type breakdowns. Also score the final packed context. Filtering, deduplication, and token budgets can remove a highly ranked passage after the reranker. Those ordering gains assume the target was on the list the reranker scored. What happens if that list stops at two?

Candidate count sets the ceiling and the bill

An ordering metric can't rescue an item that never reaches the reranker. If the reranker receives only the first two candidates, api-token-legacy-v2-rule is cut before pair scoring. Use a candidate-recall gate before comparing reranker models.

For relevant-set $R$ and candidate-set $C_k$, candidate recall is $|R\cap C_k|/|R|$. This value bounds the recall of any context selected from $C_k$. With two required passages but only one retrieved, even an oracle reranker can't exceed 50% passage recall. For multi-hop questions, measure whether every required passage is present.

Predict the budget tradeoff: top 2 is too small to score the target; top 3 gives it a chance. The lab keeps every other variable fixed so that ceiling is visible.

def target_present(candidates: list[Candidate]) -> bool:
    return TARGET_ID in [candidate.chunk_id for candidate in candidates]

def recall_at_k(ids: list[str], relevant_ids: set[str], k: int) -> float:
    nonnegative_integer(k, "candidate cutoff")
    if len(ids) != len(set(ids)):
        raise ValueError("unique candidate IDs required")
    return len(set(ids[:k]) & relevant_ids) / len(relevant_ids) if relevant_ids else 0.0

RERANK_CANDIDATE_BUDGET = 3
too_small = first_stage[:2]
release_input = first_stage[:RERANK_CANDIDATE_BUDGET]
limited_ids = [result.candidate.chunk_id for result in rerank(
    REQUEST, too_small, ConstraintAwarePairScorer()
)]
release_reranked = rerank(REQUEST, release_input, ConstraintAwarePairScorer())
release_reranked_ids = [result.candidate.chunk_id for result in release_reranked]
print("top-2 includes target:", target_present(too_small), limited_ids)
print("top-3 includes target:", target_present(release_input), release_reranked_ids)
print(f"Candidate recall@2: {recall_at_k(first_stage_ids, RELEVANT_IDS, 2):.1f}")
print(f"Candidate recall@3: {recall_at_k(first_stage_ids, RELEVANT_IDS, 3):.1f}")
assert not target_present(too_small)
assert release_reranked_ids[0] == TARGET_ID

top-2 includes target: False ['api-password-reset-v1', 'api-token-troubleshooting-v1']
top-3 includes target: True ['api-token-legacy-v2-rule', 'api-password-reset-v1', 'api-token-troubleshooting-v1']
Candidate recall@2: 0.0
Candidate recall@3: 1.0

Compute and latency budgeting in the two-stage funnel

A cross-encoder with independent pair scoring evaluates each query-candidate pair online. Increasing candidate depth increases total work, but wall-clock latency also depends on batch scheduling and hardware. For dense attention with fixed layer count $L$ and hidden width, the attention component scales as $O(LT^2)$ for pair length $T$. Doubling $T$ multiplies that component by roughly four, not necessarily the complete request latency.

Choose a retrieval shortlist by measuring the recall/latency frontier. There is no universal GPU millisecond range, CPU depth limit, or safe candidate count. Include actual query and passage lengths, queueing, concurrency, precision, model loading, and endpoint overhead in the measurement.

Batching also needs a concrete contract:

• Padding a batch to its longest sequence can waste work. Length bucketing reduces that waste, while waiting to form batches can add latency.
• Triton's default dynamic batching requires compatible input shapes. Ragged batching is optional configuration; the backend must handle concatenated inputs and their boundaries.^{[13]Reference 13Triton Inference Server: Ragged Batchinghttps://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/ragged_batching.html} Selecting a runtime doesn't automatically remove padding.
• FlashAttention improves attention memory access; it doesn't remove dense attention's quadratic arithmetic. Whether a particular reranker supports variable-length execution must be checked in its implementation.
• Compare smaller checkpoints, quantization, and different batching strategies on relevance and serving measurements. Recheck ranking quality after each speed optimization.

This fixture uses a sequential cost model to make the candidate-depth trade-off visible. It isn't a p95 benchmark: production serving batches pairs, and percentiles must be measured end to end. Cohere recommends against sending more than 1,000 documents in one Rerank request; that's a hosted-API cousin of the same $k$ knob.^{[10]Reference 10Rerank API.https://docs.cohere.com/v2/reference/rerank}

SEQUENTIAL_PAIR_COST_MS = 7.5
REQUEST_OVERHEAD_MS = 4.0
FIXTURE_LATENCY_BUDGET_MS = 30.0

def estimated_fixture_latency_ms(candidate_count: int) -> float:
    nonnegative_integer(candidate_count, "candidate count")
    if not finite_number(candidate_count):
        raise ValueError("candidate count is too large for this float cost model")
    latency = REQUEST_OVERHEAD_MS + SEQUENTIAL_PAIR_COST_MS * candidate_count
    if not finite_number(latency):
        raise ValueError("fixture latency overflow")
    return latency

release_fixture_latency = estimated_fixture_latency_ms(len(release_input))
too_wide_fixture_latency = estimated_fixture_latency_ms(len(first_stage))
print(
    f"top-3 fixture latency: {release_fixture_latency:.1f} ms, "
    f"pass={release_fixture_latency <= FIXTURE_LATENCY_BUDGET_MS}"
)
print(
    f"top-4 fixture latency: {too_wide_fixture_latency:.1f} ms, "
    f"pass={too_wide_fixture_latency <= FIXTURE_LATENCY_BUDGET_MS}"
)
assert release_fixture_latency <= FIXTURE_LATENCY_BUDGET_MS
assert too_wide_fixture_latency > FIXTURE_LATENCY_BUDGET_MS

top-3 fixture latency: 26.5 ms, pass=True
top-4 fixture latency: 34.0 ms, pass=False

For this fixture, three candidates recover the rule and fit the synthetic budget. The context cap is two, but only one passage clears the fixture threshold. A real service chooses these limits using held-out relevance labels and measured end-to-end p95 latency by candidate count, input length, concurrency, and batch strategy. With dense attention, a pair's attention work grows roughly quadratically in its token length; equal candidate counts can therefore have very different costs. Measure queueing, network, tokenization, model execution, and packing together.

Define failure behavior before serving: on a timeout, either fall back to the original authorized order and mark the trace as degraded, or abstain if policy requires reranking. Never fill the gap with unscored restricted evidence. On equal scores, this lab preserves first-stage rank and then uses chunk ID as a deterministic tie-breaker.

Depth decides how many chunks get a score. It doesn't decide when the query is allowed to meet the chunk. Can that comparison be stored ahead of the request, or does it have to wait until the query and the passage are read together?

Choose an interaction design deliberately

Anything computed without the query can be indexed. A score that needs this query and this chunk has to wait for request time. Predict that storage tradeoff before comparing the three designs below.

Three ranking designs make different choices across the interaction spectrum:

Bi-encoders compress a passage into one vector before comparing it with the query.^{[14]Reference 14Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.https://arxiv.org/abs/1908.10084} Encoding every condition into that representation can be challenging, but this doesn't mean a bi-encoder necessarily loses negation or entity roles. Those behaviors need evaluation on the actual checkpoint.

BERT-style cross-encoders let query and document tokens interact before the pooled score is computed. This interaction can capture fine distinctions. Measure the trained model on ambiguous pairs. Its output is still a compressed scalar.

ColBERT bridges the gap through late interaction.^{[15]Reference 15ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.https://arxiv.org/abs/2004.12832} It keeps document representations indexable offline as individual contextual token vectors, while preserving token-level comparison at query time. For each query token vector, the MaxSim operator finds its maximum dot product across all document token vectors, then sums those per-token maxima:

Original ColBERT L2-normalizes token embeddings, making token dot products equal cosine similarities.^{[15]Reference 15ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.https://arxiv.org/abs/2004.12832} Token dimension $m$ is a configuration choice, not always 128. PLAID is a search engine for ColBERTv2 representations that uses centroid interaction and pruning; its reported speedups depend on the configuration and workload.^{[16]Reference 16PLAID: An Efficient Engine for Late Interaction Retrievalhttps://arxiv.org/abs/2205.09707} Don't infer a universal sub-10-ms result.

View larger

Two query vectors find their best matches independently: $q_1$ matches $d_1$ at $0.9$, $q_2$ matches $d_2$ at $0.8$, and the score is their sum. These invented 2-D vectors all have unit length; $\sqrt{0.19}\approx0.436$ is chosen so $0.9^2+0.19=1$. The lab illustrates MaxSim, not embeddings from a trained ColBERT checkpoint.

from math import hypot, sqrt

query_vecs = [(1.0, 0.0), (0.0, 1.0)]
doc_vecs = [(0.9, sqrt(0.19)), (0.6, 0.8), (-0.8, -0.6)]

def dot(left: tuple[float, float], right: tuple[float, float]) -> float:
    return left[0] * right[0] + left[1] * right[1]

def maxsim(
    query: list[tuple[float, float]],
    document: list[tuple[float, float]],
) -> float:
    if not query or not document:
        raise ValueError("query and document need token vectors")
    for vector in query + document:
        if len(vector) != 2 or not all(finite_number(value) for value in vector):
            raise ValueError("this lab requires finite 2-D token vectors")
        if abs(hypot(*vector) - 1.0) > 1e-9:
            raise ValueError("this lab requires unit-length token vectors")
    return sum(max(dot(query_vec, doc_vec) for doc_vec in document) for query_vec in query)

score = maxsim(query_vecs, doc_vecs)
assert all(abs(dot(v, v) - 1.0) < 1e-9 for v in query_vecs + doc_vecs)
print(f"MaxSim: {score:.1f}")
assert abs(score - 1.7) < 1e-9

MaxSim: 1.7

Cache invalidation and multi-tenant security

Cross-encoder scores are expensive to recompute, which makes a cache attractive. Caching a score only by chunk ID, or only by (query, chunk_id), is a severe correctness and security failure.

Three changes make a key that thin wrong:

1. Document updates: If a policy document changes but keeps its chunk ID, an un-invalidated cache entry continues serving scores calculated against the obsolete version.
2. Model upgrades: A checkpoint or scoring-method change can shift score scales and ordering. Mixing incompatible cached scores can corrupt ranking.
3. Cross-tenant data leakage: In multi-tenant environments, Tenant A and Tenant B might issue identical queries. If Tenant A has access to a confidential contract and caches its high relevance score, Tenant B's retrieval pipeline must not see that score, receive that document ID, or learn of its existence through side-channel timing probes.

Fingerprint the exact scoring input, immutable model and tokenizer revisions, scoring settings, and trusted tenant/access namespace. Include instructions, templates, truncation or windows, aggregation, adapters, and relevant precision settings. A mutable model name isn't an immutable revision. Avoid normalizing away input distinctions unless the scorer deliberately uses that same normalization.

For an independently scored pair, the input contains one query and passage. For a listwise model such as Jina Reranker v3.5, it contains the complete ordered candidate list: changing a neighbor or its position can change the score.^{[9]Reference 9Jina Reranker v3.5 model cardhttps://huggingface.co/jinaai/jina-reranker-v3.5} Pairwise cache entries aren't interchangeable with whole-list results.

Use structured serialization rather than ambiguous string concatenation. This small fixture hashes already-rendered model input and explicit configuration. The revision strings below are invented teaching values, not real checkpoint pins.

def score_cache_key(
    model_input: str,
    *,
    model_revision: str,
    tokenizer_revision: str,
    scoring_settings: dict[str, object],
    tenant_id: str,
    security_scope: tuple[str, ...],
    grant_revision: str,
) -> str:
    strings = (model_input, model_revision, tokenizer_revision, tenant_id, grant_revision)
    if any(not isinstance(value, str) or not value for value in strings):
        raise ValueError("cache input and revisions must be nonempty strings")
    if (
        not isinstance(scoring_settings, dict)
        or not all(isinstance(key, str) for key in scoring_settings)
        or not isinstance(security_scope, tuple)
        or not all(isinstance(grant, str) and grant for grant in security_scope)
    ):
        raise ValueError("invalid scoring settings or security scope")
    payload = {
        "schema": 1,
        "input_sha256": sha256(model_input.encode("utf-8")).hexdigest(),
        "model_revision": model_revision,
        "tokenizer_revision": tokenizer_revision,
        "scoring_settings": scoring_settings,
        "tenant_id": tenant_id,
        "security_scope": sorted(set(security_scope)),
        "grant_revision": grant_revision,
    }
    try:
        serialized = json.dumps(
            payload, sort_keys=True, ensure_ascii=False,
            separators=(",", ":"), allow_nan=False,
        )
    except (TypeError, ValueError, OverflowError) as error:
        raise ValueError("cache configuration must be finite JSON data") from error
    return sha256(serialized.encode("utf-8")).hexdigest()

cache_config = {
    "model_revision": "fixture-model-revision-1",
    "tokenizer_revision": "fixture-tokenizer-revision-1",
    "scoring_settings": {"template": "fixture-v1", "truncation": "refuse"},
    "tenant_id": "policy-lab",
    "security_scope": ("read:api-auth-v2",),
    "grant_revision": "fixture-grants-1",
}
pair_input = json.dumps([QUERY, CANDIDATES[2].text])
pair_key = score_cache_key(pair_input, **cache_config)
changed_text_key = score_cache_key(pair_input + " revised", **cache_config)
other_tenant_key = score_cache_key(
    pair_input, **{**cache_config, "tenant_id": "other-fixture-tenant"}
)
list_input = json.dumps([QUERY, [candidate.text for candidate in CANDIDATES]])
reversed_list_input = json.dumps([QUERY, [candidate.text for candidate in reversed(CANDIDATES)]])
print("Changed text changes key:", pair_key != changed_text_key)
print("Changed tenant changes key:", pair_key != other_tenant_key)
print("Changed list order changes key:", score_cache_key(list_input, **cache_config)
      != score_cache_key(reversed_list_input, **cache_config))
assert pair_key != changed_text_key and pair_key != other_tenant_key
assert score_cache_key(list_input, **cache_config) != score_cache_key(reversed_list_input, **cache_config)

Changed text changes key: True
Changed tenant changes key: True
Changed list order changes key: True

The key identifies inputs. Authenticate callers, encrypt sensitive material, and validate source text separately. Derive access fields from trusted authorization, and recheck current access before exposing cached IDs, scores, or text, as well as before generation. Test timing side channels alongside cache isolation. Keep source revisions in cached result records for traceability as well.

A correct key still doesn't explain a bad answer. If the selected chunk is right and the reply is wrong, which record do you open?

Emit the trace that evaluation needs

Open the trace. Without evidence-level records, a bad answer could come from retrieval order, pair scoring, context packing, or source freshness.

Record first-stage IDs, actual reranker input, candidate source identity, policy versions, pair scores, model versions, selected context IDs, and release gates. A document version is part of correctness: a well-ranked stale rule is still unsafe context.

SCORER_VERSION = "phrase-rule-fixture-v1"
selected_context = [
    result.candidate
    for result in release_reranked
    if result.score >= MIN_CONTEXT_SCORE
][:CONTEXT_BUDGET]
release_trace = {
    "query_id": "api-token-legacy-access-001",
    "fixture_authority": {
        "tenant": "policy-lab",
        "caller": "service-account-reader",
        "grant_revision": "fixture-grants-1",
        "replay_date": "2026-05-27",
    },
    "versions": {
        "retriever": "policy-retriever-v2",
        "index": "policy-index/2026-05-27",
        "sparse": "bm25-tokenizer-v1",
        "dense": "fixture-embeddings-v1",
        "fusion": "rrf-k60",
        "reranker": SCORER_VERSION,
    },
    "first_stage_ids": first_stage_ids,
    "rerank_input_ids": [candidate.chunk_id for candidate in release_input],
    "reranked_ids": release_reranked_ids,
    "candidate_records": [
        {
            "chunk_id": result.candidate.chunk_id,
            "document_id": result.candidate.document_id,
            "parent_id": result.candidate.parent_id,
            "version": result.candidate.version,
            "first_stage_rank": result.candidate.first_stage_rank,
            "rerank_score": result.score,
            "reasons": result.reasons,
        }
        for result in release_reranked
    ],
    "selected_context_ids": [candidate.chunk_id for candidate in selected_context],
    "selected_versions": [candidate.version for candidate in selected_context],
    "gates": {
        "permitted_and_current": all(
            in_snapshot(candidate) for candidate in release_input
        ),
        "target_in_context": TARGET_ID in [
            candidate.chunk_id for candidate in selected_context
        ],
        "ordering_lift": ndcg_at_k(release_reranked_ids, RELEVANT_IDS, CONTEXT_BUDGET)
        > ndcg_at_k([candidate.chunk_id for candidate in release_input], RELEVANT_IDS, CONTEXT_BUDGET),
        "latency_budget": release_fixture_latency <= FIXTURE_LATENCY_BUDGET_MS,
    },
}
stores_raw_policy_text = any(
    candidate.text in str(release_trace)
    for candidate in SOURCE_STORE
)
print("Versions:", release_trace["versions"])
print("Rerank input:", release_trace["rerank_input_ids"])
print("Selected context:", release_trace["selected_context_ids"])
print("Trace stores raw policy text:", stores_raw_policy_text)
print("Gates:", release_trace["gates"])
assert set(release_trace["reranked_ids"]) == set(release_trace["rerank_input_ids"])
assert not stores_raw_policy_text
assert all(release_trace["gates"].values())

Versions: {'retriever': 'policy-retriever-v2', 'index': 'policy-index/2026-05-27', 'sparse': 'bm25-tokenizer-v1', 'dense': 'fixture-embeddings-v1', 'fusion': 'rrf-k60', 'reranker': 'phrase-rule-fixture-v1'}
Rerank input: ['api-token-troubleshooting-v1', 'api-password-reset-v1', 'api-token-legacy-v2-rule']
Selected context: ['api-token-legacy-v2-rule']
Trace stores raw policy text: False
Gates: {'permitted_and_current': True, 'target_in_context': True, 'ordering_lift': True, 'latency_budget': True}

The trace records the actual three-candidate input and compares ordering on that subset. Its authority labels describe a trusted teaching snapshot, not implemented authentication. Gold-target and ordering gates belong to offline evaluation, where labels exist. Live traces record IDs, versions, latency, and selection, but normally don't know the correct answer.

The substring check only detects complete fixture text. It misses fragments or encoded content and isn't a privacy audit. IDs, scores, and traces still require access controls.

Try to break the receipt

What if a scorer returns a different candidate with a very high score? Or someone edits text while preserving the ID and version? These tests exercise the actual boundary under score manipulation and text substitution. The final case also shows why a relevant rule can support a denial.

from dataclasses import replace

def rejected(action) -> bool:
    try:
        action()
    except ValueError:
        return True
    return False

class RebindingScorer(ConstraintAwarePairScorer):
    def score(self, request: Request, candidate: Candidate) -> PairScore:
        return PairScore(SOURCE_STORE[-2], 999.0, ("wrong-candidate",))

class BooleanScoreScorer(ConstraintAwarePairScorer):
    def score(self, request: Request, candidate: Candidate) -> PairScore:
        return PairScore(candidate, True, ())

changed_record = replace(CANDIDATES[2], text="Changed rule under unchanged ID/version")
assert rejected(lambda: rerank(REQUEST, [changed_record], ConstraintAwarePairScorer()))
assert rejected(lambda: rerank(REQUEST, CANDIDATES, RebindingScorer()))
assert rejected(lambda: rerank(REQUEST, CANDIDATES, BooleanScoreScorer()))
assert rejected(lambda: pack_context(CANDIDATES, True))
print("Changed record, rebound result, Boolean score/budget: rejected")

for days, audit in [(0, True), (14, True), (15, True), (10, False)]:
    request = replace(REQUEST, days_since_deprecation=days, audit_enabled=audit)
    target_score = next(
        result.score for result in rerank(request, CANDIDATES, ConstraintAwarePairScorer())
        if result.candidate.chunk_id == TARGET_ID
    )
    permitted_by_rule = 0 <= days <= 14 and audit
    print(f"day={days}, audit={audit}: score={target_score}, policy_allows={permitted_by_rule}")

Changed record, rebound result, Boolean score/budget: rejected
day=0, audit=True: score=7, policy_allows=True
day=14, audit=True: score=7, policy_allows=True
day=15, audit=True: score=5, policy_allows=False
day=10, audit=False: score=5, policy_allows=False

The fictional rule treats days 0 through 14 as inclusive and assumes the caller is a service account. These are this lab's interpretation rules, not general API policy. At day 15 or with auditing disabled, the chunk remains relevant and clears the +5 admission gate, while access is denied. The score selects evidence; interpreting that evidence is a separate operation.

Release checks for policy-answerer-v3

Use the gates as a diagnosis path. A missing target is a recall failure; a target below the context cutoff is an ordering failure; a score-text mismatch is a truncation failure; correct context with an unsupported answer belongs to the next generation and evaluation stage.

Before this reranker serves another legacy-token question:

Caching only by chunk ID is unsafe. Include the exact scoring input's checksum, immutable model/tokenizer revisions, scoring settings, and trusted access scope. For listwise scoring, input identity covers the full ordered list. Recheck access before disclosing a cache hit. Replay labeled cases after source changes. A high score for obsolete policy is still obsolete evidence.

A missing target is a recall miss, and a target below the cutoff is an ordering miss. The scored span has to match the packed text, and a cache key has to include tenant, revision, and list membership. A score of +7 can admit Rule AUTH-14 and still leave access denied on day 15. The generated sentence has to follow the selected rule, which the score does not check.