Personalize this lesson
Adapt explanations and teaching visuals to your background and preferred voice.
In this fictional policy lab, Maya asks what DEPLOY-17 requires for a payment-service production deploy during a release freeze. The recorded hybrid search candidates contain the rule, and the reranker selects it. Consider this hand-written answer:
The release-freeze rule applies. You can deploy without a linked rollback plan. [DEPLOY-17]
That answer sounds plausible because its citation names a rule in the fixture. It contradicts that rule: both incident commander (IC) approval and a linked rollback plan are required before rollout. This lab blocks release when any emitted policy claim lacks support.
In retrieval-augmented generation (RAG), retrieved text supplies evidence to a language model. The previous lesson's policy-answerer-v3 recorded an evidence trace showing which chunks were retrieved, selected, and tied to source versions. Use that trace alongside claim-level support checks to evaluate whether the answer stayed inside the evidence.
policy-answerer-v4-eval is the evaluation harness for this request. It reports the first failed check instead of averaging everything into one quality score. Its decisions use trusted fixture labels, not a neural entailment model. Those gates still matter when a judge later scores paraphrases the labels can't cover.
The RAG Triad groups three useful measurements: context relevance, groundedness (which RAGAS calls faithfulness), and answer relevance.[1] Map those measurements to your product's risks, then add checks for access, citations, and completeness.
- Context relevance: Is supplied context relevant to the question?
- Groundedness / faithfulness: Are the answer's claims supported by supplied evidence? Claim extraction and support assessment can both make mistakes; validate high-scoring outputs against human-labeled examples and representative cases.[2]
- Answer relevance: Does the response address the question? Topic relevance alone doesn't certify every required condition, factual correctness, or task completion.
For this policy explanation, we add access, citation, and required-point checks alongside the Triad:
- Admissibility and authorization: Did every chunk in the evidence trace pass access control (such as Fine-Grained Authorization) and freshness checks before reaching the model?
- Context selection: Did hybrid search retrieve the gold rule, and did reranking preserve it within the admitted context budget?
- Claim faithfulness: Does the admitted context logically entail every atomic claim asserted by the model?
- Citation support: Does each attached citation link to an admitted chunk that actually establishes that specific claim?
- Task completeness: Did the model cover every required policy point instead of silently dropping mandatory conditions?
A fluent answer can pass one gate and fail another. Faithfulness measures support from the supplied evidence. A faithful answer can repeat an incorrect source, and a true fact from pre-training can lack support in the request's context. A rule that requires approval and a rollback plan also leaves deployment subject to the policy's other conditions. This lesson evaluates a policy explanation. Execution still requires its own authorization.
Replay the freeze-deploy trace
Maya's question for this replay is:
What does DEPLOY-17 require for a payment-service production deploy during a release freeze?
The fixture starts after hybrid retrieval found the correct rule and reranking selected it. Evaluation consumes that recorded evidence path instead of reconstructing retrieval inside the answer-quality harness. The field names match the previous lesson's release trace: first_stage_ids, reranked_ids, selected_context_ids, and component versions.
Before running the fixture, predict which chunk ID should survive in selected_context_ids. Check that tuple, its source version, and the component versions against the printed trace.
from __future__ import annotations
from collections import defaultdict
from dataclasses import dataclass, replace
from types import MappingProxyType
TARGET_ID = "deploy-freeze-approval-rule"
@dataclass(frozen=True)
class EvidenceChunk:
chunk_id: str
document_id: str
parent_id: str
version: str
permitted: bool
current: bool
text: str
@dataclass(frozen=True)
class GoldCase:
case_id: str
question: str
required_source_ids: frozenset[str]
required_points: frozenset[str]
@dataclass(frozen=True)
class RagTrace:
case_id: str
question: str
first_stage_ids: tuple[str, ...]
rerank_input_ids: tuple[str, ...]
reranked_ids: tuple[str, ...]
selected_context_ids: tuple[str, ...]
selected_versions: tuple[str, ...]
versions: tuple[tuple[str, str], ...]
EVIDENCE = {
TARGET_ID: EvidenceChunk(
TARGET_ID,
"deploy-policy",
"deploy-policy-v2",
"deploy-policy/2026-06-01",
True,
True,
(
"Rule DEPLOY-17. Payment-service production deploys during a release "
"freeze require incident commander approval and a linked rollback plan "
"before rollout."
),
),
"payment-service-rollback-runbook": EvidenceChunk(
"payment-service-rollback-runbook",
"payment-rollback",
"payment-rollback-v1",
"payment-rollback/2026-05-20",
True,
True,
(
"Payment-service rollback drills must keep the previous artifact "
"available for 30 minutes. This runbook does not authorize freeze deploys."
),
),
"frontend-docs-deploy-rule": EvidenceChunk(
"frontend-docs-deploy-rule",
"frontend-docs",
"frontend-docs-v1",
"frontend-docs/2026-03-01",
True,
True,
"Frontend documentation deploys may ship during normal business hours without freeze approval.",
),
"restricted-breakglass-note": EvidenceChunk(
"restricted-breakglass-note",
"breakglass-terms",
"breakglass-terms",
"breakglass/2026-05-01",
False,
True,
"Payment-service deploys may bypass freeze approval during executive escalations.",
),
}
# Trusted replay records. Frozen records plus a read-only mapping protect this
# in-memory fixture; a real service needs versioned, access-controlled storage.
REPLAY_SNAPSHOT = MappingProxyType(dict(EVIDENCE))
GOLD = GoldCase(
case_id="payment-freeze-deploy-001",
question=(
"What does DEPLOY-17 require for a payment-service production deploy "
"during a release freeze?"
),
required_source_ids=frozenset({TARGET_ID}),
required_points=frozenset({"freeze-scope", "approval", "rollback-plan"}),
)
TRACE = RagTrace(
case_id=GOLD.case_id,
question=GOLD.question,
first_stage_ids=(
"payment-service-rollback-runbook",
"frontend-docs-deploy-rule",
TARGET_ID,
),
rerank_input_ids=(
"payment-service-rollback-runbook",
"frontend-docs-deploy-rule",
TARGET_ID,
),
reranked_ids=(TARGET_ID, "payment-service-rollback-runbook", "frontend-docs-deploy-rule"),
selected_context_ids=(TARGET_ID,),
selected_versions=("deploy-policy/2026-06-01",),
versions=(
("retriever", "policy-retriever-v2"),
("index", "policy-index/2026-06-02"),
("sparse", "bm25-tokenizer-v1"),
("dense", "fixture-embeddings-v1"),
("fusion", "rrf-k60"),
("reranker", "fixture-cross-encoder-v1"),
),
)
selected_sources = [
(
chunk.chunk_id,
chunk.document_id,
chunk.parent_id,
chunk.version,
)
for chunk in (EVIDENCE[chunk_id] for chunk_id in TRACE.selected_context_ids)
]
print("Selected context:", TRACE.selected_context_ids)
print("Selected sources:", selected_sources)
print("Pipeline versions:", dict(TRACE.versions))
assert TRACE.selected_context_ids == (TARGET_ID,)Selected context: ('deploy-freeze-approval-rule',)
Selected sources: [('deploy-freeze-approval-rule', 'deploy-policy', 'deploy-policy-v2', 'deploy-policy/2026-06-01')]
Pipeline versions: {'retriever': 'policy-retriever-v2', 'index': 'policy-index/2026-06-02', 'sparse': 'bm25-tokenizer-v1', 'dense': 'fixture-embeddings-v1', 'fusion': 'rrf-k60', 'reranker': 'fixture-cross-encoder-v1'}The trace identifies the sources and retrieval configuration to inspect. This in-memory fixture has one immutable revision per chunk. A real replay store must resolve the recorded revision, not substitute today's text. To reproduce generation, also save the exact assembled prompt, generator version, decoding settings, and original output. Even those records don't promise identical resampling from a hosted model.
The trace names the right chunk and the right versions. Were those candidates allowed to be on the path at all?
Gate admissibility before scoring quality
A relevance score can't make forbidden evidence acceptable. Here, first_stage_ids means candidates returned across the request's authorization boundary, not every record an index scans internally. Restricted, stale, or unknown evidence in that returned path fails this fixture's contract, even if selection later drops it. permitted and current are trusted test labels; these booleans don't implement access control or freshness checks.
def admissible_evidence_path(trace: RagTrace, gold: GoldCase) -> bool:
if trace.case_id != gold.case_id or trace.question != gold.question:
return False
if not trace.selected_context_ids:
return False
if len(trace.selected_context_ids) != len(trace.selected_versions):
return False
required_version_keys = {"retriever", "index", "sparse", "dense", "fusion", "reranker"}
if any(
not isinstance(record, tuple)
or len(record) != 2
or any(not isinstance(value, str) or not value.strip() for value in record)
for record in trace.versions
):
return False
version_map = dict(trace.versions)
if len(version_map) != len(trace.versions):
return False
if not required_version_keys.issubset(version_map):
return False
if any(not version_map[key].strip() for key in required_version_keys):
return False
stage_ids = (
trace.first_stage_ids,
trace.rerank_input_ids,
trace.reranked_ids,
trace.selected_context_ids,
)
if any(len(ids) != len(set(ids)) for ids in stage_ids):
return False
traced_ids = tuple(chunk_id for ids in stage_ids for chunk_id in ids)
if any(chunk_id not in EVIDENCE for chunk_id in traced_ids):
return False
traced = [EVIDENCE[chunk_id] for chunk_id in traced_ids]
selected = [EVIDENCE[chunk_id] for chunk_id in trace.selected_context_ids]
rerank_input_came_from_retrieval = set(trace.rerank_input_ids).issubset(
trace.first_stage_ids
)
reranked_same_candidates = set(trace.reranked_ids) == set(trace.rerank_input_ids)
returned_by_reranker = set(trace.selected_context_ids).issubset(trace.reranked_ids)
versions_match = all(
chunk.version == version
for chunk, version in zip(selected, trace.selected_versions)
)
allowed_and_current = all(
chunk.permitted is True
and chunk.current is True
and REPLAY_SNAPSHOT.get(chunk.chunk_id) == chunk
for chunk in traced
)
return (
rerank_input_came_from_retrieval
and reranked_same_candidates
and returned_by_reranker
and versions_match
and allowed_and_current
)
restricted_trace = replace(
TRACE,
first_stage_ids=("restricted-breakglass-note",),
rerank_input_ids=("restricted-breakglass-note",),
reranked_ids=("restricted-breakglass-note",),
selected_context_ids=("restricted-breakglass-note",),
selected_versions=("breakglass/2026-05-01",),
)
blocked_candidate_trace = replace(
TRACE,
first_stage_ids=TRACE.first_stage_ids + ("restricted-breakglass-note",),
rerank_input_ids=TRACE.rerank_input_ids + ("restricted-breakglass-note",),
reranked_ids=TRACE.reranked_ids + ("restricted-breakglass-note",),
)
unknown_candidate_trace = replace(TRACE, first_stage_ids=TRACE.first_stage_ids + ("missing",))
stale_version_trace = replace(TRACE, selected_versions=("deploy-policy/2025-02-01",))
missing_version_trace = replace(TRACE, versions=TRACE.versions[:-1])
wrong_case_trace = replace(TRACE, case_id="payment-freeze-deploy-002")
duplicate_candidate_trace = replace(
TRACE,
first_stage_ids=TRACE.first_stage_ids + (TARGET_ID,),
)
print("Fixture path admissible:", admissible_evidence_path(TRACE, GOLD))
print("Restricted context admissible:", admissible_evidence_path(restricted_trace, GOLD))
print("Blocked candidate admissible:", admissible_evidence_path(blocked_candidate_trace, GOLD))
print("Unknown candidate admissible:", admissible_evidence_path(unknown_candidate_trace, GOLD))
print("Wrong-version path admissible:", admissible_evidence_path(stale_version_trace, GOLD))
print("Incomplete trace admissible:", admissible_evidence_path(missing_version_trace, GOLD))
print("Wrong-case trace admissible:", admissible_evidence_path(wrong_case_trace, GOLD))
print("Duplicate candidate admissible:", admissible_evidence_path(duplicate_candidate_trace, GOLD))
assert admissible_evidence_path(TRACE, GOLD)
assert not admissible_evidence_path(restricted_trace, GOLD)
assert not admissible_evidence_path(blocked_candidate_trace, GOLD)
assert not admissible_evidence_path(unknown_candidate_trace, GOLD)
assert not admissible_evidence_path(stale_version_trace, GOLD)
assert not admissible_evidence_path(missing_version_trace, GOLD)
assert not admissible_evidence_path(wrong_case_trace, GOLD)
assert not admissible_evidence_path(duplicate_candidate_trace, GOLD)Fixture path admissible: True
Restricted context admissible: False
Blocked candidate admissible: False
Unknown candidate admissible: False
Wrong-version path admissible: False
Incomplete trace admissible: False
Wrong-case trace admissible: False
Duplicate candidate admissible: FalseThe case and question checks bind these labels to this replay; they don't establish that the recorded text is the caller's actual request or authenticate that caller. Complete record comparisons reject edited source text, parents, or metadata under an unchanged ID/version. The component names are teaching version labels, not immutable artifact pins. A service also needs trusted caller/grant records and current access checks.
The path can pass those checks and still be the wrong evidence for the answer. Is the required rule in the candidate list, and did it survive into the prompt?
A restricted break-glass rule would answer the question perfectly. Why must the evaluation fail before computing relevance?
Answer
Authorization determines which evidence Maya may use. Once restricted text enters retrieval or reranking, a score, cache, trace, or response can expose evidence Maya isn't allowed to use. Authorization and freshness are hard gates across the full evidence path.
Separate candidates from selected context
Admissibility establishes a permitted, current evidence path. Next, measure whether retrieval found the required rule and reranking kept it inside the prompt.
Context relevance asks whether supplied text is about the question. Evidence sufficiency also needs a coverage check. We inspect candidate retrieval and selected context separately.
Pause on that distinction: if the rule appears in first-stage candidates but disappears before the prompt, which recall should fall? Keep that prediction in mind while reviewing the metrics:
| Layer | Gate | Meaning on this trace |
|---|---|---|
| Candidate retrieval | candidate_recall | Did hybrid search pull the required rule into initial candidates? |
| Selected context | context_recall | Did the rule survive reranking and token budget admission? |
| Selected context | selected_context_precision | Did admitted context filter out irrelevant distractors? |
Selected-context precision directly measures noise reduction:
Distractors can increase context cost and introduce competing evidence. Lost in the Middle found position-dependent degradation on its tested tasks and models, especially when relevant information appeared in the middle of long contexts.[3] Extra irrelevant chunks don't automatically trigger that failure in every model; test position and distraction separately.
Ranking metrics such as mean reciprocal rank (MRR) and normalized discounted cumulative gain (NDCG) ask whether top evidence appears early in a ranked list.[4] At generation evaluation time, we care about admission: did the evidence make it into the prompt, and how clean is the prompt?
def coverage(ids: tuple[str, ...], required_ids: frozenset[str]) -> float:
if not required_ids:
raise ValueError("Source recall needs a nonempty reference set")
if len(ids) != len(set(ids)):
raise ValueError("Source recall needs unique admitted IDs")
return len(set(ids) & required_ids) / len(required_ids)
def selected_context_precision(ids: tuple[str, ...], useful_ids: frozenset[str]) -> float:
"""ID-set precision, not rank-weighted Ragas ContextPrecision."""
if len(ids) != len(set(ids)):
raise ValueError("Selected precision needs unique admitted IDs")
return len(set(ids) & useful_ids) / len(ids) if ids else 0.0
candidate_recall = coverage(TRACE.first_stage_ids, GOLD.required_source_ids)
context_recall = coverage(TRACE.selected_context_ids, GOLD.required_source_ids)
selected_precision = selected_context_precision(
TRACE.selected_context_ids, GOLD.required_source_ids
)
print(f"Candidate recall: {candidate_recall:.1f}")
print(f"Selected-context recall: {context_recall:.1f}")
print(f"Selected-context precision: {selected_precision:.1f}")
assert (candidate_recall, context_recall, selected_precision) == (1.0, 1.0, 1.0)Candidate recall: 1.0
Selected-context recall: 1.0
Selected-context precision: 1.0
This result describes the evidence path. Inspect the generated answer separately. These are set metrics over labeled chunk IDs; the original RAGAS context-relevance metric uses a different method.[2] An alternative chunk could contain equally good evidence yet score as a miss against an incomplete gold set. Label acceptable alternatives or assess coverage of answer-bearing facts before treating that miss as a retrieval defect.
Metric names can hide different denominators. Current Ragas documents a rank-weighted ContextPrecision and a separate IDBasedContextPrecision using the matching-ID fraction.[5] Our unique-ID precision follows the latter idea. With one useful chunk and two distractors, set precision is regardless of order. The rank-weighted metric can change when the useful chunk moves.
Context can be sufficient and the written answer can still be wrong. How do you score the sentences themselves?
Turn an answer into testable claims
Consider two responses produced from the same valid context:
| Response | What changed? |
|---|---|
unsafe-bypass | Adds a no-rollback-plan claim that the deploy rule never states |
supported-deploy | States only the deploy scope and conditions present in DEPLOY-17 |
Context can be sufficient while the answer isn't. Split the answer into atomic policy claims so you can score it. Each claim below carries its text and citation. A separate, trusted annotation will say what the evidence supports. FActScore similarly evaluates factual precision at the level of atomic facts rather than treating a whole passage as uniformly correct.[6]
Before looking at the output, decide which answer should score lower. Both use the same selected context, but the bypass replaces required conditions with an unsupported exemption; count supported claims, not citation count or answer length.
@dataclass(frozen=True)
class Claim:
claim_id: str
text: str
citation_id: str | None
@dataclass(frozen=True)
class Answer:
answer_id: str
claims: tuple[Claim, ...]
UNSAFE_BYPASS = Answer(
"unsafe-bypass",
(
Claim(
"freeze-scope",
"The request is governed by the release-freeze deploy rule.",
TARGET_ID,
),
Claim(
"bypass",
"The deploy can start without a linked rollback plan.",
TARGET_ID,
),
),
)
SUPPORTED_DEPLOY = Answer(
"supported-deploy",
(
Claim(
"freeze-scope",
"The request is governed by the release-freeze deploy rule.",
TARGET_ID,
),
Claim(
"approval",
"Incident commander approval is required before rollout.",
TARGET_ID,
),
Claim(
"rollback-plan",
"A linked rollback plan is required before rollout.",
TARGET_ID,
),
),
)
PARTIAL_DEPLOY = Answer(
"partial-deploy",
(
Claim(
"freeze-scope",
"The request is governed by the release-freeze deploy rule.",
TARGET_ID,
),
Claim(
"approval",
"Incident commander approval is required before rollout.",
TARGET_ID,
),
),
)
EMPTY_ANSWER = Answer("empty", ())
print("Unsafe claims:", [claim.claim_id for claim in UNSAFE_BYPASS.claims])
print("Supported claims:", [claim.claim_id for claim in SUPPORTED_DEPLOY.claims])
print("Partial claims:", [claim.claim_id for claim in PARTIAL_DEPLOY.claims])
print("Empty claims:", [claim.claim_id for claim in EMPTY_ANSWER.claims])Unsafe claims: ['freeze-scope', 'bypass']
Supported claims: ['freeze-scope', 'approval', 'rollback-plan']
Partial claims: ['freeze-scope', 'approval']
Empty claims: []These claims are hand-written test fixtures, not outputs from a model call. In live evaluation, inspect claim extraction too: dropping a qualifier such as "only" or skipping an unsupported sentence can inflate every later score.
Faithfulness checks claims against context
Selected context can still lose to a generator that outruns it. Check each claim against the selected DEPLOY-17 sentence before writing a scorer:
| Claim | Evidence relationship | Supported? |
|---|---|---|
freeze-scope | The rule explicitly covers this service and freeze | yes |
bypass | The rule requires the plan; the answer says it can be absent | no |
One of two unsafe claims is supported, so faithfulness is . RAGAS formalizes this ratio after an LLM extracts atomic statements and evaluates which ones are logically inferred from context.[2]
Verifying whether context supports claim is formally modeled as a Natural Language Inference (NLI) task. Treating the admitted context as the premise and atomic claim as the hypothesis, the evaluator (an NLI cross-encoder or calibrated LLM judge) assigns one of three categorical relations:
- Entailment (): The context logically necessitates the claim, including all scopes and conditions.
- Contradiction (): The context directly conflicts with the claim. Here, DEPLOY-17 requires a linked rollback plan; the answer asserts the plan can be absent.
- Neutral / Not Established (): The context doesn't contain enough evidence to confirm or refute the claim (such as speculating about unmentioned executive waivers).
A learned evaluator estimates these relations; its verdict isn't a formal proof. The claim-level faithfulness metric is the fraction of emitted claims assessed as entailed by the admitted context:
where is the total count of atomic claims in the response.
Token overlap alone can't distinguish support here: "approval is required" and "approval is not required" share most of their tokens while stating opposite rules. Entailment requires agreement with scope and conditions, not lexical similarity alone.
The code uses a hand-labeled teaching answer key. Each supported text maps to an exact source revision and a required point. An unknown text is unadjudicated, not automatically false. This fixture refuses to certify it until reviewed. In a real study, independently review the labels; never let the answer supply its own support verdict.
REVIEWED_SUPPORT = {
"The request is governed by the release-freeze deploy rule.": (
TARGET_ID, "deploy-policy/2026-06-01", "freeze-scope"
),
"Incident commander approval is required before rollout.": (
TARGET_ID, "deploy-policy/2026-06-01", "approval"
),
"A linked rollback plan is required before rollout.": (
TARGET_ID, "deploy-policy/2026-06-01", "rollback-plan"
),
}
REVIEWED_UNSUPPORTED = {"The deploy can start without a linked rollback plan."}
def source_supports(claim: Claim, chunk: EvidenceChunk) -> bool:
annotation = REVIEWED_SUPPORT.get(claim.text)
return (
annotation is not None
and annotation[:2] == (chunk.chunk_id, chunk.version)
and REPLAY_SNAPSHOT.get(chunk.chunk_id) == chunk
)
def claim_supported_by_context(claim: Claim, trace: RagTrace) -> bool:
return any(
source_supports(claim, EVIDENCE[chunk_id])
for chunk_id in trace.selected_context_ids
)
def faithfulness(answer: Answer, trace: RagTrace) -> float:
if any(
claim.text not in REVIEWED_SUPPORT and claim.text not in REVIEWED_UNSUPPORTED
for claim in answer.claims
):
raise ValueError("Unadjudicated claim: review before scoring")
supported = sum(
claim_supported_by_context(claim, trace) for claim in answer.claims
)
return supported / len(answer.claims) if answer.claims else 0.0
print(f"unsafe-bypass faithfulness: {faithfulness(UNSAFE_BYPASS, TRACE):.2f}")
print(
"supported-deploy faithfulness: "
f"{faithfulness(SUPPORTED_DEPLOY, TRACE):.2f}"
)
print(f"empty faithfulness: {faithfulness(EMPTY_ANSWER, TRACE):.2f}")
assert faithfulness(UNSAFE_BYPASS, TRACE) == 0.5
assert faithfulness(SUPPORTED_DEPLOY, TRACE) == 1.0
assert faithfulness(EMPTY_ANSWER, TRACE) == 0.0unsafe-bypass faithfulness: 0.50
supported-deploy faithfulness: 1.00
empty faithfulness: 0.00The unsafe response is on topic and cites a real selected rule. It still fails because one claim contradicts that rule. The empty-claims score of 0.0 is a deliberate fixture convention, not a zero-denominator ratio. Current Ragas also defines faithfulness through extracted-claim support; its extraction and support models can err.[7] This answer key binds each claim to one source. A claim requiring several sources needs an evidence-set annotation or another evaluator. An empty score is not a justified refusal.
The citation is the remaining question. Does the attached chunk ID establish the sentence, or only name a document?
Verify that each citation supports its claim
Faithfulness asks whether selected context could support the sentence. Citation support asks whether the attached source is the one that does. RAGAS faithfulness doesn't check citation IDs, so this gate is extra.
A citation metric needs two checks:
- Coverage: Does every policy claim cite a source?
- Support: Does the cited selected source establish that claim?
A fabricated rollback exemption can get perfect citation coverage by attaching the correct-looking chunk ID. Presence alone is a weak gate.
Before running the next cell, predict what changes for MIS_CITED: faithfulness, citation support, or both.
MIS_CITED = Answer(
"mis-cited",
tuple(
replace(claim, citation_id="payment-service-rollback-runbook")
for claim in SUPPORTED_DEPLOY.claims
),
)
def citation_coverage(answer: Answer) -> float:
cited = sum(
isinstance(claim.citation_id, str) and bool(claim.citation_id.strip())
for claim in answer.claims
)
return cited / len(answer.claims) if answer.claims else 0.0
def citation_support(answer: Answer, trace: RagTrace) -> float:
supported = 0
for claim in answer.claims:
if claim.citation_id not in trace.selected_context_ids:
continue
if source_supports(claim, EVIDENCE[claim.citation_id]):
supported += 1
return supported / len(answer.claims) if answer.claims else 0.0
print(f"unsafe citation coverage: {citation_coverage(UNSAFE_BYPASS):.2f}")
print(f"unsafe citation support: {citation_support(UNSAFE_BYPASS, TRACE):.2f}")
print(f"mis-cited answer faithfulness: {faithfulness(MIS_CITED, TRACE):.2f}")
print(f"mis-cited citation support: {citation_support(MIS_CITED, TRACE):.2f}")
assert citation_coverage(UNSAFE_BYPASS) == 1.0
assert citation_support(UNSAFE_BYPASS, TRACE) == 0.5
assert faithfulness(MIS_CITED, TRACE) == 1.0
assert citation_support(MIS_CITED, TRACE) == 0.0unsafe citation coverage: 1.00
unsafe citation support: 0.50
mis-cited answer faithfulness: 1.00
mis-cited citation support: 0.00In the mis-cited response, selected context supports every sentence, so faithfulness is 1.0. Its citations still fail because they point at payment-service-rollback-runbook, which is not selected context and does not support those claims. Adding that unrelated runbook to context would leave its citations irrelevant to Maya's question.
An answer gets faithfulness 1.0 but citation support 0.0. What happened?
Answer
The answer may match evidence somewhere in context while its attached citations point somewhere else, or outside the admitted context entirely. Inspect claim-to-citation alignment rather than changing retrieval first.
A cited sentence can be fully supported and still omit a condition Maya needs. Did the answer finish the policy?
Check whether the answer finished the task
A 1.0 faithfulness score can still fail release if a required point never appears as a supported claim. For this golden case, Maya needs three answer points: deploy scope, approval, and rollback plan. Treat point coverage as a labeled completeness gate:
def supported_point_coverage(answer: Answer, trace: RagTrace, gold: GoldCase) -> float:
if not gold.required_points:
raise ValueError("Point coverage needs nonempty required points")
supported_points = {
REVIEWED_SUPPORT[claim.text][2]
for claim in answer.claims
if claim_supported_by_context(claim, trace)
}
return len(supported_points & gold.required_points) / len(gold.required_points)
unsafe_coverage = supported_point_coverage(UNSAFE_BYPASS, TRACE, GOLD)
supported_coverage = supported_point_coverage(SUPPORTED_DEPLOY, TRACE, GOLD)
partial_coverage = supported_point_coverage(PARTIAL_DEPLOY, TRACE, GOLD)
print(f"unsafe supported point coverage: {unsafe_coverage:.2f}")
print(f"supported answer point coverage: {supported_coverage:.2f}")
print(f"partial supported point coverage: {partial_coverage:.2f}")
assert unsafe_coverage == 1 / 3
assert supported_coverage == 1.0
assert partial_coverage == 2 / 3unsafe supported point coverage: 0.33
supported answer point coverage: 1.00
partial supported point coverage: 0.67Point coverage does a different job from answer relevance. A response can sound on-topic while omitting the rollback condition; RAGAS answer relevance asks whether the response addresses the question, while this labeled gate checks whether each required policy point appears.[2]
PARTIAL_DEPLOY is the useful contrast: faithfulness is 1.0 while point coverage is only because the rollback-plan claim never appears. Conversely, a correctly quoted frontend deployment rule could be fully supported yet irrelevant to Maya's payment-service question. The required points are a task-specific relevance and completeness rubric, not a general semantic relevance detector.
Before checking the partial answer's output, predict whether its missing rollback point should be blamed on faithfulness or completeness.
An answer emits no policy claims. Does its lack of unsupported claims make it releasable?
Answer
Not on this answerable case. Silence doesn't invent policy, but it doesn't finish Maya's task either. Block it as incomplete. A clear, justified abstention on an unanswerable question is a different behavior and needs a different label.
When declining to answer is correct
Suppose Maya asks whether an executive escalation waives the rollback requirement. The admitted rule contains no such exception. "The available policy lists no waiver; ask the policy owner" is a useful abstention. "No waiver exists anywhere" is an unsupported universal claim.
Label both corpus answerability and selected-evidence sufficiency. If the corpus contains the needed rule but retrieval misses it, abstaining can be the safe generation behavior while retrieval still fails. Report that distinction instead of rewarding every refusal or forcing every question into an answer-required gate.
| Corpus can answer? | Selected evidence sufficient? | Response | Interpretation |
|---|---|---|---|
| yes | yes | Complete, supported answer | Task success |
| no | no | Explain the evidence limit | Justified abstention |
| yes | no | Explain the evidence limit | Safe response, retrieval/selection failure |
| yes | yes | Refuse without a reason | Unnecessary abstention |
Replay that distinction with five reviewed outcomes. corpus_answerable is an annotation about the permitted, current corpus for this caller, not everything the organization stores. context_sufficient describes the evidence actually admitted. The response labels come from a reviewer; a model cannot certify its own refusal merely by calling it an abstention.
from dataclasses import dataclass
@dataclass(frozen=True)
class ReviewedOutcome:
name: str
corpus_answerable: bool
context_sufficient: bool
response: str
def outcome_labels(case: ReviewedOutcome) -> tuple[bool, bool, bool]:
if case.context_sufficient and not case.corpus_answerable:
raise ValueError("Sufficient context contradicts the corpus label")
if case.response not in {"complete-supported", "bounded-abstention", "unsupported"}:
raise ValueError("Response needs independent review")
supported_answer = case.response == "complete-supported" and case.context_sufficient
safe_response = supported_answer or case.response == "bounded-abstention"
evidence_path_ok = not case.corpus_answerable or case.context_sufficient
task_success = supported_answer or (
not case.corpus_answerable and case.response == "bounded-abstention"
)
return safe_response, evidence_path_ok, task_success
cases = (
ReviewedOutcome("complete answer", True, True, "complete-supported"),
ReviewedOutcome("waiver not established", False, False, "bounded-abstention"),
ReviewedOutcome("missed rule, safe refusal", True, False, "bounded-abstention"),
ReviewedOutcome("rule available, unnecessary refusal", True, True, "bounded-abstention"),
ReviewedOutcome("no waiver exists anywhere", False, False, "unsupported"),
)
for case in cases:
safe, evidence_ok, success = outcome_labels(case)
print(f"{case.name}: safe={safe}, evidence_path_ok={evidence_ok}, task_success={success}")
assert outcome_labels(cases[2]) == (True, False, False)
assert outcome_labels(cases[3]) == (True, True, False)
assert outcome_labels(cases[4]) == (False, True, False)complete answer: safe=True, evidence_path_ok=True, task_success=True
waiver not established: safe=True, evidence_path_ok=True, task_success=True
missed rule, safe refusal: safe=True, evidence_path_ok=False, task_success=False
rule available, unnecessary refusal: safe=True, evidence_path_ok=True, task_success=False
no waiver exists anywhere: safe=False, evidence_path_ok=True, task_success=FalseHere, safe means only that the reviewed response avoids unsupported policy claims; it doesn't certify general safety. evidence_path_ok=True on an unanswerable case means there was no labeled answer-bearing evidence to miss, not that every retrieval decision was optimal. A bounded refusal and an unsupported universal claim need different labels even when both contain no affirmative waiver.
The main harness below remains scoped to Maya's answerable requirements question. This separate replay demonstrates abstention accounting without treating empty context as an authorization failure or a perfect faithfulness score. A production suite must validate the actual empty-context trace, source-access decisions, and response text, then report unsupported answering and unnecessary abstention separately.
Attribute the first failure
When a RAG answer fails, start with one useful question: where did its evidence path first break? If you score the system as one black box, you won't know whether to inspect retrieval, reranking, or the answer itself.
Pipeline order gives an investigation priority. It doesn't resolve every cause. If retrieval misses a rule and the answer invents a waiver, both defects matter: the model should handle insufficient evidence safely. Short-circuiting a release decision can save work, while a diagnostic run can still assess downstream behavior on the exact original context.
This answerable-case fixture checks:
- Admissibility: Did unauthorized or stale evidence enter the pipeline?
- Candidate retrieval: Did hybrid search place the required chunk in initial candidates?
- Context selection: Did reranking keep the chunk in the admitted prompt context?
- Claim presence: Are there policy claims to evaluate?
- Claim review: Does the fixed answer key cover their exact text?
- Answer faithfulness: Are emitted claims supported by admitted evidence?
- Citation support: Do the attached sources support their claims?
- Task completeness: Are all required points covered?
Each failure narrows the next inspection:
- A missing candidate calls for checking the index, query, retrieval, and gold labels.
- A dropped rule calls for checking shortlist depth, reranking, filtering, deduplication, and packing.
- No extracted policy claims calls for inspecting the original output, extraction, and API response to identify where the text went missing.
- Unreviewed text needs adjudication; the answer key can't distinguish a new contradiction from a new valid paraphrase.
- For an invented bypass, inspect generation and its support labels. Test lower temperature or another model against the same cases before adopting the change.
- A wrong citation calls for inspecting source attachment and validation.
- A missing required point calls for inspecting answer coverage and the rubric.
RETRIEVAL_MISS = replace(
TRACE,
first_stage_ids=("payment-service-rollback-runbook", "frontend-docs-deploy-rule"),
rerank_input_ids=("payment-service-rollback-runbook", "frontend-docs-deploy-rule"),
reranked_ids=("payment-service-rollback-runbook", "frontend-docs-deploy-rule"),
selected_context_ids=("payment-service-rollback-runbook",),
selected_versions=("payment-rollback/2026-05-20",),
)
SELECTION_MISS = replace(
TRACE,
selected_context_ids=("payment-service-rollback-runbook",),
selected_versions=("payment-rollback/2026-05-20",),
)
def first_failed_stage(trace: RagTrace, answer: Answer, gold: GoldCase) -> str:
if not admissible_evidence_path(trace, gold):
return "admissibility"
if coverage(trace.first_stage_ids, gold.required_source_ids) < 1.0:
return "candidate retrieval"
if coverage(trace.selected_context_ids, gold.required_source_ids) < 1.0:
return "context selection"
if not answer.claims:
return "claim presence"
if any(
claim.text not in REVIEWED_SUPPORT and claim.text not in REVIEWED_UNSUPPORTED
for claim in answer.claims
):
return "claim review"
if faithfulness(answer, trace) < 1.0:
return "answer faithfulness"
if citation_support(answer, trace) < 1.0:
return "citation support"
if supported_point_coverage(answer, trace, gold) < 1.0:
return "answer completeness"
return "pass"
diagnoses = {
"missing candidate": first_failed_stage(RETRIEVAL_MISS, SUPPORTED_DEPLOY, GOLD),
"dropped context": first_failed_stage(SELECTION_MISS, SUPPORTED_DEPLOY, GOLD),
"invented bypass": first_failed_stage(TRACE, UNSAFE_BYPASS, GOLD),
"wrong citation": first_failed_stage(TRACE, MIS_CITED, GOLD),
"empty answer": first_failed_stage(TRACE, EMPTY_ANSWER, GOLD),
"missing rollback point": first_failed_stage(TRACE, PARTIAL_DEPLOY, GOLD),
"supported answer": first_failed_stage(TRACE, SUPPORTED_DEPLOY, GOLD),
}
for variant, stage in diagnoses.items():
print(f"{variant}: {stage}")
assert diagnoses["missing candidate"] == "candidate retrieval"
assert diagnoses["dropped context"] == "context selection"
assert diagnoses["invented bypass"] == "answer faithfulness"
assert diagnoses["wrong citation"] == "citation support"
assert diagnoses["empty answer"] == "claim presence"
assert diagnoses["missing rollback point"] == "answer completeness"
assert diagnoses["supported answer"] == "pass"missing candidate: candidate retrieval
dropped context: context selection
invented bypass: answer faithfulness
wrong citation: citation support
empty answer: claim presence
missing rollback point: answer completeness
supported answer: passEmpty claims fail claim presence. The partial answer emits supported claims but misses the rollback point, so it fails answer completeness. An unseen paraphrase returns claim review; the fixed key can't decide its meaning. These labels name observed checks, not proven causes.
Try two attempts to fool the evaluator. The first keeps a valid claim ID but negates its text. The second repeats an approval claim under the rollback ID. Neither should gain support or completeness from its identifier.
negated = replace(
SUPPORTED_DEPLOY.claims[1], text="Incident commander approval is not required."
)
spoofed_text = Answer("negated", (negated,))
spoofed_point = Answer(
"renamed-approval",
PARTIAL_DEPLOY.claims + (replace(SUPPORTED_DEPLOY.claims[1], claim_id="rollback-plan"),),
)
paraphrase = Answer("unreviewed-paraphrase", (
Claim("approval", "Approval from the incident commander is mandatory.", TARGET_ID),
))
print("Negated known claim:", first_failed_stage(TRACE, spoofed_text, GOLD))
print("Renamed duplicate:", first_failed_stage(TRACE, spoofed_point, GOLD))
print("Valid but unreviewed paraphrase:", first_failed_stage(TRACE, paraphrase, GOLD))
assert first_failed_stage(TRACE, spoofed_text, GOLD) == "claim review"
assert first_failed_stage(TRACE, spoofed_point, GOLD) == "answer completeness"
assert first_failed_stage(TRACE, paraphrase, GOLD) == "claim review"
assert supported_point_coverage(spoofed_point, TRACE, GOLD) == 2 / 3Negated known claim: claim review
Renamed duplicate: answer completeness
Valid but unreviewed paraphrase: claim reviewAn unreviewed contradiction and an unreviewed correct paraphrase both require review. This fixture routes both to review; after a reviewer distinguishes them, store separate labels. Keep atomic-claim segmentation and deduplication consistent so repetition can't inflate a reported average.
Challenge the source and citation receipts
Predict whether an unchanged source ID/version is sufficient after its text changes. Then compare a blank citation string with a citation that actually names a source.
original_chunk = EVIDENCE[TARGET_ID]
EVIDENCE[TARGET_ID] = replace(original_chunk, text="Edited policy under an unchanged version")
try:
print("Edited source diagnosis:", first_failed_stage(TRACE, SUPPORTED_DEPLOY, GOLD))
assert first_failed_stage(TRACE, SUPPORTED_DEPLOY, GOLD) == "admissibility"
assert not source_supports(SUPPORTED_DEPLOY.claims[1], EVIDENCE[TARGET_ID])
finally:
EVIDENCE[TARGET_ID] = original_chunk
blank_citations = Answer(
"blank-citations",
tuple(replace(claim, citation_id=" ") for claim in SUPPORTED_DEPLOY.claims),
)
wrong_question = replace(TRACE, question="Does a different service have a freeze exemption?")
print(f"Blank citation coverage: {citation_coverage(blank_citations):.1f}")
print("Changed question admissible:", admissible_evidence_path(wrong_question, GOLD))
assert citation_coverage(blank_citations) == 0.0
assert first_failed_stage(TRACE, blank_citations, GOLD) == "citation support"
assert not admissible_evidence_path(wrong_question, GOLD)Edited source diagnosis: admissibility
Blank citation coverage: 0.0
Changed question admissible: FalseMake bad behaviors part of release testing
Evaluation should show both that a supported answer is accepted and that known bad answers are blocked. These variants become a tiny regression suite:
@dataclass(frozen=True)
class ReleaseCase:
name: str
trace: RagTrace
answer: Answer
should_release: bool
def can_release(trace: RagTrace, answer: Answer, gold: GoldCase) -> bool:
return first_failed_stage(trace, answer, gold) == "pass"
RELEASE_CASES = (
ReleaseCase("supported answer", TRACE, SUPPORTED_DEPLOY, True),
ReleaseCase("unsupported bypass", TRACE, UNSAFE_BYPASS, False),
ReleaseCase("wrong citation", TRACE, MIS_CITED, False),
ReleaseCase("empty answer", TRACE, EMPTY_ANSWER, False),
ReleaseCase("dropped source", SELECTION_MISS, SUPPORTED_DEPLOY, False),
ReleaseCase("missing rollback point", TRACE, PARTIAL_DEPLOY, False),
)
regression_passes = 0
for case in RELEASE_CASES:
observed = can_release(case.trace, case.answer, GOLD)
regression_passes += observed == case.should_release
print(f"{case.name}: release={observed}, expected={case.should_release}")
print(f"Regression checks passed: {regression_passes}/{len(RELEASE_CASES)}")
assert regression_passes == len(RELEASE_CASES)supported answer: release=True, expected=True
unsupported bypass: release=False, expected=False
wrong citation: release=False, expected=False
empty answer: release=False, expected=False
dropped source: release=False, expected=False
missing rollback point: release=False, expected=False
Regression checks passed: 6/6Release behavior is now concrete: pass the supported response and reject the five controlled defects.
Slice results before trusting an average
A larger golden set should include release-freeze deploys, incident hotfixes, schema migrations, and data backfills. Aggregate quality can remain high while a costly policy slice regresses.
@dataclass(frozen=True)
class LabeledOutcome:
workflow: str
released_correctly: bool
def __post_init__(self) -> None:
if not isinstance(self.workflow, str) or not self.workflow.strip():
raise ValueError("workflow must be a nonempty string")
if type(self.released_correctly) is not bool:
raise ValueError("outcome must be a Boolean")
OUTCOMES = (
LabeledOutcome("release-freeze", True),
LabeledOutcome("release-freeze", False),
LabeledOutcome("incident-hotfix", True),
LabeledOutcome("incident-hotfix", True),
LabeledOutcome("schema-migration", False),
)
by_workflow: dict[str, list[bool]] = defaultdict(list)
for outcome in OUTCOMES:
by_workflow[outcome.workflow].append(outcome.released_correctly)
overall = sum(outcome.released_correctly for outcome in OUTCOMES) / len(OUTCOMES)
print(f"Overall pass rate: {overall:.0%}")
for workflow, results in sorted(by_workflow.items()):
print(f"{workflow}: {sum(results)}/{len(results)} ({sum(results) / len(results):.0%})")
assert overall == 0.6
assert sum(by_workflow["schema-migration"]) == 0Overall pass rate: 60%
incident-hotfix: 2/2 (100%)
release-freeze: 1/2 (50%)
schema-migration: 0/1 (0%)REQUIRED_FIXTURE_SLICES = frozenset({"release-freeze", "incident-hotfix", "schema-migration"})
def slice_regression_clear(results_by_workflow: dict[str, list[bool]]) -> bool:
# The local 95/100 rule is exact count arithmetic, not a population bound.
if not REQUIRED_FIXTURE_SLICES.issubset(results_by_workflow):
return False
for workflow in REQUIRED_FIXTURE_SLICES:
results = results_by_workflow[workflow]
if (
not isinstance(results, list)
or not results
or any(type(result) is not bool for result in results)
or sum(results) * 100 < 95 * len(results)
):
return False
return True
release_report = {
"service_version": "policy-answerer-v4-eval",
"source_trace": TRACE.case_id,
"required_policy_version": EVIDENCE[TARGET_ID].version,
"checks": {
"admissible_evidence_path": admissible_evidence_path(TRACE, GOLD),
"supported_answer_released": can_release(TRACE, SUPPORTED_DEPLOY, GOLD),
"known_bad_answers_blocked": all(
not can_release(case.trace, case.answer, GOLD)
for case in RELEASE_CASES
if not case.should_release
),
"slice_regression_clear": slice_regression_clear(by_workflow),
},
}
print("Release checks:", release_report["checks"])
print("Release allowed:", all(release_report["checks"].values()))
assert not all(release_report["checks"].values())Release checks: {'admissible_evidence_path': True, 'supported_answer_released': True, 'known_bad_answers_blocked': True, 'slice_regression_clear': False}
Release allowed: FalseThe supported trace passes, but this synthetic five-case report blocks release. Two required slices miss the local 95% threshold. Missing or empty required slices also fail; an empty report mustn't pass through all([]). The two incident-hotfix successes don't establish a 95% population success rate. This lab only requires a nonempty slice to demonstrate logic. A real study must predefine thresholds, required slices, adequate sample sizes, and uncertainty rules.
Choose the aggregation unit with equal care. For the formulas below, counts answers with atomic claims; report empty-claims cases and abstentions separately rather than silently assigning an undefined ratio. Two averages answer different questions:
- Pooled-claim faithfulness (micro-average): All claims from all answers are dumped into a single pool:
- Mean per-answer faithfulness (macro-average): Each answer's faithfulness ratio is computed individually, and those ratios are averaged across queries:
Pooling weights answers by claim count; a per-answer mean weights answers equally. Neither is inherently the correct estimand for every study. Label the unit and show both when the distinction affects interpretation.
Consider two hypothetical test cases:
- Query 1: The model generates a wordy, 9-claim explanation of a basic policy rule. Every single claim is entailed by the context (9/9 supported, 100% faithfulness).
- Query 2: The model generates a concise, 1-claim response on a critical freeze rule, but hallucinates an unauthorized bypass (0/1 supported, 0% faithfulness).
Under pooled-claim aggregation:
Under mean per-answer aggregation:
The fraction of nonempty answers whose claims are all supported is:
The first answer contributes nine times the weight under pooling. These two cases don't establish a population failure rate. Complete claim support also doesn't detect omissions: PARTIAL_DEPLOY has no unsupported claims but misses rollback. Avoid labeling this pass fraction "safety" or treating it as complete task success.
annotated_counts = [(9, 9), (0, 1)] # (supported, emitted), hypothetical cases
pooled = sum(supported for supported, _ in annotated_counts) / sum(
emitted for _, emitted in annotated_counts
)
per_answer = sum(supported / emitted for supported, emitted in annotated_counts) / len(annotated_counts)
claim_support_pass = sum(supported == emitted for supported, emitted in annotated_counts) / len(annotated_counts)
print(f"Pooled claim support: {pooled:.0%}")
print(f"Mean per-answer support: {per_answer:.0%}")
print(f"All-claims-supported fraction: {claim_support_pass:.0%}")
print(f"Partial answer: all claims supported={faithfulness(PARTIAL_DEPLOY, TRACE) == 1.0}, "
f"task complete={supported_point_coverage(PARTIAL_DEPLOY, TRACE, GOLD) == 1.0}")
assert (pooled, per_answer, claim_support_pass) == (0.9, 0.5, 0.5)
assert not can_release(TRACE, PARTIAL_DEPLOY, GOLD)
assert not slice_regression_clear({})
assert not slice_regression_clear({workflow: [] for workflow in REQUIRED_FIXTURE_SLICES})Pooled claim support: 90%
Mean per-answer support: 50%
All-claims-supported fraction: 50%
Partial answer: all claims supported=True, task complete=FalseChoose claim-support, completeness, and critical-slice rules for the actual product before comparing systems. Evaluate the same questions and retain paired results. Claims within an answer and paraphrases of the same case aren't independent test questions. Estimate uncertainty at the question or case-family level, keep calibration separate from held-out evaluation, and show slice counts. A high average over near-duplicates isn't broad coverage.
Where automated judges enter
The harness's answer key makes the arithmetic testable. It doesn't solve semantic support checking. Once questions or answers change, humans or a calibrated judge must decide whether each claim is entailed, contradicted, or not established by the admitted evidence.
The original RAGAS paper estimates faithfulness from claim support, answer relevance from embedding similarity to questions generated from the answer, and context relevance from extracted relevant sentences. "Reference-free" means the metric needs no gold answer. It still judges claims against retrieved evidence.[2] ARES trains judges on synthetic examples and uses human labels with prediction-powered inference to estimate aggregate evaluation scores with confidence intervals. Those intervals quantify aggregate system-level estimates. Evaluate each answer against its evidence and labels.[8]
Predict the boundary before adding a judge: which decision can semantic scoring help with, and which decisions must remain hard gates? A judge may recognize a valid paraphrase, but it can't authorize a restricted chunk or repair missing provenance.
Those approaches don't remove the need for this trace design. An automated judge still needs:
| Input | Why the judge needs it |
|---|---|
| Question and expected workflow | Decide whether the answer addressed the request |
| Admitted context and version | Decide whether claims are grounded in allowed evidence |
| Atomic claims and citations | Explain which claim failed and which source was cited |
| Human-reviewed calibration set | Detect judge mistakes and drift |
For a support judge, include supported claims, contradictions, missing evidence, valid paraphrases, wrong citations, and appropriate abstentions. A set containing only good answers can't measure false acceptance. Give reviewers the same question and versioned evidence as the judge; resolve ambiguous labels and measure both false accepts and false rejects on held-out cases. Record the judge model, prompt, rubric, and parsing failures. Retrieved text is evidence to assess, not instructions the judge should obey.
For a separate clarity comparison, use two already-supported answers and randomize their presentation order. Don't substitute a preference for fluent writing for evidence entailment. Authorization and trace integrity stay independent of any later semantic judge. What still has to pass before this answerer serves another freeze-deploy question?
Release gates for policy-answerer-v4
Before this answerer serves another freeze-deploy question:
| Gate | Minimum evidence | First fix when it fails |
|---|---|---|
| Admissibility | Retrieved, reranked, and selected chunk IDs; versions; ACL/freshness decision | Evidence filtering |
| Candidate recall | Required chunk in retrieved candidates | Chunking, query, or retrieval lane |
| Context admission | Sufficient admitted evidence; measure distracting context separately | Reranker or context gate |
| Faithfulness | Supported claim ledger | Prompt, abstention, or generation policy |
| Citation support | Claim-to-source validation | Citation attachment or validator |
| Completeness | Required points covered, or justified abstention on an unanswerable case | Answer policy and coverage rubric |
| Slice health | Workflow-level regression report | Block release and debug affected slice |
Don't compress these gates into one quality percentage. A dashboard should tell an engineer what to repair first.
Use the first failed check to choose the next inspection. A supported omission is a different miss from a contradiction, and an average can hide a broken policy slice. The answer key still cannot score a paraphrase it has never seen.