RAG Evaluation for Reliable Answers

In this fictional policy lab, Maya asks what DEPLOY-17 requires for a payment-service production deploy during a release freeze. The recorded hybrid search candidates contain the rule, and the reranker selects it. Consider this hand-written answer:

The release-freeze rule applies. You can deploy without a linked rollback plan. [DEPLOY-17]

That answer sounds plausible because its citation names a rule in the fixture. It contradicts that rule: both incident commander (IC) approval and a linked rollback plan are required before rollout. This lab blocks release when any emitted policy claim lacks support.

In retrieval-augmented generation (RAG), retrieved text supplies evidence to a language model. The previous lesson's policy-answerer-v3 recorded an evidence trace showing which chunks were retrieved, selected, and tied to source versions. A trustworthy trace lets you inspect that path. It doesn't prove the answer stayed inside the evidence.

policy-answerer-v4-eval is the evaluation harness for this request. It reports the first failed check instead of averaging everything into one quality score. Its decisions use trusted fixture labels, not a neural entailment model. Those gates still matter when the next lesson adds large language model (LLM) judges for paraphrases the labels can't cover.

TruLens calls three evaluation dimensions the RAG Triad. It is a useful framework, not a universal production standard:^{[1]Reference 1The RAG Triadhttps://www.trulens.org/getting_started/core_concepts/rag_triad/}

1. Context relevance: Is supplied context relevant to the question?
2. Groundedness / faithfulness: Are the answer's claims supported by supplied evidence? Claim extraction and support assessment can both make mistakes; a high estimated score doesn't guarantee zero hallucinations.^{[2]Reference 2RAGAS: Automated Evaluation of Retrieval Augmented Generation.https://arxiv.org/abs/2309.15217}
3. Answer relevance: Does the response address the question? Topic relevance alone doesn't certify every required condition, factual correctness, or task completion.

For this policy explanation, we add access, citation, and required-point checks. These are this lesson's contracts, not definitions of the Triad:

• Admissibility and authorization: Did every chunk in the evidence trace pass access control (such as Fine-Grained Authorization) and freshness checks before reaching the model?
• Context selection: Did hybrid search retrieve the gold rule, and did reranking preserve it within the admitted context budget?
• Claim faithfulness: Does the admitted context logically entail every atomic claim asserted by the model?
• Citation support: Does each attached citation link to an admitted chunk that actually establishes that specific claim?
• Task completeness: Did the model cover every required policy point instead of silently dropping mandatory conditions?

A fluent answer can pass one gate and fail another. Faithfulness is relative to supplied evidence, not an absolute guarantee of real-world truth. An answer can faithfully repeat an incorrect source; a true fact from the model's pre-training memory can be unsupported by this specific context. Nor does a rule saying approval and a rollback plan are required prove that meeting those conditions alone authorizes deployment. This lesson evaluates a policy explanation, not an execution permit.

Replay the freeze-deploy trace

Maya's question for this replay is:

What does DEPLOY-17 require for a payment-service production deploy during a release freeze?

The fixture starts after hybrid retrieval found the correct rule and reranking selected it. Evaluation consumes that recorded evidence path instead of reconstructing retrieval inside the answer-quality harness. The field names match the previous lesson's release trace: first_stage_ids, reranked_ids, selected_context_ids, and component versions.

Before running the fixture, predict which chunk ID should survive in selected_context_ids. Check that tuple, its source version, and the component versions against the printed trace.

The trace identifies the sources and retrieval configuration to inspect. This in-memory fixture has one immutable revision per chunk. A real replay store must resolve the recorded revision, not substitute today's text. To reproduce generation, also save the exact assembled prompt, generator version, decoding settings, and original output. Even those records don't promise identical resampling from a hosted model.

Gate admissibility before scoring quality

A relevance score can't make forbidden evidence acceptable. Here, first_stage_ids means candidates returned across the request's authorization boundary, not every record an index scans internally. Restricted, stale, or unknown evidence in that returned path fails this fixture's contract, even if selection later drops it. permitted and current are trusted test labels; these booleans don't implement access control or freshness checks.

The case and question checks bind these labels to this replay; they don't establish that the recorded text is the caller's actual request or authenticate that caller. Complete record comparisons reject edited source text, parents, or metadata under an unchanged ID/version. The component names are teaching version labels, not immutable artifact pins. A service also needs trusted caller/grant records and current access checks.

Separate candidates from selected context

The path is allowed. That still doesn't say the required rule is in the candidate list, or that it survived into the prompt.

Context relevance assesses relevance to the question; evidence sufficiency also needs a coverage check. We inspect candidate retrieval and selected context separately.

Pause on that distinction: if the rule appears in first-stage candidates but disappears before the prompt, which recall should fall? Keep that prediction in mind while reviewing the metrics:

Selected-context precision directly measures noise reduction:

$$\operatorname{Precision}_{\text{selected}} = \frac{|\text{admitted chunk IDs} \cap \text{useful chunk IDs}|}{|\text{admitted chunk IDs}|}$$

Distractors can increase context cost and introduce competing evidence. Lost in the Middle found position-dependent degradation on its tested tasks and models, especially when relevant information appeared in the middle of long contexts.^{[3]Reference 3Lost in the Middle: How Language Models Use Long Contextshttps://arxiv.org/abs/2307.03172} Extra irrelevant chunks don't automatically trigger that failure in every model; test position and distraction separately.

Ranking metrics such as mean reciprocal rank (MRR) and normalized discounted cumulative gain (NDCG) ask whether top evidence appears early in a ranked list.^{[4]Reference 4Introduction to Information Retrieval.https://nlp.stanford.edu/IR-book/} At generation evaluation time, we care about admission: did the evidence make it into the prompt, and how clean is the prompt?

This result describes only the evidence path. It doesn't say what the model wrote. These are set metrics over labeled chunk IDs, not a reproduction of the original RAGAS context-relevance metric.^{[2]Reference 2RAGAS: Automated Evaluation of Retrieval Augmented Generation.https://arxiv.org/abs/2309.15217} An alternative chunk could contain equally good evidence yet score as a miss against an incomplete gold set. Label acceptable alternatives or assess coverage of answer-bearing facts before treating that miss as a retrieval defect.

Metric names can hide different denominators. Current Ragas documents a rank-weighted ContextPrecision and a separate IDBasedContextPrecision using the matching-ID fraction.^{[5]Reference 5Ragas: Context Precisionhttps://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/} Our unique-ID precision follows the latter idea. With one useful chunk and two distractors, set precision is $1/3$ regardless of order; the rank-weighted metric can change when the useful chunk moves.

Turn an answer into testable claims

Consider two responses produced from the same valid context:

Context can be sufficient while the answer isn't. Split the answer into atomic policy claims so you can score it. Each claim below carries its text and citation. A separate, trusted annotation will say what the evidence supports. FActScore similarly evaluates factual precision at the level of atomic facts rather than treating a whole passage as uniformly correct.^{[6]Reference 6FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation.https://arxiv.org/abs/2305.14251}

Before looking at the output, decide which answer should score lower. Both use the same selected context, but the bypass replaces required conditions with an unsupported exemption; count supported claims, not citation count or answer length.

These claims are hand-written test fixtures, not outputs from a model call. In live evaluation, inspect claim extraction too: dropping a qualifier such as "only" or skipping an unsupported sentence can inflate every later score.

Faithfulness checks claims against context

Selected context can still lose to a generator that outruns it. Check each claim against the selected DEPLOY-17 sentence before writing a scorer:

One of two unsafe claims is supported, so faithfulness is $1/2$. RAGAS formalizes this ratio after an LLM extracts atomic statements and evaluates which ones are logically inferred from context.^{[2]Reference 2RAGAS: Automated Evaluation of Retrieval Augmented Generation.https://arxiv.org/abs/2309.15217}

Verifying whether context $E$ supports claim $c_j$ is formally modeled as a Natural Language Inference (NLI) task. Treating the admitted context $E$ as the premise and atomic claim $c_j$ as the hypothesis, the evaluator (an NLI cross-encoder or calibrated LLM judge) assigns one of three categorical relations:

1. Entailment ($E \implies c_j$): The context logically necessitates the claim, including all scopes and conditions.
2. Contradiction ($E \implies \neg c_j$): The context directly conflicts with the claim. Here, DEPLOY-17 requires a linked rollback plan; the answer asserts the plan can be absent.
3. Neutral / Not Established ($E \not\implies c_j \land E \not\implies \neg c_j$): The context doesn't contain enough evidence to confirm or refute the claim (such as speculating about unmentioned executive waivers).

A learned evaluator estimates these relations; its verdict isn't a formal proof. The claim-level faithfulness metric is the fraction of emitted claims assessed as entailed by the admitted context:

$$\operatorname{faithfulness} = \frac{\sum_{j=1}^m \mathbb{I}(\text{entailed}(c_j, E))}{m}$$

where $m$ is the total count of atomic claims in the response.

Token overlap alone can't distinguish support here: "approval is required" and "approval is not required" share most of their tokens while stating opposite rules. Entailment requires agreement with scope and conditions, not just lexical similarity.

The code uses a hand-labeled teaching answer key. Each supported text maps to an exact source revision and a required point. An unknown text is unadjudicated, not automatically false. This fixture refuses to certify it until reviewed. In a real study, independently review the labels; never let the answer supply its own support verdict.

The unsafe response is on topic and cites a real selected rule. It still fails because one claim contradicts that rule. The empty-claims score of 0.0 is a deliberate fixture convention, not a zero-denominator ratio. Current Ragas also defines faithfulness through extracted-claim support; its extraction and support models can err.^{[7]Reference 7Ragas: Faithfulnesshttps://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/} This answer key binds each claim to one source. A claim requiring several sources needs an evidence-set annotation or another evaluator. We'll handle justified abstention separately.

Citation presence isn't citation support

Faithfulness asks whether selected context could support the sentence. Citation support asks whether the attached source is the one that does. RAGAS faithfulness doesn't check citation IDs, so this gate is extra.

A citation metric needs two checks:

1. Coverage: Does every policy claim cite a source?
2. Support: Does the cited selected source establish that claim?

A fabricated rollback exemption can get perfect citation coverage by attaching the correct-looking chunk ID. Presence alone is a weak gate.

Before running the next cell, predict what changes for MIS_CITED: faithfulness, citation support, or both.

In the mis-cited response, selected context supports every sentence, so faithfulness is 1.0. Its citations still fail because they point at payment-service-rollback-runbook, which isn't selected and doesn't establish those claims. Adding that unrelated runbook to context would fix neither the citation support nor its relevance to Maya's question.

Check whether the answer finished the task

A 1.0 faithfulness score can still fail release if a required point never appears as a supported claim. For this golden case, Maya needs three answer points: deploy scope, approval, and rollback plan. Treat point coverage as a labeled completeness gate:

Point coverage does a different job from answer relevance. A response can sound on-topic while omitting the rollback condition; RAGAS answer relevance asks whether the response addresses the question, while this labeled gate checks whether each required policy point appears.^{[2]Reference 2RAGAS: Automated Evaluation of Retrieval Augmented Generation.https://arxiv.org/abs/2309.15217}

PARTIAL_DEPLOY is the useful contrast: faithfulness is 1.0 while point coverage is only $2/3$ because the rollback-plan claim never appears. Conversely, a correctly quoted frontend deployment rule could be fully supported yet irrelevant to Maya's payment-service question. The required points are a task-specific relevance and completeness rubric, not a general semantic relevance detector.

Before checking the partial answer's output, predict whether its missing rollback point should be blamed on faithfulness or completeness.

When declining to answer is correct

Suppose Maya asks whether an executive escalation waives the rollback requirement. The admitted rule doesn't establish that exception. "The available policy doesn't establish a waiver; ask the policy owner" is a useful abstention. "No waiver exists anywhere" is an unsupported universal claim.

Label both corpus answerability and selected-evidence sufficiency. If the corpus contains the needed rule but retrieval misses it, abstaining can be the safe generation behavior while retrieval still fails. Report that distinction instead of rewarding every refusal or forcing every question into an answer-required gate.

The harness below remains scoped to Maya's answerable requirements question. A production suite needs an explicit abstention branch, including safe handling of empty context, and separate rates for unsupported answering and unnecessary abstention. Don't turn an undefined claim ratio into a perfect faithfulness score.

Attribute the first failure

Evaluating a multi-component RAG system as a monolithic black box leads to circular debugging. When a generated answer fails in production, you can't immediately tell whether to blame the vector index, the reranker, the generator's temperature, or the prompt template.

Pipeline order gives an investigation priority. It doesn't resolve every cause. If retrieval misses a rule and the answer invents a waiver, both defects matter: the model should handle insufficient evidence safely. Short-circuiting a release decision can save work, while a diagnostic run can still assess downstream behavior on the exact original context.

This answerable-case fixture checks:

1. Admissibility: Did unauthorized or stale evidence enter the pipeline?
2. Candidate retrieval: Did hybrid search place the required chunk in initial candidates?
3. Context selection: Did reranking keep the chunk in the admitted prompt context?
4. Claim presence: Are there policy claims to evaluate?
5. Claim review: Does the fixed answer key cover their exact text?
6. Answer faithfulness: Are emitted claims supported by admitted evidence?
7. Citation support: Do the attached sources support their claims?
8. Task completeness: Are all required points covered?

Each failure narrows the next inspection:

• A missing candidate calls for checking the index, query, retrieval, and gold labels.
• A dropped rule calls for checking shortlist depth, reranking, filtering, deduplication, and packing.
• No extracted policy claims calls for inspecting the original output and extraction. It doesn't prove an API returned blank text.
• Unreviewed text needs adjudication; the answer key can't distinguish a new contradiction from a new valid paraphrase.
• An invented bypass calls for checking generation and its support labels. Lower temperature or a different model isn't a guaranteed repair.
• A wrong citation calls for inspecting source attachment and validation.
• A missing required point calls for inspecting answer coverage and the rubric.

Empty claims fail claim presence. The partial answer emits supported claims but misses the rollback point, so it fails answer completeness. An unseen paraphrase returns claim review; the fixed key can't decide its meaning. These labels name observed checks, not proven causes.

Try two attempts to fool the evaluator. The first keeps a valid claim ID but negates its text. The second repeats an approval claim under the rollback ID. Neither should gain support or completeness from its identifier.

An unreviewed contradiction and an unreviewed correct paraphrase both require review. That is an honest limit of this fixture, not evidence that their meanings are equivalent. Once reviewed, store separate labels. Keep atomic-claim segmentation and deduplication consistent so repetition can't inflate a reported average.

Challenge the source and citation receipts

Predict whether an unchanged source ID/version is sufficient after its text changes. Then compare a blank citation string with a citation that actually names a source.

Make bad behaviors part of release testing

Evaluation should show both that a supported answer is accepted and that known bad answers are blocked. These variants become a tiny regression suite:

Release behavior is now concrete: pass the supported response and reject the five controlled defects.

Slice results before trusting an average

A larger golden set should include release-freeze deploys, incident hotfixes, schema migrations, and data backfills. Aggregate quality can remain high while a costly policy slice regresses.

The supported trace passes, but this synthetic five-case report blocks release. Two required slices miss the local 95% threshold. Missing or empty required slices also fail; an empty report mustn't pass through all([]). The two incident-hotfix successes don't establish a 95% population success rate. This lab only requires a nonempty slice to demonstrate logic. A real study must predefine thresholds, required slices, adequate sample sizes, and uncertainty rules.

Choose the aggregation unit with equal care. For the formulas below, $N$ counts answers with $m_i>0$ atomic claims; report empty-claims cases and abstentions separately rather than silently assigning an undefined ratio. Two averages answer different questions:

1. Pooled-claim faithfulness (micro-average): All claims from all answers are dumped into a single pool: $$\operatorname{Faithfulness}_{\text{pooled}} = \frac{\sum_{i=1}^N \sum_{j=1}^{m_i} \mathbb{I}(\text{entailed}(c_{i,j}, E_i))}{\sum_{i=1}^N m_i}$$
2. Mean per-answer faithfulness (macro-average): Each answer's faithfulness ratio is computed individually, and those ratios are averaged across queries: $$\operatorname{Faithfulness}_{\text{macro}} = \frac{1}{N} \sum_{i=1}^N \left( \frac{\sum_{j=1}^{m_i} \mathbb{I}(\text{entailed}(c_{i,j}, E_i))}{m_i} \right)$$

Pooling weights answers by claim count; a per-answer mean weights answers equally. Neither is inherently the correct estimand for every study. Label the unit and show both when the distinction affects interpretation.

Consider two hypothetical test cases:

• Query 1: The model generates a wordy, 9-claim explanation of a basic policy rule. Every single claim is entailed by the context (9/9 supported, 100% faithfulness).
• Query 2: The model generates a concise, 1-claim response on a critical freeze rule, but hallucinates an unauthorized bypass (0/1 supported, 0% faithfulness).

Under pooled-claim aggregation:

$$\operatorname{Faithfulness}_{\text{pooled}} = \frac{9 + 0}{9 + 1} = \frac{9}{10} = 90\%$$

Under mean per-answer aggregation:

$$\operatorname{Faithfulness}_{\text{macro}} = \frac{1.0 + 0.0}{2} = 50\%$$

The fraction of nonempty answers whose claims are all supported is:

$$\operatorname{ClaimSupportPass} = \frac{1 + 0}{2} = 50\%$$

The first answer contributes nine times the weight under pooling. These two cases don't establish a population failure rate. Complete claim support also doesn't detect omissions: PARTIAL_DEPLOY has no unsupported claims but misses rollback. Avoid labeling this pass fraction "safety" or treating it as complete task success.

Choose claim-support, completeness, and critical-slice rules for the actual product before comparing systems. Evaluate the same questions and retain paired results. Claims within an answer and paraphrases of the same case aren't independent test questions. Estimate uncertainty at the question or case-family level, keep calibration separate from held-out evaluation, and show slice counts. A high average over near-duplicates isn't broad coverage.

Where automated judges enter

The harness's answer key makes the arithmetic testable. It doesn't solve semantic support checking. Once questions or answers change, humans or a calibrated judge must decide whether each claim is entailed, contradicted, or not established by the admitted evidence.

The original RAGAS paper estimates faithfulness from claim support, answer relevance from embedding similarity to questions generated from the answer, and context relevance from extracted relevant sentences. "Reference-free" here means no gold answer is required, not that no evidence is required.^{[2]Reference 2RAGAS: Automated Evaluation of Retrieval Augmented Generation.https://arxiv.org/abs/2309.15217} ARES trains judges on synthetic examples and uses human labels with prediction-powered inference to estimate aggregate evaluation scores with confidence intervals. Those intervals concern system-level estimates, not a guarantee that an individual answer is correct.^{[8]Reference 8ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems.https://aclanthology.org/2024.naacl-long.20/}

Predict the boundary before adding a judge: which decision can semantic scoring help with, and which decisions must remain hard gates? A judge may recognize a valid paraphrase, but it can't authorize a restricted chunk or repair missing provenance.

Those approaches don't remove the need for this trace design. An automated judge still needs:

For a support judge, include supported claims, contradictions, missing evidence, valid paraphrases, wrong citations, and appropriate abstentions. A set containing only good answers can't measure false acceptance. Give reviewers the same question and versioned evidence as the judge; resolve ambiguous labels and measure both false accepts and false rejects on held-out cases. Record the judge model, prompt, rubric, and parsing failures. Retrieved text is evidence to assess, not instructions the judge should obey.

For a separate clarity comparison, use two already-supported answers and randomize their presentation order. Don't substitute a preference for fluent writing for evidence entailment. The next lesson develops rubric-driven LLM judgments; authorization and trace-integrity checks remain independent of those judgments.

Release gates for policy-answerer-v4

Before this answerer serves another freeze-deploy question:

Don't compress these gates into one quality percentage. A dashboard should tell an engineer what to repair first.