RAG Evaluation for Reliable Answers

policy-answerer-v3 selects one current, permitted policy chunk: deploy-freeze-approval-rule. That's necessary evidence for Maya's reply, but it isn't proof of a reliable answer. A generator can still say the deploy may start without a rollback plan, or cite the right rule beside a claim it doesn't support.

The next step is an evaluation harness for that trace. Retrieval-augmented generation (RAG) evaluation should tell you where a response first became unsafe: evidence access, candidate retrieval, context selection, answer claims, or citations. You'll turn each boundary into an executable eval gate before the next lesson introduces large language model (LLM) judges for semantic cases that deterministic labels can't cover.

Carry the trace forward

Maya's question remains unchanged:

Can I deploy payment-service during the release freeze? Incident commander approval is attached, and the rollback plan is linked.

policy-answerer-v3 already established that hybrid retrieval found the correct rule and reranking selected it for context. The evaluation fixture carries that evidence path forward instead of inventing a new example.

The trace is an evaluation input, not a log decoration. It binds a question to exact source identities and pipeline versions without storing raw policy text. During replay, the evaluator loads permitted source text from the versioned evidence store.

Gate admissibility before scoring quality

An answer should fail immediately if restricted, stale, or unknown evidence entered retrieval, reranking, or selected context. A relevance score can't make forbidden evidence acceptable, even when a later stage happens to drop it.

The case-ID check prevents labels from one replay from grading another replay. The uniqueness check rejects malformed rankings before duplicated IDs distort later metrics.

Separate candidates from selected context

The previous two lessons already taught ranking metrics such as Mean Reciprocal Rank (MRR) and Normalized Discounted Cumulative Gain (NDCG). Those metrics answer whether the best evidence appears early in a ranking.^[1] At answer evaluation time, retain two simpler questions:

This result clears only the evidence path. It doesn't say what the model wrote.

Turn an answer into testable claims

Consider two responses produced from the same valid context:

For a labeled release case, represent an answer as atomic policy claims. Each claim names its cited source, the source phrases that must establish it, and the expected answer point it covers.

This is a golden-set contract. It's intentionally strict and inspectable. It won't recognize every correct paraphrase in live traffic; that softer matching problem belongs after you understand the release invariant.

Faithfulness checks claims against context

Faithfulness asks whether selected context supports the answer's claims. RAGAS defines this family of evaluation by decomposing a response into claims and checking support from retrieved context.^[2] For a labeled high-impact policy case, exact required phrases give a deterministic first gate:

$$\operatorname{faithfulness} = \frac{\text{claims supported by selected context}} {\text{claims in the answer}}$$

The unsafe response is on topic and cites a real selected rule. It still fails because one claim outruns the rule.

Citation presence isn't citation support

A citation metric needs two checks:

1. Coverage: Does every policy claim cite a source?
2. Support: Does the cited selected source establish that claim?

A fabricated rollback exemption can achieve perfect citation coverage by attaching the correct-looking chunk ID. That's why presence alone is a weak gate.

In the mis-cited response, selected context could support every sentence, so faithfulness is high. Its citations still fail because they don't point to the selected source that proves those sentences.

Check whether the answer finished the task

A faithful response can still omit a required condition. For this golden case, Maya needs three answer points: deploy scope, approval, and rollback plan. Treat point coverage as a labeled completeness gate:

This metric is stronger than asking whether an answer sounds relevant. It knows exactly which policy conditions this regression case must preserve. On live questions without labels, you'll need sampled human review or a calibrated judge, which is the next lesson's problem.

Attribute the first failure

One trace can fail at several layers after the first defect. If selected context drops the target rule, later unsupported claims are consequences, not the first fix. Diagnose in pipeline order.

The ordering is important. If the required rule never reaches context, a claim-support failure doesn't justify tuning the generation prompt yet.

Make bad behaviors part of release testing

Evaluation should show both that a supported answer is accepted and that known bad answers are blocked. These variants become a tiny regression suite:

Release behavior is now concrete: pass the supported response and reject the four controlled defects.

Slice results before trusting an average

A larger golden set should include release-freeze deploys, incident hotfixes, schema migrations, and data backfills. Aggregate quality can remain high while a costly policy slice regresses.

The supported single trace passes, and bad-answer detection works. The broader fixture still blocks release because schema-migration has a failed slice. This is the correct result: a demonstration case can't overrule a failing workflow.

Where automated judges enter

The harness used exact labels because policy release cases should be unambiguous. At scale, operators paraphrase questions and models paraphrase valid answers. RAGAS presents reference-free RAG metrics including faithfulness and answer relevance; ARES proposes learned judges for context relevance, answer faithfulness, and answer relevance.^[2][3]

Those approaches don't remove the need for this trace design. An automated judge still needs:

The next lesson will replace selected deterministic judgments with rubric-driven LLM judgments while keeping hard evidence and citation gates visible.

Production checklist

Before releasing a RAG answer pipeline:

Don't compress these gates into one quality percentage. A dashboard should tell an engineer what to repair first.

Mastery check

You're ready to evaluate a RAG answer pipeline when you can:

• Carry versioned source identity and pipeline versions from retrieval into answer evaluation.
• Treat authorization, freshness, trace integrity, and case identity as hard gates before quality scores.
• Separate candidate recall, selected-context recall, claim support, citation support, and answer completeness.
• Diagnose the first failing layer instead of tuning every component at once.
• Build regression cases that accept supported answers and block known bad behaviors.
• Explain why automated judges require calibration and can't replace deterministic gates.

Evaluation rubric

Follow-up questions

Common pitfalls

Evaluation starts from answer text only

• Symptom: A wrong response is visible, but nobody can tell whether evidence was missing, dropped, or ignored.
• Cause: The system didn't retain source identities, retrieved IDs, reranker inputs, selected context IDs, pipeline versions, and citations beside the answer.
• Fix: Store and replay a full evidence trace for every labeled evaluation case.

Citation coverage is mistaken for grounding

• Symptom: Responses show citations on every sentence, yet reviewers find unsupported rollback-bypass or deploy-approval promises.
• Cause: Tests check that a citation exists, not that the cited selected chunk supports the claim.
• Fix: Validate claim-to-source support and block unsupported policy claims.

Judge scores replace hard safety gates

• Symptom: A high judge score releases an answer based on restricted or stale policy text.
• Cause: Semantic quality was evaluated before admission and source-version invariants.
• Fix: Enforce authorization and freshness deterministically; apply judges only after evidence is admissible.