Capstone: Eval Dashboard

In the preceding capstone, you built document_qa_for_support_policies. It answered one required refund-policy question with a cited policy record, abstained when evidence was missing, and rejected a private-note prompt injection. Most importantly, it emitted one replayable evaluation row for each fixture, including dataset, run, and corpus identity.

This capstone starts from those rows. Your job isn't to make three green checks look impressive. Your job is to answer a harder question: when retrieval changes to handle paraphrased customer questions, can you see both the improvement and any safety regression before the support agent consumes it?

The change that needs a decision

The document-QA baseline intentionally left one upgrade target: it can find a policy when the wording overlaps strongly, but it may miss a paraphrase such as "Can a cracked tablet be refunded before a specialist reviews it?"

Suppose you add hybrid retrieval to improve recall. You now have three runs:

The dashboard needs to make the middle run impossible to mistake for an improvement. hybrid-v1 answers more questions, but it's a worse product because it crosses the evidence boundary.

That is the first design rule: a release decision is not the highest average score. It is a documented set of non-negotiable gates.

Extend the frozen fixture set fairly

You can't compare the baseline on three rows with a candidate on four rows and call the numbers comparable. Add the new paraphrase fixture to the frozen dataset, then rerun every version against that same set.

The original three rows remain the consumer contract from the preceding capstone. The fourth row measures a new capability. Rerunning the baseline on all four rows tells you whether the candidate actually fixed a known weakness.

Row contract

A dashboard row should preserve enough evidence for another engineer to recompute the grade and reject an invalid comparison. This is the smallest useful schema:

grader_version identifies the deterministic code contract. judge_rubric_version is null because this gate doesn't need a model judge. If you later add a clarity judge for already-grounded answers, stamp its rubric version on those rows. Notice what is not in this row: an unexplained quality score. For this product contract, status and citation behavior are objectively checkable. Don't ask a model judge whether citing a private note is acceptable when a deterministic rule already proves that it isn't.

The release path has two different stop conditions. First reject invalid comparisons. Only then ask whether a valid candidate receipt passes product gates.

Grade evidence before aggregating it

Write the grader before the dashboard. It compares observed behavior with the fixture contract and records failure codes that a UI can display.

The failures say more than a chart could. The baseline needs better recall. The first hybrid candidate needs its evidence boundary repaired. The second candidate passes the small contract, but that still doesn't authorize production traffic.

Aggregate without hiding safety slices

pass@1 means the first answer shown to the consumer passed its checks. For document QA, it's the primary metric because the refund support agent expects one response, not a basket of drafts.

The next script aggregates the graded outcomes. It repeats only the compact graded table, as a dashboard service would load it from JSON Lines or a database.

extractive-v1 and hybrid-v1 have the same overall score. They are not equally acceptable. One lacks a new capability; the other leaks a private-note instruction into customer-facing evidence. A dashboard that sorts candidates by the top-line percentage would hide the most important conclusion.

Write gates in product language

A passing metric is not yet a release policy. For this product, define these gates before looking at candidate results:

1. Every compared run must use the same dataset, deterministic grader, corpus snapshot, and exact fixture inventory.
2. supported_policy, unsupported_question, and untrusted_instruction must not regress.
3. A retrieval upgrade intended to fix paraphrases must pass paraphrase_recall.
4. A candidate that passes four teaching fixtures may proceed to an expanded offline evaluation, not production deployment.
5. Latency must be recorded for comparison, but a four-row sample is not a defensible production latency benchmark.

That fourth gate matters. If every observed row passes, a bootstrap interval can describe sensitivity to those four represented rows; it can't tell you about missing countries, policy versions, document formats, or attacks you never wrote down.

This decision is stricter than a glossy dashboard demo, and it's more credible. hybrid-v2 has earned a larger frozen eval set. It hasn't earned customer traffic.

Uncertainty needs coverage beside it

Bootstrap resampling estimates sensitivity to the rows you observed: sample rows with replacement, recompute the metric many times, and read an interval from the simulated scores.^[1] With four deliberately chosen teaching fixtures, one failure moves pass@1 by 25 percentage points. Treat the interval as a sensitivity warning, not as an estimate of production accuracy or permission to ship.

The hybrid-v2 interval is a trap if you read it carelessly. Every resample of four passing rows still passes, so the interval is 100% to 100%. It says nothing about cases absent from the set. Put both row count and coverage warnings beside intervals in the dashboard.

When pass@k belongs on this dashboard

HumanEval popularized pass@k for sampled code solutions: a task succeeds if at least one of k generated candidates passes its tests.^[2] A document-QA endpoint normally displays its first answer, so pass@1 is the honest headline metric here.

You may report pass@k as a secondary experiment if the serving system actually generates multiple answers and ranks them before exposing one. Don't let retries erase a safety failure. An answer citing private text is a failed attempt even if a later retry abstains correctly.

That distinction keeps metric design aligned with product behavior. Retries may improve benign recall after ranking is tested. They must not hide evidence-boundary violations.

Where judgment helps, and where it can't

The hard gates in this capstone are deterministic:

• Did the system abstain when it should?
• Did a grounded answer cite exactly the allowed policy document?
• Did the answer include the required policy condition?

Later, you might want to grade whether two grounded answers are equally clear for a support specialist. A model-based judge can help triage that fuzzy property, but only after deterministic safety checks pass. Comparative judge studies have documented position and verbosity biases, so preserve rubric version, randomize presentation order where relevant, and audit disagreements with humans.^[3]

A useful judge record stores verdict, reason_code, rubric version, and cited spans. It doesn't ask the judge to provide hidden step-by-step reasoning, and it never upgrades an answer that failed citation or abstention checks.

Build the dashboard around decisions

Your UI can be React, Streamlit, or a notebook report. Its first screen should have the same information regardless of framework:

A small API view model keeps the frontend honest. It should expose the decision and the evidence that produced it, instead of computing gates inside a chart component. The serialized view is also a contract, so stamp both its schema version and the release-policy version that produced the decision.

Turn a coverage warning into work

expand_offline_eval needs an actionable queue. Before release review, choose required slices with the support and policy owners, then collect and label enough examples in each one. The target counts below are a project plan, not a statistical guarantee.

The numbers force a useful conversation. If regional policies are high risk, fifteen cases may still be far too few. The dashboard should display the approved plan and its owner, not imply that any arbitrary threshold proves coverage.

Make a hold decision inspectable

Each failed gate should open the row that caused it. A drill-down payload can be tiny as long as it preserves expected behavior, observed evidence, and a repair hint.

Turn the boundary into a regression test

The most valuable dashboard behavior should also fail in continuous integration. This small assertion prevents an unsafe retrieval candidate from advancing even if its average improves later.

Package a reviewer can run

The artifact is more than a screenshot. Submit a small repository surface that proves every dashboard card came from stored evidence:

Add one test that would have stopped hybrid-v1: any untrusted_instruction row with a citation must block the candidate. That test is worth more than another decorative chart.

Practice: Try to fool the dashboard

Run the relevant cells again after each mutation. Revert one mutation before trying the next.

1. Remove private_note_injection from repaired_rows. Does the candidate advance? Why should missing safety evidence count as a hold?
2. Remove expected_documents, cited_documents, and answer from EvalRow. What review task becomes impossible if stored rows retain only passed and failure_codes?
3. Add one hundred easy passing rows to unsafe_rows, including a second passing private_note_injection row after its failed private-note row. Should a higher average or later duplicate change the release decision?
4. Change only hybrid-v2 to a new corpus or grader version. Can its percentage be compared directly with extractive-v1?

Carry the contract into the next capstone

The fine-tuned classifier in the next lesson predicts whether a support ticket should be escalated to a person. Its checks are different from document QA, but its dashboard row shape is familiar: version, slice, expected decision, actual decision, latency, and failure code.

Failure modes to catch

Submission checklist

A strong portfolio submission gives a reviewer concrete answers:

Evaluation rubric

• Strong: Reuses the document-QA row contract, compares runs on one frozen receipt, rejects identity or fixture drift, blocks evidence-boundary regressions, and explains why a small passing set earns more testing rather than deployment.
• Partial: Computes overall and slice results, but leaves comparison identity, failed-row drill-down, or coverage limits unclear.
• Weak: Starts with generic charts or judge scores, hides required abstentions in an average, or promotes a candidate without inspectable evidence.

Common failures

Optimizing recall past the evidence boundary

Symptom: Hybrid retrieval solves the paraphrase row but cites a seller's private note. Cause: Retrieval was upgraded before the admission rule was preserved. Fix: Make untrusted_instruction a hard gate and keep evidence filtering before retrieval.

Treating uncertainty as coverage

Symptom: Dashboard says 100% and shows a tight interval for four passing rows. Cause: Bootstrap results were read as proof about cases the dataset doesn't contain. Fix: Display slice inventory, row count, and expansion requirements beside the interval.

Separating the decision from its row

Symptom: An engineer sees hold but can't locate the answer or citation that caused it. Cause: Dashboard aggregated away evidence instead of linking to it. Fix: Store failure codes and make every failed gate open its exact row.

Comparing receipts that drifted

Symptom: Candidate and baseline percentages appear side by side even though one run used a newer corpus snapshot, grader, or fixture inventory. Cause: Dashboard grouped rows by run name without validating the comparison receipt first. Fix: Reject identity drift, missing fixtures, duplicates, and unexpected rows before calculating a release decision.

Self-check questions