Capstone: Eval Dashboard

In the preceding capstone, you built document_qa_for_access_policies. It answered one required access-policy question with a cited policy record, abstained when evidence was missing, and rejected a private-note prompt injection. It also emitted one replayable evaluation row for each fixture, including dataset, run, and corpus identity.

This capstone starts from those rows. Your job isn't to make three green checks look impressive. Your job is to answer a harder question: when retrieval changes to handle paraphrased operator questions, can you see both the improvement and any safety regression before the access agent consumes it?

The change that needs a decision

The document-QA baseline intentionally left one upgrade target: it can find a policy when the wording overlaps strongly, but it may miss a paraphrase such as "Can a locked production API key regain access after identity verification?"

Suppose hybrid retrieval is added to improve recall. The dashboard compares three runs:

The dashboard needs to make the middle run impossible to mistake for an improvement. hybrid-v1 answers more questions, but it's a worse product because it crosses the evidence boundary.

That's the first design rule: a release decision isn't the highest average score. It's a documented set of non-negotiable gates.

Extend the frozen fixture set fairly

You can't compare the baseline on three rows with a candidate on four rows and call the numbers comparable. Add the new paraphrase fixture to the frozen dataset, then rerun every version against that same set.

The original three rows remain the consumer contract from the preceding capstone. The fourth row measures a new capability. Rerunning the baseline on all four rows tells you whether the candidate fixed a known weakness.

Row contract

A dashboard row should preserve enough evidence for another engineer to recompute the grade and reject an invalid comparison. This is the smallest useful schema:

grader_version identifies the deterministic code contract. judge_rubric_version is null because this gate doesn't need a model judge. If you later add a clarity judge for already-grounded answers, stamp its rubric version on those rows. Notice what isn't in this row: an unexplained quality score. For this product contract, status and citation behavior are objectively checkable. Don't ask a model judge whether citing a private note is acceptable when a deterministic rule already proves that it isn't.

Why must the baseline run on the new paraphrase fixture too?

Answer

A candidate and baseline are comparable only on the same frozen cases and grading rules. Otherwise an apparent gain could come from changing the test rather than improving the system.

The release path has two different stop conditions. First reject invalid comparisons. Only then ask whether a valid candidate receipt passes product gates.

Grade evidence before aggregating it

Write the grader before the dashboard. It compares observed behavior with the fixture contract and records failure codes that a UI can display.

The failures say more than a chart could. Baseline recall needs work, and the first hybrid candidate has to repair its evidence boundary. The second candidate passes the small contract, but that still doesn't authorize production traffic.

Aggregate without hiding safety slices

pass@1 means the first answer shown to the consumer passed its checks. For document QA, it's the primary metric because the access agent expects one response, not a basket of drafts.

The next script aggregates the graded outcomes. It repeats only the compact graded table, as a dashboard service would load it from JSON Lines or a database.

extractive-v1 and hybrid-v1 have the same overall score. They aren't equally acceptable. One lacks a new capability; the other leaks a private-note instruction into operator-facing evidence. A dashboard that sorts candidates by the top-line percentage would hide the most important conclusion.

Write gates in product language

A passing metric isn't yet a release policy. For this product, define these gates before looking at candidate results:

1. Every compared run must use the same dataset, deterministic grader, corpus snapshot, and exact fixture set.
2. supported_policy, unsupported_question, and untrusted_instruction must not regress.
3. A retrieval upgrade intended to fix paraphrases must pass paraphrase_recall.
4. A candidate that passes four teaching fixtures may proceed to an expanded offline evaluation, not production deployment.
5. Latency must be recorded for comparison, but a four-row sample isn't a defensible production latency benchmark.

That fourth gate matters. If every observed row passes, a bootstrap interval can describe sensitivity to those four represented rows; it can't tell you about missing countries, policy versions, document formats, or attacks you never wrote down.

This decision is stricter than a glossy dashboard demo, and it's more credible. hybrid-v2 has earned a larger frozen eval set. It hasn't earned operator traffic.

Uncertainty needs coverage beside it

Bootstrap resampling estimates sensitivity to the rows you observed: sample rows with replacement, recompute the metric many times, and read an interval from the simulated scores.^{[1]Reference 1Bootstrap Methods: Another Look at the Jackknife.https://projecteuclid.org/journals/annals-of-statistics/volume-7/issue-1/Bootstrap-Methods--Another-Look-at-the-Jackknife/10.1214/aos/1176344552.full} With four deliberately chosen teaching fixtures, one failure moves pass@1 by 25 percentage points. Treat the interval as a sensitivity warning, not as an estimate of production accuracy or permission to ship.

The hybrid-v2 interval is a trap if you read it carelessly. Every resample of four passing rows still passes, so the interval is 100% to 100%. It says nothing about cases absent from the set. Put both row count and coverage warnings beside intervals in the dashboard.

Why does a 100% to 100% bootstrap interval not prove hybrid-v2 is ready for production?

Answer

The interval resamples only four already-passing fixtures. It can't reveal missing policy versions, regions, formats, or adversarial cases, so it measures sample stability without proving coverage.

When pass@k belongs on this dashboard

HumanEval popularized pass@k for sampled code solutions: a task succeeds if at least one of k generated candidates passes its tests.^{[2]Reference 2Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374} A document-QA endpoint normally displays its first answer, so pass@1 is the honest headline metric here.

You may report pass@k as a secondary experiment if the serving system generates multiple answers and ranks them before exposing one. Don't let retries erase a safety failure. An answer citing private text is a failed attempt even if a later retry abstains correctly.

That distinction keeps metric design aligned with product behavior. Retries may improve benign recall after ranking is tested. They must not hide evidence-boundary violations.

Where judgment helps, and where it can't

The hard gates in this capstone are deterministic:

• Did the system abstain when it should?
• Did a grounded answer cite exactly the allowed policy document?
• Did the answer include the required policy condition?

Later, you might want to grade whether two grounded answers are equally clear for a support specialist. A model-based judge can help triage that fuzzy property, but only after deterministic safety checks pass. Comparative judge studies have documented position and verbosity biases, so preserve rubric version, randomize presentation order where relevant, and audit disagreements with humans.^{[3]Reference 3Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}

A useful judge record stores verdict, reason_code, rubric version, and cited spans. It doesn't ask the judge to provide hidden step-by-step reasoning, and it never upgrades an answer that failed citation or abstention checks.

Build the dashboard around decisions

Your UI can be React, Streamlit, or a notebook report. Its first screen should have the same information regardless of framework:

A small API view model keeps the frontend honest. It should expose the decision and the evidence that produced it, instead of computing gates inside a chart component. The serialized view is also a contract, so stamp both its schema version and the release-policy version that produced the decision.

Turn a coverage warning into work

expand_offline_eval needs an actionable queue. Before release review, choose required slices with the support and policy owners, then collect and label enough examples in each one. The target counts below are a project plan, not a statistical guarantee.

The numbers force a useful conversation. If regional policies are high risk, fifteen cases may still be far too few. The dashboard should display the approved plan and its owner, not imply that any arbitrary threshold proves coverage.

Make a hold decision inspectable

Each failed gate should open the row that caused it. A drill-down payload can be tiny as long as it preserves expected behavior, observed evidence, and a repair hint.

Turn the boundary into a regression test

The most important dashboard behavior should also fail in continuous integration. This small assertion prevents an unsafe retrieval candidate from advancing even if its average improves later.

Package a reviewer can run

The artifact is more than a screenshot. Submit a small repository surface that proves every dashboard card came from stored evidence:

Add one test that would have stopped hybrid-v1: any untrusted_instruction row with a citation must block the candidate. That test is worth more than another decorative chart.

Practice: try to fool the dashboard

Run the relevant cells again after each mutation. Revert one mutation before trying the next.

1. Remove private_note_injection from repaired_rows. Does the candidate advance? Why should missing safety evidence count as a hold?
2. Remove expected_documents, cited_documents, and answer from EvalRow. What review task becomes impossible if stored rows retain only passed and failure_codes?
3. Add one hundred easy passing rows to unsafe_rows, including a second passing private_note_injection row after its failed private-note row. Should a higher average or later duplicate change the release decision?
4. Change only hybrid-v2 to a new corpus or grader version. Can its percentage be compared directly with extractive-v1?

What should each mutation teach you?

Answer

1. The candidate stays blocked. A required safety row that vanished in collection or aggregation is missing evidence, not evidence of a pass.
2. A reviewer can't recompute the grade or inspect the cited source. Keep expected behavior and observed evidence on every stored row.
3. The candidate stays blocked. A hard evidence-boundary failure can't be averaged away by adding easy successes.
4. The percentages aren't directly comparable until both runs execute the same frozen dataset and corpus snapshot under the same deterministic grader.

Carry the contract into the next capstone

The fine-tuned classifier in the next lesson predicts whether an access ticket should be escalated to a person. Its checks are different from document QA, but its dashboard row shape is familiar: version, slice, expected decision, actual decision, latency, and failure code.

Failure modes to catch

Submission checklist

A strong portfolio submission gives a reviewer concrete answers: