LLM-as-a-Judge Evaluation

The previous lesson ended with policy-answerer-v4-eval proving a hard fact: the current, permitted policy source authorizes a replacement, not an immediate refund. A claim ledger can block an unsupported refund promise. It can't decide which of two safe replies from a large language model (LLM) is clearer for the customer.

Consider these two answers to Luna's refurbished-laptop case:

Both respect the selected evidence. The remaining question is softer: does the added next step make the second reply more useful without making it wordy or confusing?

An LLM-as-a-judge uses another LLM as an evaluator for quality that can't be fully decided by an exact assertion. It can compare clarity, helpfulness, or tone under a rubric. It must not decide whether restricted context was allowed or whether a policy claim is supported. Those remain deterministic gates.

Zheng et al. found that strong LLM judges could exceed 80% agreement with human preferences on their MT-Bench and Chatbot Arena experiments. The same work reports position, verbosity, self-enhancement, and reasoning limitations. A judge is useful measurement equipment, not ground truth.^[1]

Keep facts outside the judge

The boundary matters more than the model name. In a customer-support answer pipeline, different questions need different evaluators:

This lesson builds only the third layer, while carrying the first two layers forward.

Figure 1: Semantic judging starts after admissibility and claim support have passed. A judge score never reopens a blocked evidence path.

Start with two supported answers

The lab uses an abbreviated hard gate so the boundary is visible in one screen. The previous lesson built the complete evidence-path validator; here we reuse its result and add one unsafe counterexample to prove it still wins over any soft score.

If a judge later says unsafe_refund sounds friendlier, the answer stays blocked. That invariant makes the judge safe to experiment with.

Choose the evaluator before writing the rubric

Not every evaluation question should be routed to an LLM. Choose the measurement tool from the decision you need to make.

Write a rubric for the remaining question

A vague instruction such as "pick the better answer" lets the evaluator reward length, politeness, or formatting arbitrarily. A rubric should name what remains undecided after hard checks and include anchors for a tie.

G-Eval studied LLM evaluation with task-specific criteria and a form-filling output design. The practical lesson here is modest: a criterion and a structured answer are easier to audit than a free-form impression.^[2]

The next cell builds the packet that would be sent to a model API. Notice two decisions:

1. Candidate names are anonymous slots, not model or prompt-version names.
2. Protected facts are displayed as already validated context, not handed to the judge for re-litigation.

In a deployed evaluator, serialize this packet, request structured output from the chosen judge model, and store the raw packet plus parsed verdict. Don't rely on a hidden prompt that can't be reproduced during a regression.

Treat the judge output as untrusted data

The judge is another model. Its JSON can be malformed, its evidence can be irrelevant, and its preference can contradict its own rationale. Parse and validate it just as you would validate a tool result from an agent.

The output above is a stored fixture, not proof that a particular hosted model will agree. The engineering problem is to make an evaluator run observable and testable before plugging in any provider.

A preference must survive swapping A and B

Pairwise comparison is useful because the evaluator chooses between two concrete alternatives. It also exposes position bias: a judge may prefer the first slot instead of the better reply. Zheng et al. identify this bias in LLM judging, so every pairwise comparison in this lab is run twice with the candidates swapped.^[1]

The crucial detail is normalization. A verdict of B in the first pass and A in the swapped pass can represent the same underlying answer.

Keep those states separate in your report. An explicit tie is a valid rubric outcome, needs_human_review is an escalation, and unstable_after_swap is evidence that slot order changed a decisive preference.

Probe the biases you expect

One clean comparison doesn't establish that a judge is trustworthy. Build probe cases where an undesirable shortcut is easy to observe.

Length is not only a hypothetical confounder. Length-Controlled AlpacaEval proposes a regression-based adjustment intended to answer what preference would have been if compared answers had equal length.^[3] In a local product eval, the smaller first step is to add same-information length probes and report when padding wins.

The following fixtures model stored judge returns from two probes. The code doesn't pretend to detect bias from text alone; it asks whether the judge failed a case whose expected behavior you defined in advance.

This is a valuable negative result. Releasing a judge because it produced pleasing scores would make the evaluation system worse. A failed probe tells you exactly what to repair.

Calibrate the measurement against people

Hard gates have test oracles. Soft judgments need a labeled calibration set: humans apply the same rubric to a representative sample, then the judge is scored against those labels.

Raw agreement is easy to understand, but can overstate reliability when one label dominates. Cohen's kappa corrects for agreement expected from each rater's label frequencies:^[4]

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

Here, $p_o$ is observed agreement and $p_e$ is agreement expected from label prevalence. Kappa isn't a universal release threshold. Your baseline is human-human agreement on the same rubric and the same workflow slices.

This tiny calibration set is intentionally too small to approve a real metric. It shows the computation and demonstrates why a promising number alone can't release an evaluator.

An aggregate can now conceal the exact problem that requires attention. Report the calibration set by workflow slice before allowing the judge metric to guide any experiment.

For an actual evaluation program:

1. Freeze a rubric and collect human labels for easy wins, real ties, and known failures.
2. Include workflow slices such as replacement, delivery delay, and address change.
3. Record human-human agreement before comparing the judge to people.
4. Re-run calibration after prompt, judge-model, rubric, or traffic-distribution changes.
5. Escalate slices where agreement or bias probes fail, even if aggregate agreement looks healthy.

Conversation quality still needs the trace

Once a support conversation has multiple turns, a fluent final reply can conceal a bad evidence path. A judge packet should include relevant conversation turns, selected evidence identifiers, hard-gate outcomes, and the safe candidates being compared.

The next cell blocks a conversation before semantic judging if its trace isn't admissible. This is the same contract as the single-turn example, applied to a fuller packet.

Use judges offline before letting them guide changes

LLM judging is usually most defensible as an offline experiment metric: compare prompt versions or model releases over a frozen dataset, investigate disagreements, and let humans approve consequential changes. It is rarely a good reason to make a real-time policy decision for one customer.

Define an explicit promotion contract. The numbers below are illustrative requirements for this lab, not universal industry thresholds:

A blocked promotion is the correct result. The lab has produced a useful candidate preference, but it hasn't established that its judge deserves to influence prompt selection across real customer workflows.

A practical evaluation report

When you implement this pattern in a real project, store a report with these sections:

The scientist's habit is to evaluate the evaluator. A judge score is one observation; a calibrated, stress-tested metric with recorded failure modes is evidence.

Mastery check

What you can now do

Evaluation rubric

• Keeps policy truth in deterministic gates and sends only supported answers to the judge
• Parses structured verdicts as untrusted data and preserves ties, escalation, and swap instability separately
• Probes position and verbosity shortcuts before promoting the metric
• Compares judge decisions with human labels by workflow slice
• Blocks metric promotion when calibration, trace, or bias-probe evidence is inadequate

Follow-up questions

Common pitfalls

The judge is asked to authorize policy truth

Symptom: A polished but unsupported refund reply receives a high score. Cause: The pipeline sends all answers to the judge before deterministic policy checks. Fix: Block inadmissible evidence and unsupported claims first; judge only remaining soft differences.

Pairwise wins follow answer position

Symptom: A prompt variant wins when placed in slot A, then loses when placed in slot B. Cause: The evaluation reports one ordering and ignores position bias. Fix: Run both orderings, normalize to candidate identity, and record flips as unstable or route them to humans.

Longer replies win by repeating the same facts

Symptom: Apologies and duplicated policy text improve judge score without helping the customer. Cause: The rubric doesn't make concision measurable, and there is no length probe. Fix: Add same-information padding probes, track response length, and block metric promotion while padding wins.

A high aggregate agreement hides a weak slice

Symptom: Overall calibration looks acceptable, but address-change replies are frequently misjudged. Cause: Evaluation reports only one aggregate number. Fix: Label and report agreement by workflow slice, then escalate or repair failed slices before release.

A demonstration becomes a release metric too early

Symptom: Eight hand-picked cases become the quality gate for a new prompt. Cause: The team treats a runnable example as a validation dataset. Fix: Write a promotion contract with calibration size, probe, trace, and human-review requirements.