LLM-as-a-Judge Evaluation

Maya's payment service crossed its rollback threshold. The selected DEP-27 policy requires a rollback runbook, not continued deployment, and the evidence checks have already confirmed that boundary. Now a large language model (LLM) judge compares two generated replies that preserve the same remedy. Which one would you want an on-call engineer to read first?

Read the two candidates before naming the evaluator:

Both respect the selected evidence. The remaining question is softer: does the second wording make the required action clearer without becoming wordy or confusing?

Make one prediction before we add a model to this path: if a third reply said “keep deploying,” should a high helpfulness score rescue it? No. Eligibility and policy truth need exact assertions; clarity, actionability, and concision are the remaining questions for a soft evaluator.

An LLM-as-a-judge uses another LLM as an evaluator for quality that can't be fully decided by an exact assertion. It can compare clarity, helpfulness, or tone under a rubric. It mustn't decide whether restricted context was allowed or whether a policy claim is supported. Those remain deterministic gates.

In their MT-Bench and Chatbot Arena study, Zheng et al. report over 80% agreement between GPT-4 judgments and human preferences, at the level of human-human agreement in their setup. The same paper reports position bias (most judges favored the first slot), verbosity bias, self-enhancement (preferring a model's own answers), and limited reasoning. A judge is useful measurement equipment, not ground truth.^{[1]Reference 1Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}

Keep facts outside the judge

Start by naming the decision, not the model. In this deploy-policy pipeline, each question has a different owner:

Only the third row changes in this lesson. The previous lesson split retrieval failures from generation failures with candidate recall, selected-context recall, claim support, and citation support.

Soft scores for context relevance, groundedness, and answer relevance can help with paraphrase-heavy residual risk, but they mustn't reopen authorization or DEP-27 policy truth.

⚠️ Common mistake: Claim support for release-critical policy stays deterministic (the claim ledger and source gates from RAG Evaluation for Reliable Answers). Judge "faithfulness" or groundedness scores are only for paraphrase-heavy residual risk after those hard gates pass. A soft faithfulness judge mustn't replace claim-to-source verification for DEP-27-style policy truth.

Three checkpoints make the contract explicit: deterministic gates decide eligibility, swapped comparisons test preference stability, and calibration plus bias probes decide whether the metric may guide a release.

Why should an exact required phrase or permission rule be checked outside an LLM judge?

Answer

Deterministic facts don't need probabilistic interpretation. Keep schema, exact evidence, and policy checks in code so the judge handles only the subjective dimensions that remain.

With eligibility separated from preference, we can now run the smallest useful lab: two safe answers and one answer that must never reach the judge.

Start with two supported answers

Before opening the code, predict the route: brief and actionable should pass, while unsafe_continue should be blocked. The lab uses an abbreviated hard gate so that boundary fits on one screen. The previous lesson built the complete evidence-path validator; here we reuse its result and add one unsafe counterexample to prove that no soft score can override it.

The output gives us the first invariant: only supported answers enter soft judging. If a judge later says unsafe_continue sounds friendlier, the answer still stays blocked, so we can experiment with the judge without moving the safety boundary.

The gate has reduced the problem to a safe pair. Before asking which reply wins, choose whether the evaluator should score one answer or compare two.

Choose the evaluator before writing the rubric

Not every evaluation question belongs to an LLM. Choose the measurement tool from the decision you need to make.

Zheng et al.'s paper names three shapes: pairwise comparison, single-answer grading, and reference-guided grading. This lab uses the first two as pairwise and pointwise shapes, while policy truth stays outside both.^{[1]Reference 1Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}

Here are the two soft-evaluation shapes that matter for the DEP-27 question:

Predict the choice before running the next cell. rollback authorization has an exact oracle, clearer supported reply compares two safe variants, and new exception policy has no tested rubric anchor. Their routes should be code, pairwise judge, and human review in that order.

The three printed routes follow the unresolved question, not a preferred vendor. With the evaluator shape chosen, the next job is to define what a win means.

Write a rubric for the remaining question

A vague instruction such as "pick the better answer" lets the evaluator reward length, politeness, or formatting arbitrarily. A rubric should name what remains undecided after hard checks and include anchors for a tie.

Turn those labels into observations a reviewer can repeat. After hard gates establish the rollback remedy, actionability passes when the reply names the immediate next step, is partial when the step is vague, and fails when no usable step appears.

Clarity passes when an engineer can identify outcome and action on first read, is partial when their relationship takes a second read, and fails when wording supports multiple actions. Concision passes when every sentence adds decision-useful detail, is partial when an extra sentence changes no action, and fails when repeated or ornamental text wins. A tie stays available whenever the difference is stylistic.

G-Eval asks a judge to generate evaluation steps and then fill a structured form for task-specific criteria. A named criterion plus a structured verdict is easier to audit than a free-form impression.^{[2]Reference 2G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment.https://arxiv.org/abs/2303.16634}

Freeze those anchors in the packet before sending it to a model API. Candidate names become anonymous slots, not model or prompt-version names. Protected facts appear as already validated context, so the judge can stay focused on preference instead of re-litigating authorization.

Before reading the output, predict what the packet should contain: the evidence version and hard-check result should remain visible for provenance, while literal names such as brief and actionable should be absent from the candidate slots.

The output confirms that separation: the packet carries provenance and an explicit verdict set, but the candidates are only A and B. In a deployed evaluator, serialize this packet, request structured output from the chosen judge model, and store the raw packet plus parsed verdict. Don't rely on a hidden prompt that can't be reproduced during a regression.

Treat the judge output as untrusted data

The judge is another model. Its JSON can be malformed, its evidence can be irrelevant, and its preference can contradict its own rationale. Parse and validate it as you would validate a tool result from an agent.

Predict the failure path: a decisive B with an evidence string instead of a list should be rejected before any winner is counted. The parser below treats the complete result as a contract, not the verdict field alone.

The output is a stored fixture, not proof that a particular hosted model will agree. It does prove that malformed evidence is stopped at the integration boundary. Make the evaluator run observable and testable before plugging in any provider.

Valid structure still says nothing about preference quality. The next check keeps the same two candidates but changes their display order.

A preference must survive swapping A and B

Pairwise comparison asks the evaluator to choose between two concrete alternatives. That makes position bias visible: a judge may prefer the first slot instead of the better reply. Zheng et al. identify this bias in LLM judging, so every pairwise comparison in this lab runs twice with candidates swapped.^{[1]Reference 1Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}

Predict the two verdicts before reading the figure. A stable judge might return B for (brief, actionable) and A for (actionable, brief). Those letters differ, but they identify the same reply. A slot-following judge would return A both times, which maps to two different replies and must be treated as unstable.

Translate each slot back to reply identity before aggregating. A verdict of B in the first pass and A in the swapped pass can represent the same underlying answer; comparing raw letters would falsely call that disagreement.

Keep those states separate in your report. An explicit tie is a valid rubric outcome, needs_human_review is an escalation, and unstable_after_swap is evidence that slot order changed a decisive preference.

That distinction gives operators a diagnosis, not only a score. Stable preference can continue to calibration, an explicit tie can remain unresolved, and an unstable pair becomes a regression case for the judge prompt or model.

Judge output prefers answer A, but after swapping display order it prefers the same screen position rather than the same answer. What does that show?

Answer

Changing answer order changes the verdict, so the first result isn't stable preference evidence. Repeat with swapped order and treat inconsistent pairs as biased or inconclusive.

Probe the biases you expect

One clean comparison doesn't establish that a judge is trustworthy. Build probe cases where an undesirable shortcut is easy to observe.

Predict the release outcome from the controlled changes. Position should pass because reply identity stays stable. Padding, leaked identity, and same-family preference should fail because each change gives the judge a shortcut unrelated to answer quality. Ambiguity should route to a tie or a reviewer rather than force a winner.

Self-preference and same-family judge bias

A judge is a conditional predictor, not an objective standard. Panickssery et al. found self-preference in several evaluated judge settings: evaluators could recognize and favor their own generations, even without explicit model labels. On their summarization probes, GPT-4 distinguished its own summaries from two other models and humans at 73.5% accuracy, and self-preference strength rose with self-recognition. Effect size varied by model and task, so treat self-preference as a bias to measure rather than a universal ordering rule.^{[3]Reference 3LLM Evaluators Recognize and Favor Their Own Generations.https://arxiv.org/abs/2404.13076}

This bias can mask regressions during a model swap or upgrade. Anonymize model-specific markers and templates, compare judges from a different model family as an additional probe, and calibrate the results against human labels. A cross-family judge isn't automatically neutral.

Length isn't only a hypothetical confounder. Length-Controlled AlpacaEval fits a regression so it can ask what the preference would have been if the two answers had equal length. That adjustment raised Spearman correlation with Chatbot Arena from 0.94 to 0.98 on their reported leaderboard snapshot.^{[4]Reference 4Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators.https://arxiv.org/abs/2404.04475} In a local product eval, the smaller first step is to add same-information length probes and report when padding wins.

These fixtures stand in for stored judge returns from probes wired into promotion. Before running the report, predict three failures: padding, identity, and self-preference. The code doesn't detect bias from text alone; it compares each observed result with behavior you defined in advance. Identity and self-preference sit in both the matrix and runnable report instead of being left as narrative claims.

Three failed probes are release evidence, not an inconvenience to hide. The padding failure points to a concision rubric or judge that rewards extra words; the identity failure points to leaked labels; the self-preference failure points to a judge-family risk. Position swap can pass while those other shortcuts still block promotion.

Calibrate the measurement against people

Hard gates have test oracles. Soft judgments need a labeled calibration set: people apply the same rubric to a representative sample, then the judge is scored against those labels.

Raw agreement is easy to understand, but it can overstate reliability when one label dominates. Cohen's kappa corrects for agreement expected from each rater's label frequencies:^{[5]Reference 5A Coefficient of Agreement for Nominal Scaleshttps://doi.org/10.1177/001316446002000104}

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

Here, $p_o$ is observed agreement and $p_e$ is agreement expected from label prevalence. Kappa isn't a universal release threshold. Your baseline is human-human agreement on the same rubric and the same workflow slices.

In this fixture, the judge and human labels agree on six of eight rows, so $p_o = 0.75$. Their label counts produce $p_e = 0.359$, which gives $\kappa \approx 0.610$. The arithmetic explains why kappa is lower than raw agreement; the eight rows are still far too small to approve a real metric.

Predict the printed result before running the cell: six matches, raw agreement 0.75, kappa near 0.610, and a warning that the sample is insufficient.

The aggregate hides where the misses happen. Rows r6 and r7 disagree inside retry_deploy, so inspect their evidence, criterion labels, slot order, and judge rationale before changing the prompt. Keep each disagreement as a diagnostic example instead of averaging it away.

Predict the slice report next: rollback should be perfect, while retry_deploy should land at 0.50 and become the only weak slice.

For a real evaluation program, freeze the rubric and collect labels for easy wins, genuine ties, and known failures across workflows such as rollback, access review, and retry-after-rollback. Measure human-human agreement before comparing the judge to people. Re-run after any prompt, judge-model, rubric, or traffic-distribution change, and escalate a weak slice even when the aggregate looks healthy.

A judge agrees with human labels 90% overall but misses most unsafe-answer cases. Is it calibrated for a safety release gate?

Answer

No. Overall agreement hides the critical slice. Measure confusion, precision, and recall against reviewed human labels for each release-critical category before trusting the judge there.

Conversation quality still needs the trace

Once a developer conversation has multiple turns, a fluent final reply can conceal a bad evidence path. A judge packet should include relevant conversation turns, selected evidence identifiers, hard-gate outcomes, and the safe candidates being compared.

The next cell blocks a conversation before semantic judging if its trace isn't admissible. This is the same contract as the single-turn example, applied to a fuller packet. When a bundle is blocked, keep the selected version and gate outcome in the record so a reviewer can reproduce the path. Don't hand those fields to the judge as a freshness question.

Predict the two routes before running it. The current policy should reach soft judging with its version preserved. The stale policy should stop before scoring, while its version and failed gate remain available for diagnosis.

The output keeps provenance attached to both outcomes. A reviewer can see why the current bundle was eligible and why the stale one never reached a semantic score.

Use judges offline before letting them guide changes

The trace is now safe to measure, but that doesn't make each judge result a live policy decision. Use judging first as an offline experiment metric: compare prompt versions or model releases over a frozen dataset, investigate disagreements, and let people approve consequential changes.

Define the promotion contract before collecting scores. The numbers below are illustrative requirements for this lab, not universal industry thresholds:

Predict the gate result from the table: hard checks and an escalation path pass, but eight calibration rows miss the 50-row minimum and three bias probes fail. Promotion should be blocked for exactly those two reasons.

A blocked promotion is the correct result. The lab has produced a useful candidate preference, but it hasn't established that its judge deserves to influence prompt selection across real developer workflows. The gate separates a promising observation from release evidence.

Even after those gaps close, agreement with reviewers isn't the last question. The next lesson asks whether the same judge routes equivalent requests evenly when wording or language variety changes.

A practical evaluation report

When you implement this pattern in a real project, store a report with these sections:

Treat that report as an operational handoff, not a dashboard screenshot. Keep the case_id, source and version, rubric and prompt versions, judge model version, slot order, raw response, parsed verdict, human label, workflow slice, and final disposition together. The application owner can then diagnose eligibility, the evaluation owner can inspect disagreement, and the release owner can see exactly why promotion passed or stopped.

A judge score is one observation. Experiment decisions need a calibrated, stress-tested metric and a record of its failure modes. Keep judging asynchronous or offline until those records and gates are in place; don't let one soft score change a live policy path.

Run the completed chain on one trace: hard gates choose eligible candidates, the rubric defines the comparison, anonymous order swaps test stability, probes expose shortcuts, human labels calibrate slices, and the promotion gate decides whether the metric may guide an experiment. The judge can rank supported wording. It can't grant permission, repair stale evidence, or ship itself.