LLM-as-a-Judge Evaluation

policy-answerer-v4-eval proved a hard fact: the current, permitted policy source requires a rollback runbook, not continued deployment. A claim ledger can block unsupported "keep deploying" advice. It can't decide which of two safe replies from a large language model (LLM) is clearer for the engineer.

Consider these two answers to Maya's payment-service incident:

Both respect the selected evidence. The remaining question is softer: does the added next step make the second reply more useful without making it wordy or confusing?

An LLM-as-a-judge uses another LLM as an evaluator for quality that can't be fully decided by an exact assertion. It can compare clarity, helpfulness, or tone under a rubric. It must not decide whether restricted context was allowed or whether a policy claim is supported. Those remain deterministic gates.

Zheng et al. found that strong LLM judges could exceed 80% agreement with human preferences on their MT-Bench and Chatbot Arena experiments. The same work reports position bias, verbosity bias, preference for model-like answers, and reasoning limitations. A judge is useful measurement equipment, not ground truth.^{[1]Reference 1Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}

Keep facts outside the judge

The boundary matters more than the model name. In a deploy-policy answer pipeline, different questions need different evaluators:

Only the third layer changes here, while the first two layers carry forward. The overview below shows the complete contract: deterministic gates decide eligibility, swapped comparisons test preference stability, and calibration plus bias probes decide whether the resulting metric may guide a release.

Separate retrieval, grounding, and answer relevance

For Retrieval-Augmented Generation (RAG) applications, evaluate three relationships separately:

1. Context Relevance: Evaluates whether the retrieved context is relevant and sufficient to answer the user's query. This isolates retrieval-quality problems from generation flaws.
2. Groundedness / Faithfulness: Evaluates whether the generated response is entirely supported by the retrieved context. A low groundedness score indicates the model is using its parametric memory to hallucinate claims not present in the retrieved documents.
3. Answer Relevance: Evaluates whether the final response directly addresses the user's original query. This detects cases where the model generates a factual but unhelpful or off-topic reply.

These measurements help distinguish retriever failures from generator failures. None replaces deterministic authorization, freshness, or claim-support checks.

Self-preference and same-family judge bias

LLM judges are conditional probability engines, not objective standards. Panickssery et al. found self-preference in several evaluated judge settings: evaluators could recognize and favor their own generations, even without explicit model labels. Effect size varied by model and task, so treat self-preference as a bias to measure rather than a universal ordering rule.^{[2]Reference 2LLM Evaluators Recognize and Favor Their Own Generations.https://arxiv.org/abs/2404.13076}

This bias can mask regressions during a model swap or upgrade. Mitigations include:

• Anonymize candidates: Strip model-specific markers, templates, or signatures before evaluation.
• Cross-model evaluation: Compare judges from model families different from the generators; a different family is a probe, not automatic neutrality.
• Calibrate with humans: Regularly compare the automated judge's scores against a human-graded gold dataset to measure drift.

Start with two supported answers

The lab uses an abbreviated hard gate so the boundary is visible in one screen. The previous lesson built the complete evidence-path validator; here we reuse its result and add one unsafe counterexample to prove it still wins over any soft score.

If a judge later says unsafe_continue sounds friendlier, the answer stays blocked. That invariant makes the judge safe to experiment with.

Choose the evaluator before writing the rubric

Not every evaluation question should be routed to an LLM. Choose the measurement tool from the decision you need to make.

Two soft-evaluation shapes matter here:

The DEP-27 example uses pairwise judging because brief and actionable are two safe variants of the same answer.

Write a rubric for the remaining question

A vague instruction such as "pick the better answer" lets the evaluator reward length, politeness, or formatting arbitrarily. A rubric should name what remains undecided after hard checks and include anchors for a tie.

G-Eval studied LLM evaluation with task-specific criteria and a form-filling output design. A criterion and a structured answer are easier to audit than a free-form impression.^{[3]Reference 3G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment.https://arxiv.org/abs/2303.16634}

The next cell builds the packet that would be sent to a model API. Notice two decisions:

1. Candidate names are anonymous slots, not model or prompt-version names.
2. Protected facts are displayed as already validated context, not handed to the judge for re-litigation.

In a deployed evaluator, serialize this packet, request structured output from the chosen judge model, and store the raw packet plus parsed verdict. Don't rely on a hidden prompt that can't be reproduced during a regression.

Treat the judge output as untrusted data

The judge is another model. Its JSON can be malformed, its evidence can be irrelevant, and its preference can contradict its own rationale. Parse and validate it just as you would validate a tool result from an agent.

The output above is a stored fixture, not proof that a particular hosted model will agree. The engineering problem is to make an evaluator run observable and testable before plugging in any provider.

A preference must survive swapping A and B

Pairwise comparison is useful because the evaluator chooses between two concrete alternatives. It also exposes position bias: a judge may prefer the first slot instead of the better reply. Zheng et al. identify this bias in LLM judging, so every pairwise comparison in this lab is run twice with the candidates swapped.^{[1]Reference 1Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}

The detail that matters is normalization. A verdict of B in the first pass and A in the swapped pass can represent the same underlying answer.

Keep those states separate in your report. An explicit tie is a valid rubric outcome, needs_human_review is an escalation, and unstable_after_swap is evidence that slot order changed a decisive preference.

Probe the biases you expect

One clean comparison doesn't establish that a judge is trustworthy. Build probe cases where an undesirable shortcut is easy to observe.

Length isn't only a hypothetical confounder. Length-Controlled AlpacaEval proposes a regression-based adjustment intended to answer what preference would have been if compared answers had equal length.^{[4]Reference 4Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators.https://arxiv.org/abs/2404.04475} In a local product eval, the smaller first step is to add same-information length probes and report when padding wins.

These fixtures model stored judge returns from two probes. The code doesn't pretend to detect bias from text alone; it asks whether the judge failed a case whose expected behavior you defined in advance.

This is a useful negative result. Releasing a judge because it produced pleasing scores would make the evaluation system worse. A failed probe tells you exactly what to repair.

Calibrate the measurement against people

Hard gates have test oracles. Soft judgments need a labeled calibration set: humans apply the same rubric to a representative sample, then the judge is scored against those labels.

Raw agreement is easy to understand, but can overstate reliability when one label dominates. Cohen's kappa corrects for agreement expected from each rater's label frequencies:^{[5]Reference 5A Coefficient of Agreement for Nominal Scaleshttps://doi.org/10.1177/001316446002000104}

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

Here, $p_o$ is observed agreement and $p_e$ is agreement expected from label prevalence. Kappa isn't a universal release threshold. Your baseline is human-human agreement on the same rubric and the same workflow slices.

This tiny calibration set is intentionally too small to approve a real metric. It shows the computation and demonstrates why a promising number alone can't release an evaluator.

An aggregate can now conceal the exact problem that requires attention. Report the calibration set by workflow slice before allowing the judge metric to guide any experiment.

For an actual evaluation program:

1. Freeze a rubric and collect human labels for easy wins, real ties, and known failures.
2. Include workflow slices such as rollback, access review, and address change.
3. Record human-human agreement before comparing the judge to people.
4. Re-run calibration after prompt, judge-model, rubric, or traffic-distribution changes.
5. Escalate slices where agreement or bias probes fail, even if aggregate agreement looks healthy.

Conversation quality still needs the trace

Once a developer conversation has multiple turns, a fluent final reply can conceal a bad evidence path. A judge packet should include relevant conversation turns, selected evidence identifiers, hard-gate outcomes, and the safe candidates being compared.

The next cell blocks a conversation before semantic judging if its trace isn't admissible. This is the same contract as the single-turn example, applied to a fuller packet.

Use judges offline before letting them guide changes

LLM judging is usually most defensible as an offline experiment metric: compare prompt versions or model releases over a frozen dataset, investigate disagreements, and let humans approve consequential changes. It's rarely a good reason to make a real-time policy decision for one engineer.

Define an explicit promotion contract. The numbers below are illustrative requirements for this lab, not universal industry thresholds:

A blocked promotion is the correct result. The lab has produced a useful candidate preference, but it hasn't established that its judge deserves to influence prompt selection across real developer workflows.

A practical evaluation report

When you implement this pattern in a real project, store a report with these sections:

The scientist's habit is to evaluate the evaluator. A judge score is one observation; a calibrated, stress-tested metric with recorded failure modes is evidence.

Mastery check

Mastery outcomes

Evaluation rubric

• Keeps policy truth in deterministic gates and sends only supported answers to the judge
• Parses structured verdicts as untrusted data and preserves ties, escalation, and swap instability separately
• Probes position and verbosity shortcuts before promoting the metric
• Compares judge decisions with human labels by workflow slice
• Blocks metric promotion when calibration, trace, or bias-probe evidence is inadequate

Follow-up questions

Common pitfalls

The judge is asked to authorize policy truth

• Symptom: A polished but unsupported keep-deploying reply receives a high score.
• Cause: The pipeline sends all answers to the judge before deterministic policy checks.
• Fix: Block inadmissible evidence and unsupported claims first; judge only remaining soft differences.

Pairwise wins follow answer position

• Symptom: A prompt variant wins when placed in slot A, then loses when placed in slot B.
• Cause: The evaluation reports one ordering and ignores position bias.
• Fix: Run both orderings, normalize to candidate identity, and record flips as unstable or route them to humans.

Longer replies win by repeating the same facts

• Symptom: Apologies and duplicated policy text improve judge score without helping the engineer.
• Cause: The rubric doesn't make concision measurable, and no length probe exists.
• Fix: Add same-information padding probes, track response length, and block metric promotion while padding wins.

A high aggregate agreement hides a weak slice

• Symptom: Overall calibration looks acceptable, but address-change replies are frequently misjudged.
• Cause: Evaluation reports only one aggregate number.
• Fix: Label and report agreement by workflow slice, then escalate or repair failed slices before release.

A demonstration becomes a release metric too early

• Symptom: Eight hand-picked cases become the quality check for a new prompt.
• Cause: The team treats a runnable example as a validation dataset.
• Fix: Write a promotion contract with calibration size, probe, trace, and human-review requirements.