Evaluating AI Agents

In the previous lesson, you built refund-feedback-v12 and protected a frozen set named refund-eval-v5. That frozen set becomes useful now. A better preference dataset can improve a candidate model, but only an agent evaluation can answer the release question: when the model may use tool calling to change order state, does it finish the right task without breaking policy?

ShopFlow's refund assistant is a good test case. A polished reply means little if the agent issued an unauthorized refund, skipped an accessibility path, retried until the bill exploded, or failed only when a tool timed out. You need evidence about the complete run.

An episode is one isolated task with its starting state, user request, allowed actions, and expected checks. A trajectory is the observable record of one attempt: tool calls, redacted arguments, observations, approvals, final state, latency, and cost. Agent evaluation grades an episode from that evidence, not from private chain-of-thought.

Turn frozen examples into executable episodes

The earlier dataset lesson separated training records from frozen evaluation records. That split prevents memorization from looking like improvement. For an agent, each frozen record now needs an environment setup and assertions about what may change.

Use three small ShopFlow episodes throughout this chapter:

An episode isn't a preferred response. It's a test contract:

Treat allowed and forbidden tools as a permission boundary, not as suggestions in a prompt. Enforce authority outside the model and grade any attempted bypass as a failure.

The test loop should reset state, run the candidate, capture its trace, and score hard gates before any softer judgment:

Key insight: The training artifact taught the model. The frozen episode suite judges the agent that wraps the model, its tools, prompts, permissions, and recovery behavior.

Grade outcomes before style

Agent evaluation becomes much clearer when metrics have roles. Some evidence can block a release. Other evidence helps you diagnose or choose among safe candidates.

A weighted average is dangerous here. A beautiful reply shouldn't compensate for issue_refund appearing in a trace where refund authority was absent. Set hard constraints first; rank only candidates that satisfy them.

Record observable evidence, not private reasoning

An agent trace should contain the facts needed to replay and grade a run:

This schema intentionally doesn't request hidden reasoning. Reasoning text may be unavailable, unfaithful, private, or unsafe to retain. Actions and resulting state are stronger evidence: you can determine whether a refund occurred, whether approval existed, and whether the candidate verified its write.

Before scoring behavior, reject traces that can't become safe evaluation evidence. The trace validator below requires a candidate ID for reproducibility and rejects argument keys reserved for raw customer identifiers.

Now score behavior. The code below defines three episode contracts, feeds it one run for each episode, and refuses the injected refund attempt even though it could have produced a polite response.

The first two runs satisfy their final-state and tool-contract gates. The third doesn't get partial credit: issue_refund is a forbidden side effect in attack-014, and the final state is wrong.

Process checks are most useful when they explain a failure. Here one run recovers from a temporary policy lookup timeout and verifies its write. Another burns its retry budget without producing state evidence. The third verifies stale state before its write, which doesn't prove the write succeeded.

Keep outcome, safety, and economics separate

Once every run has a verdict, aggregate the run set without hiding critical failures. Cost per successful task (CPST) is total evaluation cost divided by the number of successful episodes. It's useful, but it doesn't forgive harm.

For three runs with costs 0.032, 0.038, and 0.041, total cost is 0.111. Two episodes pass, so:

$$\text{CPST} = \frac{\$0.111}{2} = \$0.0555$$

The following small report computes that number and retains the safety failure as a separate count.

If a cheaper agent fails more cases, CPST may still decrease. That doesn't establish product value. Failed refunds can require human review, delay a return, or create unauthorized money movement. Track remediation cost and safety separately rather than folding everything into one friendly number.

Evaluate repeatability, not one lucky trace

Agents are stochastic: sampling, tool errors, and observation ordering can change a run. A candidate that passes once and fails the next time isn't ready for a consequential action.

Two metrics answer different questions:

For sampled code solutions, the unbiased pass@k estimator from HumanEval is based on how many of n samples pass.^[1] Tau-Bench introduced pass^k to make repeated reliability visible for tool-using support agents.^[2] They point in opposite directions: more attempts help pass@k, while more required clean reruns make pass^k stricter.

This runnable example calculates each protocol independently. Candidate patches use five sampled attempts for pass@3. Refund-agent reliability uses three reruns per episode and counts an episode only if all three are safe successes.

pass@3 looks high because a verifier can choose one good patch from several attempts. The refund agent's pass^3 is low because two customer episodes fail at least once. The latter is the release warning.

Three episodes make failures concrete, but they don't yield a precise estimate of production success. Report uncertainty as the suite grows. A Wilson interval is useful for a binary pass rate because it behaves sensibly with modest sample sizes. With z = 1.96, the example below reports a two-sided 95% interval.

The pilot is excellent for debugging, not for a confident release estimate. Expanding the suite narrows uncertainty, but safety-critical failures still block directly rather than waiting for a confidence interval.

Use judges only where deterministic checks stop

Some quality dimensions aren't database fields. Was the refusal clear? Did the specialist handoff explain what happens next? A human rubric can label those messages, and a model judge can help scale routine scoring.

Judges need calibration. Studies of model-based judging document position and verbosity biases, so an untested judge shouldn't decide whether a risky tool action was acceptable.^[3] In this lesson:

• Hard facts stay deterministic: tool permissions, final state, PII redaction, timeout, cost.
• Soft communication quality may use a judge after comparison with human labels.
• Swapping response order tests whether pairwise judgments are stable.
• Any unsafe tool action overrides a good communication score.

The following calibration fixture includes four human-labeled comparisons. The swapped judgment is normalized back to the original A or B identity before comparison. A judge that changes its winner when display order swaps remains advisory.

This judge can still surface cases for human review. It can't promote a candidate or override issue_refund without authority.

Match public benchmarks to the behavior you ship

Private episodes answer "may we release this ShopFlow change?" Public benchmarks answer narrower comparative questions. Their value depends on matching the tested surface to the product surface.

AgentBench helped establish broad interactive evaluation across multiple environments, but a production scorecard still has to choose tests that resemble its actual permissions and failures.^[10] A support agent shouldn't claim readiness from a coding leaderboard, and a coding agent shouldn't claim readiness from a browser task.

Run each episode in a clean environment

An agent evaluation isn't reproducible if the second run inherits the first run's writes. If damaged-221 already has a label because the previous attempt created one, the next candidate may appear to succeed without calling any tool.

A practical harness has five stages:

1. Load a frozen episode and its expected checks.
2. Start an isolated sandbox or disposable test tenant from a known snapshot.
3. Run the candidate with tool permissions, timeout, and spending budget enforced outside the model.
4. Export a redacted trace plus final state.
5. Destroy the environment, then score the exported evidence.

For a local unit test, an in-memory state reset is enough to make the rule visible:

In a real harness, the same principle means disposable databases, sandboxed filesystems, fake payment tools, bounded network access, and replayable tool responses. Don't run autonomous write-capable evals against a personal machine or live customer state.

Produce a release report that can say no

An evaluation report should be a versioned artifact, just like the feedback dataset that produced the candidate. Include:

This final gate uses the metrics produced above. Candidate v7 must be blocked because a single critical refund bypass is enough, even before reliability and judge calibration are considered.

The next engineering task isn't tuning the judge until the score turns green. Repair the candidate so attack-014 opens a security review without attempting a refund, then rerun the unchanged frozen suite.

The repaired trace below makes that delta concrete. v8 changes the action for attack-014, then earns promotion only after the same frozen cases pass repeatedly.

Build the ShopFlow evaluation artifact

Complete the chapter by turning the ten runnable fragments into a small evaluation package:

1. Write three episode JSON records for damaged-221, appeal-009, and attack-014, including initial state, tool permissions, expected final state, and budgets.
2. Capture observable trace fields only: redacted tool calls, observations, approvals, state snapshots, timing, and cost.
3. Make one deliberately unsafe candidate trace fail because it calls issue_refund for attack-014.
4. Run at least three independent attempts per episode, then report both ordinary success rate and pass^3.
5. Label four final messages with a human rubric and test a judge under response-order swaps.
6. Emit a release report for refund-agent-v7 that blocks promotion and lists exact reasons.
7. Repair the candidate behavior, rerun the same frozen suite, and explain which evidence changed.

What a strong submission contains

• Frozen episode fixtures that never enter training.
• A deterministic evaluator for final state, forbidden tools, verification, timeout, and spend.
• An aggregate report that never averages away a critical safety failure.
• A repeatability result that distinguishes safe default behavior from best-of-many rescue.
• A calibrated, limited role for any model judge.
• One failing trace and one repaired trace that a reviewer can reproduce.

Mastery Check

You are ready to evaluate an agent when you can:

• Turn a frozen feedback case into an isolated episode with explicit final-state and authorization checks.
• Explain why observable trajectories are more appropriate evaluation evidence than hidden reasoning.
• Keep hard safety gates separate from cost and communication diagnostics.
• Compute CPST without claiming it captures harm or remediation cost.
• Explain the difference between pass@k search capacity and pass^k repeatability.
• Calibrate a message judge against humans and order swaps before relying on its scores.
• Choose a public benchmark by task surface while keeping private episodes as the release gate.
• Write a candidate report that blocks an unsafe but fluent agent.

Evaluation rubric

• Foundational: Defines an episode, trajectory, final-state assertion, and forbidden tool check for a refund case.
• Intermediate: Executes a deterministic scorecard and produces a cost-aware report from multiple traces.
• Applied: Adds rerun reliability, sandbox reset, and judge calibration without weakening hard gates.
• Advanced: Maintains a private suite and explains how public benchmark slices complement, but don't replace, product release evidence.

Common pitfalls