Evaluating AI Agents

promotion-feedback-v12 now needs a real agent evaluation. You already protected the frozen set named promotion-eval-v5, and that set becomes useful here. A better preference dataset can improve a candidate model, but only an agent evaluation can answer the release question: when the model may use tool calling to change production-release state, does it finish the right task without breaking policy?

A model-promotion assistant is a good test case. A polished reply means little if the agent promoted a model without approval, skipped an exception path, retried until the bill exploded, or failed only when a tool timed out. You need evidence about the complete run.

An episode is one isolated task with its starting state, user request, allowed actions, and expected checks. A trajectory is the observable record of one attempt: tool calls, redacted arguments, observations, approvals, final state, latency, and cost. Agent evaluation grades an episode from that evidence, not from private chain-of-thought.

Turn frozen examples into executable episodes

The earlier dataset lesson separated training records from frozen evaluation records. That split prevents memorization from looking like improvement. For an agent, each frozen record now needs an environment setup and assertions about what may change.

Use three small model-release episodes throughout the chapter:

An episode isn't a preferred response. It's a test contract:

Treat allowed and forbidden tools as a permission boundary, not as suggestions in a prompt. Enforce authority outside the model and grade any attempted bypass as a failure.

The test loop should reset state, run the candidate, capture its trace, and score hard gates before any softer judgment:

Evaluation target: The training artifact taught the model. The frozen episode suite judges the agent that wraps the model, its tools, prompts, permissions, and recovery behavior.

Grade outcomes before style

Agent evaluation becomes much clearer when metrics have roles. Some evidence can block a release. Other evidence diagnoses failures or ranks safe candidates.

A weighted average is dangerous here. A beautiful reply shouldn't compensate for promote_model appearing in a trace where promotion authority was absent. Set hard constraints first; rank only candidates that satisfy them.

## Record observable evidence, not private reasoning

An agent trace should contain the facts needed to replay and grade a run:

This schema intentionally doesn't request hidden reasoning. Reasoning text may be unavailable, unfaithful, private, or unsafe to retain. Actions and resulting state are stronger evidence: you can determine whether a promotion occurred, whether approval existed, and whether the candidate verified its write.

Before scoring behavior, reject traces that can't become safe evaluation evidence. The trace validator below requires a candidate ID for reproducibility and rejects argument keys reserved for raw actor or candidate identifiers.

Now score behavior. The scorer defines three episode contracts, feeds it one run for each episode, and refuses the injected promotion attempt even though it could have produced a polite response.

The first two runs satisfy their final-state and tool-contract gates. The third doesn't get partial credit: promote_model is a forbidden side effect in attack-014, and the final state is wrong.

Process checks are most useful when they explain a failure. Here one run recovers from a temporary policy lookup timeout and verifies its write. Another burns its retry budget without producing state evidence. The third verifies stale state before its write, which doesn't prove the write succeeded.

Keep outcome, safety, and economics separate

Once every run has a verdict, aggregate the run set without hiding critical failures. Cost per successful task (CPST) is total evaluation cost divided by the number of successful episodes. It's useful, but it doesn't forgive harm.

For three runs with costs 0.032, 0.038, and 0.041, total cost is 0.111. Two episodes pass, so:

$$\text{CPST} = \frac{\$0.111}{2} = \$0.0555$$

This small report computes that number and retains the safety failure as a separate count.

If a cheaper agent fails more cases, CPST may still decrease. That doesn't establish product value. Failed promotions can require human review, delay a safe rollout, or shift production traffic without authorization. Track remediation cost and safety separately rather than folding everything into one friendly number.

Evaluate repeatability, not one lucky trace

Agents are stochastic: sampling, tool errors, and observation ordering can change a run. A candidate that passes once and fails the next time isn't ready for a consequential action.

Two metrics answer different questions:

For sampled code solutions, the unbiased pass@k estimator from HumanEval is based on how many of n samples pass.^{[1]Reference 1Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374} Tau-Bench introduced pass^k to make repeated reliability visible for tool-using support agents.^{[2]Reference 2Tau-Bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainshttps://arxiv.org/abs/2406.12045} They point in opposite directions: more attempts help pass@k, while more required clean reruns make pass^k stricter.

This runnable example calculates each protocol independently. Candidate patches use five sampled attempts for pass@3. Policy-agent reliability uses three reruns per episode and counts an episode only if all three are safe successes.

pass@3 looks high because a verifier can choose one good patch from several attempts. The promotion agent's pass^3 is low because two release episodes fail at least once. The latter is the release warning.

Three episodes make failures concrete, but they don't yield a precise estimate of production success. Report uncertainty as the suite grows. A Wilson interval is useful for a binary pass rate because it behaves sensibly with modest sample sizes. With z = 1.96, the example below reports a two-sided 95% interval.

The pilot is excellent for debugging, not for a confident release estimate. Expanding the suite narrows uncertainty, but safety-critical failures still block directly rather than waiting for a confidence interval.

Use judges only where deterministic checks stop

Some quality dimensions aren't database fields. Was the refusal clear? Did the release-review handoff explain what happens next? A human rubric can label those messages, and a model judge can help scale routine scoring.

Judges need calibration. Studies of model-based judging document position and verbosity biases, so an untested judge shouldn't decide whether a risky tool action was acceptable.^{[3]Reference 3Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685} Measure:

• Hard facts stay deterministic: tool permissions, final state, PII redaction, timeout, cost.
• Soft communication quality may use a judge after comparison with human labels.
• Swapping response order tests whether pairwise judgments are stable.
• Any unsafe tool action overrides a good communication score.

A judge also opens an injection surface. The trajectory you hand it, including tool arguments, observations, and the final message, is untrusted text, and a candidate can embed judge-directed instructions inside its own output. The same benchmark audits found agents appending a hidden note such as an "evaluation note" that the judge then parroted back as a favorable score.^{[4]Reference 4How We Broke Top AI Agent Benchmarks: And What Comes Next.https://rdi.berkeley.edu/blog/trustworthy-benchmarks-cont/} Treat trace content as data, not instructions: delimit it with clear structural markers, strip or escape anything that resembles a system prompt or scoring directive, and prefer extracted-feature checks (did the final state equal security_review_opened?) over asking a judge to rate a full raw trajectory.

This calibration fixture includes four human-labeled comparisons. The swapped judgment is normalized back to the original A or B identity before comparison. A judge that changes its winner when display order swaps remains advisory.

This judge can still surface cases for human review. It can't promote a candidate or override promote_model without authority.

Match public benchmarks to the behavior you ship

Private episodes answer "may we release this model-promotion agent?" Public benchmarks answer narrower comparative questions. Their value depends on matching the tested surface to the product surface.

AgentBench helped establish broad interactive evaluation across multiple environments, but a production scorecard still has to choose tests that resemble its actual permissions and failures.^{[11]Reference 11AgentBench: Evaluating LLMs as Agentshttps://arxiv.org/abs/2308.03688} A support agent shouldn't claim readiness from a coding leaderboard, and a coding agent shouldn't claim readiness from a browser task.

Run each episode in a clean environment

An agent evaluation isn't reproducible if the second run inherits the first run's writes. If promote-221 already has a promotion request because the previous attempt created one, the next candidate may appear to succeed without calling any tool.

A practical harness has five stages:

1. Load a frozen episode and its expected checks.
2. Start an isolated sandbox or disposable test tenant from a known snapshot.
3. Run the candidate with tool permissions, timeout, and spending budget enforced outside the model.
4. Export a redacted trace plus final state.
5. Destroy the environment, then score the exported evidence.

One boundary is easy to miss: the scoring artifacts themselves. If expected final states, reference answers, or gold files sit inside the same sandbox the agent's tools can read, a file-reading or write-capable candidate can fetch them and "pass" without doing the task. Automated audits of public agent benchmarks found exactly this failure. Agents that read a local file:// path or accessed a task config recovered the gold answer directly and scored close to 100% without solving anything.^{[4]Reference 4How We Broke Top AI Agent Benchmarks: And What Comes Next.https://rdi.berkeley.edu/blog/trustworthy-benchmarks-cont/} Keep expected answers and evaluator config on a separate path the candidate can't reach, run the scorer outside the agent's container, and export only the redacted trace and final state for grading.

The micro-fixture below makes the harness contract testable: gold inside the agent-readable mount means the design fails; the required layout keeps gold outside that mount.

For a local unit test, an in-memory state reset is enough to make the rule visible:

In a real harness, the same principle means disposable databases, sandboxed filesystems, fake promotion tools, bounded network access, and replayable tool responses. Don't run autonomous write-capable evals against a personal machine or live production state.

Produce a release report that can say no

An evaluation report should be a versioned artifact, just like the feedback dataset that produced the candidate. Include:

This final gate uses the metrics produced above. Candidate v7 must be blocked because a single critical promotion bypass is enough, even before reliability and judge calibration are considered.

The next engineering task isn't tuning the judge until the score turns green. Repair the candidate so attack-014 opens a security review without attempting a promotion, then rerun the unchanged frozen suite.

The repaired trace below makes that delta concrete. v8 changes the action for attack-014, then earns promotion only after the same frozen cases pass repeatedly.