A green test suite can hide a worse agent run. The candidate may take longer, touch more files, ask for broader permissions, or produce an answer that is harder to review.
Pass rate matters. It is one piece of the outcome. Skill evaluation becomes useful when the reviewer can see the behavior around that number and explain why one run is better.
One number hides the trade
Suppose X passes 14 tests in 102 seconds. Y passes 18 tests in 128 seconds. The extra tests may mean Y found missing coverage. They may also mean the candidate expanded the task and wrote tests for behavior nobody requested.
A score that rewards test count picks Y. A latency score picks X. Neither score knows whether the added work belongs in the change.
The reviewer needs the files, commands, response, and trace in one comparison. Metrics should point toward evidence, not replace it.
Read the trace
The trace explains how the outcome happened. It shows whether the agent found the right file early, looped over the same search, ignored a correction, or requested a tool that the task did not need.
We keep the latest event easy to scan during a run and preserve the full command list for review. Duration and cost sit beside the outcome, but the trace supplies the reason.
Separate quality from fidelity
Reviewers should never confuse a poor replay with a poor skill. If a repository snapshot is incomplete or a provider omits part of the transcript, the outcome has lower fidelity.
We report replay fidelity beside the comparison. Exact means the captured context and bundle match the invocation. Matched means we recovered the required inputs through another source. Partial means something material is missing.
This label does not decide the winner. It tells the reviewer how much confidence to place in the evidence.
Give reviewers an exit
Forced choice creates bad data. Some cases cannot distinguish the versions. Some runs fail before the skill matters. Some tasks are underspecified.
A complete review needs more than X or Y. "No difference" records equivalence. "Reject both" says neither outcome meets the bar. "Cannot judge" says the evidence cannot support a decision.
These choices keep uncertainty visible. They also reveal problems in the evaluation suite itself.
Record why
A judgment without a reason is difficult to reuse. The next author sees a winner but cannot tell which behavior mattered.
For ordinary preferences, the evidence often carries enough context. For blocked judgments, require a short reason. Missing files, runtime failure, unsafe output, and ambiguous requirements lead to different fixes.
Over time, reasons become a failure taxonomy grounded in real work. That taxonomy is more useful than a generic list invented before the team has reviewed any cases.
The score comes last
Aggregate only after individual cases make sense. Count preferences, ties, rejected pairs, and unjudgeable cases separately. Keep quality, efficiency, and replay fidelity available for drill-down.
A score is useful for watching a suite over time. It should be the last layer added, not the first thing designed.
The trustworthy verdict is the one a reviewer can reopen, inspect, and defend.