A prompt benchmark can tell you whether a model answered a question. A skill evaluation has to answer a harder one. Did this instruction change make an agent more reliable inside the work that caused the instruction to matter?
The difference is context. Agent work arrives with a repository, a conversation, tool results, permission boundaries, and a point in time. Remove those conditions and the evaluation may still look rigorous. It is testing a different task.
The prompt is not the case
Consider a skill that tells an agent how to modify a provider adapter. The final user message may be only a sentence long. The useful evidence sits earlier in the session: which files the agent read, what the user rejected, which tests failed, and which skill version the runtime loaded.
Copying the last message into a fresh chat throws that evidence away. The new run may choose another file, miss a constraint, or solve a broader problem. A better answer does not prove that the candidate skill improved the original workflow.
This definition changes how cases enter an evaluation suite. We do not begin with a spreadsheet of prompts. We begin with sessions where a skill was invoked and ask whether enough evidence exists to reconstruct the moment before the agent acted.
Hold the world still
A useful comparison changes one variable. For skill work, that variable is the skill bundle. Everything else should remain as close as the runtime allows.
| Keep fixed | Why it matters |
|---|---|
| Conversation | The agent should inherit the same decisions, corrections, and constraints. |
| Starting files | Different repository state can change the plan before the skill has any effect. |
| Runtime and model | A model change can be larger than the instruction change under review. |
| Tool policy | Different permissions alter which paths the agent can take. |
| Repetitions | Both versions need the same opportunity to expose variance. |
Perfect replay is rarely available. Files disappear. Providers omit events. A native runtime may not expose its historical state. The honest response is to report fidelity, not quietly fill the gaps.
Rebuild the replay point
We recover the transcript up to the skill invocation, not through the completed answer. We stage a repository snapshot. We load the current bundle into one attempt and the candidate bundle into another. Then we let the original runtime do the work.
The runtime matters because a mock executor erases the behavior we want to observe. Tool selection, context management, permission requests, and stopping decisions are part of the outcome. If the skill saves six tool calls, that is evidence. If it causes a new approval prompt, that is evidence too.
case = {
transcript: session.before(skill_invocation),
workspace: snapshot(session.workspace),
runtime: session.runtime,
policy: session.tool_policy
}
replay(case, current_skill)
replay(case, candidate_skill)The important line is the empty space between the two replays. The attempts must not share mutable state. A file written by X cannot become a starting condition for Y.
Judge the outcome blind
Names invite stories. A reviewer who sees "candidate" starts looking for improvement. A reviewer who remembers writing the change starts defending it. Blind comparison removes the easiest source of bias.
We label outcomes X and Y until the reviewer records a judgment. The review puts the response, file changes, commands, tests, duration, cost, and tool failures in the same frame. No single artifact gets to stand in for the whole run.
"Cannot judge" is not a weak answer. It protects the dataset. A missing repository, failed runtime, or ambiguous requirement should not become a noisy vote.
Failures belong in the result
Evaluation systems often treat infrastructure failure as an exception outside the score. Agent workflows make that separation less useful. A candidate that requests broader access, times out, or depends on a missing command may be telling us something important about the instruction.
We keep three facts separate. Outcome quality describes what the agent produced. Replay fidelity describes how closely we reconstructed the case. Runtime health describes whether the attempt completed under the intended conditions.
Combining them into one number makes the dashboard tidy and the result hard to trust. Keep the dimensions visible until a human can explain the trade.
Start with one case
A small evaluation can still be useful if the case is real and the evidence is complete. Start with the session that exposed the failure. Replay it against the proposed change. Read both traces. Decide what improved and what got worse.
Then save the case. The next skill change should face it again. Over time, the suite becomes a record of mistakes the system has already paid to learn from.
The goal is not a large benchmark. It is a short feedback loop between a change, the work it affects, and a decision someone can defend.