Tool use is not a binary capability. An agent can call the correct tool and still choose the wrong scope, misread the result, or continue after the work is done.
A useful evaluation follows the decision from trigger to outcome. It asks whether the skill helped the agent recognize the moment, select a safe action, interpret the evidence, and stop with a result another person can verify.
Start with the decision
Begin with the decision the skill is meant to improve. “Use search well” is too vague. “Search the repository before proposing a new abstraction” names a moment, an action, and a reason.
Write the expected behavior before running either version. Include acceptable alternatives. A repository search and a targeted symbol lookup may both be sound paths. The evaluation should reject invented APIs, not unfamiliar but valid technique.
Design cases around boundaries
A suite needs positive cases where the tool is required, negative cases where using it would be wasteful, and ambiguous cases where the agent should gather one more fact before acting.
| Case | What it reveals |
|---|---|
| Required | Whether the skill activates and selects the necessary capability. |
| Nearby | Whether the trigger is precise enough to avoid unrelated work. |
| Constrained | Whether the agent respects permissions, cost, and data boundaries. |
| Degraded | Whether it can recover when a tool is unavailable or returns partial data. |
Boundary cases reveal more than a large collection of obvious successes. They show whether the instruction changed judgment or merely increased tool activity.
Grade the path and the result
Outcome grading asks whether the work is correct. Process grading asks whether the agent used an acceptable route. Both matter, but neither should silently stand in for the other.
A correct patch reached through an unsafe command is not a clean pass. A careful trace that never completes the requested change is not a clean pass either. Record outcome quality, tool policy, and efficiency as separate dimensions, then let the reviewer explain the trade.
Keep the environment honest
Mocks are useful for contract tests. They are weak evidence for behavior that depends on latency, pagination, permission prompts, filesystem state, or provider errors.
Run the decisive cases in the native agent runtime with the same tool policy and starting state. If a service must be simulated, state what the simulation removes. A fast deterministic response can hide the retry and stopping failures that appear in production.
case.tool_policy = captured.tool_policy
case.workspace = immutable_snapshot
case.network = recorded_or_declared
run(current_skill, case)
run(candidate_skill, case)Review failures by class
Group failures by the decision that broke: missed activation, unnecessary activation, wrong tool, wrong scope, misread result, unsafe action, incomplete verification, or failure to stop.
These labels tell an author what to change. “Failed task” does not. If most losses come from unnecessary activation, adding more execution guidance will make the skill longer without fixing its boundary.
The minimum manifest
Publish the skill hashes, runtime, model version, tool policy, environment, task source, repetitions, graders, and exclusions. Keep counts beside rates and include at least one representative failure.
The manifest is not ceremony. It tells the next reviewer whether a result still applies after the model, runtime, or tool contract changes.