Skill Evaluation Field Notes
Better agent work starts with better evidence.
We study how skills change real AI workflows. These are the methods, failures, and design decisions behind trustworthy evaluations.
MethodsAugust 28, 20269 min read
Replay the work, not the prompt
A skill changes the path an agent takes. A fair evaluation preserves the work around it, then changes one thing.
Captured session · turn 41
same prompt · same files · same runtime
XCurrent skill14 tests · 1m 42s
YCandidate skill18 tests · 2m 08s
Identity stays hidden until review
Latest
Notes from the evaluation loop
SKILL.md
01TriggerWhen a module owns a decision
02OwnName the boundary before editing
03AvoidProvider details outside adapters
04CheckRun the narrow contract tests
Every rule leaves evidence
Skills8 min read
Longer instructions often hide the real interface. A useful skill states when it applies, what it owns, and how success can be checked.
Skill Evaluation team · August 26, 2026
Blind review · case 08Evidence before score
Outcome quality88%
Tests100%
Replay fidelity96%
Tool restraint72%
2 tool calls added1m 12s slowerReview needed
Field notes7 min read
Tests can pass while the workflow gets slower, noisier, or less trustworthy. Review the evidence that sits around the score.
Skill Evaluation team · August 21, 2026
tool-use.eval
01TriggerThe decision needs external evidence
02ScopeUse the smallest capable tool
03ReadInterpret result before acting
04StopVerify the requested outcome
Path and outcome remain inspectable
Methods10 min read
Test whether an agent chose the right tool, used it with the right scope, and turned its result into a defensible outcome.
Skill Evaluation team · August 18, 2026
Failure review · primary classLabel the decision
Activation32%
Planning68%
Execution54%
Verification81%
1 primary class2 contributorsTrace linked
Field notes8 min read
A useful failure label identifies the decision that broke, not just the symptom visible at the end of the run.
Skill Evaluation team · August 12, 2026
Regression case · preserved source
capture · reduce · replay
01Original failuresource retained
02Durable caseverified regression
Source and reductions remain auditable
Methods7 min read
Preserve the smallest real context that reproduces the failure, then keep it in the suite after the immediate fix ships.
Skill Evaluation team · August 5, 2026
Replay real work
Use captured sessions instead of idealized prompts.
01Change one thing
Freeze the context, runtime, and starting files.
02Keep the evidence
Review output, tools, files, tests, cost, and failures.
03Written for people who build and review agent workflows.