A failure is expensive evidence. If the team fixes the immediate output but loses the session that produced it, the next skill version can buy the same lesson again.
A durable test case preserves the smallest faithful context that can reproduce the decision. It stays understandable enough for a reviewer to know why it exists.
Capture before explaining
Save the transcript, skill bundle, repository state, runtime, tool policy, and relevant provider events before editing the skill. Explanations written after the fix tend to overfit the author’s theory.
Keep the raw capture immutable. Derive a smaller case from it, but retain the source and a record of every removal.
Find the smallest faithful case
Remove context in layers and replay after each change. If deleting an earlier correction changes the agent’s plan, that correction belongs in the case. If a large generated directory never enters the trace, it may not.
Small does not mean synthetic. The case must retain the conditions that made the failure possible, including awkward names, partial state, and permission boundaries.
Write the expected decision
Describe the decision the improved skill should change, not the exact answer it should copy. For example: “inspect the existing adapter boundary before adding provider logic” leaves room for several correct implementations.
Add rejection conditions for unsafe or misleading outcomes. A case with only a golden response rewards imitation and punishes valid alternatives.
Prove the case can fail
Run the captured skill version against the case and confirm that the relevant failure still appears. If it does not, the reduction removed something material or the original outcome depended on uncontrolled variance.
Repeat both versions enough to see whether the result is stable. One lucky run should not turn a noisy case into a permanent gate.
Keep the regression visible
Name the case after the decision, not the incident ticket. Store its source, creation date, owner, expected behavior, rejection conditions, and known fidelity limits beside the runnable inputs.
When a runtime or model changes, rerun the case and record the new verification date. Retire it only when the underlying behavior is impossible or covered by a stronger case.
A case is a memory
Good suites are not piles of prompts. They are a memory of decisions the system once got wrong and the evidence that taught the team how to recognize them.
One carefully preserved failure is a modest start. Kept through the next ten changes, it becomes part of the product’s judgment.