alexion
5db98dcab9
Collapse duplication so each rule has one home: the pass-rule threshold and cost footnote defer to the core, and the case-contract detail defers to CASE-FORMAT.md. Drop editor-facing sediment from the arms step. Fixes surfaced by a verify walk-through: define the hard-assertion gate as passing only when every predicate exits zero (an earlier failure was maskable by a later success), state that in-session arms run at the session default temperature, name the fixture directory, and note the steps run from the repo root.
2.7 KiB
The case.md authoring contract
Tests live in a top-level tests/ tree that mirrors skills/ by full path.
A skill at skills/<maybe/cosmetic/nesting>/<skill>/ has its tests at tests/skills/<maybe/cosmetic/nesting>/<skill>/.
Under that mirror point, each case is its own directory holding a case.md and an optional fixture.
Tests sit outside skills/ on purpose: a skill is packaged and placed on its own, and its tests must never ride along.
case.md
---
description: <one-line scenario>
---
## Prompt
<the realistic user request, given identically to every arm>
## Seed (optional; conversation-based skills only)
**User:** ...
**Assistant:** ...
## Hard assertions (executable; $OUTPUT = arm's final message, $WORLD = its fixture copy)
```sh
test -f "$WORLD/review.md"
grep -qE '<pattern>' "$OUTPUT"
```
## Soft criteria
- <a statement the judge grounds the comparisons on>
description— a one-line summary of the scenario, in the frontmatter.## Prompt— the realistic user request. It is authored once and arm-agnostically, and every arm receives it verbatim. Do not mention the skill by name or hint that a skill exists, or the baseline arm stops being an honest counterfactual.## Seed— optional, for conversation-driven skills whose input is a discussion rather than a file tree. It is a role-tagged transcript (**User:**/**Assistant:**) injected as the subagent's prior context before the prompt.## Hard assertions— optional, ashcode block whose lines are each a separate predicate, together forming a deterministic gate. Each predicate runs with$OUTPUTbound to the path of the arm's captured final message and$WORLDbound to the path of that arm's fresh fixture copy. The gate passes only when every predicate exits zero — evaluate them so an earlier failure is never masked by a later success, not by a block's trailing exit code alone. Any non-zero exit fails the gate, and a failed gate fails the case outright, against the new-skill arm only.## Soft criteria— at least one natural-language statement the blind judge grounds its comparisons on. Describe what a good answer looks like, not which arm should win.
Fixtures
A case ships a hermetic, committed fixture so a run is reproducible and needs no live external state.
File-and-tree skills get a fixture directory named fixture/ beside case.md, whose contents populate each arm's world root — so a $WORLD/<file> predicate names a file directly under fixture/.
Conversation-driven skills use ## Seed instead.
Every arm-and-trial combination runs against a fresh copy of the fixture, so writes never leak between runs.