skills/benchmark-skill/CASE-FORMAT.md

@@ -35,16 +35,16 @@ grep -qE '<pattern>' "$OUTPUT"
   Do not mention the skill by name or hint that a skill exists, or the baseline arm stops being an honest counterfactual.
 - **`## Seed`** — optional, for conversation-driven skills whose input is a discussion rather than a file tree.
   It is a role-tagged transcript (`**User:**` / `**Assistant:**`) injected as the subagent's prior context before the prompt.
-- **`## Hard assertions`** — optional, a `sh` code block of executable predicates forming a deterministic gate.
+- **`## Hard assertions`** — optional, a `sh` code block whose lines are each a separate predicate, together forming a deterministic gate.
-  Each runs with `$OUTPUT` bound to the path of the arm's captured final message and `$WORLD` bound to the path of that arm's fresh fixture copy.
+  Each predicate runs with `$OUTPUT` bound to the path of the arm's captured final message and `$WORLD` bound to the path of that arm's fresh fixture copy.
-  A non-zero exit fails the assertion, and a failed hard assertion fails the case outright.
+  The gate passes only when **every** predicate exits zero — evaluate them so an earlier failure is never masked by a later success, not by a block's trailing exit code alone.
-  The gate runs against the new-skill arm only.
+  Any non-zero exit fails the gate, and a failed gate fails the case outright, against the new-skill arm only.
 - **`## Soft criteria`** — at least one natural-language statement the blind judge grounds its comparisons on.
   Describe what a good answer looks like, not which arm should win.
 
 ## Fixtures
 
 A case ships a hermetic, committed fixture so a run is reproducible and needs no live external state.
-File-and-tree skills get a fixture directory beside `case.md`.
+File-and-tree skills get a fixture directory named `fixture/` beside `case.md`, whose contents populate each arm's world root — so a `$WORLD/<file>` predicate names a file directly under `fixture/`.
 Conversation-driven skills use `## Seed` instead.
 Every arm-and-trial combination runs against a **fresh copy** of the fixture, so writes never leak between runs.

skills/benchmark-skill/SKILL.md

@@ -19,6 +19,7 @@ The authoring contract for a test case is [`CASE-FORMAT.md`](CASE-FORMAT.md).
 Read it before you read the target skill's tests.
 
 Report artifacts live under a git-ignored `tests/.reports/` directory at the repo root, flat and keyed by skill name.
+Run from the repo root: every path in the steps below is relative to it.
 
 ## 1. Resolve the target and validate its tests tree
 
@@ -29,7 +30,7 @@ If the skill exists but has no tests directory or no case directories, report th
 
 Validate the shape of the tests tree at run time:
 
-- Each case directory has a `case.md` that parses per `CASE-FORMAT.md` — a `description`, a `## Prompt`, at least one `## Soft criteria` entry, and optional `## Seed` and `## Hard assertions` blocks.
+- Each case directory has a `case.md` that parses per `CASE-FORMAT.md`.
 - Any fixture a case references exists.
 - The test directory maps to a real skill.
 
@@ -40,12 +41,10 @@ Done when every case parses, its fixture exists, and you have the case list in s
 
 ## 2. Fix the arms and per-run settings
 
-This slice is **two-arm efficacy only**: a new-skill arm and a no-skill baseline arm.
+Run **two arms**: a new-skill arm and a no-skill baseline arm.
-(The previous-version regression arm is a later slice.
-The core already handles extra arms, so do not remove the two-arm shape.)
 
-Choose the trial temperature: a realistic temperature (around 1.0) by default, so the result reflects whether the skill *reliably* helps across variance.
+Arms run at the session's realistic default temperature (around 1.0), so the result reflects whether the skill *reliably* helps across variance rather than helping once by luck.
-A skill may pin a temperature-zero override when it wraps a genuinely mechanical task — honor that override if the skill declares one.
+Per-arm temperature is not settable through the in-session workflow surface, so a skill's temperature-zero override for a genuinely mechanical task is a documented knob for the future headless path, not something to set here.
 
 Each case runs **5 paired trials**.
 Trial *i*'s new-skill output is judged against trial *i*'s no-skill output.
@@ -82,11 +81,8 @@ Have it return a single winner: A, B, or tie.
 
 ## 4. Run the hard-assertion gate
 
-Run the case's `## Hard assertions` against the **new arm only**, once per trial.
+Run the case's `## Hard assertions` against the **new arm only**, once per trial, with `$OUTPUT` and `$WORLD` bound as `CASE-FORMAT.md` defines — that trial's new-arm final message and its world.
-Bind `$OUTPUT` to the path of that trial's new-arm final message and `$WORLD` to that trial's new-arm world (its fixture copy, which is the arm's working directory).
+Record each trial's pass or fail into the bundle for the core to score.
-Run each predicate.
-A non-zero exit fails the assertion.
-Record pass/fail per trial — a failed hard assertion fails the case outright regardless of the head-to-head.
 A case with no `## Hard assertions` block simply has no gate.
 
 ## 5. Collect the run data
@@ -146,9 +142,9 @@ python3 skills/benchmark-skill/core/benchmark_core.py \
   --html tests/.reports/<name>.html
 ```
 
-The core collapses each efficacy trial to WIN, TIE, or LOSS for the new arm (a tie is a non-win), applies the pass rule — **efficacy passes for a case when wins ≥ 3 and losses ≤ 1** — flags any loss for human review, and yields a skill-level **Efficacy verdict** that is green only when every case passes.
+The core collapses each trial to a per-arm WIN, TIE, or LOSS, applies the efficacy pass rule, flags any loss for human review, and yields the skill-level **Efficacy verdict**.
-It renders a self-contained HTML report: the Efficacy badge and run metadata, a two-row per-arm cost table (footnoted that absolute cost is inflated by shared-context cache overhead, so the trustworthy signal is the new-vs-no-skill ratio), and the cases in stable authored order.
+It renders the self-contained HTML report — the verdict badge, run metadata, the per-arm cost table, and the cases in stable authored order — alongside the machine-readable model.
-Cost is reported alongside quality but never gates the verdict.
+Cost is reported but never gates the verdict.
 
 Clean up the scratch when done: remove the `tests/.reports/.work/` directory and every per-arm world and skill materialization you created under the system temp path.