diff --git a/skills/benchmark-skill/CASE-FORMAT.md b/skills/benchmark-skill/CASE-FORMAT.md index 5cf4d8a..0194ab7 100644 --- a/skills/benchmark-skill/CASE-FORMAT.md +++ b/skills/benchmark-skill/CASE-FORMAT.md @@ -35,16 +35,16 @@ grep -qE '' "$OUTPUT" Do not mention the skill by name or hint that a skill exists, or the baseline arm stops being an honest counterfactual. - **`## Seed`** — optional, for conversation-driven skills whose input is a discussion rather than a file tree. It is a role-tagged transcript (`**User:**` / `**Assistant:**`) injected as the subagent's prior context before the prompt. -- **`## Hard assertions`** — optional, a `sh` code block of executable predicates forming a deterministic gate. - Each runs with `$OUTPUT` bound to the path of the arm's captured final message and `$WORLD` bound to the path of that arm's fresh fixture copy. - A non-zero exit fails the assertion, and a failed hard assertion fails the case outright. - The gate runs against the new-skill arm only. +- **`## Hard assertions`** — optional, a `sh` code block whose lines are each a separate predicate, together forming a deterministic gate. + Each predicate runs with `$OUTPUT` bound to the path of the arm's captured final message and `$WORLD` bound to the path of that arm's fresh fixture copy. + The gate passes only when **every** predicate exits zero — evaluate them so an earlier failure is never masked by a later success, not by a block's trailing exit code alone. + Any non-zero exit fails the gate, and a failed gate fails the case outright, against the new-skill arm only. - **`## Soft criteria`** — at least one natural-language statement the blind judge grounds its comparisons on. Describe what a good answer looks like, not which arm should win. ## Fixtures A case ships a hermetic, committed fixture so a run is reproducible and needs no live external state. -File-and-tree skills get a fixture directory beside `case.md`. +File-and-tree skills get a fixture directory named `fixture/` beside `case.md`, whose contents populate each arm's world root — so a `$WORLD/` predicate names a file directly under `fixture/`. Conversation-driven skills use `## Seed` instead. Every arm-and-trial combination runs against a **fresh copy** of the fixture, so writes never leak between runs. diff --git a/skills/benchmark-skill/SKILL.md b/skills/benchmark-skill/SKILL.md index ed5239c..a5216f6 100644 --- a/skills/benchmark-skill/SKILL.md +++ b/skills/benchmark-skill/SKILL.md @@ -19,6 +19,7 @@ The authoring contract for a test case is [`CASE-FORMAT.md`](CASE-FORMAT.md). Read it before you read the target skill's tests. Report artifacts live under a git-ignored `tests/.reports/` directory at the repo root, flat and keyed by skill name. +Run from the repo root: every path in the steps below is relative to it. ## 1. Resolve the target and validate its tests tree @@ -29,7 +30,7 @@ If the skill exists but has no tests directory or no case directories, report th Validate the shape of the tests tree at run time: -- Each case directory has a `case.md` that parses per `CASE-FORMAT.md` — a `description`, a `## Prompt`, at least one `## Soft criteria` entry, and optional `## Seed` and `## Hard assertions` blocks. +- Each case directory has a `case.md` that parses per `CASE-FORMAT.md`. - Any fixture a case references exists. - The test directory maps to a real skill. @@ -40,12 +41,10 @@ Done when every case parses, its fixture exists, and you have the case list in s ## 2. Fix the arms and per-run settings -This slice is **two-arm efficacy only**: a new-skill arm and a no-skill baseline arm. -(The previous-version regression arm is a later slice. -The core already handles extra arms, so do not remove the two-arm shape.) +Run **two arms**: a new-skill arm and a no-skill baseline arm. -Choose the trial temperature: a realistic temperature (around 1.0) by default, so the result reflects whether the skill *reliably* helps across variance. -A skill may pin a temperature-zero override when it wraps a genuinely mechanical task — honor that override if the skill declares one. +Arms run at the session's realistic default temperature (around 1.0), so the result reflects whether the skill *reliably* helps across variance rather than helping once by luck. +Per-arm temperature is not settable through the in-session workflow surface, so a skill's temperature-zero override for a genuinely mechanical task is a documented knob for the future headless path, not something to set here. Each case runs **5 paired trials**. Trial *i*'s new-skill output is judged against trial *i*'s no-skill output. @@ -82,11 +81,8 @@ Have it return a single winner: A, B, or tie. ## 4. Run the hard-assertion gate -Run the case's `## Hard assertions` against the **new arm only**, once per trial. -Bind `$OUTPUT` to the path of that trial's new-arm final message and `$WORLD` to that trial's new-arm world (its fixture copy, which is the arm's working directory). -Run each predicate. -A non-zero exit fails the assertion. -Record pass/fail per trial — a failed hard assertion fails the case outright regardless of the head-to-head. +Run the case's `## Hard assertions` against the **new arm only**, once per trial, with `$OUTPUT` and `$WORLD` bound as `CASE-FORMAT.md` defines — that trial's new-arm final message and its world. +Record each trial's pass or fail into the bundle for the core to score. A case with no `## Hard assertions` block simply has no gate. ## 5. Collect the run data @@ -146,9 +142,9 @@ python3 skills/benchmark-skill/core/benchmark_core.py \ --html tests/.reports/.html ``` -The core collapses each efficacy trial to WIN, TIE, or LOSS for the new arm (a tie is a non-win), applies the pass rule — **efficacy passes for a case when wins ≥ 3 and losses ≤ 1** — flags any loss for human review, and yields a skill-level **Efficacy verdict** that is green only when every case passes. -It renders a self-contained HTML report: the Efficacy badge and run metadata, a two-row per-arm cost table (footnoted that absolute cost is inflated by shared-context cache overhead, so the trustworthy signal is the new-vs-no-skill ratio), and the cases in stable authored order. -Cost is reported alongside quality but never gates the verdict. +The core collapses each trial to a per-arm WIN, TIE, or LOSS, applies the efficacy pass rule, flags any loss for human review, and yields the skill-level **Efficacy verdict**. +It renders the self-contained HTML report — the verdict badge, run metadata, the per-arm cost table, and the cases in stable authored order — alongside the machine-readable model. +Cost is reported but never gates the verdict. Clean up the scratch when done: remove the `tests/.reports/.work/` directory and every per-arm world and skill materialization you created under the system temp path.