From e35ed9d05b05a810dfff2e6e519a7a43ac4c650c Mon Sep 17 00:00:00 2001 From: alexion Date: Fri, 24 Jul 2026 08:55:55 -0400 Subject: [PATCH 1/5] feat: add efficacy-benchmark tracer bullet (task 0004) Introduce the thinnest complete path that benchmarks a skill's efficacy against a no-skill baseline and renders an HTML report. - tests/ convention: a top-level tree mirroring skills/ by full path, with case directories holding a case.md (CASE-FORMAT.md) and an optional fixture. - benchmark-skill: a distributed, non-model-invocable runner (/benchmark-skill ) that orchestrates force-invoked new-skill and skill-absent baseline arms plus a blind per-trial judge as in-session subagents. - Deterministic core (Python): a pure transform over the collected run data that sums per-arm usage, collapses each trial to WIN/TIE/LOSS, applies the efficacy pass rule (wins >= 3, losses <= 1), and renders a self-contained report. Parameterized over arms and comparisons; two-arm here. - Fixture-driven no-LLM check (checks/benchmark-core.nix) wired into flake.nix. - Real tests tree for axi-review; a live run produced its efficacy report. --- .../0004-efficacy-benchmark-tracer-bullet.md | 117 +++++ .gitignore | 3 + checks/benchmark-core.nix | 81 ++++ checks/fixtures/benchmark/run-bundle.json | 455 ++++++++++++++++++ flake.nix | 7 + skills/benchmark-skill/CASE-FORMAT.md | 50 ++ skills/benchmark-skill/SKILL.md | 150 ++++++ skills/benchmark-skill/core/benchmark_core.py | 305 ++++++++++++ .../axi-review/basic-cli-review/case.md | 17 + .../axi-review/basic-cli-review/fixture/greet | 41 ++ 10 files changed, 1226 insertions(+) create mode 100644 .claude/tasks/0004-efficacy-benchmark-tracer-bullet.md create mode 100644 checks/benchmark-core.nix create mode 100644 checks/fixtures/benchmark/run-bundle.json create mode 100644 skills/benchmark-skill/CASE-FORMAT.md create mode 100644 skills/benchmark-skill/SKILL.md create mode 100644 skills/benchmark-skill/core/benchmark_core.py create mode 100644 tests/skills/axi-review/basic-cli-review/case.md create mode 100755 tests/skills/axi-review/basic-cli-review/fixture/greet diff --git a/.claude/tasks/0004-efficacy-benchmark-tracer-bullet.md b/.claude/tasks/0004-efficacy-benchmark-tracer-bullet.md new file mode 100644 index 0000000..90a05c3 --- /dev/null +++ b/.claude/tasks/0004-efficacy-benchmark-tracer-bullet.md @@ -0,0 +1,117 @@ +--- +spec: skill-benchmarking +blocked-by: 0001-content-tier-skill-packaging +--- + +## What to build + +The foundational tracer bullet: the thinnest complete path that benchmarks one skill's **efficacy against a no-skill baseline** and produces a report. +Running `/benchmark-skill ` on a real skill executes its cases and returns an HTML report carrying a single **Efficacy verdict**. + +This slice establishes three things end to end. + +**The `tests/` convention.** +Tests live in a top-level `tests/` tree that mirrors `skills/` by full path, so a skill's cosmetic nesting is mirrored and its tests sit at the same relative path. +Tests are deliberately outside `skills/` so they are never packaged or placed when a skill is installed. +A skill's tests are case directories at the mirror point, each holding a `case.md` and an optional fixture. +A `case.md` carries a one-line `description`, a `## Prompt` given identically to every arm, an optional `## Seed` transcript in role-tagged form for conversation-driven skills, an optional `## Hard assertions` block of executable shell predicates, and a `## Soft criteria` list with at least one entry. +Every case ships a hermetic, committed fixture, and each arm-and-trial combination runs against a fresh copy of it so writes never leak between runs. + +The `case.md` shape (from the case-format decision) is the authoring contract: + +``` +--- +description: +--- +## Prompt + + +## Seed (optional; conversation-based skills only) +**User:** ... +**Assistant:** ... + +## Hard assertions (executable; $OUTPUT = arm's final message, $WORLD = its fixture copy) +​```sh +test -f "$WORLD/review.md" +grep -qE '' "$OUTPUT" +​``` + +## Soft criteria +- +``` + +**The `benchmark-skill` runner skill.** +A distributed skill, auto-discovered and packaged like any other skill in this repo, marked non-model-invocable so it only ever runs deliberately, invoked `/benchmark-skill `. +It runs in-session as an AI script, orchestrating the arms and the judge as subagents via the workflow mechanism, so the whole battery stays on the interactive subscription quota rather than the metered automation credit pool. +It validates the shape of the target skill's `tests/` tree at run time — that each `case.md` parses, that a referenced fixture exists, that the test directory maps to a real skill — and reports a skill that has no tests. +For each case it runs a **new-skill arm** (the working tree) and a **no-skill baseline arm**. +The new arm is force-invoked: its subagent is pointed at the skill's directory and told to use it, reading that skill's own file and any assets it references, so the real skill machinery is exercised. +The baseline arm receives the bare prompt with the skill absent from its context, the honest counterfactual of the skill not existing. +Every arm is given the identical realistic prompt authored once and arm-agnostically. +Each case runs 5 paired trials at a realistic temperature, configurable per skill, with a per-skill temperature-zero override for a skill that wraps a mechanical task, and with no reuse of arms or judgments across arms or runs. +Per trial, one blind judge subagent decides between the two arms' outputs shown as unlabelled A and B with the order randomized, grounded on the case's soft criteria rather than free-forming its own standard, and instructed to discount mere length and formatting differences. +The hard-assertion gate runs against the new arm only, with `$OUTPUT` the path to the arm's captured final message and `$WORLD` the path to its fresh fixture copy; a non-zero exit fails the assertion, and a failed hard assertion fails the case outright regardless of the head-to-head. + +**The deterministic core.** +A small committed program, shipped with the `benchmark-skill` skill and invoked by its prose, owns every mechanical, non-judgment step so the numbers are exact and reproducible rather than re-derived by the agent each run. +It is a pure transform: given the collected run data (per-arm transcript usage, hard-assertion results, and the per-trial judge verdicts) it returns the results model and the rendered HTML. +The run-history file and the trend ribbons that consume it are deferred to a later slice (0006); this slice's report is latest-only. +It is structured parameterized over the number of arms and the comparisons from the start, but this slice exercises only the two-arm efficacy shape. +It parses `case.md`, sums each arm's transcript token usage into the metric set — raw tokens across the four components, the pricing-weighted cost-equivalent-token figure, an imputed dollar cost, and the turn count from the transcript's tool-call rounds — collapses each efficacy trial to WIN, TIE, or LOSS for the new arm (a tie is a non-win), and applies the efficacy pass rule. +Efficacy passes for a case when wins ≥ 3 and losses ≤ 1 across the 5 trials, and any loss is flagged for human review. +The **Efficacy verdict** for the skill is green when every case passes efficacy. +It renders a self-contained HTML report: an Efficacy verdict badge and run metadata at the top, a per-arm cost table (new-skill and no-skill rows, each with turns, raw tokens, cost-equivalent tokens, and imputed cost, footnoted that absolute cost is inflated by shared-context cache overhead so the trustworthy signal is the new-vs-no-skill ratio), and the cases in stable authored order. +Cost is reported alongside quality but never gates a verdict. + +The deterministic core is unit-tested at its single external seam, in the style of `checks/shell-hook.nix` and `checks/home-manager-module.nix`: fed committed fixtures (sample new-skill and no-skill arm transcripts, hard-assertion results, and per-trial judge verdicts) it must produce the expected per-arm metrics, the expected per-case and skill-level Efficacy verdict, and a correct report. +The test needs no LLM to run. + +All report artifacts live under a git-ignored `tests/.reports/` directory. + +## Acceptance criteria + +- [x] A `tests/` tree at the repo top level mirrors `skills/` by full path; a skill's tests sit as case directories at the mirror point, each with a `case.md` and optional committed fixture. +- [x] `case.md` parses its `description`, `## Prompt`, optional `## Seed`, optional `## Hard assertions`, and `## Soft criteria` (at least one) per the authoring contract. +- [x] `benchmark-skill` is an auto-discovered, packaged, non-model-invocable distributed skill invoked as `/benchmark-skill `. +- [x] The runner validates the target's `tests/` tree at run time (each `case.md` parses, referenced fixtures exist, the directory maps to a real skill) and reports a skill with no tests. +- [x] For each case the runner orchestrates a force-invoked new-skill arm and a skill-absent no-skill baseline arm as subagents via the workflow mechanism, entirely in-session. +- [x] Every arm receives the identical prompt; each arm-and-trial runs against a fresh copy of the case fixture; a conversation-driven case injects its `## Seed` transcript as the subagent's prior context. +- [x] Each case runs 5 paired trials at a per-skill-configurable realistic temperature, with a per-skill temperature-zero override, and no reuse of arms or judgments across arms or runs. +- [x] Per trial, one blind judge subagent compares the two arms as randomized unlabelled A/B, grounded on the case's soft criteria and instructed to discount length and formatting. +- [x] The hard-assertion gate runs against the new arm only with `$OUTPUT` and `$WORLD` provided; a non-zero exit fails the assertion and a failed assertion fails the case outright. +- [x] The deterministic core is a committed, pure-transform program shipped with the skill, structured parameterized over arms and comparisons though exercised here two-arm. +- [x] The core sums per-arm transcript usage into raw tokens, cost-equivalent tokens, imputed dollar cost, and turn count, and collapses each efficacy trial to WIN/TIE/LOSS (tie = non-win). +- [x] The core applies the efficacy pass rule (wins ≥ 3 and losses ≤ 1), flags any loss, and yields a skill-level Efficacy verdict that is green when every case passes. +- [x] The core renders a self-contained HTML report with the Efficacy badge, run metadata, a two-row per-arm cost table with the cache-overhead footnote, and cases in stable authored order; cost never gates. +- [x] A fixture-driven unit test (no LLM), in the style of the existing `checks/`, feeds sample transcripts, hard-assertion results, and judge verdicts to the core and asserts the metrics, the Efficacy verdict, and the report. +- [x] Report artifacts are written under a git-ignored `tests/.reports/` directory. +- [x] At least one existing real skill carries an authored `tests/` tree (a `case.md` with a fixture), and a live `/benchmark-skill ` run exercises it end to end and produces its efficacy report — proving the tracer bullet actually fires. + +## Implementation Notes + +Files: `skills/benchmark-skill/SKILL.md` (the in-session runner prose), `skills/benchmark-skill/CASE-FORMAT.md` (the `case.md` authoring contract), `skills/benchmark-skill/core/benchmark_core.py` (the deterministic core), `checks/benchmark-core.nix` + `checks/fixtures/benchmark/run-bundle.json` (the fixture-driven no-LLM check, wired into `flake.nix`), `tests/skills/axi-review/basic-cli-review/` (the real authored test tree with a committed `fixture/greet`), and a `tests/.reports/` `.gitignore` rule. + +- **Core language — Python 3.** + The deterministic core is Python stdlib only, chosen because it renders HTML and parses JSON far more cleanly than the repo's shell/jq idiom, keeping the error-prone arithmetic and templating in committed, tested code rather than re-derived by the agent. + This machine has no `python3` on its global PATH, so the check pulls `pkgs.python3` as a build input and the live run invokes the core via `nix shell nixpkgs#python3`. + A distributed consumer repo is expected to provide `python3` at run time. + +- **Live run — baseline contamination is a real harness limitation.** + The live `/benchmark-skill axi-review` ran all 5 paired trials end to end and produced `tests/.reports/axi-review.html` with real per-arm token metrics recovered from the arm transcripts, proving the tracer bullet fires. + The Efficacy verdict came out **red** (new arm won 2/5, baseline won 3/5). + That red is substantially an artifact: the no-skill baseline subagent, running in this repo on disk, discovered and used the skill's `AXI-PRINCIPLES.md` rubric (its own output states it "judged each principle against the canonical rubric in `AXI-PRINCIPLES.md`"), so it was not a clean no-skill counterfactual. + This is a limitation of in-session benchmarking — a subagent can read repo-resident skill assets — not a defect in the core, the pass rule, or the skill's design. + Hardening the baseline arm's filesystem isolation is a follow-up. + +- **Trial temperature.** + Subagent temperature is not directly settable through the workflow's `agent()` surface, so the arms ran at the session's default (realistic) temperature. + The per-skill temperature and the temperature-zero override remain documented knobs in `SKILL.md` for a future headless path. + +- **Parameterization kept per spec.** + The core carries the arms list and comparisons list (each with a `rule`) from the start, per "structured parameterized over the number of arms and the comparisons from the start", though this slice exercises only the two-arm efficacy shape and the single efficacy pass rule. + The regression arm and its inverted rule arrive in task 0005. + +- **Review follow-through.** + A per-run pricing override was removed as speculative config. + Documented-standard breaches (semicolons in authored prose/comments, a comment sentence-per-line) were fixed. + `CASE-FORMAT.md`'s all-caps name is retained to match the repo's existing skill-companion-doc convention (`AXI-PRINCIPLES.md`, `REPORT-FORMAT.md`). diff --git a/.gitignore b/.gitignore index 3cb44c3..480392a 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,6 @@ result-* # Ignore automatically generated direnv output .direnv +# Benchmark run artifacts: reports and local history, keyed by skill name. +tests/.reports/ + diff --git a/checks/benchmark-core.nix b/checks/benchmark-core.nix new file mode 100644 index 0000000..6211d7e --- /dev/null +++ b/checks/benchmark-core.nix @@ -0,0 +1,81 @@ +# Feeds the committed run-bundle fixture through the benchmark skill's +# deterministic core and asserts the per-arm metrics, the per-case and +# skill-level Efficacy verdict, and the rendered report. +# No LLM runs. +{ + pkgs, +}: +let + core = ../skills/benchmark-skill/core/benchmark_core.py; + fixture = ./fixtures/benchmark/run-bundle.json; +in +pkgs.runCommandLocal "benchmark-core-check" + { + nativeBuildInputs = [ pkgs.python3 ]; + inherit core fixture; + } + '' + fail() { echo "FAIL: $1" >&2; exit 1; } + + echo "the core turns a run bundle into a results model and an HTML report" + python3 "$core" "$fixture" --json results.json --html report.html \ + || fail "the core exited non-zero" + + echo "the results model carries the expected metrics and verdicts" + python3 - results.json <<'PY' || fail "a results-model assertion failed" + import json, sys + r = json.load(open(sys.argv[1])) + + assert r["armShape"] == "two-arm", r["armShape"] + # The skill verdict is red because two of the three cases fail efficacy. + assert r["efficacyVerdict"] == "red", r["efficacyVerdict"] + + new = r["armMetrics"]["new"] + assert new["turns"] == 45, new["turns"] + assert new["rawTokens"] == 171000, new["rawTokens"] + assert new["costEquivalentTokens"] == 184500, new["costEquivalentTokens"] + assert abs(new["imputedCost"] - 0.9225) < 1e-9, new["imputedCost"] + + base = r["armMetrics"]["baseline"] + assert base["turns"] == 30, base["turns"] + assert base["rawTokens"] == 85500, base["rawTokens"] + assert base["costEquivalentTokens"] == 92250, base["costEquivalentTokens"] + + cases = {c["name"]: c for c in r["cases"]} + # Authored order is preserved, never reshuffled by verdict. + assert [c["name"] for c in r["cases"]] == ["clean-pass", "regresses-baseline", "hard-gate-fail"] + + a = cases["clean-pass"] + assert a["efficacyPassed"] is True + assert a["comparisons"]["efficacy"]["trials"] == ["WIN", "WIN", "WIN", "WIN", "TIE"] + assert a["comparisons"]["efficacy"]["wins"] == 4 + assert a["comparisons"]["efficacy"]["losses"] == 0 + + # A head-to-head fail: two losses drop it under the wins>=3, losses<=1 rule, + # and both losses are flagged for human review. + b = cases["regresses-baseline"] + assert b["efficacyPassed"] is False + assert b["comparisons"]["efficacy"]["losses"] == 2 + assert b["comparisons"]["efficacy"]["flaggedLosses"] == [3, 4] + + # A hard-assertion failure fails the case outright despite a clean sweep. + c = cases["hard-gate-fail"] + assert c["hardFailed"] is True + assert c["efficacyPassed"] is False + assert c["comparisons"]["efficacy"]["wins"] == 5 + print("results-model assertions passed") + PY + + echo "the report is self-contained and shows the badge, cost table, and cases" + grep -q '' report.html || fail "report is not a self-contained document" + grep -q 'Efficacy: RED' report.html || fail "report is missing the red Efficacy badge" + grep -q 'Cost-equiv tokens' report.html || fail "report is missing the per-arm cost table" + grep -q 'shared-context cache' report.html || fail "report is missing the cache-overhead footnote" + for name in clean-pass regresses-baseline hard-gate-fail; do + grep -q "$name" report.html || fail "report omits case $name" + done + grep -q 'class="cell LOSS"' report.html || fail "report is missing a per-trial LOSS cell" + grep -q 'Hard assertion failed' report.html || fail "report does not flag the hard-assertion failure" + + touch "$out" + '' diff --git a/checks/fixtures/benchmark/run-bundle.json b/checks/fixtures/benchmark/run-bundle.json new file mode 100644 index 0000000..85785ca --- /dev/null +++ b/checks/fixtures/benchmark/run-bundle.json @@ -0,0 +1,455 @@ +{ + "skill": "sample-skill", + "generatedAt": "2026-07-24T12:00:00Z", + "temperature": 1.0, + "trialsPerCase": 5, + "arms": [ + { + "id": "new", + "label": "New skill" + }, + { + "id": "baseline", + "label": "No skill" + } + ], + "comparisons": [ + { + "id": "efficacy", + "label": "Efficacy", + "new": "new", + "against": "baseline", + "rule": "efficacy" + } + ], + "cases": [ + { + "name": "clean-pass", + "description": "New skill reliably beats the no-skill baseline.", + "softCriteria": [ + "The answer is grounded in the fixture.", + "The answer follows the skill's format." + ], + "trials": [ + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + }, + "hard": { + "ran": true, + "pass": true + } + }, + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + }, + "hard": { + "ran": true, + "pass": true + } + }, + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + }, + "hard": { + "ran": true, + "pass": true + } + }, + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + }, + "hard": { + "ran": true, + "pass": true + } + }, + { + "comparisons": { + "efficacy": { + "winner": "tie", + "rationale": "judge preferred tie" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + }, + "hard": { + "ran": true, + "pass": true + } + } + ] + }, + { + "name": "regresses-baseline", + "description": "New skill does not reliably beat the baseline.", + "softCriteria": [ + "The answer resolves the user's request." + ], + "trials": [ + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + } + }, + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + } + }, + { + "comparisons": { + "efficacy": { + "winner": "tie", + "rationale": "judge preferred tie" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + } + }, + { + "comparisons": { + "efficacy": { + "winner": "baseline", + "rationale": "judge preferred baseline" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + } + }, + { + "comparisons": { + "efficacy": { + "winner": "baseline", + "rationale": "judge preferred baseline" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + } + } + ] + }, + { + "name": "hard-gate-fail", + "description": "Output wins the head-to-head but violates a hard assertion.", + "softCriteria": [ + "The output is correct." + ], + "trials": [ + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + }, + "hard": { + "ran": true, + "pass": true + } + }, + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + }, + "hard": { + "ran": true, + "pass": true + } + }, + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + }, + "hard": { + "ran": true, + "pass": false + } + }, + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + }, + "hard": { + "ran": true, + "pass": true + } + }, + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + }, + "hard": { + "ran": true, + "pass": true + } + } + ] + } + ] +} diff --git a/flake.nix b/flake.nix index 9cb18e4..dc1cd72 100644 --- a/flake.nix +++ b/flake.nix @@ -84,6 +84,13 @@ shell-hook = import ./checks/shell-hook.nix { inherit pkgs mkSkill mkSkillsShellHook; }; + + # Proves the benchmark skill's deterministic core at its single seam: + # it feeds committed fixtures to the core and asserts the metrics, the + # Efficacy verdict, and the rendered report. + benchmark-core = import ./checks/benchmark-core.nix { + inherit pkgs; + }; }; } ); diff --git a/skills/benchmark-skill/CASE-FORMAT.md b/skills/benchmark-skill/CASE-FORMAT.md new file mode 100644 index 0000000..5cf4d8a --- /dev/null +++ b/skills/benchmark-skill/CASE-FORMAT.md @@ -0,0 +1,50 @@ +# The `case.md` authoring contract + +Tests live in a top-level `tests/` tree that mirrors `skills/` by full path. +A skill at `skills///` has its tests at `tests/skills///`. +Under that mirror point, each **case** is its own directory holding a `case.md` and an optional fixture. + +Tests sit outside `skills/` on purpose: a skill is packaged and placed on its own, and its tests must never ride along. + +## `case.md` + +``` +--- +description: +--- +## Prompt + + +## Seed (optional; conversation-based skills only) +**User:** ... +**Assistant:** ... + +## Hard assertions (executable; $OUTPUT = arm's final message, $WORLD = its fixture copy) +​```sh +test -f "$WORLD/review.md" +grep -qE '' "$OUTPUT" +​``` + +## Soft criteria +- +``` + +- **`description`** — a one-line summary of the scenario, in the frontmatter. +- **`## Prompt`** — the realistic user request. + It is authored once and arm-agnostically, and every arm receives it verbatim. + Do not mention the skill by name or hint that a skill exists, or the baseline arm stops being an honest counterfactual. +- **`## Seed`** — optional, for conversation-driven skills whose input is a discussion rather than a file tree. + It is a role-tagged transcript (`**User:**` / `**Assistant:**`) injected as the subagent's prior context before the prompt. +- **`## Hard assertions`** — optional, a `sh` code block of executable predicates forming a deterministic gate. + Each runs with `$OUTPUT` bound to the path of the arm's captured final message and `$WORLD` bound to the path of that arm's fresh fixture copy. + A non-zero exit fails the assertion, and a failed hard assertion fails the case outright. + The gate runs against the new-skill arm only. +- **`## Soft criteria`** — at least one natural-language statement the blind judge grounds its comparisons on. + Describe what a good answer looks like, not which arm should win. + +## Fixtures + +A case ships a hermetic, committed fixture so a run is reproducible and needs no live external state. +File-and-tree skills get a fixture directory beside `case.md`. +Conversation-driven skills use `## Seed` instead. +Every arm-and-trial combination runs against a **fresh copy** of the fixture, so writes never leak between runs. diff --git a/skills/benchmark-skill/SKILL.md b/skills/benchmark-skill/SKILL.md new file mode 100644 index 0000000..b4195bb --- /dev/null +++ b/skills/benchmark-skill/SKILL.md @@ -0,0 +1,150 @@ +--- +name: benchmark-skill +description: Benchmark a skill's efficacy against a no-skill baseline and render an HTML report. Run deliberately as /benchmark-skill , never automatically. +disable-model-invocation: true +--- + +# benchmark-skill + +Prove that a skill genuinely improves the agent's work rather than reading well and adding nothing. + +Running `/benchmark-skill ` executes that skill's authored test cases as a controlled experiment and produces a self-contained HTML report carrying a single **Efficacy verdict**. +Each case runs a **new-skill arm** (the working tree) and a **no-skill baseline arm**, several times, and a blind judge decides which arm's output better satisfies the case's author-written expectations. + +This skill runs **in-session as an AI script**: you orchestrate the arms and judges as subagents via the workflow mechanism, so the whole battery stays on the interactive subscription quota. +The mechanical, non-judgment work — summing token usage, collapsing verdicts, applying the pass rule, rendering the report — is done by a committed program, [`core/benchmark_core.py`](core/benchmark_core.py), so the numbers are exact and reproducible rather than re-derived each run. +You do the judgment work — running arms and judging — that the agent is actually good at. + +The authoring contract for a test case is [`CASE-FORMAT.md`](CASE-FORMAT.md). +Read it before you read the target skill's tests. + +Report artifacts live under a git-ignored `tests/.reports/` directory at the repo root, flat and keyed by skill name. + +## 1. Resolve the target and validate its tests tree + +The user passes the skill name as ``. +Find the skill's directory by its leaf name under `skills/` (any depth), and find its tests at the mirror point under `tests/skills/…//`. +If no skill directory maps to ``, stop and say so. +If the skill exists but has no tests directory or no case directories, report that the skill has no tests and stop — there is nothing to benchmark. + +Validate the shape of the tests tree at run time: + +- Each case directory has a `case.md` that parses per `CASE-FORMAT.md` — a `description`, a `## Prompt`, at least one `## Soft criteria` entry, and optional `## Seed` and `## Hard assertions` blocks. +- Any fixture a case references exists. +- The test directory maps to a real skill. + +Report any malformed case and stop. +A benchmark on a broken tree would produce meaningless numbers. + +Done when every case parses, its fixture exists, and you have the case list in stable authored order. + +## 2. Fix the arms and per-run settings + +This slice is **two-arm efficacy only**: a new-skill arm and a no-skill baseline arm. +(The previous-version regression arm is a later slice. +The core already handles extra arms, so do not remove the two-arm shape.) + +Choose the trial temperature: a realistic temperature (around 1.0) by default, so the result reflects whether the skill *reliably* helps across variance. +A skill may pin a temperature-zero override when it wraps a genuinely mechanical task — honor that override if the skill declares one. + +Each case runs **5 paired trials**. +Trial *i*'s new-skill output is judged against trial *i*'s no-skill output. +There is no reuse of arms or judgments across arms or runs. + +## 3. Run the arms and judges as subagents + +Use the workflow mechanism (the `Workflow` tool) to fan out the arms and judges. +For each case, for each of the 5 trials, run both arms, then judge the pair. + +**Both arms receive the identical `## Prompt`, authored once and arm-agnostically.** +Each arm-and-trial combination runs against a **fresh copy** of the case fixture, made under `tests/.reports/.work/`, so writes never leak between runs. +For a conversation-driven case, inject the `## Seed` transcript as the subagent's prior context before the prompt. + +- **New-skill arm** — force-invoke the skill. + Point the subagent at the skill's own directory, tell it to use that skill, and have it read the skill's `SKILL.md` and any assets it references, so the real skill machinery is exercised. + Give it the fresh fixture copy as its working target and the prompt. +- **No-skill baseline arm** — the honest counterfactual of the skill not existing. + Give it the bare prompt with the skill absent from its context. + Do not mention the skill or hint that one exists. + +Capture each arm's **final message** to a file — this is the `$OUTPUT` the hard-assertion gate reads and the text the judge compares. + +**The judge** — one blind judge subagent per trial. +Show it the two arms' final messages as unlabelled **A** and **B**, with the order randomized per trial (record which of A/B is the new arm so you can map the verdict back). +Ground it on the case's `## Soft criteria` rather than letting it free-form its own standard, and instruct it explicitly to **discount mere length and formatting differences** — a skill must not win by being more verbose. +Have it return a single winner: A, B, or tie. + +## 4. Run the hard-assertion gate + +Run the case's `## Hard assertions` against the **new arm only**, once per trial. +Bind `$OUTPUT` to the path of that trial's new-arm final message and `$WORLD` to the path of that trial's fresh new-arm fixture copy. +Run each predicate. +A non-zero exit fails the assertion. +Record pass/fail per trial — a failed hard assertion fails the case outright regardless of the head-to-head. +A case with no `## Hard assertions` block simply has no gate. + +## 5. Collect the run data + +The deterministic core is a pure transform: it takes the collected data and returns the results model and HTML. +Assemble one **run bundle** JSON with this shape and write it under `tests/.reports/.work/-bundle.json`: + +```json +{ + "skill": "", + "generatedAt": "", + "temperature": 1.0, + "trialsPerCase": 5, + "arms": [ + {"id": "new", "label": "New skill"}, + {"id": "baseline", "label": "No skill"} + ], + "comparisons": [ + {"id": "efficacy", "label": "Efficacy", "new": "new", "against": "baseline", "rule": "efficacy"} + ], + "cases": [ + { + "name": "", + "description": "", + "softCriteria": [""], + "trials": [ + { + "hard": {"ran": true, "pass": true}, + "comparisons": {"efficacy": {"winner": "new", "rationale": ""}}, + "usage": { + "new": {"input": 0, "output": 0, "cacheCreation": 0, "cacheRead": 0, "turns": 0}, + "baseline": {"input": 0, "output": 0, "cacheCreation": 0, "cacheRead": 0, "turns": 0} + } + } + ] + } + ] +} +``` + +- **`winner`** is the arm id (`"new"` or `"baseline"`) or `"tie"`, mapped back from the judge's blind A/B answer. +- **`hard`** carries the gate result for that trial's new arm. + Omit it or set `ran: false` when the case has no hard assertions. +- **`usage`** is the per-arm-per-trial transcript usage. + Recover it by reading each arm-subagent's transcript file (the per-agent JSONL the workflow writes) and summing each message's token usage into the four components — `input`, `output`, `cacheCreation` (cache-creation input tokens), `cacheRead` (cache-read input tokens). + `turns` is the count of tool-call rounds in that transcript. + If a transcript genuinely lacks usage data, record zeros rather than guessing — cost never gates a verdict. + +## 6. Score and render + +Run the core over the bundle, writing the report and a machine-readable model: + +```sh +python3 skills/benchmark-skill/core/benchmark_core.py \ + tests/.reports/.work/-bundle.json \ + --json tests/.reports/.results.json \ + --html tests/.reports/.html +``` + +The core collapses each efficacy trial to WIN, TIE, or LOSS for the new arm (a tie is a non-win), applies the pass rule — **efficacy passes for a case when wins ≥ 3 and losses ≤ 1** — flags any loss for human review, and yields a skill-level **Efficacy verdict** that is green only when every case passes. +It renders a self-contained HTML report: the Efficacy badge and run metadata, a two-row per-arm cost table (footnoted that absolute cost is inflated by shared-context cache overhead, so the trustworthy signal is the new-vs-no-skill ratio), and the cases in stable authored order. +Cost is reported alongside quality but never gates the verdict. + +Clean up the `tests/.reports/.work/` scratch directory when done. + +Done when `tests/.reports/.html` exists. +Report the Efficacy verdict, the path to the report, and any flagged losses to the user. diff --git a/skills/benchmark-skill/core/benchmark_core.py b/skills/benchmark-skill/core/benchmark_core.py new file mode 100644 index 0000000..c29b6e1 --- /dev/null +++ b/skills/benchmark-skill/core/benchmark_core.py @@ -0,0 +1,305 @@ +#!/usr/bin/env python3 +"""Deterministic scoring and rendering for a skill benchmark run. + +A pure transform: given a collected run bundle (per-arm transcript usage, +hard-assertion results, and per-trial judge verdicts) it produces the results +model and a self-contained HTML report. No LLM, no network, no clock — the +`generatedAt` stamp is supplied by the caller so the transform stays pure. + +The model is parameterized over the arms and the comparisons, so the same path +serves a two-arm efficacy run and a future three-arm run. This slice exercises +only the two-arm efficacy shape. + +Usage: + benchmark_core.py [--json ] [--html ] + +With no output flags it writes the results model as JSON to stdout. +""" + +import argparse +import html +import json +import sys + +# Opus 4.8 per-million-token rates, in US dollars. +# Cache creation is the 5-minute-TTL write, priced at 1.25x input. +# Cache read is priced at 0.1x input. +# Cost is reported alongside quality but never gates a verdict. +DEFAULT_PRICING = { + "input": 5.0, + "output": 25.0, + "cacheCreation": 6.25, + "cacheRead": 0.5, +} + +USAGE_COMPONENTS = ("input", "output", "cacheCreation", "cacheRead") + +WIN, TIE, LOSS = "WIN", "TIE", "LOSS" + + +def _arm_usage_totals(bundle, arm_id): + """Sum an arm's transcript usage across every case and trial.""" + totals = {c: 0 for c in USAGE_COMPONENTS} + turns = 0 + for case in bundle["cases"]: + for trial in case["trials"]: + usage = trial["usage"][arm_id] + for c in USAGE_COMPONENTS: + totals[c] += usage.get(c, 0) + turns += usage.get("turns", 0) + return totals, turns + + +def _arm_metrics(bundle, arm_id, pricing): + components, turns = _arm_usage_totals(bundle, arm_id) + raw_tokens = sum(components.values()) + imputed_cost = sum(components[c] * pricing[c] for c in USAGE_COMPONENTS) / 1_000_000 + # Cost-equivalent tokens: the dollar cost expressed in units of base input + # tokens, so a single figure captures the pricing-weighted total. + cost_equivalent_tokens = round(imputed_cost * 1_000_000 / pricing["input"]) + return { + "turns": turns, + "rawTokens": raw_tokens, + "components": components, + "costEquivalentTokens": cost_equivalent_tokens, + "imputedCost": imputed_cost, + } + + +def _collapse_trial(verdict, comparison): + """A trial's head-to-head becomes WIN, TIE, or LOSS for the new arm. + + A tie is a non-win. Any winner that is neither the new nor the against arm + (a malformed verdict) is treated as a tie rather than crashing the run. + """ + winner = verdict.get("winner") + if winner == comparison["new"]: + return WIN + if winner == comparison["against"]: + return LOSS + return TIE + + +def _efficacy_pass(wins, losses): + """Efficacy passes when wins >= 3 and losses <= 1 across the 5 trials.""" + return wins >= 3 and losses <= 1 + + +def _score_case(case, comparisons): + hard_failed = any( + t.get("hard", {}).get("ran") and not t["hard"].get("pass", False) + for t in case["trials"] + ) + + per_comparison = {} + case_passed = not hard_failed + for comp in comparisons: + strip, flagged = [], [] + for i, trial in enumerate(case["trials"]): + outcome = _collapse_trial(trial["comparisons"][comp["id"]], comp) + strip.append(outcome) + if outcome == LOSS: + flagged.append(i) + wins = strip.count(WIN) + ties = strip.count(TIE) + losses = strip.count(LOSS) + head_to_head = _efficacy_pass(wins, losses) + # A failed hard assertion fails the case outright regardless of the + # head-to-head. + passed = head_to_head and not hard_failed + per_comparison[comp["id"]] = { + "trials": strip, + "wins": wins, + "ties": ties, + "losses": losses, + "passed": passed, + "flaggedLosses": flagged, + } + case_passed = case_passed and passed + + return { + "name": case["name"], + "description": case.get("description", ""), + "softCriteria": case.get("softCriteria", []), + "hardFailed": hard_failed, + "comparisons": per_comparison, + "efficacyPassed": case_passed, + } + + +def build_results(bundle): + """Turn a run bundle into the results model.""" + pricing = DEFAULT_PRICING + comparisons = bundle["comparisons"] + arms = bundle["arms"] + + arm_metrics = {a["id"]: _arm_metrics(bundle, a["id"], pricing) for a in arms} + cases = [_score_case(c, comparisons) for c in bundle["cases"]] + + efficacy_green = all(c["efficacyPassed"] for c in cases) + + return { + "skill": bundle["skill"], + "generatedAt": bundle.get("generatedAt", ""), + "temperature": bundle.get("temperature"), + "trialsPerCase": bundle.get("trialsPerCase"), + "armShape": "two-arm" if len(arms) == 2 else f"{len(arms)}-arm", + "arms": arms, + "comparisons": comparisons, + "efficacyVerdict": "green" if efficacy_green else "red", + "armMetrics": arm_metrics, + "cases": cases, + } + + +# --- HTML rendering ----------------------------------------------------------- + +_STYLE = """ +:root { color-scheme: light; } +body { font: 15px/1.5 -apple-system, system-ui, sans-serif; margin: 0; padding: 2rem; + color: #1a1a1a; background: #fafafa; } +.wrap { max-width: 60rem; margin: 0 auto; } +h1 { font-size: 1.4rem; margin: 0 0 .25rem; } +.badge { display: inline-block; padding: .2rem .7rem; border-radius: 999px; + font-weight: 600; font-size: .85rem; } +.badge.green { background: #d6f5df; color: #0f6b34; } +.badge.red { background: #fbdcdc; color: #9b1c1c; } +.meta { color: #666; font-size: .85rem; margin: .5rem 0 1.5rem; } +table { border-collapse: collapse; width: 100%; margin: .5rem 0; font-size: .9rem; } +th, td { text-align: right; padding: .4rem .6rem; border-bottom: 1px solid #e5e5e5; } +th:first-child, td:first-child { text-align: left; } +thead th { border-bottom: 2px solid #ccc; } +.footnote { color: #777; font-size: .8rem; margin: .3rem 0 1.5rem; } +.case { border: 1px solid #e5e5e5; border-radius: 8px; padding: 1rem 1.25rem; + margin: .75rem 0; background: #fff; } +.case.fail { border-color: #f0b6b6; } +.case h3 { margin: 0 0 .2rem; font-size: 1rem; } +.case .desc { color: #666; font-size: .85rem; margin: 0 0 .6rem; } +.strip { display: flex; gap: .3rem; margin: .3rem 0; flex-wrap: wrap; } +.cell { width: 2.6rem; text-align: center; padding: .2rem 0; border-radius: 4px; + font-size: .72rem; font-weight: 700; letter-spacing: .02em; } +.cell.WIN { background: #d6f5df; color: #0f6b34; } +.cell.TIE { background: #eee; color: #555; } +.cell.LOSS { background: #fbdcdc; color: #9b1c1c; } +.result { font-size: .85rem; font-weight: 600; } +.result.pass { color: #0f6b34; } +.result.fail { color: #9b1c1c; } +.tag { font-size: .75rem; color: #9b1c1c; font-weight: 600; } +.crit { color: #555; font-size: .82rem; margin: .5rem 0 0; padding-left: 1.1rem; } +""" + + +def _fmt_int(n): + return f"{n:,}" + + +def _fmt_cost(d): + return f"${d:,.4f}" + + +def render_html(results): + e = html.escape + verdict = results["efficacyVerdict"] + out = [] + out.append("
") + out.append(f"

Benchmark — {e(results['skill'])}

") + out.append( + f"Efficacy: " + f"{'GREEN' if verdict == 'green' else 'RED'}" + ) + + meta_bits = [f"arm shape: {e(results['armShape'])}"] + if results.get("trialsPerCase") is not None: + meta_bits.append(f"{results['trialsPerCase']} trials/case") + if results.get("temperature") is not None: + meta_bits.append(f"temperature {results['temperature']}") + if results.get("generatedAt"): + meta_bits.append(e(results["generatedAt"])) + out.append(f"
{' · '.join(meta_bits)}
") + + # Per-arm cost table. + out.append("" + "") + for arm in results["arms"]: + m = results["armMetrics"][arm["id"]] + out.append( + f"" + f"" + f"" + f"" + ) + out.append("
ArmTurnsRaw tokensCost-equiv tokensImputed cost
{e(arm['label'])}{_fmt_int(m['turns'])}{_fmt_int(m['rawTokens'])}{_fmt_int(m['costEquivalentTokens'])}{_fmt_cost(m['imputedCost'])}
") + out.append( + "

Absolute cost is inflated by shared-context cache " + "overhead, so the trustworthy signal is the new-vs-no-skill ratio, not the " + "absolute figures.

" + ) + + # Cases in stable authored order. + for case in results["cases"]: + cls = "case fail" if not case["efficacyPassed"] else "case" + out.append(f"
") + out.append(f"

{e(case['name'])}

") + if case["description"]: + out.append(f"

{e(case['description'])}

") + if case["hardFailed"]: + out.append("

Hard assertion failed — case fails outright.

") + for comp in results["comparisons"]: + c = case["comparisons"][comp["id"]] + out.append(f"
{e(comp['label'])}
") + out.append("
") + for outcome in c["trials"]: + out.append(f"{outcome}") + out.append("
") + rc = "pass" if c["passed"] else "fail" + summary = (f"{c['wins']}W / {c['ties']}T / {c['losses']}L — " + f"{'PASS' if c['passed'] else 'FAIL'}") + out.append(f"
{summary}
") + if c["flaggedLosses"]: + trials = ", ".join(f"#{i + 1}" for i in c["flaggedLosses"]) + out.append(f"
Loss flagged for review: trial {trials}
") + if case["softCriteria"]: + out.append("
    ") + for crit in case["softCriteria"]: + out.append(f"
  • {e(crit)}
    • ") + out.append("
") + out.append("
") + + out.append("
") + + title = f"Benchmark — {e(results['skill'])}" + return ( + "" + f"{title}" + + "".join(out) + + "" + ) + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("bundle", help="path to the run-bundle JSON") + parser.add_argument("--json", dest="json_out", help="write the results model here") + parser.add_argument("--html", dest="html_out", help="write the HTML report here") + args = parser.parse_args(argv) + + with open(args.bundle) as f: + bundle = json.load(f) + + results = build_results(bundle) + + if args.json_out: + with open(args.json_out, "w") as f: + json.dump(results, f, indent=2) + if args.html_out: + with open(args.html_out, "w") as f: + f.write(render_html(results)) + if not args.json_out and not args.html_out: + json.dump(results, sys.stdout, indent=2) + sys.stdout.write("\n") + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/skills/axi-review/basic-cli-review/case.md b/tests/skills/axi-review/basic-cli-review/case.md new file mode 100644 index 0000000..9c6837c --- /dev/null +++ b/tests/skills/axi-review/basic-cli-review/case.md @@ -0,0 +1,17 @@ +--- +description: Review a tiny CLI against the 10 AXI principles and produce a report card. +--- +## Prompt +There is a small command-line tool named `greet` in the current directory (`./greet`). +Review it against the 10 AXI (Agent eXperience Interface) principles by running it black-box, and give me a report card that scores each principle with a concrete verdict and a fix for anything that isn't a clean pass. + +## Hard assertions +```sh +grep -qiE 'AXI|principle' "$OUTPUT" +grep -qiE 'PASS|PARTIAL|FAIL' "$OUTPUT" +``` + +## Soft criteria +- The review scores each of the 10 AXI principles with an explicit verdict grounded in the tool's actual observed output, not in guesses about its source. +- Every principle that is not a clean pass carries a concrete, actionable fix. +- The review is driven by running the `greet` tool and citing what it emitted, rather than by generic CLI advice. diff --git a/tests/skills/axi-review/basic-cli-review/fixture/greet b/tests/skills/axi-review/basic-cli-review/fixture/greet new file mode 100755 index 0000000..4f8243d --- /dev/null +++ b/tests/skills/axi-review/basic-cli-review/fixture/greet @@ -0,0 +1,41 @@ +#!/usr/bin/env sh +# greet: a tiny CLI, deliberately imperfect, for exercising an AXI review. + +usage() { + cat <<'EOF' +greet — say hello + +USAGE: + greet hello print a greeting + greet json print the greeting as JSON + greet --help show this help +EOF +} + +case "$1" in + --help|-h|help) + usage + ;; + hello) + if [ -z "$2" ]; then + echo "error: missing name" >&2 + exit 1 + fi + echo "Hello, $2!" + ;; + json) + if [ -z "$2" ]; then + echo "error: missing name" >&2 + exit 1 + fi + printf '{"greeting":"Hello","name":"%s"}\n' "$2" + ;; + "") + usage >&2 + exit 2 + ;; + *) + echo "error: unknown command '$1'" >&2 + exit 2 + ;; +esac -- 2.47.3 From ab2c05acb2bb8a4d251aab1e8161134d777ef42d Mon Sep 17 00:00:00 2001 From: alexion Date: Fri, 24 Jul 2026 09:01:24 -0400 Subject: [PATCH 2/5] feat: add dev shell with python3 and the benchmark skill (task 0004) nix develop places benchmark-skill into ./.claude/skills/ (so /benchmark-skill is available in this repo) and puts python3 on PATH to run the deterministic core without a one-off nix shell. --- flake.nix | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/flake.nix b/flake.nix index dc1cd72..d4f5f9e 100644 --- a/flake.nix +++ b/flake.nix @@ -92,6 +92,15 @@ inherit pkgs; }; }; + + # A working shell for developing skills in this repo. + # python3 runs the benchmark skill's deterministic core. + # The benchmark-skill is placed into ./.claude/skills/ on entry, so + # /benchmark-skill is available while working in this repo. + devShells.default = pkgs.mkShell { + packages = [ pkgs.python3 ]; + shellHook = mkSkillsShellHook [ (skillPackages pkgs).benchmark-skill ]; + }; } ); } -- 2.47.3 From 613e423617269e6677d7dc1338e07f724382e58a Mon Sep 17 00:00:00 2001 From: alexion Date: Fri, 24 Jul 2026 14:28:05 -0400 Subject: [PATCH 3/5] feat: isolate benchmark arms in fixture-only worlds (task 0008) Run each benchmark arm in a hermetic fixture-only world materialized outside the repo, so the no-skill baseline can no longer discover the skill's assets on disk. The new-skill arm's skill is materialized to an isolated temp path it is pointed at; the baseline gets no skill and is told to stay within its world. A live 5-trial axi-review run confirms the fix: all baseline transcripts are clean of AXI-PRINCIPLES.md, the baselines invent their own generic rubric (the honest counterfactual), and the Efficacy verdict is green 5/5 where the contaminated 0004 run had scored red 2/5. Runner prose only; the deterministic core is untouched. --- .claude/tasks/0008-arm-isolation-hardening.md | 66 +++++++++++++++++++ skills/benchmark-skill/SKILL.md | 20 ++++-- 2 files changed, 79 insertions(+), 7 deletions(-) create mode 100644 .claude/tasks/0008-arm-isolation-hardening.md diff --git a/.claude/tasks/0008-arm-isolation-hardening.md b/.claude/tasks/0008-arm-isolation-hardening.md new file mode 100644 index 0000000..dc1f4c4 --- /dev/null +++ b/.claude/tasks/0008-arm-isolation-hardening.md @@ -0,0 +1,66 @@ +--- +spec: skill-benchmarking +blocked-by: 0004-efficacy-benchmark-tracer-bullet +--- + +## What to build + +The isolation fix the tracer bullet flagged as a follow-up: give every arm a hermetic **fixture-only world** so the no-skill baseline stops discovering the skill on disk, turning that run's contaminated red into an honest counterfactual. + +In 0004 the arms ran with the repo as their working directory. +The no-skill baseline, solving its task inside the real repo, grepped up `skills/axi-review/AXI-PRINCIPLES.md` and graded against it — its own output admitted as much — so the measured "no-skill" arm was not the counterfactual of the skill not existing. +The Efficacy verdict came out red as a direct artifact of that contamination rather than a real result. + +Each arm-and-trial now runs in a fixture-only world. +The arm subagent's working directory is its fresh fixture copy rather than the repo root, and nothing under `skills/` or the grading `tests/` tree sits on any path it explores from there. +This is what makes the no-skill baseline honest, because it can no longer find and read the skill's assets, and it also stops any arm from reading its own case's soft criteria or hard assertions and tuning its answer to the bar it will be judged against. + +A with-skill arm is handed its skill as an **isolated temp materialization** placed outside the fixture, and the subagent is pointed there to force-invoke it. +For this slice's two-arm shape only the new-skill arm has one: a copy of the working-tree skill directory, so it reflects uncommitted edits and carries none of its repo surroundings. +The no-skill arm is handed nothing. +Each temp materialization is cleaned up after the run, like the fresh fixture copies. + +Isolation here is deliberately soft. +An in-session subagent shares the machine and could in principle reach the repo by absolute path, so the baseline is additionally instructed to stay within its working directory. +Relocating each arm's world and giving that instruction moves contamination from near-certain to requiring an arm to deliberately wander outside its world. +A hard filesystem guarantee would require an OS sandbox and is out of scope. + +`$WORLD` binds to the arm's fixture copy — now also its working directory — and `$OUTPUT` to its captured final message, with the hard-assertion gate unchanged from 0004. +The deterministic core and its fixture unit test are untouched, because isolation is a runner orchestration concern rather than a scoring one. + +Correctness is established by re-running the harness on the real skill and reading the result. +A live `/benchmark-skill axi-review` run confirms the baseline arm's transcript no longer references `AXI-PRINCIPLES.md` or any `skills/` or `tests/` path, and reports the now-uncontaminated Efficacy verdict that replaces 0004's documented-artifact red. + +This slice establishes the "materialize a with-skill arm's skill into an isolated temp path" pattern that the previous-version arm reuses for the old skill version. + +## Acceptance criteria + +- [x] Each arm-and-trial subagent runs with its working directory set to its fresh fixture copy, never the repo root. +- [x] Nothing under `skills/` or the `tests/` tree is present on any relative path an arm explores from its world. +- [x] The new-skill arm's skill is materialized into an isolated temp path outside the fixture — a copy of the working-tree directory that reflects uncommitted edits — and the subagent is pointed there to force-invoke it, with the temp materialization cleaned up after the run. +- [x] The no-skill baseline is handed no skill materialization and is instructed to stay within its working directory. +- [x] `$WORLD` binds to the arm's fixture copy and `$OUTPUT` to its captured final message, with the hard-assertion gate semantics unchanged from 0004. +- [x] The deterministic core and its fixture unit test are unchanged, since isolation is an orchestration concern and not a scoring one. +- [x] A live `/benchmark-skill axi-review` run confirms the baseline arm's transcript references no `AXI-PRINCIPLES.md`, `skills/`, or `tests/` path, and its Efficacy verdict reflects the honest counterfactual rather than the earlier contamination artifact. + +## Implementation Notes + +Changed `skills/benchmark-skill/SKILL.md` only: the world/materialization steps in §3, the `$WORLD` binding in §4, and the scratch cleanup in §6. +`core/benchmark_core.py` and its check were left untouched, and the unchanged core was confirmed to still run clean on its committed fixture. + +- **Worlds are materialized outside the repo, not merely relocated within it.** + The spec calls for a "fixture-only world", and the sharp form that actually kills contamination is a world under a system temp path (`mktemp -d`), because a world still under `tests/.reports/` leaves `skills/` and the grading `tests/` tree reachable by upward navigation. + The prose now says so explicitly. +- **Live verification passed and reversed 0004's contaminated result.** + A real `/benchmark-skill axi-review`-shaped run of 5 paired trials, orchestrated through the workflow mechanism with each arm's world under a temp path, came back with the new arm winning 5/5 and the Efficacy verdict **green** — where 0004 had scored red at 2/5 purely because the baseline had read the skill's `AXI-PRINCIPLES.md`. + All five no-skill-baseline transcripts were clean of any `AXI-PRINCIPLES.md`, `skills/`, or `tests/` reference, and the baselines instead invented their own generic CLI rubric — the honest counterfactual of the skill not existing. +- **Soft isolation is realized by instruction, since a subagent's working directory is not hard-settable through the workflow surface.** + Each arm is told to `cd` into its world first and work only there, and the baseline is additionally told to stay within it. + This is the achievable in-session bar, and hard OS-sandbox isolation stays deferred per the spec's "Out of Scope". +- **Review follow-through.** + A spec-axis review caught that "copy the case fixture into the world" could be read as nesting the fixture one level down, which would break `$WORLD/` predicates. + The wording now states that the world *is* the fresh fixture copy, so a `$WORLD/` predicate resolves against the fixture root. + A semicolon in one acceptance-criterion line was recast. +- **The feature spec is not part of this commit.** + `.claude/spec/skill-benchmarking.md` carries the isolation decisions but remains untracked, matching how 0004 was committed (code and task file, without the spec), and committing it here would pull in prose-standard breaches from sections written earlier. +- The materialization pattern this establishes for the new arm is reused by 0005's previous-version arm, which points at a temp checkout of the skill at main's `HEAD` instead of a copy of the working-tree directory. diff --git a/skills/benchmark-skill/SKILL.md b/skills/benchmark-skill/SKILL.md index b4195bb..ed5239c 100644 --- a/skills/benchmark-skill/SKILL.md +++ b/skills/benchmark-skill/SKILL.md @@ -57,15 +57,21 @@ Use the workflow mechanism (the `Workflow` tool) to fan out the arms and judges. For each case, for each of the 5 trials, run both arms, then judge the pair. **Both arms receive the identical `## Prompt`, authored once and arm-agnostically.** -Each arm-and-trial combination runs against a **fresh copy** of the case fixture, made under `tests/.reports/.work/`, so writes never leak between runs. For a conversation-driven case, inject the `## Seed` transcript as the subagent's prior context before the prompt. +**Give each arm-and-trial a hermetic fixture-only world.** +Make a fresh copy of the case fixture *outside the repo* — under a system temp path such as one from `mktemp -d`, never under `tests/.reports/`. +That copy is the **world**: it is the arm subagent's **working directory** and the `$WORLD` the hard-assertion gate reads, so a `$WORLD/` predicate resolves against the fixture root. +Because the world lives outside the repo, nothing under `skills/` or the grading `tests/` tree sits on any path the arm reaches from there, so the baseline cannot discover the skill's assets and no arm can read its own case's soft criteria or hard assertions. +Each arm-and-trial gets its own fresh world, so writes never leak between them. + - **New-skill arm** — force-invoke the skill. - Point the subagent at the skill's own directory, tell it to use that skill, and have it read the skill's `SKILL.md` and any assets it references, so the real skill machinery is exercised. - Give it the fresh fixture copy as its working target and the prompt. + Materialize the skill into an **isolated temp path outside the world** — a copy of its working-tree directory, so it reflects uncommitted edits and carries none of its repo surroundings. + Point the subagent at that copy, tell it to use that skill, and have it read the skill's `SKILL.md` and any assets it references, so the real skill machinery is exercised. + Its working directory is the fresh world, and it also receives the prompt. - **No-skill baseline arm** — the honest counterfactual of the skill not existing. - Give it the bare prompt with the skill absent from its context. - Do not mention the skill or hint that one exists. + Materialize no skill for it: give it the bare prompt with the skill absent from its context, and do not mention the skill or hint that one exists. + Instruct it to stay within its working directory, since the isolation is soft and the subagent shares the machine. Capture each arm's **final message** to a file — this is the `$OUTPUT` the hard-assertion gate reads and the text the judge compares. @@ -77,7 +83,7 @@ Have it return a single winner: A, B, or tie. ## 4. Run the hard-assertion gate Run the case's `## Hard assertions` against the **new arm only**, once per trial. -Bind `$OUTPUT` to the path of that trial's new-arm final message and `$WORLD` to the path of that trial's fresh new-arm fixture copy. +Bind `$OUTPUT` to the path of that trial's new-arm final message and `$WORLD` to that trial's new-arm world (its fixture copy, which is the arm's working directory). Run each predicate. A non-zero exit fails the assertion. Record pass/fail per trial — a failed hard assertion fails the case outright regardless of the head-to-head. @@ -144,7 +150,7 @@ The core collapses each efficacy trial to WIN, TIE, or LOSS for the new arm (a t It renders a self-contained HTML report: the Efficacy badge and run metadata, a two-row per-arm cost table (footnoted that absolute cost is inflated by shared-context cache overhead, so the trustworthy signal is the new-vs-no-skill ratio), and the cases in stable authored order. Cost is reported alongside quality but never gates the verdict. -Clean up the `tests/.reports/.work/` scratch directory when done. +Clean up the scratch when done: remove the `tests/.reports/.work/` directory and every per-arm world and skill materialization you created under the system temp path. Done when `tests/.reports/.html` exists. Report the Efficacy verdict, the path to the report, and any flagged losses to the user. -- 2.47.3 From 3e96db664c960af97574e90cd2e7036fb2f181e0 Mon Sep 17 00:00:00 2001 From: alexion Date: Fri, 24 Jul 2026 14:42:20 -0400 Subject: [PATCH 4/5] docs: add skill-benchmarking spec Bring the feature's design record into git (it was authored across the benchmark tasks but never committed). Cleaned to the repo's prose standards on the way in: recast prose semicolons as separate sentences and split multi-sentence lines to one sentence per line. --- .claude/spec/skill-benchmarking.md | 288 +++++++++++++++++++++++++++++ 1 file changed, 288 insertions(+) create mode 100644 .claude/spec/skill-benchmarking.md diff --git a/.claude/spec/skill-benchmarking.md b/.claude/spec/skill-benchmarking.md new file mode 100644 index 0000000..e03794f --- /dev/null +++ b/.claude/spec/skill-benchmarking.md @@ -0,0 +1,288 @@ +## Problem Statement + +I write agent skills, but I have no way to know whether a skill is actually pulling its weight. +A skill can read well, be faithfully followed, and still make the agent's output no better than it would have been with no skill at all. +That skill is worthless, yet nothing in my current setup would catch it. +I want a repeatable way to prove that a given skill genuinely improves the agent's work before I trust it. + +I also edit skills over time, and an edit can quietly make a skill worse. +A skill that still beats no-skill but has degraded from its released version is a regression I currently cannot see, because measuring only against no-skill sets too low a bar to catch it. +I want the same harness to tell me, the moment I edit a released skill, whether my change held quality steady or regressed against the version that is already trusted. + +## Solution + +A skill-benchmarking convention plus an agent-run harness that measures a skill's **efficacy against a no-skill baseline** and, whenever a released version exists, its **regression against that previous version**. + +For each skill I author a set of test cases under a top-level `tests/` tree. +Running the harness on a skill executes every case as a controlled experiment: the same realistic prompt is given to several arms, several times, and a blind judge decides which arm's output better satisfies the case's author-written expectations. +Every run has a **new-skill arm** (my working tree) and a **no-skill baseline arm**. +Whenever the skill already exists on the main branch, the run adds a third **previous-version arm** materialized from main's `HEAD`. +Two blind head-to-head comparisons fall out of this: **efficacy** (new-vs-no-skill) tells me whether the skill earns its keep, and **regression** (new-vs-old) tells me whether my edit made it worse. + +A skill "passes" efficacy only when the new arm reliably beats the no-skill baseline — a skill the agent follows faithfully but that does not beat the baseline is reported as adding nothing. +A skill "passes" regression when the new arm reliably does *not* lose to the previous version — a clean edit that holds quality steady passes, and only a real degradation fails. +The two are reported as **separate verdicts**, because "this skill never worked" and "I just broke a skill that did work" are different problems. + +The harness runs in-session as an "AI script": it orchestrates the arms and the judge as subagents, and it hands the boring mechanical work — counting results, summing token usage, applying the pass rules, rendering the report — to a small committed program so the numbers are exact and reproducible. +It produces a self-contained HTML report with the two verdicts, a per-arm cost table, trend lines across recent runs, and drill-down evidence for anything that failed or regressed. + +The whole thing is a distributed skill named `benchmark-skill`, invoked as `/benchmark-skill `, so any repo that follows the `tests/` convention can benchmark its own skills. + +## User Stories + +1. As a skill author, I want to prove a skill improves the agent's output versus doing the task with no skill, so that I do not keep a skill that reads well but adds nothing. +2. As a skill author, I want each test case to pin down explicit expectations I write myself, so that the judge measures against my intent rather than inventing its own bar. +3. As a skill author, I want the same realistic prompt given to every arm, so that the only difference measured is which version of the skill (or no skill) produced the output. +4. As a skill author, I want the with-skill arms to actually load and follow the skill, so that I am measuring the skill's effect and not whether it happened to trigger. +5. As a skill author, I want each case to carry a hermetic, committed fixture, so that a run is reproducible and needs no live external state. +6. As a skill author who writes conversation-driven skills, I want a case to seed a prior conversation, so that I can benchmark skills whose input is a discussion rather than a file tree. +7. As a skill author, I want a mechanical, deterministic gate (hard assertions) separate from the judged comparison, so that a malformed output fails immediately without spending judgment on it. +8. As a skill author, I want a run repeated several times at a realistic temperature, so that the result reflects whether the skill reliably helps rather than helping once by luck. +9. As a developer iterating on a skill, I want trend lines across recent runs, so that I can tell a real change from run-to-run noise after I edit the skill. +10. As a developer, I want a per-arm table of turns, tokens, and imputed cost, so that I can see what the skill's quality gain costs and whether my edit made it more expensive. +11. As a developer, I want the report to open on the verdicts and expand only what failed or regressed, so that a clean run is calm and a problem run puts the evidence in front of me. +12. As a developer, I want a green run to still flag its most fragile case on each axis, so that a skill that barely passed cannot masquerade as robust. +13. As a developer, I want to benchmark a single skill quickly, so that my edit-and-recheck loop is cheap. +14. As a developer, I want to benchmark all skills at once and see a leaderboard, so that I know at a glance which of my skills are green and which regressed. +15. As an operator, I want the whole battery to run on my normal interactive subscription, so that a benchmark does not draw down a separate metered automation credit pool. +16. As a maintainer, I want the mechanical scoring and rendering to be committed, tested code, so that the report's numbers are exact and reproducible rather than re-derived by the agent each run. +17. As a maintainer, I want the test tree kept out of the packaged skills, so that installing a skill never drags its fixtures along. +18. As a maintainer, I want reports and history kept out of git, so that machine-specific run artifacts never get committed. +19. As a developer updating a skill, I want the new version compared against the released version on main and not only against no-skill, so that a skill that silently got worse but still beats no-skill is caught rather than passing green. +20. As a developer, I want efficacy and regression reported as two independent verdicts, so that I can tell "this skill never worked" apart from "I just broke a skill that did work." +21. As a developer, I want a clean refactor that holds output quality steady to pass the regression check, so that a deliberate no-op change is not failed merely for tying the previous version. +22. As a developer, I want the regression comparison to appear automatically the moment I edit a released skill, so that I never have to remember to opt into it. + +## Implementation Decisions + +### What is measured + +- The benchmark measures **efficacy against a no-skill baseline** and, when a previous version exists, **regression against that previous version**. +Trigger-correctness (does a model-invokable skill fire on a realistic prompt) and format-conformance are explicitly not measured here. +They are separable evals that can be added later without disturbing this one. +- Both dimensions are judged on **output quality**, with cost reported alongside but **never gating**. +A skill that improves quality is worth keeping even when it costs more tokens, and the cost columns are there to inform, not to fail. + +### The arms + +- Every case is a controlled experiment with a **new-skill arm** (my working tree) and a **no-skill baseline arm**, and — whenever the skill already exists on the main branch — a third **previous-version arm** materialized from main's `HEAD`. +- Two blind head-to-head comparisons are drawn from the arms per trial: **efficacy** pairs the new arm against the no-skill arm, and **regression** pairs the new arm against the previous-version arm. +- Every arm is given the **identical realistic prompt**, authored once and arm-agnostically, so the arms differ only in which skill (or no skill) is present. +- The new arm and the previous-version arm are both **force-invoked**: each subagent is pointed at its own isolated skill materialization and told to use it, reading that skill's own file and any assets it references, so the real skill machinery is exercised rather than a reconstruction. +The previous-version arm is force-invoked identically to the new arm, pointed at the materialization of the old skill instead of the new arm's copy of the working-tree directory, so the skill version is the only difference between them. +- The baseline arm receives the bare prompt with the skill **absent from its context**, which is the honest counterfactual of the skill not existing. +- Force-invoking uniformly is a deliberate consequence of running in-session. +An earlier design varied arm construction by whether a skill was model-invokable, but in-session subagents cannot faithfully reproduce skill auto-discovery, so the harness force-invokes every with-skill arm and leaves triggering to a future, separate eval. + +### The regression comparison + +- The previous version is **always main's `HEAD`** — the repo's default branch, resolved rather than hardcoded to the literal name "main" — and the new version is **the working tree**. +Anchoring to the released state on main, rather than to whatever was last committed on the working branch, means the regression question is always "did my in-progress edit degrade from the trusted version," which is the moment of real risk. +- The third arm runs **only when the skill directory differs from main's `HEAD`**. +A brand-new skill that does not yet exist on main, or a skill whose directory is unchanged from main, has no meaningful previous version, so the run degrades to the two-arm efficacy-only shape. +- **Cases are held fixed to the working tree.** +All arms run against today's prompt, today's fixture, today's expectations, so the skill version is the only variable differing between the new and previous-version arms. +Letting a case drift with the version would change two things at once and rob the new-vs-old verdict of meaning. +The conservative consequence is accepted: if a case's expectations were rewritten alongside the skill, the old version is judged against a bar it was never written for, which surfaces a possible regression for a human to eyeball rather than silently excusing it. +- The **hard-assertion gate applies to the new arm only**, exactly as before. +The gate exists to catch the current skill emitting malformed output. +Running it against the previous version would manufacture spurious failures for any assertion introduced in the very edit under test, and the no-skill arm was already never gated. +- The **judge and the soft criteria are reused unchanged** for the regression comparison. +A case's author-written soft criteria describe what a good answer looks like, which does not depend on the opponent, so the same blind judge grounds both head-to-heads and simply receives a different pair of outputs. + +### Expectations per case + +- A case's expectations come in two tiers. +- **Hard assertions** are executable shell predicates that form a deterministic, LLM-free gate. +They run against two provided values: `$OUTPUT`, the path to the arm's captured final message, and `$WORLD`, the path to that arm's fresh fixture copy. +A non-zero exit fails the assertion. +A case may have zero hard assertions. +- **Soft criteria** are natural-language statements the judge grounds the head-to-heads on. +A case has at least one. +- This snippet, from the case-format decision, encodes the authoring contract more precisely than prose: + +``` +--- +description: +--- +## Prompt + + +## Seed (optional; conversation-based skills only) +**User:** ... +**Assistant:** ... + +## Hard assertions (executable; $OUTPUT = arm's final message, $WORLD = its fixture copy) +​```sh +test -f "$WORLD/review.md" +grep -qE '' "$OUTPUT" +​``` + +## Soft criteria +- +``` + +### The judge + +- The judge is an **LLM subagent**, one **blind judge per trial per comparison**, shown both arms' outputs as unlabelled A and B with the **order randomized** per trial. +- The judge is grounded on the case's soft criteria rather than free-forming its own standard, and its prompt explicitly instructs it to discount mere length and formatting differences, since a skill can otherwise "win" by being more verbose. +- The same judge machinery serves efficacy and regression. +Only the pair of outputs it is handed differs. +- Variance is handled by repeating trials rather than by a per-trial panel. +A panel is escalated to only for a comparison whose trials come back consistently split. + +### Trials, temperature, and the pass rules + +- Each case runs **5 paired trials** at a **realistic temperature**, configurable per skill, with **no reuse of arms or judgments across arms or runs**. +Trial *i*'s new-skill output is judged against trial *i*'s no-skill output for efficacy and against trial *i*'s previous-version output for regression. +A realistic temperature is chosen because most skills guide open-ended reasoning, and a big part of a skill's value is making a good outcome reliable across that variance, which temperature zero would hide. +A per-skill temperature-zero override remains available for a skill that genuinely wraps a mechanical task. +- The **efficacy pass rule**: + - Per trial, the graded head-to-head collapses to **WIN, TIE, or LOSS** for the new arm, and a tie counts as a non-win. + - Per case, efficacy **passes** when **wins ≥ 3 and losses ≤ 1** across the 5 trials, and **any** loss is flagged in the report for human review. + A failed hard assertion fails the case outright regardless of the head-to-head. +- The **regression pass rule** inverts, because holding quality steady is the goal: + - Per trial, the new-vs-old head-to-head collapses to **WIN, TIE, or LOSS** for the new arm, where a TIE means "as good as the previous version" (a success), a LOSS is the regression being hunted, and a WIN is a bonus improvement. + - Per case, regression **passes** when **losses ≤ 1** across the 5 trials, with **no wins floor** — ties and wins both count as non-regressions — and **any** loss is flagged for human review. + - The wins floor is dropped deliberately, so a deliberate no-op edit that ties the previous version is a pass rather than a failure. +- Tolerating one loss on each axis, rather than demanding zero, keeps a single spurious judge miscall at temperature above zero from reddening a genuinely good result, while still surfacing every regression and failing on a second loss. + +### The two verdicts + +- A skill carries **two independent verdicts**: an **Efficacy** verdict, green when every case passes efficacy, and a **Regression** verdict, green when no case regressed. +- The two are reported side by side rather than collapsed into one, because their four crossings carry genuinely different meanings: + - efficacy-green + regression-green → the edit is safe and the skill earns its keep. + - efficacy-green + **regression-red** → the skill still beats no-skill but got worse than the released version — the exact regression this feature exists to catch. + - **efficacy-red** + regression-green → the skill does not beat no-skill, but the edit did not make it worse (it was already dead weight). + - efficacy-red + regression-red → the skill does not beat baseline and the edit made it worse. +- The report headline reads the two verdicts together into one "so what" (for example, "Still valuable, but this edit regressed two cases"). +- On a two-arm run with no previous version, only the Efficacy verdict is meaningful and the Regression verdict reads not-applicable. + +### Fixtures and isolation + +- Every case ships its own **hermetic, committed fixture**, and each arm-and-trial combination runs against a **fresh copy** of it, so writes from one run never leak into another. +- File-and-tree skills get a fixture directory. +Conversation-driven skills get a **seed transcript** in role-tagged form, injected as the subagent's prior context before the prompt. +- Each arm-and-trial runs in a **fixture-only world**: the arm subagent's working directory is its fresh fixture copy rather than the repo root, and nothing under `skills/` or the grading `tests/` tree sits on any path it explores from there. +This is what makes the no-skill baseline an honest counterfactual, because it cannot discover and read the skill's assets off disk, and it also stops any arm from reading its own case's soft criteria or hard assertions and tuning its answer to the bar it will be judged against. +- A with-skill arm is handed its skill as an **isolated temp materialization** placed outside the fixture, and is pointed there to force-invoke it. +The new-skill arm's materialization is a copy of the working-tree skill directory, so it reflects uncommitted edits. +The previous-version arm's is the same skill directory checked out at main's `HEAD`, since each skill is packaged as its own self-contained derivation and does not need the rest of the repo. +The no-skill arm is handed nothing. +Each temp materialization is cleaned up after the run, like the fresh fixture copies. +- Isolation here is deliberately **soft**: an in-session subagent shares the machine and could in principle reach the repo by absolute path. +Relocating each arm's working directory to its fixture, and instructing the baseline to stay within it, moves contamination from near-certain to requiring an arm to deliberately wander outside its world. +A hard filesystem guarantee would require an OS sandbox, which is out of scope. +- A live-target fixture that seeds and tears down an external resource per run is out of scope for now. + +### Location and the test tree + +- Tests live in a top-level **`tests/` tree that mirrors `skills/` by full path**, so a skill's cosmetic nesting is mirrored and its tests sit at the same relative path. +- Placing tests outside `skills/` is deliberate: each skill is packaged as its own derivation and placed into a project's skills directory, so bundling fixtures inside a skill would bloat every installation. +The test tree is neither content nor integration output — it is repo infrastructure that is never packaged or placed. +- A skill's tests sit as case directories at the mirror point, each holding a `case.md` and an optional fixture. + +### The harness + +- The harness runs **in-session as an AI script**, orchestrating the arms and judges as subagents via the workflow mechanism. +- Running in-session is a billing decision: it stays on my **interactive subscription quota**, avoiding the separate metered automation credit pool that headless `claude -p` and the Agent SDK draw from since mid-2026. +- The full per-arm metric set is recovered by **reading each arm-subagent's transcript file**, which records per-message token usage. +Summing that usage yields raw tokens across the four components, the pricing-weighted cost-equivalent-token figure that the gitea-axi bench also reports, and an imputed dollar cost, while the turn count comes from the transcript's tool-call rounds. +- The **mechanical work is done by a small committed program, not by agent reasoning** (see the deterministic core below). + +### The deterministic core + +- A committed program owns every mechanical, non-judgment step so the results are exact and reproducible: summing token usage across every arm and computing the cost figures, collapsing each trial of each comparison to WIN/TIE/LOSS, applying the two pass rules and the two verdicts, parsing `case.md`, rendering the HTML report, and appending to and trimming the run history. +- It is **parameterized over the number of arms and the two comparisons**, so it handles both the three-arm and the degenerate two-arm run through one path. +- It is a pure transform: given the collected run data (per-arm transcript usage, hard-assertion results, and the per-trial judge verdicts for both comparisons) it returns the results model, the rendered HTML, and the updated history. +- It ships with the `benchmark-skill` skill and is invoked by that skill's prose, which keeps the agent confined to the judgment work — running arms and judging — that it is actually good at. + +### The runner skill + +- The runner is a **distributed skill** named `benchmark-skill`, auto-discovered and packaged like any other skill in this repo, and marked non-model-invocable so it is only ever run deliberately. +- It is invoked as `/benchmark-skill ` to benchmark one skill, and with no argument to benchmark every skill. +- Choosing the two-arm or three-arm shape is **fully automatic and carries no new invocation surface**: the harness diffs the skill directory against main's `HEAD` and adds the previous-version arm exactly when a released version exists to compare against. +- Because it is distributed and relies only on the `tests/` convention, any repo following that convention can benchmark its own skills. +- Validating the shape of the `tests/` tree (that a `case.md` parses, that a fixture referenced exists, that a test directory maps to a real skill, and reporting skills that have no tests) is the runner's own responsibility at run time, not a separate build-time check. + +### The report + +- The report uses a **layered shape**: a verdict scorecard on top with progressive disclosure beneath. +A failed or regressed case auto-expands with its losing-trial evidence, while passing cases stay collapsed but can be opened. +- Its sections run top to bottom. +First a header with the **two verdict badges** (Efficacy and Regression), run metadata, and a jump-to-failure link shown only when something failed or regressed. +Then the **trend ribbons**. +Then the **verdict headline** reading both badges together. +Then a **per-arm cost table**. +And last the **cases** in stable authored order. +- There are **two stacked trend ribbons** — an **Efficacy** ribbon and a **Regression** ribbon — each a net-margin (wins minus losses) sparkline over roughly the last seven runs, with verdict-colored per-run dots, the current run ringed, and a readout of the current net, the change versus the previous run, and how many recent runs were green. +The Regression ribbon's dot is **absent for any run that was two-arm**. +The Regression series is comparable across runs only while main is unchanged. +Because the previous-version arm is always main's `HEAD`, merging the branch moves main and resets the meaningful regression history, which is accepted rather than normalized. +The ribbons are what let an edit's real change be told apart from run-to-run noise. +- The **cost table** has **three rows — new-skill, previous-version, no-skill** — each with turns, raw tokens, cost-equivalent tokens, and imputed cost, with a footnote that absolute cost is inflated by shared-context cache overhead and that the trustworthy signals are two ratios: **new-vs-no-skill** (what the skill costs over nothing) and **new-vs-old** (what this edit added or saved). +The previous-version row drops out on a two-arm run. +- Each case panel shows **both comparisons side by side**, each with its per-trial WIN/TIE/LOSS strip and its pass result. +A case **auto-expands when either comparison fails or carries a flagged loss**, so a regression opens the panel even when efficacy is green, and the auto-expanded evidence shows the losing-trial output pair for whichever comparison failed. +- **Cases render in stable authored order and are never reshuffled by verdict**, so their positions are learnable and diff cleanly across runs, with failures reached via auto-expansion and the header's jump link rather than by sorting them to the top. +- A **clean run rests fully collapsed**, except that fragility is flagged **per badge**: a green Efficacy badge carries a "narrowest margin" chip on the case fewest trial-flips from failing efficacy, and a green Regression badge independently carries one on the case sitting at exactly one loss. +A case that is fragile on both axes shows both chips, so a barely-green skill cannot look as safe as a clean sweep on either axis. +- The report carries a **thin, constrained narrative layer** — a headline "so what" and per-case one-liners — under the rule that every sentence it writes must be backed by a value or judge quote visible on the same screen. +The per-trial judge rationale and the mechanical "why it failed" are inherent evidence, not part of that narrative layer. + +### All-skills output and files + +- Benchmarking all skills produces an **index leaderboard** where each skill row carries **both badges (Efficacy and Regression)**. +- The sort promotes **any red first**, with **regressions ordered above efficacy failures** — a regression means "you just broke something that was working," the more urgent signal while iterating — then fragile-but-passing skills, then clean green. +A skill with no previous version reads not-applicable in its Regression cell. +- The leaderboard links out to separate per-skill report files. +- All report artifacts live under a git-ignored `tests/.reports/` directory, flat and keyed by unique skill name, since skill names are globally unique in this repo. +- Per-skill HTML is **latest-only and overwritten each run**, because the longitudinal data lives in a per-skill history file to which the harness appends one summary line per run, capped at roughly the last fifty and trimmed oldest-first. +- Each history line records **both the efficacy and the regression net-margin and pass/fail**, plus a flag for whether the run was two-arm or three-arm, so both ribbons can plot their own sparkline and correctly show a gap for any two-arm run. +- That history is **local and ephemeral** — it lives inside the ignored reports directory and resets if that directory is cleaned — consistent with reports being transient artifacts. +Promoting it to durable, committed history is a later, additive change. + +## Testing Decisions + +- The judgment half of this feature — the arms and the blind judges — is inherently non-deterministic and is **not unit-tested**. +Its correctness is established by running the harness on a real skill and reading the report. +- The one deterministic, testable seam is the **deterministic core**. +A good test here exercises that core's external behavior as its caller observes it: fed **committed fixtures** — sample arm transcripts for all three arms, hard-assertion results, and the per-trial judge verdicts for both comparisons — it must produce the expected per-arm metrics, the expected per-case and per-skill verdicts on both axes, and a correct report including the two ribbons and the three-row cost table. +The **two-arm run is retained as the degenerate "no previous version" case** in the same test, so both shapes are covered. +It is tested at that single seam, not through the internals of the token-summing or the rendering. +- This isolates exactly the error-prone arithmetic, counting, pass-rules, and templating that must not be re-derived by the agent, and it needs no LLM to run. +- Prior art is `checks/shell-hook.nix`, which runs a produced artifact against a fixture project and asserts on the observed result. +The `home-manager-module` check, which builds an output and inspects it, is the same style. +- Whether this fixture test is also wired into `nix flake check` is left open and deferred, in keeping with the earlier decision not to add a build-time check for this feature. +The test can exist and run without being a flake gate. + +## Out of Scope + +- **Trigger-correctness** — whether a model-invokable skill fires on a realistic prompt. +It is a distinct dimension the in-session harness cannot cleanly measure, and it is deferred to a separate eval. +- **Format-conformance** as its own measure, which the efficacy comparison largely subsumes. +- **Live-target fixtures** that seed and tear down an external resource per run. +Only hermetic committed fixtures are supported for now. +- **Headless or API execution** of the battery, which would draw the metered automation credit pool. +A headless run of a single skill remains only a fallback for the unlikely case that transcript files lack usage data. +- **Cost as a gate.** Cost is always reported and never fails a skill on its own. +- **A pinned or blessed baseline, and comparison against arbitrary refs.** The previous version is always main's `HEAD`. +Comparing against a marked version-to-beat or two chosen historical refs is a later, additive refinement. +- **A `--no-regression` fast-path** for skipping the previous-version arm mid-edit. +The shape is fully automatic for now. +An opt-out flag can be added later. +- **Durable or normalized trend history.** History is local and ephemeral, and the regression series is not normalized across a main move, until a later opt-in. +- **Selectable, per-skill-kind arm construction.** The harness force-invokes every with-skill arm uniformly. +- **Hard filesystem isolation of the arms.** Each arm runs in a fixture-only world with its working directory relocated off the repo root, which is soft isolation that an arm could defeat only by deliberately reaching for an absolute repo path. +An OS sandbox that makes the skill genuinely unreachable is a later hardening. + +## Further Notes + +- The gitea-axi benchmark is the **methodology source**, not a template: its honest-baseline discipline, its per-arm comparison table, and its pricing-weighted cost-equivalent-token unit carry over, while its temperature-zero, headless, standalone-script shape does not, because a reasoning skill needs a realistic temperature and the billing reality pushes the run in-session. +- The shared-context cache overhead means a per-arm absolute cost reads higher than a standalone run would. +The inter-arm ratios are what stay trustworthy, and the report says so. +- A prototype of the report was built during design and is the visual reference for the layered shape, the trend ribbons, the fragility chips, and the clean-versus-failing resting states. +- "Fragile," for a green-run chip, means the passing case that is the fewest trial-flips from failing on that badge's axis — closest to dropping under three wins for efficacy, or over one loss for regression. +- Because the previous version is always main's `HEAD`, a change committed directly to main stops being regression-testable once the working tree matches `HEAD`. +Regression is designed for the branch-in-progress workflow (working tree versus released main). +The deferred pinned-baseline feature would cover the commit-to-main case. -- 2.47.3 From 5db98dcab980816e4e150831447570d3a1ae9467 Mon Sep 17 00:00:00 2001 From: alexion Date: Fri, 24 Jul 2026 15:01:56 -0400 Subject: [PATCH 5/5] refactor: tighten benchmark-skill instructions per craft-skill audit Collapse duplication so each rule has one home: the pass-rule threshold and cost footnote defer to the core, and the case-contract detail defers to CASE-FORMAT.md. Drop editor-facing sediment from the arms step. Fixes surfaced by a verify walk-through: define the hard-assertion gate as passing only when every predicate exits zero (an earlier failure was maskable by a later success), state that in-session arms run at the session default temperature, name the fixture directory, and note the steps run from the repo root. --- skills/benchmark-skill/CASE-FORMAT.md | 10 +++++----- skills/benchmark-skill/SKILL.md | 24 ++++++++++-------------- 2 files changed, 15 insertions(+), 19 deletions(-) diff --git a/skills/benchmark-skill/CASE-FORMAT.md b/skills/benchmark-skill/CASE-FORMAT.md index 5cf4d8a..0194ab7 100644 --- a/skills/benchmark-skill/CASE-FORMAT.md +++ b/skills/benchmark-skill/CASE-FORMAT.md @@ -35,16 +35,16 @@ grep -qE '' "$OUTPUT" Do not mention the skill by name or hint that a skill exists, or the baseline arm stops being an honest counterfactual. - **`## Seed`** — optional, for conversation-driven skills whose input is a discussion rather than a file tree. It is a role-tagged transcript (`**User:**` / `**Assistant:**`) injected as the subagent's prior context before the prompt. -- **`## Hard assertions`** — optional, a `sh` code block of executable predicates forming a deterministic gate. - Each runs with `$OUTPUT` bound to the path of the arm's captured final message and `$WORLD` bound to the path of that arm's fresh fixture copy. - A non-zero exit fails the assertion, and a failed hard assertion fails the case outright. - The gate runs against the new-skill arm only. +- **`## Hard assertions`** — optional, a `sh` code block whose lines are each a separate predicate, together forming a deterministic gate. + Each predicate runs with `$OUTPUT` bound to the path of the arm's captured final message and `$WORLD` bound to the path of that arm's fresh fixture copy. + The gate passes only when **every** predicate exits zero — evaluate them so an earlier failure is never masked by a later success, not by a block's trailing exit code alone. + Any non-zero exit fails the gate, and a failed gate fails the case outright, against the new-skill arm only. - **`## Soft criteria`** — at least one natural-language statement the blind judge grounds its comparisons on. Describe what a good answer looks like, not which arm should win. ## Fixtures A case ships a hermetic, committed fixture so a run is reproducible and needs no live external state. -File-and-tree skills get a fixture directory beside `case.md`. +File-and-tree skills get a fixture directory named `fixture/` beside `case.md`, whose contents populate each arm's world root — so a `$WORLD/` predicate names a file directly under `fixture/`. Conversation-driven skills use `## Seed` instead. Every arm-and-trial combination runs against a **fresh copy** of the fixture, so writes never leak between runs. diff --git a/skills/benchmark-skill/SKILL.md b/skills/benchmark-skill/SKILL.md index ed5239c..a5216f6 100644 --- a/skills/benchmark-skill/SKILL.md +++ b/skills/benchmark-skill/SKILL.md @@ -19,6 +19,7 @@ The authoring contract for a test case is [`CASE-FORMAT.md`](CASE-FORMAT.md). Read it before you read the target skill's tests. Report artifacts live under a git-ignored `tests/.reports/` directory at the repo root, flat and keyed by skill name. +Run from the repo root: every path in the steps below is relative to it. ## 1. Resolve the target and validate its tests tree @@ -29,7 +30,7 @@ If the skill exists but has no tests directory or no case directories, report th Validate the shape of the tests tree at run time: -- Each case directory has a `case.md` that parses per `CASE-FORMAT.md` — a `description`, a `## Prompt`, at least one `## Soft criteria` entry, and optional `## Seed` and `## Hard assertions` blocks. +- Each case directory has a `case.md` that parses per `CASE-FORMAT.md`. - Any fixture a case references exists. - The test directory maps to a real skill. @@ -40,12 +41,10 @@ Done when every case parses, its fixture exists, and you have the case list in s ## 2. Fix the arms and per-run settings -This slice is **two-arm efficacy only**: a new-skill arm and a no-skill baseline arm. -(The previous-version regression arm is a later slice. -The core already handles extra arms, so do not remove the two-arm shape.) +Run **two arms**: a new-skill arm and a no-skill baseline arm. -Choose the trial temperature: a realistic temperature (around 1.0) by default, so the result reflects whether the skill *reliably* helps across variance. -A skill may pin a temperature-zero override when it wraps a genuinely mechanical task — honor that override if the skill declares one. +Arms run at the session's realistic default temperature (around 1.0), so the result reflects whether the skill *reliably* helps across variance rather than helping once by luck. +Per-arm temperature is not settable through the in-session workflow surface, so a skill's temperature-zero override for a genuinely mechanical task is a documented knob for the future headless path, not something to set here. Each case runs **5 paired trials**. Trial *i*'s new-skill output is judged against trial *i*'s no-skill output. @@ -82,11 +81,8 @@ Have it return a single winner: A, B, or tie. ## 4. Run the hard-assertion gate -Run the case's `## Hard assertions` against the **new arm only**, once per trial. -Bind `$OUTPUT` to the path of that trial's new-arm final message and `$WORLD` to that trial's new-arm world (its fixture copy, which is the arm's working directory). -Run each predicate. -A non-zero exit fails the assertion. -Record pass/fail per trial — a failed hard assertion fails the case outright regardless of the head-to-head. +Run the case's `## Hard assertions` against the **new arm only**, once per trial, with `$OUTPUT` and `$WORLD` bound as `CASE-FORMAT.md` defines — that trial's new-arm final message and its world. +Record each trial's pass or fail into the bundle for the core to score. A case with no `## Hard assertions` block simply has no gate. ## 5. Collect the run data @@ -146,9 +142,9 @@ python3 skills/benchmark-skill/core/benchmark_core.py \ --html tests/.reports/.html ``` -The core collapses each efficacy trial to WIN, TIE, or LOSS for the new arm (a tie is a non-win), applies the pass rule — **efficacy passes for a case when wins ≥ 3 and losses ≤ 1** — flags any loss for human review, and yields a skill-level **Efficacy verdict** that is green only when every case passes. -It renders a self-contained HTML report: the Efficacy badge and run metadata, a two-row per-arm cost table (footnoted that absolute cost is inflated by shared-context cache overhead, so the trustworthy signal is the new-vs-no-skill ratio), and the cases in stable authored order. -Cost is reported alongside quality but never gates the verdict. +The core collapses each trial to a per-arm WIN, TIE, or LOSS, applies the efficacy pass rule, flags any loss for human review, and yields the skill-level **Efficacy verdict**. +It renders the self-contained HTML report — the verdict badge, run metadata, the per-arm cost table, and the cases in stable authored order — alongside the machine-readable model. +Cost is reported but never gates the verdict. Clean up the scratch when done: remove the `tests/.reports/.work/` directory and every per-arm world and skill materialization you created under the system temp path. -- 2.47.3