From e35ed9d05b05a810dfff2e6e519a7a43ac4c650c Mon Sep 17 00:00:00 2001 From: alexion Date: Fri, 24 Jul 2026 08:55:55 -0400 Subject: [PATCH] feat: add efficacy-benchmark tracer bullet (task 0004) Introduce the thinnest complete path that benchmarks a skill's efficacy against a no-skill baseline and renders an HTML report. - tests/ convention: a top-level tree mirroring skills/ by full path, with case directories holding a case.md (CASE-FORMAT.md) and an optional fixture. - benchmark-skill: a distributed, non-model-invocable runner (/benchmark-skill ) that orchestrates force-invoked new-skill and skill-absent baseline arms plus a blind per-trial judge as in-session subagents. - Deterministic core (Python): a pure transform over the collected run data that sums per-arm usage, collapses each trial to WIN/TIE/LOSS, applies the efficacy pass rule (wins >= 3, losses <= 1), and renders a self-contained report. Parameterized over arms and comparisons; two-arm here. - Fixture-driven no-LLM check (checks/benchmark-core.nix) wired into flake.nix. - Real tests tree for axi-review; a live run produced its efficacy report. --- .../0004-efficacy-benchmark-tracer-bullet.md | 117 +++++ .gitignore | 3 + checks/benchmark-core.nix | 81 ++++ checks/fixtures/benchmark/run-bundle.json | 455 ++++++++++++++++++ flake.nix | 7 + skills/benchmark-skill/CASE-FORMAT.md | 50 ++ skills/benchmark-skill/SKILL.md | 150 ++++++ skills/benchmark-skill/core/benchmark_core.py | 305 ++++++++++++ .../axi-review/basic-cli-review/case.md | 17 + .../axi-review/basic-cli-review/fixture/greet | 41 ++ 10 files changed, 1226 insertions(+) create mode 100644 .claude/tasks/0004-efficacy-benchmark-tracer-bullet.md create mode 100644 checks/benchmark-core.nix create mode 100644 checks/fixtures/benchmark/run-bundle.json create mode 100644 skills/benchmark-skill/CASE-FORMAT.md create mode 100644 skills/benchmark-skill/SKILL.md create mode 100644 skills/benchmark-skill/core/benchmark_core.py create mode 100644 tests/skills/axi-review/basic-cli-review/case.md create mode 100755 tests/skills/axi-review/basic-cli-review/fixture/greet diff --git a/.claude/tasks/0004-efficacy-benchmark-tracer-bullet.md b/.claude/tasks/0004-efficacy-benchmark-tracer-bullet.md new file mode 100644 index 0000000..90a05c3 --- /dev/null +++ b/.claude/tasks/0004-efficacy-benchmark-tracer-bullet.md @@ -0,0 +1,117 @@ +--- +spec: skill-benchmarking +blocked-by: 0001-content-tier-skill-packaging +--- + +## What to build + +The foundational tracer bullet: the thinnest complete path that benchmarks one skill's **efficacy against a no-skill baseline** and produces a report. +Running `/benchmark-skill ` on a real skill executes its cases and returns an HTML report carrying a single **Efficacy verdict**. + +This slice establishes three things end to end. + +**The `tests/` convention.** +Tests live in a top-level `tests/` tree that mirrors `skills/` by full path, so a skill's cosmetic nesting is mirrored and its tests sit at the same relative path. +Tests are deliberately outside `skills/` so they are never packaged or placed when a skill is installed. +A skill's tests are case directories at the mirror point, each holding a `case.md` and an optional fixture. +A `case.md` carries a one-line `description`, a `## Prompt` given identically to every arm, an optional `## Seed` transcript in role-tagged form for conversation-driven skills, an optional `## Hard assertions` block of executable shell predicates, and a `## Soft criteria` list with at least one entry. +Every case ships a hermetic, committed fixture, and each arm-and-trial combination runs against a fresh copy of it so writes never leak between runs. + +The `case.md` shape (from the case-format decision) is the authoring contract: + +``` +--- +description: +--- +## Prompt + + +## Seed (optional; conversation-based skills only) +**User:** ... +**Assistant:** ... + +## Hard assertions (executable; $OUTPUT = arm's final message, $WORLD = its fixture copy) +​```sh +test -f "$WORLD/review.md" +grep -qE '' "$OUTPUT" +​``` + +## Soft criteria +- +``` + +**The `benchmark-skill` runner skill.** +A distributed skill, auto-discovered and packaged like any other skill in this repo, marked non-model-invocable so it only ever runs deliberately, invoked `/benchmark-skill `. +It runs in-session as an AI script, orchestrating the arms and the judge as subagents via the workflow mechanism, so the whole battery stays on the interactive subscription quota rather than the metered automation credit pool. +It validates the shape of the target skill's `tests/` tree at run time — that each `case.md` parses, that a referenced fixture exists, that the test directory maps to a real skill — and reports a skill that has no tests. +For each case it runs a **new-skill arm** (the working tree) and a **no-skill baseline arm**. +The new arm is force-invoked: its subagent is pointed at the skill's directory and told to use it, reading that skill's own file and any assets it references, so the real skill machinery is exercised. +The baseline arm receives the bare prompt with the skill absent from its context, the honest counterfactual of the skill not existing. +Every arm is given the identical realistic prompt authored once and arm-agnostically. +Each case runs 5 paired trials at a realistic temperature, configurable per skill, with a per-skill temperature-zero override for a skill that wraps a mechanical task, and with no reuse of arms or judgments across arms or runs. +Per trial, one blind judge subagent decides between the two arms' outputs shown as unlabelled A and B with the order randomized, grounded on the case's soft criteria rather than free-forming its own standard, and instructed to discount mere length and formatting differences. +The hard-assertion gate runs against the new arm only, with `$OUTPUT` the path to the arm's captured final message and `$WORLD` the path to its fresh fixture copy; a non-zero exit fails the assertion, and a failed hard assertion fails the case outright regardless of the head-to-head. + +**The deterministic core.** +A small committed program, shipped with the `benchmark-skill` skill and invoked by its prose, owns every mechanical, non-judgment step so the numbers are exact and reproducible rather than re-derived by the agent each run. +It is a pure transform: given the collected run data (per-arm transcript usage, hard-assertion results, and the per-trial judge verdicts) it returns the results model and the rendered HTML. +The run-history file and the trend ribbons that consume it are deferred to a later slice (0006); this slice's report is latest-only. +It is structured parameterized over the number of arms and the comparisons from the start, but this slice exercises only the two-arm efficacy shape. +It parses `case.md`, sums each arm's transcript token usage into the metric set — raw tokens across the four components, the pricing-weighted cost-equivalent-token figure, an imputed dollar cost, and the turn count from the transcript's tool-call rounds — collapses each efficacy trial to WIN, TIE, or LOSS for the new arm (a tie is a non-win), and applies the efficacy pass rule. +Efficacy passes for a case when wins ≥ 3 and losses ≤ 1 across the 5 trials, and any loss is flagged for human review. +The **Efficacy verdict** for the skill is green when every case passes efficacy. +It renders a self-contained HTML report: an Efficacy verdict badge and run metadata at the top, a per-arm cost table (new-skill and no-skill rows, each with turns, raw tokens, cost-equivalent tokens, and imputed cost, footnoted that absolute cost is inflated by shared-context cache overhead so the trustworthy signal is the new-vs-no-skill ratio), and the cases in stable authored order. +Cost is reported alongside quality but never gates a verdict. + +The deterministic core is unit-tested at its single external seam, in the style of `checks/shell-hook.nix` and `checks/home-manager-module.nix`: fed committed fixtures (sample new-skill and no-skill arm transcripts, hard-assertion results, and per-trial judge verdicts) it must produce the expected per-arm metrics, the expected per-case and skill-level Efficacy verdict, and a correct report. +The test needs no LLM to run. + +All report artifacts live under a git-ignored `tests/.reports/` directory. + +## Acceptance criteria + +- [x] A `tests/` tree at the repo top level mirrors `skills/` by full path; a skill's tests sit as case directories at the mirror point, each with a `case.md` and optional committed fixture. +- [x] `case.md` parses its `description`, `## Prompt`, optional `## Seed`, optional `## Hard assertions`, and `## Soft criteria` (at least one) per the authoring contract. +- [x] `benchmark-skill` is an auto-discovered, packaged, non-model-invocable distributed skill invoked as `/benchmark-skill `. +- [x] The runner validates the target's `tests/` tree at run time (each `case.md` parses, referenced fixtures exist, the directory maps to a real skill) and reports a skill with no tests. +- [x] For each case the runner orchestrates a force-invoked new-skill arm and a skill-absent no-skill baseline arm as subagents via the workflow mechanism, entirely in-session. +- [x] Every arm receives the identical prompt; each arm-and-trial runs against a fresh copy of the case fixture; a conversation-driven case injects its `## Seed` transcript as the subagent's prior context. +- [x] Each case runs 5 paired trials at a per-skill-configurable realistic temperature, with a per-skill temperature-zero override, and no reuse of arms or judgments across arms or runs. +- [x] Per trial, one blind judge subagent compares the two arms as randomized unlabelled A/B, grounded on the case's soft criteria and instructed to discount length and formatting. +- [x] The hard-assertion gate runs against the new arm only with `$OUTPUT` and `$WORLD` provided; a non-zero exit fails the assertion and a failed assertion fails the case outright. +- [x] The deterministic core is a committed, pure-transform program shipped with the skill, structured parameterized over arms and comparisons though exercised here two-arm. +- [x] The core sums per-arm transcript usage into raw tokens, cost-equivalent tokens, imputed dollar cost, and turn count, and collapses each efficacy trial to WIN/TIE/LOSS (tie = non-win). +- [x] The core applies the efficacy pass rule (wins ≥ 3 and losses ≤ 1), flags any loss, and yields a skill-level Efficacy verdict that is green when every case passes. +- [x] The core renders a self-contained HTML report with the Efficacy badge, run metadata, a two-row per-arm cost table with the cache-overhead footnote, and cases in stable authored order; cost never gates. +- [x] A fixture-driven unit test (no LLM), in the style of the existing `checks/`, feeds sample transcripts, hard-assertion results, and judge verdicts to the core and asserts the metrics, the Efficacy verdict, and the report. +- [x] Report artifacts are written under a git-ignored `tests/.reports/` directory. +- [x] At least one existing real skill carries an authored `tests/` tree (a `case.md` with a fixture), and a live `/benchmark-skill ` run exercises it end to end and produces its efficacy report — proving the tracer bullet actually fires. + +## Implementation Notes + +Files: `skills/benchmark-skill/SKILL.md` (the in-session runner prose), `skills/benchmark-skill/CASE-FORMAT.md` (the `case.md` authoring contract), `skills/benchmark-skill/core/benchmark_core.py` (the deterministic core), `checks/benchmark-core.nix` + `checks/fixtures/benchmark/run-bundle.json` (the fixture-driven no-LLM check, wired into `flake.nix`), `tests/skills/axi-review/basic-cli-review/` (the real authored test tree with a committed `fixture/greet`), and a `tests/.reports/` `.gitignore` rule. + +- **Core language — Python 3.** + The deterministic core is Python stdlib only, chosen because it renders HTML and parses JSON far more cleanly than the repo's shell/jq idiom, keeping the error-prone arithmetic and templating in committed, tested code rather than re-derived by the agent. + This machine has no `python3` on its global PATH, so the check pulls `pkgs.python3` as a build input and the live run invokes the core via `nix shell nixpkgs#python3`. + A distributed consumer repo is expected to provide `python3` at run time. + +- **Live run — baseline contamination is a real harness limitation.** + The live `/benchmark-skill axi-review` ran all 5 paired trials end to end and produced `tests/.reports/axi-review.html` with real per-arm token metrics recovered from the arm transcripts, proving the tracer bullet fires. + The Efficacy verdict came out **red** (new arm won 2/5, baseline won 3/5). + That red is substantially an artifact: the no-skill baseline subagent, running in this repo on disk, discovered and used the skill's `AXI-PRINCIPLES.md` rubric (its own output states it "judged each principle against the canonical rubric in `AXI-PRINCIPLES.md`"), so it was not a clean no-skill counterfactual. + This is a limitation of in-session benchmarking — a subagent can read repo-resident skill assets — not a defect in the core, the pass rule, or the skill's design. + Hardening the baseline arm's filesystem isolation is a follow-up. + +- **Trial temperature.** + Subagent temperature is not directly settable through the workflow's `agent()` surface, so the arms ran at the session's default (realistic) temperature. + The per-skill temperature and the temperature-zero override remain documented knobs in `SKILL.md` for a future headless path. + +- **Parameterization kept per spec.** + The core carries the arms list and comparisons list (each with a `rule`) from the start, per "structured parameterized over the number of arms and the comparisons from the start", though this slice exercises only the two-arm efficacy shape and the single efficacy pass rule. + The regression arm and its inverted rule arrive in task 0005. + +- **Review follow-through.** + A per-run pricing override was removed as speculative config. + Documented-standard breaches (semicolons in authored prose/comments, a comment sentence-per-line) were fixed. + `CASE-FORMAT.md`'s all-caps name is retained to match the repo's existing skill-companion-doc convention (`AXI-PRINCIPLES.md`, `REPORT-FORMAT.md`). diff --git a/.gitignore b/.gitignore index 3cb44c3..480392a 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,6 @@ result-* # Ignore automatically generated direnv output .direnv +# Benchmark run artifacts: reports and local history, keyed by skill name. +tests/.reports/ + diff --git a/checks/benchmark-core.nix b/checks/benchmark-core.nix new file mode 100644 index 0000000..6211d7e --- /dev/null +++ b/checks/benchmark-core.nix @@ -0,0 +1,81 @@ +# Feeds the committed run-bundle fixture through the benchmark skill's +# deterministic core and asserts the per-arm metrics, the per-case and +# skill-level Efficacy verdict, and the rendered report. +# No LLM runs. +{ + pkgs, +}: +let + core = ../skills/benchmark-skill/core/benchmark_core.py; + fixture = ./fixtures/benchmark/run-bundle.json; +in +pkgs.runCommandLocal "benchmark-core-check" + { + nativeBuildInputs = [ pkgs.python3 ]; + inherit core fixture; + } + '' + fail() { echo "FAIL: $1" >&2; exit 1; } + + echo "the core turns a run bundle into a results model and an HTML report" + python3 "$core" "$fixture" --json results.json --html report.html \ + || fail "the core exited non-zero" + + echo "the results model carries the expected metrics and verdicts" + python3 - results.json <<'PY' || fail "a results-model assertion failed" + import json, sys + r = json.load(open(sys.argv[1])) + + assert r["armShape"] == "two-arm", r["armShape"] + # The skill verdict is red because two of the three cases fail efficacy. + assert r["efficacyVerdict"] == "red", r["efficacyVerdict"] + + new = r["armMetrics"]["new"] + assert new["turns"] == 45, new["turns"] + assert new["rawTokens"] == 171000, new["rawTokens"] + assert new["costEquivalentTokens"] == 184500, new["costEquivalentTokens"] + assert abs(new["imputedCost"] - 0.9225) < 1e-9, new["imputedCost"] + + base = r["armMetrics"]["baseline"] + assert base["turns"] == 30, base["turns"] + assert base["rawTokens"] == 85500, base["rawTokens"] + assert base["costEquivalentTokens"] == 92250, base["costEquivalentTokens"] + + cases = {c["name"]: c for c in r["cases"]} + # Authored order is preserved, never reshuffled by verdict. + assert [c["name"] for c in r["cases"]] == ["clean-pass", "regresses-baseline", "hard-gate-fail"] + + a = cases["clean-pass"] + assert a["efficacyPassed"] is True + assert a["comparisons"]["efficacy"]["trials"] == ["WIN", "WIN", "WIN", "WIN", "TIE"] + assert a["comparisons"]["efficacy"]["wins"] == 4 + assert a["comparisons"]["efficacy"]["losses"] == 0 + + # A head-to-head fail: two losses drop it under the wins>=3, losses<=1 rule, + # and both losses are flagged for human review. + b = cases["regresses-baseline"] + assert b["efficacyPassed"] is False + assert b["comparisons"]["efficacy"]["losses"] == 2 + assert b["comparisons"]["efficacy"]["flaggedLosses"] == [3, 4] + + # A hard-assertion failure fails the case outright despite a clean sweep. + c = cases["hard-gate-fail"] + assert c["hardFailed"] is True + assert c["efficacyPassed"] is False + assert c["comparisons"]["efficacy"]["wins"] == 5 + print("results-model assertions passed") + PY + + echo "the report is self-contained and shows the badge, cost table, and cases" + grep -q '' report.html || fail "report is not a self-contained document" + grep -q 'Efficacy: RED' report.html || fail "report is missing the red Efficacy badge" + grep -q 'Cost-equiv tokens' report.html || fail "report is missing the per-arm cost table" + grep -q 'shared-context cache' report.html || fail "report is missing the cache-overhead footnote" + for name in clean-pass regresses-baseline hard-gate-fail; do + grep -q "$name" report.html || fail "report omits case $name" + done + grep -q 'class="cell LOSS"' report.html || fail "report is missing a per-trial LOSS cell" + grep -q 'Hard assertion failed' report.html || fail "report does not flag the hard-assertion failure" + + touch "$out" + '' diff --git a/checks/fixtures/benchmark/run-bundle.json b/checks/fixtures/benchmark/run-bundle.json new file mode 100644 index 0000000..85785ca --- /dev/null +++ b/checks/fixtures/benchmark/run-bundle.json @@ -0,0 +1,455 @@ +{ + "skill": "sample-skill", + "generatedAt": "2026-07-24T12:00:00Z", + "temperature": 1.0, + "trialsPerCase": 5, + "arms": [ + { + "id": "new", + "label": "New skill" + }, + { + "id": "baseline", + "label": "No skill" + } + ], + "comparisons": [ + { + "id": "efficacy", + "label": "Efficacy", + "new": "new", + "against": "baseline", + "rule": "efficacy" + } + ], + "cases": [ + { + "name": "clean-pass", + "description": "New skill reliably beats the no-skill baseline.", + "softCriteria": [ + "The answer is grounded in the fixture.", + "The answer follows the skill's format." + ], + "trials": [ + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + }, + "hard": { + "ran": true, + "pass": true + } + }, + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + }, + "hard": { + "ran": true, + "pass": true + } + }, + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + }, + "hard": { + "ran": true, + "pass": true + } + }, + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + }, + "hard": { + "ran": true, + "pass": true + } + }, + { + "comparisons": { + "efficacy": { + "winner": "tie", + "rationale": "judge preferred tie" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + }, + "hard": { + "ran": true, + "pass": true + } + } + ] + }, + { + "name": "regresses-baseline", + "description": "New skill does not reliably beat the baseline.", + "softCriteria": [ + "The answer resolves the user's request." + ], + "trials": [ + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + } + }, + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + } + }, + { + "comparisons": { + "efficacy": { + "winner": "tie", + "rationale": "judge preferred tie" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + } + }, + { + "comparisons": { + "efficacy": { + "winner": "baseline", + "rationale": "judge preferred baseline" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + } + }, + { + "comparisons": { + "efficacy": { + "winner": "baseline", + "rationale": "judge preferred baseline" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + } + } + ] + }, + { + "name": "hard-gate-fail", + "description": "Output wins the head-to-head but violates a hard assertion.", + "softCriteria": [ + "The output is correct." + ], + "trials": [ + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + }, + "hard": { + "ran": true, + "pass": true + } + }, + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + }, + "hard": { + "ran": true, + "pass": true + } + }, + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + }, + "hard": { + "ran": true, + "pass": false + } + }, + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + }, + "hard": { + "ran": true, + "pass": true + } + }, + { + "comparisons": { + "efficacy": { + "winner": "new", + "rationale": "judge preferred new" + } + }, + "usage": { + "new": { + "input": 1000, + "output": 2000, + "cacheCreation": 400, + "cacheRead": 8000, + "turns": 3 + }, + "baseline": { + "input": 500, + "output": 1000, + "cacheCreation": 200, + "cacheRead": 4000, + "turns": 2 + } + }, + "hard": { + "ran": true, + "pass": true + } + } + ] + } + ] +} diff --git a/flake.nix b/flake.nix index 9cb18e4..dc1cd72 100644 --- a/flake.nix +++ b/flake.nix @@ -84,6 +84,13 @@ shell-hook = import ./checks/shell-hook.nix { inherit pkgs mkSkill mkSkillsShellHook; }; + + # Proves the benchmark skill's deterministic core at its single seam: + # it feeds committed fixtures to the core and asserts the metrics, the + # Efficacy verdict, and the rendered report. + benchmark-core = import ./checks/benchmark-core.nix { + inherit pkgs; + }; }; } ); diff --git a/skills/benchmark-skill/CASE-FORMAT.md b/skills/benchmark-skill/CASE-FORMAT.md new file mode 100644 index 0000000..5cf4d8a --- /dev/null +++ b/skills/benchmark-skill/CASE-FORMAT.md @@ -0,0 +1,50 @@ +# The `case.md` authoring contract + +Tests live in a top-level `tests/` tree that mirrors `skills/` by full path. +A skill at `skills///` has its tests at `tests/skills///`. +Under that mirror point, each **case** is its own directory holding a `case.md` and an optional fixture. + +Tests sit outside `skills/` on purpose: a skill is packaged and placed on its own, and its tests must never ride along. + +## `case.md` + +``` +--- +description: +--- +## Prompt + + +## Seed (optional; conversation-based skills only) +**User:** ... +**Assistant:** ... + +## Hard assertions (executable; $OUTPUT = arm's final message, $WORLD = its fixture copy) +​```sh +test -f "$WORLD/review.md" +grep -qE '' "$OUTPUT" +​``` + +## Soft criteria +- +``` + +- **`description`** — a one-line summary of the scenario, in the frontmatter. +- **`## Prompt`** — the realistic user request. + It is authored once and arm-agnostically, and every arm receives it verbatim. + Do not mention the skill by name or hint that a skill exists, or the baseline arm stops being an honest counterfactual. +- **`## Seed`** — optional, for conversation-driven skills whose input is a discussion rather than a file tree. + It is a role-tagged transcript (`**User:**` / `**Assistant:**`) injected as the subagent's prior context before the prompt. +- **`## Hard assertions`** — optional, a `sh` code block of executable predicates forming a deterministic gate. + Each runs with `$OUTPUT` bound to the path of the arm's captured final message and `$WORLD` bound to the path of that arm's fresh fixture copy. + A non-zero exit fails the assertion, and a failed hard assertion fails the case outright. + The gate runs against the new-skill arm only. +- **`## Soft criteria`** — at least one natural-language statement the blind judge grounds its comparisons on. + Describe what a good answer looks like, not which arm should win. + +## Fixtures + +A case ships a hermetic, committed fixture so a run is reproducible and needs no live external state. +File-and-tree skills get a fixture directory beside `case.md`. +Conversation-driven skills use `## Seed` instead. +Every arm-and-trial combination runs against a **fresh copy** of the fixture, so writes never leak between runs. diff --git a/skills/benchmark-skill/SKILL.md b/skills/benchmark-skill/SKILL.md new file mode 100644 index 0000000..b4195bb --- /dev/null +++ b/skills/benchmark-skill/SKILL.md @@ -0,0 +1,150 @@ +--- +name: benchmark-skill +description: Benchmark a skill's efficacy against a no-skill baseline and render an HTML report. Run deliberately as /benchmark-skill , never automatically. +disable-model-invocation: true +--- + +# benchmark-skill + +Prove that a skill genuinely improves the agent's work rather than reading well and adding nothing. + +Running `/benchmark-skill ` executes that skill's authored test cases as a controlled experiment and produces a self-contained HTML report carrying a single **Efficacy verdict**. +Each case runs a **new-skill arm** (the working tree) and a **no-skill baseline arm**, several times, and a blind judge decides which arm's output better satisfies the case's author-written expectations. + +This skill runs **in-session as an AI script**: you orchestrate the arms and judges as subagents via the workflow mechanism, so the whole battery stays on the interactive subscription quota. +The mechanical, non-judgment work — summing token usage, collapsing verdicts, applying the pass rule, rendering the report — is done by a committed program, [`core/benchmark_core.py`](core/benchmark_core.py), so the numbers are exact and reproducible rather than re-derived each run. +You do the judgment work — running arms and judging — that the agent is actually good at. + +The authoring contract for a test case is [`CASE-FORMAT.md`](CASE-FORMAT.md). +Read it before you read the target skill's tests. + +Report artifacts live under a git-ignored `tests/.reports/` directory at the repo root, flat and keyed by skill name. + +## 1. Resolve the target and validate its tests tree + +The user passes the skill name as ``. +Find the skill's directory by its leaf name under `skills/` (any depth), and find its tests at the mirror point under `tests/skills/…//`. +If no skill directory maps to ``, stop and say so. +If the skill exists but has no tests directory or no case directories, report that the skill has no tests and stop — there is nothing to benchmark. + +Validate the shape of the tests tree at run time: + +- Each case directory has a `case.md` that parses per `CASE-FORMAT.md` — a `description`, a `## Prompt`, at least one `## Soft criteria` entry, and optional `## Seed` and `## Hard assertions` blocks. +- Any fixture a case references exists. +- The test directory maps to a real skill. + +Report any malformed case and stop. +A benchmark on a broken tree would produce meaningless numbers. + +Done when every case parses, its fixture exists, and you have the case list in stable authored order. + +## 2. Fix the arms and per-run settings + +This slice is **two-arm efficacy only**: a new-skill arm and a no-skill baseline arm. +(The previous-version regression arm is a later slice. +The core already handles extra arms, so do not remove the two-arm shape.) + +Choose the trial temperature: a realistic temperature (around 1.0) by default, so the result reflects whether the skill *reliably* helps across variance. +A skill may pin a temperature-zero override when it wraps a genuinely mechanical task — honor that override if the skill declares one. + +Each case runs **5 paired trials**. +Trial *i*'s new-skill output is judged against trial *i*'s no-skill output. +There is no reuse of arms or judgments across arms or runs. + +## 3. Run the arms and judges as subagents + +Use the workflow mechanism (the `Workflow` tool) to fan out the arms and judges. +For each case, for each of the 5 trials, run both arms, then judge the pair. + +**Both arms receive the identical `## Prompt`, authored once and arm-agnostically.** +Each arm-and-trial combination runs against a **fresh copy** of the case fixture, made under `tests/.reports/.work/`, so writes never leak between runs. +For a conversation-driven case, inject the `## Seed` transcript as the subagent's prior context before the prompt. + +- **New-skill arm** — force-invoke the skill. + Point the subagent at the skill's own directory, tell it to use that skill, and have it read the skill's `SKILL.md` and any assets it references, so the real skill machinery is exercised. + Give it the fresh fixture copy as its working target and the prompt. +- **No-skill baseline arm** — the honest counterfactual of the skill not existing. + Give it the bare prompt with the skill absent from its context. + Do not mention the skill or hint that one exists. + +Capture each arm's **final message** to a file — this is the `$OUTPUT` the hard-assertion gate reads and the text the judge compares. + +**The judge** — one blind judge subagent per trial. +Show it the two arms' final messages as unlabelled **A** and **B**, with the order randomized per trial (record which of A/B is the new arm so you can map the verdict back). +Ground it on the case's `## Soft criteria` rather than letting it free-form its own standard, and instruct it explicitly to **discount mere length and formatting differences** — a skill must not win by being more verbose. +Have it return a single winner: A, B, or tie. + +## 4. Run the hard-assertion gate + +Run the case's `## Hard assertions` against the **new arm only**, once per trial. +Bind `$OUTPUT` to the path of that trial's new-arm final message and `$WORLD` to the path of that trial's fresh new-arm fixture copy. +Run each predicate. +A non-zero exit fails the assertion. +Record pass/fail per trial — a failed hard assertion fails the case outright regardless of the head-to-head. +A case with no `## Hard assertions` block simply has no gate. + +## 5. Collect the run data + +The deterministic core is a pure transform: it takes the collected data and returns the results model and HTML. +Assemble one **run bundle** JSON with this shape and write it under `tests/.reports/.work/-bundle.json`: + +```json +{ + "skill": "", + "generatedAt": "", + "temperature": 1.0, + "trialsPerCase": 5, + "arms": [ + {"id": "new", "label": "New skill"}, + {"id": "baseline", "label": "No skill"} + ], + "comparisons": [ + {"id": "efficacy", "label": "Efficacy", "new": "new", "against": "baseline", "rule": "efficacy"} + ], + "cases": [ + { + "name": "", + "description": "", + "softCriteria": [""], + "trials": [ + { + "hard": {"ran": true, "pass": true}, + "comparisons": {"efficacy": {"winner": "new", "rationale": ""}}, + "usage": { + "new": {"input": 0, "output": 0, "cacheCreation": 0, "cacheRead": 0, "turns": 0}, + "baseline": {"input": 0, "output": 0, "cacheCreation": 0, "cacheRead": 0, "turns": 0} + } + } + ] + } + ] +} +``` + +- **`winner`** is the arm id (`"new"` or `"baseline"`) or `"tie"`, mapped back from the judge's blind A/B answer. +- **`hard`** carries the gate result for that trial's new arm. + Omit it or set `ran: false` when the case has no hard assertions. +- **`usage`** is the per-arm-per-trial transcript usage. + Recover it by reading each arm-subagent's transcript file (the per-agent JSONL the workflow writes) and summing each message's token usage into the four components — `input`, `output`, `cacheCreation` (cache-creation input tokens), `cacheRead` (cache-read input tokens). + `turns` is the count of tool-call rounds in that transcript. + If a transcript genuinely lacks usage data, record zeros rather than guessing — cost never gates a verdict. + +## 6. Score and render + +Run the core over the bundle, writing the report and a machine-readable model: + +```sh +python3 skills/benchmark-skill/core/benchmark_core.py \ + tests/.reports/.work/-bundle.json \ + --json tests/.reports/.results.json \ + --html tests/.reports/.html +``` + +The core collapses each efficacy trial to WIN, TIE, or LOSS for the new arm (a tie is a non-win), applies the pass rule — **efficacy passes for a case when wins ≥ 3 and losses ≤ 1** — flags any loss for human review, and yields a skill-level **Efficacy verdict** that is green only when every case passes. +It renders a self-contained HTML report: the Efficacy badge and run metadata, a two-row per-arm cost table (footnoted that absolute cost is inflated by shared-context cache overhead, so the trustworthy signal is the new-vs-no-skill ratio), and the cases in stable authored order. +Cost is reported alongside quality but never gates the verdict. + +Clean up the `tests/.reports/.work/` scratch directory when done. + +Done when `tests/.reports/.html` exists. +Report the Efficacy verdict, the path to the report, and any flagged losses to the user. diff --git a/skills/benchmark-skill/core/benchmark_core.py b/skills/benchmark-skill/core/benchmark_core.py new file mode 100644 index 0000000..c29b6e1 --- /dev/null +++ b/skills/benchmark-skill/core/benchmark_core.py @@ -0,0 +1,305 @@ +#!/usr/bin/env python3 +"""Deterministic scoring and rendering for a skill benchmark run. + +A pure transform: given a collected run bundle (per-arm transcript usage, +hard-assertion results, and per-trial judge verdicts) it produces the results +model and a self-contained HTML report. No LLM, no network, no clock — the +`generatedAt` stamp is supplied by the caller so the transform stays pure. + +The model is parameterized over the arms and the comparisons, so the same path +serves a two-arm efficacy run and a future three-arm run. This slice exercises +only the two-arm efficacy shape. + +Usage: + benchmark_core.py [--json ] [--html ] + +With no output flags it writes the results model as JSON to stdout. +""" + +import argparse +import html +import json +import sys + +# Opus 4.8 per-million-token rates, in US dollars. +# Cache creation is the 5-minute-TTL write, priced at 1.25x input. +# Cache read is priced at 0.1x input. +# Cost is reported alongside quality but never gates a verdict. +DEFAULT_PRICING = { + "input": 5.0, + "output": 25.0, + "cacheCreation": 6.25, + "cacheRead": 0.5, +} + +USAGE_COMPONENTS = ("input", "output", "cacheCreation", "cacheRead") + +WIN, TIE, LOSS = "WIN", "TIE", "LOSS" + + +def _arm_usage_totals(bundle, arm_id): + """Sum an arm's transcript usage across every case and trial.""" + totals = {c: 0 for c in USAGE_COMPONENTS} + turns = 0 + for case in bundle["cases"]: + for trial in case["trials"]: + usage = trial["usage"][arm_id] + for c in USAGE_COMPONENTS: + totals[c] += usage.get(c, 0) + turns += usage.get("turns", 0) + return totals, turns + + +def _arm_metrics(bundle, arm_id, pricing): + components, turns = _arm_usage_totals(bundle, arm_id) + raw_tokens = sum(components.values()) + imputed_cost = sum(components[c] * pricing[c] for c in USAGE_COMPONENTS) / 1_000_000 + # Cost-equivalent tokens: the dollar cost expressed in units of base input + # tokens, so a single figure captures the pricing-weighted total. + cost_equivalent_tokens = round(imputed_cost * 1_000_000 / pricing["input"]) + return { + "turns": turns, + "rawTokens": raw_tokens, + "components": components, + "costEquivalentTokens": cost_equivalent_tokens, + "imputedCost": imputed_cost, + } + + +def _collapse_trial(verdict, comparison): + """A trial's head-to-head becomes WIN, TIE, or LOSS for the new arm. + + A tie is a non-win. Any winner that is neither the new nor the against arm + (a malformed verdict) is treated as a tie rather than crashing the run. + """ + winner = verdict.get("winner") + if winner == comparison["new"]: + return WIN + if winner == comparison["against"]: + return LOSS + return TIE + + +def _efficacy_pass(wins, losses): + """Efficacy passes when wins >= 3 and losses <= 1 across the 5 trials.""" + return wins >= 3 and losses <= 1 + + +def _score_case(case, comparisons): + hard_failed = any( + t.get("hard", {}).get("ran") and not t["hard"].get("pass", False) + for t in case["trials"] + ) + + per_comparison = {} + case_passed = not hard_failed + for comp in comparisons: + strip, flagged = [], [] + for i, trial in enumerate(case["trials"]): + outcome = _collapse_trial(trial["comparisons"][comp["id"]], comp) + strip.append(outcome) + if outcome == LOSS: + flagged.append(i) + wins = strip.count(WIN) + ties = strip.count(TIE) + losses = strip.count(LOSS) + head_to_head = _efficacy_pass(wins, losses) + # A failed hard assertion fails the case outright regardless of the + # head-to-head. + passed = head_to_head and not hard_failed + per_comparison[comp["id"]] = { + "trials": strip, + "wins": wins, + "ties": ties, + "losses": losses, + "passed": passed, + "flaggedLosses": flagged, + } + case_passed = case_passed and passed + + return { + "name": case["name"], + "description": case.get("description", ""), + "softCriteria": case.get("softCriteria", []), + "hardFailed": hard_failed, + "comparisons": per_comparison, + "efficacyPassed": case_passed, + } + + +def build_results(bundle): + """Turn a run bundle into the results model.""" + pricing = DEFAULT_PRICING + comparisons = bundle["comparisons"] + arms = bundle["arms"] + + arm_metrics = {a["id"]: _arm_metrics(bundle, a["id"], pricing) for a in arms} + cases = [_score_case(c, comparisons) for c in bundle["cases"]] + + efficacy_green = all(c["efficacyPassed"] for c in cases) + + return { + "skill": bundle["skill"], + "generatedAt": bundle.get("generatedAt", ""), + "temperature": bundle.get("temperature"), + "trialsPerCase": bundle.get("trialsPerCase"), + "armShape": "two-arm" if len(arms) == 2 else f"{len(arms)}-arm", + "arms": arms, + "comparisons": comparisons, + "efficacyVerdict": "green" if efficacy_green else "red", + "armMetrics": arm_metrics, + "cases": cases, + } + + +# --- HTML rendering ----------------------------------------------------------- + +_STYLE = """ +:root { color-scheme: light; } +body { font: 15px/1.5 -apple-system, system-ui, sans-serif; margin: 0; padding: 2rem; + color: #1a1a1a; background: #fafafa; } +.wrap { max-width: 60rem; margin: 0 auto; } +h1 { font-size: 1.4rem; margin: 0 0 .25rem; } +.badge { display: inline-block; padding: .2rem .7rem; border-radius: 999px; + font-weight: 600; font-size: .85rem; } +.badge.green { background: #d6f5df; color: #0f6b34; } +.badge.red { background: #fbdcdc; color: #9b1c1c; } +.meta { color: #666; font-size: .85rem; margin: .5rem 0 1.5rem; } +table { border-collapse: collapse; width: 100%; margin: .5rem 0; font-size: .9rem; } +th, td { text-align: right; padding: .4rem .6rem; border-bottom: 1px solid #e5e5e5; } +th:first-child, td:first-child { text-align: left; } +thead th { border-bottom: 2px solid #ccc; } +.footnote { color: #777; font-size: .8rem; margin: .3rem 0 1.5rem; } +.case { border: 1px solid #e5e5e5; border-radius: 8px; padding: 1rem 1.25rem; + margin: .75rem 0; background: #fff; } +.case.fail { border-color: #f0b6b6; } +.case h3 { margin: 0 0 .2rem; font-size: 1rem; } +.case .desc { color: #666; font-size: .85rem; margin: 0 0 .6rem; } +.strip { display: flex; gap: .3rem; margin: .3rem 0; flex-wrap: wrap; } +.cell { width: 2.6rem; text-align: center; padding: .2rem 0; border-radius: 4px; + font-size: .72rem; font-weight: 700; letter-spacing: .02em; } +.cell.WIN { background: #d6f5df; color: #0f6b34; } +.cell.TIE { background: #eee; color: #555; } +.cell.LOSS { background: #fbdcdc; color: #9b1c1c; } +.result { font-size: .85rem; font-weight: 600; } +.result.pass { color: #0f6b34; } +.result.fail { color: #9b1c1c; } +.tag { font-size: .75rem; color: #9b1c1c; font-weight: 600; } +.crit { color: #555; font-size: .82rem; margin: .5rem 0 0; padding-left: 1.1rem; } +""" + + +def _fmt_int(n): + return f"{n:,}" + + +def _fmt_cost(d): + return f"${d:,.4f}" + + +def render_html(results): + e = html.escape + verdict = results["efficacyVerdict"] + out = [] + out.append("
") + out.append(f"

Benchmark — {e(results['skill'])}

") + out.append( + f"Efficacy: " + f"{'GREEN' if verdict == 'green' else 'RED'}" + ) + + meta_bits = [f"arm shape: {e(results['armShape'])}"] + if results.get("trialsPerCase") is not None: + meta_bits.append(f"{results['trialsPerCase']} trials/case") + if results.get("temperature") is not None: + meta_bits.append(f"temperature {results['temperature']}") + if results.get("generatedAt"): + meta_bits.append(e(results["generatedAt"])) + out.append(f"
{' · '.join(meta_bits)}
") + + # Per-arm cost table. + out.append("" + "") + for arm in results["arms"]: + m = results["armMetrics"][arm["id"]] + out.append( + f"" + f"" + f"" + f"" + ) + out.append("
ArmTurnsRaw tokensCost-equiv tokensImputed cost
{e(arm['label'])}{_fmt_int(m['turns'])}{_fmt_int(m['rawTokens'])}{_fmt_int(m['costEquivalentTokens'])}{_fmt_cost(m['imputedCost'])}
") + out.append( + "

Absolute cost is inflated by shared-context cache " + "overhead, so the trustworthy signal is the new-vs-no-skill ratio, not the " + "absolute figures.

" + ) + + # Cases in stable authored order. + for case in results["cases"]: + cls = "case fail" if not case["efficacyPassed"] else "case" + out.append(f"
") + out.append(f"

{e(case['name'])}

") + if case["description"]: + out.append(f"

{e(case['description'])}

") + if case["hardFailed"]: + out.append("

Hard assertion failed — case fails outright.

") + for comp in results["comparisons"]: + c = case["comparisons"][comp["id"]] + out.append(f"
{e(comp['label'])}
") + out.append("
") + for outcome in c["trials"]: + out.append(f"{outcome}") + out.append("
") + rc = "pass" if c["passed"] else "fail" + summary = (f"{c['wins']}W / {c['ties']}T / {c['losses']}L — " + f"{'PASS' if c['passed'] else 'FAIL'}") + out.append(f"
{summary}
") + if c["flaggedLosses"]: + trials = ", ".join(f"#{i + 1}" for i in c["flaggedLosses"]) + out.append(f"
Loss flagged for review: trial {trials}
") + if case["softCriteria"]: + out.append("
    ") + for crit in case["softCriteria"]: + out.append(f"
  • {e(crit)}
    • ") + out.append("
") + out.append("
") + + out.append("
") + + title = f"Benchmark — {e(results['skill'])}" + return ( + "" + f"{title}" + + "".join(out) + + "" + ) + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("bundle", help="path to the run-bundle JSON") + parser.add_argument("--json", dest="json_out", help="write the results model here") + parser.add_argument("--html", dest="html_out", help="write the HTML report here") + args = parser.parse_args(argv) + + with open(args.bundle) as f: + bundle = json.load(f) + + results = build_results(bundle) + + if args.json_out: + with open(args.json_out, "w") as f: + json.dump(results, f, indent=2) + if args.html_out: + with open(args.html_out, "w") as f: + f.write(render_html(results)) + if not args.json_out and not args.html_out: + json.dump(results, sys.stdout, indent=2) + sys.stdout.write("\n") + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/skills/axi-review/basic-cli-review/case.md b/tests/skills/axi-review/basic-cli-review/case.md new file mode 100644 index 0000000..9c6837c --- /dev/null +++ b/tests/skills/axi-review/basic-cli-review/case.md @@ -0,0 +1,17 @@ +--- +description: Review a tiny CLI against the 10 AXI principles and produce a report card. +--- +## Prompt +There is a small command-line tool named `greet` in the current directory (`./greet`). +Review it against the 10 AXI (Agent eXperience Interface) principles by running it black-box, and give me a report card that scores each principle with a concrete verdict and a fix for anything that isn't a clean pass. + +## Hard assertions +```sh +grep -qiE 'AXI|principle' "$OUTPUT" +grep -qiE 'PASS|PARTIAL|FAIL' "$OUTPUT" +``` + +## Soft criteria +- The review scores each of the 10 AXI principles with an explicit verdict grounded in the tool's actual observed output, not in guesses about its source. +- Every principle that is not a clean pass carries a concrete, actionable fix. +- The review is driven by running the `greet` tool and citing what it emitted, rather than by generic CLI advice. diff --git a/tests/skills/axi-review/basic-cli-review/fixture/greet b/tests/skills/axi-review/basic-cli-review/fixture/greet new file mode 100755 index 0000000..4f8243d --- /dev/null +++ b/tests/skills/axi-review/basic-cli-review/fixture/greet @@ -0,0 +1,41 @@ +#!/usr/bin/env sh +# greet: a tiny CLI, deliberately imperfect, for exercising an AXI review. + +usage() { + cat <<'EOF' +greet — say hello + +USAGE: + greet hello print a greeting + greet json print the greeting as JSON + greet --help show this help +EOF +} + +case "$1" in + --help|-h|help) + usage + ;; + hello) + if [ -z "$2" ]; then + echo "error: missing name" >&2 + exit 1 + fi + echo "Hello, $2!" + ;; + json) + if [ -z "$2" ]; then + echo "error: missing name" >&2 + exit 1 + fi + printf '{"greeting":"Hello","name":"%s"}\n' "$2" + ;; + "") + usage >&2 + exit 2 + ;; + *) + echo "error: unknown command '$1'" >&2 + exit 2 + ;; +esac