Collapse duplication so each rule has one home: the pass-rule threshold
and cost footnote defer to the core, and the case-contract detail defers
to CASE-FORMAT.md. Drop editor-facing sediment from the arms step.
Fixes surfaced by a verify walk-through: define the hard-assertion gate
as passing only when every predicate exits zero (an earlier failure was
maskable by a later success), state that in-session arms run at the
session default temperature, name the fixture directory, and note the
steps run from the repo root.
Bring the feature's design record into git (it was authored across the
benchmark tasks but never committed). Cleaned to the repo's prose
standards on the way in: recast prose semicolons as separate sentences
and split multi-sentence lines to one sentence per line.
Run each benchmark arm in a hermetic fixture-only world materialized
outside the repo, so the no-skill baseline can no longer discover the
skill's assets on disk. The new-skill arm's skill is materialized to an
isolated temp path it is pointed at; the baseline gets no skill and is
told to stay within its world.
A live 5-trial axi-review run confirms the fix: all baseline transcripts
are clean of AXI-PRINCIPLES.md, the baselines invent their own generic
rubric (the honest counterfactual), and the Efficacy verdict is green
5/5 where the contaminated 0004 run had scored red 2/5.
Runner prose only; the deterministic core is untouched.
nix develop places benchmark-skill into ./.claude/skills/ (so /benchmark-skill
is available in this repo) and puts python3 on PATH to run the deterministic
core without a one-off nix shell.
Introduce the thinnest complete path that benchmarks a skill's efficacy
against a no-skill baseline and renders an HTML report.
- tests/ convention: a top-level tree mirroring skills/ by full path, with
case directories holding a case.md (CASE-FORMAT.md) and an optional fixture.
- benchmark-skill: a distributed, non-model-invocable runner (/benchmark-skill
<name>) that orchestrates force-invoked new-skill and skill-absent baseline
arms plus a blind per-trial judge as in-session subagents.
- Deterministic core (Python): a pure transform over the collected run data
that sums per-arm usage, collapses each trial to WIN/TIE/LOSS, applies the
efficacy pass rule (wins >= 3, losses <= 1), and renders a self-contained
report. Parameterized over arms and comparisons; two-arm here.
- Fixture-driven no-LLM check (checks/benchmark-core.nix) wired into flake.nix.
- Real tests tree for axi-review; a live run produced its efficacy report.
