Add a second pure transform to the benchmark core: a `--leaderboard` mode
that ranks per-skill results models into an index leaderboard, one row per
skill carrying both verdicts, links out to each per-skill report, and sorts
any red first (regressions above efficacy failures), then fragile-but-passing,
then clean green.
The fragile tier reuses the 0006 per-badge chips: the efficacy chip now
carries its win count so the leaderboard reads it against the pass floor
without recomputing margins, and fragility is scoped to the passing tier so a
red row never carries a chip. The runner's SKILL.md gains the no-argument
batch flow and the index invocation. The fixture test covers the tiered sort,
the not-applicable Regression cell, and the per-skill links.
Add the longitudinal layer to the benchmark core: a per-skill JSON-lines
history that appends one summary line per run and trims oldest-first at
fifty, two stacked net-margin trend ribbons (Efficacy and Regression, the
latter leaving a gap for two-arm runs) with the current run ringed and a
net/delta/green-count readout, and per-badge fragility chips that flag the
narrowest passing case on each green axis so a barely-green skill cannot
look robust.
History persistence is opt-in via --history and is the core's only side
effect. Without it the core stays a pure transform. Two committed fixtures
(clean and fragile) and new fixture-test sections cover the append-and-trim,
both ribbons including the two-arm gap, and the chips.
Turn a benchmark run into a three-arm experiment: alongside the new-skill
and no-skill arms, add a previous-version arm materialized from the
default branch's HEAD, drawn automatically whenever the skill's directory
differs from HEAD and degrading to the two-arm efficacy-only shape
otherwise. Two blind head-to-heads now fall out per trial — efficacy
(new-vs-no-skill) and regression (new-vs-old).
The deterministic core dispatches a pass rule per comparison (efficacy at
wins>=3 and losses<=1, regression at losses<=1 with no wins floor),
gates only Efficacy on the hard assertion, and yields a second
skill-level Regression verdict, green when no case regressed and
not-applicable on a two-arm run. The report gains a second badge, a
headline reading both verdicts together, a previous-version cost row with
a two-ratio footnote, and per-case side-by-side comparisons that
auto-expand on any failure or flagged loss to show the losing-trial
output pair.
The fixture test covers both shapes in one test: a three-arm run
asserting per-arm metrics, both per-case and skill-level verdicts, the
net margins, the three-row cost table and losing-trial evidence, and the
two-arm run retained as the degenerate no-previous-version case.
Collapse duplication so each rule has one home: the pass-rule threshold
and cost footnote defer to the core, and the case-contract detail defers
to CASE-FORMAT.md. Drop editor-facing sediment from the arms step.
Fixes surfaced by a verify walk-through: define the hard-assertion gate
as passing only when every predicate exits zero (an earlier failure was
maskable by a later success), state that in-session arms run at the
session default temperature, name the fixture directory, and note the
steps run from the repo root.
Run each benchmark arm in a hermetic fixture-only world materialized
outside the repo, so the no-skill baseline can no longer discover the
skill's assets on disk. The new-skill arm's skill is materialized to an
isolated temp path it is pointed at; the baseline gets no skill and is
told to stay within its world.
A live 5-trial axi-review run confirms the fix: all baseline transcripts
are clean of AXI-PRINCIPLES.md, the baselines invent their own generic
rubric (the honest counterfactual), and the Efficacy verdict is green
5/5 where the contaminated 0004 run had scored red 2/5.
Runner prose only; the deterministic core is untouched.
Introduce the thinnest complete path that benchmarks a skill's efficacy
against a no-skill baseline and renders an HTML report.
- tests/ convention: a top-level tree mirroring skills/ by full path, with
case directories holding a case.md (CASE-FORMAT.md) and an optional fixture.
- benchmark-skill: a distributed, non-model-invocable runner (/benchmark-skill
<name>) that orchestrates force-invoked new-skill and skill-absent baseline
arms plus a blind per-trial judge as in-session subagents.
- Deterministic core (Python): a pure transform over the collected run data
that sums per-arm usage, collapses each trial to WIN/TIE/LOSS, applies the
efficacy pass rule (wins >= 3, losses <= 1), and renders a self-contained
report. Parameterized over arms and comparisons; two-arm here.
- Fixture-driven no-LLM check (checks/benchmark-core.nix) wired into flake.nix.
- Real tests tree for axi-review; a live run produced its efficacy report.
