Add the longitudinal layer to the benchmark core: a per-skill JSON-lines
history that appends one summary line per run and trims oldest-first at
fifty, two stacked net-margin trend ribbons (Efficacy and Regression, the
latter leaving a gap for two-arm runs) with the current run ringed and a
net/delta/green-count readout, and per-badge fragility chips that flag the
narrowest passing case on each green axis so a barely-green skill cannot
look robust.
History persistence is opt-in via --history and is the core's only side
effect. Without it the core stays a pure transform. Two committed fixtures
(clean and fragile) and new fixture-test sections cover the append-and-trim,
both ribbons including the two-arm gap, and the chips.
Turn a benchmark run into a three-arm experiment: alongside the new-skill
and no-skill arms, add a previous-version arm materialized from the
default branch's HEAD, drawn automatically whenever the skill's directory
differs from HEAD and degrading to the two-arm efficacy-only shape
otherwise. Two blind head-to-heads now fall out per trial — efficacy
(new-vs-no-skill) and regression (new-vs-old).
The deterministic core dispatches a pass rule per comparison (efficacy at
wins>=3 and losses<=1, regression at losses<=1 with no wins floor),
gates only Efficacy on the hard assertion, and yields a second
skill-level Regression verdict, green when no case regressed and
not-applicable on a two-arm run. The report gains a second badge, a
headline reading both verdicts together, a previous-version cost row with
a two-ratio footnote, and per-case side-by-side comparisons that
auto-expand on any failure or flagged loss to show the losing-trial
output pair.
The fixture test covers both shapes in one test: a three-arm run
asserting per-arm metrics, both per-case and skill-level verdicts, the
net margins, the three-row cost table and losing-trial evidence, and the
two-arm run retained as the degenerate no-previous-version case.
Introduce the thinnest complete path that benchmarks a skill's efficacy
against a no-skill baseline and renders an HTML report.
- tests/ convention: a top-level tree mirroring skills/ by full path, with
case directories holding a case.md (CASE-FORMAT.md) and an optional fixture.
- benchmark-skill: a distributed, non-model-invocable runner (/benchmark-skill
<name>) that orchestrates force-invoked new-skill and skill-absent baseline
arms plus a blind per-trial judge as in-session subagents.
- Deterministic core (Python): a pure transform over the collected run data
that sums per-arm usage, collapses each trial to WIN/TIE/LOSS, applies the
efficacy pass rule (wins >= 3, losses <= 1), and renders a self-contained
report. Parameterized over arms and comparisons; two-arm here.
- Fixture-driven no-LLM check (checks/benchmark-core.nix) wired into flake.nix.
- Real tests tree for axi-review; a live run produced its efficacy report.
Add the content tier: a standalone flake that auto-discovers each skill
(a directory containing a SKILL.md, at any depth) and exposes it as an
individually addressable derivation built by lib.mkSkill. Directories
without a SKILL.md are descended through as cosmetic containers; once a
SKILL.md is found, that directory's subfolders are its assets, not
further skills. Skill names must be globally unique across the tree — a
collision is a hard eval-time error, not a warning.
A fixture-driven skill-build check under `nix flake check` exercises the
recursive walk, the builder, the SKILL.md-at-$out-root contract, the
eval-time name passthru, and the collision error. The repo ships no real
skill content yet, so packages.<system> is empty today.
