feat: add all-skills leaderboard (task 0007)

Add a second pure transform to the benchmark core: a `--leaderboard` mode
that ranks per-skill results models into an index leaderboard, one row per
skill carrying both verdicts, links out to each per-skill report, and sorts
any red first (regressions above efficacy failures), then fragile-but-passing,
then clean green.

The fragile tier reuses the 0006 per-badge chips: the efficacy chip now
carries its win count so the leaderboard reads it against the pass floor
without recomputing margins, and fragility is scoped to the passing tier so a
red row never carries a chip. The runner's SKILL.md gains the no-argument
batch flow and the index invocation. The fixture test covers the tiered sort,
the not-applicable Regression cell, and the per-skill links.
This commit was merged in pull request #7.
This commit is contained in:
2026-07-24 16:58:02 -04:00
parent f4df5d31c9
commit 97c0eeb055
4 changed files with 325 additions and 25 deletions

View File

@@ -0,0 +1,46 @@
---
spec: skill-benchmarking
blocked-by: 0006-trend-history-and-ribbons
---
## What to build
The batch view: `/benchmark-skill` with no argument benchmarks every skill and produces an index leaderboard, so an author knows at a glance which skills are green and which regressed.
Invoked with no argument, the runner benchmarks every skill that follows the `tests/` convention and renders an index leaderboard.
Each skill row carries both badges, Efficacy and Regression, and a skill with no previous version reads not-applicable in its Regression cell.
The sort promotes any red first, with regressions ordered above efficacy failures — a regression means "you just broke something that was working," the more urgent signal while iterating — then fragile-but-passing skills, then clean green.
The fragile-but-passing tier reuses the per-badge fragility signal introduced with the trend layer (0006) rather than recomputing it here.
The leaderboard links out to the separate per-skill report files produced by a single-skill run.
The per-skill report files and the index live flat under the `tests/.reports/` directory (its git-ignore established in 0004), keyed by unique skill name, since skill names are globally unique in this repo.
Per-skill HTML is latest-only and overwritten each run, because the longitudinal data already lives in the per-skill history file.
The core's fixture unit test is extended to cover the leaderboard rendering: the sort order across a mix of regressed, efficacy-failed, fragile, and clean skills, the not-applicable Regression cell for a skill with no previous version, and the links to per-skill files.
## Acceptance criteria
- [x] `/benchmark-skill` with no argument benchmarks every skill following the `tests/` convention and renders an index leaderboard.
- [x] Each leaderboard row carries both the Efficacy and Regression badges; a skill with no previous version reads not-applicable in its Regression cell.
- [x] The sort promotes any red first with regressions above efficacy failures, then fragile-but-passing skills, then clean green.
- [x] The leaderboard links out to the separate per-skill report files.
- [x] The per-skill report files and the leaderboard index live flat under `tests/.reports/` (its git-ignore established in 0004), keyed by unique skill name.
- [x] Per-skill HTML is latest-only and overwritten each run.
- [x] The core's fixture unit test covers the leaderboard sort order, the not-applicable Regression cell, and the per-skill links.
## Implementation Notes
The leaderboard is a second pure transform in the deterministic core, `core/benchmark_core.py`, fully covered by the extended `checks/benchmark-core.nix` fixture test.
The in-session orchestration half — enumerate every skill with tests, run steps 16 for each, then render the index — is documented in `SKILL.md` step 7 rather than unit-tested, following the 0006 split where the workflow half is established by running the harness, not the fixture test.
Per the invocation, the benchmark harness itself was not run.
- **The leaderboard consumes per-skill results JSONs, not bundles, so the core stays a pure transform.**
The single-skill flow already writes `tests/.reports/<name>.results.json`; the batch flow feeds every one of those to `--leaderboard`, which sorts and renders the index to `tests/.reports/index.html`.
- **The CLI gains a `--leaderboard` mode that reuses `--json`/`--html`.**
The positional argument was widened from a single `bundle` to `inputs` (`nargs="+"`) with an explicit count guard, so the single-skill contract `core.py <bundle> --json … --html …` is unchanged and every existing call site still works.
- **The fragile-but-passing tier reuses the 0006 per-badge chips rather than recomputing margins.**
The regression chip already means "one loss from regressing." A green run always chips its narrowest efficacy case, so chip presence alone cannot tell a barely-green skill from a roomy one; the efficacy chip therefore now carries its win count, and the leaderboard reads that count against the existing `EFFICACY_WINS_FLOOR` to decide efficacy fragility — the spec's own definition, "closest to dropping under three wins."
- **Fragility is scoped to the fragile-but-passing tier (review finding).**
A red skill carries no fragility chip even when a still-passing axis sits at its edge, so the chip stays the marker of tier 2 rather than leaking onto a red row.
- **Two leaderboard fixtures are authored inline in the check, alongside the four scored models.**
No committed bundle reaches the clean-green tier (the clean bundle's narrowest efficacy case sits on the floor, so it is itself fragile) or the efficacy-red-with-a-still-fragile-regression-axis crossing, so `robust-skill` and `leaky-skill` are synthesized as small results JSONs the way 0006 synthesized its 55-line history seed.