skills/.claude/tasks/0007-all-skills-leaderboard.md

spec, blocked-by

spec	blocked-by
skill-benchmarking	0006-trend-history-and-ribbons

What to build

The batch view: /benchmark-skill with no argument benchmarks every skill and produces an index leaderboard, so an author knows at a glance which skills are green and which regressed.

Invoked with no argument, the runner benchmarks every skill that follows the tests/ convention and renders an index leaderboard. Each skill row carries both badges, Efficacy and Regression, and a skill with no previous version reads not-applicable in its Regression cell. The sort promotes any red first, with regressions ordered above efficacy failures — a regression means "you just broke something that was working," the more urgent signal while iterating — then fragile-but-passing skills, then clean green. The fragile-but-passing tier reuses the per-badge fragility signal introduced with the trend layer (0006) rather than recomputing it here. The leaderboard links out to the separate per-skill report files produced by a single-skill run.

The per-skill report files and the index live flat under the tests/.reports/ directory (its git-ignore established in 0004), keyed by unique skill name, since skill names are globally unique in this repo. Per-skill HTML is latest-only and overwritten each run, because the longitudinal data already lives in the per-skill history file.

The core's fixture unit test is extended to cover the leaderboard rendering: the sort order across a mix of regressed, efficacy-failed, fragile, and clean skills, the not-applicable Regression cell for a skill with no previous version, and the links to per-skill files.

Acceptance criteria

• /benchmark-skill with no argument benchmarks every skill following the tests/ convention and renders an index leaderboard.
• Each leaderboard row carries both the Efficacy and Regression badges; a skill with no previous version reads not-applicable in its Regression cell.
• The sort promotes any red first with regressions above efficacy failures, then fragile-but-passing skills, then clean green.
• The leaderboard links out to the separate per-skill report files.
• The per-skill report files and the leaderboard index live flat under tests/.reports/ (its git-ignore established in 0004), keyed by unique skill name.
• Per-skill HTML is latest-only and overwritten each run.
• The core's fixture unit test covers the leaderboard sort order, the not-applicable Regression cell, and the per-skill links.

Implementation Notes

The leaderboard is a second pure transform in the deterministic core, core/benchmark_core.py, fully covered by the extended checks/benchmark-core.nix fixture test. The in-session orchestration half — enumerate every skill with tests, run steps 1–6 for each, then render the index — is documented in SKILL.md step 7 rather than unit-tested, following the 0006 split where the workflow half is established by running the harness, not the fixture test. Per the invocation, the benchmark harness itself was not run.

• The leaderboard consumes per-skill results JSONs, not bundles, so the core stays a pure transform. The single-skill flow already writes tests/.reports/<name>.results.json; the batch flow feeds every one of those to --leaderboard, which sorts and renders the index to tests/.reports/index.html.
• The CLI gains a --leaderboard mode that reuses --json/--html. The positional argument was widened from a single bundle to inputs (nargs="+") with an explicit count guard, so the single-skill contract core.py <bundle> --json … --html … is unchanged and every existing call site still works.
• The fragile-but-passing tier reuses the 0006 per-badge chips rather than recomputing margins. The regression chip already means "one loss from regressing." A green run always chips its narrowest efficacy case, so chip presence alone cannot tell a barely-green skill from a roomy one; the efficacy chip therefore now carries its win count, and the leaderboard reads that count against the existing EFFICACY_WINS_FLOOR to decide efficacy fragility — the spec's own definition, "closest to dropping under three wins."
• Fragility is scoped to the fragile-but-passing tier (review finding). A red skill carries no fragility chip even when a still-passing axis sits at its edge, so the chip stays the marker of tier 2 rather than leaking onto a red row.
• Two leaderboard fixtures are authored inline in the check, alongside the four scored models. No committed bundle reaches the clean-green tier (the clean bundle's narrowest efficacy case sits on the floor, so it is itself fragile) or the efficacy-red-with-a-still-fragile-regression-axis crossing, so robust-skill and leaky-skill are synthesized as small results JSONs the way 0006 synthesized its 55-line history seed.