---
spec: skill-benchmarking
blocked-by: 0006-trend-history-and-ribbons
---

## What to build

The batch view: `/benchmark-skill` with no argument benchmarks every skill and produces an index leaderboard, so an author knows at a glance which skills are green and which regressed.

Invoked with no argument, the runner benchmarks every skill that follows the `tests/` convention and renders an index leaderboard.
Each skill row carries both badges, Efficacy and Regression, and a skill with no previous version reads not-applicable in its Regression cell.
The sort promotes any red first, with regressions ordered above efficacy failures — a regression means "you just broke something that was working," the more urgent signal while iterating — then fragile-but-passing skills, then clean green.
The fragile-but-passing tier reuses the per-badge fragility signal introduced with the trend layer (0006) rather than recomputing it here.
The leaderboard links out to the separate per-skill report files produced by a single-skill run.

The per-skill report files and the index live flat under the `tests/.reports/` directory (its git-ignore established in 0004), keyed by unique skill name, since skill names are globally unique in this repo.
Per-skill HTML is latest-only and overwritten each run, because the longitudinal data already lives in the per-skill history file.

The core's fixture unit test is extended to cover the leaderboard rendering: the sort order across a mix of regressed, efficacy-failed, fragile, and clean skills, the not-applicable Regression cell for a skill with no previous version, and the links to per-skill files.

## Acceptance criteria

- [x] `/benchmark-skill` with no argument benchmarks every skill following the `tests/` convention and renders an index leaderboard.
- [x] Each leaderboard row carries both the Efficacy and Regression badges; a skill with no previous version reads not-applicable in its Regression cell.
- [x] The sort promotes any red first with regressions above efficacy failures, then fragile-but-passing skills, then clean green.
- [x] The leaderboard links out to the separate per-skill report files.
- [x] The per-skill report files and the leaderboard index live flat under `tests/.reports/` (its git-ignore established in 0004), keyed by unique skill name.
- [x] Per-skill HTML is latest-only and overwritten each run.
- [x] The core's fixture unit test covers the leaderboard sort order, the not-applicable Regression cell, and the per-skill links.

## Implementation Notes

The leaderboard is a second pure transform in the deterministic core, `core/benchmark_core.py`, fully covered by the extended `checks/benchmark-core.nix` fixture test.
The in-session orchestration half — enumerate every skill with tests, run steps 1–6 for each, then render the index — is documented in `SKILL.md` step 7 rather than unit-tested, following the 0006 split where the workflow half is established by running the harness, not the fixture test.
Per the invocation, the benchmark harness itself was not run.

- **The leaderboard consumes per-skill results JSONs, not bundles, so the core stays a pure transform.**
  The single-skill flow already writes `tests/.reports/<name>.results.json`; the batch flow feeds every one of those to `--leaderboard`, which sorts and renders the index to `tests/.reports/index.html`.
- **The CLI gains a `--leaderboard` mode that reuses `--json`/`--html`.**
  The positional argument was widened from a single `bundle` to `inputs` (`nargs="+"`) with an explicit count guard, so the single-skill contract `core.py <bundle> --json … --html …` is unchanged and every existing call site still works.
- **The fragile-but-passing tier reuses the 0006 per-badge chips rather than recomputing margins.**
  The regression chip already means "one loss from regressing." A green run always chips its narrowest efficacy case, so chip presence alone cannot tell a barely-green skill from a roomy one; the efficacy chip therefore now carries its win count, and the leaderboard reads that count against the existing `EFFICACY_WINS_FLOOR` to decide efficacy fragility — the spec's own definition, "closest to dropping under three wins."
- **Fragility is scoped to the fragile-but-passing tier (review finding).**
  A red skill carries no fragility chip even when a still-passing axis sits at its edge, so the chip stays the marker of tier 2 rather than leaking onto a red row.
- **Two leaderboard fixtures are authored inline in the check, alongside the four scored models.**
  No committed bundle reaches the clean-green tier (the clean bundle's narrowest efficacy case sits on the floor, so it is itself fragile) or the efficacy-red-with-a-still-fragile-regression-axis crossing, so `robust-skill` and `leaky-skill` are synthesized as small results JSONs the way 0006 synthesized its 55-line history seed.