alexion
97c0eeb055
Add a second pure transform to the benchmark core: a `--leaderboard` mode that ranks per-skill results models into an index leaderboard, one row per skill carrying both verdicts, links out to each per-skill report, and sorts any red first (regressions above efficacy failures), then fragile-but-passing, then clean green. The fragile tier reuses the 0006 per-badge chips: the efficacy chip now carries its win count so the leaderboard reads it against the pass floor without recomputing margins, and fragility is scoped to the passing tier so a red row never carries a chip. The runner's SKILL.md gains the no-argument batch flow and the index invocation. The fixture test covers the tiered sort, the not-applicable Regression cell, and the per-skill links.
47 lines
4.8 KiB
Markdown
47 lines
4.8 KiB
Markdown
---
|
||
spec: skill-benchmarking
|
||
blocked-by: 0006-trend-history-and-ribbons
|
||
---
|
||
|
||
## What to build
|
||
|
||
The batch view: `/benchmark-skill` with no argument benchmarks every skill and produces an index leaderboard, so an author knows at a glance which skills are green and which regressed.
|
||
|
||
Invoked with no argument, the runner benchmarks every skill that follows the `tests/` convention and renders an index leaderboard.
|
||
Each skill row carries both badges, Efficacy and Regression, and a skill with no previous version reads not-applicable in its Regression cell.
|
||
The sort promotes any red first, with regressions ordered above efficacy failures — a regression means "you just broke something that was working," the more urgent signal while iterating — then fragile-but-passing skills, then clean green.
|
||
The fragile-but-passing tier reuses the per-badge fragility signal introduced with the trend layer (0006) rather than recomputing it here.
|
||
The leaderboard links out to the separate per-skill report files produced by a single-skill run.
|
||
|
||
The per-skill report files and the index live flat under the `tests/.reports/` directory (its git-ignore established in 0004), keyed by unique skill name, since skill names are globally unique in this repo.
|
||
Per-skill HTML is latest-only and overwritten each run, because the longitudinal data already lives in the per-skill history file.
|
||
|
||
The core's fixture unit test is extended to cover the leaderboard rendering: the sort order across a mix of regressed, efficacy-failed, fragile, and clean skills, the not-applicable Regression cell for a skill with no previous version, and the links to per-skill files.
|
||
|
||
## Acceptance criteria
|
||
|
||
- [x] `/benchmark-skill` with no argument benchmarks every skill following the `tests/` convention and renders an index leaderboard.
|
||
- [x] Each leaderboard row carries both the Efficacy and Regression badges; a skill with no previous version reads not-applicable in its Regression cell.
|
||
- [x] The sort promotes any red first with regressions above efficacy failures, then fragile-but-passing skills, then clean green.
|
||
- [x] The leaderboard links out to the separate per-skill report files.
|
||
- [x] The per-skill report files and the leaderboard index live flat under `tests/.reports/` (its git-ignore established in 0004), keyed by unique skill name.
|
||
- [x] Per-skill HTML is latest-only and overwritten each run.
|
||
- [x] The core's fixture unit test covers the leaderboard sort order, the not-applicable Regression cell, and the per-skill links.
|
||
|
||
## Implementation Notes
|
||
|
||
The leaderboard is a second pure transform in the deterministic core, `core/benchmark_core.py`, fully covered by the extended `checks/benchmark-core.nix` fixture test.
|
||
The in-session orchestration half — enumerate every skill with tests, run steps 1–6 for each, then render the index — is documented in `SKILL.md` step 7 rather than unit-tested, following the 0006 split where the workflow half is established by running the harness, not the fixture test.
|
||
Per the invocation, the benchmark harness itself was not run.
|
||
|
||
- **The leaderboard consumes per-skill results JSONs, not bundles, so the core stays a pure transform.**
|
||
The single-skill flow already writes `tests/.reports/<name>.results.json`; the batch flow feeds every one of those to `--leaderboard`, which sorts and renders the index to `tests/.reports/index.html`.
|
||
- **The CLI gains a `--leaderboard` mode that reuses `--json`/`--html`.**
|
||
The positional argument was widened from a single `bundle` to `inputs` (`nargs="+"`) with an explicit count guard, so the single-skill contract `core.py <bundle> --json … --html …` is unchanged and every existing call site still works.
|
||
- **The fragile-but-passing tier reuses the 0006 per-badge chips rather than recomputing margins.**
|
||
The regression chip already means "one loss from regressing." A green run always chips its narrowest efficacy case, so chip presence alone cannot tell a barely-green skill from a roomy one; the efficacy chip therefore now carries its win count, and the leaderboard reads that count against the existing `EFFICACY_WINS_FLOOR` to decide efficacy fragility — the spec's own definition, "closest to dropping under three wins."
|
||
- **Fragility is scoped to the fragile-but-passing tier (review finding).**
|
||
A red skill carries no fragility chip even when a still-passing axis sits at its edge, so the chip stays the marker of tier 2 rather than leaking onto a red row.
|
||
- **Two leaderboard fixtures are authored inline in the check, alongside the four scored models.**
|
||
No committed bundle reaches the clean-green tier (the clean bundle's narrowest efficacy case sits on the floor, so it is itself fragile) or the efficacy-red-with-a-still-fragile-regression-axis crossing, so `robust-skill` and `leaky-skill` are synthesized as small results JSONs the way 0006 synthesized its 55-line history seed.
|