feat: add all-skills leaderboard (task 0007)

Add a second pure transform to the benchmark core: a `--leaderboard` mode
that ranks per-skill results models into an index leaderboard, one row per
skill carrying both verdicts, links out to each per-skill report, and sorts
any red first (regressions above efficacy failures), then fragile-but-passing,
then clean green.

The fragile tier reuses the 0006 per-badge chips: the efficacy chip now
carries its win count so the leaderboard reads it against the pass floor
without recomputing margins, and fragility is scoped to the passing tier so a
red row never carries a chip. The runner's SKILL.md gains the no-argument
batch flow and the index invocation. The fixture test covers the tiered sort,
the not-applicable Regression cell, and the per-skill links.
This commit was merged in pull request #7.
This commit is contained in:
2026-07-24 16:58:02 -04:00
parent f4df5d31c9
commit 97c0eeb055
4 changed files with 325 additions and 25 deletions

View File

@@ -1,6 +1,6 @@
---
name: benchmark-skill
description: Benchmark a skill's efficacy against a no-skill baseline and, when a released version exists, its regression against that version, rendering an HTML report. Run deliberately as /benchmark-skill <name>, never automatically.
description: Benchmark a skill's efficacy against a no-skill baseline and, when a released version exists, its regression against that version, rendering an HTML report. Run deliberately as /benchmark-skill <name> for one skill, or /benchmark-skill with no argument for the all-skills leaderboard, never automatically.
disable-model-invocation: true
---
@@ -9,6 +9,7 @@ disable-model-invocation: true
Prove that a skill genuinely improves the agent's work rather than reading well and adding nothing.
Running `/benchmark-skill <name>` executes that skill's authored test cases as a controlled experiment and produces a self-contained HTML report carrying two independent verdicts: an **Efficacy verdict** (does the skill beat no-skill) and a **Regression verdict** (did my in-progress edit degrade it from the released version).
Running `/benchmark-skill` with **no argument** benchmarks every skill that follows the `tests/` convention and renders an index leaderboard over their reports (step 7).
Each case runs a **new-skill arm** (the working tree) and a **no-skill baseline arm**, and — whenever the skill already exists on main and its directory differs from that released state — a third **previous-version arm** materialized from main's `HEAD`.
Each arm runs several times, and a blind judge decides which arm's output better satisfies the case's author-written expectations.
Two head-to-heads fall out of the arms per trial: **efficacy** pairs the new arm against no-skill, and **regression** pairs it against the previous version.
@@ -27,6 +28,7 @@ Run from the repo root: every path in the steps below is relative to it.
## 1. Resolve the target and validate its tests tree
The user passes the skill name as `<name>`.
With **no `<name>`** you are in the all-skills mode: skip to step 7, which enumerates the skills and drives steps 16 for each.
Find the skill's directory by its leaf name under `skills/` (any depth), and find its tests at the mirror point under `tests/skills/…/<name>/`.
If no skill directory maps to `<name>`, stop and say so.
If the skill exists but has no tests directory or no case directories, report that the skill has no tests and stop — there is nothing to benchmark.
@@ -188,3 +190,27 @@ Clean up the scratch when done: remove the `tests/.reports/.work/` directory and
Done when `tests/.reports/<name>.html` exists.
Report both verdicts, the path to the report, and any flagged losses or regressions to the user.
## 7. Benchmark every skill and render the leaderboard
This step runs only in the all-skills mode — `/benchmark-skill` with no `<name>`.
Enumerate every skill the same way step 1 resolves one: each skill directory under `skills/` (any depth) whose mirror point under `tests/skills/…/` holds case directories.
A skill with no tests directory is skipped rather than failing the batch; note which skills were skipped.
Run steps 16 for each skill in turn, so each writes its own per-skill report and results model under `tests/.reports/`, flat and keyed by skill name — the per-skill HTML is latest-only and overwritten each run, since the longitudinal data lives in the per-skill history file.
Then render the index over every per-skill results model:
```sh
python3 skills/benchmark-skill/core/benchmark_core.py --leaderboard \
tests/.reports/*.results.json \
--html tests/.reports/index.html \
--generated-at "<ISO-8601 timestamp>"
```
The core sorts the rows — any red first with regressions ordered above efficacy failures, then fragile-but-passing skills, then clean green — and links each row out to its `<name>.html`.
A skill with no previous version reads not-applicable in its Regression cell.
The fragile-but-passing tier reuses the per-badge fragility signal from the per-skill reports: a case one loss from regressing, or a narrowest efficacy margin sitting on the pass floor.
Done when `tests/.reports/index.html` exists.
Report the leaderboard path, and per skill both verdicts and any flagged losses or regressions, to the user.