feat: add all-skills leaderboard (task 0007)
Add a second pure transform to the benchmark core: a `--leaderboard` mode that ranks per-skill results models into an index leaderboard, one row per skill carrying both verdicts, links out to each per-skill report, and sorts any red first (regressions above efficacy failures), then fragile-but-passing, then clean green. The fragile tier reuses the 0006 per-badge chips: the efficacy chip now carries its win count so the leaderboard reads it against the pass floor without recomputing margins, and fragility is scoped to the passing tier so a red row never carries a chip. The runner's SKILL.md gains the no-argument batch flow and the index invocation. The fixture test covers the tiered sort, the not-applicable Regression cell, and the per-skill links.
This commit was merged in pull request #7.
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: benchmark-skill
|
||||
description: Benchmark a skill's efficacy against a no-skill baseline and, when a released version exists, its regression against that version, rendering an HTML report. Run deliberately as /benchmark-skill <name>, never automatically.
|
||||
description: Benchmark a skill's efficacy against a no-skill baseline and, when a released version exists, its regression against that version, rendering an HTML report. Run deliberately as /benchmark-skill <name> for one skill, or /benchmark-skill with no argument for the all-skills leaderboard, never automatically.
|
||||
disable-model-invocation: true
|
||||
---
|
||||
|
||||
@@ -9,6 +9,7 @@ disable-model-invocation: true
|
||||
Prove that a skill genuinely improves the agent's work rather than reading well and adding nothing.
|
||||
|
||||
Running `/benchmark-skill <name>` executes that skill's authored test cases as a controlled experiment and produces a self-contained HTML report carrying two independent verdicts: an **Efficacy verdict** (does the skill beat no-skill) and a **Regression verdict** (did my in-progress edit degrade it from the released version).
|
||||
Running `/benchmark-skill` with **no argument** benchmarks every skill that follows the `tests/` convention and renders an index leaderboard over their reports (step 7).
|
||||
Each case runs a **new-skill arm** (the working tree) and a **no-skill baseline arm**, and — whenever the skill already exists on main and its directory differs from that released state — a third **previous-version arm** materialized from main's `HEAD`.
|
||||
Each arm runs several times, and a blind judge decides which arm's output better satisfies the case's author-written expectations.
|
||||
Two head-to-heads fall out of the arms per trial: **efficacy** pairs the new arm against no-skill, and **regression** pairs it against the previous version.
|
||||
@@ -27,6 +28,7 @@ Run from the repo root: every path in the steps below is relative to it.
|
||||
## 1. Resolve the target and validate its tests tree
|
||||
|
||||
The user passes the skill name as `<name>`.
|
||||
With **no `<name>`** you are in the all-skills mode: skip to step 7, which enumerates the skills and drives steps 1–6 for each.
|
||||
Find the skill's directory by its leaf name under `skills/` (any depth), and find its tests at the mirror point under `tests/skills/…/<name>/`.
|
||||
If no skill directory maps to `<name>`, stop and say so.
|
||||
If the skill exists but has no tests directory or no case directories, report that the skill has no tests and stop — there is nothing to benchmark.
|
||||
@@ -188,3 +190,27 @@ Clean up the scratch when done: remove the `tests/.reports/.work/` directory and
|
||||
|
||||
Done when `tests/.reports/<name>.html` exists.
|
||||
Report both verdicts, the path to the report, and any flagged losses or regressions to the user.
|
||||
|
||||
## 7. Benchmark every skill and render the leaderboard
|
||||
|
||||
This step runs only in the all-skills mode — `/benchmark-skill` with no `<name>`.
|
||||
|
||||
Enumerate every skill the same way step 1 resolves one: each skill directory under `skills/` (any depth) whose mirror point under `tests/skills/…/` holds case directories.
|
||||
A skill with no tests directory is skipped rather than failing the batch; note which skills were skipped.
|
||||
Run steps 1–6 for each skill in turn, so each writes its own per-skill report and results model under `tests/.reports/`, flat and keyed by skill name — the per-skill HTML is latest-only and overwritten each run, since the longitudinal data lives in the per-skill history file.
|
||||
|
||||
Then render the index over every per-skill results model:
|
||||
|
||||
```sh
|
||||
python3 skills/benchmark-skill/core/benchmark_core.py --leaderboard \
|
||||
tests/.reports/*.results.json \
|
||||
--html tests/.reports/index.html \
|
||||
--generated-at "<ISO-8601 timestamp>"
|
||||
```
|
||||
|
||||
The core sorts the rows — any red first with regressions ordered above efficacy failures, then fragile-but-passing skills, then clean green — and links each row out to its `<name>.html`.
|
||||
A skill with no previous version reads not-applicable in its Regression cell.
|
||||
The fragile-but-passing tier reuses the per-badge fragility signal from the per-skill reports: a case one loss from regressing, or a narrowest efficacy margin sitting on the pass floor.
|
||||
|
||||
Done when `tests/.reports/index.html` exists.
|
||||
Report the leaderboard path, and per skill both verdicts and any flagged losses or regressions, to the user.
|
||||
|
||||
Reference in New Issue
Block a user