alexion/skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add trend history and ribbons (task 0006)

Browse Source 
Add the longitudinal layer to the benchmark core: a per-skill JSON-lines
history that appends one summary line per run and trims oldest-first at
fifty, two stacked net-margin trend ribbons (Efficacy and Regression, the
latter leaving a gap for two-arm runs) with the current run ringed and a
net/delta/green-count readout, and per-badge fragility chips that flag the
narrowest passing case on each green axis so a barely-green skill cannot
look robust.

History persistence is opt-in via --history and is the core's only side
effect. Without it the core stays a pure transform. Two committed fixtures
(clean and fragile) and new fixture-test sections cover the append-and-trim,
both ribbons including the two-arm gap, and the chips.
This commit was merged in pull request #6.
This commit is contained in:
alexion
2026-07-24 16:37:11 -04:00
parent 9b40d9f4e5
commit f4df5d31c9
6 changed files with 1645 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
skills/benchmark-skill/SKILL.md
View File

@@ -173,11 +173,15 @@ Run the core over the bundle, writing the report and a machine-readable model:
python3 skills/benchmark-skill/core/benchmark_core.py \
tests/.reports/.work/<name>-bundle.json \
--json tests/.reports/<name>.results.json \
--html tests/.reports/<name>.html
--html tests/.reports/<name>.html \
--history tests/.reports/<name>.history.jsonl
```
The core collapses each trial of each comparison to a WIN, TIE, or LOSS for the new arm, applies each comparison's pass rule — efficacy passes at wins ≥ 3 and losses ≤ 1, regression passes at losses ≤ 1 with no wins floor — flags any loss for human review, and yields the two skill-level verdicts: **Efficacy** (green when every case beats no-skill) and **Regression** (green when no case degraded, not-applicable on a two-arm run).
It renders the self-contained HTML report — the two verdict badges, a headline reading them together, run metadata, the per-arm cost table (a previous-version row on a three-arm run), and the cases in stable authored order, each auto-expanding on any failure or flagged loss to show the losing-trial output pair — alongside the machine-readable model.
With `--history` it appends this run's summary line — each axis's net margin and pass/fail plus the two-arm/three-arm flag — to the per-skill history file, trimming it oldest-first at roughly the last fifty runs.
That history file lives in the git-ignored reports directory and is ephemeral: cleaning the directory resets it, consistent with reports being transient artifacts.
It renders the self-contained HTML report — the two verdict badges, two stacked trend ribbons plotting each axis's net-margin over the recent runs (the regression ribbon leaving a gap for any two-arm run), a headline reading the badges together, run metadata, the per-arm cost table (a previous-version row on a three-arm run), and the cases in stable authored order, each auto-expanding on any failure or flagged loss to show the losing-trial output pair — alongside the machine-readable model.
A clean run rests fully collapsed, and each green badge still flags its most fragile case with a chip so a barely-green skill cannot look robust.
Cost is reported but never gates a verdict.
Clean up the scratch when done: remove the `tests/.reports/.work/` directory and every per-arm world and skill materialization you created under the system temp path — including the previous-version checkout.