feat: add trend history and ribbons (task 0006) #6
59
.claude/tasks/0006-trend-history-and-ribbons.md
Normal file
59
.claude/tasks/0006-trend-history-and-ribbons.md
Normal file
@@ -0,0 +1,59 @@
|
||||
---
|
||||
spec: skill-benchmarking
|
||||
blocked-by: 0005-regression-arm-and-two-verdicts
|
||||
---
|
||||
|
||||
## What to build
|
||||
|
||||
The longitudinal layer that lets a real change be told apart from run-to-run noise, plus the fragility flags that stop a barely-green skill from masquerading as robust.
|
||||
|
||||
**History.**
|
||||
The deterministic core appends one summary line per run to a per-skill history file and trims it, oldest-first, at roughly the last fifty.
|
||||
Each line records both the efficacy and the regression net-margin (wins minus losses) and pass/fail, plus a flag for whether the run was two-arm or three-arm, so both ribbons can plot their own sparkline and correctly show a gap for any two-arm run.
|
||||
The history is local and ephemeral: it lives inside the git-ignored reports directory and resets if that directory is cleaned, consistent with reports being transient artifacts.
|
||||
|
||||
**Trend ribbons.**
|
||||
The report grows two stacked trend ribbons — an Efficacy ribbon and a Regression ribbon — each a net-margin sparkline over roughly the last seven runs, with verdict-colored per-run dots, the current run ringed, and a readout of the current net, the change versus the previous run, and how many recent runs were green.
|
||||
The Regression ribbon's dot is absent for any run that was two-arm.
|
||||
The Regression series is comparable across runs only while main is unchanged; because the previous-version arm is always main's `HEAD`, merging the branch moves main and resets the meaningful regression history, which is accepted rather than normalized.
|
||||
|
||||
**Fragility chips and the clean resting state.**
|
||||
A clean run rests fully collapsed, and fragility is flagged per badge.
|
||||
A green Efficacy badge carries a "narrowest margin" chip on the case fewest trial-flips from failing efficacy — closest to dropping under three wins.
|
||||
A green Regression badge independently carries a chip on the case sitting at exactly one loss.
|
||||
A case fragile on both axes shows both chips, so a barely-green skill cannot look as safe as a clean sweep on either axis.
|
||||
|
||||
The core's fixture unit test is extended to cover the history append-and-trim, the two ribbons' rendering including the two-arm gap in the regression series, and the per-badge fragility chips.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [x] The core appends one summary line per run to a per-skill history file and trims oldest-first at roughly fifty lines.
|
||||
- [x] Each history line records the efficacy and regression net-margins and pass/fail plus a two-arm/three-arm flag.
|
||||
- [x] The history file lives inside the git-ignored reports directory and is treated as ephemeral (resets when that directory is cleaned).
|
||||
- [x] The report renders two stacked ribbons (Efficacy and Regression), each a net-margin sparkline over ~7 runs with verdict-colored dots, the current run ringed, and a net/delta/green-count readout.
|
||||
- [x] The Regression ribbon shows no dot for any run that was two-arm.
|
||||
- [x] A clean run rests fully collapsed.
|
||||
- [x] A green Efficacy badge carries a narrowest-margin chip on the case fewest trial-flips from failing efficacy; a green Regression badge independently carries a chip on the case at exactly one loss; a both-axes-fragile case shows both.
|
||||
- [x] The core's fixture unit test covers the history append-and-trim, both ribbons (including the two-arm gap), and the per-badge fragility chips.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The whole slice lives in the deterministic core, `core/benchmark_core.py`, and is fully covered by the extended `checks/benchmark-core.nix` fixture test.
|
||||
The runner's `SKILL.md` step 6 gains the `--history tests/.reports/<name>.history.jsonl` argument and a description of the append-trim and the ribbons, since the orchestration half runs in-session and is established by running the harness rather than unit-tested.
|
||||
|
||||
- **The standard `/implement` branching flow was followed this time.**
|
||||
Task 0005's PR had merged to `main`, so the blocker was reachable: `main` was fast-forwarded to `origin/main` and `task-0006` branched off it, without the manual stack-tip fast-forward 0005 needed.
|
||||
- **History persistence is opt-in via `--history` and is the core's only side effect.**
|
||||
Without the flag the core stays a pure transform and renders no ribbons, so the 0005-era report shape is unchanged.
|
||||
The history is JSON-lines — one run summary object per line — read, appended, trimmed to fifty, and rewritten.
|
||||
- **The ribbon delta compares against the previous run that carries a value on that axis, not the literal previous line.**
|
||||
On the regression axis this skips a two-arm gap rather than blanking the change readout, consistent with the spec accepting regression discontinuity across a main move.
|
||||
- **The green-count readout shows `green/applicable`, not a bare green count.**
|
||||
The denominator is the number of runs in the window that have a value on that axis, so a regression readout excludes two-arm runs it cannot score, which reads more honestly than counting them against the total.
|
||||
- **The sparkline draws a faint dashed zero baseline** when the plotted window straddles zero, as a readability aid for telling a positive net-margin run from a negative one.
|
||||
- **Two committed fixtures were added, machine-generated for accuracy with the generator not committed** (the 0005 precedent).
|
||||
The existing red fixtures carry no green axis and so exercise no chips, so `clean-bundle.json` (fully clean, green/green) pins the fully-collapsed resting state and the lone narrowest-margin efficacy chip, and `fragile-bundle.json` (green but fragile) pins the independent per-badge chips and the both-axes case carrying both.
|
||||
- **The efficacy chip is a single case, the regression chip is every case at exactly one loss.**
|
||||
This asymmetry follows the wording: "narrowest margin" is comparative and picks one case (ties broken by authored order), while "the case sitting at exactly one loss" is an absolute condition any number of cases can meet.
|
||||
- **A case at exactly one regression loss both auto-expands and carries its chip.**
|
||||
"A clean run rests fully collapsed" holds because a clean run has zero losses, so nothing trips the 0005 flagged-loss auto-expand — the chip is what still flags fragility on a case that has no loss to open it (a green efficacy case sitting at exactly three wins).
|
||||
@@ -3,6 +3,9 @@
|
||||
# skill-level Efficacy and Regression verdicts, and the rendered report — for
|
||||
# both the three-arm shape and the degenerate two-arm "no previous version"
|
||||
# shape.
|
||||
# It also drives the longitudinal layer: the per-skill history append-and-trim,
|
||||
# the two trend ribbons including the two-arm gap in the regression series, and
|
||||
# the per-badge fragility chips on a clean run and a green-but-fragile one.
|
||||
# No LLM runs.
|
||||
{
|
||||
pkgs,
|
||||
@@ -11,11 +14,13 @@ let
|
||||
core = ../skills/benchmark-skill/core/benchmark_core.py;
|
||||
threeArm = ./fixtures/benchmark/three-arm-bundle.json;
|
||||
twoArm = ./fixtures/benchmark/two-arm-bundle.json;
|
||||
clean = ./fixtures/benchmark/clean-bundle.json;
|
||||
fragile = ./fixtures/benchmark/fragile-bundle.json;
|
||||
in
|
||||
pkgs.runCommandLocal "benchmark-core-check"
|
||||
{
|
||||
nativeBuildInputs = [ pkgs.python3 ];
|
||||
inherit core threeArm twoArm;
|
||||
inherit core threeArm twoArm clean fragile;
|
||||
}
|
||||
''
|
||||
fail() { echo "FAIL: $1" >&2; exit 1; }
|
||||
@@ -180,5 +185,97 @@ pkgs.runCommandLocal "benchmark-core-check"
|
||||
grep -q 'OUT-new-regresses-baseline-t3' two.html || fail "missing new evidence for the two-arm efficacy loss"
|
||||
grep -q 'OUT-baseline-regresses-baseline-t3' two.html || fail "missing baseline evidence for the two-arm efficacy loss"
|
||||
|
||||
# --- clean run: fully collapsed, one narrowest-margin chip, ribbons ---------
|
||||
echo "the core rests a clean run fully collapsed with a lone narrowest-margin chip"
|
||||
python3 "$core" "$clean" --json clean.json --html clean.html --history clean.history.jsonl \
|
||||
|| fail "the core exited non-zero on the clean bundle"
|
||||
|
||||
python3 - clean.json <<'PY' || fail "a clean-run assertion failed"
|
||||
import json, sys
|
||||
r = json.load(open(sys.argv[1]))
|
||||
assert r["efficacyVerdict"] == "green", r["efficacyVerdict"]
|
||||
assert r["regressionVerdict"] == "green", r["regressionVerdict"]
|
||||
cases = {c["name"]: c for c in r["cases"]}
|
||||
# The narrowest efficacy case (fewest wins) carries the chip, the roomy one does not.
|
||||
assert [ch["axis"] for ch in cases["narrowest"]["chips"]] == ["efficacy"], cases["narrowest"]["chips"]
|
||||
assert cases["roomy"]["chips"] == [], cases["roomy"]["chips"]
|
||||
print("clean-run assertions passed")
|
||||
PY
|
||||
|
||||
if grep -q ' open>' clean.html; then fail "a clean run did not rest fully collapsed"; fi
|
||||
grep -q 'class="chip efficacy"' clean.html || fail "clean run omits the narrowest-margin efficacy chip"
|
||||
if grep -q 'class="chip regression"' clean.html; then fail "clean run shows a spurious regression chip"; fi
|
||||
grep -q 'class="ribbons"' clean.html || fail "clean run omits the trend ribbons"
|
||||
grep -q 'class="spark"' clean.html || fail "clean run omits a sparkline"
|
||||
grep -q 'class="ring"' clean.html || fail "clean run does not ring the current run"
|
||||
|
||||
# --- history append-and-trim and the two-ribbon window with a two-arm gap ---
|
||||
echo "the core appends to and trims the per-skill history and plots the ribbon window"
|
||||
python3 - seed.history.jsonl <<'PY' || fail "seeding the history fixture failed"
|
||||
import json, sys
|
||||
# 55 prior runs, all three-arm except the most recent, which is two-arm and so
|
||||
# must plot a gap on the regression axis once it lands inside the 7-run window.
|
||||
lines = []
|
||||
for i in range(55):
|
||||
two = i == 54
|
||||
lines.append({
|
||||
"generatedAt": f"seed-{i}",
|
||||
"armShape": "two-arm" if two else "three-arm",
|
||||
"efficacyNet": i % 5,
|
||||
"efficacyPass": True,
|
||||
"regressionNet": None if two else i % 3,
|
||||
"regressionPass": None if two else True,
|
||||
})
|
||||
open(sys.argv[1], "w").write("".join(json.dumps(l) + "\n" for l in lines))
|
||||
PY
|
||||
|
||||
python3 "$core" "$clean" --json trend.json --history seed.history.jsonl \
|
||||
|| fail "the core exited non-zero on the seeded-history run"
|
||||
|
||||
python3 - seed.history.jsonl trend.json <<'PY' || fail "a history/trend assertion failed"
|
||||
import json, sys
|
||||
lines = [json.loads(l) for l in open(sys.argv[1]) if l.strip()]
|
||||
# 55 prior + this run = 56, trimmed oldest-first back to the cap of 50.
|
||||
assert len(lines) == 50, len(lines)
|
||||
gens = [l["generatedAt"] for l in lines]
|
||||
assert "seed-0" not in gens and "seed-5" not in gens, "oldest runs were not trimmed"
|
||||
assert "seed-6" in gens and "seed-54" in gens, "recent runs were wrongly trimmed"
|
||||
assert lines[-1]["armShape"] == "three-arm", lines[-1]
|
||||
assert lines[-1]["efficacyNet"] == 8 and lines[-1]["regressionNet"] == 1, lines[-1]
|
||||
|
||||
t = json.load(open(sys.argv[2]))["trend"]
|
||||
eff, reg = t["efficacy"]["points"], t["regression"]["points"]
|
||||
assert len(eff) == 7 and len(reg) == 7, (len(eff), len(reg))
|
||||
# Efficacy has a value every run, while regression breaks at the two-arm run.
|
||||
assert all(p["net"] is not None for p in eff), "efficacy series should have no gaps"
|
||||
assert eff[-1]["current"] and reg[-1]["current"], "the current run is ringed on both axes"
|
||||
assert reg[5]["net"] is None, [p["net"] for p in reg]
|
||||
assert reg[-1]["net"] is not None, "the current three-arm run has a regression dot"
|
||||
assert t["regression"]["applicable"] == 6, t["regression"]["applicable"]
|
||||
print("history/trend assertions passed")
|
||||
PY
|
||||
|
||||
# --- green-but-fragile run: independent per-badge chips, both on one case ----
|
||||
echo "the core chips each fragility axis independently, and both on a doubly-fragile case"
|
||||
python3 "$core" "$fragile" --json fragile.json --html fragile.html \
|
||||
|| fail "the core exited non-zero on the fragile bundle"
|
||||
|
||||
python3 - fragile.json <<'PY' || fail "a fragility-chip assertion failed"
|
||||
import json, sys
|
||||
r = json.load(open(sys.argv[1]))
|
||||
assert r["efficacyVerdict"] == "green" and r["regressionVerdict"] == "green"
|
||||
cases = {c["name"]: c for c in r["cases"]}
|
||||
# One case is both the narrowest efficacy margin and at exactly one regression
|
||||
# loss, so it carries both chips.
|
||||
# The others carry at most their own axis.
|
||||
assert {ch["axis"] for ch in cases["both"]["chips"]} == {"efficacy", "regression"}, cases["both"]["chips"]
|
||||
assert [ch["axis"] for ch in cases["reg-only"]["chips"]] == ["regression"], cases["reg-only"]["chips"]
|
||||
assert cases["none"]["chips"] == [], cases["none"]["chips"]
|
||||
print("fragility-chip assertions passed")
|
||||
PY
|
||||
|
||||
grep -q 'class="chip efficacy"' fragile.html || fail "fragile run omits the efficacy chip"
|
||||
grep -q 'class="chip regression"' fragile.html || fail "fragile run omits the regression chip"
|
||||
|
||||
touch "$out"
|
||||
''
|
||||
|
||||
496
checks/fixtures/benchmark/clean-bundle.json
Normal file
496
checks/fixtures/benchmark/clean-bundle.json
Normal file
@@ -0,0 +1,496 @@
|
||||
{
|
||||
"skill": "clean-skill",
|
||||
"generatedAt": "2026-07-24T10:00:00Z",
|
||||
"temperature": 1.0,
|
||||
"trialsPerCase": 5,
|
||||
"arms": [
|
||||
{
|
||||
"id": "new",
|
||||
"label": "New skill"
|
||||
},
|
||||
{
|
||||
"id": "previous",
|
||||
"label": "Previous version"
|
||||
},
|
||||
{
|
||||
"id": "baseline",
|
||||
"label": "No skill"
|
||||
}
|
||||
],
|
||||
"comparisons": [
|
||||
{
|
||||
"id": "efficacy",
|
||||
"label": "Efficacy",
|
||||
"new": "new",
|
||||
"against": "baseline",
|
||||
"rule": "efficacy"
|
||||
},
|
||||
{
|
||||
"id": "regression",
|
||||
"label": "Regression",
|
||||
"new": "new",
|
||||
"against": "previous",
|
||||
"rule": "regression"
|
||||
}
|
||||
],
|
||||
"cases": [
|
||||
{
|
||||
"name": "narrowest",
|
||||
"description": "narrowest scenario",
|
||||
"softCriteria": [
|
||||
"a good answer for narrowest"
|
||||
],
|
||||
"trials": [
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-narrowest-t0",
|
||||
"previous": "OUT-previous-narrowest-t0",
|
||||
"baseline": "OUT-baseline-narrowest-t0"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-narrowest-t1",
|
||||
"previous": "OUT-previous-narrowest-t1",
|
||||
"baseline": "OUT-baseline-narrowest-t1"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-narrowest-t2",
|
||||
"previous": "OUT-previous-narrowest-t2",
|
||||
"baseline": "OUT-baseline-narrowest-t2"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-narrowest-t3",
|
||||
"previous": "OUT-previous-narrowest-t3",
|
||||
"baseline": "OUT-baseline-narrowest-t3"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-narrowest-t4",
|
||||
"previous": "OUT-previous-narrowest-t4",
|
||||
"baseline": "OUT-baseline-narrowest-t4"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "roomy",
|
||||
"description": "roomy scenario",
|
||||
"softCriteria": [
|
||||
"a good answer for roomy"
|
||||
],
|
||||
"trials": [
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-roomy-t0",
|
||||
"previous": "OUT-previous-roomy-t0",
|
||||
"baseline": "OUT-baseline-roomy-t0"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-roomy-t1",
|
||||
"previous": "OUT-previous-roomy-t1",
|
||||
"baseline": "OUT-baseline-roomy-t1"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-roomy-t2",
|
||||
"previous": "OUT-previous-roomy-t2",
|
||||
"baseline": "OUT-baseline-roomy-t2"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-roomy-t3",
|
||||
"previous": "OUT-previous-roomy-t3",
|
||||
"baseline": "OUT-baseline-roomy-t3"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-roomy-t4",
|
||||
"previous": "OUT-previous-roomy-t4",
|
||||
"baseline": "OUT-baseline-roomy-t4"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
725
checks/fixtures/benchmark/fragile-bundle.json
Normal file
725
checks/fixtures/benchmark/fragile-bundle.json
Normal file
@@ -0,0 +1,725 @@
|
||||
{
|
||||
"skill": "fragile-skill",
|
||||
"generatedAt": "2026-07-24T11:00:00Z",
|
||||
"temperature": 1.0,
|
||||
"trialsPerCase": 5,
|
||||
"arms": [
|
||||
{
|
||||
"id": "new",
|
||||
"label": "New skill"
|
||||
},
|
||||
{
|
||||
"id": "previous",
|
||||
"label": "Previous version"
|
||||
},
|
||||
{
|
||||
"id": "baseline",
|
||||
"label": "No skill"
|
||||
}
|
||||
],
|
||||
"comparisons": [
|
||||
{
|
||||
"id": "efficacy",
|
||||
"label": "Efficacy",
|
||||
"new": "new",
|
||||
"against": "baseline",
|
||||
"rule": "efficacy"
|
||||
},
|
||||
{
|
||||
"id": "regression",
|
||||
"label": "Regression",
|
||||
"new": "new",
|
||||
"against": "previous",
|
||||
"rule": "regression"
|
||||
}
|
||||
],
|
||||
"cases": [
|
||||
{
|
||||
"name": "both",
|
||||
"description": "both scenario",
|
||||
"softCriteria": [
|
||||
"a good answer for both"
|
||||
],
|
||||
"trials": [
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-both-t0",
|
||||
"previous": "OUT-previous-both-t0",
|
||||
"baseline": "OUT-baseline-both-t0"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-both-t1",
|
||||
"previous": "OUT-previous-both-t1",
|
||||
"baseline": "OUT-baseline-both-t1"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-both-t2",
|
||||
"previous": "OUT-previous-both-t2",
|
||||
"baseline": "OUT-baseline-both-t2"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-both-t3",
|
||||
"previous": "OUT-previous-both-t3",
|
||||
"baseline": "OUT-baseline-both-t3"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "previous",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-both-t4",
|
||||
"previous": "OUT-previous-both-t4",
|
||||
"baseline": "OUT-baseline-both-t4"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "reg-only",
|
||||
"description": "reg-only scenario",
|
||||
"softCriteria": [
|
||||
"a good answer for reg-only"
|
||||
],
|
||||
"trials": [
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-reg-only-t0",
|
||||
"previous": "OUT-previous-reg-only-t0",
|
||||
"baseline": "OUT-baseline-reg-only-t0"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-reg-only-t1",
|
||||
"previous": "OUT-previous-reg-only-t1",
|
||||
"baseline": "OUT-baseline-reg-only-t1"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-reg-only-t2",
|
||||
"previous": "OUT-previous-reg-only-t2",
|
||||
"baseline": "OUT-baseline-reg-only-t2"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-reg-only-t3",
|
||||
"previous": "OUT-previous-reg-only-t3",
|
||||
"baseline": "OUT-baseline-reg-only-t3"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "previous",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-reg-only-t4",
|
||||
"previous": "OUT-previous-reg-only-t4",
|
||||
"baseline": "OUT-baseline-reg-only-t4"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "none",
|
||||
"description": "none scenario",
|
||||
"softCriteria": [
|
||||
"a good answer for none"
|
||||
],
|
||||
"trials": [
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-none-t0",
|
||||
"previous": "OUT-previous-none-t0",
|
||||
"baseline": "OUT-baseline-none-t0"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-none-t1",
|
||||
"previous": "OUT-previous-none-t1",
|
||||
"baseline": "OUT-baseline-none-t1"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-none-t2",
|
||||
"previous": "OUT-previous-none-t2",
|
||||
"baseline": "OUT-baseline-none-t2"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-none-t3",
|
||||
"previous": "OUT-previous-none-t3",
|
||||
"baseline": "OUT-baseline-none-t3"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
},
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "fixture"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-none-t4",
|
||||
"previous": "OUT-previous-none-t4",
|
||||
"baseline": "OUT-baseline-none-t4"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 500,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 300,
|
||||
"turns": 3
|
||||
},
|
||||
"previous": {
|
||||
"input": 900,
|
||||
"output": 450,
|
||||
"cacheCreation": 180,
|
||||
"cacheRead": 270,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 600,
|
||||
"output": 300,
|
||||
"cacheCreation": 0,
|
||||
"cacheRead": 0,
|
||||
"turns": 2
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -173,11 +173,15 @@ Run the core over the bundle, writing the report and a machine-readable model:
|
||||
python3 skills/benchmark-skill/core/benchmark_core.py \
|
||||
tests/.reports/.work/<name>-bundle.json \
|
||||
--json tests/.reports/<name>.results.json \
|
||||
--html tests/.reports/<name>.html
|
||||
--html tests/.reports/<name>.html \
|
||||
--history tests/.reports/<name>.history.jsonl
|
||||
```
|
||||
|
||||
The core collapses each trial of each comparison to a WIN, TIE, or LOSS for the new arm, applies each comparison's pass rule — efficacy passes at wins ≥ 3 and losses ≤ 1, regression passes at losses ≤ 1 with no wins floor — flags any loss for human review, and yields the two skill-level verdicts: **Efficacy** (green when every case beats no-skill) and **Regression** (green when no case degraded, not-applicable on a two-arm run).
|
||||
It renders the self-contained HTML report — the two verdict badges, a headline reading them together, run metadata, the per-arm cost table (a previous-version row on a three-arm run), and the cases in stable authored order, each auto-expanding on any failure or flagged loss to show the losing-trial output pair — alongside the machine-readable model.
|
||||
With `--history` it appends this run's summary line — each axis's net margin and pass/fail plus the two-arm/three-arm flag — to the per-skill history file, trimming it oldest-first at roughly the last fifty runs.
|
||||
That history file lives in the git-ignored reports directory and is ephemeral: cleaning the directory resets it, consistent with reports being transient artifacts.
|
||||
It renders the self-contained HTML report — the two verdict badges, two stacked trend ribbons plotting each axis's net-margin over the recent runs (the regression ribbon leaving a gap for any two-arm run), a headline reading the badges together, run metadata, the per-arm cost table (a previous-version row on a three-arm run), and the cases in stable authored order, each auto-expanding on any failure or flagged loss to show the losing-trial output pair — alongside the machine-readable model.
|
||||
A clean run rests fully collapsed, and each green badge still flags its most fragile case with a chip so a barely-green skill cannot look robust.
|
||||
Cost is reported but never gates a verdict.
|
||||
|
||||
Clean up the scratch when done: remove the `tests/.reports/.work/` directory and every per-arm world and skill materialization you created under the system temp path — including the previous-version checkout.
|
||||
|
||||
@@ -11,8 +11,14 @@ The model is parameterized over the arms and the comparisons, so one path serves
|
||||
both the three-arm run (efficacy against no-skill and regression against the
|
||||
previous version) and the degenerate two-arm run with no previous version.
|
||||
|
||||
When `--history` names a per-skill history file, the run's summary line is
|
||||
appended to it, the file is trimmed oldest-first, and the report grows two trend
|
||||
ribbons plotted from the recent tail. That file read-and-rewrite is the only side
|
||||
effect, opt-in and deterministic given the same file contents and inputs.
|
||||
|
||||
Usage:
|
||||
benchmark_core.py <bundle.json> [--json <out.json>] [--html <out.html>]
|
||||
[--history <hist.jsonl>]
|
||||
|
||||
With no output flags it writes the results model as JSON to stdout.
|
||||
"""
|
||||
@@ -37,6 +43,16 @@ USAGE_COMPONENTS = ("input", "output", "cacheCreation", "cacheRead")
|
||||
|
||||
WIN, TIE, LOSS = "WIN", "TIE", "LOSS"
|
||||
|
||||
# The per-skill history is a JSON-lines tail of run summaries kept in the
|
||||
# git-ignored reports directory.
|
||||
# It is capped and trimmed oldest-first, and the ribbons plot only its most
|
||||
# recent window.
|
||||
HISTORY_CAP = 50
|
||||
RIBBON_WINDOW = 7
|
||||
|
||||
# Efficacy passes at three wins, so this is the floor a green case sits above.
|
||||
EFFICACY_WINS_FLOOR = 3
|
||||
|
||||
|
||||
def _arm_usage_totals(bundle, arm_id):
|
||||
"""Sum an arm's transcript usage across every case and trial."""
|
||||
@@ -160,6 +176,35 @@ def _net_margin(cases, comparison_id):
|
||||
return total
|
||||
|
||||
|
||||
def _flag_fragility(cases, efficacy_id, regression, efficacy_green, regression_green):
|
||||
"""Chip the most fragile passing case on each green axis.
|
||||
|
||||
A green run still names its weakest case so a barely-green skill cannot pass
|
||||
for robust. The efficacy chip lands on the single case closest to dropping
|
||||
under the three-win floor, ties broken by authored order. The regression chip
|
||||
lands on every case sitting at its one tolerated loss, so a case fragile on
|
||||
both axes carries both.
|
||||
"""
|
||||
for case in cases:
|
||||
case["chips"] = []
|
||||
|
||||
if efficacy_green and cases:
|
||||
narrowest = min(cases, key=lambda c: c["comparisons"][efficacy_id]["wins"])
|
||||
wins = narrowest["comparisons"][efficacy_id]["wins"]
|
||||
narrowest["chips"].append({
|
||||
"axis": "efficacy",
|
||||
"label": f"narrowest efficacy margin · {wins}W",
|
||||
})
|
||||
|
||||
if regression is not None and regression_green:
|
||||
for case in cases:
|
||||
if case["comparisons"][regression["id"]]["losses"] == 1:
|
||||
case["chips"].append({
|
||||
"axis": "regression",
|
||||
"label": "one loss from regressing",
|
||||
})
|
||||
|
||||
|
||||
def build_results(bundle):
|
||||
"""Turn a run bundle into the results model."""
|
||||
pricing = DEFAULT_PRICING
|
||||
@@ -184,6 +229,10 @@ def build_results(bundle):
|
||||
)
|
||||
regression_net = _net_margin(cases, regression["id"])
|
||||
|
||||
_flag_fragility(
|
||||
cases, efficacy_id, regression, efficacy_green, regression_verdict == "green"
|
||||
)
|
||||
|
||||
return {
|
||||
"skill": bundle["skill"],
|
||||
"generatedAt": bundle.get("generatedAt", ""),
|
||||
@@ -201,6 +250,87 @@ def build_results(bundle):
|
||||
}
|
||||
|
||||
|
||||
# --- History and trend --------------------------------------------------------
|
||||
|
||||
|
||||
def summary_line(results):
|
||||
"""The one-line run summary appended to the per-skill history.
|
||||
|
||||
It records each axis's net margin and pass/fail plus the arm shape — all the
|
||||
ribbons need to redraw both sparklines and show a gap for any two-arm run. A
|
||||
two-arm run has no regression measurement, so both its regression fields are
|
||||
null.
|
||||
"""
|
||||
three_arm = results["armShape"] == "three-arm"
|
||||
return {
|
||||
"generatedAt": results.get("generatedAt", ""),
|
||||
"armShape": results["armShape"],
|
||||
"efficacyNet": results["efficacyNetMargin"],
|
||||
"efficacyPass": results["efficacyVerdict"] == "green",
|
||||
"regressionNet": results["regressionNetMargin"],
|
||||
"regressionPass": (results["regressionVerdict"] == "green") if three_arm else None,
|
||||
}
|
||||
|
||||
|
||||
def load_history(path):
|
||||
"""Read the JSON-lines history, tolerating a missing or empty file."""
|
||||
try:
|
||||
with open(path) as f:
|
||||
return [json.loads(line) for line in f if line.strip()]
|
||||
except FileNotFoundError:
|
||||
return []
|
||||
|
||||
|
||||
def trim_history(lines):
|
||||
"""Keep the most recent runs, dropping the oldest past the cap."""
|
||||
return lines[-HISTORY_CAP:]
|
||||
|
||||
|
||||
def write_history(path, lines):
|
||||
"""Rewrite the history file with the given lines, oldest first."""
|
||||
with open(path, "w") as f:
|
||||
for line in lines:
|
||||
f.write(json.dumps(line) + "\n")
|
||||
|
||||
|
||||
def build_trend(series):
|
||||
"""Ribbon data for both axes over the most recent window of runs.
|
||||
|
||||
A run with no value on an axis — a two-arm run on the regression axis — plots
|
||||
no dot and breaks the line, so the gap reads as absent rather than being
|
||||
interpolated across. The delta compares against the previous run that actually
|
||||
carries a value on that axis, so a two-arm gap does not blank the comparison.
|
||||
"""
|
||||
window = series[-RIBBON_WINDOW:]
|
||||
|
||||
def axis(net_key, pass_key):
|
||||
points = [
|
||||
{
|
||||
"net": run.get(net_key),
|
||||
"pass": run.get(pass_key),
|
||||
"current": i == len(window) - 1,
|
||||
}
|
||||
for i, run in enumerate(window)
|
||||
]
|
||||
valued = [p for p in points if p["net"] is not None]
|
||||
current_net = points[-1]["net"] if points else None
|
||||
delta = None
|
||||
if current_net is not None and len(valued) >= 2:
|
||||
delta = valued[-1]["net"] - valued[-2]["net"]
|
||||
return {
|
||||
"points": points,
|
||||
"current": current_net,
|
||||
"delta": delta,
|
||||
"green": sum(1 for p in points if p["pass"] is True),
|
||||
"applicable": len(valued),
|
||||
}
|
||||
|
||||
return {
|
||||
"efficacy": axis("efficacyNet", "efficacyPass"),
|
||||
"regression": axis("regressionNet", "regressionPass"),
|
||||
}
|
||||
|
||||
|
||||
# --- HTML rendering -----------------------------------------------------------
|
||||
|
||||
_STYLE = """
|
||||
@@ -216,7 +346,26 @@ h1 { font-size: 1.4rem; margin: 0 0 .25rem; }
|
||||
.badge.red { background: #fbdcdc; color: #9b1c1c; }
|
||||
.badge.na { background: #eee; color: #666; }
|
||||
.headline { font-size: 1.02rem; font-weight: 600; margin: .8rem 0 .2rem; }
|
||||
.meta { color: #666; font-size: .85rem; margin: .5rem 0 1.5rem; }
|
||||
.meta { color: #666; font-size: .85rem; margin: .5rem 0 1rem; }
|
||||
.ribbons { display: grid; gap: .5rem; margin: .5rem 0 1.5rem; }
|
||||
.ribbon { display: grid; grid-template-columns: 6.5rem 1fr auto; align-items: center;
|
||||
gap: 1rem; border: 1px solid #e5e5e5; border-radius: 8px;
|
||||
padding: .5rem .9rem; background: #fff; }
|
||||
.rlabel { font-weight: 600; font-size: .85rem; }
|
||||
.spark { width: 100%; height: 44px; display: block; }
|
||||
.spark .line { fill: none; stroke: #c4c4c4; stroke-width: 1.5; }
|
||||
.spark .zero { stroke: #e5e5e5; stroke-width: 1; stroke-dasharray: 2 3; }
|
||||
.spark .dot.pass { fill: #2f9e5f; }
|
||||
.spark .dot.fail { fill: #c0392b; }
|
||||
.spark .ring { fill: none; stroke: #1a1a1a; stroke-width: 1.5; }
|
||||
.readout { font-size: .8rem; color: #555; white-space: nowrap; text-align: right; }
|
||||
.readout .up { color: #0f6b34; font-weight: 600; }
|
||||
.readout .down { color: #9b1c1c; font-weight: 600; }
|
||||
.chip { display: inline-block; margin-left: .5rem; padding: .05rem .55rem;
|
||||
border-radius: 999px; font-size: .7rem; font-weight: 600;
|
||||
vertical-align: middle; }
|
||||
.chip.efficacy { background: #fff1cf; color: #8a5a00; }
|
||||
.chip.regression { background: #e9e2fb; color: #5b3ea8; }
|
||||
table { border-collapse: collapse; width: 100%; margin: .5rem 0; font-size: .9rem; }
|
||||
th, td { text-align: right; padding: .4rem .6rem; border-bottom: 1px solid #e5e5e5; }
|
||||
th:first-child, td:first-child { text-align: left; }
|
||||
@@ -301,6 +450,100 @@ def _badge(label, state):
|
||||
return f"<span class=\"badge {cls}\">{label}: {text}</span>"
|
||||
|
||||
|
||||
def _sparkline(points, width=240, height=44):
|
||||
"""An inline SVG net-margin sparkline where gaps break the line and drop no dot."""
|
||||
nets = [p["net"] for p in points if p["net"] is not None]
|
||||
if not nets:
|
||||
return f"<svg viewBox=\"0 0 {width} {height}\" class=\"spark\"></svg>"
|
||||
|
||||
pad_x, pad_y = 10, 9
|
||||
lo, hi = min(nets), max(nets)
|
||||
n = len(points)
|
||||
|
||||
def x(i):
|
||||
return width / 2 if n == 1 else pad_x + i * (width - 2 * pad_x) / (n - 1)
|
||||
|
||||
def y(v):
|
||||
if hi == lo:
|
||||
return height / 2
|
||||
return height - pad_y - (v - lo) / (hi - lo) * (height - 2 * pad_y)
|
||||
|
||||
parts = [
|
||||
f"<svg viewBox=\"0 0 {width} {height}\" class=\"spark\" "
|
||||
"preserveAspectRatio=\"xMidYMid meet\" xmlns=\"http://www.w3.org/2000/svg\">"
|
||||
]
|
||||
if lo < 0 < hi:
|
||||
zy = y(0)
|
||||
parts.append(f"<line x1=\"0\" y1=\"{zy:.1f}\" x2=\"{width}\" y2=\"{zy:.1f}\" class=\"zero\"/>")
|
||||
|
||||
# Split the run into contiguous valued segments so a gap leaves a real break.
|
||||
segments, current = [], []
|
||||
for i, p in enumerate(points):
|
||||
if p["net"] is None:
|
||||
if current:
|
||||
segments.append(current)
|
||||
current = []
|
||||
else:
|
||||
current.append((x(i), y(p["net"])))
|
||||
if current:
|
||||
segments.append(current)
|
||||
for seg in segments:
|
||||
if len(seg) >= 2:
|
||||
coords = " ".join(f"{px:.1f},{py:.1f}" for px, py in seg)
|
||||
parts.append(f"<polyline points=\"{coords}\" class=\"line\"/>")
|
||||
|
||||
for i, p in enumerate(points):
|
||||
if p["net"] is None:
|
||||
continue
|
||||
cx, cy = x(i), y(p["net"])
|
||||
if p["current"]:
|
||||
parts.append(f"<circle cx=\"{cx:.1f}\" cy=\"{cy:.1f}\" r=\"5.5\" class=\"ring\"/>")
|
||||
cls = "dot pass" if p["pass"] else "dot fail"
|
||||
parts.append(f"<circle cx=\"{cx:.1f}\" cy=\"{cy:.1f}\" r=\"3\" class=\"{cls}\"/>")
|
||||
|
||||
parts.append("</svg>")
|
||||
return "".join(parts)
|
||||
|
||||
|
||||
def _ribbon_readout(axis):
|
||||
"""The net / delta-vs-previous / green-count line beneath a ribbon."""
|
||||
net = "net —" if axis["current"] is None else f"net {axis['current']:+d}"
|
||||
|
||||
delta = axis["delta"]
|
||||
if delta is None:
|
||||
change = ""
|
||||
elif delta > 0:
|
||||
change = f"<span class=\"up\">▲ {delta}</span> vs prev"
|
||||
elif delta < 0:
|
||||
change = f"<span class=\"down\">▼ {abs(delta)}</span> vs prev"
|
||||
else:
|
||||
change = "±0 vs prev"
|
||||
|
||||
if axis["applicable"]:
|
||||
green = f"{axis['green']}/{axis['applicable']} green"
|
||||
else:
|
||||
green = "no runs yet"
|
||||
|
||||
return " · ".join(bit for bit in (net, change, green) if bit)
|
||||
|
||||
|
||||
def _render_ribbons(results):
|
||||
"""Two stacked net-margin ribbons, empty when no history was supplied."""
|
||||
trend = results.get("trend")
|
||||
if not trend:
|
||||
return ""
|
||||
out = ["<div class=\"ribbons\">"]
|
||||
for key, label in (("efficacy", "Efficacy"), ("regression", "Regression")):
|
||||
axis = trend[key]
|
||||
out.append("<div class=\"ribbon\">")
|
||||
out.append(f"<div class=\"rlabel\">{label}</div>")
|
||||
out.append(_sparkline(axis["points"]))
|
||||
out.append(f"<div class=\"readout\">{_ribbon_readout(axis)}</div>")
|
||||
out.append("</div>")
|
||||
out.append("</div>")
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def render_html(results):
|
||||
e = html.escape
|
||||
arm_labels = {a["id"]: a["label"] for a in results["arms"]}
|
||||
@@ -322,6 +565,8 @@ def render_html(results):
|
||||
meta_bits.append(e(results["generatedAt"]))
|
||||
out.append(f"<div class=\"meta\">{' · '.join(meta_bits)}</div>")
|
||||
|
||||
out.append(_render_ribbons(results))
|
||||
|
||||
out.append(f"<p class=\"headline\">{e(_headline(results))}</p>")
|
||||
|
||||
# Per-arm cost table.
|
||||
@@ -357,8 +602,12 @@ def render_html(results):
|
||||
failed = (not case["efficacyPassed"]) or case["regressionPassed"] is False
|
||||
cls = "case fail" if failed else "case"
|
||||
opened = " open" if expanded else ""
|
||||
chips = "".join(
|
||||
f"<span class=\"chip {ch['axis']}\">{e(ch['label'])}</span>"
|
||||
for ch in case.get("chips", [])
|
||||
)
|
||||
out.append(f"<details class=\"{cls}\"{opened}>")
|
||||
out.append(f"<summary>{e(case['name'])}</summary>")
|
||||
out.append(f"<summary>{e(case['name'])}{chips}</summary>")
|
||||
if case["description"]:
|
||||
out.append(f"<p class=\"desc\">{e(case['description'])}</p>")
|
||||
if case["hardFailed"]:
|
||||
@@ -418,6 +667,11 @@ def main(argv=None):
|
||||
parser.add_argument("bundle", help="path to the run-bundle JSON")
|
||||
parser.add_argument("--json", dest="json_out", help="write the results model here")
|
||||
parser.add_argument("--html", dest="html_out", help="write the HTML report here")
|
||||
parser.add_argument(
|
||||
"--history",
|
||||
dest="history",
|
||||
help="append this run to the JSON-lines history file and draw the trend ribbons",
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
with open(args.bundle) as f:
|
||||
@@ -425,6 +679,11 @@ def main(argv=None):
|
||||
|
||||
results = build_results(bundle)
|
||||
|
||||
if args.history:
|
||||
series = load_history(args.history) + [summary_line(results)]
|
||||
write_history(args.history, trim_history(series))
|
||||
results["trend"] = build_trend(series)
|
||||
|
||||
if args.json_out:
|
||||
with open(args.json_out, "w") as f:
|
||||
json.dump(results, f, indent=2)
|
||||
|
||||
Reference in New Issue
Block a user