feat: add regression arm and two verdicts (task 0005)
Turn a benchmark run into a three-arm experiment: alongside the new-skill and no-skill arms, add a previous-version arm materialized from the default branch's HEAD, drawn automatically whenever the skill's directory differs from HEAD and degrading to the two-arm efficacy-only shape otherwise. Two blind head-to-heads now fall out per trial — efficacy (new-vs-no-skill) and regression (new-vs-old). The deterministic core dispatches a pass rule per comparison (efficacy at wins>=3 and losses<=1, regression at losses<=1 with no wins floor), gates only Efficacy on the hard assertion, and yields a second skill-level Regression verdict, green when no case regressed and not-applicable on a two-arm run. The report gains a second badge, a headline reading both verdicts together, a previous-version cost row with a two-ratio footnote, and per-case side-by-side comparisons that auto-expand on any failure or flagged loss to show the losing-trial output pair. The fixture test covers both shapes in one test: a three-arm run asserting per-arm metrics, both per-case and skill-level verdicts, the net margins, the three-row cost table and losing-trial evidence, and the two-arm run retained as the degenerate no-previous-version case.
This commit was merged in pull request #5.
This commit is contained in:
77
.claude/tasks/0005-regression-arm-and-two-verdicts.md
Normal file
77
.claude/tasks/0005-regression-arm-and-two-verdicts.md
Normal file
@@ -0,0 +1,77 @@
|
||||
---
|
||||
spec: skill-benchmarking
|
||||
blocked-by: 0008-arm-isolation-hardening
|
||||
---
|
||||
|
||||
## What to build
|
||||
|
||||
The second head-to-head: a **previous-version arm** that turns a run into a three-arm experiment and adds a **Regression verdict** beside the Efficacy one, so an edit that silently made a trusted skill worse is caught rather than passing green.
|
||||
|
||||
The previous version is always main's `HEAD` — the repo's default branch, resolved rather than hardcoded to the literal name "main" — and the new version is the working tree.
|
||||
Anchoring to the released state on main makes the regression question always "did my in-progress edit degrade from the trusted version," which is the moment of real risk.
|
||||
The third arm is added **fully automatically and carries no new invocation surface**: the runner diffs the skill's directory against main's `HEAD` and adds the previous-version arm exactly when the two differ.
|
||||
A brand-new skill absent from main, or a skill unchanged from `HEAD`, has no meaningful previous version, so the run degrades to the two-arm efficacy-only shape.
|
||||
|
||||
The previous-version arm's skill is materialized by checking out just the skill's directory at main's `HEAD` into a temp path, since each skill is a self-contained derivation that does not need the rest of the repo.
|
||||
That temp materialization is cleaned up after the run, like the fresh fixture copies.
|
||||
The previous-version arm is force-invoked identically to the new arm, pointed at this temp checkout instead of the new arm's copy of the working-tree directory, so the skill version is the only difference between them.
|
||||
Cases are held fixed to the working tree — all arms run against today's prompt, fixture, and expectations — so the skill version is the only variable differing between the new and previous-version arms.
|
||||
The hard-assertion gate stays on the new arm only, and the blind judge and the case's soft criteria are reused unchanged for the regression comparison; only the pair of outputs handed to the judge differs.
|
||||
|
||||
Two blind head-to-heads are now drawn per trial: **efficacy** pairs the new arm against the no-skill arm, and **regression** pairs the new arm against the previous-version arm.
|
||||
Trial *i*'s new-skill output is judged against trial *i*'s no-skill output for efficacy and against trial *i*'s previous-version output for regression.
|
||||
|
||||
The regression pass rule inverts, because holding quality steady is the goal.
|
||||
Per trial, the new-vs-old head-to-head collapses to WIN, TIE, or LOSS for the new arm, where a TIE means "as good as the previous version" (a success), a LOSS is the regression being hunted, and a WIN is a bonus improvement.
|
||||
Per case, regression passes when losses ≤ 1 across the 5 trials, with no wins floor — ties and wins both count as non-regressions — and any loss is flagged for human review.
|
||||
Dropping the wins floor means a deliberate no-op edit that ties the previous version passes rather than failing merely for tying.
|
||||
|
||||
The deterministic core is exercised across its full three-arm, two-comparison path (the same parameterized program from 0004, now run with both comparisons rather than re-architected).
|
||||
It gains the regression pass rule and a skill-level **Regression verdict**, green when no case regressed.
|
||||
The per-run result now carries the regression net-margin and a two-arm/three-arm flag, but persisting those into the durable per-run history line is deferred to the trend-history slice (0006), which owns history persistence; this slice's report remains latest-only with no ribbons.
|
||||
The two verdicts are reported side by side as independent badges, because their four crossings carry genuinely different meanings, and the report headline reads both badges together into one "so what."
|
||||
On a two-arm run with no previous version the Regression verdict reads not-applicable.
|
||||
The cost table gains a **previous-version row** (turns, raw tokens, cost-equivalent tokens, imputed cost), which drops out on a two-arm run, and the footnote names the two trustworthy ratios: new-vs-no-skill and new-vs-old.
|
||||
Each case panel shows both comparisons side by side, each with its per-trial WIN/TIE/LOSS strip and its pass result, and auto-expands when either comparison fails or carries a flagged loss — so a regression opens the panel even when efficacy is green — surfacing the losing-trial output pair for whichever comparison failed.
|
||||
|
||||
The core's fixture unit test is extended so the same test covers both shapes: the three-arm run (sample transcripts for all three arms and per-trial judge verdicts for both comparisons) asserting the per-arm metrics, both per-case and skill-level verdicts, and a report with the three-row cost table, and the two-arm run retained as the degenerate "no previous version" case.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [x] The previous version resolves to the default branch's `HEAD` dynamically, never a hardcoded "main" literal, and the new version is the working tree.
|
||||
- [x] The third arm is added automatically, with no new flag or argument, exactly when the skill directory differs from `HEAD`; otherwise the run stays two-arm efficacy-only.
|
||||
- [x] The previous-version arm materializes by checking out just the skill's directory at `HEAD` into a temp path, force-invoked against it identically to the new arm, and the temp path is cleaned up after the run.
|
||||
- [x] All arms run against the working-tree case (prompt, fixture, expectations), so the skill version is the only variable between the new and previous-version arms.
|
||||
- [x] The hard-assertion gate remains new-arm-only; the same blind judge and soft criteria are reused for the regression comparison, differing only in the output pair.
|
||||
- [x] Per trial, efficacy is judged new-vs-no-skill and regression new-vs-previous, using that trial's own outputs with no cross-arm reuse.
|
||||
- [x] The regression pass rule passes a case when losses ≤ 1 across 5 trials with no wins floor (ties and wins are non-regressions), flagging any loss.
|
||||
- [x] The core, unchanged in structure, runs the full three-arm two-comparison path and yields a skill-level Regression verdict green when no case regressed.
|
||||
- [x] The report shows Efficacy and Regression as two independent badges with a headline reading both together; Regression reads not-applicable on a two-arm run.
|
||||
- [x] The cost table gains a previous-version row that drops out on a two-arm run, with a footnote naming the new-vs-no-skill and new-vs-old ratios.
|
||||
- [x] Each case panel shows both comparisons side by side with per-trial WIN/TIE/LOSS strips, auto-expanding on any fail or flagged loss and showing the losing-trial output pair for the failed comparison.
|
||||
- [x] The core's fixture unit test covers both the three-arm and the degenerate two-arm shapes in the same test, asserting metrics, both verdicts, and the three-row cost table.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The deterministic half — the pass rules, the two verdicts, the net margins, and the report — lives in `core/benchmark_core.py` and is fully covered by the extended `checks/benchmark-core.nix` fixture test.
|
||||
The orchestration half — resolving `HEAD`, diff-gating the third arm, materializing the previous-version checkout, and drawing the two head-to-heads (criteria 1–6) — is specified in the runner's `SKILL.md` prose, since that half runs in-session as an AI script and, per the spec's Testing Decisions, is not unit-tested but established by running the harness.
|
||||
|
||||
- **Branching deviated from the standard `/implement` flow, at the user's direction.**
|
||||
Task 0005 is blocked by 0008, which stacks on 0004, and none of that had been merged to `main` — so branching 0005 off a fresh `main` would have lost the entire benchmark foundation.
|
||||
`main` was fast-forwarded to the `task-0004` stack tip (a clean linear superset) and pushed, then `task-0005` was branched off it.
|
||||
- **The hard-assertion gate feeds Efficacy only, not Regression.**
|
||||
The spec defines the regression pass rule purely as losses ≤ 1 and never mentions the gate, and the gate exists to catch the *current* skill emitting malformed output — an absolute property of the new arm, whereas regression is relative to the previous version.
|
||||
So a hard failure fails the case's Efficacy axis outright while its Regression axis is judged purely on the head-to-head.
|
||||
The three-arm fixture's `hard-gate-fail` case pins this: Efficacy fails on the gate, Regression passes.
|
||||
- **The run bundle gained a per-trial `outputs` map (arm id → final message).**
|
||||
0004 carried no output text; the losing-trial evidence pair this slice requires needs it, so the core reads `outputs` to surface the new arm's output beside the one it lost to, only for flagged-loss trials of a failed or flagged comparison.
|
||||
- **Arm order is canonical: new-skill, previous-version, no-skill.**
|
||||
The cost table renders arms in bundle order, and a Spec-axis review caught that the initial new/no-skill/previous order put the previous-version row last rather than in the spec's stated middle position.
|
||||
The fixtures and the `SKILL.md` example now emit arms in the spec order.
|
||||
- **Comparisons render side by side via a CSS grid.**
|
||||
A Spec-axis review caught that the two comparisons were stacked vertically rather than laid out side by side as the report shape requires; a `.cmps` grid container now places them beside each other, wrapping to one column on a narrow viewport.
|
||||
- **Arm shape is derived from the presence of a regression comparison, not the arm count**, so a run is two-arm exactly when no regression comparison was drawn.
|
||||
- **The two committed fixtures replace 0004's single `run-bundle.json`.**
|
||||
`three-arm-bundle.json` is the primary shape and `two-arm-bundle.json` is retained as the degenerate "no previous version" case, both exercised by the one test.
|
||||
They are machine-generated for accuracy; the generator is not committed.
|
||||
- **A live three-arm `/benchmark-skill` run is the natural follow-up validation** (as 0008 did for isolation) but is not gated by this slice's acceptance criteria, so it was not run here.
|
||||
@@ -1,34 +1,146 @@
|
||||
# Feeds the committed run-bundle fixture through the benchmark skill's
|
||||
# Feeds the committed run-bundle fixtures through the benchmark skill's
|
||||
# deterministic core and asserts the per-arm metrics, the per-case and
|
||||
# skill-level Efficacy verdict, and the rendered report.
|
||||
# skill-level Efficacy and Regression verdicts, and the rendered report — for
|
||||
# both the three-arm shape and the degenerate two-arm "no previous version"
|
||||
# shape.
|
||||
# No LLM runs.
|
||||
{
|
||||
pkgs,
|
||||
}:
|
||||
let
|
||||
core = ../skills/benchmark-skill/core/benchmark_core.py;
|
||||
fixture = ./fixtures/benchmark/run-bundle.json;
|
||||
threeArm = ./fixtures/benchmark/three-arm-bundle.json;
|
||||
twoArm = ./fixtures/benchmark/two-arm-bundle.json;
|
||||
in
|
||||
pkgs.runCommandLocal "benchmark-core-check"
|
||||
{
|
||||
nativeBuildInputs = [ pkgs.python3 ];
|
||||
inherit core fixture;
|
||||
inherit core threeArm twoArm;
|
||||
}
|
||||
''
|
||||
fail() { echo "FAIL: $1" >&2; exit 1; }
|
||||
|
||||
echo "the core turns a run bundle into a results model and an HTML report"
|
||||
python3 "$core" "$fixture" --json results.json --html report.html \
|
||||
|| fail "the core exited non-zero"
|
||||
# --- three-arm run --------------------------------------------------------
|
||||
echo "the core scores a three-arm run and renders its report"
|
||||
python3 "$core" "$threeArm" --json three.json --html three.html \
|
||||
|| fail "the core exited non-zero on the three-arm bundle"
|
||||
|
||||
echo "the results model carries the expected metrics and verdicts"
|
||||
python3 - results.json <<'PY' || fail "a results-model assertion failed"
|
||||
echo "the three-arm results model carries both verdicts, net margins, and metrics"
|
||||
python3 - three.json <<'PY' || fail "a three-arm results-model assertion failed"
|
||||
import json, sys
|
||||
r = json.load(open(sys.argv[1]))
|
||||
|
||||
assert r["armShape"] == "three-arm", r["armShape"]
|
||||
# Efficacy is red: dead-weight fails the wins floor and hard-gate-fail trips
|
||||
# the gate. Regression is red: regressed-still-valuable loses twice to the
|
||||
# previous version.
|
||||
assert r["efficacyVerdict"] == "red", r["efficacyVerdict"]
|
||||
assert r["regressionVerdict"] == "red", r["regressionVerdict"]
|
||||
assert r["efficacyNetMargin"] == 12, r["efficacyNetMargin"]
|
||||
assert r["regressionNetMargin"] == 3, r["regressionNetMargin"]
|
||||
|
||||
new = r["armMetrics"]["new"]
|
||||
assert new["turns"] == 60, new["turns"]
|
||||
assert new["rawTokens"] == 228000, new["rawTokens"]
|
||||
assert new["costEquivalentTokens"] == 246000, new["costEquivalentTokens"]
|
||||
assert abs(new["imputedCost"] - 1.23) < 1e-9, new["imputedCost"]
|
||||
|
||||
base = r["armMetrics"]["baseline"]
|
||||
assert base["turns"] == 40, base["turns"]
|
||||
assert base["rawTokens"] == 114000, base["rawTokens"]
|
||||
assert base["costEquivalentTokens"] == 123000, base["costEquivalentTokens"]
|
||||
|
||||
prev = r["armMetrics"]["previous"]
|
||||
assert prev["turns"] == 60, prev["turns"]
|
||||
assert prev["rawTokens"] == 174000, prev["rawTokens"]
|
||||
assert prev["costEquivalentTokens"] == 195500, prev["costEquivalentTokens"]
|
||||
assert abs(prev["imputedCost"] - 0.9775) < 1e-9, prev["imputedCost"]
|
||||
|
||||
# Cost-table rows render in arm order: new-skill, previous-version, no-skill.
|
||||
assert [a["id"] for a in r["arms"]] == ["new", "previous", "baseline"], \
|
||||
[a["id"] for a in r["arms"]]
|
||||
|
||||
cases = {c["name"]: c for c in r["cases"]}
|
||||
# Authored order is preserved, never reshuffled by verdict.
|
||||
assert [c["name"] for c in r["cases"]] == [
|
||||
"clean-both", "regressed-still-valuable", "dead-weight", "hard-gate-fail"
|
||||
]
|
||||
|
||||
a = cases["clean-both"]
|
||||
assert a["efficacyPassed"] is True
|
||||
assert a["regressionPassed"] is True
|
||||
assert a["comparisons"]["efficacy"]["trials"] == ["WIN", "WIN", "WIN", "WIN", "TIE"]
|
||||
assert a["comparisons"]["regression"]["trials"] == ["TIE", "TIE", "WIN", "TIE", "TIE"]
|
||||
|
||||
# Efficacy green but regression red — the exact crossing this feature exists
|
||||
# to catch: still beats no-skill, yet degraded from the released version.
|
||||
b = cases["regressed-still-valuable"]
|
||||
assert b["efficacyPassed"] is True
|
||||
assert b["regressionPassed"] is False
|
||||
assert b["comparisons"]["regression"]["losses"] == 2
|
||||
assert b["comparisons"]["regression"]["flaggedLosses"] == [0, 1]
|
||||
|
||||
# Efficacy red but regression green — already dead weight, but the edit did
|
||||
# not make it worse.
|
||||
c = cases["dead-weight"]
|
||||
assert c["efficacyPassed"] is False
|
||||
assert c["regressionPassed"] is True
|
||||
assert c["comparisons"]["efficacy"]["wins"] == 0
|
||||
assert c["comparisons"]["efficacy"]["losses"] == 1
|
||||
assert c["comparisons"]["efficacy"]["flaggedLosses"] == [2]
|
||||
assert c["comparisons"]["regression"]["wins"] == 3
|
||||
|
||||
# The hard gate fails efficacy outright but does not gate regression, which
|
||||
# is judged purely on losses.
|
||||
d = cases["hard-gate-fail"]
|
||||
assert d["hardFailed"] is True
|
||||
assert d["efficacyPassed"] is False
|
||||
assert d["regressionPassed"] is True
|
||||
assert d["comparisons"]["efficacy"]["wins"] == 5
|
||||
print("three-arm results-model assertions passed")
|
||||
PY
|
||||
|
||||
echo "the three-arm report shows both badges, the three-row table, and evidence"
|
||||
grep -q '<!doctype html>' three.html || fail "three-arm report is not self-contained"
|
||||
grep -q 'Efficacy: RED' three.html || fail "three-arm report is missing the red Efficacy badge"
|
||||
grep -q 'Regression: RED' three.html || fail "three-arm report is missing the red Regression badge"
|
||||
grep -qi 'regress' three.html || fail "three-arm report is missing the verdict headline"
|
||||
grep -q 'Cost-equiv tokens' three.html || fail "three-arm report is missing the cost table"
|
||||
grep -q 'Previous version' three.html || fail "three-arm report is missing the previous-version row"
|
||||
grep -q 'shared-context cache' three.html || fail "three-arm report is missing the cache footnote"
|
||||
grep -q 'new-vs-no-skill' three.html || fail "three-arm footnote omits the new-vs-no-skill ratio"
|
||||
grep -q 'new-vs-old' three.html || fail "three-arm footnote omits the new-vs-old ratio"
|
||||
for name in clean-both regressed-still-valuable dead-weight hard-gate-fail; do
|
||||
grep -q "$name" three.html || fail "three-arm report omits case $name"
|
||||
done
|
||||
grep -q 'class="cell LOSS"' three.html || fail "three-arm report is missing a LOSS cell"
|
||||
grep -q 'Hard assertion failed' three.html || fail "three-arm report does not flag the hard failure"
|
||||
grep -q ' open>' three.html || fail "three-arm report does not auto-expand a failing case"
|
||||
grep -q 'class="cmps"' three.html || fail "three-arm report does not lay comparisons side by side"
|
||||
# Evidence for the failed regression comparison: new vs previous on trial 1.
|
||||
grep -q 'OUT-new-regressed-still-valuable-t0' three.html || fail "missing new evidence for the regression loss"
|
||||
grep -q 'OUT-previous-regressed-still-valuable-t0' three.html || fail "missing previous evidence for the regression loss"
|
||||
# Evidence for the failed efficacy comparison: new vs baseline on trial 3.
|
||||
grep -q 'OUT-new-dead-weight-t2' three.html || fail "missing new evidence for the efficacy loss"
|
||||
grep -q 'OUT-baseline-dead-weight-t2' three.html || fail "missing baseline evidence for the efficacy loss"
|
||||
# A clean, collapsed case emits no losing-trial evidence.
|
||||
if grep -q 'OUT-new-clean-both' three.html; then fail "a clean case leaked losing-trial evidence"; fi
|
||||
|
||||
# --- two-arm run ----------------------------------------------------------
|
||||
echo "the core scores the degenerate two-arm run"
|
||||
python3 "$core" "$twoArm" --json two.json --html two.html \
|
||||
|| fail "the core exited non-zero on the two-arm bundle"
|
||||
|
||||
echo "the two-arm results model reads regression as not-applicable"
|
||||
python3 - two.json <<'PY' || fail "a two-arm results-model assertion failed"
|
||||
import json, sys
|
||||
r = json.load(open(sys.argv[1]))
|
||||
|
||||
assert r["armShape"] == "two-arm", r["armShape"]
|
||||
# The skill verdict is red because two of the three cases fail efficacy.
|
||||
assert r["efficacyVerdict"] == "red", r["efficacyVerdict"]
|
||||
assert r["regressionVerdict"] == "not-applicable", r["regressionVerdict"]
|
||||
assert r["regressionNetMargin"] is None, r["regressionNetMargin"]
|
||||
assert r["efficacyNetMargin"] == 9, r["efficacyNetMargin"]
|
||||
|
||||
new = r["armMetrics"]["new"]
|
||||
assert new["turns"] == 45, new["turns"]
|
||||
@@ -41,41 +153,32 @@ pkgs.runCommandLocal "benchmark-core-check"
|
||||
assert base["rawTokens"] == 85500, base["rawTokens"]
|
||||
assert base["costEquivalentTokens"] == 92250, base["costEquivalentTokens"]
|
||||
|
||||
cases = {c["name"]: c for c in r["cases"]}
|
||||
# Authored order is preserved, never reshuffled by verdict.
|
||||
assert [c["name"] for c in r["cases"]] == ["clean-pass", "regresses-baseline", "hard-gate-fail"]
|
||||
assert "previous" not in r["armMetrics"], "two-arm run has no previous arm"
|
||||
|
||||
a = cases["clean-pass"]
|
||||
assert a["efficacyPassed"] is True
|
||||
assert a["comparisons"]["efficacy"]["trials"] == ["WIN", "WIN", "WIN", "WIN", "TIE"]
|
||||
assert a["comparisons"]["efficacy"]["wins"] == 4
|
||||
assert a["comparisons"]["efficacy"]["losses"] == 0
|
||||
assert [c["name"] for c in r["cases"]] == [
|
||||
"clean-pass", "regresses-baseline", "hard-gate-fail"
|
||||
]
|
||||
for c in r["cases"]:
|
||||
assert c["regressionPassed"] is None, (c["name"], c["regressionPassed"])
|
||||
|
||||
# A head-to-head fail: two losses drop it under the wins>=3, losses<=1 rule,
|
||||
# and both losses are flagged for human review.
|
||||
b = cases["regresses-baseline"]
|
||||
b = {c["name"]: c for c in r["cases"]}["regresses-baseline"]
|
||||
assert b["efficacyPassed"] is False
|
||||
assert b["comparisons"]["efficacy"]["losses"] == 2
|
||||
assert b["comparisons"]["efficacy"]["flaggedLosses"] == [3, 4]
|
||||
|
||||
# A hard-assertion failure fails the case outright despite a clean sweep.
|
||||
c = cases["hard-gate-fail"]
|
||||
assert c["hardFailed"] is True
|
||||
assert c["efficacyPassed"] is False
|
||||
assert c["comparisons"]["efficacy"]["wins"] == 5
|
||||
print("results-model assertions passed")
|
||||
h = {c["name"]: c for c in r["cases"]}["hard-gate-fail"]
|
||||
assert h["hardFailed"] is True
|
||||
print("two-arm results-model assertions passed")
|
||||
PY
|
||||
|
||||
echo "the report is self-contained and shows the badge, cost table, and cases"
|
||||
grep -q '<!doctype html>' report.html || fail "report is not a self-contained document"
|
||||
grep -q 'Efficacy: RED' report.html || fail "report is missing the red Efficacy badge"
|
||||
grep -q 'Cost-equiv tokens' report.html || fail "report is missing the per-arm cost table"
|
||||
grep -q 'shared-context cache' report.html || fail "report is missing the cache-overhead footnote"
|
||||
for name in clean-pass regresses-baseline hard-gate-fail; do
|
||||
grep -q "$name" report.html || fail "report omits case $name"
|
||||
done
|
||||
grep -q 'class="cell LOSS"' report.html || fail "report is missing a per-trial LOSS cell"
|
||||
grep -q 'Hard assertion failed' report.html || fail "report does not flag the hard-assertion failure"
|
||||
echo "the two-arm report reads Regression as not-applicable and drops the previous row"
|
||||
grep -q 'Efficacy: RED' two.html || fail "two-arm report is missing the Efficacy badge"
|
||||
grep -q 'Regression: N/A' two.html || fail "two-arm report does not read Regression as not-applicable"
|
||||
grep -q 'new-vs-no-skill' two.html || fail "two-arm footnote omits the new-vs-no-skill ratio"
|
||||
if grep -q 'new-vs-old' two.html; then fail "two-arm footnote names a new-vs-old ratio that does not apply"; fi
|
||||
if grep -q 'Previous version' two.html; then fail "two-arm report shows a previous-version row"; fi
|
||||
grep -q 'OUT-new-regresses-baseline-t3' two.html || fail "missing new evidence for the two-arm efficacy loss"
|
||||
grep -q 'OUT-baseline-regresses-baseline-t3' two.html || fail "missing baseline evidence for the two-arm efficacy loss"
|
||||
|
||||
touch "$out"
|
||||
''
|
||||
|
||||
954
checks/fixtures/benchmark/three-arm-bundle.json
Normal file
954
checks/fixtures/benchmark/three-arm-bundle.json
Normal file
@@ -0,0 +1,954 @@
|
||||
{
|
||||
"skill": "sample-skill",
|
||||
"generatedAt": "2026-07-24T12:00:00Z",
|
||||
"temperature": 1.0,
|
||||
"trialsPerCase": 5,
|
||||
"arms": [
|
||||
{
|
||||
"id": "new",
|
||||
"label": "New skill"
|
||||
},
|
||||
{
|
||||
"id": "previous",
|
||||
"label": "Previous version"
|
||||
},
|
||||
{
|
||||
"id": "baseline",
|
||||
"label": "No skill"
|
||||
}
|
||||
],
|
||||
"comparisons": [
|
||||
{
|
||||
"id": "efficacy",
|
||||
"label": "Efficacy",
|
||||
"new": "new",
|
||||
"against": "baseline",
|
||||
"rule": "efficacy"
|
||||
},
|
||||
{
|
||||
"id": "regression",
|
||||
"label": "Regression",
|
||||
"new": "new",
|
||||
"against": "previous",
|
||||
"rule": "regression"
|
||||
}
|
||||
],
|
||||
"cases": [
|
||||
{
|
||||
"name": "clean-both",
|
||||
"description": "New beats baseline and holds steady versus the previous version.",
|
||||
"softCriteria": [
|
||||
"The answer is grounded in the fixture."
|
||||
],
|
||||
"trials": [
|
||||
{
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge: efficacy new"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge: regression tie"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-clean-both-t0",
|
||||
"previous": "OUT-previous-clean-both-t0",
|
||||
"baseline": "OUT-baseline-clean-both-t0"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 2000,
|
||||
"cacheCreation": 400,
|
||||
"cacheRead": 8000,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 500,
|
||||
"output": 1000,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 4000,
|
||||
"turns": 2
|
||||
},
|
||||
"previous": {
|
||||
"input": 800,
|
||||
"output": 1600,
|
||||
"cacheCreation": 300,
|
||||
"cacheRead": 6000,
|
||||
"turns": 3
|
||||
}
|
||||
},
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge: efficacy new"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge: regression tie"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-clean-both-t1",
|
||||
"previous": "OUT-previous-clean-both-t1",
|
||||
"baseline": "OUT-baseline-clean-both-t1"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 2000,
|
||||
"cacheCreation": 400,
|
||||
"cacheRead": 8000,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 500,
|
||||
"output": 1000,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 4000,
|
||||
"turns": 2
|
||||
},
|
||||
"previous": {
|
||||
"input": 800,
|
||||
"output": 1600,
|
||||
"cacheCreation": 300,
|
||||
"cacheRead": 6000,
|
||||
"turns": 3
|
||||
}
|
||||
},
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge: efficacy new"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "new",
|
||||
"rationale": "judge: regression new"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-clean-both-t2",
|
||||
"previous": "OUT-previous-clean-both-t2",
|
||||
"baseline": "OUT-baseline-clean-both-t2"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 2000,
|
||||
"cacheCreation": 400,
|
||||
"cacheRead": 8000,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 500,
|
||||
"output": 1000,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 4000,
|
||||
"turns": 2
|
||||
},
|
||||
"previous": {
|
||||
"input": 800,
|
||||
"output": 1600,
|
||||
"cacheCreation": 300,
|
||||
"cacheRead": 6000,
|
||||
"turns": 3
|
||||
}
|
||||
},
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge: efficacy new"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge: regression tie"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-clean-both-t3",
|
||||
"previous": "OUT-previous-clean-both-t3",
|
||||
"baseline": "OUT-baseline-clean-both-t3"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 2000,
|
||||
"cacheCreation": 400,
|
||||
"cacheRead": 8000,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 500,
|
||||
"output": 1000,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 4000,
|
||||
"turns": 2
|
||||
},
|
||||
"previous": {
|
||||
"input": 800,
|
||||
"output": 1600,
|
||||
"cacheCreation": 300,
|
||||
"cacheRead": 6000,
|
||||
"turns": 3
|
||||
}
|
||||
},
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge: efficacy tie"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge: regression tie"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-clean-both-t4",
|
||||
"previous": "OUT-previous-clean-both-t4",
|
||||
"baseline": "OUT-baseline-clean-both-t4"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 2000,
|
||||
"cacheCreation": 400,
|
||||
"cacheRead": 8000,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 500,
|
||||
"output": 1000,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 4000,
|
||||
"turns": 2
|
||||
},
|
||||
"previous": {
|
||||
"input": 800,
|
||||
"output": 1600,
|
||||
"cacheCreation": 300,
|
||||
"cacheRead": 6000,
|
||||
"turns": 3
|
||||
}
|
||||
},
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "regressed-still-valuable",
|
||||
"description": "New still beats baseline but has degraded from the previous version.",
|
||||
"softCriteria": [
|
||||
"The answer resolves the user's request."
|
||||
],
|
||||
"trials": [
|
||||
{
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge: efficacy new"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "previous",
|
||||
"rationale": "judge: regression previous"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-regressed-still-valuable-t0",
|
||||
"previous": "OUT-previous-regressed-still-valuable-t0",
|
||||
"baseline": "OUT-baseline-regressed-still-valuable-t0"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 2000,
|
||||
"cacheCreation": 400,
|
||||
"cacheRead": 8000,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 500,
|
||||
"output": 1000,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 4000,
|
||||
"turns": 2
|
||||
},
|
||||
"previous": {
|
||||
"input": 800,
|
||||
"output": 1600,
|
||||
"cacheCreation": 300,
|
||||
"cacheRead": 6000,
|
||||
"turns": 3
|
||||
}
|
||||
},
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge: efficacy new"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "previous",
|
||||
"rationale": "judge: regression previous"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-regressed-still-valuable-t1",
|
||||
"previous": "OUT-previous-regressed-still-valuable-t1",
|
||||
"baseline": "OUT-baseline-regressed-still-valuable-t1"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 2000,
|
||||
"cacheCreation": 400,
|
||||
"cacheRead": 8000,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 500,
|
||||
"output": 1000,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 4000,
|
||||
"turns": 2
|
||||
},
|
||||
"previous": {
|
||||
"input": 800,
|
||||
"output": 1600,
|
||||
"cacheCreation": 300,
|
||||
"cacheRead": 6000,
|
||||
"turns": 3
|
||||
}
|
||||
},
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge: efficacy new"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge: regression tie"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-regressed-still-valuable-t2",
|
||||
"previous": "OUT-previous-regressed-still-valuable-t2",
|
||||
"baseline": "OUT-baseline-regressed-still-valuable-t2"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 2000,
|
||||
"cacheCreation": 400,
|
||||
"cacheRead": 8000,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 500,
|
||||
"output": 1000,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 4000,
|
||||
"turns": 2
|
||||
},
|
||||
"previous": {
|
||||
"input": 800,
|
||||
"output": 1600,
|
||||
"cacheCreation": 300,
|
||||
"cacheRead": 6000,
|
||||
"turns": 3
|
||||
}
|
||||
},
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge: efficacy tie"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "new",
|
||||
"rationale": "judge: regression new"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-regressed-still-valuable-t3",
|
||||
"previous": "OUT-previous-regressed-still-valuable-t3",
|
||||
"baseline": "OUT-baseline-regressed-still-valuable-t3"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 2000,
|
||||
"cacheCreation": 400,
|
||||
"cacheRead": 8000,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 500,
|
||||
"output": 1000,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 4000,
|
||||
"turns": 2
|
||||
},
|
||||
"previous": {
|
||||
"input": 800,
|
||||
"output": 1600,
|
||||
"cacheCreation": 300,
|
||||
"cacheRead": 6000,
|
||||
"turns": 3
|
||||
}
|
||||
},
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge: efficacy new"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge: regression tie"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-regressed-still-valuable-t4",
|
||||
"previous": "OUT-previous-regressed-still-valuable-t4",
|
||||
"baseline": "OUT-baseline-regressed-still-valuable-t4"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 2000,
|
||||
"cacheCreation": 400,
|
||||
"cacheRead": 8000,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 500,
|
||||
"output": 1000,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 4000,
|
||||
"turns": 2
|
||||
},
|
||||
"previous": {
|
||||
"input": 800,
|
||||
"output": 1600,
|
||||
"cacheCreation": 300,
|
||||
"cacheRead": 6000,
|
||||
"turns": 3
|
||||
}
|
||||
},
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "dead-weight",
|
||||
"description": "New does not beat baseline, but the edit did not make it worse.",
|
||||
"softCriteria": [
|
||||
"The answer is correct."
|
||||
],
|
||||
"trials": [
|
||||
{
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge: efficacy tie"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "new",
|
||||
"rationale": "judge: regression new"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-dead-weight-t0",
|
||||
"previous": "OUT-previous-dead-weight-t0",
|
||||
"baseline": "OUT-baseline-dead-weight-t0"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 2000,
|
||||
"cacheCreation": 400,
|
||||
"cacheRead": 8000,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 500,
|
||||
"output": 1000,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 4000,
|
||||
"turns": 2
|
||||
},
|
||||
"previous": {
|
||||
"input": 800,
|
||||
"output": 1600,
|
||||
"cacheCreation": 300,
|
||||
"cacheRead": 6000,
|
||||
"turns": 3
|
||||
}
|
||||
},
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge: efficacy tie"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "new",
|
||||
"rationale": "judge: regression new"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-dead-weight-t1",
|
||||
"previous": "OUT-previous-dead-weight-t1",
|
||||
"baseline": "OUT-baseline-dead-weight-t1"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 2000,
|
||||
"cacheCreation": 400,
|
||||
"cacheRead": 8000,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 500,
|
||||
"output": 1000,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 4000,
|
||||
"turns": 2
|
||||
},
|
||||
"previous": {
|
||||
"input": 800,
|
||||
"output": 1600,
|
||||
"cacheCreation": 300,
|
||||
"cacheRead": 6000,
|
||||
"turns": 3
|
||||
}
|
||||
},
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "baseline",
|
||||
"rationale": "judge: efficacy baseline"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge: regression tie"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-dead-weight-t2",
|
||||
"previous": "OUT-previous-dead-weight-t2",
|
||||
"baseline": "OUT-baseline-dead-weight-t2"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 2000,
|
||||
"cacheCreation": 400,
|
||||
"cacheRead": 8000,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 500,
|
||||
"output": 1000,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 4000,
|
||||
"turns": 2
|
||||
},
|
||||
"previous": {
|
||||
"input": 800,
|
||||
"output": 1600,
|
||||
"cacheCreation": 300,
|
||||
"cacheRead": 6000,
|
||||
"turns": 3
|
||||
}
|
||||
},
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge: efficacy tie"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "new",
|
||||
"rationale": "judge: regression new"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-dead-weight-t3",
|
||||
"previous": "OUT-previous-dead-weight-t3",
|
||||
"baseline": "OUT-baseline-dead-weight-t3"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 2000,
|
||||
"cacheCreation": 400,
|
||||
"cacheRead": 8000,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 500,
|
||||
"output": 1000,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 4000,
|
||||
"turns": 2
|
||||
},
|
||||
"previous": {
|
||||
"input": 800,
|
||||
"output": 1600,
|
||||
"cacheCreation": 300,
|
||||
"cacheRead": 6000,
|
||||
"turns": 3
|
||||
}
|
||||
},
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge: efficacy tie"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge: regression tie"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-dead-weight-t4",
|
||||
"previous": "OUT-previous-dead-weight-t4",
|
||||
"baseline": "OUT-baseline-dead-weight-t4"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 2000,
|
||||
"cacheCreation": 400,
|
||||
"cacheRead": 8000,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 500,
|
||||
"output": 1000,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 4000,
|
||||
"turns": 2
|
||||
},
|
||||
"previous": {
|
||||
"input": 800,
|
||||
"output": 1600,
|
||||
"cacheCreation": 300,
|
||||
"cacheRead": 6000,
|
||||
"turns": 3
|
||||
}
|
||||
},
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "hard-gate-fail",
|
||||
"description": "New sweeps both head-to-heads but violates a hard assertion.",
|
||||
"softCriteria": [
|
||||
"The output is well-formed."
|
||||
],
|
||||
"trials": [
|
||||
{
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge: efficacy new"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge: regression tie"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-hard-gate-fail-t0",
|
||||
"previous": "OUT-previous-hard-gate-fail-t0",
|
||||
"baseline": "OUT-baseline-hard-gate-fail-t0"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 2000,
|
||||
"cacheCreation": 400,
|
||||
"cacheRead": 8000,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 500,
|
||||
"output": 1000,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 4000,
|
||||
"turns": 2
|
||||
},
|
||||
"previous": {
|
||||
"input": 800,
|
||||
"output": 1600,
|
||||
"cacheCreation": 300,
|
||||
"cacheRead": 6000,
|
||||
"turns": 3
|
||||
}
|
||||
},
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge: efficacy new"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge: regression tie"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-hard-gate-fail-t1",
|
||||
"previous": "OUT-previous-hard-gate-fail-t1",
|
||||
"baseline": "OUT-baseline-hard-gate-fail-t1"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 2000,
|
||||
"cacheCreation": 400,
|
||||
"cacheRead": 8000,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 500,
|
||||
"output": 1000,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 4000,
|
||||
"turns": 2
|
||||
},
|
||||
"previous": {
|
||||
"input": 800,
|
||||
"output": 1600,
|
||||
"cacheCreation": 300,
|
||||
"cacheRead": 6000,
|
||||
"turns": 3
|
||||
}
|
||||
},
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge: efficacy new"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge: regression tie"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-hard-gate-fail-t2",
|
||||
"previous": "OUT-previous-hard-gate-fail-t2",
|
||||
"baseline": "OUT-baseline-hard-gate-fail-t2"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 2000,
|
||||
"cacheCreation": 400,
|
||||
"cacheRead": 8000,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 500,
|
||||
"output": 1000,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 4000,
|
||||
"turns": 2
|
||||
},
|
||||
"previous": {
|
||||
"input": 800,
|
||||
"output": 1600,
|
||||
"cacheCreation": 300,
|
||||
"cacheRead": 6000,
|
||||
"turns": 3
|
||||
}
|
||||
},
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge: efficacy new"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge: regression tie"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-hard-gate-fail-t3",
|
||||
"previous": "OUT-previous-hard-gate-fail-t3",
|
||||
"baseline": "OUT-baseline-hard-gate-fail-t3"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 2000,
|
||||
"cacheCreation": 400,
|
||||
"cacheRead": 8000,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 500,
|
||||
"output": 1000,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 4000,
|
||||
"turns": 2
|
||||
},
|
||||
"previous": {
|
||||
"input": 800,
|
||||
"output": 1600,
|
||||
"cacheCreation": 300,
|
||||
"cacheRead": 6000,
|
||||
"turns": 3
|
||||
}
|
||||
},
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge: efficacy new"
|
||||
},
|
||||
"regression": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge: regression tie"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-hard-gate-fail-t4",
|
||||
"previous": "OUT-previous-hard-gate-fail-t4",
|
||||
"baseline": "OUT-baseline-hard-gate-fail-t4"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
"output": 2000,
|
||||
"cacheCreation": 400,
|
||||
"cacheRead": 8000,
|
||||
"turns": 3
|
||||
},
|
||||
"baseline": {
|
||||
"input": 500,
|
||||
"output": 1000,
|
||||
"cacheCreation": 200,
|
||||
"cacheRead": 4000,
|
||||
"turns": 2
|
||||
},
|
||||
"previous": {
|
||||
"input": 800,
|
||||
"output": 1600,
|
||||
"cacheCreation": 300,
|
||||
"cacheRead": 6000,
|
||||
"turns": 3
|
||||
}
|
||||
},
|
||||
"hard": {
|
||||
"ran": true,
|
||||
"pass": true
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"skill": "sample-skill",
|
||||
"skill": "sample-skill-2arm",
|
||||
"generatedAt": "2026-07-24T12:00:00Z",
|
||||
"temperature": 1.0,
|
||||
"trialsPerCase": 5,
|
||||
@@ -35,9 +35,13 @@
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge preferred new"
|
||||
"rationale": "judge: efficacy new"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-clean-pass-t0",
|
||||
"baseline": "OUT-baseline-clean-pass-t0"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
@@ -63,9 +67,13 @@
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge preferred new"
|
||||
"rationale": "judge: efficacy new"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-clean-pass-t1",
|
||||
"baseline": "OUT-baseline-clean-pass-t1"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
@@ -91,9 +99,13 @@
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge preferred new"
|
||||
"rationale": "judge: efficacy new"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-clean-pass-t2",
|
||||
"baseline": "OUT-baseline-clean-pass-t2"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
@@ -119,9 +131,13 @@
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge preferred new"
|
||||
"rationale": "judge: efficacy new"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-clean-pass-t3",
|
||||
"baseline": "OUT-baseline-clean-pass-t3"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
@@ -147,9 +163,13 @@
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge preferred tie"
|
||||
"rationale": "judge: efficacy tie"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-clean-pass-t4",
|
||||
"baseline": "OUT-baseline-clean-pass-t4"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
@@ -184,9 +204,13 @@
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge preferred new"
|
||||
"rationale": "judge: efficacy new"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-regresses-baseline-t0",
|
||||
"baseline": "OUT-baseline-regresses-baseline-t0"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
@@ -208,9 +232,13 @@
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge preferred new"
|
||||
"rationale": "judge: efficacy new"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-regresses-baseline-t1",
|
||||
"baseline": "OUT-baseline-regresses-baseline-t1"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
@@ -232,9 +260,13 @@
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "tie",
|
||||
"rationale": "judge preferred tie"
|
||||
"rationale": "judge: efficacy tie"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-regresses-baseline-t2",
|
||||
"baseline": "OUT-baseline-regresses-baseline-t2"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
@@ -256,9 +288,13 @@
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "baseline",
|
||||
"rationale": "judge preferred baseline"
|
||||
"rationale": "judge: efficacy baseline"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-regresses-baseline-t3",
|
||||
"baseline": "OUT-baseline-regresses-baseline-t3"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
@@ -280,9 +316,13 @@
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "baseline",
|
||||
"rationale": "judge preferred baseline"
|
||||
"rationale": "judge: efficacy baseline"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-regresses-baseline-t4",
|
||||
"baseline": "OUT-baseline-regresses-baseline-t4"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
@@ -313,9 +353,13 @@
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge preferred new"
|
||||
"rationale": "judge: efficacy new"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-hard-gate-fail-t0",
|
||||
"baseline": "OUT-baseline-hard-gate-fail-t0"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
@@ -341,9 +385,13 @@
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge preferred new"
|
||||
"rationale": "judge: efficacy new"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-hard-gate-fail-t1",
|
||||
"baseline": "OUT-baseline-hard-gate-fail-t1"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
@@ -369,9 +417,13 @@
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge preferred new"
|
||||
"rationale": "judge: efficacy new"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-hard-gate-fail-t2",
|
||||
"baseline": "OUT-baseline-hard-gate-fail-t2"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
@@ -397,9 +449,13 @@
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge preferred new"
|
||||
"rationale": "judge: efficacy new"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-hard-gate-fail-t3",
|
||||
"baseline": "OUT-baseline-hard-gate-fail-t3"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
@@ -425,9 +481,13 @@
|
||||
"comparisons": {
|
||||
"efficacy": {
|
||||
"winner": "new",
|
||||
"rationale": "judge preferred new"
|
||||
"rationale": "judge: efficacy new"
|
||||
}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "OUT-new-hard-gate-fail-t4",
|
||||
"baseline": "OUT-baseline-hard-gate-fail-t4"
|
||||
},
|
||||
"usage": {
|
||||
"new": {
|
||||
"input": 1000,
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: benchmark-skill
|
||||
description: Benchmark a skill's efficacy against a no-skill baseline and render an HTML report. Run deliberately as /benchmark-skill <name>, never automatically.
|
||||
description: Benchmark a skill's efficacy against a no-skill baseline and, when a released version exists, its regression against that version, rendering an HTML report. Run deliberately as /benchmark-skill <name>, never automatically.
|
||||
disable-model-invocation: true
|
||||
---
|
||||
|
||||
@@ -8,11 +8,14 @@ disable-model-invocation: true
|
||||
|
||||
Prove that a skill genuinely improves the agent's work rather than reading well and adding nothing.
|
||||
|
||||
Running `/benchmark-skill <name>` executes that skill's authored test cases as a controlled experiment and produces a self-contained HTML report carrying a single **Efficacy verdict**.
|
||||
Each case runs a **new-skill arm** (the working tree) and a **no-skill baseline arm**, several times, and a blind judge decides which arm's output better satisfies the case's author-written expectations.
|
||||
Running `/benchmark-skill <name>` executes that skill's authored test cases as a controlled experiment and produces a self-contained HTML report carrying two independent verdicts: an **Efficacy verdict** (does the skill beat no-skill) and a **Regression verdict** (did my in-progress edit degrade it from the released version).
|
||||
Each case runs a **new-skill arm** (the working tree) and a **no-skill baseline arm**, and — whenever the skill already exists on main and its directory differs from that released state — a third **previous-version arm** materialized from main's `HEAD`.
|
||||
Each arm runs several times, and a blind judge decides which arm's output better satisfies the case's author-written expectations.
|
||||
Two head-to-heads fall out of the arms per trial: **efficacy** pairs the new arm against no-skill, and **regression** pairs it against the previous version.
|
||||
When the skill is brand-new or unchanged from main, there is no previous version to compare against and the run degrades to the two-arm efficacy-only shape, with the Regression verdict reading not-applicable.
|
||||
|
||||
This skill runs **in-session as an AI script**: you orchestrate the arms and judges as subagents via the workflow mechanism, so the whole battery stays on the interactive subscription quota.
|
||||
The mechanical, non-judgment work — summing token usage, collapsing verdicts, applying the pass rule, rendering the report — is done by a committed program, [`core/benchmark_core.py`](core/benchmark_core.py), so the numbers are exact and reproducible rather than re-derived each run.
|
||||
The mechanical, non-judgment work — summing token usage, collapsing verdicts, applying the pass rules, rendering the report — is done by a committed program, [`core/benchmark_core.py`](core/benchmark_core.py), so the numbers are exact and reproducible rather than re-derived each run.
|
||||
You do the judgment work — running arms and judging — that the agent is actually good at.
|
||||
|
||||
The authoring contract for a test case is [`CASE-FORMAT.md`](CASE-FORMAT.md).
|
||||
@@ -41,21 +44,26 @@ Done when every case parses, its fixture exists, and you have the case list in s
|
||||
|
||||
## 2. Fix the arms and per-run settings
|
||||
|
||||
Run **two arms**: a new-skill arm and a no-skill baseline arm.
|
||||
Always run a **new-skill arm** and a **no-skill baseline arm**.
|
||||
Add a third **previous-version arm** exactly when a released version exists to compare against, decided automatically with no flag or argument:
|
||||
|
||||
- Resolve the repo's default branch dynamically — `git symbolic-ref refs/remotes/origin/HEAD` (or `git remote show origin`), never a hardcoded `"main"` literal — and call its tip `HEAD`.
|
||||
- Diff the skill's directory against `HEAD` (`git diff --quiet HEAD -- <skill-dir>`). Add the previous-version arm only when the directory both **exists at `HEAD`** and **differs** from it.
|
||||
- A brand-new skill absent from `HEAD`, or a skill unchanged from it, has no meaningful previous version, so the run stays two-arm efficacy-only.
|
||||
|
||||
Arms run at the session's realistic default temperature (around 1.0), so the result reflects whether the skill *reliably* helps across variance rather than helping once by luck.
|
||||
Per-arm temperature is not settable through the in-session workflow surface, so a skill's temperature-zero override for a genuinely mechanical task is a documented knob for the future headless path, not something to set here.
|
||||
|
||||
Each case runs **5 paired trials**.
|
||||
Trial *i*'s new-skill output is judged against trial *i*'s no-skill output.
|
||||
For each trial, trial *i*'s new-skill output is judged against trial *i*'s no-skill output for efficacy and — on a three-arm run — against trial *i*'s previous-version output for regression.
|
||||
There is no reuse of arms or judgments across arms or runs.
|
||||
|
||||
## 3. Run the arms and judges as subagents
|
||||
|
||||
Use the workflow mechanism (the `Workflow` tool) to fan out the arms and judges.
|
||||
For each case, for each of the 5 trials, run both arms, then judge the pair.
|
||||
For each case, for each of the 5 trials, run every arm, then judge each comparison's pair.
|
||||
|
||||
**Both arms receive the identical `## Prompt`, authored once and arm-agnostically.**
|
||||
**Every arm receives the identical `## Prompt`, authored once and arm-agnostically.**
|
||||
For a conversation-driven case, inject the `## Seed` transcript as the subagent's prior context before the prompt.
|
||||
|
||||
**Give each arm-and-trial a hermetic fixture-only world.**
|
||||
@@ -68,14 +76,20 @@ Each arm-and-trial gets its own fresh world, so writes never leak between them.
|
||||
Materialize the skill into an **isolated temp path outside the world** — a copy of its working-tree directory, so it reflects uncommitted edits and carries none of its repo surroundings.
|
||||
Point the subagent at that copy, tell it to use that skill, and have it read the skill's `SKILL.md` and any assets it references, so the real skill machinery is exercised.
|
||||
Its working directory is the fresh world, and it also receives the prompt.
|
||||
- **Previous-version arm** (three-arm runs only) — force-invoked identically to the new-skill arm, differing only in which skill it points at.
|
||||
Materialize the skill's directory **at the default branch's `HEAD`** into its own isolated temp path (`git --work-tree=<temp> checkout HEAD -- <skill-dir>`, or `git archive HEAD <skill-dir>` piped into the temp path), since each skill is self-contained and needs none of the rest of the repo.
|
||||
Point the subagent at that checkout and otherwise treat it exactly as the new-skill arm — same world, same prompt.
|
||||
- **No-skill baseline arm** — the honest counterfactual of the skill not existing.
|
||||
Materialize no skill for it: give it the bare prompt with the skill absent from its context, and do not mention the skill or hint that one exists.
|
||||
Instruct it to stay within its working directory, since the isolation is soft and the subagent shares the machine.
|
||||
|
||||
The cases are held fixed to the working tree: all arms run against today's prompt, fixture, and expectations, so the skill version is the only variable between the new and previous-version arms.
|
||||
|
||||
Capture each arm's **final message** to a file — this is the `$OUTPUT` the hard-assertion gate reads and the text the judge compares.
|
||||
|
||||
**The judge** — one blind judge subagent per trial.
|
||||
Show it the two arms' final messages as unlabelled **A** and **B**, with the order randomized per trial (record which of A/B is the new arm so you can map the verdict back).
|
||||
**The judge** — one blind judge subagent per trial per comparison, so a three-arm trial draws two judgments (efficacy and regression) and a two-arm trial draws one.
|
||||
For each comparison, show the judge the two arms' final messages as unlabelled **A** and **B**, with the order randomized per trial (record which of A/B is the new arm so you can map the verdict back).
|
||||
The same judge machinery serves both comparisons; only the pair of outputs handed over differs — efficacy pairs new against no-skill, regression pairs new against the previous version.
|
||||
Ground it on the case's `## Soft criteria` rather than letting it free-form its own standard, and instruct it explicitly to **discount mere length and formatting differences** — a skill must not win by being more verbose.
|
||||
Have it return a single winner: A, B, or tie.
|
||||
|
||||
@@ -90,6 +104,9 @@ A case with no `## Hard assertions` block simply has no gate.
|
||||
The deterministic core is a pure transform: it takes the collected data and returns the results model and HTML.
|
||||
Assemble one **run bundle** JSON with this shape and write it under `tests/.reports/.work/<name>-bundle.json`:
|
||||
|
||||
The shape below is the **three-arm** run.
|
||||
For a two-arm run, drop the `previous` arm, drop the `regression` comparison, and drop the `previous` key from each trial's `outputs` and `usage` — the core reads the arm shape off the presence of a regression comparison.
|
||||
|
||||
```json
|
||||
{
|
||||
"skill": "<name>",
|
||||
@@ -98,10 +115,12 @@ Assemble one **run bundle** JSON with this shape and write it under `tests/.repo
|
||||
"trialsPerCase": 5,
|
||||
"arms": [
|
||||
{"id": "new", "label": "New skill"},
|
||||
{"id": "previous", "label": "Previous version"},
|
||||
{"id": "baseline", "label": "No skill"}
|
||||
],
|
||||
"comparisons": [
|
||||
{"id": "efficacy", "label": "Efficacy", "new": "new", "against": "baseline", "rule": "efficacy"}
|
||||
{"id": "efficacy", "label": "Efficacy", "new": "new", "against": "baseline", "rule": "efficacy"},
|
||||
{"id": "regression", "label": "Regression", "new": "new", "against": "previous", "rule": "regression"}
|
||||
],
|
||||
"cases": [
|
||||
{
|
||||
@@ -111,9 +130,18 @@ Assemble one **run bundle** JSON with this shape and write it under `tests/.repo
|
||||
"trials": [
|
||||
{
|
||||
"hard": {"ran": true, "pass": true},
|
||||
"comparisons": {"efficacy": {"winner": "new", "rationale": "<judge's one line>"}},
|
||||
"comparisons": {
|
||||
"efficacy": {"winner": "new", "rationale": "<judge's one line>"},
|
||||
"regression": {"winner": "tie", "rationale": "<judge's one line>"}
|
||||
},
|
||||
"outputs": {
|
||||
"new": "<new arm's final message>",
|
||||
"previous": "<previous-version arm's final message>",
|
||||
"baseline": "<no-skill arm's final message>"
|
||||
},
|
||||
"usage": {
|
||||
"new": {"input": 0, "output": 0, "cacheCreation": 0, "cacheRead": 0, "turns": 0},
|
||||
"previous": {"input": 0, "output": 0, "cacheCreation": 0, "cacheRead": 0, "turns": 0},
|
||||
"baseline": {"input": 0, "output": 0, "cacheCreation": 0, "cacheRead": 0, "turns": 0}
|
||||
}
|
||||
}
|
||||
@@ -123,9 +151,15 @@ Assemble one **run bundle** JSON with this shape and write it under `tests/.repo
|
||||
}
|
||||
```
|
||||
|
||||
- **`winner`** is the arm id (`"new"` or `"baseline"`) or `"tie"`, mapped back from the judge's blind A/B answer.
|
||||
- **`comparisons`** carries one entry per head-to-head, keyed by the comparison id.
|
||||
`regression` is present only on a three-arm run.
|
||||
Each **`winner`** is an arm id (`"new"`, `"baseline"`, or `"previous"`) or `"tie"`, mapped back from that comparison's blind A/B answer.
|
||||
- **`hard`** carries the gate result for that trial's new arm.
|
||||
Omit it or set `ran: false` when the case has no hard assertions.
|
||||
The gate feeds the Efficacy axis only.
|
||||
The Regression axis is judged purely on the head-to-head.
|
||||
- **`outputs`** is each arm's captured final message, keyed by arm id.
|
||||
The core surfaces the losing-trial pair as report evidence, so a failed comparison shows the new arm's output beside the one it lost to.
|
||||
- **`usage`** is the per-arm-per-trial transcript usage.
|
||||
Recover it by reading each arm-subagent's transcript file (the per-agent JSONL the workflow writes) and summing each message's token usage into the four components — `input`, `output`, `cacheCreation` (cache-creation input tokens), `cacheRead` (cache-read input tokens).
|
||||
`turns` is the count of tool-call rounds in that transcript.
|
||||
@@ -142,11 +176,11 @@ python3 skills/benchmark-skill/core/benchmark_core.py \
|
||||
--html tests/.reports/<name>.html
|
||||
```
|
||||
|
||||
The core collapses each trial to a per-arm WIN, TIE, or LOSS, applies the efficacy pass rule, flags any loss for human review, and yields the skill-level **Efficacy verdict**.
|
||||
It renders the self-contained HTML report — the verdict badge, run metadata, the per-arm cost table, and the cases in stable authored order — alongside the machine-readable model.
|
||||
Cost is reported but never gates the verdict.
|
||||
The core collapses each trial of each comparison to a WIN, TIE, or LOSS for the new arm, applies each comparison's pass rule — efficacy passes at wins ≥ 3 and losses ≤ 1, regression passes at losses ≤ 1 with no wins floor — flags any loss for human review, and yields the two skill-level verdicts: **Efficacy** (green when every case beats no-skill) and **Regression** (green when no case degraded, not-applicable on a two-arm run).
|
||||
It renders the self-contained HTML report — the two verdict badges, a headline reading them together, run metadata, the per-arm cost table (a previous-version row on a three-arm run), and the cases in stable authored order, each auto-expanding on any failure or flagged loss to show the losing-trial output pair — alongside the machine-readable model.
|
||||
Cost is reported but never gates a verdict.
|
||||
|
||||
Clean up the scratch when done: remove the `tests/.reports/.work/` directory and every per-arm world and skill materialization you created under the system temp path.
|
||||
Clean up the scratch when done: remove the `tests/.reports/.work/` directory and every per-arm world and skill materialization you created under the system temp path — including the previous-version checkout.
|
||||
|
||||
Done when `tests/.reports/<name>.html` exists.
|
||||
Report the Efficacy verdict, the path to the report, and any flagged losses to the user.
|
||||
Report both verdicts, the path to the report, and any flagged losses or regressions to the user.
|
||||
|
||||
@@ -2,13 +2,14 @@
|
||||
"""Deterministic scoring and rendering for a skill benchmark run.
|
||||
|
||||
A pure transform: given a collected run bundle (per-arm transcript usage,
|
||||
hard-assertion results, and per-trial judge verdicts) it produces the results
|
||||
model and a self-contained HTML report. No LLM, no network, no clock — the
|
||||
`generatedAt` stamp is supplied by the caller so the transform stays pure.
|
||||
hard-assertion results, and the per-trial judge verdicts for each comparison) it
|
||||
produces the results model and a self-contained HTML report. No LLM, no network,
|
||||
no clock — the `generatedAt` stamp is supplied by the caller so the transform
|
||||
stays pure.
|
||||
|
||||
The model is parameterized over the arms and the comparisons, so the same path
|
||||
serves a two-arm efficacy run and a future three-arm run. This slice exercises
|
||||
only the two-arm efficacy shape.
|
||||
The model is parameterized over the arms and the comparisons, so one path serves
|
||||
both the three-arm run (efficacy against no-skill and regression against the
|
||||
previous version) and the degenerate two-arm run with no previous version.
|
||||
|
||||
Usage:
|
||||
benchmark_core.py <bundle.json> [--json <out.json>] [--html <out.html>]
|
||||
@@ -80,9 +81,18 @@ def _collapse_trial(verdict, comparison):
|
||||
return TIE
|
||||
|
||||
|
||||
def _efficacy_pass(wins, losses):
|
||||
"""Efficacy passes when wins >= 3 and losses <= 1 across the 5 trials."""
|
||||
return wins >= 3 and losses <= 1
|
||||
# Each comparison names a pass rule keyed here.
|
||||
# Efficacy hunts for a reliable win, so it needs a wins floor.
|
||||
# Regression hunts for a degradation, so a tie is already a success and only losses matter.
|
||||
PASS_RULES = {
|
||||
"efficacy": lambda wins, losses: wins >= 3 and losses <= 1,
|
||||
"regression": lambda wins, losses: losses <= 1,
|
||||
}
|
||||
|
||||
|
||||
def _comparison_by_rule(comparisons, rule):
|
||||
"""The comparison carrying a given pass rule, or None if absent."""
|
||||
return next((c for c in comparisons if c["rule"] == rule), None)
|
||||
|
||||
|
||||
def _score_case(case, comparisons):
|
||||
@@ -92,21 +102,29 @@ def _score_case(case, comparisons):
|
||||
)
|
||||
|
||||
per_comparison = {}
|
||||
case_passed = not hard_failed
|
||||
for comp in comparisons:
|
||||
strip, flagged = [], []
|
||||
strip, flagged, evidence = [], [], []
|
||||
for i, trial in enumerate(case["trials"]):
|
||||
outcome = _collapse_trial(trial["comparisons"][comp["id"]], comp)
|
||||
strip.append(outcome)
|
||||
if outcome == LOSS:
|
||||
flagged.append(i)
|
||||
# The losing-trial output pair lets the report surface the new
|
||||
# arm's output beside the one it lost to, for a human to eyeball.
|
||||
outputs = trial.get("outputs", {})
|
||||
evidence.append({
|
||||
"trial": i,
|
||||
"new": outputs.get(comp["new"], ""),
|
||||
"against": outputs.get(comp["against"], ""),
|
||||
})
|
||||
wins = strip.count(WIN)
|
||||
ties = strip.count(TIE)
|
||||
losses = strip.count(LOSS)
|
||||
head_to_head = _efficacy_pass(wins, losses)
|
||||
# A failed hard assertion fails the case outright regardless of the
|
||||
# head-to-head.
|
||||
passed = head_to_head and not hard_failed
|
||||
head_to_head = PASS_RULES[comp["rule"]](wins, losses)
|
||||
# The hard gate is new-arm-only and fails efficacy outright.
|
||||
# It catches the current skill emitting malformed output.
|
||||
# Regression is relative to the previous version, so it is judged purely on losses.
|
||||
passed = head_to_head and not (hard_failed and comp["rule"] == "efficacy")
|
||||
per_comparison[comp["id"]] = {
|
||||
"trials": strip,
|
||||
"wins": wins,
|
||||
@@ -114,8 +132,11 @@ def _score_case(case, comparisons):
|
||||
"losses": losses,
|
||||
"passed": passed,
|
||||
"flaggedLosses": flagged,
|
||||
"evidence": evidence,
|
||||
}
|
||||
case_passed = case_passed and passed
|
||||
|
||||
efficacy_id = _comparison_by_rule(comparisons, "efficacy")["id"]
|
||||
regression = _comparison_by_rule(comparisons, "regression")
|
||||
|
||||
return {
|
||||
"name": case["name"],
|
||||
@@ -123,30 +144,58 @@ def _score_case(case, comparisons):
|
||||
"softCriteria": case.get("softCriteria", []),
|
||||
"hardFailed": hard_failed,
|
||||
"comparisons": per_comparison,
|
||||
"efficacyPassed": case_passed,
|
||||
"efficacyPassed": per_comparison[efficacy_id]["passed"],
|
||||
"regressionPassed": (
|
||||
per_comparison[regression["id"]]["passed"] if regression else None
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _net_margin(cases, comparison_id):
|
||||
"""Wins minus losses for a comparison, summed across every case."""
|
||||
total = 0
|
||||
for case in cases:
|
||||
comp = case["comparisons"][comparison_id]
|
||||
total += comp["wins"] - comp["losses"]
|
||||
return total
|
||||
|
||||
|
||||
def build_results(bundle):
|
||||
"""Turn a run bundle into the results model."""
|
||||
pricing = DEFAULT_PRICING
|
||||
comparisons = bundle["comparisons"]
|
||||
arms = bundle["arms"]
|
||||
|
||||
regression = _comparison_by_rule(comparisons, "regression")
|
||||
|
||||
arm_metrics = {a["id"]: _arm_metrics(bundle, a["id"], pricing) for a in arms}
|
||||
cases = [_score_case(c, comparisons) for c in bundle["cases"]]
|
||||
|
||||
efficacy_id = _comparison_by_rule(comparisons, "efficacy")["id"]
|
||||
efficacy_green = all(c["efficacyPassed"] for c in cases)
|
||||
# No previous-version arm means the regression axis has nothing to measure,
|
||||
# so it reads not-applicable rather than green or red.
|
||||
if regression is None:
|
||||
regression_verdict = "not-applicable"
|
||||
regression_net = None
|
||||
else:
|
||||
regression_verdict = (
|
||||
"green" if all(c["regressionPassed"] for c in cases) else "red"
|
||||
)
|
||||
regression_net = _net_margin(cases, regression["id"])
|
||||
|
||||
return {
|
||||
"skill": bundle["skill"],
|
||||
"generatedAt": bundle.get("generatedAt", ""),
|
||||
"temperature": bundle.get("temperature"),
|
||||
"trialsPerCase": bundle.get("trialsPerCase"),
|
||||
"armShape": "two-arm" if len(arms) == 2 else f"{len(arms)}-arm",
|
||||
"armShape": "three-arm" if regression is not None else "two-arm",
|
||||
"arms": arms,
|
||||
"comparisons": comparisons,
|
||||
"efficacyVerdict": "green" if efficacy_green else "red",
|
||||
"regressionVerdict": regression_verdict,
|
||||
"efficacyNetMargin": _net_margin(cases, efficacy_id),
|
||||
"regressionNetMargin": regression_net,
|
||||
"armMetrics": arm_metrics,
|
||||
"cases": cases,
|
||||
}
|
||||
@@ -160,10 +209,13 @@ body { font: 15px/1.5 -apple-system, system-ui, sans-serif; margin: 0; padding:
|
||||
color: #1a1a1a; background: #fafafa; }
|
||||
.wrap { max-width: 60rem; margin: 0 auto; }
|
||||
h1 { font-size: 1.4rem; margin: 0 0 .25rem; }
|
||||
.badges { display: flex; gap: .5rem; margin: .25rem 0; flex-wrap: wrap; }
|
||||
.badge { display: inline-block; padding: .2rem .7rem; border-radius: 999px;
|
||||
font-weight: 600; font-size: .85rem; }
|
||||
.badge.green { background: #d6f5df; color: #0f6b34; }
|
||||
.badge.red { background: #fbdcdc; color: #9b1c1c; }
|
||||
.badge.na { background: #eee; color: #666; }
|
||||
.headline { font-size: 1.02rem; font-weight: 600; margin: .8rem 0 .2rem; }
|
||||
.meta { color: #666; font-size: .85rem; margin: .5rem 0 1.5rem; }
|
||||
table { border-collapse: collapse; width: 100%; margin: .5rem 0; font-size: .9rem; }
|
||||
th, td { text-align: right; padding: .4rem .6rem; border-bottom: 1px solid #e5e5e5; }
|
||||
@@ -173,7 +225,6 @@ thead th { border-bottom: 2px solid #ccc; }
|
||||
.case { border: 1px solid #e5e5e5; border-radius: 8px; padding: 1rem 1.25rem;
|
||||
margin: .75rem 0; background: #fff; }
|
||||
.case.fail { border-color: #f0b6b6; }
|
||||
.case h3 { margin: 0 0 .2rem; font-size: 1rem; }
|
||||
.case .desc { color: #666; font-size: .85rem; margin: 0 0 .6rem; }
|
||||
.strip { display: flex; gap: .3rem; margin: .3rem 0; flex-wrap: wrap; }
|
||||
.cell { width: 2.6rem; text-align: center; padding: .2rem 0; border-radius: 4px;
|
||||
@@ -186,6 +237,22 @@ thead th { border-bottom: 2px solid #ccc; }
|
||||
.result.fail { color: #9b1c1c; }
|
||||
.tag { font-size: .75rem; color: #9b1c1c; font-weight: 600; }
|
||||
.crit { color: #555; font-size: .82rem; margin: .5rem 0 0; padding-left: 1.1rem; }
|
||||
.case > summary { cursor: pointer; font-weight: 600; font-size: 1rem;
|
||||
list-style-position: outside; }
|
||||
.case > summary::marker { color: #999; }
|
||||
.cmps { display: grid; grid-template-columns: repeat(auto-fit, minmax(18rem, 1fr));
|
||||
gap: 1rem; align-items: start; margin: .5rem 0 .2rem; }
|
||||
.cmp { min-width: 0; }
|
||||
.evidence { display: grid; grid-template-columns: repeat(auto-fit, minmax(12rem, 1fr));
|
||||
gap: .6rem; margin: .4rem 0 0; }
|
||||
.evidence figure { margin: 0; }
|
||||
.evidence figcaption { font-size: .72rem; color: #666; font-weight: 600;
|
||||
margin: 0 0 .2rem; }
|
||||
.evidence pre { margin: 0; padding: .5rem .6rem; background: #f6f6f6;
|
||||
border: 1px solid #e5e5e5; border-radius: 4px; font-size: .78rem;
|
||||
white-space: pre-wrap; overflow-wrap: anywhere; }
|
||||
.evidence .lead { grid-column: 1 / -1; font-size: .78rem; color: #9b1c1c;
|
||||
font-weight: 600; margin: .3rem 0 0; }
|
||||
"""
|
||||
|
||||
|
||||
@@ -197,16 +264,54 @@ def _fmt_cost(d):
|
||||
return f"${d:,.4f}"
|
||||
|
||||
|
||||
def _plural(n, noun):
|
||||
return f"{n} {noun}" if n == 1 else f"{n} {noun}s"
|
||||
|
||||
|
||||
def _headline(results):
|
||||
"""One 'so what' sentence reading the two verdicts together.
|
||||
|
||||
Every clause it states is backed by a verdict or a case count rendered on the
|
||||
same screen.
|
||||
"""
|
||||
eff = results["efficacyVerdict"]
|
||||
reg = results["regressionVerdict"]
|
||||
eff_fail = sum(1 for c in results["cases"] if not c["efficacyPassed"])
|
||||
reg_fail = sum(1 for c in results["cases"] if c["regressionPassed"] is False)
|
||||
|
||||
if reg == "not-applicable":
|
||||
if eff == "green":
|
||||
return "This skill earns its keep against no-skill, with no previous version to regress against."
|
||||
return (f"This skill does not beat no-skill on {_plural(eff_fail, 'case')}, "
|
||||
"with no previous version to regress against.")
|
||||
if eff == "green" and reg == "green":
|
||||
return "This skill earns its keep and this edit held quality steady."
|
||||
if eff == "green" and reg == "red":
|
||||
return f"Still valuable, but this edit regressed {_plural(reg_fail, 'case')}."
|
||||
if eff == "red" and reg == "green":
|
||||
return (f"This skill does not beat no-skill on {_plural(eff_fail, 'case')}, "
|
||||
"but this edit did not make it worse.")
|
||||
return (f"This skill does not beat no-skill on {_plural(eff_fail, 'case')} "
|
||||
f"and this edit regressed {_plural(reg_fail, 'case')}.")
|
||||
|
||||
|
||||
def _badge(label, state):
|
||||
text = {"green": "GREEN", "red": "RED", "not-applicable": "N/A"}[state]
|
||||
cls = {"green": "green", "red": "red", "not-applicable": "na"}[state]
|
||||
return f"<span class=\"badge {cls}\">{label}: {text}</span>"
|
||||
|
||||
|
||||
def render_html(results):
|
||||
e = html.escape
|
||||
verdict = results["efficacyVerdict"]
|
||||
arm_labels = {a["id"]: a["label"] for a in results["arms"]}
|
||||
three_arm = results["armShape"] == "three-arm"
|
||||
out = []
|
||||
out.append("<div class=\"wrap\">")
|
||||
out.append(f"<h1>Benchmark — {e(results['skill'])}</h1>")
|
||||
out.append(
|
||||
f"<span class=\"badge {verdict}\">Efficacy: "
|
||||
f"{'GREEN' if verdict == 'green' else 'RED'}</span>"
|
||||
)
|
||||
out.append("<div class=\"badges\">")
|
||||
out.append(_badge("Efficacy", results["efficacyVerdict"]))
|
||||
out.append(_badge("Regression", results["regressionVerdict"]))
|
||||
out.append("</div>")
|
||||
|
||||
meta_bits = [f"arm shape: {e(results['armShape'])}"]
|
||||
if results.get("trialsPerCase") is not None:
|
||||
@@ -217,6 +322,8 @@ def render_html(results):
|
||||
meta_bits.append(e(results["generatedAt"]))
|
||||
out.append(f"<div class=\"meta\">{' · '.join(meta_bits)}</div>")
|
||||
|
||||
out.append(f"<p class=\"headline\">{e(_headline(results))}</p>")
|
||||
|
||||
# Per-arm cost table.
|
||||
out.append("<table><thead><tr><th>Arm</th><th>Turns</th><th>Raw tokens</th>"
|
||||
"<th>Cost-equiv tokens</th><th>Imputed cost</th></tr></thead><tbody>")
|
||||
@@ -229,23 +336,37 @@ def render_html(results):
|
||||
f"<td>{_fmt_cost(m['imputedCost'])}</td></tr>"
|
||||
)
|
||||
out.append("</tbody></table>")
|
||||
out.append(
|
||||
"<p class=\"footnote\">Absolute cost is inflated by shared-context cache "
|
||||
"overhead, so the trustworthy signal is the new-vs-no-skill ratio, not the "
|
||||
"absolute figures.</p>"
|
||||
)
|
||||
if three_arm:
|
||||
footnote = ("Absolute cost is inflated by shared-context cache overhead, "
|
||||
"so the trustworthy signals are two ratios: new-vs-no-skill "
|
||||
"(what the skill costs over nothing) and new-vs-old (what this "
|
||||
"edit added or saved), not the absolute figures.")
|
||||
else:
|
||||
footnote = ("Absolute cost is inflated by shared-context cache overhead, "
|
||||
"so the trustworthy signal is the new-vs-no-skill ratio, not "
|
||||
"the absolute figures.")
|
||||
out.append(f"<p class=\"footnote\">{footnote}</p>")
|
||||
|
||||
# Cases in stable authored order.
|
||||
# Cases in stable authored order, each a collapsible panel that auto-expands
|
||||
# when either comparison fails or carries a flagged loss.
|
||||
for case in results["cases"]:
|
||||
cls = "case fail" if not case["efficacyPassed"] else "case"
|
||||
out.append(f"<div class=\"{cls}\">")
|
||||
out.append(f"<h3>{e(case['name'])}</h3>")
|
||||
comps = case["comparisons"]
|
||||
expanded = case["hardFailed"] or any(
|
||||
c["flaggedLosses"] or not c["passed"] for c in comps.values()
|
||||
)
|
||||
failed = (not case["efficacyPassed"]) or case["regressionPassed"] is False
|
||||
cls = "case fail" if failed else "case"
|
||||
opened = " open" if expanded else ""
|
||||
out.append(f"<details class=\"{cls}\"{opened}>")
|
||||
out.append(f"<summary>{e(case['name'])}</summary>")
|
||||
if case["description"]:
|
||||
out.append(f"<p class=\"desc\">{e(case['description'])}</p>")
|
||||
if case["hardFailed"]:
|
||||
out.append("<p class=\"tag\">Hard assertion failed — case fails outright.</p>")
|
||||
out.append("<div class=\"cmps\">")
|
||||
for comp in results["comparisons"]:
|
||||
c = case["comparisons"][comp["id"]]
|
||||
c = comps[comp["id"]]
|
||||
out.append("<div class=\"cmp\">")
|
||||
out.append(f"<div><strong>{e(comp['label'])}</strong></div>")
|
||||
out.append("<div class=\"strip\">")
|
||||
for outcome in c["trials"]:
|
||||
@@ -258,12 +379,28 @@ def render_html(results):
|
||||
if c["flaggedLosses"]:
|
||||
trials = ", ".join(f"#{i + 1}" for i in c["flaggedLosses"])
|
||||
out.append(f"<div class=\"tag\">Loss flagged for review: trial {trials}</div>")
|
||||
new_label = arm_labels.get(comp["new"], comp["new"])
|
||||
against_label = arm_labels.get(comp["against"], comp["against"])
|
||||
for ev in c["evidence"]:
|
||||
out.append("<div class=\"evidence\">")
|
||||
out.append(f"<p class=\"lead\">Trial #{ev['trial'] + 1} — losing output pair</p>")
|
||||
out.append(
|
||||
f"<figure><figcaption>{e(new_label)}</figcaption>"
|
||||
f"<pre>{e(ev['new'])}</pre></figure>"
|
||||
)
|
||||
out.append(
|
||||
f"<figure><figcaption>{e(against_label)}</figcaption>"
|
||||
f"<pre>{e(ev['against'])}</pre></figure>"
|
||||
)
|
||||
out.append("</div>")
|
||||
out.append("</div>")
|
||||
out.append("</div>")
|
||||
if case["softCriteria"]:
|
||||
out.append("<ul class=\"crit\">")
|
||||
for crit in case["softCriteria"]:
|
||||
out.append(f"<li>{e(crit)}</li>")
|
||||
out.append("</ul>")
|
||||
out.append("</div>")
|
||||
out.append("</details>")
|
||||
|
||||
out.append("</div>")
|
||||
|
||||
|
||||
Reference in New Issue
Block a user