skills/.claude/tasks/0005-regression-arm-and-two-verdicts.md

---
spec: skill-benchmarking
blocked-by: 0008-arm-isolation-hardening
---

## What to build

The second head-to-head: a **previous-version arm** that turns a run into a three-arm experiment and adds a **Regression verdict** beside the Efficacy one, so an edit that silently made a trusted skill worse is caught rather than passing green.

The previous version is always main's `HEAD` — the repo's default branch, resolved rather than hardcoded to the literal name "main" — and the new version is the working tree.
Anchoring to the released state on main makes the regression question always "did my in-progress edit degrade from the trusted version," which is the moment of real risk.
The third arm is added **fully automatically and carries no new invocation surface**: the runner diffs the skill's directory against main's `HEAD` and adds the previous-version arm exactly when the two differ.
A brand-new skill absent from main, or a skill unchanged from `HEAD`, has no meaningful previous version, so the run degrades to the two-arm efficacy-only shape.

The previous-version arm's skill is materialized by checking out just the skill's directory at main's `HEAD` into a temp path, since each skill is a self-contained derivation that does not need the rest of the repo.
That temp materialization is cleaned up after the run, like the fresh fixture copies.
The previous-version arm is force-invoked identically to the new arm, pointed at this temp checkout instead of the new arm's copy of the working-tree directory, so the skill version is the only difference between them.
Cases are held fixed to the working tree — all arms run against today's prompt, fixture, and expectations — so the skill version is the only variable differing between the new and previous-version arms.
The hard-assertion gate stays on the new arm only, and the blind judge and the case's soft criteria are reused unchanged for the regression comparison; only the pair of outputs handed to the judge differs.

Two blind head-to-heads are now drawn per trial: **efficacy** pairs the new arm against the no-skill arm, and **regression** pairs the new arm against the previous-version arm.
Trial *i*'s new-skill output is judged against trial *i*'s no-skill output for efficacy and against trial *i*'s previous-version output for regression.

The regression pass rule inverts, because holding quality steady is the goal.
Per trial, the new-vs-old head-to-head collapses to WIN, TIE, or LOSS for the new arm, where a TIE means "as good as the previous version" (a success), a LOSS is the regression being hunted, and a WIN is a bonus improvement.
Per case, regression passes when losses ≤ 1 across the 5 trials, with no wins floor — ties and wins both count as non-regressions — and any loss is flagged for human review.
Dropping the wins floor means a deliberate no-op edit that ties the previous version passes rather than failing merely for tying.

The deterministic core is exercised across its full three-arm, two-comparison path (the same parameterized program from 0004, now run with both comparisons rather than re-architected).
It gains the regression pass rule and a skill-level **Regression verdict**, green when no case regressed.
The per-run result now carries the regression net-margin and a two-arm/three-arm flag, but persisting those into the durable per-run history line is deferred to the trend-history slice (0006), which owns history persistence; this slice's report remains latest-only with no ribbons.
The two verdicts are reported side by side as independent badges, because their four crossings carry genuinely different meanings, and the report headline reads both badges together into one "so what."
On a two-arm run with no previous version the Regression verdict reads not-applicable.
The cost table gains a **previous-version row** (turns, raw tokens, cost-equivalent tokens, imputed cost), which drops out on a two-arm run, and the footnote names the two trustworthy ratios: new-vs-no-skill and new-vs-old.
Each case panel shows both comparisons side by side, each with its per-trial WIN/TIE/LOSS strip and its pass result, and auto-expands when either comparison fails or carries a flagged loss — so a regression opens the panel even when efficacy is green — surfacing the losing-trial output pair for whichever comparison failed.

The core's fixture unit test is extended so the same test covers both shapes: the three-arm run (sample transcripts for all three arms and per-trial judge verdicts for both comparisons) asserting the per-arm metrics, both per-case and skill-level verdicts, and a report with the three-row cost table, and the two-arm run retained as the degenerate "no previous version" case.

## Acceptance criteria

- [x] The previous version resolves to the default branch's `HEAD` dynamically, never a hardcoded "main" literal, and the new version is the working tree.
- [x] The third arm is added automatically, with no new flag or argument, exactly when the skill directory differs from `HEAD`; otherwise the run stays two-arm efficacy-only.
- [x] The previous-version arm materializes by checking out just the skill's directory at `HEAD` into a temp path, force-invoked against it identically to the new arm, and the temp path is cleaned up after the run.
- [x] All arms run against the working-tree case (prompt, fixture, expectations), so the skill version is the only variable between the new and previous-version arms.
- [x] The hard-assertion gate remains new-arm-only; the same blind judge and soft criteria are reused for the regression comparison, differing only in the output pair.
- [x] Per trial, efficacy is judged new-vs-no-skill and regression new-vs-previous, using that trial's own outputs with no cross-arm reuse.
- [x] The regression pass rule passes a case when losses ≤ 1 across 5 trials with no wins floor (ties and wins are non-regressions), flagging any loss.
- [x] The core, unchanged in structure, runs the full three-arm two-comparison path and yields a skill-level Regression verdict green when no case regressed.
- [x] The report shows Efficacy and Regression as two independent badges with a headline reading both together; Regression reads not-applicable on a two-arm run.
- [x] The cost table gains a previous-version row that drops out on a two-arm run, with a footnote naming the new-vs-no-skill and new-vs-old ratios.
- [x] Each case panel shows both comparisons side by side with per-trial WIN/TIE/LOSS strips, auto-expanding on any fail or flagged loss and showing the losing-trial output pair for the failed comparison.
- [x] The core's fixture unit test covers both the three-arm and the degenerate two-arm shapes in the same test, asserting metrics, both verdicts, and the three-row cost table.

## Implementation Notes

The deterministic half — the pass rules, the two verdicts, the net margins, and the report — lives in `core/benchmark_core.py` and is fully covered by the extended `checks/benchmark-core.nix` fixture test.
The orchestration half — resolving `HEAD`, diff-gating the third arm, materializing the previous-version checkout, and drawing the two head-to-heads (criteria 1–6) — is specified in the runner's `SKILL.md` prose, since that half runs in-session as an AI script and, per the spec's Testing Decisions, is not unit-tested but established by running the harness.

- **Branching deviated from the standard `/implement` flow, at the user's direction.**
  Task 0005 is blocked by 0008, which stacks on 0004, and none of that had been merged to `main` — so branching 0005 off a fresh `main` would have lost the entire benchmark foundation.
  `main` was fast-forwarded to the `task-0004` stack tip (a clean linear superset) and pushed, then `task-0005` was branched off it.
- **The hard-assertion gate feeds Efficacy only, not Regression.**
  The spec defines the regression pass rule purely as losses ≤ 1 and never mentions the gate, and the gate exists to catch the *current* skill emitting malformed output — an absolute property of the new arm, whereas regression is relative to the previous version.
  So a hard failure fails the case's Efficacy axis outright while its Regression axis is judged purely on the head-to-head.
  The three-arm fixture's `hard-gate-fail` case pins this: Efficacy fails on the gate, Regression passes.
- **The run bundle gained a per-trial `outputs` map (arm id → final message).**
  0004 carried no output text; the losing-trial evidence pair this slice requires needs it, so the core reads `outputs` to surface the new arm's output beside the one it lost to, only for flagged-loss trials of a failed or flagged comparison.
- **Arm order is canonical: new-skill, previous-version, no-skill.**
  The cost table renders arms in bundle order, and a Spec-axis review caught that the initial new/no-skill/previous order put the previous-version row last rather than in the spec's stated middle position.
  The fixtures and the `SKILL.md` example now emit arms in the spec order.
- **Comparisons render side by side via a CSS grid.**
  A Spec-axis review caught that the two comparisons were stacked vertically rather than laid out side by side as the report shape requires; a `.cmps` grid container now places them beside each other, wrapping to one column on a narrow viewport.
- **Arm shape is derived from the presence of a regression comparison, not the arm count**, so a run is two-arm exactly when no regression comparison was drawn.
- **The two committed fixtures replace 0004's single `run-bundle.json`.**
  `three-arm-bundle.json` is the primary shape and `two-arm-bundle.json` is retained as the degenerate "no previous version" case, both exercised by the one test.
  They are machine-generated for accuracy; the generator is not committed.
- **A live three-arm `/benchmark-skill` run is the natural follow-up validation** (as 0008 did for isolation) but is not gated by this slice's acceptance criteria, so it was not run here.