feat: add regression arm and two verdicts (task 0005)

Turn a benchmark run into a three-arm experiment: alongside the new-skill
and no-skill arms, add a previous-version arm materialized from the
default branch's HEAD, drawn automatically whenever the skill's directory
differs from HEAD and degrading to the two-arm efficacy-only shape
otherwise. Two blind head-to-heads now fall out per trial — efficacy
(new-vs-no-skill) and regression (new-vs-old).

The deterministic core dispatches a pass rule per comparison (efficacy at
wins>=3 and losses<=1, regression at losses<=1 with no wins floor),
gates only Efficacy on the hard assertion, and yields a second
skill-level Regression verdict, green when no case regressed and
not-applicable on a two-arm run. The report gains a second badge, a
headline reading both verdicts together, a previous-version cost row with
a two-ratio footnote, and per-case side-by-side comparisons that
auto-expand on any failure or flagged loss to show the losing-trial
output pair.

The fixture test covers both shapes in one test: a three-arm run
asserting per-arm metrics, both per-case and skill-level verdicts, the
net margins, the three-row cost table and losing-trial evidence, and the
two-arm run retained as the degenerate no-previous-version case.

skills/benchmark-skill/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: benchmark-skill
-description: Benchmark a skill's efficacy against a no-skill baseline and render an HTML report. Run deliberately as /benchmark-skill <name>, never automatically.
+description: Benchmark a skill's efficacy against a no-skill baseline and, when a released version exists, its regression against that version, rendering an HTML report. Run deliberately as /benchmark-skill <name>, never automatically.
 disable-model-invocation: true
 ---
 
@@ -8,11 +8,14 @@ disable-model-invocation: true
 
 Prove that a skill genuinely improves the agent's work rather than reading well and adding nothing.
 
-Running `/benchmark-skill <name>` executes that skill's authored test cases as a controlled experiment and produces a self-contained HTML report carrying a single **Efficacy verdict**.
-Each case runs a **new-skill arm** (the working tree) and a **no-skill baseline arm**, several times, and a blind judge decides which arm's output better satisfies the case's author-written expectations.
+Running `/benchmark-skill <name>` executes that skill's authored test cases as a controlled experiment and produces a self-contained HTML report carrying two independent verdicts: an **Efficacy verdict** (does the skill beat no-skill) and a **Regression verdict** (did my in-progress edit degrade it from the released version).
+Each case runs a **new-skill arm** (the working tree) and a **no-skill baseline arm**, and — whenever the skill already exists on main and its directory differs from that released state — a third **previous-version arm** materialized from main's `HEAD`.
+Each arm runs several times, and a blind judge decides which arm's output better satisfies the case's author-written expectations.
+Two head-to-heads fall out of the arms per trial: **efficacy** pairs the new arm against no-skill, and **regression** pairs it against the previous version.
+When the skill is brand-new or unchanged from main, there is no previous version to compare against and the run degrades to the two-arm efficacy-only shape, with the Regression verdict reading not-applicable.
 
 This skill runs **in-session as an AI script**: you orchestrate the arms and judges as subagents via the workflow mechanism, so the whole battery stays on the interactive subscription quota.
-The mechanical, non-judgment work — summing token usage, collapsing verdicts, applying the pass rule, rendering the report — is done by a committed program, [`core/benchmark_core.py`](core/benchmark_core.py), so the numbers are exact and reproducible rather than re-derived each run.
+The mechanical, non-judgment work — summing token usage, collapsing verdicts, applying the pass rules, rendering the report — is done by a committed program, [`core/benchmark_core.py`](core/benchmark_core.py), so the numbers are exact and reproducible rather than re-derived each run.
 You do the judgment work — running arms and judging — that the agent is actually good at.
 
 The authoring contract for a test case is [`CASE-FORMAT.md`](CASE-FORMAT.md).
@@ -41,21 +44,26 @@ Done when every case parses, its fixture exists, and you have the case list in s
 
 ## 2. Fix the arms and per-run settings
 
-Run **two arms**: a new-skill arm and a no-skill baseline arm.
+Always run a **new-skill arm** and a **no-skill baseline arm**.
+Add a third **previous-version arm** exactly when a released version exists to compare against, decided automatically with no flag or argument:
+
+- Resolve the repo's default branch dynamically — `git symbolic-ref refs/remotes/origin/HEAD` (or `git remote show origin`), never a hardcoded `"main"` literal — and call its tip `HEAD`.
+- Diff the skill's directory against `HEAD` (`git diff --quiet HEAD -- <skill-dir>`). Add the previous-version arm only when the directory both **exists at `HEAD`** and **differs** from it.
+- A brand-new skill absent from `HEAD`, or a skill unchanged from it, has no meaningful previous version, so the run stays two-arm efficacy-only.
 
 Arms run at the session's realistic default temperature (around 1.0), so the result reflects whether the skill *reliably* helps across variance rather than helping once by luck.
 Per-arm temperature is not settable through the in-session workflow surface, so a skill's temperature-zero override for a genuinely mechanical task is a documented knob for the future headless path, not something to set here.
 
 Each case runs **5 paired trials**.
-Trial *i*'s new-skill output is judged against trial *i*'s no-skill output.
+For each trial, trial *i*'s new-skill output is judged against trial *i*'s no-skill output for efficacy and — on a three-arm run — against trial *i*'s previous-version output for regression.
 There is no reuse of arms or judgments across arms or runs.
 
 ## 3. Run the arms and judges as subagents
 
 Use the workflow mechanism (the `Workflow` tool) to fan out the arms and judges.
-For each case, for each of the 5 trials, run both arms, then judge the pair.
+For each case, for each of the 5 trials, run every arm, then judge each comparison's pair.
 
-**Both arms receive the identical `## Prompt`, authored once and arm-agnostically.**
+**Every arm receives the identical `## Prompt`, authored once and arm-agnostically.**
 For a conversation-driven case, inject the `## Seed` transcript as the subagent's prior context before the prompt.
 
 **Give each arm-and-trial a hermetic fixture-only world.**
@@ -68,14 +76,20 @@ Each arm-and-trial gets its own fresh world, so writes never leak between them.
   Materialize the skill into an **isolated temp path outside the world** — a copy of its working-tree directory, so it reflects uncommitted edits and carries none of its repo surroundings.
   Point the subagent at that copy, tell it to use that skill, and have it read the skill's `SKILL.md` and any assets it references, so the real skill machinery is exercised.
   Its working directory is the fresh world, and it also receives the prompt.
+- **Previous-version arm** (three-arm runs only) — force-invoked identically to the new-skill arm, differing only in which skill it points at.
+  Materialize the skill's directory **at the default branch's `HEAD`** into its own isolated temp path (`git --work-tree=<temp> checkout HEAD -- <skill-dir>`, or `git archive HEAD <skill-dir>` piped into the temp path), since each skill is self-contained and needs none of the rest of the repo.
+  Point the subagent at that checkout and otherwise treat it exactly as the new-skill arm — same world, same prompt.
 - **No-skill baseline arm** — the honest counterfactual of the skill not existing.
   Materialize no skill for it: give it the bare prompt with the skill absent from its context, and do not mention the skill or hint that one exists.
   Instruct it to stay within its working directory, since the isolation is soft and the subagent shares the machine.
 
+The cases are held fixed to the working tree: all arms run against today's prompt, fixture, and expectations, so the skill version is the only variable between the new and previous-version arms.
+
 Capture each arm's **final message** to a file — this is the `$OUTPUT` the hard-assertion gate reads and the text the judge compares.
 
-**The judge** — one blind judge subagent per trial.
-Show it the two arms' final messages as unlabelled **A** and **B**, with the order randomized per trial (record which of A/B is the new arm so you can map the verdict back).
+**The judge** — one blind judge subagent per trial per comparison, so a three-arm trial draws two judgments (efficacy and regression) and a two-arm trial draws one.
+For each comparison, show the judge the two arms' final messages as unlabelled **A** and **B**, with the order randomized per trial (record which of A/B is the new arm so you can map the verdict back).
+The same judge machinery serves both comparisons; only the pair of outputs handed over differs — efficacy pairs new against no-skill, regression pairs new against the previous version.
 Ground it on the case's `## Soft criteria` rather than letting it free-form its own standard, and instruct it explicitly to **discount mere length and formatting differences** — a skill must not win by being more verbose.
 Have it return a single winner: A, B, or tie.
 
@@ -90,6 +104,9 @@ A case with no `## Hard assertions` block simply has no gate.
 The deterministic core is a pure transform: it takes the collected data and returns the results model and HTML.
 Assemble one **run bundle** JSON with this shape and write it under `tests/.reports/.work/<name>-bundle.json`:
 
+The shape below is the **three-arm** run.
+For a two-arm run, drop the `previous` arm, drop the `regression` comparison, and drop the `previous` key from each trial's `outputs` and `usage` — the core reads the arm shape off the presence of a regression comparison.
+
 ```json
 {
   "skill": "<name>",
@@ -98,10 +115,12 @@ Assemble one **run bundle** JSON with this shape and write it under `tests/.repo
   "trialsPerCase": 5,
   "arms": [
     {"id": "new", "label": "New skill"},
+    {"id": "previous", "label": "Previous version"},
     {"id": "baseline", "label": "No skill"}
   ],
   "comparisons": [
-    {"id": "efficacy", "label": "Efficacy", "new": "new", "against": "baseline", "rule": "efficacy"}
+    {"id": "efficacy",   "label": "Efficacy",   "new": "new", "against": "baseline", "rule": "efficacy"},
+    {"id": "regression", "label": "Regression", "new": "new", "against": "previous", "rule": "regression"}
   ],
   "cases": [
     {
@@ -111,9 +130,18 @@ Assemble one **run bundle** JSON with this shape and write it under `tests/.repo
       "trials": [
         {
           "hard": {"ran": true, "pass": true},
-          "comparisons": {"efficacy": {"winner": "new", "rationale": "<judge's one line>"}},
+          "comparisons": {
+            "efficacy":   {"winner": "new", "rationale": "<judge's one line>"},
+            "regression": {"winner": "tie", "rationale": "<judge's one line>"}
+          },
+          "outputs": {
+            "new":      "<new arm's final message>",
+            "previous": "<previous-version arm's final message>",
+            "baseline": "<no-skill arm's final message>"
+          },
           "usage": {
             "new":      {"input": 0, "output": 0, "cacheCreation": 0, "cacheRead": 0, "turns": 0},
+            "previous": {"input": 0, "output": 0, "cacheCreation": 0, "cacheRead": 0, "turns": 0},
             "baseline": {"input": 0, "output": 0, "cacheCreation": 0, "cacheRead": 0, "turns": 0}
           }
         }
@@ -123,9 +151,15 @@ Assemble one **run bundle** JSON with this shape and write it under `tests/.repo
 }
 ```
 
-- **`winner`** is the arm id (`"new"` or `"baseline"`) or `"tie"`, mapped back from the judge's blind A/B answer.
+- **`comparisons`** carries one entry per head-to-head, keyed by the comparison id.
+  `regression` is present only on a three-arm run.
+  Each **`winner`** is an arm id (`"new"`, `"baseline"`, or `"previous"`) or `"tie"`, mapped back from that comparison's blind A/B answer.
 - **`hard`** carries the gate result for that trial's new arm.
   Omit it or set `ran: false` when the case has no hard assertions.
+  The gate feeds the Efficacy axis only.
+  The Regression axis is judged purely on the head-to-head.
+- **`outputs`** is each arm's captured final message, keyed by arm id.
+  The core surfaces the losing-trial pair as report evidence, so a failed comparison shows the new arm's output beside the one it lost to.
 - **`usage`** is the per-arm-per-trial transcript usage.
   Recover it by reading each arm-subagent's transcript file (the per-agent JSONL the workflow writes) and summing each message's token usage into the four components — `input`, `output`, `cacheCreation` (cache-creation input tokens), `cacheRead` (cache-read input tokens).
   `turns` is the count of tool-call rounds in that transcript.
@@ -142,11 +176,11 @@ python3 skills/benchmark-skill/core/benchmark_core.py \
   --html tests/.reports/<name>.html
 ```
 
-The core collapses each trial to a per-arm WIN, TIE, or LOSS, applies the efficacy pass rule, flags any loss for human review, and yields the skill-level **Efficacy verdict**.
-It renders the self-contained HTML report — the verdict badge, run metadata, the per-arm cost table, and the cases in stable authored order — alongside the machine-readable model.
-Cost is reported but never gates the verdict.
+The core collapses each trial of each comparison to a WIN, TIE, or LOSS for the new arm, applies each comparison's pass rule — efficacy passes at wins ≥ 3 and losses ≤ 1, regression passes at losses ≤ 1 with no wins floor — flags any loss for human review, and yields the two skill-level verdicts: **Efficacy** (green when every case beats no-skill) and **Regression** (green when no case degraded, not-applicable on a two-arm run).
+It renders the self-contained HTML report — the two verdict badges, a headline reading them together, run metadata, the per-arm cost table (a previous-version row on a three-arm run), and the cases in stable authored order, each auto-expanding on any failure or flagged loss to show the losing-trial output pair — alongside the machine-readable model.
+Cost is reported but never gates a verdict.
 
-Clean up the scratch when done: remove the `tests/.reports/.work/` directory and every per-arm world and skill materialization you created under the system temp path.
+Clean up the scratch when done: remove the `tests/.reports/.work/` directory and every per-arm world and skill materialization you created under the system temp path — including the previous-version checkout.
 
 Done when `tests/.reports/<name>.html` exists.
-Report the Efficacy verdict, the path to the report, and any flagged losses to the user.
+Report both verdicts, the path to the report, and any flagged losses or regressions to the user.