feat: add regression arm and two verdicts (task 0005) #5

alexion · 2026-07-24T15:23:59-04:00

alexion commented

2026-07-24 15:23:59 -04:00

Task file: .claude/tasks/0005-regression-arm-and-two-verdicts.md

Summary

Turns a benchmark run into a three-arm experiment and adds a second Regression verdict beside Efficacy, so an edit that silently made a trusted skill worse is caught rather than passing green.

A previous-version arm is materialized from the default branch's HEAD and added automatically whenever the skill's directory differs from HEAD; a brand-new or unchanged skill degrades to the two-arm efficacy-only shape.
Two blind head-to-heads per trial: efficacy (new-vs-no-skill) and regression (new-vs-old), reusing the same judge and soft criteria — only the output pair differs.
The deterministic core dispatches a pass rule per comparison (efficacy wins≥3 ∧ losses≤1; regression losses≤1, no wins floor), gates only Efficacy on the hard assertion, and yields a skill-level Regression verdict — green when no case regressed, not-applicable on a two-arm run.
The report gains a second badge, a headline reading both verdicts together, a previous-version cost row with a two-ratio footnote (new-vs-no-skill and new-vs-old), and per-case side-by-side comparisons that auto-expand on any failure or flagged loss to show the losing-trial output pair.
The fixture test covers both the three-arm and the degenerate two-arm shapes in one test.

Deviations

Branching: 0005 is blocked by 0008, which stacks on 0004, none of it merged. Rather than lose the benchmark foundation by branching off a bare main, main was fast-forwarded to the task-0004 stack tip (a clean linear superset) and pushed, then task-0005 was branched off it — at the user's direction.
Hard gate feeds Efficacy only, not Regression: the spec's regression pass rule is defined purely on losses, and the gate catches the current skill's malformed output (absolute), whereas regression is relative to the previous version. Pinned by the hard-gate-fail fixture case (Efficacy fails on the gate, Regression passes).
Run bundle gained a per-trial outputs map (arm id → final message) to carry the losing-trial evidence the report shows; 0004 had no output text.
Live three-arm run is the natural follow-up validation (as 0008 did for isolation) but is not gated by this slice's acceptance criteria, so it was not run here. The deterministic half is unit-tested; the orchestration half is specified in SKILL.md prose per the spec's Testing Decisions.

Review

Risk

Overall: MEDIUM

Blast radius: Low — self-contained tooling under skills/benchmark-skill/ and checks/; the core is a pure transform with no external callers.
Reversibility: Low — no migrations or published API; a pure function plus docs, trivial to revert (the one fixture rename aside).
Test coverage: Low — the nix check exercises both the three-arm and two-arm shapes with dense assertions on metrics, verdicts, net margins, evidence surfacing, and report HTML.
Sensitive domain: Low — no auth, payments, permissions, concurrency, or data migration.
Size & complexity: Medium — branchy verdict/headline permutations, per-comparison pass rules, and evidence/HTML rendering; moderate but readable control flow.
Runtime criticality: Low — dev-only, deliberately manual (disable-model-invocation) skill-authoring tooling, off any production path.

Unaddressed findings

None. All Standards findings (sentence-per-line in two comments, a prose semicolon, _headline display-string semicolons, and a repeated comparison-by-rule lookup) and both Spec findings (comparisons stacked instead of side-by-side; cost-table row order) were fixed in the diff.

Task file: `.claude/tasks/0005-regression-arm-and-two-verdicts.md` ## Summary Turns a benchmark run into a three-arm experiment and adds a second **Regression verdict** beside Efficacy, so an edit that silently made a trusted skill worse is caught rather than passing green. - A **previous-version arm** is materialized from the default branch's `HEAD` and added **automatically** whenever the skill's directory differs from `HEAD`; a brand-new or unchanged skill degrades to the two-arm efficacy-only shape. - Two blind head-to-heads per trial: **efficacy** (new-vs-no-skill) and **regression** (new-vs-old), reusing the same judge and soft criteria — only the output pair differs. - The deterministic core dispatches a pass rule per comparison (efficacy `wins≥3 ∧ losses≤1`; regression `losses≤1`, no wins floor), gates **only Efficacy** on the hard assertion, and yields a skill-level **Regression verdict** — green when no case regressed, not-applicable on a two-arm run. - The report gains a second badge, a headline reading both verdicts together, a previous-version cost row with a two-ratio footnote (new-vs-no-skill and new-vs-old), and per-case **side-by-side** comparisons that auto-expand on any failure or flagged loss to show the losing-trial output pair. - The fixture test covers both the three-arm and the degenerate two-arm shapes in one test. ### Deviations - **Branching:** 0005 is blocked by 0008, which stacks on 0004, none of it merged. Rather than lose the benchmark foundation by branching off a bare `main`, `main` was fast-forwarded to the `task-0004` stack tip (a clean linear superset) and pushed, then `task-0005` was branched off it — at the user's direction. - **Hard gate feeds Efficacy only, not Regression:** the spec's regression pass rule is defined purely on losses, and the gate catches the *current* skill's malformed output (absolute), whereas regression is relative to the previous version. Pinned by the `hard-gate-fail` fixture case (Efficacy fails on the gate, Regression passes). - **Run bundle gained a per-trial `outputs` map** (arm id → final message) to carry the losing-trial evidence the report shows; 0004 had no output text. - **Live three-arm run** is the natural follow-up validation (as 0008 did for isolation) but is not gated by this slice's acceptance criteria, so it was not run here. The deterministic half is unit-tested; the orchestration half is specified in `SKILL.md` prose per the spec's Testing Decisions. ## Review ### Risk **Overall: MEDIUM** - Blast radius: Low — self-contained tooling under `skills/benchmark-skill/` and `checks/`; the core is a pure transform with no external callers. - Reversibility: Low — no migrations or published API; a pure function plus docs, trivial to revert (the one fixture rename aside). - Test coverage: Low — the nix check exercises both the three-arm and two-arm shapes with dense assertions on metrics, verdicts, net margins, evidence surfacing, and report HTML. - Sensitive domain: Low — no auth, payments, permissions, concurrency, or data migration. - Size & complexity: Medium — branchy verdict/headline permutations, per-comparison pass rules, and evidence/HTML rendering; moderate but readable control flow. - Runtime criticality: Low — dev-only, deliberately manual (`disable-model-invocation`) skill-authoring tooling, off any production path. ### Unaddressed findings None. All Standards findings (sentence-per-line in two comments, a prose semicolon, `_headline` display-string semicolons, and a repeated comparison-by-rule lookup) and both Spec findings (comparisons stacked instead of side-by-side; cost-table row order) were fixed in the diff.

alexion added 1 commit 2026-07-24 15:23:59 -04:00

feat: add regression arm and two verdicts (task 0005) 9b40d9f4e5

Turn a benchmark run into a three-arm experiment: alongside the new-skill
and no-skill arms, add a previous-version arm materialized from the
default branch's HEAD, drawn automatically whenever the skill's directory
differs from HEAD and degrading to the two-arm efficacy-only shape
otherwise. Two blind head-to-heads now fall out per trial — efficacy
(new-vs-no-skill) and regression (new-vs-old).

The deterministic core dispatches a pass rule per comparison (efficacy at
wins>=3 and losses<=1, regression at losses<=1 with no wins floor),
gates only Efficacy on the hard assertion, and yields a second
skill-level Regression verdict, green when no case regressed and
not-applicable on a two-arm run. The report gains a second badge, a
headline reading both verdicts together, a previous-version cost row with
a two-ratio footnote, and per-case side-by-side comparisons that
auto-expand on any failure or flagged loss to show the losing-trial
output pair.

The fixture test covers both shapes in one test: a three-arm run
asserting per-arm metrics, both per-case and skill-level verdicts, the
net margins, the three-row cost table and losing-trial evidence, and the
two-arm run retained as the degenerate no-previous-version case.

alexion merged commit 9b40d9f4e5 into main

2026-07-24 15:53:51 -04:00

alexion deleted branch task-0005-regression-arm-and-two-verdicts

2026-07-24 15:53:51 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: alexion/skills#5