feat: isolate benchmark arms in fixture-only worlds (task 0008)

Run each benchmark arm in a hermetic fixture-only world materialized
outside the repo, so the no-skill baseline can no longer discover the
skill's assets on disk. The new-skill arm's skill is materialized to an
isolated temp path it is pointed at; the baseline gets no skill and is
told to stay within its world.

A live 5-trial axi-review run confirms the fix: all baseline transcripts
are clean of AXI-PRINCIPLES.md, the baselines invent their own generic
rubric (the honest counterfactual), and the Efficacy verdict is green
5/5 where the contaminated 0004 run had scored red 2/5.

Runner prose only; the deterministic core is untouched.
This commit is contained in:
2026-07-24 14:28:05 -04:00
parent ab2c05acb2
commit 613e423617
2 changed files with 79 additions and 7 deletions

View File

@@ -57,15 +57,21 @@ Use the workflow mechanism (the `Workflow` tool) to fan out the arms and judges.
For each case, for each of the 5 trials, run both arms, then judge the pair.
**Both arms receive the identical `## Prompt`, authored once and arm-agnostically.**
Each arm-and-trial combination runs against a **fresh copy** of the case fixture, made under `tests/.reports/.work/`, so writes never leak between runs.
For a conversation-driven case, inject the `## Seed` transcript as the subagent's prior context before the prompt.
**Give each arm-and-trial a hermetic fixture-only world.**
Make a fresh copy of the case fixture *outside the repo* — under a system temp path such as one from `mktemp -d`, never under `tests/.reports/`.
That copy is the **world**: it is the arm subagent's **working directory** and the `$WORLD` the hard-assertion gate reads, so a `$WORLD/<file>` predicate resolves against the fixture root.
Because the world lives outside the repo, nothing under `skills/` or the grading `tests/` tree sits on any path the arm reaches from there, so the baseline cannot discover the skill's assets and no arm can read its own case's soft criteria or hard assertions.
Each arm-and-trial gets its own fresh world, so writes never leak between them.
- **New-skill arm** — force-invoke the skill.
Point the subagent at the skill's own directory, tell it to use that skill, and have it read the skill's `SKILL.md` and any assets it references, so the real skill machinery is exercised.
Give it the fresh fixture copy as its working target and the prompt.
Materialize the skill into an **isolated temp path outside the world** — a copy of its working-tree directory, so it reflects uncommitted edits and carries none of its repo surroundings.
Point the subagent at that copy, tell it to use that skill, and have it read the skill's `SKILL.md` and any assets it references, so the real skill machinery is exercised.
Its working directory is the fresh world, and it also receives the prompt.
- **No-skill baseline arm** — the honest counterfactual of the skill not existing.
Give it the bare prompt with the skill absent from its context.
Do not mention the skill or hint that one exists.
Materialize no skill for it: give it the bare prompt with the skill absent from its context, and do not mention the skill or hint that one exists.
Instruct it to stay within its working directory, since the isolation is soft and the subagent shares the machine.
Capture each arm's **final message** to a file — this is the `$OUTPUT` the hard-assertion gate reads and the text the judge compares.
@@ -77,7 +83,7 @@ Have it return a single winner: A, B, or tie.
## 4. Run the hard-assertion gate
Run the case's `## Hard assertions` against the **new arm only**, once per trial.
Bind `$OUTPUT` to the path of that trial's new-arm final message and `$WORLD` to the path of that trial's fresh new-arm fixture copy.
Bind `$OUTPUT` to the path of that trial's new-arm final message and `$WORLD` to that trial's new-arm world (its fixture copy, which is the arm's working directory).
Run each predicate.
A non-zero exit fails the assertion.
Record pass/fail per trial — a failed hard assertion fails the case outright regardless of the head-to-head.
@@ -144,7 +150,7 @@ The core collapses each efficacy trial to WIN, TIE, or LOSS for the new arm (a t
It renders a self-contained HTML report: the Efficacy badge and run metadata, a two-row per-arm cost table (footnoted that absolute cost is inflated by shared-context cache overhead, so the trustworthy signal is the new-vs-no-skill ratio), and the cases in stable authored order.
Cost is reported alongside quality but never gates the verdict.
Clean up the `tests/.reports/.work/` scratch directory when done.
Clean up the scratch when done: remove the `tests/.reports/.work/` directory and every per-arm world and skill materialization you created under the system temp path.
Done when `tests/.reports/<name>.html` exists.
Report the Efficacy verdict, the path to the report, and any flagged losses to the user.