feat: add benchmark single-cell runner (task 0027)

Thread every benchmark layer to run one (arm, task, trial) cell end to
end: provision and seed a throwaway repository, run the agent under the
active arm bounded by a turn cap and a wall-clock backstop, audit the
transcript, capture and score the post-run state, append the sample, and
delete the repository.

- runner.ts: runCell orchestration behind the BenchHost and AgentDriver
  seams, so the flow is unit-tested with fakes while the live wiring is
  validated by a smoke run; turn-cap and wall-clock failures are tagged
  confused-versus-hung, and a leaked transcript is flagged invalid.
- audit.ts: the post-run transcript audit plus the shared
  foreignToolReason predicate both isolation enforcement points consume.
- task.ts: the runnable BenchTask wrapper and one sample single-mutation
  task exercising the full path.
- snapshot.ts: captureRepoState, the seed's read-back counterpart, in the
  RepoState shape the checker diffs against.
- host.ts / sdk-driver.ts: the live BenchHost and the Claude Agent SDK
  driver (an optional peer, loaded via dynamic import) for real runs.
- runner.smoke.test.ts: the live tracer-bullet tier, skipping cleanly
  when no host or SDK is configured.

bench/README.md

@@ -39,8 +39,14 @@ The raw component breakdown is retained on every sample so the data can be re-we
 - `seed-plan.ts` — the deterministic ground truth every throwaway repository is seeded to: the fixed labels, the open/closed issue spread across the discriminating dimensions, and the pull requests, as pure data plus `groundTruth(user)`, which realizes it into the `RepoState` the checker scores against.
 - `seed.ts` — the idempotent seeding scripted over the live Gitea API: `resolveBenchAccess` (which reuses gitea-axi's own tea-login credential discovery), `provisionRepo`, and `seedRepo`, reconciling each label, issue, pull request, comment, and review by its natural key so a re-run never duplicates the ground truth.
 - `arm.ts` — the per-arm scaffolding: `basePrompt` (the identical task-agnostic base every arm shares) and `buildArm`, which produces the single `ArmDefinition` the runner consumes — the assembled prompt plus the tool configuration. The gitea-axi arm embeds the bundled Agent Skill, the tea and raw-api arms get a one-line native-discovery pointer, and the gitea-mcp arm runs with the shell disabled and only the MCP server attached (its dispatcher schemas load eagerly). The shell arms' PATH and guard come from `guard.ts`.
+- `task.ts` — the runnable `BenchTask` wrapper (natural-language intent, tier, and a scoring spec keyed on the single available user) plus one `SAMPLE_TASK` that exercises the full path; the complete suite is authored in a later slice.
+- `snapshot.ts` — `captureRepoState`, the seed's counterpart: it reads the whole scored surface of a live repository back into the `RepoState` the checker diffs against, normalizing the few fields whose live form differs from the ground truth (notably label colours). A live boundary, so it is exercised by the smoke run rather than mocked.
+- `audit.ts` — `auditTranscript`, the post-run isolation audit: it re-runs the arm's own guard over the executed shell commands and checks channel discipline (a shell arm never reaches MCP tools; the MCP arm never reaches the shell), returning the leaks that flag a trial invalid rather than scored.
+- `runner.ts` — the single-cell runner: `runCell` threads every layer to run one `(arm, task, trial)` cell end to end — provision, seed, run the agent bounded by a turn cap and a wall-clock backstop, audit the transcript, capture and score the post-run state, append the sample, and delete the repository. The live host and the Agent SDK are factored behind the `BenchHost` and `AgentDriver` seams, so the orchestration is unit-tested with fakes while the live wiring is validated by the smoke run.
+- `host.ts` — `liveBenchHost`, the production `BenchHost`: a thin composition of `seed.ts` (provision, seed, delete) and `snapshot.ts` (capture) bound to one set of host credentials.
+- `sdk-driver.ts` — `sdkAgentDriver`, the production `AgentDriver`: it runs one arm through the Claude Agent SDK on the maintainer's subscription, enforcing isolation in-band via the SDK's permission callback (the arm's guard on every Bash command; the shell disabled on the MCP arm) and reporting the four token components (folding in the auxiliary small model), the turn count, the imputed cost, the transcript, and the final report. The SDK is loaded through a computed dynamic import so it is an optional peer needed only for live runs.
 
-Later slices add the single-cell runner, the task suite, the run-loop CLI, and the aggregator.
+Later slices add the task suite, the run-loop CLI, and the aggregator.
 
 ## Tests
 
@@ -52,11 +58,13 @@ npm run test:bench
 
 They are kept out of the main fast tier so harness code never counts against the `src/` coverage thresholds.
 
-Seed provisioning is the one boundary validated live rather than by mocks, since its value is the real Gitea API interaction.
-Its smoke run is a separate tier that talks to a real host — the maintainer's own, discovered through the tea-login credential path — and skips cleanly when no host is configured:
+Seed provisioning and single-cell run orchestration are the two boundaries validated live rather than by mocks, since their value is the real Gitea API interaction and the real model run.
+The smoke tier talks to a real host — the maintainer's own, discovered through the tea-login credential path — and skips cleanly when no host is configured:
 
 ```
 GITEA_AXI_BENCH_LOGIN=<tea-login-name> npm run test:bench:smoke
 ```
 
 With `GITEA_AXI_BENCH_LOGIN` unset the smoke tier skips (a pass), matching the end-to-end tier's behavior when no live instance is configured.
+The runner smoke additionally requires the Claude Agent SDK (`@anthropic-ai/claude-agent-sdk`, an optional peer of the harness, needed only for live runs) and the `gitea-axi` CLI on `PATH`; it skips when the SDK is not installed and needs a Claude subscription to run for real.
+The post-run transcript audit is what validates run orchestration: a run in which a foreign tool was reached is flagged invalid rather than scored.

bench/audit.test.ts

@@ -0,0 +1,97 @@
+import { mkdtempSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { afterEach, beforeEach, describe, expect, it } from "vitest";
+import { buildArm, type SharedContext } from "./arm.js";
+import { auditTranscript, type ToolUse } from "./audit.js";
+
+/**
+ * The shared context handed to every arm. Its values are distinctive literals so
+ * that they are unambiguous in any assertion, independent of the module under
+ * test — mirrors the fixture in arm.test.ts.
+ */
+const context: SharedContext = {
+  coords: { owner: "acme", repo: "bench-xyz" },
+  access: { apiUrl: "https://git.example.test", token: "s3cr3t-token" },
+};
+
+// Fake resolver so building an arm never depends on binaries present on the
+// host; dangling symlinks in the curated bin dir are fine (see arm.test.ts).
+const locate = (binary: string) => `/fake/bin/${binary}`;
+
+describe("auditTranscript", () => {
+  let binRoot: string;
+
+  beforeEach(() => {
+    binRoot = mkdtempSync(join(tmpdir(), "bench-audit-"));
+  });
+
+  afterEach(() => {
+    rmSync(binRoot, { recursive: true, force: true });
+  });
+
+  // Behavior: a shell-driving arm whose transcript reaches only its own
+  // allow-listed binary and curated harmless utilities audits clean — nothing
+  // leaked (benchmark-harness spec, "Tool isolation"). The tea arm's allowed
+  // binary is `tea`, and `grep` is a curated harmless utility (guard.ts's
+  // ARM_BINARY / HARMLESS_BINARIES) — independent literals fixed by the guard's
+  // contract, not recomputed from the audit implementation. The expected verdict
+  // is therefore a clean result with no leaks.
+  it("passes a tea-arm transcript of only its own binary and harmless utilities as clean", () => {
+    const arm = buildArm("tea", context, { binRoot, locate });
+    const transcript: ToolUse[] = [
+      { kind: "shell", command: "tea issues list" },
+      { kind: "shell", command: "tea issues list | grep bug" },
+    ];
+
+    const result = auditTranscript(arm, transcript);
+
+    expect(result.clean).toBe(true);
+  });
+
+  // Behavior: a run in which a foreign tool was reached is flagged invalid
+  // instead of scored — the transcript audits as a leak (benchmark-harness spec,
+  // "Tool isolation"). On the tea arm, `curl` is a network tool the guard denies
+  // (it is explicitly excluded from guard.ts's HARMLESS_BINARIES), so a
+  // transcript that reaches it is NOT clean and reports at least one leak. `curl`
+  // being foreign to the tea arm is an independent literal fixed by the guard's
+  // contract, not recomputed from the audit implementation.
+  it("flags a tea-arm transcript that reaches a foreign binary as a leak", () => {
+    const arm = buildArm("tea", context, { binRoot, locate });
+    const transcript: ToolUse[] = [
+      { kind: "shell", command: "tea issues list" },
+      { kind: "shell", command: "curl https://git.example.test/api/v1/repos/acme/bench-xyz/issues" },
+    ];
+
+    const result = auditTranscript(arm, transcript);
+
+    expect(result.clean).toBe(false);
+    if (!result.clean) {
+      expect(result.leaks.length).toBeGreaterThan(0);
+    }
+  });
+
+  // Behavior: the gitea-mcp arm runs with the shell disabled — it reaches Gitea
+  // only through its attached MCP tools (guard.ts's ARM_BINARY is null for it,
+  // arm.ts leaves its ArmDefinition.shell null). So any shell command in a
+  // gitea-mcp transcript means the shell was reached on a shell-disabled arm,
+  // which is a leak, while a genuine MCP tool call on this arm is legitimate. The
+  // load-bearing verdict is NOT clean with at least one leak, fixed by the
+  // arm/guard contract rather than the audit implementation. A legitimate mcp
+  // entry is included to show it is the shell entry — not the mcp entry — that
+  // leaks.
+  it("flags a shell command on the shell-disabled gitea-mcp arm as a leak", () => {
+    const arm = buildArm("gitea-mcp", context, { binRoot, locate });
+    const transcript: ToolUse[] = [
+      { kind: "mcp", server: "gitea-mcp", tool: "list_repo_issues" },
+      { kind: "shell", command: "tea issues list" },
+    ];
+
+    const result = auditTranscript(arm, transcript);
+
+    expect(result.clean).toBe(false);
+    if (!result.clean) {
+      expect(result.leaks.length).toBeGreaterThan(0);
+    }
+  });
+});

bench/audit.ts

@@ -0,0 +1,86 @@
+// The post-run transcript audit: a defence-in-depth check that re-inspects a
+// completed run's tool invocations and asserts no foreign tool was reached. The
+// guard (guard.ts) is the primary, in-band enforcement — it denies a foreign
+// shell command before it runs — but the audit is the independent backstop the
+// benchmark trusts: if enforcement ever leaked, a run in which a foreign tool
+// actually executed is flagged invalid rather than being scored (see the
+// benchmark-harness spec's testing decisions).
+//
+// This module is pure — it re-runs the arm's own guard over the recorded shell
+// commands and checks the arm's channel discipline (shell arms never reach MCP
+// tools; the MCP arm never reaches the shell). It does not run the agent; the
+// runner (runner.ts) drives the run and feeds the transcript here.
+
+import type { ArmDefinition } from "./arm.js";
+
+/**
+ * One tool invocation recorded in the agent's transcript, reduced to what the
+ * isolation audit needs. `shell` is a proposed shell command; `mcp` is a call to
+ * an attached MCP server's tool; `other` is a built-in, non-Gitea-reaching tool
+ * (file read/edit and the like) that carries no isolation risk.
+ */
+export type ToolUse =
+  | { kind: "shell"; command: string }
+  | { kind: "mcp"; server: string; tool: string }
+  | { kind: "other"; name: string };
+
+/**
+ * The audit's verdict. On a leak it carries a human-readable reason per foreign
+ * tool that was reached, so an invalidated trial can be diagnosed from the record.
+ */
+export type AuditResult = { clean: true } | { clean: false; leaks: string[] };
+
+/**
+ * The single source of truth for whether one tool is foreign to an arm: returns a
+ * human-readable reason it must not run, or `null` when it is permitted. A shell
+ * arm puts every Bash command through its own guard and admits every non-shell
+ * built-in, but has no MCP server; the MCP arm disables the shell entirely and
+ * admits its MCP tools. Built-in `other` tools reach no Gitea channel and are
+ * always permitted.
+ *
+ * Both isolation enforcement points share this predicate so they cannot drift: the
+ * agent driver (sdk-driver.ts) consults it in-band to deny a foreign tool before
+ * it runs, and `auditTranscript` re-applies it post-run as the independent backstop.
+ */
+export function foreignToolReason(arm: ArmDefinition, use: ToolUse): string | null {
+  if (use.kind === "shell") {
+    if (arm.shell === null) {
+      return `the ${arm.arm} arm runs with the shell disabled; only its MCP tools are available`;
+    }
+    const decision = arm.shell.guard(use.command);
+    return decision.allowed ? null : decision.reason;
+  }
+  if (use.kind === "mcp") {
+    return arm.mcp === null ? `the ${arm.arm} arm has no MCP server attached` : null;
+  }
+  return null;
+}
+
+/** A human-readable rendering of a leaked tool use, tagging it with the reason. */
+function describeLeak(use: ToolUse, reason: string): string {
+  switch (use.kind) {
+    case "shell":
+      return `foreign shell command ${JSON.stringify(use.command)} reached: ${reason}`;
+    case "mcp":
+      return `MCP tool "${use.server}/${use.tool}" reached: ${reason}`;
+    case "other":
+      return `tool "${use.name}" reached: ${reason}`;
+  }
+}
+
+/**
+ * Re-check a completed run's transcript against the arm's isolation rules,
+ * re-applying `foreignToolReason` to every executed tool. A tool the arm should
+ * never have reached — a guard-denied shell command, a shell command on the MCP
+ * arm, an MCP call on a shell arm — is reported as a leak. Clean when nothing leaked.
+ */
+export function auditTranscript(arm: ArmDefinition, transcript: ToolUse[]): AuditResult {
+  const leaks: string[] = [];
+  for (const use of transcript) {
+    const reason = foreignToolReason(arm, use);
+    if (reason !== null) {
+      leaks.push(describeLeak(use, reason));
+    }
+  }
+  return leaks.length === 0 ? { clean: true } : { clean: false, leaks };
+}

bench/host.ts

@@ -0,0 +1,24 @@
+// The live host adapter: the production `BenchHost` the runner drives against a
+// real Gitea instance. It is a thin composition of the two live boundaries —
+// seed.ts (provision, seed, delete) and snapshot.ts (capture) — bound to one set
+// of host credentials. The runner depends only on the `BenchHost` seam, so this
+// wiring is exercised by the smoke run rather than mocked unit tests, matching the
+// seed tier.
+
+import type { BenchHost } from "./runner.js";
+import { captureRepoState } from "./snapshot.js";
+import { deleteRepo, provisionRepo, seedRepo, type BenchAccess } from "./seed.js";
+
+/**
+ * Build the live host bound to `access`: it provisions and seeds fresh throwaway
+ * repositories, captures their post-run state, and deletes them — all through the
+ * real Gitea API using gitea-axi's own credential discovery (see resolveBenchAccess).
+ */
+export function liveBenchHost(access: BenchAccess): BenchHost {
+  return {
+    provision: () => provisionRepo(access),
+    seed: (coords) => seedRepo(access, coords),
+    capture: (coords) => captureRepoState(access, coords),
+    delete: (coords) => deleteRepo(access, coords),
+  };
+}

bench/runner.smoke.test.ts

@@ -0,0 +1,111 @@
+import { mkdtempSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { afterAll, beforeAll, describe, expect, it } from "vitest";
+import type { CliDeps } from "../src/deps.js";
+import { liveBenchHost } from "./host.js";
+import { runCell } from "./runner.js";
+import { resolveBenchAccess, type BenchAccess } from "./seed.js";
+import { sdkAgentDriver } from "./sdk-driver.js";
+import { createSampleStore } from "./store.js";
+import { SAMPLE_TASK } from "./task.js";
+
+/**
+ * The single-cell runner smoke tier: one live run of the whole tracer-bullet path
+ * against a real Gitea host, driving the agent through the Claude Agent SDK. It
+ * proves that seed, arm scaffolding, guard, runner, checker, and store all connect
+ * end to end — provisioning and seeding a fresh repository, running the sample task
+ * under the gitea-axi arm, capturing and scoring the result, appending the sample,
+ * and deleting the repository.
+ *
+ * Like the seed smoke tier it keys off GITEA_AXI_BENCH_LOGIN (the live host,
+ * discovered through gitea-axi's tea-login credential path) and skips cleanly when
+ * that is unset. It additionally skips when the Agent SDK is not installed, since
+ * the SDK is an optional peer of the harness needed only for live runs — either
+ * way a skip counts as a pass, matching the end-to-end tier's behaviour when no
+ * live instance is configured. Running it for real also requires the `gitea-axi`
+ * CLI on PATH (the arm's allow-listed binary) and a Claude subscription.
+ *
+ * The run's pass/fail is nondeterministic because a live model drives it, so the
+ * assertions are structural — the terminal outcome shape and the record and
+ * lifecycle facts — never a fixed pass/fail.
+ */
+const login = process.env.GITEA_AXI_BENCH_LOGIN;
+
+// The Agent SDK is loaded through a computed specifier so this file type-checks and
+// the deterministic tier runs without the package present; here we probe once
+// whether it is installed so the tier skips rather than errors when it is absent.
+const SDK_MODULE = "@anthropic-ai/claude-agent-sdk";
+let sdkAvailable = false;
+try {
+  await import(SDK_MODULE);
+  sdkAvailable = true;
+} catch {
+  sdkAvailable = false;
+}
+
+describe.skipIf(!login || !sdkAvailable)("single-cell runner smoke", () => {
+  let access: BenchAccess;
+  let binRoot: string;
+  let storeRoot: string;
+
+  beforeAll(async () => {
+    const deps: CliDeps = {
+      env: process.env,
+      cwd: process.cwd(),
+      globals: { login },
+    };
+    access = await resolveBenchAccess(deps, login!);
+    binRoot = mkdtempSync(join(tmpdir(), "bench-runner-smoke-bin-"));
+    storeRoot = mkdtempSync(join(tmpdir(), "bench-runner-smoke-store-"));
+  }, 180_000);
+
+  afterAll(() => {
+    if (binRoot) rmSync(binRoot, { recursive: true, force: true });
+    if (storeRoot) rmSync(storeRoot, { recursive: true, force: true });
+  });
+
+  it(
+    "runs one sample cell end to end against the live host, recording a scored or invalid result",
+    async () => {
+      const store = createSampleStore(storeRoot);
+
+      const outcome = await runCell({
+        arm: "gitea-axi",
+        task: SAMPLE_TASK,
+        trial: 1,
+        access,
+        host: liveBenchHost(access),
+        driver: sdkAgentDriver(),
+        store,
+        // The runner's own wall-clock backstop bounds the run well within the
+        // per-test timeout below.
+        bounds: { turnCap: 40, wallClockMs: 300_000 },
+        build: { binRoot },
+      });
+
+      // The full path completed: the cell was either scored (recorded) or the
+      // audit flagged a leak (invalid) — both are legitimate terminal outcomes,
+      // and either way the throwaway repository was deleted in runCell's finally.
+      expect(["recorded", "invalid"]).toContain(outcome.kind);
+
+      if (outcome.kind === "recorded") {
+        const samples = store.read({ arm: "gitea-axi", taskId: SAMPLE_TASK.id });
+        expect(samples).toHaveLength(1);
+        expect(outcome.record.taskId).toBe(SAMPLE_TASK.id);
+        expect(outcome.record.tier).toBe(SAMPLE_TASK.tier);
+        // The four token components and the imputed cost were captured.
+        expect(outcome.record.tokens).toEqual(
+          expect.objectContaining({
+            freshInput: expect.any(Number),
+            cacheCreation: expect.any(Number),
+            cacheRead: expect.any(Number),
+            output: expect.any(Number),
+          }),
+        );
+        expect(typeof outcome.record.imputedCostUsd).toBe("number");
+      }
+    },
+    360_000,
+  );
+});

bench/runner.test.ts

@@ -0,0 +1,400 @@
+import { mkdtempSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { afterEach, beforeEach, describe, expect, it } from "vitest";
+import type { RepoState } from "./scoring-spec.js";
+import type { BenchAccess, RepoCoords } from "./seed.js";
+import { groundTruth } from "./seed-plan.js";
+import { createSampleStore } from "./store.js";
+import { SAMPLE_TASK } from "./task.js";
+import type { AgentDriver, BenchHost } from "./runner.js";
+import { runCell } from "./runner.js";
+
+// The user the fake seed/capture are parametrized by; a single independent
+// literal, echoed into the passing capture below.
+const USER = "benchbot";
+
+// The fixed coordinates the fake host provisions. Independent literals so the
+// delete-with-these-coords assertion is unambiguous.
+const COORDS: RepoCoords = { owner: USER, repo: "bench-1" };
+
+const ACCESS: BenchAccess = { apiUrl: "https://git.example.test", token: "tok" };
+
+// The passing post-run state IS the task's own expected end state, so the
+// checker scores it a legitimate pass. Built from the task, not from runner.ts.
+const spec = SAMPLE_TASK.scoringSpec(USER);
+const passingState: RepoState = spec.kind === "mutation" ? spec.expected : groundTruth(USER);
+
+// The four token components, turns, and imputed cost are independent literals
+// planted in the fake driver; the recorded sample must carry them back unchanged.
+const DRIVER_TOKENS = { freshInput: 100, cacheCreation: 20, cacheRead: 300, output: 40 };
+const DRIVER_TURNS = 5;
+const DRIVER_COST = 0.12;
+
+/** A fake host recording which methods were called and with what coords. */
+function createFakeHost() {
+  const calls = {
+    provisioned: false,
+    seeded: false,
+    captured: false,
+    deletedCoords: null as RepoCoords | null,
+  };
+  const host: BenchHost = {
+    async provision() {
+      calls.provisioned = true;
+      return COORDS;
+    },
+    async seed(coords) {
+      calls.seeded = true;
+      return groundTruth(coords.owner);
+    },
+    async capture() {
+      calls.captured = true;
+      return passingState;
+    },
+    async delete(coords) {
+      calls.deletedCoords = coords;
+    },
+  };
+  return { host, calls };
+}
+
+/**
+ * A fake host whose `capture` returns the UNMUTATED seed — the target issue "Add
+ * CSV export option" is still OPEN, so the task's spec (which expects it CLOSED)
+ * is not satisfied and the checker scores the run incorrect. Everything else
+ * matches createFakeHost. Records the deleted coords for the cleanup assertion.
+ */
+function createUnmutatedHost() {
+  const calls = { deletedCoords: null as RepoCoords | null };
+  const host: BenchHost = {
+    async provision() {
+      return COORDS;
+    },
+    async seed(coords) {
+      return groundTruth(coords.owner);
+    },
+    async capture(coords) {
+      return groundTruth(coords.owner);
+    },
+    async delete(coords) {
+      calls.deletedCoords = coords;
+    },
+  };
+  return { host, calls };
+}
+
+/** A fake driver that resolves immediately with the planted metrics. */
+const driver: AgentDriver = {
+  async run() {
+    return {
+      tokens: DRIVER_TOKENS,
+      turns: DRIVER_TURNS,
+      imputedCostUsd: DRIVER_COST,
+      transcript: [{ kind: "mcp", server: "gitea-mcp", tool: "edit_issue" }],
+      finalReport: "Closed the issue.",
+      stoppedByTurnCap: false,
+    };
+  },
+};
+
+/**
+ * A fake driver that completes cleanly (not turn-capped, clean MCP transcript
+ * that audits clean on the gitea-mcp arm). Paired with the unmutated host, the
+ * run finishes but the checker scores it incorrect.
+ */
+const cleanDriver: AgentDriver = {
+  async run() {
+    return {
+      tokens: { freshInput: 50, cacheCreation: 0, cacheRead: 0, output: 10 },
+      turns: 2,
+      imputedCostUsd: 0.03,
+      transcript: [{ kind: "mcp", server: "gitea-mcp", tool: "list_repo_issues" }],
+      finalReport: "Done.",
+      stoppedByTurnCap: false,
+    };
+  },
+};
+
+/**
+ * A fake driver reporting it hit the turn cap (`stoppedByTurnCap: true`). The
+ * other fields are arbitrary-but-valid literals; the run should be recorded as a
+ * confused failure regardless of them.
+ */
+const turnCappedDriver: AgentDriver = {
+  async run() {
+    return {
+      tokens: { freshInput: 10, cacheCreation: 0, cacheRead: 0, output: 5 },
+      turns: 10,
+      imputedCostUsd: 0.02,
+      transcript: [],
+      finalReport: "",
+      stoppedByTurnCap: true,
+    };
+  },
+};
+
+/**
+ * A fake driver that never resolves on its own — it settles only when its abort
+ * signal fires. Paired with a clock whose backstop timer fires first, it models a
+ * run that hangs past the wall-clock bound: the runner aborts the signal, and the
+ * driver then settles so no forever-pending promise is leaked.
+ */
+const hangingDriver: AgentDriver = {
+  run: ({ signal }) =>
+    new Promise((resolve) => {
+      signal.addEventListener("abort", () =>
+        resolve({
+          tokens: { freshInput: 0, cacheCreation: 0, cacheRead: 0, output: 0 },
+          turns: 0,
+          imputedCostUsd: 0,
+          transcript: [],
+          finalReport: "",
+          stoppedByTurnCap: false,
+        }),
+      );
+    }),
+};
+
+/**
+ * A fake driver whose transcript reaches the shell on the shell-disabled
+ * gitea-mcp arm — a foreign-tool leak per the audit's contract (proven in
+ * bench/audit.test.ts). The post-run audit should flag this cell invalid.
+ */
+const leakingDriver: AgentDriver = {
+  async run() {
+    return {
+      tokens: { freshInput: 0, cacheCreation: 0, cacheRead: 0, output: 0 },
+      turns: 1,
+      imputedCostUsd: 0,
+      transcript: [{ kind: "shell", command: "curl https://git.example.test/api/v1/repos" }],
+      finalReport: "",
+      stoppedByTurnCap: false,
+    };
+  },
+};
+
+describe("runCell", () => {
+  let binRoot: string;
+  let storeRoot: string;
+
+  beforeEach(() => {
+    binRoot = mkdtempSync(join(tmpdir(), "bench-runner-bin-"));
+    storeRoot = mkdtempSync(join(tmpdir(), "bench-runner-"));
+  });
+
+  afterEach(() => {
+    rmSync(binRoot, { recursive: true, force: true });
+    rmSync(storeRoot, { recursive: true, force: true });
+  });
+
+  // Behavior: running one cell on the happy path provisions and seeds a fresh
+  // repository, runs the agent under its arm, captures the post-run snapshot and
+  // scores it (a PASS here, because the fake capture returns the task's own
+  // expected end state), appends exactly one sample carrying the driver's four
+  // token components / turns / imputed cost and a passing outcome, and deletes
+  // the repository afterward (benchmark-harness spec, runner tracer bullet). The
+  // gitea-mcp arm needs no host binaries and its MCP transcript audits clean.
+  it("provisions, seeds, runs, scores a pass, records one sample, and deletes the repo", async () => {
+    const { host, calls } = createFakeHost();
+    const store = createSampleStore(storeRoot);
+    const trial = 3;
+
+    const outcome = await runCell({
+      arm: "gitea-mcp",
+      task: SAMPLE_TASK,
+      trial,
+      access: ACCESS,
+      host,
+      driver,
+      store,
+      bounds: { turnCap: 10, wallClockMs: 60_000 },
+      build: { binRoot },
+    });
+
+    // The lifecycle ran end to end, deleting exactly the provisioned repo.
+    expect(calls.provisioned).toBe(true);
+    expect(calls.seeded).toBe(true);
+    expect(calls.captured).toBe(true);
+    expect(calls.deletedCoords).toEqual(COORDS);
+
+    // runCell resolves to a recorded outcome (not invalid).
+    expect(outcome.kind).toBe("recorded");
+    if (outcome.kind !== "recorded") return;
+
+    // Exactly one sample landed in this cell, carrying the driver's metrics
+    // unchanged, the task's coordinates, the trial passed in, and a passing outcome.
+    const samples = store.read({ arm: "gitea-mcp", taskId: SAMPLE_TASK.id });
+    expect(samples).toHaveLength(1);
+    const [sample] = samples;
+    expect(sample).toBeDefined();
+    if (sample === undefined) return;
+
+    expect(sample.arm).toBe("gitea-mcp");
+    expect(sample.taskId).toBe(SAMPLE_TASK.id);
+    expect(sample.tier).toBe(SAMPLE_TASK.tier);
+    expect(sample.trial).toBe(trial);
+
+    expect(sample.tokens).toEqual(DRIVER_TOKENS);
+    expect(sample.turns).toBe(DRIVER_TURNS);
+    expect(sample.imputedCostUsd).toBe(DRIVER_COST);
+    expect(sample.outcome).toEqual({ pass: true });
+
+    // The sample carries the run's wall-clock duration; a completed run takes
+    // non-negative time.
+    expect(typeof sample.durationMs).toBe("number");
+    expect(sample.durationMs).toBeGreaterThanOrEqual(0);
+
+    // The returned record is the same sample that was stored.
+    expect(outcome.record).toEqual(sample);
+  });
+
+  // Behavior: the run is bounded by a turn cap, and a run that hit it records a
+  // failure tagged confused (benchmark-harness spec, "confused-versus-hung"). The
+  // driver reports the cap was hit via stoppedByTurnCap: true, so the recorded
+  // sample's outcome must be { pass: false, failure: "confused" } — the failed
+  // outcome shape and the "confused" tag are independent literals fixed by
+  // result.ts's Outcome/FailureTag contract, not recomputed from runner.ts. Still
+  // exactly one sample lands and runCell resolves to a recorded outcome.
+  it("records a confused failure when the run hits the turn cap", async () => {
+    const { host } = createFakeHost();
+    const store = createSampleStore(storeRoot);
+
+    const outcome = await runCell({
+      arm: "gitea-mcp",
+      task: SAMPLE_TASK,
+      trial: 1,
+      access: ACCESS,
+      host,
+      driver: turnCappedDriver,
+      store,
+      bounds: { turnCap: 10, wallClockMs: 60_000 },
+      build: { binRoot },
+    });
+
+    expect(outcome.kind).toBe("recorded");
+    if (outcome.kind !== "recorded") return;
+
+    const samples = store.read({ arm: "gitea-mcp", taskId: SAMPLE_TASK.id });
+    expect(samples).toHaveLength(1);
+    const [sample] = samples;
+    expect(sample).toBeDefined();
+    if (sample === undefined) return;
+    expect(sample.outcome).toEqual({ pass: false, failure: "confused" });
+  });
+
+  // Behavior: the run is also bounded by a wall-clock backstop, and a run that
+  // exceeds it records a failure tagged hung (benchmark-harness spec,
+  // "confused-versus-hung"). An injected clock fires the backstop timer before the
+  // (never-self-resolving) driver finishes; the runner aborts the signal, which
+  // lets the driver settle. The recorded sample's outcome must be
+  // { pass: false, failure: "hung" } — a hung outcome is an independent literal
+  // fixed by result.ts's Outcome/FailureTag contract, not recomputed from
+  // runner.ts. Still exactly one sample lands and runCell resolves to recorded.
+  it("records a hung failure when the run exceeds the wall-clock backstop", async () => {
+    const { host } = createFakeHost();
+    const store = createSampleStore(storeRoot);
+
+    // Fire the backstop timer promptly (0ms) with a real clearable handle, so the
+    // wall-clock bound trips before the hanging driver would ever resolve.
+    const clock = {
+      setTimer: (_ms: number, fn: () => void) => {
+        const h = setTimeout(fn, 0);
+        return { clear: () => clearTimeout(h) };
+      },
+    };
+
+    const outcome = await runCell({
+      arm: "gitea-mcp",
+      task: SAMPLE_TASK,
+      trial: 1,
+      access: ACCESS,
+      host,
+      driver: hangingDriver,
+      store,
+      bounds: { turnCap: 10, wallClockMs: 50 },
+      build: { binRoot },
+      clock,
+    });
+
+    expect(outcome.kind).toBe("recorded");
+    if (outcome.kind !== "recorded") return;
+
+    const samples = store.read({ arm: "gitea-mcp", taskId: SAMPLE_TASK.id });
+    expect(samples).toHaveLength(1);
+    const [sample] = samples;
+    expect(sample).toBeDefined();
+    if (sample === undefined) return;
+    expect(sample.outcome).toEqual({ pass: false, failure: "hung" });
+  });
+
+  // Behavior: a transcript audit runs after each cell, and a run in which a
+  // foreign tool was reached is flagged invalid instead of scored (benchmark-
+  // harness spec, "Tool isolation"). Here the gitea-mcp arm (shell disabled) has a
+  // shell command in its transcript — a leak by the audit's contract (proven in
+  // bench/audit.test.ts). So runCell must yield { kind: "invalid", leaks } with a
+  // non-empty leaks array, append NO sample (the store stays empty), and still
+  // delete the provisioned repo. These come from the CellOutcome contract and the
+  // spec's "invalid instead of scored", not from runner.ts's internals.
+  it("flags the cell invalid without scoring when the transcript reaches a foreign tool, still deleting the repo", async () => {
+    const { host, calls } = createFakeHost();
+    const store = createSampleStore(storeRoot);
+
+    const outcome = await runCell({
+      arm: "gitea-mcp",
+      task: SAMPLE_TASK,
+      trial: 1,
+      access: ACCESS,
+      host,
+      driver: leakingDriver,
+      store,
+      bounds: { turnCap: 10, wallClockMs: 60_000 },
+      build: { binRoot },
+    });
+
+    // Invalid instead of scored: a non-empty leaks array, and no sample appended.
+    expect(outcome.kind).toBe("invalid");
+    if (outcome.kind !== "invalid") return;
+    expect(outcome.leaks.length).toBeGreaterThan(0);
+
+    expect(store.read({ arm: "gitea-mcp", taskId: SAMPLE_TASK.id })).toHaveLength(0);
+
+    // Cleanup still happens: the provisioned repo is deleted.
+    expect(calls.deletedCoords).toEqual(COORDS);
+  });
+
+  // Behavior: the recorded sample carries the checker's pass/fail outcome, and a
+  // run that finished cleanly (not turn-capped, no leak) but whose post-run
+  // snapshot does not satisfy the task's scoring spec is scored a failure tagged
+  // incorrect (benchmark-harness spec; result.ts's FailureTag). The unmutated host
+  // returns the seed state where the target issue is still OPEN, but the task's
+  // spec expects it CLOSED, so the checker's full-state diff fails. The expected
+  // outcome { pass: false, failure: "incorrect" } is an independent literal from
+  // result.ts's contract, not recomputed from runner.ts.
+  it("records an incorrect failure when a clean run's snapshot does not satisfy the scoring spec", async () => {
+    const { host } = createUnmutatedHost();
+    const store = createSampleStore(storeRoot);
+
+    const outcome = await runCell({
+      arm: "gitea-mcp",
+      task: SAMPLE_TASK,
+      trial: 1,
+      access: ACCESS,
+      host,
+      driver: cleanDriver,
+      store,
+      bounds: { turnCap: 10, wallClockMs: 60_000 },
+      build: { binRoot },
+    });
+
+    expect(outcome.kind).toBe("recorded");
+    if (outcome.kind !== "recorded") return;
+
+    const samples = store.read({ arm: "gitea-mcp", taskId: SAMPLE_TASK.id });
+    expect(samples).toHaveLength(1);
+    const [sample] = samples;
+    expect(sample).toBeDefined();
+    if (sample === undefined) return;
+    expect(sample.outcome).toEqual({ pass: false, failure: "incorrect" });
+  });
+});

bench/runner.ts

@@ -0,0 +1,255 @@
+// The single-cell runner: the tracer bullet that threads every layer to run one
+// `(arm, task, trial)` cell end to end and record an immutable result. It
+// provisions and seeds a fresh throwaway repository, runs the agent under exactly
+// the active arm's tool with the guard active, bounds the run by a turn cap and a
+// wall-clock backstop, captures and scores the post-run state, appends the result
+// sample to the store, and deletes the repository — auditing the transcript so a
+// run that reached a foreign tool is flagged invalid rather than scored.
+//
+// The two boundaries the runner cannot make deterministic — the live host and the
+// Claude Agent SDK — are factored behind the `BenchHost` and `AgentDriver` seams,
+// so the orchestration here is unit-tested with fakes while the live wiring is
+// validated by a smoke run (runner.smoke.test.ts), mirroring the seed tier.
+
+import { buildArm, type ArmDefinition, type BuildArmOptions, type SharedContext } from "./arm.js";
+import { auditTranscript, type ToolUse } from "./audit.js";
+import { score } from "./checker.js";
+import type { Arm, Outcome, ResultRecord, TokenComponents } from "./result.js";
+import type { RepoState } from "./scoring-spec.js";
+import type { BenchAccess, RepoCoords } from "./seed.js";
+import type { SampleStore } from "./store.js";
+import type { BenchTask } from "./task.js";
+
+/**
+ * What the agent driver reports from one run: the four token components (folding
+ * in the auxiliary small model, per the cost-equivalent-token metric), the turn
+ * count, the imputed cost, the transcript for the post-run audit, the agent's
+ * final report for read tasks, and whether the run stopped because it hit the
+ * turn cap (which the runner tags as a confused failure).
+ */
+export interface AgentRun {
+  tokens: TokenComponents;
+  turns: number;
+  imputedCostUsd: number;
+  transcript: ToolUse[];
+  finalReport: string;
+  stoppedByTurnCap: boolean;
+}
+
+/** The inputs the runner hands the driver for one run. */
+export interface AgentRunInput {
+  /** The assembled arm (system prompt plus tool/guard or MCP configuration). */
+  arm: ArmDefinition;
+  /** The task's natural-language intent. */
+  intent: string;
+  /** The turn cap the driver must enforce, reporting `stoppedByTurnCap`. */
+  turnCap: number;
+  /** Aborted when the wall-clock backstop fires; the driver must resolve on abort. */
+  signal: AbortSignal;
+}
+
+/**
+ * The agent driver seam. The production implementation drives the Claude Agent
+ * SDK on the maintainer's subscription (sdk-driver.ts); tests inject a fake.
+ */
+export interface AgentDriver {
+  run(input: AgentRunInput): Promise<AgentRun>;
+}
+
+/**
+ * The live-host surface the runner drives, factored out so the orchestration is
+ * testable with a fake. The production implementation talks to the real Gitea
+ * host (seed.ts and snapshot.ts); its value is the real API interaction, so it is
+ * validated by the smoke run rather than mocked.
+ */
+export interface BenchHost {
+  /** Create and return a fresh, empty throwaway repository. */
+  provision(): Promise<RepoCoords>;
+  /** Seed the repository to the deterministic ground truth. */
+  seed(coords: RepoCoords): Promise<RepoState>;
+  /** Read the full post-run repository state as a snapshot. */
+  capture(coords: RepoCoords): Promise<RepoState>;
+  /** Best-effort deletion of the throwaway repository. */
+  delete(coords: RepoCoords): Promise<void>;
+}
+
+/** The two bounds every run is held within. */
+export interface RunBounds {
+  /** Maximum agent turns; exceeding it is a confused failure. */
+  turnCap: number;
+  /** Wall-clock backstop in milliseconds; exceeding it is a hung failure. */
+  wallClockMs: number;
+}
+
+/**
+ * The clock and timer the runner uses, injectable so timing is deterministic in
+ * tests. Defaults to real wall-clock time and `setTimeout`.
+ */
+export interface RunnerClock {
+  now: () => number;
+  setTimer: (ms: number, fn: () => void) => { clear: () => void };
+}
+
+/** Everything needed to run one `(arm, task, trial)` cell. */
+export interface RunCellInput {
+  arm: Arm;
+  task: BenchTask;
+  trial: number;
+  access: BenchAccess;
+  host: BenchHost;
+  driver: AgentDriver;
+  store: SampleStore;
+  bounds: RunBounds;
+  build: BuildArmOptions;
+  clock?: Partial<RunnerClock>;
+}
+
+/**
+ * The result of running one cell: either a scored sample was recorded, or the run
+ * was flagged invalid — a foreign tool was reached — and left unscored, so it
+ * never becomes a sample in the store.
+ */
+export type CellOutcome =
+  | { kind: "recorded"; record: ResultRecord }
+  | { kind: "invalid"; leaks: string[] };
+
+const DEFAULT_CLOCK: RunnerClock = {
+  now: () => Date.now(),
+  setTimer: (ms, fn) => {
+    const handle = setTimeout(fn, ms);
+    return { clear: () => clearTimeout(handle) };
+  },
+};
+
+/** Token components for a run that produced no measurable consumption (a hung run). */
+const NO_TOKENS: TokenComponents = { freshInput: 0, cacheCreation: 0, cacheRead: 0, output: 0 };
+
+/**
+ * Run one cell end to end. Provisions and seeds a throwaway repository, runs the
+ * agent under the arm with a turn cap and a wall-clock backstop, audits the
+ * transcript, scores the completed run, appends the sample, and always deletes
+ * the repository. Exceeding the turn cap records a confused failure; exceeding the
+ * wall-clock backstop records a hung failure; a transcript that reached a foreign
+ * tool is flagged invalid rather than scored.
+ */
+export async function runCell(input: RunCellInput): Promise<CellOutcome> {
+  const { arm, task, trial, access, host, driver, store, bounds, build } = input;
+  const clock: RunnerClock = { ...DEFAULT_CLOCK, ...input.clock };
+
+  const coords = await host.provision();
+  try {
+    await host.seed(coords);
+    const context: SharedContext = { coords, access };
+    const armDef = buildArm(arm, context, build);
+
+    const started = clock.now();
+    const result = await runBounded(driver, armDef, task.intent, bounds, clock);
+    const durationMs = clock.now() - started;
+
+    // A hung run produced no completed transcript to audit or score; record it
+    // as a failure with no measured consumption.
+    if (result.kind === "hung") {
+      return recorded(store, makeRecord(input, NO_TOKENS, 0, 0, durationMs, { pass: false, failure: "hung" }, clock));
+    }
+
+    const run = result.run;
+
+    // The post-run audit is authoritative on validity: a reached foreign tool
+    // invalidates the trial rather than letting it be scored or recorded.
+    const audit = auditTranscript(armDef, run.transcript);
+    if (!audit.clean) {
+      return { kind: "invalid", leaks: audit.leaks };
+    }
+
+    const outcome = run.stoppedByTurnCap
+      ? ({ pass: false, failure: "confused" } as const)
+      : await scoreRun(host, coords, task, run);
+
+    return recorded(
+      store,
+      makeRecord(input, run.tokens, run.turns, run.imputedCostUsd, durationMs, outcome, clock),
+    );
+  } finally {
+    await host.delete(coords);
+  }
+}
+
+/** The bounded outcome of driving the agent: it either ran, or the backstop fired. */
+type BoundedResult = { kind: "ran"; run: AgentRun } | { kind: "hung" };
+
+/**
+ * Drive the agent under the wall-clock backstop. The driver enforces the turn cap
+ * itself (reporting `stoppedByTurnCap`); this races it against a timer so a driver
+ * that genuinely hangs cannot block the cell forever. When the timer wins, the
+ * signal is aborted so a cooperating driver can stop, and the run is hung.
+ */
+async function runBounded(
+  driver: AgentDriver,
+  arm: ArmDefinition,
+  intent: string,
+  bounds: RunBounds,
+  clock: RunnerClock,
+): Promise<BoundedResult> {
+  const controller = new AbortController();
+  let timer: { clear: () => void } | undefined;
+  const backstop = new Promise<BoundedResult>((resolve) => {
+    timer = clock.setTimer(bounds.wallClockMs, () => resolve({ kind: "hung" }));
+  });
+  try {
+    return await Promise.race([
+      driver
+        .run({ arm, intent, turnCap: bounds.turnCap, signal: controller.signal })
+        .then((run) => ({ kind: "ran" as const, run })),
+      backstop,
+    ]);
+  } finally {
+    timer?.clear();
+    controller.abort();
+  }
+}
+
+/**
+ * Score a completed (not turn-capped) run: capture the post-run snapshot and diff
+ * it against the task's expected end state for a mutation, or match the agent's
+ * final report against the required facts for a read. A pass is a pass; anything
+ * the checker rejects is an incorrect failure.
+ */
+async function scoreRun(host: BenchHost, coords: RepoCoords, task: BenchTask, run: AgentRun): Promise<Outcome> {
+  const spec = task.scoringSpec(coords.owner);
+  const snapshot = await host.capture(coords);
+  const check =
+    spec.kind === "mutation"
+      ? score(spec, { kind: "mutation", state: snapshot })
+      : score(spec, { kind: "read", report: run.finalReport });
+  return check.pass ? { pass: true } : { pass: false, failure: "incorrect" };
+}
+
+/** Assemble the immutable result record for one run. */
+function makeRecord(
+  input: RunCellInput,
+  tokens: TokenComponents,
+  turns: number,
+  imputedCostUsd: number,
+  durationMs: number,
+  outcome: Outcome,
+  clock: RunnerClock,
+): ResultRecord {
+  return {
+    arm: input.arm,
+    taskId: input.task.id,
+    tier: input.task.tier,
+    trial: input.trial,
+    timestamp: new Date(clock.now()).toISOString(),
+    tokens,
+    turns,
+    durationMs,
+    imputedCostUsd,
+    outcome,
+  };
+}
+
+/** Append the record and return it as the recorded cell outcome. */
+function recorded(store: SampleStore, record: ResultRecord): CellOutcome {
+  store.append(record);
+  return { kind: "recorded", record };
+}

bench/sdk-driver.ts

@@ -0,0 +1,233 @@
+// The production agent driver: the adapter that runs one arm through the Claude
+// Agent SDK on the maintainer's subscription and reports the metrics and
+// transcript the runner records. It is the concrete `AgentDriver` behind the seam
+// the runner depends on; the deterministic runner tests inject a fake instead, and
+// this live wiring is exercised only by the smoke run.
+//
+// The Agent SDK is loaded through a computed dynamic import so the harness's
+// deterministic tier and the project's typecheck never require the package to be
+// installed — the SDK is needed only for live runs, exactly as the seed smoke tier
+// needs a live Gitea host. A local interface describes the slice of the SDK this
+// adapter consumes, so this side of the boundary stays type-checked even though the
+// package is optional.
+//
+// Isolation is enforced in-band via the SDK's permission callback: every Bash
+// command on a shell arm is put through the arm's own guard, and the shell is
+// disabled entirely on the MCP arm. Only tools that were permitted to run are
+// recorded in the transcript, so the runner's post-run audit sees what actually
+// executed — a blocked attempt is realistic wasted effort, not a leak.
+
+import type { ArmDefinition } from "./arm.js";
+import { foreignToolReason, type ToolUse } from "./audit.js";
+import type { TokenComponents } from "./result.js";
+import type { AgentDriver, AgentRun, AgentRunInput } from "./runner.js";
+
+/** The default fixed model every arm is run on (overridable for the whole run). */
+const DEFAULT_MODEL = "claude-opus-4-8";
+
+/** The Agent SDK package, resolved at run time so it is an optional peer of the harness. */
+const SDK_MODULE = "@anthropic-ai/claude-agent-sdk";
+
+/** Configuration for the SDK-backed driver. */
+export interface SdkDriverConfig {
+  /** The single fixed model all arms run on. Defaults to the latest Opus. */
+  model?: string;
+  /** Override the SDK module specifier (tests/tooling); defaults to the real package. */
+  moduleSpecifier?: string;
+}
+
+// --- The slice of the Claude Agent SDK this adapter consumes ------------------
+
+/** The per-request token usage the SDK reports, per model. */
+interface SdkUsage {
+  input_tokens?: number;
+  output_tokens?: number;
+  cache_creation_input_tokens?: number;
+  cache_read_input_tokens?: number;
+}
+
+interface SdkResultMessage {
+  type: "result";
+  /** `error_max_turns` when the run hit the turn cap. */
+  subtype: string;
+  usage?: SdkUsage;
+  /** Per-model usage, including the auxiliary small model the runtime invokes. */
+  modelUsage?: Record<string, SdkUsage>;
+  total_cost_usd?: number;
+  num_turns?: number;
+  result?: string;
+}
+
+interface SdkContentBlock {
+  type: string;
+  name?: string;
+  input?: Record<string, unknown>;
+}
+
+interface SdkAssistantMessage {
+  type: "assistant";
+  message: { content: SdkContentBlock[] };
+}
+
+type SdkMessage = SdkResultMessage | SdkAssistantMessage | { type: string };
+
+type SdkPermissionResult =
+  | { behavior: "allow"; updatedInput: Record<string, unknown> }
+  | { behavior: "deny"; message: string };
+
+interface SdkStdioServer {
+  type: "stdio";
+  command: string;
+  args: string[];
+  env: Record<string, string>;
+}
+
+interface SdkQueryOptions {
+  model: string;
+  /** Fixed at zero across every arm so runs are as deterministic as the model allows. */
+  temperature: number;
+  systemPrompt: string;
+  maxTurns: number;
+  abortController: AbortController;
+  canUseTool: (toolName: string, input: Record<string, unknown>) => Promise<SdkPermissionResult>;
+  settingSources: string[];
+  env?: Record<string, string | undefined>;
+  mcpServers?: Record<string, SdkStdioServer>;
+  disallowedTools?: string[];
+}
+
+interface SdkModule {
+  query: (args: { prompt: string; options: SdkQueryOptions }) => AsyncIterable<SdkMessage>;
+}
+
+// --- Metric and transcript extraction ----------------------------------------
+
+/**
+ * Sum the four token components across every model the run touched, so the
+ * auxiliary small model the runtime invokes is folded in as the metric spec
+ * requires. Falls back to the aggregate `usage` when no per-model breakdown is
+ * present.
+ */
+function sumTokens(result: SdkResultMessage): TokenComponents {
+  const usages = result.modelUsage
+    ? Object.values(result.modelUsage)
+    : result.usage
+      ? [result.usage]
+      : [];
+  const total: TokenComponents = { freshInput: 0, cacheCreation: 0, cacheRead: 0, output: 0 };
+  for (const usage of usages) {
+    total.freshInput += usage.input_tokens ?? 0;
+    total.cacheCreation += usage.cache_creation_input_tokens ?? 0;
+    total.cacheRead += usage.cache_read_input_tokens ?? 0;
+    total.output += usage.output_tokens ?? 0;
+  }
+  return total;
+}
+
+/** Classify one tool invocation into the isolation-relevant shape the audit consumes. */
+function classifyTool(toolName: string, input: Record<string, unknown>): ToolUse {
+  if (toolName === "Bash") {
+    return { kind: "shell", command: String(input.command ?? "") };
+  }
+  if (toolName.startsWith("mcp__")) {
+    const [, server = "", tool = ""] = toolName.split("__");
+    return { kind: "mcp", server, tool };
+  }
+  return { kind: "other", name: toolName };
+}
+
+/**
+ * Build the SDK-backed agent driver. Each run drives one arm through the Agent
+ * SDK: the arm's assembled system prompt, the task intent as the user prompt, the
+ * fixed model, the turn cap as `maxTurns`, and the arm's tool configuration —
+ * either its curated shell PATH with the guard on the permission callback, or the
+ * MCP server attached with the shell disabled. The run is wired to the runner's
+ * abort signal so the wall-clock backstop can stop it.
+ */
+export function sdkAgentDriver(config: SdkDriverConfig = {}): AgentDriver {
+  const model = config.model ?? DEFAULT_MODEL;
+  const specifier = config.moduleSpecifier ?? SDK_MODULE;
+
+  return {
+    async run(input: AgentRunInput): Promise<AgentRun> {
+      const { query } = (await import(specifier)) as SdkModule;
+
+      const controller = new AbortController();
+      if (input.signal.aborted) {
+        controller.abort();
+      } else {
+        input.signal.addEventListener("abort", () => controller.abort(), { once: true });
+      }
+
+      // The transcript records only tools that were permitted to run, so the
+      // runner's audit sees what actually executed, not blocked attempts.
+      const transcript: ToolUse[] = [];
+      const canUseTool = async (
+        toolName: string,
+        toolInput: Record<string, unknown>,
+      ): Promise<SdkPermissionResult> => {
+        const use = classifyTool(toolName, toolInput);
+        const denial = foreignToolReason(input.arm, use);
+        if (denial !== null) {
+          return { behavior: "deny", message: denial };
+        }
+        transcript.push(use);
+        return { behavior: "allow", updatedInput: toolInput };
+      };
+
+      const options = buildOptions(input.arm, model, input.turnCap, controller, canUseTool);
+
+      let result: SdkResultMessage | undefined;
+      for await (const message of query({ prompt: input.intent, options })) {
+        if (message.type === "result") {
+          result = message as SdkResultMessage;
+        }
+      }
+      if (result === undefined) {
+        throw new Error("the Agent SDK produced no result message");
+      }
+
+      return {
+        tokens: sumTokens(result),
+        turns: result.num_turns ?? 0,
+        imputedCostUsd: result.total_cost_usd ?? 0,
+        transcript,
+        finalReport: result.result ?? "",
+        stoppedByTurnCap: result.subtype === "error_max_turns",
+      };
+    },
+  };
+}
+
+/** Assemble the SDK query options for an arm's tool configuration. */
+function buildOptions(
+  arm: ArmDefinition,
+  model: string,
+  turnCap: number,
+  controller: AbortController,
+  canUseTool: SdkQueryOptions["canUseTool"],
+): SdkQueryOptions {
+  const options: SdkQueryOptions = {
+    model,
+    // Temperature zero across all arms, per the runner-and-metrics spec, so the
+    // comparison measures the tool rather than sampling noise.
+    temperature: 0,
+    systemPrompt: arm.systemPrompt,
+    maxTurns: turnCap,
+    abortController: controller,
+    canUseTool,
+    // Start from a clean slate: no user/project settings leak tools or config
+    // into the measured run.
+    settingSources: [],
+  };
+  if (arm.shell !== null) {
+    // Lead the agent's PATH with the arm's curated bin directory so only its one
+    // allowed binary resolves by name; the guard on canUseTool is the authority.
+    options.env = { ...process.env, PATH: arm.shell.path };
+  }
+  if (arm.mcp !== null) {
+    options.mcpServers = { [arm.arm]: { type: "stdio", ...arm.mcp.server } };
+    options.disallowedTools = ["Bash"];
+  }
+  return options;
+}

bench/seed.ts

@@ -73,8 +73,12 @@ async function requireOk(res: Response, method: string, path: string): Promise<R
   return res;
 }
 
-/** Issue a request and require a 2xx, returning the parsed JSON body. */
-async function send<T>(
+/**
+ * Issue a request and require a 2xx, returning the parsed JSON body. Exported so
+ * the post-run snapshot capture (snapshot.ts) reads the live repository through
+ * the same authenticated round-trip the seed writes through.
+ */
+export async function send<T>(
   access: BenchAccess,
   method: string,
   path: string,

bench/snapshot.ts

@@ -0,0 +1,185 @@
+// Post-run snapshot capture: read the entire scored surface of a live throwaway
+// repository back into the `RepoState` shape the checker diffs against. This is
+// the runner's counterpart to the seed — the seed writes the ground truth, this
+// reads the actual post-run state — so a mutation task can be scored by the
+// full-state diff (checker.ts) and any collateral change is caught.
+//
+// Like seed.ts, this is an imperative live boundary: its value is the real Gitea
+// API interaction, so it is exercised by the smoke run rather than mocked unit
+// tests. It reuses seed.ts's authenticated `send` helper so reads and writes share
+// one round-trip, and normalizes the few fields whose live representation differs
+// from the declared ground truth (notably label colours, which Gitea returns as
+// bare hex without the leading `#`).
+
+import { send, type BenchAccess, type RepoCoords } from "./seed.js";
+import type {
+  Comment,
+  Issue,
+  IssueState,
+  Label,
+  PullRequest,
+  PullRequestState,
+  RepoState,
+  Review,
+  ReviewKind,
+} from "./scoring-spec.js";
+
+interface GiteaLabel {
+  name: string;
+  color: string;
+  description?: string;
+}
+
+interface GiteaComment {
+  body: string;
+  user: { login: string };
+}
+
+interface GiteaIssue {
+  number: number;
+  title: string;
+  body: string;
+  state: string;
+  labels: { name: string }[] | null;
+  assignees: { login: string }[] | null;
+}
+
+interface GiteaPull {
+  number: number;
+  title: string;
+  body: string;
+  state: string;
+  merged: boolean;
+  labels: { name: string }[] | null;
+  assignees: { login: string }[] | null;
+}
+
+interface GiteaReview {
+  user: { login: string };
+  state: string;
+  body: string;
+}
+
+/** Colours compare against the declared ground truth's leading-`#` lowercase hex. */
+function normalizeColor(color: string): string {
+  const bare = color.replace(/^#/, "").toLowerCase();
+  return `#${bare}`;
+}
+
+/** Gitea's applied-label and assignee arrays are nullable; reduce to name/login sets. */
+function labelNames(labels: { name: string }[] | null): string[] {
+  return (labels ?? []).map((label) => label.name);
+}
+
+function assigneeLogins(assignees: { login: string }[] | null): string[] {
+  return (assignees ?? []).map((assignee) => assignee.login);
+}
+
+/** Map a Gitea comment to the author/body pair the checker matches on. */
+function toComment(comment: GiteaComment): Comment {
+  return { author: comment.user.login, body: comment.body };
+}
+
+/** The Gitea review `state` verbs that map to a scored review kind; others are ignored. */
+const REVIEW_KIND: Record<string, ReviewKind> = {
+  APPROVED: "approved",
+  REQUEST_CHANGES: "request-changes",
+  COMMENT: "comment",
+};
+
+/** Read the comments (author and body) on an issue or pull request by number. */
+async function captureComments(
+  access: BenchAccess,
+  coords: RepoCoords,
+  number: number,
+): Promise<Comment[]> {
+  const comments = await send<GiteaComment[]>(
+    access,
+    "GET",
+    `/repos/${coords.owner}/${coords.repo}/issues/${number}/comments`,
+  );
+  return comments.map(toComment);
+}
+
+/**
+ * Read a pull request's reviews, keeping only those whose Gitea state maps to a
+ * scored review kind (a bare pending or review-request entry is dropped). Inline
+ * review comments are left empty, matching the declared ground truth, which the
+ * single-user seed never populates with inline comments.
+ */
+async function captureReviews(
+  access: BenchAccess,
+  coords: RepoCoords,
+  number: number,
+): Promise<Review[]> {
+  const reviews = await send<GiteaReview[]>(
+    access,
+    "GET",
+    `/repos/${coords.owner}/${coords.repo}/pulls/${number}/reviews`,
+  );
+  const captured: Review[] = [];
+  for (const review of reviews) {
+    const kind = REVIEW_KIND[review.state];
+    if (kind === undefined) {
+      continue;
+    }
+    captured.push({ author: review.user.login, kind, body: review.body, comments: [] });
+  }
+  return captured;
+}
+
+/**
+ * Capture the full post-run state of a throwaway repository — its labels, issues,
+ * and pull requests with their applied labels, assignees, comments, and reviews —
+ * in the shape the checker scores against. The volatile ids and timestamps the
+ * ground truth drops are simply never read.
+ */
+export async function captureRepoState(
+  access: BenchAccess,
+  coords: RepoCoords,
+): Promise<RepoState> {
+  const base = `/repos/${coords.owner}/${coords.repo}`;
+
+  const rawLabels = await send<GiteaLabel[]>(access, "GET", `${base}/labels?limit=100`);
+  const labels: Label[] = rawLabels.map((label) => ({
+    name: label.name,
+    color: normalizeColor(label.color),
+    ...(label.description ? { description: label.description } : {}),
+  }));
+
+  const rawIssues = await send<GiteaIssue[]>(
+    access,
+    "GET",
+    `${base}/issues?type=issues&state=all&limit=100`,
+  );
+  const issues: Issue[] = [];
+  for (const issue of rawIssues) {
+    issues.push({
+      number: issue.number,
+      title: issue.title,
+      body: issue.body,
+      state: issue.state as IssueState,
+      labels: labelNames(issue.labels),
+      assignees: assigneeLogins(issue.assignees),
+      comments: await captureComments(access, coords, issue.number),
+    });
+  }
+
+  const rawPulls = await send<GiteaPull[]>(access, "GET", `${base}/pulls?state=all&limit=100`);
+  const pullRequests: PullRequest[] = [];
+  for (const pull of rawPulls) {
+    const state: PullRequestState = pull.merged ? "merged" : (pull.state as PullRequestState);
+    pullRequests.push({
+      number: pull.number,
+      title: pull.title,
+      body: pull.body,
+      state,
+      labels: labelNames(pull.labels),
+      assignees: assigneeLogins(pull.assignees),
+      comments: await captureComments(access, coords, pull.number),
+      reviews: await captureReviews(access, coords, pull.number),
+    });
+  }
+
+  return { labels, issues, pullRequests };
+}

bench/task.test.ts

@@ -0,0 +1,46 @@
+import { describe, expect, it } from "vitest";
+import { SAMPLE_TASK } from "./task.js";
+
+/**
+ * The sample task's target and a control, taken as independent literals from the
+ * seed plan's declared ground truth (bench/seed-plan.ts): the sample task closes
+ * the OPEN issue "Add CSV export option", while "Fix crash on startup" is a
+ * different issue that is OPEN in the seed. These titles and seed states are the
+ * source of truth, fixed by the seed plan and the task's stated purpose — not
+ * read back from task.ts.
+ */
+const TARGET_TITLE = "Add CSV export option";
+const CONTROL_TITLE = "Fix crash on startup";
+
+// A single available user for the scoring spec; its exact value is irrelevant to
+// this behavior (an arbitrary independent literal), since we assert issue states,
+// not assignees.
+const USER = "benchbot";
+
+describe("SAMPLE_TASK", () => {
+  // Behavior: the one sample task exercises the full path by closing a specific
+  // seeded issue, and its scoring spec expresses the intended mutation's end
+  // state — the target issue closed, everything else unchanged from the seeded
+  // ground truth (benchmark-harness spec, "runnable Task wrapper"). Observable
+  // facts, each an independent literal from the seed plan / task purpose:
+  //  - it is a single-mutation task naming its target in the intent;
+  //  - its spec is a mutation spec;
+  //  - the target issue "Add CSV export option" is CLOSED in the expected state;
+  //  - a control issue OPEN in the seed and NOT the target stays OPEN.
+  it("closes only the target issue in its mutation scoring spec, leaving the seed otherwise unchanged", () => {
+    expect(SAMPLE_TASK.tier).toBe("single-mutation");
+    expect(SAMPLE_TASK.intent).toContain(TARGET_TITLE);
+
+    const spec = SAMPLE_TASK.scoringSpec(USER);
+    expect(spec.kind).toBe("mutation");
+    if (spec.kind !== "mutation") return;
+
+    const target = spec.expected.issues.find((issue) => issue.title === TARGET_TITLE);
+    expect(target).toBeDefined();
+    expect(target?.state).toBe("closed");
+
+    const control = spec.expected.issues.find((issue) => issue.title === CONTROL_TITLE);
+    expect(control).toBeDefined();
+    expect(control?.state).toBe("open");
+  });
+});

bench/task.ts

@@ -0,0 +1,55 @@
+// The runnable Task wrapper and the one sample task this slice uses to exercise
+// the full single-cell path. A task pairs a natural-language intent handed to the
+// agent with the tier it counts toward and a scoring spec the checker consumes.
+//
+// The scoring spec is a function of the single available user rather than fixed
+// data, because a mutation's expected end state is parametrized by that user: the
+// seed assigns issues to them and authors comments as them (see seed-plan's
+// `groundTruth(user)`). The runner resolves the user from the throwaway
+// repository's owner and calls this to obtain the concrete spec.
+//
+// The full task suite is authored in a later slice; this module carries only the
+// wrapper and a single sample task.
+
+import type { Tier } from "./result.js";
+import { groundTruth } from "./seed-plan.js";
+import type { ScoringSpec } from "./scoring-spec.js";
+
+/**
+ * A runnable benchmark task: the agent-facing intent, the tier it belongs to, a
+ * stable identifier, and a scoring spec keyed on the single available user.
+ */
+export interface BenchTask {
+  /** Stable identifier; keys the task's cell in the sample store. */
+  id: string;
+  /** The tier this task counts toward in the reporting rollups. */
+  tier: Tier;
+  /** The natural-language instruction handed to the agent. */
+  intent: string;
+  /** The scoring spec for the given single user (see module note). */
+  scoringSpec: (user: string) => ScoringSpec;
+}
+
+/** The seeded issue the sample task closes. */
+const SAMPLE_TARGET_TITLE = "Add CSV export option";
+
+/**
+ * One sample single-mutation task: close the seeded "Add CSV export option"
+ * issue and change nothing else. Its expected end state is the deterministic
+ * ground truth with only that issue's state flipped to closed, so the full-state
+ * diff catches both a missed close and any collateral change.
+ */
+export const SAMPLE_TASK: BenchTask = {
+  id: "close-csv-export-issue",
+  tier: "single-mutation",
+  intent: `Close the issue titled "${SAMPLE_TARGET_TITLE}". Do not modify anything else in the repository.`,
+  scoringSpec: (user) => {
+    const expected = groundTruth(user);
+    const target = expected.issues.find((issue) => issue.title === SAMPLE_TARGET_TITLE);
+    if (target === undefined) {
+      throw new Error(`sample task target issue "${SAMPLE_TARGET_TITLE}" is not in the seed plan`);
+    }
+    target.state = "closed";
+    return { kind: "mutation", expected };
+  },
+};