feat: add benchmark single-cell runner (task 0027)
All checks were successful
CI / test (pull_request) Successful in 50s
CI / test (push) Successful in 52s

Thread every benchmark layer to run one (arm, task, trial) cell end to
end: provision and seed a throwaway repository, run the agent under the
active arm bounded by a turn cap and a wall-clock backstop, audit the
transcript, capture and score the post-run state, append the sample, and
delete the repository.

- runner.ts: runCell orchestration behind the BenchHost and AgentDriver
  seams, so the flow is unit-tested with fakes while the live wiring is
  validated by a smoke run; turn-cap and wall-clock failures are tagged
  confused-versus-hung, and a leaked transcript is flagged invalid.
- audit.ts: the post-run transcript audit plus the shared
  foreignToolReason predicate both isolation enforcement points consume.
- task.ts: the runnable BenchTask wrapper and one sample single-mutation
  task exercising the full path.
- snapshot.ts: captureRepoState, the seed's read-back counterpart, in the
  RepoState shape the checker diffs against.
- host.ts / sdk-driver.ts: the live BenchHost and the Claude Agent SDK
  driver (an optional peer, loaded via dynamic import) for real runs.
- runner.smoke.test.ts: the live tracer-bullet tier, skipping cleanly
  when no host or SDK is configured.
This commit was merged in pull request #28.
This commit is contained in:
2026-07-16 09:17:38 -04:00
parent 9a2ba40657
commit c6a972734e
13 changed files with 1529 additions and 11 deletions

46
bench/task.test.ts Normal file
View File

@@ -0,0 +1,46 @@
import { describe, expect, it } from "vitest";
import { SAMPLE_TASK } from "./task.js";
/**
* The sample task's target and a control, taken as independent literals from the
* seed plan's declared ground truth (bench/seed-plan.ts): the sample task closes
* the OPEN issue "Add CSV export option", while "Fix crash on startup" is a
* different issue that is OPEN in the seed. These titles and seed states are the
* source of truth, fixed by the seed plan and the task's stated purpose — not
* read back from task.ts.
*/
const TARGET_TITLE = "Add CSV export option";
const CONTROL_TITLE = "Fix crash on startup";
// A single available user for the scoring spec; its exact value is irrelevant to
// this behavior (an arbitrary independent literal), since we assert issue states,
// not assignees.
const USER = "benchbot";
describe("SAMPLE_TASK", () => {
// Behavior: the one sample task exercises the full path by closing a specific
// seeded issue, and its scoring spec expresses the intended mutation's end
// state — the target issue closed, everything else unchanged from the seeded
// ground truth (benchmark-harness spec, "runnable Task wrapper"). Observable
// facts, each an independent literal from the seed plan / task purpose:
// - it is a single-mutation task naming its target in the intent;
// - its spec is a mutation spec;
// - the target issue "Add CSV export option" is CLOSED in the expected state;
// - a control issue OPEN in the seed and NOT the target stays OPEN.
it("closes only the target issue in its mutation scoring spec, leaving the seed otherwise unchanged", () => {
expect(SAMPLE_TASK.tier).toBe("single-mutation");
expect(SAMPLE_TASK.intent).toContain(TARGET_TITLE);
const spec = SAMPLE_TASK.scoringSpec(USER);
expect(spec.kind).toBe("mutation");
if (spec.kind !== "mutation") return;
const target = spec.expected.issues.find((issue) => issue.title === TARGET_TITLE);
expect(target).toBeDefined();
expect(target?.state).toBe("closed");
const control = spec.expected.issues.find((issue) => issue.title === CONTROL_TITLE);
expect(control).toBeDefined();
expect(control?.state).toBe("open");
});
});