alexion c6a972734e feat: add benchmark single-cell runner (task 0027) ...

Thread every benchmark layer to run one (arm, task, trial) cell end to
end: provision and seed a throwaway repository, run the agent under the
active arm bounded by a turn cap and a wall-clock backstop, audit the
transcript, capture and score the post-run state, append the sample, and
delete the repository.

- runner.ts: runCell orchestration behind the BenchHost and AgentDriver
  seams, so the flow is unit-tested with fakes while the live wiring is
  validated by a smoke run; turn-cap and wall-clock failures are tagged
  confused-versus-hung, and a leaked transcript is flagged invalid.
- audit.ts: the post-run transcript audit plus the shared
  foreignToolReason predicate both isolation enforcement points consume.
- task.ts: the runnable BenchTask wrapper and one sample single-mutation
  task exercising the full path.
- snapshot.ts: captureRepoState, the seed's read-back counterpart, in the
  RepoState shape the checker diffs against.
- host.ts / sdk-driver.ts: the live BenchHost and the Claude Agent SDK
  driver (an optional peer, loaded via dynamic import) for real runs.
- runner.smoke.test.ts: the live tracer-bullet tier, skipping cleanly
  when no host or SDK is configured.

2026-07-16 09:17:38 -04:00

5.7 KiB

Raw Blame History

View Raw