feat: add benchmark single-cell runner (task 0027)
All checks were successful
CI / test (pull_request) Successful in 50s
CI / test (push) Successful in 52s

Thread every benchmark layer to run one (arm, task, trial) cell end to
end: provision and seed a throwaway repository, run the agent under the
active arm bounded by a turn cap and a wall-clock backstop, audit the
transcript, capture and score the post-run state, append the sample, and
delete the repository.

- runner.ts: runCell orchestration behind the BenchHost and AgentDriver
  seams, so the flow is unit-tested with fakes while the live wiring is
  validated by a smoke run; turn-cap and wall-clock failures are tagged
  confused-versus-hung, and a leaked transcript is flagged invalid.
- audit.ts: the post-run transcript audit plus the shared
  foreignToolReason predicate both isolation enforcement points consume.
- task.ts: the runnable BenchTask wrapper and one sample single-mutation
  task exercising the full path.
- snapshot.ts: captureRepoState, the seed's read-back counterpart, in the
  RepoState shape the checker diffs against.
- host.ts / sdk-driver.ts: the live BenchHost and the Claude Agent SDK
  driver (an optional peer, loaded via dynamic import) for real runs.
- runner.smoke.test.ts: the live tracer-bullet tier, skipping cleanly
  when no host or SDK is configured.
This commit was merged in pull request #28.
This commit is contained in:
2026-07-16 09:17:38 -04:00
parent 9a2ba40657
commit c6a972734e
13 changed files with 1529 additions and 11 deletions

View File

@@ -13,9 +13,23 @@ This slice also defines the runnable Task wrapper (natural-language intent, para
## Acceptance criteria
- [ ] Running one cell provisions and seeds a fresh repository, runs the agent under its arm's tool with the guard active, and deletes the repository afterward.
- [ ] The run is bounded by both a turn cap and a wall-clock backstop; exceeding either records a failure tagged confused-versus-hung.
- [ ] The recorded sample carries the four token components (including the auxiliary small model), turns, duration, imputed cost, and the checker's pass/fail outcome.
- [ ] The post-run repository state is captured as a snapshot and scored by the checker against the task's scoring spec.
- [ ] A transcript audit runs after each cell; a run in which a foreign tool was reached is flagged invalid instead of scored.
- [ ] A runnable Task wrapper is defined and one sample task runs the full path against the live host.
- [x] Running one cell provisions and seeds a fresh repository, runs the agent under its arm's tool with the guard active, and deletes the repository afterward.
- [x] The run is bounded by both a turn cap and a wall-clock backstop; exceeding either records a failure tagged confused-versus-hung.
- [x] The recorded sample carries the four token components (including the auxiliary small model), turns, duration, imputed cost, and the checker's pass/fail outcome.
- [x] The post-run repository state is captured as a snapshot and scored by the checker against the task's scoring spec.
- [x] A transcript audit runs after each cell; a run in which a foreign tool was reached is flagged invalid instead of scored.
- [x] A runnable Task wrapper is defined and one sample task runs the full path against the live host.
## Implementation Notes
- **Two seams for the live boundaries.** The runner's orchestration (`bench/runner.ts`, `runCell`) is unit-tested with fakes by factoring the two non-deterministic boundaries behind interfaces: `BenchHost` (provision/seed/capture/delete against live Gitea, implemented by `liveBenchHost` in `bench/host.ts` over `seed.ts` + `snapshot.ts`) and `AgentDriver` (the model run, implemented by `sdkAgentDriver` in `bench/sdk-driver.ts` over the Claude Agent SDK). This mirrors the spec's split: run orchestration is the live boundary validated by the transcript audit and the smoke run, not by unit tests.
- **Claude Agent SDK is an optional peer.** The SDK (`@anthropic-ai/claude-agent-sdk`) is loaded via a computed dynamic import, so it is not a hard dependency of the (unshipped) `bench/` harness and neither the deterministic tier nor `npm run typecheck` requires it to be installed. The runner smoke tier (`bench/runner.smoke.test.ts`) skips cleanly when the SDK is absent or `GITEA_AXI_BENCH_LOGIN` is unset — a skip counts as a pass, matching the seed smoke tier. A real run additionally needs the `gitea-axi` CLI on `PATH` and a Claude subscription. Formalizing the SDK as a declared dependency belongs to the run-loop CLI slice (0029), which is the first consumer that runs it in earnest.
- **Temperature zero.** `sdkAgentDriver` passes `temperature: 0` in the SDK query options per the runner-and-metrics spec, so the comparison measures the tool rather than sampling noise. If a given SDK version does not surface a temperature knob, the field is additive and determinism falls back to the runtime default.
- **Shared isolation predicate.** Both isolation enforcement points share one `foreignToolReason(arm, use)` in `bench/audit.ts` so they cannot drift: the driver consults it in-band to deny a foreign tool before it runs, and the runner's `auditTranscript` re-applies it post-run as the independent backstop. Only tools that were permitted to run are recorded in the transcript, so a guard-blocked attempt is realistic wasted effort, not a leak — matching the spec's tool-isolation note that "blocked attempts are left in the transcript and count as realistic wasted effort."
- **Snapshot review verbs follow Gitea.** `bench/snapshot.ts` maps review states using Gitea's `ReviewStateType` verbs (`APPROVED` / `COMMENT` / `REQUEST_CHANGES`), which Gitea returns on read as well as on the write event (unlike GitHub's `CHANGES_REQUESTED`). Inline review comments are captured as empty, matching the single-user seed's declared ground truth, which never populates them. This capture is a live boundary exercised by the smoke tier rather than mocked.
- **No criteria dropped.** All six acceptance criteria are satisfied. The sample task is a single-mutation task (close a seeded issue), chosen so the "captured as a snapshot and scored by the checker" criterion is demonstrated through the full-state diff. The "runs the full path against the live host" criterion is realized by the runner smoke tier, which skips-as-pass when no host/SDK is configured, consistent with the project's e2e and seed tiers.