feat: add benchmark single-cell runner (task 0027) #28

alexion · 2026-07-16T09:18:22-04:00

alexion commented

2026-07-16 09:18:22 -04:00

Implements .claude/tasks/0027-bench-single-cell-runner.md.

Summary

The tracer bullet that threads every benchmark layer to run one (arm, task, trial) cell end to end and record an immutable result — proving seed, arm scaffolding, guard, runner, checker, and store all connect.

runCell (bench/runner.ts) provisions and seeds a fresh throwaway repository, runs the agent under exactly the active arm's tool with the guard enforced, bounds the run by a turn cap and a wall-clock backstop (tagging a failure confused-versus-hung), audits the transcript, captures the post-run state and scores it with the checker, appends the sample, and always deletes the repository. A run in which a foreign tool was reached is flagged invalid rather than scored.

The two non-deterministic boundaries are factored behind seams so the orchestration is unit-tested with fakes while the live wiring is validated by a smoke run:

BenchHost — liveBenchHost (host.ts) over seed.ts (provision/seed/delete) and the new snapshot.ts (captureRepoState, the seed's read-back counterpart).
AgentDriver — sdkAgentDriver (sdk-driver.ts) over the Claude Agent SDK, enforcing isolation in-band via the SDK permission callback and reporting the four token components (folding in the auxiliary small model), turns, duration, imputed cost, transcript, and final report.

Also adds task.ts (the runnable BenchTask wrapper and one sample single-mutation task) and audit.ts (the post-run transcript audit plus the shared foreignToolReason predicate both enforcement points consume).

Deviations

Claude Agent SDK is an optional peer, loaded via a computed dynamic import, so the deterministic tier and npm run typecheck never require it. The runner smoke tier (runner.smoke.test.ts) skips cleanly when the SDK is absent or GITEA_AXI_BENCH_LOGIN is unset; a real run also needs the gitea-axi CLI on PATH and a Claude subscription. Declaring the SDK as a dependency belongs to the run-loop CLI slice (0029), its first real consumer.
The sample task is a single-mutation task (close a seeded issue), chosen so the snapshot-and-score criterion is demonstrated through the full-state diff.

Review

Risk

Overall: MEDIUM

Blast radius: Low — entirely new files under bench/ plus one send visibility bump in seed.ts; nothing in src/ or the shipped package is touched.
Reversibility: Low — all additions, no migrations/deletions/published API; the send export is superset-compatible.
Test coverage: Low — runner.ts/audit.ts/task.ts have deterministic unit tests (happy path, turn-cap, hung, leak, incorrect); live seams covered by the skip-clean smoke tier.
Sensitive domain: Medium — a tool-isolation guard/audit backstop plus a permission callback and credential-bearing host access, though dev-only and defense-in-depth.
Size & complexity: Medium — ~1600 lines, mostly declarative; the one tangled spot is runBounded's abort/timer race.
Runtime criticality: Low — benchmark harness, excluded from dist, never runs in production.

Standards findings (unaddressed)

Duplicated Code (minor, judgement call) — snapshot.ts's captureComments and the two capture loops share a labels/assignees/comments shape. Left as-is per the reviewer's own recommendation ("Leave it"): the pull loop additionally captures reviews and PR state, so extracting would be a thin, low-value abstraction.

(The flagged duplication between the driver's enforcement and the audit was addressed by extracting the shared foreignToolReason predicate.)

Spec findings (unaddressed)

Snapshot review-state verb (live-boundary, to verify) — snapshot.ts maps review states with Gitea's ReviewStateType verbs (APPROVED/COMMENT/REQUEST_CHANGES). Not changed because Gitea returns REQUEST_CHANGES on read as well as write (unlike GitHub's CHANGES_REQUESTED); it does not affect the current sample task (no reviews) and is exercised by the smoke tier against the live host. The temperature-zero and duration-assertion findings were addressed.

Implements `.claude/tasks/0027-bench-single-cell-runner.md`. ## Summary The tracer bullet that threads every benchmark layer to run one `(arm, task, trial)` cell end to end and record an immutable result — proving seed, arm scaffolding, guard, runner, checker, and store all connect. `runCell` (`bench/runner.ts`) provisions and seeds a fresh throwaway repository, runs the agent under exactly the active arm's tool with the guard enforced, bounds the run by a turn cap and a wall-clock backstop (tagging a failure confused-versus-hung), audits the transcript, captures the post-run state and scores it with the checker, appends the sample, and always deletes the repository. A run in which a foreign tool was reached is flagged invalid rather than scored. The two non-deterministic boundaries are factored behind seams so the orchestration is unit-tested with fakes while the live wiring is validated by a smoke run: - `BenchHost` — `liveBenchHost` (`host.ts`) over `seed.ts` (provision/seed/delete) and the new `snapshot.ts` (`captureRepoState`, the seed's read-back counterpart). - `AgentDriver` — `sdkAgentDriver` (`sdk-driver.ts`) over the Claude Agent SDK, enforcing isolation in-band via the SDK permission callback and reporting the four token components (folding in the auxiliary small model), turns, duration, imputed cost, transcript, and final report. Also adds `task.ts` (the runnable `BenchTask` wrapper and one sample single-mutation task) and `audit.ts` (the post-run transcript audit plus the shared `foreignToolReason` predicate both enforcement points consume). ### Deviations - **Claude Agent SDK is an optional peer**, loaded via a computed dynamic import, so the deterministic tier and `npm run typecheck` never require it. The runner smoke tier (`runner.smoke.test.ts`) skips cleanly when the SDK is absent or `GITEA_AXI_BENCH_LOGIN` is unset; a real run also needs the `gitea-axi` CLI on `PATH` and a Claude subscription. Declaring the SDK as a dependency belongs to the run-loop CLI slice (0029), its first real consumer. - The sample task is a **single-mutation** task (close a seeded issue), chosen so the snapshot-and-score criterion is demonstrated through the full-state diff. ## Review ### Risk **Overall: MEDIUM** - Blast radius: Low — entirely new files under `bench/` plus one `send` visibility bump in `seed.ts`; nothing in `src/` or the shipped package is touched. - Reversibility: Low — all additions, no migrations/deletions/published API; the `send` export is superset-compatible. - Test coverage: Low — `runner.ts`/`audit.ts`/`task.ts` have deterministic unit tests (happy path, turn-cap, hung, leak, incorrect); live seams covered by the skip-clean smoke tier. - Sensitive domain: Medium — a tool-isolation guard/audit backstop plus a permission callback and credential-bearing host access, though dev-only and defense-in-depth. - Size & complexity: Medium — ~1600 lines, mostly declarative; the one tangled spot is `runBounded`'s abort/timer race. - Runtime criticality: Low — benchmark harness, excluded from `dist`, never runs in production. ### Standards findings (unaddressed) - **Duplicated Code (minor, judgement call)** — `snapshot.ts`'s `captureComments` and the two capture loops share a labels/assignees/comments shape. Left as-is per the reviewer's own recommendation ("Leave it"): the pull loop additionally captures reviews and PR state, so extracting would be a thin, low-value abstraction. (The flagged duplication between the driver's enforcement and the audit was addressed by extracting the shared `foreignToolReason` predicate.) ### Spec findings (unaddressed) - **Snapshot review-state verb (live-boundary, to verify)** — `snapshot.ts` maps review states with Gitea's `ReviewStateType` verbs (`APPROVED`/`COMMENT`/`REQUEST_CHANGES`). Not changed because Gitea returns `REQUEST_CHANGES` on read as well as write (unlike GitHub's `CHANGES_REQUESTED`); it does not affect the current sample task (no reviews) and is exercised by the smoke tier against the live host. The temperature-zero and duration-assertion findings were addressed.

alexion added 1 commit 2026-07-16 09:18:22 -04:00

feat: add benchmark single-cell runner (task 0027)

CI / test (pull_request) Successful in 50s

Details

CI / test (push) Successful in 52s

Details

c6a972734e

Thread every benchmark layer to run one (arm, task, trial) cell end to
end: provision and seed a throwaway repository, run the agent under the
active arm bounded by a turn cap and a wall-clock backstop, audit the
transcript, capture and score the post-run state, append the sample, and
delete the repository.

- runner.ts: runCell orchestration behind the BenchHost and AgentDriver
  seams, so the flow is unit-tested with fakes while the live wiring is
  validated by a smoke run; turn-cap and wall-clock failures are tagged
  confused-versus-hung, and a leaked transcript is flagged invalid.
- audit.ts: the post-run transcript audit plus the shared
  foreignToolReason predicate both isolation enforcement points consume.
- task.ts: the runnable BenchTask wrapper and one sample single-mutation
  task exercising the full path.
- snapshot.ts: captureRepoState, the seed's read-back counterpart, in the
  RepoState shape the checker diffs against.
- host.ts / sdk-driver.ts: the live BenchHost and the Claude Agent SDK
  driver (an optional peer, loaded via dynamic import) for real runs.
- runner.smoke.test.ts: the live tracer-bullet tier, skipping cleanly
  when no host or SDK is configured.

alexion merged commit c6a972734e into main

2026-07-16 09:19:30 -04:00

alexion deleted branch task-0027-bench-single-cell-runner

2026-07-16 09:19:30 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: alexion/gitea-axi#28