feat: add benchmark run-loop command (task 0029) #30

alexion · 2026-07-16T10:45:44-04:00

alexion commented

2026-07-16 10:45:44 -04:00

Task: .claude/tasks/0029-bench-run-loop-cli.md

Summary

Adds the maintainer-facing command that runs one chosen benchmark cell on demand, so only the token budget available at that moment is spent.

runCells (bench/run-loop.ts) runs one (arm, task) cell for a batch of trials — defaulting to five with a reporting floor of three — by driving the existing single-cell runCell and the append-only sample store rather than reimplementing orchestration.
Re-running a cell deepens it: trial numbering continues past the highest trial the cell already holds (robust to gaps left by invalid attempts), and the new samples append without overwriting priors.
bench/run.ts is the command: parseRunArgs is the pure, unit-tested argument seam; runBenchCommand is the live boundary that resolves host access, probes self-review support to resolve the scored suite, selects the task, and drives the run loop. Invoked via the new bench:run npm script under tsx.

Deviations from the plan

tsx devDependency + bench:run script: node's native TS stripping doesn't rewrite .js specifiers to .ts, which the whole bench/ tree relies on, so the command runs under tsx.
Claude Agent SDK declared as an optional peerDependency: documented but neither installed for package consumers nor pulled into CI's npm ci; the maintainer installs it for live runs.
--model flag dropped after review (see below); bench/results/ gitignored.
TDD sequencing: the run loop is a small cohesive unit, so its first GREEN carried the trial-numbering/floor logic; the later run-loop and parser tests are passing characterization/regression guards, each still written test-first from the public interface with independent expected literals.

Review

Risk

Overall: Medium

Blast radius: Low — self-contained new files in bench/; only shared touch is package.json/lockfile, and the new peer dep is optional so it can't break published-package installs.
Reversibility: Low — all-new files plus additive config; clean revert, no migrations/deletions; the store only appends.
Test coverage: Low — run-loop.ts and the pure parseRunArgs seam are well covered; the live runBenchCommand boundary is deliberately validated by running, not mocked.
Sensitive domain: Medium — resolves live tea credentials and spends a real Claude token budget; a loop bug or bad --trials burns budget against production credentials.
Size & complexity: Low — ~390 net new lines, a single trial loop plus flat arg parsing.
Runtime criticality: Low — dev-only maintainer tooling, excluded from the published package.

Standards — unaddressed (judgement calls, left as-is)

Multi-copy 5/3 defaults: the trial default and reporting floor appear in ~6 prose/test copies. Left deliberately — the code constants (DEFAULT_TRIALS, REPORTING_FLOOR) are single-sourced; the prose and test copies are intentional (tests must not recompute expected values from source).
Mild Primitive Obsession (arm + taskId as a loose pair): a Cell type could exist. Left as-is — the store already accepts the { arm, taskId } shape, and minting a suite-wide Cell type is broader than this slice warrants.
Minor Data Clumps (run-context septet forwarded verbatim): already grouped by the pre-existing RunCellInput; no new bundling needed.

Spec — unaddressed

Hung-vs-confused failure tags not surfaced in the CLI tally: left to the aggregator (slice 0030). The tags are on every stored sample; the tally intentionally reports only recorded/invalid counts and floor status.
--turn-cap / --wall-clock-ms are unrequested surface: kept deliberately. They are safe-defaulted bounds a maintainer may legitimately need to raise for a heavier task, and they do not affect cross-arm comparability (unlike --model, which was removed).

Task: `.claude/tasks/0029-bench-run-loop-cli.md` ## Summary Adds the maintainer-facing command that runs one chosen benchmark cell on demand, so only the token budget available at that moment is spent. - `runCells` (`bench/run-loop.ts`) runs one `(arm, task)` cell for a batch of trials — defaulting to five with a reporting floor of three — by driving the existing single-cell `runCell` and the append-only sample store rather than reimplementing orchestration. - Re-running a cell **deepens** it: trial numbering continues past the highest trial the cell already holds (robust to gaps left by invalid attempts), and the new samples append without overwriting priors. - `bench/run.ts` is the command: `parseRunArgs` is the pure, unit-tested argument seam; `runBenchCommand` is the live boundary that resolves host access, probes self-review support to resolve the scored suite, selects the task, and drives the run loop. Invoked via the new `bench:run` npm script under `tsx`. ### Deviations from the plan - **`tsx` devDependency + `bench:run` script:** node's native TS stripping doesn't rewrite `.js` specifiers to `.ts`, which the whole `bench/` tree relies on, so the command runs under `tsx`. - **Claude Agent SDK declared as an optional `peerDependency`:** documented but neither installed for package consumers nor pulled into CI's `npm ci`; the maintainer installs it for live runs. - **`--model` flag dropped after review** (see below); `bench/results/` gitignored. - **TDD sequencing:** the run loop is a small cohesive unit, so its first GREEN carried the trial-numbering/floor logic; the later run-loop and parser tests are passing characterization/regression guards, each still written test-first from the public interface with independent expected literals. ## Review ### Risk **Overall: Medium** - Blast radius: Low — self-contained new files in `bench/`; only shared touch is `package.json`/lockfile, and the new peer dep is optional so it can't break published-package installs. - Reversibility: Low — all-new files plus additive config; clean revert, no migrations/deletions; the store only appends. - Test coverage: Low — `run-loop.ts` and the pure `parseRunArgs` seam are well covered; the live `runBenchCommand` boundary is deliberately validated by running, not mocked. - Sensitive domain: Medium — resolves live tea credentials and spends a real Claude token budget; a loop bug or bad `--trials` burns budget against production credentials. - Size & complexity: Low — ~390 net new lines, a single trial loop plus flat arg parsing. - Runtime criticality: Low — dev-only maintainer tooling, excluded from the published package. ### Standards — unaddressed (judgement calls, left as-is) - **Multi-copy `5`/`3` defaults:** the trial default and reporting floor appear in ~6 prose/test copies. Left deliberately — the code constants (`DEFAULT_TRIALS`, `REPORTING_FLOOR`) are single-sourced; the prose and test copies are intentional (tests must not recompute expected values from source). - **Mild Primitive Obsession (`arm` + `taskId` as a loose pair):** a `Cell` type could exist. Left as-is — the store already accepts the `{ arm, taskId }` shape, and minting a suite-wide `Cell` type is broader than this slice warrants. - **Minor Data Clumps (run-context septet forwarded verbatim):** already grouped by the pre-existing `RunCellInput`; no new bundling needed. ### Spec — unaddressed - **Hung-vs-confused failure tags not surfaced in the CLI tally:** left to the aggregator (slice 0030). The tags are on every stored sample; the tally intentionally reports only recorded/invalid counts and floor status. - **`--turn-cap` / `--wall-clock-ms` are unrequested surface:** kept deliberately. They are safe-defaulted bounds a maintainer may legitimately need to raise for a heavier task, and they do not affect cross-arm comparability (unlike `--model`, which was removed).

alexion added 1 commit 2026-07-16 10:45:45 -04:00

feat: add benchmark run-loop command (task 0029)

CI / test (pull_request) Successful in 51s

Details

CI / test (push) Successful in 52s

Details

80a4fafa06

Add the maintainer-facing command that runs one chosen benchmark cell on
demand, so only the token budget available at that moment is spent.

runCells (bench/run-loop.ts) runs one (arm, task) cell for a batch of
trials — defaulting to five with a reporting floor of three — by driving
the existing single-cell runner and the append-only sample store rather
than reimplementing orchestration. Re-running a cell deepens it: trial
numbering continues past the highest trial the cell already holds and the
new samples append, so a cell's sample size grows across sittings without
overwriting prior runs.

bench/run.ts is the command: parseRunArgs is the pure, unit-tested
argument seam, and runBenchCommand is the live boundary that resolves host
access, resolves the scored suite against the host's self-review support,
selects the task, and drives the run loop. It is invoked via the new
bench:run npm script, run under tsx (a new devDependency) because the
harness's .js-specifier imports need a TypeScript-aware runner. The Claude
Agent SDK is now declared as an optional peerDependency — documented but
neither installed for package consumers nor pulled into CI.

Every arm runs on the driver's single fixed model; the command exposes no
per-cell model override that could break cross-arm comparability. The
default store root bench/results/ is gitignored.

alexion merged commit 80a4fafa06 into main

2026-07-16 12:19:03 -04:00

alexion deleted branch task-0029-bench-run-loop-cli

2026-07-16 12:19:03 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: alexion/gitea-axi#30