feat: add benchmark run-loop command (task 0029)
All checks were successful
CI / test (pull_request) Successful in 51s
CI / test (push) Successful in 52s

Add the maintainer-facing command that runs one chosen benchmark cell on
demand, so only the token budget available at that moment is spent.

runCells (bench/run-loop.ts) runs one (arm, task) cell for a batch of
trials — defaulting to five with a reporting floor of three — by driving
the existing single-cell runner and the append-only sample store rather
than reimplementing orchestration. Re-running a cell deepens it: trial
numbering continues past the highest trial the cell already holds and the
new samples append, so a cell's sample size grows across sittings without
overwriting prior runs.

bench/run.ts is the command: parseRunArgs is the pure, unit-tested
argument seam, and runBenchCommand is the live boundary that resolves host
access, resolves the scored suite against the host's self-review support,
selects the task, and drives the run loop. It is invoked via the new
bench:run npm script, run under tsx (a new devDependency) because the
harness's .js-specifier imports need a TypeScript-aware runner. The Claude
Agent SDK is now declared as an optional peerDependency — documented but
neither installed for package consumers nor pulled into CI.

Every arm runs on the driver's single fixed model; the command exposes no
per-cell model override that could break cross-arm comparability. The
default store root bench/results/ is gitignored.
This commit was merged in pull request #30.
This commit is contained in:
2026-07-16 10:44:52 -04:00
parent e8310fb616
commit 80a4fafa06
9 changed files with 784 additions and 6 deletions

View File

@@ -36,7 +36,8 @@
"test:e2e": "vitest run --config vitest.e2e.config.ts",
"test:pack": "vitest run --config vitest.packaging.config.ts",
"test:bench": "vitest run --config vitest.bench.config.ts",
"test:bench:smoke": "vitest run --config vitest.bench-smoke.config.ts"
"test:bench:smoke": "vitest run --config vitest.bench-smoke.config.ts",
"bench:run": "tsx bench/run.ts"
},
"dependencies": {
"@toon-format/toon": "^2.3.0",
@@ -46,7 +47,16 @@
"devDependencies": {
"@types/node": "^20.19.0",
"@vitest/coverage-v8": "^3.2.7",
"tsx": "^4.23.1",
"typescript": "^5.8.0",
"vitest": "^3.2.0"
},
"peerDependencies": {
"@anthropic-ai/claude-agent-sdk": ">=0.3.0"
},
"peerDependenciesMeta": {
"@anthropic-ai/claude-agent-sdk": {
"optional": true
}
}
}