feat: add benchmark scaffold and result store (task 0022)
All checks were successful
CI / test (pull_request) Successful in 50s
CI / test (push) Successful in 50s

Lay the foundation the benchmark harness reads and writes: a bench/
directory (excluded from the published npm package), the immutable
result-record shape, and an append-only per-cell sample store.

- bench/result.ts: the ResultRecord shape — four token components,
  turns, duration, imputed cost, tagged pass/fail outcome, and the
  arm/task/tier/trial/timestamp tags. Arm and Tier are typed unions.
- bench/store.ts: append-only sample store, one JSONL file per cell at
  <root>/<arm>/<taskId>.jsonl; deepening a cell only ever adds samples.
- bench/README.md: harness working docs and benchmark vocabulary, kept
  out of the tool's domain glossary per the spec.
- Dedicated bench test tier (vitest.bench.config.ts, npm run test:bench)
  kept out of the fast tier; tsconfig typechecks bench.
- Packaging tier asserts bench/ never ships in the tarball.
This commit was merged in pull request #23.
This commit is contained in:
2026-07-15 10:20:15 -04:00
parent d445b2bd2b
commit 9bf8c85dc3
9 changed files with 355 additions and 6 deletions

13
vitest.bench.config.ts Normal file
View File

@@ -0,0 +1,13 @@
import { defineConfig } from "vitest/config";
export default defineConfig({
test: {
// The benchmark harness tier. The harness lives in bench/ (excluded from the
// published npm package) and its deterministic seams — the sample store,
// checker, guard, and aggregator — are unit-tested here, colocated with the
// source. It runs on its own via `test:bench` and is kept out of the main
// fast tier so bench code never counts against src coverage thresholds.
include: ["bench/**/*.test.ts"],
passWithNoTests: true,
},
});