feat: add benchmark scaffold and result store (task 0022) #23

alexion · 2026-07-15T10:20:54-04:00

alexion commented

2026-07-15 10:20:54 -04:00

Task: .claude/tasks/0022-bench-scaffold-and-result-store.md

Summary

Lays the foundation the whole benchmark harness reads and writes:

bench/ directory, excluded from the published npm package by the existing files allow-list; a new packaging-tier assertion locks in that package/bench never ships.
bench/result.ts — the immutable ResultRecord shape: the four token components, turns, duration, imputed cost, a tagged pass/fail outcome, and the arm/task/tier/trial/timestamp tags. Arm and Tier are typed unions; the failure tag is "incorrect" | "confused" | "hung" (confused = turn cap, hung = wall-clock backstop, plus incorrect for the checker-scored-wrong case the later runner records).
bench/store.ts — an append-only sample store, one newline-delimited JSON file per cell at <root>/<arm>/<taskId>.jsonl. Immutability is structural: only append/read/cells are exposed and append is a bare file append, so deepening a cell can only add samples. cells() enumeration is foundational for the aggregator slice (0030).
bench/README.md — harness working docs plus the benchmark vocabulary, deliberately kept out of the tool's domain glossary per the spec.
A dedicated bench test tier (vitest.bench.config.ts, npm run test:bench) colocated with the source and kept out of the fast tier, so harness code never counts against the src/ coverage thresholds; tsconfig.json now typechecks bench.

All four acceptance criteria are met. No deviations from the plan; the only additions beyond the literal criteria (CellKey, store.cells(), the README) are foundational for later slices and called out in the task's Implementation Notes.

Review

Risk

Overall: Low

Blast radius: Low — all-new isolated bench/ files plus two tiny additive edits (package.json script, tsconfig.json include); no existing src/ callers touched.
Reversibility: Low — entirely additive new files and config lines; deleting the directory and reverting two lines fully undoes it.
Test coverage: Low — store.ts covered by a thorough colocated store.test.ts; packaging exclusion asserted; result.ts is types only.
Sensitive domain: Low — no auth, permissions, or migrations; only local temp-dir filesystem I/O in dev tooling.
Size & complexity: Low — ~400 lines, mostly new files and docs; store logic is simple JSONL append/read.
Runtime criticality: Low — benchmark harness excluded from the published package and kept out of the fast test tier; dev-only.

Standards — unaddressed findings

Primitive Obsession (judgement call, left as-is): cells() casts a raw directory name straight to the Arm union (arm as CellKey["arm"]) without validation, so a stray directory outside the four arms would be silently mis-typed. Left intentionally: the store is internal harness code, it writes only under known arms, and the <root>/<arm>/ layout is documented — validation here would be speculative. (The two Duplicated Code findings — append bypassing cellPath, and the repeated ENOENT handling — were fixed in the diff.)

Spec

No findings — the diff faithfully implements the task with no gaps, no scope creep, and no wrong behaviour.

Task: `.claude/tasks/0022-bench-scaffold-and-result-store.md` ## Summary Lays the foundation the whole benchmark harness reads and writes: - **`bench/` directory**, excluded from the published npm package by the existing `files` allow-list; a new packaging-tier assertion locks in that `package/bench` never ships. - **`bench/result.ts`** — the immutable `ResultRecord` shape: the four token components, turns, duration, imputed cost, a tagged pass/fail outcome, and the arm/task/tier/trial/timestamp tags. `Arm` and `Tier` are typed unions; the failure tag is `"incorrect" | "confused" | "hung"` (confused = turn cap, hung = wall-clock backstop, plus `incorrect` for the checker-scored-wrong case the later runner records). - **`bench/store.ts`** — an append-only sample store, one newline-delimited JSON file per cell at `<root>/<arm>/<taskId>.jsonl`. Immutability is structural: only `append`/`read`/`cells` are exposed and append is a bare file append, so deepening a cell can only add samples. `cells()` enumeration is foundational for the aggregator slice (0030). - **`bench/README.md`** — harness working docs plus the benchmark vocabulary, deliberately kept out of the tool's domain glossary per the spec. - A dedicated bench test tier (`vitest.bench.config.ts`, `npm run test:bench`) colocated with the source and kept out of the fast tier, so harness code never counts against the `src/` coverage thresholds; `tsconfig.json` now typechecks `bench`. All four acceptance criteria are met. No deviations from the plan; the only additions beyond the literal criteria (`CellKey`, `store.cells()`, the README) are foundational for later slices and called out in the task's Implementation Notes. ## Review ### Risk **Overall: Low** - Blast radius: Low — all-new isolated `bench/` files plus two tiny additive edits (`package.json` script, `tsconfig.json` include); no existing `src/` callers touched. - Reversibility: Low — entirely additive new files and config lines; deleting the directory and reverting two lines fully undoes it. - Test coverage: Low — `store.ts` covered by a thorough colocated `store.test.ts`; packaging exclusion asserted; `result.ts` is types only. - Sensitive domain: Low — no auth, permissions, or migrations; only local temp-dir filesystem I/O in dev tooling. - Size & complexity: Low — ~400 lines, mostly new files and docs; store logic is simple JSONL append/read. - Runtime criticality: Low — benchmark harness excluded from the published package and kept out of the fast test tier; dev-only. ### Standards — unaddressed findings - **Primitive Obsession (judgement call, left as-is):** `cells()` casts a raw directory name straight to the `Arm` union (`arm as CellKey["arm"]`) without validation, so a stray directory outside the four arms would be silently mis-typed. Left intentionally: the store is internal harness code, it writes only under known arms, and the `<root>/<arm>/` layout is documented — validation here would be speculative. (The two Duplicated Code findings — `append` bypassing `cellPath`, and the repeated ENOENT handling — were fixed in the diff.) ### Spec No findings — the diff faithfully implements the task with no gaps, no scope creep, and no wrong behaviour.

alexion added 1 commit 2026-07-15 10:20:54 -04:00

feat: add benchmark scaffold and result store (task 0022)

CI / test (pull_request) Successful in 50s

Details

CI / test (push) Successful in 50s

Details

9bf8c85dc3

Lay the foundation the benchmark harness reads and writes: a bench/
directory (excluded from the published npm package), the immutable
result-record shape, and an append-only per-cell sample store.

- bench/result.ts: the ResultRecord shape — four token components,
  turns, duration, imputed cost, tagged pass/fail outcome, and the
  arm/task/tier/trial/timestamp tags. Arm and Tier are typed unions.
- bench/store.ts: append-only sample store, one JSONL file per cell at
  <root>/<arm>/<taskId>.jsonl; deepening a cell only ever adds samples.
- bench/README.md: harness working docs and benchmark vocabulary, kept
  out of the tool's domain glossary per the spec.
- Dedicated bench test tier (vitest.bench.config.ts, npm run test:bench)
  kept out of the fast tier; tsconfig typechecks bench.
- Packaging tier asserts bench/ never ships in the tarball.

alexion merged commit 9bf8c85dc3 into main

2026-07-15 22:03:48 -04:00

alexion deleted branch task-0022-bench-scaffold-and-result-store

2026-07-15 22:03:48 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: alexion/gitea-axi#23