feat: add benchmark aggregator and reporting (task 0030) #31

alexion · 2026-07-16T12:46:42-04:00

alexion commented

2026-07-16 12:46:42 -04:00

Implements .claude/tasks/0030-bench-aggregator-and-reporting.md.

Summary

Adds bench/aggregate.ts, the pure aggregator seam that renders the accumulated benchmark sample store into a readable comparison:

Headline table — one row per arm with cost-equivalent tokens as the headline metric, plus raw tokens, turns, duration, success rate, and a coverage figure, with imputed cost as a de-emphasized (parenthesized, ~$) secondary column. Arms with no samples render as an em dash rather than a misleading zero.
Cost-equivalent tokens — weighted at render time from the four retained components per ADR 0014 ({ freshInput: 1, cacheCreation: 1.25, cacheRead: 0.1, output: 5 }), so stored records can be re-weighted without re-running. The 1.25× cache-write weight is the 5-minute-TTL multiplier the single un-TTL'd cacheCreation component maps to.
Partial matrix — renderReport renders whatever exists and annotates each arm's coverage as complete or incomplete with its partial (below-floor) and missing (unsampled) cell counts, so a half-run matrix never reads as complete.
Supporting views — per-tier and per-token-component breakdowns and the separate bonus table, all from the same records.
readAllSamples(store) drains a SampleStore into the flat record list aggregate consumes.

Unit-tested with 9 tests, including an append-order-stability test driving two real createSampleStore instances.

Deviations

Aggregator seam only, no CLI. The task's acceptance criteria are entirely about the pure aggregator, and bench/ slices split their command wiring into their own tasks (task 0029 was the dedicated run-loop CLI). A bench:report command over this seam is flagged as a natural follow-up in bench/README.md rather than folded in here.
aggregate takes the suite as Pick<BenchTask, "id" | "tier">, not the full BenchTask, since it only needs each task's id and tier and never scores — cleaner typing that the real suite still satisfies.

Review

Overall: Low

Blast radius: Low — two new dev-only files + README edit; no existing code touched, no callers yet.
Reversibility: Low — purely additive new files, trivially removable; no migrations/API/schema changes.
Test coverage: Low — thorough unit tests cover each metric, coverage annotation, rendering, and append-order stability against a real store.
Sensitive domain: Low — pure functions over in-memory records; no auth/security/permissions/concurrency.
Size & complexity: Low — ~450 lines of straightforward pure aggregation/formatting.
Runtime criticality: Low — bench/ is dev-only, excluded from the published package.

Unaddressed findings

Spec: none.

Standards (all judgement-call smells, left as deliberate trade-offs):

Duplicated Code — the per-arm records.filter(...) + metric-trio shape recurs across the headline/tier/component/bonus passes. Left flat because the four view interfaces genuinely diverge in their extra fields; the shared cost-equivalent-mean is already factored into meanCostEquivalent.
Data Clumps — { samples, costEquivalentTokens, successRate } travels together across the view types. Not extracted because each view carries different extra fields (headline: turns/duration/imputed; tier: coverage), so a shared base would obscure more than it saves.
Repeated Switches — the four token components are enumerated by hand in four places. Left explicit for readability; driving them off the weights' keys would centralize at the cost of the named, legible arithmetic.

Implements `.claude/tasks/0030-bench-aggregator-and-reporting.md`. ## Summary Adds `bench/aggregate.ts`, the pure aggregator seam that renders the accumulated benchmark sample store into a readable comparison: - **Headline table** — one row per arm with cost-equivalent tokens as the headline metric, plus raw tokens, turns, duration, success rate, and a coverage figure, with imputed cost as a de-emphasized (parenthesized, `~$`) secondary column. Arms with no samples render as an em dash rather than a misleading zero. - **Cost-equivalent tokens** — weighted at render time from the four retained components per ADR 0014 (`{ freshInput: 1, cacheCreation: 1.25, cacheRead: 0.1, output: 5 }`), so stored records can be re-weighted without re-running. The 1.25× cache-write weight is the 5-minute-TTL multiplier the single un-TTL'd `cacheCreation` component maps to. - **Partial matrix** — `renderReport` renders whatever exists and annotates each arm's coverage as complete or incomplete with its partial (below-floor) and missing (unsampled) cell counts, so a half-run matrix never reads as complete. - **Supporting views** — per-tier and per-token-component breakdowns and the separate bonus table, all from the same records. - `readAllSamples(store)` drains a `SampleStore` into the flat record list `aggregate` consumes. Unit-tested with 9 tests, including an append-order-stability test driving two real `createSampleStore` instances. ## Deviations - **Aggregator seam only, no CLI.** The task's acceptance criteria are entirely about the pure aggregator, and `bench/` slices split their command wiring into their own tasks (task 0029 was the dedicated run-loop CLI). A `bench:report` command over this seam is flagged as a natural follow-up in `bench/README.md` rather than folded in here. - `aggregate` takes the suite as `Pick<BenchTask, "id" | "tier">`, not the full `BenchTask`, since it only needs each task's id and tier and never scores — cleaner typing that the real suite still satisfies. ## Review **Overall: Low** - Blast radius: Low — two new dev-only files + README edit; no existing code touched, no callers yet. - Reversibility: Low — purely additive new files, trivially removable; no migrations/API/schema changes. - Test coverage: Low — thorough unit tests cover each metric, coverage annotation, rendering, and append-order stability against a real store. - Sensitive domain: Low — pure functions over in-memory records; no auth/security/permissions/concurrency. - Size & complexity: Low — ~450 lines of straightforward pure aggregation/formatting. - Runtime criticality: Low — `bench/` is dev-only, excluded from the published package. ### Unaddressed findings Spec: none. Standards (all judgement-call smells, left as deliberate trade-offs): - **Duplicated Code** — the per-arm `records.filter(...)` + metric-trio shape recurs across the headline/tier/component/bonus passes. Left flat because the four view interfaces genuinely diverge in their extra fields; the shared cost-equivalent-mean is already factored into `meanCostEquivalent`. - **Data Clumps** — `{ samples, costEquivalentTokens, successRate }` travels together across the view types. Not extracted because each view carries different extra fields (headline: turns/duration/imputed; tier: coverage), so a shared base would obscure more than it saves. - **Repeated Switches** — the four token components are enumerated by hand in four places. Left explicit for readability; driving them off the weights' keys would centralize at the cost of the named, legible arithmetic.

alexion added 1 commit 2026-07-16 12:46:42 -04:00

feat: add benchmark aggregator and reporting (task 0030)

CI / test (pull_request) Successful in 51s

Details

CI / test (push) Successful in 50s

Details

7216d3cc31

Add bench/aggregate.ts: the pure aggregator seam that rolls the
accumulated sample store into a readable comparison. aggregate produces
a headline table (one row per arm, cost-equivalent tokens as the
headline plus raw tokens, turns, duration, success rate, coverage, and
imputed cost as a de-emphasized secondary column), per-tier and
per-token-component breakdowns, and the separate bonus table; renderReport
renders it as stable text. Cost-equivalent tokens are weighted at render
time from the four retained components per ADR 0014, and incomplete
coverage is annotated rather than hidden. Unit-tested against synthetic
append-only sample stores.

alexion merged commit 7216d3cc31 into main

2026-07-16 16:14:06 -04:00

alexion deleted branch task-0030-bench-aggregator-and-reporting

2026-07-16 16:14:06 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: alexion/gitea-axi#31