alexion/gitea-axi
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add benchmark aggregator and reporting (task 0030)
All checks were successful
CI / test (pull_request) Successful in 51s
Details
CI / test (push) Successful in 50s
Details

Browse Source 
Add bench/aggregate.ts: the pure aggregator seam that rolls the
accumulated sample store into a readable comparison. aggregate produces
a headline table (one row per arm, cost-equivalent tokens as the
headline plus raw tokens, turns, duration, success rate, coverage, and
imputed cost as a de-emphasized secondary column), per-tier and
per-token-component breakdowns, and the separate bonus table; renderReport
renders it as stable text. Cost-equivalent tokens are weighted at render
time from the four retained components per ADR 0014, and incomplete
coverage is annotated rather than hidden. Unit-tested against synthetic
append-only sample stores.
This commit was merged in pull request #31.
This commit is contained in:
alexion
2026-07-16 12:46:03 -04:00
parent 80a4fafa06
commit 7216d3cc31
4 changed files with 1027 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
bench/README.md
View File

@@ -49,8 +49,9 @@ The raw component breakdown is retained on every sample so the data can be re-we
- `sdk-driver.ts` — `sdkAgentDriver`, the production `AgentDriver`: it runs one arm through the Claude Agent SDK on the maintainer's subscription, enforcing isolation in-band via the SDK's permission callback (the arm's guard on every Bash command; the shell disabled on the MCP arm) and reporting the four token components (folding in the auxiliary small model), the turn count, the imputed cost, the transcript, and the final report. The SDK is loaded through a computed dynamic import so it is an optional peer needed only for live runs.
- `run-loop.ts` — `runCells`, the run loop: it runs one chosen `(arm, task)` cell for a batch of trials (defaulting to five, with a reporting floor of three) by driving `runCell` and the sample store, deciding only how many trials to run and at what trial numbers. Deepening a cell continues numbering past the highest trial it already holds and appends, so a cell's sample size grows across sittings rather than being overwritten. It reimplements no orchestration — provision, run, score, and append stay in `runCell`.
- `run.ts` — the maintainer-facing run-loop command. `parseRunArgs` is the pure, unit-tested argument seam; `runBenchCommand` is the live boundary that resolves host access, resolves the scored suite against the host's self-review support, selects the task, and drives `runCells`. Invoked via `npm run bench:run` (see below).
- `aggregate.ts` — the aggregator: the pure seam that renders the accumulated sample store into a readable comparison. `readAllSamples` drains a store into a flat record list; `aggregate` rolls the records up against the task definitions into a `Report` (headline, per-tier and per-token-component breakdowns, and the bonus table); `renderReport` renders that report as stable text. The headline metric, `costEquivalentTokens`, is computed here at render time by weighting the four retained token components by ADR 0014's pricing ratios (fresh input 1×, cache-write 1.25×, cache-read 0.1×, output 5×), so the stored records can be re-weighted without re-running. It reads whatever samples exist and annotates incomplete coverage — cells below the reporting floor (partial) and unsampled cells (missing) — rather than blocking on a complete matrix. A pure function of the records plus the definitions, unit-tested against synthetic append-only sample stores.
Later slices add the aggregator.
The aggregator has no command wrapper yet; a `bench:report` CLI over it is a natural follow-up, in the same shape as `run.ts` is over `run-loop.ts`.
## Running a cell