feat: add benchmark reporting command (task 0031)
All checks were successful
CI / test (pull_request) Successful in 52s
CI / test (push) Successful in 52s

Add the maintainer-facing `bench:report` command, the offline counterpart
to `bench:run`: it opens the sample store, drains it, aggregates against the
scored suite and bonus definitions, and prints the aggregator's comparison to
stdout. It renders whatever has accumulated, annotating incomplete coverage
rather than blocking on a complete matrix.

`parseReportArgs` is the pure argument seam (`--store`, `--help`, plus a
`--self-review` / `--no-self-review` variant selector). Unlike `bench:run` the
boundary is offline — it reads only the local store, no host or Agent SDK — so
the whole `runReportCommand` is deterministic and unit-tested, not smoke-run.

Move `DEFAULT_STORE_ROOT` to `store.ts` as the single source of truth,
re-exported from `run.ts` for its existing importers.
This commit was merged in pull request #32.
This commit is contained in:
2026-07-16 16:30:20 -04:00
parent 7216d3cc31
commit 53342f67e3
7 changed files with 407 additions and 7 deletions

View File

@@ -27,18 +27,19 @@ import { DEFAULT_TRIALS, REPORTING_FLOOR, runCells, type RunCellsResult } from "
import { resolveBenchAccess } from "./seed.js";
import { detectSelfReviewSupport } from "./self-review.js";
import { sdkAgentDriver } from "./sdk-driver.js";
import { createSampleStore } from "./store.js";
import { createSampleStore, DEFAULT_STORE_ROOT } from "./store.js";
import { buildScoredSuite } from "./task-suite.js";
// Re-exported so this command's existing importers keep resolving it from here;
// its single source of truth is now the store, which owns the default root.
export { DEFAULT_STORE_ROOT };
/** The default turn cap a run is bounded by when not overridden. */
export const DEFAULT_TURN_CAP = 40;
/** The default wall-clock backstop (ms) a run is bounded by when not overridden. */
export const DEFAULT_WALL_CLOCK_MS = 300_000;
/** Where accumulated samples are stored when `--store` is not given. */
export const DEFAULT_STORE_ROOT = "bench/results";
/** Environment variable naming the tea login the benchmark authenticates through. */
export const LOGIN_ENV = "GITEA_AXI_BENCH_LOGIN";