alexion/gitea-axi
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add benchmark reporting command (task 0031) #32

Merged
alexion merged 1 commits from task-0031-bench-report-cli into main 2026-07-16 18:30:52 -04:00
Conversation 0 Commits 1 Files Changed 7 +407 -7
alexion commented 2026-07-16 16:30:58 -04:00
Owner
Copy Link

Implements .claude/tasks/0031-bench-report-cli.md.

Summary

Adds the maintainer-facing bench:report command — the reporting counterpart to bench:run — giving the aggregator seam from task 0030 a runnable entry point.
runReportCommand opens the sample store, drains it with readAllSamples, aggregates against the scored suite and bonus definitions, and prints renderReport(...) to stdout.
It renders whatever has accumulated, annotating incomplete coverage (unrun arms show an em dash) rather than blocking on a complete matrix — no aggregation, weighting, or rendering is reimplemented in the command.

parseReportArgs is the pure argument seam. bench/report.ts mirrors bench/run.ts (same main / direct-execution guard / usage helper / Parsed… union shape), and a bench:report npm script (tsx bench/report.ts) sits alongside bench:run.

Deviations from the plan

  • The boundary is offline, so the whole command is unit-tested. Criterion 5 assumed the run-command pattern where the live boundary is "validated by running it." But the report reads only the local store — no credentials, host, or Agent SDK — so runReportCommand is deterministic and covered by real unit tests over a temp-dir store (populated, empty, --help), on top of the parser tests. It was still run end-to-end (npm run bench:report). The unused deps parameter is kept only for signature parity with the command family.
  • Added --self-review / --no-self-review, beyond the written (--store, --help) seam. aggregate needs a suite and bonus, and buildScoredSuite/buildBonusTasks are parameterized by selfReviewPermitted. bench:run resolves this by probing the live host; the offline report has no host to probe, so the flag makes the choice explicit (default true) instead of silently hardcoding one variant. It is well-scoped — the scored coverage is identical either way, so it only selects the bonus capability catalog — and is documented in --help and the README. Criterion 5's wording was updated to record it.
  • DEFAULT_STORE_ROOT moved to store.ts as the single source of truth (both commands need it) and re-exported from run.ts so its existing importers are unchanged.

Full bench tier green (104 tests), typecheck clean.

Review

Three-axis /review-uncommitted.

Risk

Overall: Low

  • Blast radius: Low — a new isolated command plus one moved constant re-exported for compatibility; no existing callers change behavior.
  • Reversibility: Low — purely additive new files and a package.json script; trivial to revert, no migrations or published-API impact.
  • Test coverage: Low — the parser seam and the offline boundary (default/override/help/empty-store) are covered by tests in the diff.
  • Sensitive domain: Low — offline dev tooling reading a local sample store, explicitly no credentials, host, or SDK.
  • Size & complexity: Low — ~350 lines, mostly new; a simple linear argv loop and a thin boundary delegating to existing seams.
  • Runtime criticality: Low — maintainer-only reporting tool, not on any production path.

Standards — unaddressed findings

  • parseReportArgs flag loop partly overlaps parseRunArgs (judgement call, Duplicated Code). Left unextracted: the two have genuinely diverged (the report parser adds --no- boolean negation), so a shared parser would be premature generality — the reviewer's own recommendation.

Spec — unaddressed findings

  • None. The only spec finding (--self-review exceeding the written seam) was addressed by recording the decision in the task's Implementation Notes and updating the criterion wording; the reviewer judged the flag justified in substance.
Implements `.claude/tasks/0031-bench-report-cli.md`. ## Summary Adds the maintainer-facing `bench:report` command — the reporting counterpart to `bench:run` — giving the aggregator seam from task 0030 a runnable entry point. `runReportCommand` opens the sample store, drains it with `readAllSamples`, aggregates against the scored suite and bonus definitions, and prints `renderReport(...)` to stdout. It renders whatever has accumulated, annotating incomplete coverage (unrun arms show an em dash) rather than blocking on a complete matrix — no aggregation, weighting, or rendering is reimplemented in the command. `parseReportArgs` is the pure argument seam. `bench/report.ts` mirrors `bench/run.ts` (same `main` / direct-execution guard / `usage` helper / `Parsed…` union shape), and a `bench:report` npm script (`tsx bench/report.ts`) sits alongside `bench:run`. ### Deviations from the plan - **The boundary is offline, so the whole command is unit-tested.** Criterion 5 assumed the run-command pattern where the live boundary is "validated by running it." But the report reads only the local store — no credentials, host, or Agent SDK — so `runReportCommand` is deterministic and covered by real unit tests over a temp-dir store (populated, empty, `--help`), on top of the parser tests. It was still run end-to-end (`npm run bench:report`). The unused `deps` parameter is kept only for signature parity with the command family. - **Added `--self-review` / `--no-self-review`, beyond the written `(--store, --help)` seam.** `aggregate` needs a `suite` and `bonus`, and `buildScoredSuite`/`buildBonusTasks` are parameterized by `selfReviewPermitted`. `bench:run` resolves this by probing the live host; the offline report has no host to probe, so the flag makes the choice explicit (default `true`) instead of silently hardcoding one variant. It is well-scoped — the scored coverage is identical either way, so it only selects the bonus capability catalog — and is documented in `--help` and the README. Criterion 5's wording was updated to record it. - **`DEFAULT_STORE_ROOT` moved to `store.ts`** as the single source of truth (both commands need it) and re-exported from `run.ts` so its existing importers are unchanged. Full bench tier green (104 tests), typecheck clean. ## Review Three-axis `/review-uncommitted`. ### Risk **Overall: Low** - Blast radius: Low — a new isolated command plus one moved constant re-exported for compatibility; no existing callers change behavior. - Reversibility: Low — purely additive new files and a package.json script; trivial to revert, no migrations or published-API impact. - Test coverage: Low — the parser seam and the offline boundary (default/override/help/empty-store) are covered by tests in the diff. - Sensitive domain: Low — offline dev tooling reading a local sample store, explicitly no credentials, host, or SDK. - Size & complexity: Low — ~350 lines, mostly new; a simple linear argv loop and a thin boundary delegating to existing seams. - Runtime criticality: Low — maintainer-only reporting tool, not on any production path. ### Standards — unaddressed findings - **`parseReportArgs` flag loop partly overlaps `parseRunArgs` (judgement call, Duplicated Code).** Left unextracted: the two have genuinely diverged (the report parser adds `--no-` boolean negation), so a shared parser would be premature generality — the reviewer's own recommendation. ### Spec — unaddressed findings - None. The only spec finding (`--self-review` exceeding the written seam) was addressed by recording the decision in the task's Implementation Notes and updating the criterion wording; the reviewer judged the flag justified in substance.
alexion added 1 commit 2026-07-16 16:30:58 -04:00
feat: add benchmark reporting command (task 0031)
All checks were successful
CI / test (pull_request) Successful in 52s
Details
CI / test (push) Successful in 52s
Details
53342f67e3 
Add the maintainer-facing `bench:report` command, the offline counterpart
to `bench:run`: it opens the sample store, drains it, aggregates against the
scored suite and bonus definitions, and prints the aggregator's comparison to
stdout. It renders whatever has accumulated, annotating incomplete coverage
rather than blocking on a complete matrix.

`parseReportArgs` is the pure argument seam (`--store`, `--help`, plus a
`--self-review` / `--no-self-review` variant selector). Unlike `bench:run` the
boundary is offline — it reads only the local store, no host or Agent SDK — so
the whole `runReportCommand` is deterministic and unit-tested, not smoke-run.

Move `DEFAULT_STORE_ROOT` to `store.ts` as the single source of truth,
re-exported from `run.ts` for its existing importers.
alexion merged commit 53342f67e3 into main 2026-07-16 18:30:52 -04:00
alexion deleted branch task-0031-bench-report-cli 2026-07-16 18:30:52 -04:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
alexion
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: alexion/gitea-axi#32
Delete Branch "task-0031-bench-report-cli"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?