feat: add benchmark task suite (task 0028) #29

alexion · 2026-07-16T09:51:57-04:00

alexion commented

2026-07-16 09:51:57 -04:00

Task file: .claude/tasks/0028-bench-task-suite.md

Summary

Adds the full scored task suite and the capability-asymmetric bonus definitions for the benchmark harness, plus the self-review capability probe.

bench/task-suite.ts — buildScoredSuite({ selfReviewPermitted }) returns the 20 shared-surface tasks weighted four read / six single-mutation / six find-then-act / four multi-step. Each is a natural-language intent parametrized against the seed, carrying a tier and a scoring spec keyed on the single user. The two find-then-act review tasks are approve/request-changes when self-review is permitted and comment reviews otherwise. buildBonusTasks({ selfReviewPermitted }) returns the capability-asymmetric operations kept out of the scored suite in both directions — gitea-axi's edges (full-text search, diff, checks, checkout, issue dependencies) and the not-applicable repository/release/milestone operations — plus the approve/request-changes pair as bonus entries when self-review is unavailable.
bench/self-review.ts — probeSelfReview / detectSelfReviewSupport, the live boundary that resolves self-review support once per sweep (provision a throwaway repo, seed it, attempt a self-approval, delete, report). Smoke-validated, not mocked.
bench/seed.ts — exports the existing non-throwing request helper so the probe can read a 4xx as "not permitted" without it throwing (a network failure still propagates). Only edit outside the new files.

Deviations

Probe is a runtime seam, not a baked-in constant. The pure flag-driven builders are unit-tested; the live probe resolves the flag, matching the seed/snapshot live-boundary pattern. Wiring the probe into a full sweep belongs to the later run-loop-CLI slice.
Review tasks live in the find-then-act tier (each discovers its pull request by property before acting), toggling kind on the self-review flag.
ms-create-and-apply-stale creates a label, read as "label management" (shared surface) and kept scored rather than bonus.

No acceptance criteria were dropped. All 20 specs were cross-checked against the SEED_PLAN ground truth.

Review

Overall: Low

Blast radius: Low — self-contained to bench/; the only outside touch is exporting the existing request helper in seed.ts, with no behaviour change to callers.
Reversibility: Low — additive new files plus a visibility widening; trivially revertable, no migrations or published API.
Test coverage: Low — unit tests drive suite structure and scoring through the real checker, plus a skip-clean live smoke tier for the probe.
Sensitive domain: Low — no product auth/permissions logic; the probe hits a throwaway repo it provisions and deletes with existing bench credentials.
Size & complexity: Medium — ~1,100 lines across five files, but mostly flat declarative task/bonus data with simple control flow.
Runtime criticality: Low — bench/ is internal tooling, excluded from the published npm package.

Standards and Spec axes: no unaddressed findings. The one Standards finding (a duplicated stateWithReview helper in the test file) was fixed by hoisting it to module scope; a grammar slip in a doc-comment was corrected. The Spec axis reported clean fidelity with no defects.

Task file: `.claude/tasks/0028-bench-task-suite.md` ## Summary Adds the full scored task suite and the capability-asymmetric bonus definitions for the benchmark harness, plus the self-review capability probe. - **`bench/task-suite.ts`** — `buildScoredSuite({ selfReviewPermitted })` returns the 20 shared-surface tasks weighted four read / six single-mutation / six find-then-act / four multi-step. Each is a natural-language intent parametrized against the seed, carrying a tier and a scoring spec keyed on the single user. The two find-then-act review tasks are approve/request-changes when self-review is permitted and comment reviews otherwise. `buildBonusTasks({ selfReviewPermitted })` returns the capability-asymmetric operations kept out of the scored suite in both directions — gitea-axi's edges (full-text search, diff, checks, checkout, issue dependencies) and the not-applicable repository/release/milestone operations — plus the approve/request-changes pair as bonus entries when self-review is unavailable. - **`bench/self-review.ts`** — `probeSelfReview` / `detectSelfReviewSupport`, the live boundary that resolves self-review support once per sweep (provision a throwaway repo, seed it, attempt a self-approval, delete, report). Smoke-validated, not mocked. - **`bench/seed.ts`** — exports the existing non-throwing `request` helper so the probe can read a 4xx as "not permitted" without it throwing (a network failure still propagates). Only edit outside the new files. ### Deviations - **Probe is a runtime seam, not a baked-in constant.** The pure flag-driven builders are unit-tested; the live probe resolves the flag, matching the seed/snapshot live-boundary pattern. Wiring the probe into a full sweep belongs to the later run-loop-CLI slice. - **Review tasks live in the find-then-act tier** (each discovers its pull request by property before acting), toggling kind on the self-review flag. - **`ms-create-and-apply-stale` creates a label**, read as "label management" (shared surface) and kept scored rather than bonus. No acceptance criteria were dropped. All 20 specs were cross-checked against the `SEED_PLAN` ground truth. ## Review **Overall: Low** - Blast radius: Low — self-contained to `bench/`; the only outside touch is exporting the existing `request` helper in `seed.ts`, with no behaviour change to callers. - Reversibility: Low — additive new files plus a visibility widening; trivially revertable, no migrations or published API. - Test coverage: Low — unit tests drive suite structure and scoring through the real checker, plus a skip-clean live smoke tier for the probe. - Sensitive domain: Low — no product auth/permissions logic; the probe hits a throwaway repo it provisions and deletes with existing bench credentials. - Size & complexity: Medium — ~1,100 lines across five files, but mostly flat declarative task/bonus data with simple control flow. - Runtime criticality: Low — `bench/` is internal tooling, excluded from the published npm package. Standards and Spec axes: no unaddressed findings. The one Standards finding (a duplicated `stateWithReview` helper in the test file) was fixed by hoisting it to module scope; a grammar slip in a doc-comment was corrected. The Spec axis reported clean fidelity with no defects.

alexion added 1 commit 2026-07-16 09:51:58 -04:00

feat: add benchmark task suite (task 0028)

CI / test (pull_request) Successful in 51s

Details

CI / test (push) Successful in 52s

Details

e8310fb616

Add the full 20-task scored suite and the capability-asymmetric bonus
definitions, plus the self-review capability probe that resolves the two
review tasks.

buildScoredSuite returns the shared-surface tasks weighted four read /
six single-mutation / six find-then-act / four multi-step, each a
natural-language intent parametrized against the seed and carrying a tier
and a scoring spec keyed on the single user. The two find-then-act review
tasks are approve/request-changes when the host permits self-review and
comment reviews otherwise; buildBonusTasks emits the approve/request-changes
operations as bonus entries in the fallback case, alongside the static
both-direction bonus definitions (gitea-axi's search/diff/checks/checkout/
issue-dependency edges, and the not-applicable repository/release/milestone
operations).

self-review.ts adds probeSelfReview and detectSelfReviewSupport, the live
boundary that determines self-review support once per sweep; it reuses the
now-exported non-throwing request helper from seed.ts.

alexion merged commit e8310fb616 into main

2026-07-16 10:21:18 -04:00

alexion deleted branch task-0028-bench-task-suite

2026-07-16 10:21:18 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: alexion/gitea-axi#29