feat: add benchmark task suite (task 0028)
All checks were successful
CI / test (pull_request) Successful in 51s
CI / test (push) Successful in 52s

Add the full 20-task scored suite and the capability-asymmetric bonus
definitions, plus the self-review capability probe that resolves the two
review tasks.

buildScoredSuite returns the shared-surface tasks weighted four read /
six single-mutation / six find-then-act / four multi-step, each a
natural-language intent parametrized against the seed and carrying a tier
and a scoring spec keyed on the single user. The two find-then-act review
tasks are approve/request-changes when the host permits self-review and
comment reviews otherwise; buildBonusTasks emits the approve/request-changes
operations as bonus entries in the fallback case, alongside the static
both-direction bonus definitions (gitea-axi's search/diff/checks/checkout/
issue-dependency edges, and the not-applicable repository/release/milestone
operations).

self-review.ts adds probeSelfReview and detectSelfReviewSupport, the live
boundary that determines self-review support once per sweep; it reuses the
now-exported non-throwing request helper from seed.ts.
This commit was merged in pull request #29.
This commit is contained in:
2026-07-16 09:51:22 -04:00
parent c6a972734e
commit e8310fb616
7 changed files with 1076 additions and 8 deletions

View File

@@ -15,7 +15,25 @@ The bonus definitions cover capability-asymmetric operations in both directions:
## Acceptance criteria
- [ ] The scored suite has 20 tasks confined to the shared capability surface, phrased as natural-language intents parametrized against the seed.
- [ ] The suite is weighted roughly four read / six single-mutation / six find-then-act / four multi-step, with each task carrying a tier tag and a scoring spec.
- [ ] A self-review capability probe determines whether the two review tasks run as approve/request-changes or as comment reviews (falling back to the bonus table if self-review is not permitted).
- [ ] Bonus task definitions cover the asymmetries in both directions, including the gitea-axi not-applicable operations, and are kept separate from the scored suite.
- [x] The scored suite has 20 tasks confined to the shared capability surface, phrased as natural-language intents parametrized against the seed.
- [x] The suite is weighted roughly four read / six single-mutation / six find-then-act / four multi-step, with each task carrying a tier tag and a scoring spec.
- [x] A self-review capability probe determines whether the two review tasks run as approve/request-changes or as comment reviews (falling back to the bonus table if self-review is not permitted).
- [x] Bonus task definitions cover the asymmetries in both directions, including the gitea-axi not-applicable operations, and are kept separate from the scored suite.
## Implementation Notes
The suite and bonus definitions live in `bench/task-suite.ts`; the self-review probe lives in `bench/self-review.ts`.
**Self-review probe as a runtime seam, not a baked-in constant.**
The task says self-review support is "probed during implementation." Rather than probing the host once and hard-coding a boolean, this splits the concern the way the rest of the harness is factored: `buildScoredSuite({ selfReviewPermitted })` and `buildBonusTasks({ selfReviewPermitted })` are pure functions unit-tested against a flag, and `self-review.ts` (`probeSelfReview` / `detectSelfReviewSupport`) is the live boundary that resolves the flag once per sweep — provision a throwaway repo, seed it, attempt an approval on the user's own pull request, delete the repo, report the verdict. This matches the seed/snapshot pattern (live boundaries are smoke-validated, not mocked) and means the suite tracks whatever the host actually permits instead of a guess frozen at authoring time. Wiring the probe into a full sweep belongs to the later run-loop-CLI slice; this slice ships the probe and the flag-driven builders.
**Review tasks placed in the find-then-act tier.**
The two review tasks (`fta-review-csv-pull`, `fta-review-docs-pull`) are find-then-act rather than single-mutation: each names its target pull request by a property (implements CSV export / refreshes documentation) and forces discovery before acting, which is the tier's defining trait. Their `kind` (and the intent's verb) toggles on `selfReviewPermitted`: approved/request-changes when permitted, comment otherwise; when not permitted the approve/request-changes operations are emitted as `self-review-unavailable` bonus entries instead.
**`bench/seed.ts` change.**
`request` (the non-throwing round-trip) is now exported so the probe can read a 4xx (a host forbidding self-approval) as `false` without it being thrown, while a network-level failure still propagates. This is the only edit outside the new files.
**Label creation in a multi-step task.**
`ms-create-and-apply-stale` creates a new label, which reads "label management" (shared-surface item) at its most generous. It is deliberately kept a scored task rather than a bonus one, since label creation is within every arm's reach.
All 20 mutation/read specs were cross-checked against the `SEED_PLAN` ground truth (target titles exist, pre-states and encoded changes match the intents) during the spec-fidelity review. No acceptance criteria were dropped.