alexion
e8310fb616
Add the full 20-task scored suite and the capability-asymmetric bonus definitions, plus the self-review capability probe that resolves the two review tasks. buildScoredSuite returns the shared-surface tasks weighted four read / six single-mutation / six find-then-act / four multi-step, each a natural-language intent parametrized against the seed and carrying a tier and a scoring spec keyed on the single user. The two find-then-act review tasks are approve/request-changes when the host permits self-review and comment reviews otherwise; buildBonusTasks emits the approve/request-changes operations as bonus entries in the fallback case, alongside the static both-direction bonus definitions (gitea-axi's search/diff/checks/checkout/ issue-dependency edges, and the not-applicable repository/release/milestone operations). self-review.ts adds probeSelfReview and detectSelfReviewSupport, the live boundary that determines self-review support once per sweep; it reuses the now-exported non-throwing request helper from seed.ts.
4.8 KiB
spec, blocked-by
| spec | blocked-by | |||
|---|---|---|---|---|
| benchmark-harness |
|
What to build
The full scored task suite plus the capability-asymmetric bonus definitions, authored against the runnable Task wrapper and scored against the seed's ground truth.
The scored suite is 20 tasks drawn only from the capability surface shared by all four arms — issue and pull-request listing, viewing, creation, editing, closing and reopening, commenting and comment retrieval, label management and application, review comments, merge, and assignee changes. Tasks are phrased as natural-language intents, not command invocations, parametrized against the seed, each carrying its tier tag and scoring spec. The suite is weighted toward discovery and multi-step work: roughly four read tasks, six single-mutation tasks, six find-then-act tasks, and four multi-step workflows.
Review tasks default to comment-type reviews, which a single user can leave on their own pull request. Whether the host permits a user to approve or request changes on their own pull request is probed during implementation; if permitted, the two review tasks are promoted from comment reviews to approve and request-changes, otherwise those move to the bonus table.
The bonus definitions cover capability-asymmetric operations in both directions: where tea, gitea-mcp, or raw API fall short of gitea-axi (full-text search, diff, checks, checkout, issue dependencies), and operations outside gitea-axi's scope (repository, release, and milestone management), for which gitea-axi is reported not-applicable. These are kept out of the scored suite.
Acceptance criteria
- The scored suite has 20 tasks confined to the shared capability surface, phrased as natural-language intents parametrized against the seed.
- The suite is weighted roughly four read / six single-mutation / six find-then-act / four multi-step, with each task carrying a tier tag and a scoring spec.
- A self-review capability probe determines whether the two review tasks run as approve/request-changes or as comment reviews (falling back to the bonus table if self-review is not permitted).
- Bonus task definitions cover the asymmetries in both directions, including the gitea-axi not-applicable operations, and are kept separate from the scored suite.
Implementation Notes
The suite and bonus definitions live in bench/task-suite.ts; the self-review probe lives in bench/self-review.ts.
Self-review probe as a runtime seam, not a baked-in constant.
The task says self-review support is "probed during implementation." Rather than probing the host once and hard-coding a boolean, this splits the concern the way the rest of the harness is factored: buildScoredSuite({ selfReviewPermitted }) and buildBonusTasks({ selfReviewPermitted }) are pure functions unit-tested against a flag, and self-review.ts (probeSelfReview / detectSelfReviewSupport) is the live boundary that resolves the flag once per sweep — provision a throwaway repo, seed it, attempt an approval on the user's own pull request, delete the repo, report the verdict. This matches the seed/snapshot pattern (live boundaries are smoke-validated, not mocked) and means the suite tracks whatever the host actually permits instead of a guess frozen at authoring time. Wiring the probe into a full sweep belongs to the later run-loop-CLI slice; this slice ships the probe and the flag-driven builders.
Review tasks placed in the find-then-act tier.
The two review tasks (fta-review-csv-pull, fta-review-docs-pull) are find-then-act rather than single-mutation: each names its target pull request by a property (implements CSV export / refreshes documentation) and forces discovery before acting, which is the tier's defining trait. Their kind (and the intent's verb) toggles on selfReviewPermitted: approved/request-changes when permitted, comment otherwise; when not permitted the approve/request-changes operations are emitted as self-review-unavailable bonus entries instead.
bench/seed.ts change.
request (the non-throwing round-trip) is now exported so the probe can read a 4xx (a host forbidding self-approval) as false without it being thrown, while a network-level failure still propagates. This is the only edit outside the new files.
Label creation in a multi-step task.
ms-create-and-apply-stale creates a new label, which reads "label management" (shared-surface item) at its most generous. It is deliberately kept a scored task rather than a bonus one, since label creation is within every arm's reach.
All 20 mutation/read specs were cross-checked against the SEED_PLAN ground truth (target titles exist, pre-states and encoded changes match the intents) during the spec-fidelity review. No acceptance criteria were dropped.