From e8310fb6161387e303fa9b30232db63e81961e4e Mon Sep 17 00:00:00 2001 From: alexion Date: Thu, 16 Jul 2026 09:51:22 -0400 Subject: [PATCH] feat: add benchmark task suite (task 0028) Add the full 20-task scored suite and the capability-asymmetric bonus definitions, plus the self-review capability probe that resolves the two review tasks. buildScoredSuite returns the shared-surface tasks weighted four read / six single-mutation / six find-then-act / four multi-step, each a natural-language intent parametrized against the seed and carrying a tier and a scoring spec keyed on the single user. The two find-then-act review tasks are approve/request-changes when the host permits self-review and comment reviews otherwise; buildBonusTasks emits the approve/request-changes operations as bonus entries in the fallback case, alongside the static both-direction bonus definitions (gitea-axi's search/diff/checks/checkout/ issue-dependency edges, and the not-applicable repository/release/milestone operations). self-review.ts adds probeSelfReview and detectSelfReviewSupport, the live boundary that determines self-review support once per sweep; it reuses the now-exported non-throwing request helper from seed.ts. --- .claude/tasks/0028-bench-task-suite.md | 26 +- bench/README.md | 7 +- bench/seed.ts | 8 +- bench/self-review.smoke.test.ts | 40 ++ bench/self-review.ts | 60 +++ bench/task-suite.test.ts | 496 +++++++++++++++++++++++++ bench/task-suite.ts | 447 ++++++++++++++++++++++ 7 files changed, 1076 insertions(+), 8 deletions(-) create mode 100644 bench/self-review.smoke.test.ts create mode 100644 bench/self-review.ts create mode 100644 bench/task-suite.test.ts create mode 100644 bench/task-suite.ts diff --git a/.claude/tasks/0028-bench-task-suite.md b/.claude/tasks/0028-bench-task-suite.md index 945d087..45744dd 100644 --- a/.claude/tasks/0028-bench-task-suite.md +++ b/.claude/tasks/0028-bench-task-suite.md @@ -15,7 +15,25 @@ The bonus definitions cover capability-asymmetric operations in both directions: ## Acceptance criteria -- [ ] The scored suite has 20 tasks confined to the shared capability surface, phrased as natural-language intents parametrized against the seed. -- [ ] The suite is weighted roughly four read / six single-mutation / six find-then-act / four multi-step, with each task carrying a tier tag and a scoring spec. -- [ ] A self-review capability probe determines whether the two review tasks run as approve/request-changes or as comment reviews (falling back to the bonus table if self-review is not permitted). -- [ ] Bonus task definitions cover the asymmetries in both directions, including the gitea-axi not-applicable operations, and are kept separate from the scored suite. +- [x] The scored suite has 20 tasks confined to the shared capability surface, phrased as natural-language intents parametrized against the seed. +- [x] The suite is weighted roughly four read / six single-mutation / six find-then-act / four multi-step, with each task carrying a tier tag and a scoring spec. +- [x] A self-review capability probe determines whether the two review tasks run as approve/request-changes or as comment reviews (falling back to the bonus table if self-review is not permitted). +- [x] Bonus task definitions cover the asymmetries in both directions, including the gitea-axi not-applicable operations, and are kept separate from the scored suite. + +## Implementation Notes + +The suite and bonus definitions live in `bench/task-suite.ts`; the self-review probe lives in `bench/self-review.ts`. + +**Self-review probe as a runtime seam, not a baked-in constant.** +The task says self-review support is "probed during implementation." Rather than probing the host once and hard-coding a boolean, this splits the concern the way the rest of the harness is factored: `buildScoredSuite({ selfReviewPermitted })` and `buildBonusTasks({ selfReviewPermitted })` are pure functions unit-tested against a flag, and `self-review.ts` (`probeSelfReview` / `detectSelfReviewSupport`) is the live boundary that resolves the flag once per sweep — provision a throwaway repo, seed it, attempt an approval on the user's own pull request, delete the repo, report the verdict. This matches the seed/snapshot pattern (live boundaries are smoke-validated, not mocked) and means the suite tracks whatever the host actually permits instead of a guess frozen at authoring time. Wiring the probe into a full sweep belongs to the later run-loop-CLI slice; this slice ships the probe and the flag-driven builders. + +**Review tasks placed in the find-then-act tier.** +The two review tasks (`fta-review-csv-pull`, `fta-review-docs-pull`) are find-then-act rather than single-mutation: each names its target pull request by a property (implements CSV export / refreshes documentation) and forces discovery before acting, which is the tier's defining trait. Their `kind` (and the intent's verb) toggles on `selfReviewPermitted`: approved/request-changes when permitted, comment otherwise; when not permitted the approve/request-changes operations are emitted as `self-review-unavailable` bonus entries instead. + +**`bench/seed.ts` change.** +`request` (the non-throwing round-trip) is now exported so the probe can read a 4xx (a host forbidding self-approval) as `false` without it being thrown, while a network-level failure still propagates. This is the only edit outside the new files. + +**Label creation in a multi-step task.** +`ms-create-and-apply-stale` creates a new label, which reads "label management" (shared-surface item) at its most generous. It is deliberately kept a scored task rather than a bonus one, since label creation is within every arm's reach. + +All 20 mutation/read specs were cross-checked against the `SEED_PLAN` ground truth (target titles exist, pre-states and encoded changes match the intents) during the spec-fidelity review. No acceptance criteria were dropped. diff --git a/bench/README.md b/bench/README.md index 8bf5a89..d8f77be 100644 --- a/bench/README.md +++ b/bench/README.md @@ -39,14 +39,16 @@ The raw component breakdown is retained on every sample so the data can be re-we - `seed-plan.ts` — the deterministic ground truth every throwaway repository is seeded to: the fixed labels, the open/closed issue spread across the discriminating dimensions, and the pull requests, as pure data plus `groundTruth(user)`, which realizes it into the `RepoState` the checker scores against. - `seed.ts` — the idempotent seeding scripted over the live Gitea API: `resolveBenchAccess` (which reuses gitea-axi's own tea-login credential discovery), `provisionRepo`, and `seedRepo`, reconciling each label, issue, pull request, comment, and review by its natural key so a re-run never duplicates the ground truth. - `arm.ts` — the per-arm scaffolding: `basePrompt` (the identical task-agnostic base every arm shares) and `buildArm`, which produces the single `ArmDefinition` the runner consumes — the assembled prompt plus the tool configuration. The gitea-axi arm embeds the bundled Agent Skill, the tea and raw-api arms get a one-line native-discovery pointer, and the gitea-mcp arm runs with the shell disabled and only the MCP server attached (its dispatcher schemas load eagerly). The shell arms' PATH and guard come from `guard.ts`. -- `task.ts` — the runnable `BenchTask` wrapper (natural-language intent, tier, and a scoring spec keyed on the single available user) plus one `SAMPLE_TASK` that exercises the full path; the complete suite is authored in a later slice. +- `task.ts` — the runnable `BenchTask` wrapper (natural-language intent, tier, and a scoring spec keyed on the single available user) plus one `SAMPLE_TASK` that exercises the full path. +- `task-suite.ts` — the full scored suite and the bonus definitions. `buildScoredSuite` returns the 20 shared-surface tasks (weighted four read / six single-mutation / six find-then-act / four multi-step), each phrased as a natural-language intent parametrized against the seed and carrying a tier and a scoring spec; its two review tasks are approve/request-changes or comment reviews depending on the self-review flag. `buildBonusTasks` returns the capability-asymmetric operations kept out of the scored suite, in both directions — gitea-axi's edges where the other arms fall short (full-text search, diff, checks, checkout, issue dependencies) and the repository/release/milestone operations for which gitea-axi is reported not-applicable — plus the approve/request-changes pair when self-review is unavailable. +- `self-review.ts` — `probeSelfReview` and `detectSelfReviewSupport`, the capability probe that determines whether the host permits a user to approve or request changes on their own pull request, so the suite builder can promote the two review tasks or leave them as comment reviews. A live boundary, so it is exercised by the smoke run rather than mocked. - `snapshot.ts` — `captureRepoState`, the seed's counterpart: it reads the whole scored surface of a live repository back into the `RepoState` the checker diffs against, normalizing the few fields whose live form differs from the ground truth (notably label colours). A live boundary, so it is exercised by the smoke run rather than mocked. - `audit.ts` — `auditTranscript`, the post-run isolation audit: it re-runs the arm's own guard over the executed shell commands and checks channel discipline (a shell arm never reaches MCP tools; the MCP arm never reaches the shell), returning the leaks that flag a trial invalid rather than scored. - `runner.ts` — the single-cell runner: `runCell` threads every layer to run one `(arm, task, trial)` cell end to end — provision, seed, run the agent bounded by a turn cap and a wall-clock backstop, audit the transcript, capture and score the post-run state, append the sample, and delete the repository. The live host and the Agent SDK are factored behind the `BenchHost` and `AgentDriver` seams, so the orchestration is unit-tested with fakes while the live wiring is validated by the smoke run. - `host.ts` — `liveBenchHost`, the production `BenchHost`: a thin composition of `seed.ts` (provision, seed, delete) and `snapshot.ts` (capture) bound to one set of host credentials. - `sdk-driver.ts` — `sdkAgentDriver`, the production `AgentDriver`: it runs one arm through the Claude Agent SDK on the maintainer's subscription, enforcing isolation in-band via the SDK's permission callback (the arm's guard on every Bash command; the shell disabled on the MCP arm) and reporting the four token components (folding in the auxiliary small model), the turn count, the imputed cost, the transcript, and the final report. The SDK is loaded through a computed dynamic import so it is an optional peer needed only for live runs. -Later slices add the task suite, the run-loop CLI, and the aggregator. +Later slices add the run-loop CLI and the aggregator. ## Tests @@ -68,3 +70,4 @@ GITEA_AXI_BENCH_LOGIN= npm run test:bench:smoke With `GITEA_AXI_BENCH_LOGIN` unset the smoke tier skips (a pass), matching the end-to-end tier's behavior when no live instance is configured. The runner smoke additionally requires the Claude Agent SDK (`@anthropic-ai/claude-agent-sdk`, an optional peer of the harness, needed only for live runs) and the `gitea-axi` CLI on `PATH`; it skips when the SDK is not installed and needs a Claude subscription to run for real. The post-run transcript audit is what validates run orchestration: a run in which a foreign tool was reached is flagged invalid rather than scored. +The self-review probe is the third live boundary: its smoke run provisions and seeds a throwaway repository, attempts an approval on the user's own pull request, and asserts the probe reaches a definite verdict — whichever way the host is configured. diff --git a/bench/seed.ts b/bench/seed.ts index 9929717..7ad9a3b 100644 --- a/bench/seed.ts +++ b/bench/seed.ts @@ -48,8 +48,12 @@ export async function resolveBenchAccess(deps: CliDeps, loginName: string): Prom return { apiUrl: login.url.replace(/\/+$/, ""), token }; } -/** One authenticated Gitea API round-trip; returns the raw response unchecked. */ -async function request( +/** + * One authenticated Gitea API round-trip; returns the raw response unchecked. + * Exported so the self-review probe (self-review.ts) can inspect a non-2xx + * response — a host that forbids self-approval — without it being thrown. + */ +export async function request( access: BenchAccess, method: string, path: string, diff --git a/bench/self-review.smoke.test.ts b/bench/self-review.smoke.test.ts new file mode 100644 index 0000000..cf2be12 --- /dev/null +++ b/bench/self-review.smoke.test.ts @@ -0,0 +1,40 @@ +import { describe, expect, it } from "vitest"; +import type { CliDeps } from "../src/deps.js"; +import { resolveBenchAccess, type BenchAccess } from "./seed.js"; +import { detectSelfReviewSupport } from "./self-review.js"; + +/** + * The self-review probe smoke tier: a single live validation that the capability + * probe runs end-to-end against a real host — provisioning a throwaway + * repository, seeding it, checking whether the authenticated user may approve + * their own pull request, and cleaning the repository up — and reaches a definite + * boolean verdict without throwing. The benchmark-harness spec designates the + * self-review probe, like seed provisioning, as validated by a smoke run against + * a real host rather than by mocks, since its value is the real API interaction. + * Like the seed smoke tier, this suite skips cleanly when GITEA_AXI_BENCH_LOGIN + * is unset, which counts as a pass. + * + * The verdict itself is host-configuration-dependent — some hosts forbid a user + * from approving their own pull request, some permit it — so the assertion is + * only that a definite boolean is reached, never which value it is. + */ +const login = process.env.GITEA_AXI_BENCH_LOGIN; + +describe.skipIf(!login)("self-review smoke: capability probe", () => { + it( + "runs the self-review probe against the live host and returns a definite boolean verdict", + async () => { + const deps: CliDeps = { + env: process.env, + cwd: process.cwd(), + globals: { login }, + }; + const access: BenchAccess = await resolveBenchAccess(deps, login!); + + const result = await detectSelfReviewSupport(access); + + expect(typeof result).toBe("boolean"); + }, + 180_000, + ); +}); diff --git a/bench/self-review.ts b/bench/self-review.ts new file mode 100644 index 0000000..06f5bfa --- /dev/null +++ b/bench/self-review.ts @@ -0,0 +1,60 @@ +// The self-review capability probe: whether the live host permits a user to +// approve or request changes on their own pull request. The single-user seed can +// always leave a comment-type review on its own pull request, but approve and +// request-changes are host-gated — Gitea can be configured either way — so the +// scored suite's two review tasks must be resolved against the real host before a +// sweep: promoted to approve/request-changes where self-review is permitted, left +// as comment reviews otherwise (see task-suite.ts). +// +// This is a live boundary, like seed.ts and snapshot.ts: its value is the real +// Gitea API interaction, so it is exercised by the smoke run rather than mocked. + +import { + deleteRepo, + provisionRepo, + request, + seedRepo, + type BenchAccess, + type RepoCoords, +} from "./seed.js"; +import { SEED_PLAN } from "./seed-plan.js"; + +/** + * Attempt an approving review on one of the pull requests authored by the single + * available user, returning whether the host accepted it. A 2xx means self-review + * is permitted; a 4xx (the host forbidding self-approval) means it is not. A + * network-level failure propagates rather than being read as "not permitted", so a + * transient error never silently downgrades the scored suite. + */ +export async function probeSelfReview( + access: BenchAccess, + coords: RepoCoords, + prNumber: number, +): Promise { + const res = await request( + access, + "POST", + `/repos/${coords.owner}/${coords.repo}/pulls/${prNumber}/reviews`, + { event: "APPROVED", body: "self-review capability probe" }, + ); + return res.ok; +} + +/** + * Detect self-review support end to end against the live host: provision a + * throwaway repository, seed it to the ground truth (which creates the pull + * requests), probe an approval on the first seeded pull request, and delete the + * repository. The first pull request's number is deterministic — issues are seeded + * before pull requests in one shared number space — so it is the issue count plus + * one. Returns the probe's verdict for the suite builder to consume once per sweep. + */ +export async function detectSelfReviewSupport(access: BenchAccess): Promise { + const coords = await provisionRepo(access); + try { + await seedRepo(access, coords); + const firstPullNumber = SEED_PLAN.issues.length + 1; + return await probeSelfReview(access, coords, firstPullNumber); + } finally { + await deleteRepo(access, coords); + } +} diff --git a/bench/task-suite.test.ts b/bench/task-suite.test.ts new file mode 100644 index 0000000..b1c454f --- /dev/null +++ b/bench/task-suite.test.ts @@ -0,0 +1,496 @@ +import { describe, expect, it } from "vitest"; +import { buildScoredSuite, buildBonusTasks } from "./task-suite.js"; +import { checkReadAnswer, checkMutation } from "./checker.js"; +import { SEED_PLAN, groundTruth } from "./seed-plan.js"; +import type { Tier } from "./result.js"; + +/** + * The shape of the scored suite, asserted against the acceptance criteria as + * independent literals — NOT recomputed from the returned array. The criteria + * (benchmark-harness spec) fix the suite at 20 tasks, weighted roughly four + * read / six single-mutation / six find-then-act / four multi-step, each task + * carrying a tier tag and a scoring spec. So: + * - the total is exactly 20 (a hard literal from the criteria); + * - the per-tier counts are exactly 4 / 6 / 6 / 4 (hard literals); + * - every task has a unique, non-empty id and a non-empty intent (a runnable + * task must name itself and carry an instruction for the agent); + * - the scoring-spec kind follows the fixed tier semantics: read-tier tasks are + * scored by the required facts in their report (kind "read"), while the three + * acting tiers are scored by their expected end state (kind "mutation"). + * The tier→kind rule and the counts are the source of truth here, held as + * literals below, not derived from buildScoredSuite's own output. + */ +const EXPECTED_TOTAL = 20; +const EXPECTED_COUNT_BY_TIER: Record = { + read: 4, + "single-mutation": 6, + "find-then-act": 6, + "multi-step": 4, +}; + +// The fixed tier→scoring-spec-kind rule: a read task is scored on the facts its +// report must contain; the three acting tiers are scored on the repository's +// expected end state. +const EXPECTED_KIND_BY_TIER: Record = { + read: "read", + "single-mutation": "mutation", + "find-then-act": "mutation", + "multi-step": "mutation", +}; + +// An arbitrary single available user for the scoring-spec factory; its exact +// value is irrelevant to the structural facts under test. +const USER = "benchbot"; + +// Build a post-run state that pushes one review of the given kind onto the named +// pull request, independently from groundTruth + the seed. Shared by the review +// tasks' tests (self-review permitted and not permitted alike). +const stateWithReview = ( + pullTitle: string, + review: { kind: "comment" | "approved" | "request-changes"; body: string }, +) => { + const baseline = groundTruth(USER); + return { + ...baseline, + pullRequests: baseline.pullRequests.map((pull) => + pull.title === pullTitle + ? { + ...pull, + reviews: [ + ...pull.reviews, + { author: USER, kind: review.kind, body: review.body, comments: [] }, + ], + } + : pull, + ), + }; +}; + +describe("buildScoredSuite", () => { + it("returns 20 tier-tagged tasks weighted 4 read / 6 single-mutation / 6 find-then-act / 4 multi-step, each with a unique id, an intent, and a tier-appropriate scoring spec", () => { + const suite = buildScoredSuite({ selfReviewPermitted: true }); + + // Exactly 20 tasks. + expect(suite).toHaveLength(EXPECTED_TOTAL); + + // Exact tier distribution. + const countByTier = (tier: Tier) => suite.filter((task) => task.tier === tier).length; + expect(countByTier("read")).toBe(EXPECTED_COUNT_BY_TIER.read); + expect(countByTier("single-mutation")).toBe(EXPECTED_COUNT_BY_TIER["single-mutation"]); + expect(countByTier("find-then-act")).toBe(EXPECTED_COUNT_BY_TIER["find-then-act"]); + expect(countByTier("multi-step")).toBe(EXPECTED_COUNT_BY_TIER["multi-step"]); + + // Every task names itself (uniquely) and carries an instruction. + const ids = suite.map((task) => task.id); + expect(new Set(ids).size).toBe(EXPECTED_TOTAL); + for (const task of suite) { + expect(task.id.length).toBeGreaterThan(0); + expect(task.intent.length).toBeGreaterThan(0); + } + + // Every task's scoring spec is a factory whose kind matches its tier. + for (const task of suite) { + const spec = task.scoringSpec(USER); + expect(spec.kind).toBe(EXPECTED_KIND_BY_TIER[task.tier]); + } + }); +}); + +// The true open-issue count, derived independently from the seed plan's declared +// ground truth (bench/seed-plan.ts) rather than read back from task-suite.ts. +// This tracks the plan: if the seed's open/closed spread changes, so does the +// expected answer. +const SEED_OPEN_ISSUE_COUNT = SEED_PLAN.issues.filter((issue) => issue.state === "open").length; + +describe("buildScoredSuite read-task scoring specs", () => { + // Behavior: a read task's scoring spec is a read-type spec whose required facts + // encode the seed's ground truth, so the checker accepts a report that states + // the true answer and rejects one that states a wrong answer (benchmark-harness + // spec, "read tasks scored by required facts, no LLM judge"). We assert this + // through the real public checker (checkReadAnswer), not by inspecting the fact + // strings, so the test survives any rephrasing of the fact renderings. The + // open-issue count is an independent literal computed from SEED_PLAN above. + it("grounds the open-issue-count read task's facts in the seed so the checker accepts the true count and rejects a wrong one", () => { + const suite = buildScoredSuite({ selfReviewPermitted: true }); + + const task = suite.find((candidate) => candidate.id === "read-open-issue-count"); + expect(task).toBeDefined(); + if (task === undefined) return; + + const spec = task.scoringSpec(USER); + expect(spec.kind).toBe("read"); + if (spec.kind !== "read") return; + + const correctReport = `There are ${SEED_OPEN_ISSUE_COUNT} open issues in the repository.`; + const wrongCount = SEED_OPEN_ISSUE_COUNT + 3; + const wrongReport = `There are ${wrongCount} open issues in the repository.`; + + expect(checkReadAnswer(spec.facts, correctReport).pass).toBe(true); + expect(checkReadAnswer(spec.facts, wrongReport).pass).toBe(false); + }); +}); + +// The title of the CLOSED seeded issue the reopen task targets, taken from +// SEED_PLAN (bench/seed-plan.ts) as the source of truth: "Update README badges" +// is seeded state "closed". The task's purpose is to reopen exactly that issue. +const REOPEN_TARGET_TITLE = "Update README badges"; + +describe("buildScoredSuite single-mutation scoring specs", () => { + // Behavior: a single-mutation task's expected end state is the seed's ground + // truth with only the one described change applied — the intended mutation is + // present and nothing else is altered, so the full-state diff catches both a + // missed change and collateral damage (benchmark-harness spec, "mutation tasks + // scored by expected end state, diffed in full"). We assert this through the + // real public checker (checkMutation) against a baseline we build independently + // from groundTruth + the seed, never from task-suite.ts: + // - the expected state accepts a post-run state that reopens ONLY the target + // issue (intended mutation present, nothing else changed); + // - the expected state rejects the untouched ground truth, proving the spec + // actually encodes the reopen rather than being the identity. + it("encodes reopening only the target issue: matches the reopened state and rejects the untouched seed", () => { + const suite = buildScoredSuite({ selfReviewPermitted: true }); + + const task = suite.find((candidate) => candidate.id === "reopen-readme-badges"); + expect(task).toBeDefined(); + if (task === undefined) return; + + const spec = task.scoringSpec(USER); + expect(spec.kind).toBe("mutation"); + if (spec.kind !== "mutation") return; + + // Build the post-run state independently: the seed ground truth with ONLY the + // target issue flipped from closed to open, every other field untouched. + const baseline = groundTruth(USER); + const reopenedState = { + ...baseline, + issues: baseline.issues.map((issue) => + issue.title === REOPEN_TARGET_TITLE ? { ...issue, state: "open" as const } : issue, + ), + }; + + // Intended mutation present, nothing else altered → the spec accepts it. + expect(checkMutation(spec.expected, reopenedState).pass).toBe(true); + + // The spec is not the identity: it does not accept the untouched seed, where + // the target issue is still closed. + expect(checkMutation(spec.expected, groundTruth(USER)).pass).toBe(false); + }); +}); + +// The seeded title of the pull request the find-then-act merge task targets, +// taken from SEED_PLAN (bench/seed-plan.ts) as the source of truth: the pull +// request "Fix startup crash" is seeded open, and the task's purpose is to merge +// exactly that one. This literal doubles as the string the intent must NOT hand +// over verbatim. +const FTA_TARGET_PULL_TITLE = "Fix startup crash"; + +describe("buildScoredSuite find-then-act scoring specs", () => { + // Behavior: a find-then-act task forces discovery — it names its target by a + // property, not by the exact seeded title — and its scoring spec encodes the + // single acting change against the uniquely-matching seed entity (benchmark- + // harness spec, "find-then-act tier"). We assert both halves: + // - the intent does NOT contain the exact target title, so the agent must find + // the pull request rather than being handed it; + // - through the real public checker (checkMutation) against a baseline built + // independently from groundTruth + the seed, the expected state accepts a + // post-run state that merges ONLY the target pull request (intended change + // present, nothing else altered) and rejects the untouched seed (not the + // identity). + it("describes the merge target by property (not verbatim title) and encodes merging only that pull request", () => { + const suite = buildScoredSuite({ selfReviewPermitted: true }); + + const task = suite.find((candidate) => candidate.id === "fta-merge-startup-pull"); + expect(task).toBeDefined(); + if (task === undefined) return; + + expect(task.tier).toBe("find-then-act"); + + // Discovery is forced: the exact seeded title is not handed over in the intent. + expect(task.intent.includes(FTA_TARGET_PULL_TITLE)).toBe(false); + + const spec = task.scoringSpec(USER); + expect(spec.kind).toBe("mutation"); + if (spec.kind !== "mutation") return; + + // Build the post-run state independently: the seed ground truth with ONLY the + // target pull request set to merged, every other field/entity untouched. + const baseline = groundTruth(USER); + const mergedState = { + ...baseline, + pullRequests: baseline.pullRequests.map((pull) => + pull.title === FTA_TARGET_PULL_TITLE ? { ...pull, state: "merged" as const } : pull, + ), + }; + + // Intended merge present, nothing else altered → the spec accepts it. + expect(checkMutation(spec.expected, mergedState).pass).toBe(true); + + // The spec is not the identity: it rejects the untouched seed, where the + // target pull request is still open. + expect(checkMutation(spec.expected, groundTruth(USER)).pass).toBe(false); + }); +}); + +// The seeded issue the multi-step triage task targets and the three changes it +// applies at once, from SEED_PLAN (bench/seed-plan.ts) and the task's stated +// purpose as the source of truth: "Improve export performance" is seeded open +// with labels ["enhancement"], no assignees, no comments; triage adds the +// "priority" label, assigns it to the user, and adds one comment. +const MS_TARGET_ISSUE_TITLE = "Improve export performance"; +const MS_ADDED_LABEL = "priority"; +const MS_ADDED_COMMENT_BODY = "Prioritised for the next sprint."; + +describe("buildScoredSuite multi-step scoring specs", () => { + // Behavior: a multi-step task's expected end state encodes ALL of its several + // changes together against the seed ground truth, and nothing else (benchmark- + // harness spec, "multi-step tier"). We assert through the real public checker + // (checkMutation) against baselines built independently from groundTruth + the + // seed: + // - the expected state accepts a post-run state with ALL THREE triage changes + // applied to the target issue (label + assignee + comment), nothing else; + // - the expected state REJECTS each of the three "partial" states that apply + // only two of the three changes, proving the spec encodes all three rather + // than a subset. Labels/assignees are order-independent sets and comments + // match by author+body, so array order is irrelevant. + it("encodes all three triage changes together, rejecting any two-change partial", () => { + const suite = buildScoredSuite({ selfReviewPermitted: true }); + + const task = suite.find((candidate) => candidate.id === "ms-triage-export-perf"); + expect(task).toBeDefined(); + if (task === undefined) return; + + expect(task.tier).toBe("multi-step"); + + const spec = task.scoringSpec(USER); + expect(spec.kind).toBe("mutation"); + if (spec.kind !== "mutation") return; + + // Build a post-run state applying a chosen subset of the three triage changes + // to the target issue, independently from groundTruth + the seed. + const stateWith = (opts: { label: boolean; assignee: boolean; comment: boolean }) => { + const baseline = groundTruth(USER); + return { + ...baseline, + issues: baseline.issues.map((issue) => { + if (issue.title !== MS_TARGET_ISSUE_TITLE) return issue; + return { + ...issue, + labels: opts.label ? [...issue.labels, MS_ADDED_LABEL] : [...issue.labels], + assignees: opts.assignee ? [...issue.assignees, USER] : [...issue.assignees], + comments: opts.comment + ? [...issue.comments, { author: USER, body: MS_ADDED_COMMENT_BODY }] + : [...issue.comments], + }; + }), + }; + }; + + // All three changes present, nothing else altered → the spec accepts it. + const fullState = stateWith({ label: true, assignee: true, comment: true }); + expect(checkMutation(spec.expected, fullState).pass).toBe(true); + + // Each two-of-three partial is missing exactly one change → the spec rejects + // it, proving all three are required (not a subset). + const missingLabel = stateWith({ label: false, assignee: true, comment: true }); + const missingAssignee = stateWith({ label: true, assignee: false, comment: true }); + const missingComment = stateWith({ label: true, assignee: true, comment: false }); + expect(checkMutation(spec.expected, missingLabel).pass).toBe(false); + expect(checkMutation(spec.expected, missingAssignee).pass).toBe(false); + expect(checkMutation(spec.expected, missingComment).pass).toBe(false); + }); +}); + +// The two review tasks and the reviews they encode when self-review is permitted, +// from the task's stated purpose as the source of truth: the CSV pull request is +// APPROVED, the docs pull request gets a REQUEST-CHANGES review. Both target pull +// requests have no seeded reviews (bench/seed-plan.ts). ReviewKind and the Review +// shape come from bench/scoring-spec.ts. +const REVIEW_CSV_PULL_TITLE = "Implement CSV export"; +const REVIEW_DOCS_PULL_TITLE = "Refresh documentation"; +const REVIEW_CSV_BODY = "The CSV export path looks correct to me."; +const REVIEW_DOCS_BODY = "Please expand the installation section before this merges."; + +describe("buildScoredSuite review tasks with self-review permitted", () => { + // Behavior: when the host permits self-review, the two review tasks' expected + // states encode a promoted review — an approval on the CSV pull request and a + // request-changes on the docs pull request — rather than a plain comment review + // (benchmark-harness spec, single-user-seed self-review promotion). We assert + // through the real public checker (checkMutation) against baselines built + // independently from groundTruth + the stated review kinds/bodies: + // - each expected state accepts a post-run state carrying the promoted review; + // - each expected state REJECTS the same review demoted to kind "comment", + // proving the promotion actually took effect (not just any review). + it("encodes an approval on the CSV pull and a request-changes on the docs pull, rejecting comment-kind variants", () => { + const suite = buildScoredSuite({ selfReviewPermitted: true }); + + const csvTask = suite.find((candidate) => candidate.id === "fta-review-csv-pull"); + expect(csvTask).toBeDefined(); + if (csvTask === undefined) return; + const csvSpec = csvTask.scoringSpec(USER); + expect(csvSpec.kind).toBe("mutation"); + if (csvSpec.kind !== "mutation") return; + + // CSV pull request: promoted to an approval. + const csvApproved = stateWithReview(REVIEW_CSV_PULL_TITLE, { + kind: "approved", + body: REVIEW_CSV_BODY, + }); + const csvComment = stateWithReview(REVIEW_CSV_PULL_TITLE, { + kind: "comment", + body: REVIEW_CSV_BODY, + }); + expect(checkMutation(csvSpec.expected, csvApproved).pass).toBe(true); + expect(checkMutation(csvSpec.expected, csvComment).pass).toBe(false); + + const docsTask = suite.find((candidate) => candidate.id === "fta-review-docs-pull"); + expect(docsTask).toBeDefined(); + if (docsTask === undefined) return; + const docsSpec = docsTask.scoringSpec(USER); + expect(docsSpec.kind).toBe("mutation"); + if (docsSpec.kind !== "mutation") return; + + // Docs pull request: promoted to a request-changes. + const docsRequestChanges = stateWithReview(REVIEW_DOCS_PULL_TITLE, { + kind: "request-changes", + body: REVIEW_DOCS_BODY, + }); + const docsComment = stateWithReview(REVIEW_DOCS_PULL_TITLE, { + kind: "comment", + body: REVIEW_DOCS_BODY, + }); + expect(checkMutation(docsSpec.expected, docsRequestChanges).pass).toBe(true); + expect(checkMutation(docsSpec.expected, docsComment).pass).toBe(false); + }); +}); + +describe("buildScoredSuite/buildBonusTasks review tasks with self-review NOT permitted", () => { + // Behavior: when the host forbids self-review, the two review tasks fall back to + // comment-type reviews in the scored suite, and the approve / request-changes + // operations move to the bonus table under direction "self-review-unavailable" + // (benchmark-harness spec, single-user-seed self-review fallback). We assert + // through the real public checker (checkMutation) against baselines built + // independently from groundTruth + the stated bodies, plus the bonus table's + // public shape. Scored specs must encode COMMENT reviews (rejecting the promoted + // kinds), and the bonus table gains the two unavailable operations only when + // self-review is off. + it("falls back to comment reviews in the scored suite and moves approve/request-changes to the bonus table", () => { + const suite = buildScoredSuite({ selfReviewPermitted: false }); + + // CSV pull request: a plain comment review, NOT an approval. + const csvTask = suite.find((candidate) => candidate.id === "fta-review-csv-pull"); + expect(csvTask).toBeDefined(); + if (csvTask === undefined) return; + const csvSpec = csvTask.scoringSpec(USER); + expect(csvSpec.kind).toBe("mutation"); + if (csvSpec.kind !== "mutation") return; + const csvComment = stateWithReview(REVIEW_CSV_PULL_TITLE, { + kind: "comment", + body: REVIEW_CSV_BODY, + }); + const csvApproved = stateWithReview(REVIEW_CSV_PULL_TITLE, { + kind: "approved", + body: REVIEW_CSV_BODY, + }); + expect(checkMutation(csvSpec.expected, csvComment).pass).toBe(true); + expect(checkMutation(csvSpec.expected, csvApproved).pass).toBe(false); + + // Docs pull request: a plain comment review, NOT a request-changes. + const docsTask = suite.find((candidate) => candidate.id === "fta-review-docs-pull"); + expect(docsTask).toBeDefined(); + if (docsTask === undefined) return; + const docsSpec = docsTask.scoringSpec(USER); + expect(docsSpec.kind).toBe("mutation"); + if (docsSpec.kind !== "mutation") return; + const docsComment = stateWithReview(REVIEW_DOCS_PULL_TITLE, { + kind: "comment", + body: REVIEW_DOCS_BODY, + }); + const docsRequestChanges = stateWithReview(REVIEW_DOCS_PULL_TITLE, { + kind: "request-changes", + body: REVIEW_DOCS_BODY, + }); + expect(checkMutation(docsSpec.expected, docsComment).pass).toBe(true); + expect(checkMutation(docsSpec.expected, docsRequestChanges).pass).toBe(false); + + // The approve and request-changes operations appear in the bonus table under + // "self-review-unavailable" when self-review is off. + const bonusOff = buildBonusTasks({ selfReviewPermitted: false }); + const unavailable = bonusOff.filter( + (entry) => entry.direction === "self-review-unavailable", + ); + expect( + unavailable.some((entry) => /approv/i.test(entry.operation)), + ).toBe(true); + expect( + unavailable.some((entry) => /request.*chang/i.test(entry.operation)), + ).toBe(true); + + // When self-review works, those operations are scored, not bonus. + const bonusOn = buildBonusTasks({ selfReviewPermitted: true }); + expect( + bonusOn.some((entry) => entry.direction === "self-review-unavailable"), + ).toBe(false); + }); +}); + +// The asymmetric operations, as independent literals from the benchmark-harness +// spec (never task-suite.ts): gitea-axi outreaches the other arms on full-text +// search, diff, checks, checkout, and issue dependencies; and reports +// not-applicable for the operations outside its scope — repository, release, and +// milestone management. +const GITEA_AXI_EDGE_OPERATIONS = ["search", "diff", "checks", "checkout", "depend"]; +const OUT_OF_SCOPE_OPERATIONS = ["repository", "release", "milestone"]; + +describe("buildBonusTasks capability-asymmetry definitions", () => { + // Behavior: the bonus definitions cover the capability asymmetries in BOTH + // directions — operations where gitea-axi outreaches the other arms + // (gitea-axi-advantage / applicable) and operations outside gitea-axi's scope + // (gitea-axi-not-applicable / not-applicable) — and are kept separate from the + // scored suite (benchmark-harness spec, "bonus table"). Assessed on the static + // both-directions definitions (self-review permitted, so the self-review pair is + // absent). The named operations are independent literals from the spec. + it("covers gitea-axi's five edges and the three out-of-scope operations, disjoint from the scored suite", () => { + const bonus = buildBonusTasks({ selfReviewPermitted: true }); + + // Advantage direction: every advantage entry is applicable, and together the + // advantage operations cover each of gitea-axi's five named edges. + const advantage = bonus.filter((entry) => entry.direction === "gitea-axi-advantage"); + for (const entry of advantage) { + expect(entry.giteaAxi).toBe("applicable"); + } + const advantageText = advantage.map((entry) => entry.operation.toLowerCase()); + for (const edge of GITEA_AXI_EDGE_OPERATIONS) { + expect(advantageText.some((op) => op.includes(edge))).toBe(true); + } + + // Not-applicable direction: entries marked not-applicable cover repository, + // release, and milestone management. + const notApplicable = bonus.filter( + (entry) => entry.direction === "gitea-axi-not-applicable", + ); + for (const entry of notApplicable) { + expect(entry.giteaAxi).toBe("not-applicable"); + } + const notApplicableText = notApplicable.map((entry) => entry.operation.toLowerCase()); + for (const scope of OUT_OF_SCOPE_OPERATIONS) { + expect(notApplicableText.some((op) => op.includes(scope))).toBe(true); + } + + // Kept separate: no bonus-task id (under either option value) collides with a + // scored-suite task id (under either option value). + const bonusIds = new Set( + [ + ...buildBonusTasks({ selfReviewPermitted: true }), + ...buildBonusTasks({ selfReviewPermitted: false }), + ].map((entry) => entry.id), + ); + const scoredIds = new Set( + [ + ...buildScoredSuite({ selfReviewPermitted: true }), + ...buildScoredSuite({ selfReviewPermitted: false }), + ].map((task) => task.id), + ); + const overlap = [...bonusIds].filter((id) => scoredIds.has(id)); + expect(overlap).toEqual([]); + }); +}); diff --git a/bench/task-suite.ts b/bench/task-suite.ts new file mode 100644 index 0000000..832e1f4 --- /dev/null +++ b/bench/task-suite.ts @@ -0,0 +1,447 @@ +// The full scored task suite and the capability-asymmetric bonus definitions. +// +// The scored suite is 20 natural-language tasks drawn only from the capability +// surface every arm shares, weighted toward discovery and multi-step work; each +// task carries its tier and a scoring spec keyed on the single available user +// (see task.ts and scoring-spec.ts). The two review tasks are resolved by the +// self-review capability probe (self-review.ts): approve/request-changes where the +// host permits self-review, comment reviews otherwise, in which case the +// approve/request-changes operations fall to the bonus table instead. +// +// The bonus definitions cover the capability asymmetries kept out of the scored +// suite, in both directions: operations where gitea-axi outreaches the other arms +// and operations outside gitea-axi's scope, for which it is reported not-applicable. + +import { groundTruth, SEED_PLAN } from "./seed-plan.js"; +import type { + Issue, + PullRequest, + RepoState, + RequiredFact, + ReviewKind, + ScoringSpec, +} from "./scoring-spec.js"; +import type { BenchTask } from "./task.js"; + +/** The number a freshly created issue receives: issues and pull requests share one seed-filled number space. */ +const NEXT_ENTITY_NUMBER = SEED_PLAN.issues.length + SEED_PLAN.pullRequests.length + 1; + +/** Locate a seeded issue in an expected state by its title, failing loudly if the seed lacks it. */ +function issueByTitle(state: RepoState, title: string): Issue { + const issue = state.issues.find((candidate) => candidate.title === title); + if (issue === undefined) { + throw new Error(`no seed issue titled "${title}"`); + } + return issue; +} + +/** Locate a seeded pull request in an expected state by its title, failing loudly if the seed lacks it. */ +function pullByTitle(state: RepoState, title: string): PullRequest { + const pull = state.pullRequests.find((candidate) => candidate.title === title); + if (pull === undefined) { + throw new Error(`no seed pull request titled "${title}"`); + } + return pull; +} + +/** + * Build a mutation task's scoring spec as the seed's ground truth with a change + * applied. Because `groundTruth` returns a fresh state each call, the change is + * safe to mutate in place; the checker diffs the whole state, so anything the + * change does not touch is asserted unchanged and any collateral edit is caught. + */ +function mutationSpec(change: (state: RepoState, user: string) => void): (user: string) => ScoringSpec { + return (user) => { + const expected = groundTruth(user); + change(expected, user); + return { kind: "mutation", expected }; + }; +} + +/** Whether the host permits a user to approve or request changes on their own pull request. */ +export interface SuiteOptions { + selfReviewPermitted: boolean; +} + +/** A read task's scoring spec ignores the user; its required facts are seed-fixed. */ +const readSpec = (facts: RequiredFact[]): (() => ScoringSpec) => () => ({ kind: "read", facts }); + +/** Build a review task's scoring spec: the seed with one review of the given kind added to a pull request. */ +function reviewSpec(pullTitle: string, kind: ReviewKind, body: string): (user: string) => ScoringSpec { + return mutationSpec((state, user) => { + pullByTitle(state, pullTitle).reviews.push({ author: user, kind, body, comments: [] }); + }); +} + +/** + * The four read tasks: listing/counting, label filtering, viewing one entity's + * fields, and comment/review retrieval. Each fact's `anyOf` renderings recognise a + * natural human answer after the checker's case- and whitespace-normalization, so + * a correct report passes and a wrong one fails without an LLM judge. The five + * open issues, the three bug-labelled issue titles, the labels and state of + * "Crash when saving large files", and the seeded review body are all fixed by + * SEED_PLAN. + */ +function readTasks(): BenchTask[] { + return [ + { + id: "read-open-issue-count", + tier: "read", + intent: "Report how many issues in this repository are currently open.", + scoringSpec: readSpec([ + { + description: "the repository has 5 open issues", + anyOf: ["5 open", "five open", "open issues: 5", "open: 5", "5 issues are open"], + }, + ]), + }, + { + id: "read-bug-issue-titles", + tier: "read", + intent: 'List the titles of every issue that carries the "bug" label, including closed ones.', + scoringSpec: readSpec([ + { description: 'the bug issue "Fix crash on startup"', anyOf: ["Fix crash on startup"] }, + { + description: 'the bug issue "Crash when saving large files"', + anyOf: ["Crash when saving large files"], + }, + { description: 'the bug issue "Crash in export dialog"', anyOf: ["Crash in export dialog"] }, + ]), + }, + { + id: "read-issue-labels-and-state", + tier: "read", + intent: + 'For the issue titled "Crash when saving large files", report which labels it carries and whether it is open or closed.', + scoringSpec: readSpec([ + { description: 'the "bug" label', anyOf: ["bug"] }, + { description: 'the "priority" label', anyOf: ["priority"] }, + { description: "the issue is open", anyOf: ["open"] }, + ]), + }, + { + id: "read-review-body", + tier: "read", + intent: + 'Retrieve and report the body of the review left on the pull request titled "Fix startup crash".', + scoringSpec: readSpec([ + { description: "the seeded review body", anyOf: ["the error handling could be tightened"] }, + ]), + }, + ]; +} + +/** The six single-mutation tasks: close, reopen, comment, label, assign, and create. */ +function singleMutationTasks(_options: SuiteOptions): BenchTask[] { + return [ + { + id: "close-csv-export-issue", + tier: "single-mutation", + intent: 'Close the issue titled "Add CSV export option". Do not modify anything else in the repository.', + scoringSpec: mutationSpec((state) => { + issueByTitle(state, "Add CSV export option").state = "closed"; + }), + }, + { + id: "reopen-readme-badges", + tier: "single-mutation", + intent: 'Reopen the issue titled "Update README badges".', + scoringSpec: mutationSpec((state) => { + issueByTitle(state, "Update README badges").state = "open"; + }), + }, + { + id: "comment-save-large-files", + tier: "single-mutation", + intent: 'Add a comment saying "I can reproduce this too." to the issue titled "Crash when saving large files".', + scoringSpec: mutationSpec((state, user) => { + issueByTitle(state, "Crash when saving large files").comments.push({ + author: user, + body: "I can reproduce this too.", + }); + }), + }, + { + id: "label-startup-crash", + tier: "single-mutation", + intent: 'Add the "priority" label to the issue titled "Fix crash on startup".', + scoringSpec: mutationSpec((state) => { + issueByTitle(state, "Fix crash on startup").labels.push("priority"); + }), + }, + { + id: "assign-install-docs-typo", + tier: "single-mutation", + intent: 'Assign the issue titled "Typo in installation docs" to yourself.', + scoringSpec: mutationSpec((state, user) => { + issueByTitle(state, "Typo in installation docs").assignees.push(user); + }), + }, + { + id: "create-memory-leak-issue", + tier: "single-mutation", + intent: + 'Create a new issue titled "Investigate memory leak" with the body "Memory usage grows unbounded during long export runs."', + scoringSpec: mutationSpec((state) => { + state.issues.push({ + number: NEXT_ENTITY_NUMBER, + title: "Investigate memory leak", + body: "Memory usage grows unbounded during long export runs.", + state: "open", + labels: [], + assignees: [], + comments: [], + }); + }), + }, + ]; +} + +/** The six find-then-act tasks: each names its target by a property, not its title. */ +function findThenActTasks(options: SuiteOptions): BenchTask[] { + const reviewVerbA = options.selfReviewPermitted ? "approve it" : "leave a review"; + const reviewVerbB = options.selfReviewPermitted ? "request changes on it" : "leave a review"; + return [ + { + id: "fta-reopen-unlabelled-closed", + tier: "find-then-act", + intent: "Find the closed issue that has no labels and reopen it.", + scoringSpec: mutationSpec((state) => { + issueByTitle(state, "Typo in error message").state = "open"; + }), + }, + { + id: "fta-label-export-dialog-crash", + tier: "find-then-act", + intent: + 'Among the issues about crashes, find the one about the export dialog and add the "priority" label to it.', + scoringSpec: mutationSpec((state) => { + issueByTitle(state, "Crash in export dialog").labels.push("priority"); + }), + }, + { + id: "fta-merge-startup-pull", + tier: "find-then-act", + intent: "Find the open pull request that fixes the startup crash and merge it.", + scoringSpec: mutationSpec((state) => { + pullByTitle(state, "Fix startup crash").state = "merged"; + }), + }, + { + id: "fta-edit-install-docs-typo", + tier: "find-then-act", + intent: + `Find the issue about a typo in the installation documentation and change its body to "The install guide says 'yarn' in the setup step where it should say 'npm'."`, + scoringSpec: mutationSpec((state) => { + issueByTitle(state, "Typo in installation docs").body = + "The install guide says 'yarn' in the setup step where it should say 'npm'."; + }), + }, + { + id: "fta-review-csv-pull", + tier: "find-then-act", + intent: + `Find the pull request that implements CSV export and ${reviewVerbA} with the review comment "The CSV export path looks correct to me."`, + scoringSpec: reviewSpec( + "Implement CSV export", + options.selfReviewPermitted ? "approved" : "comment", + "The CSV export path looks correct to me.", + ), + }, + { + id: "fta-review-docs-pull", + tier: "find-then-act", + intent: + `Find the pull request that refreshes the documentation and ${reviewVerbB} with the review comment "Please expand the installation section before this merges."`, + scoringSpec: reviewSpec( + "Refresh documentation", + options.selfReviewPermitted ? "request-changes" : "comment", + "Please expand the installation section before this merges.", + ), + }, + ]; +} + +/** The four multi-step workflows: each applies several mutations in one task. */ +function multiStepTasks(_options: SuiteOptions): BenchTask[] { + return [ + { + id: "ms-triage-export-perf", + tier: "multi-step", + intent: + 'Triage the issue titled "Improve export performance": add the "priority" label, assign it to yourself, and leave a comment "Prioritised for the next sprint."', + scoringSpec: mutationSpec((state, user) => { + const issue = issueByTitle(state, "Improve export performance"); + issue.labels.push("priority"); + issue.assignees.push(user); + issue.comments.push({ author: user, body: "Prioritised for the next sprint." }); + }), + }, + { + id: "ms-close-open-crashes", + tier: "multi-step", + intent: + 'For every open issue whose title mentions a crash, add the comment "Consolidating crash reports." and then close it.', + scoringSpec: mutationSpec((state, user) => { + for (const issue of state.issues) { + if (issue.state === "open" && /crash/i.test(issue.title)) { + issue.comments.push({ author: user, body: "Consolidating crash reports." }); + issue.state = "closed"; + } + } + }), + }, + { + id: "ms-create-and-apply-stale", + tier: "multi-step", + intent: + 'Create a new label named "stale" with colour #cccccc, apply it to the issue titled "Add CSV export option", and then close that issue.', + scoringSpec: mutationSpec((state) => { + state.labels = [...state.labels, { name: "stale", color: "#cccccc" }]; + const issue = issueByTitle(state, "Add CSV export option"); + issue.labels.push("stale"); + issue.state = "closed"; + }), + }, + { + id: "ms-reopen-assign-comment-badges", + tier: "multi-step", + intent: + 'Reopen the issue titled "Update README badges", assign it to yourself, and add a comment "Reopening to refresh the badge URLs."', + scoringSpec: mutationSpec((state, user) => { + const issue = issueByTitle(state, "Update README badges"); + issue.state = "open"; + issue.assignees.push(user); + issue.comments.push({ author: user, body: "Reopening to refresh the badge URLs." }); + }), + }, + ]; +} + +/** Build the 20-task scored suite; the two review tasks reflect self-review support. */ +export function buildScoredSuite(options: SuiteOptions): BenchTask[] { + return [ + ...readTasks(), + ...singleMutationTasks(options), + ...findThenActTasks(options), + ...multiStepTasks(options), + ]; +} + +/** Which side of gitea-axi a bonus operation's capability asymmetry favours. */ +export type BonusDirection = + | "gitea-axi-advantage" + | "gitea-axi-not-applicable" + | "self-review-unavailable"; + +/** Whether a given arm can perform an operation at all. */ +export type Applicability = "applicable" | "not-applicable"; + +/** One capability-asymmetric operation reported in the bonus table, kept out of the scored suite. */ +export interface BonusTask { + id: string; + operation: string; + direction: BonusDirection; + note: string; + /** gitea-axi's own applicability for this operation. */ + giteaAxi: Applicability; +} + +/** + * The static capability-asymmetric bonus definitions, in both directions. The + * first group is where the other arms fall short of gitea-axi — full-text search, + * pull-request diff, checks, checkout, and issue dependencies — so gitea-axi is + * applicable and the asymmetry is its advantage. The second is outside gitea-axi's + * command surface entirely — repository, release, and milestone management — so + * gitea-axi is reported not-applicable. All are kept out of the scored suite (their + * ids share no namespace with it) so the headline comparison stays on the shared + * surface. + */ +const STATIC_BONUS: BonusTask[] = [ + { + id: "bonus-full-text-search", + operation: 'Search every issue for the word "crash" using full-text search and report the matches.', + direction: "gitea-axi-advantage", + note: "gitea-axi exposes the issue full-text search endpoint ergonomically; the other arms have no first-class equivalent.", + giteaAxi: "applicable", + }, + { + id: "bonus-pull-request-diff", + operation: 'Show the file diff proposed by the pull request titled "Implement CSV export".', + direction: "gitea-axi-advantage", + note: "gitea-axi renders a pull request's diff directly; the other arms must reconstruct it from raw endpoints or git.", + giteaAxi: "applicable", + }, + { + id: "bonus-pull-request-checks", + operation: 'Report the CI checks status of the pull request titled "Fix startup crash".', + direction: "gitea-axi-advantage", + note: "gitea-axi surfaces a pull request's commit-status checks; the other arms lack a dedicated affordance.", + giteaAxi: "applicable", + }, + { + id: "bonus-pull-request-checkout", + operation: 'Checkout the head branch of the pull request titled "Refresh documentation" to inspect it locally.', + direction: "gitea-axi-advantage", + note: "gitea-axi resolves a pull request to its head branch for checkout; the other arms leave this to manual git.", + giteaAxi: "applicable", + }, + { + id: "bonus-issue-dependencies", + operation: 'Add an issue dependency so that "Add CSV export option" depends on "Improve export performance".', + direction: "gitea-axi-advantage", + note: "gitea-axi manages issue dependencies; the other arms do not model them.", + giteaAxi: "applicable", + }, + { + id: "bonus-repository-management", + operation: 'Create a new repository named "archive-2026" under the current user.', + direction: "gitea-axi-not-applicable", + note: "Repository management is outside gitea-axi's command surface; it is reported not-applicable.", + giteaAxi: "not-applicable", + }, + { + id: "bonus-release-management", + operation: 'Publish a release tagged "v1.0.0" with notes on the repository.', + direction: "gitea-axi-not-applicable", + note: "Release management is outside gitea-axi's command surface; it is reported not-applicable.", + giteaAxi: "not-applicable", + }, + { + id: "bonus-milestone-management", + operation: 'Create a milestone "v2.0" and attach the open enhancement issues to it.', + direction: "gitea-axi-not-applicable", + note: "Milestone management is outside gitea-axi's command surface; it is reported not-applicable.", + giteaAxi: "not-applicable", + }, +]; + +/** + * The self-review pair, reported in the bonus table only when the host forbids a + * user from approving or requesting changes on their own pull request. When + * self-review is permitted these are scored directly (the two review tasks are + * promoted, see findThenActTasks), so they are absent from the bonus table then. + * The operation is within gitea-axi's own reach — the limitation is the host's, not + * the tool's — so gitea-axi is applicable. + */ +const SELF_REVIEW_BONUS: BonusTask[] = [ + { + id: "bonus-approve-own-pull", + operation: 'Approve your own pull request titled "Implement CSV export".', + direction: "self-review-unavailable", + note: "The host forbids approving one's own pull request, so no arm can perform it; the scored suite substitutes a comment review.", + giteaAxi: "applicable", + }, + { + id: "bonus-request-changes-own-pull", + operation: 'Request changes on your own pull request titled "Refresh documentation".', + direction: "self-review-unavailable", + note: "The host forbids requesting changes on one's own pull request, so no arm can perform it; the scored suite substitutes a comment review.", + giteaAxi: "applicable", + }, +]; + +/** The bonus definitions; the self-review pair is appended when self-review is unavailable. */ +export function buildBonusTasks(options: SuiteOptions): BonusTask[] { + return [...STATIC_BONUS, ...(options.selfReviewPermitted ? [] : SELF_REVIEW_BONUS)]; +}