feat: add benchmark task suite (task 0028)
All checks were successful
CI / test (pull_request) Successful in 51s
CI / test (push) Successful in 52s

Add the full 20-task scored suite and the capability-asymmetric bonus
definitions, plus the self-review capability probe that resolves the two
review tasks.

buildScoredSuite returns the shared-surface tasks weighted four read /
six single-mutation / six find-then-act / four multi-step, each a
natural-language intent parametrized against the seed and carrying a tier
and a scoring spec keyed on the single user. The two find-then-act review
tasks are approve/request-changes when the host permits self-review and
comment reviews otherwise; buildBonusTasks emits the approve/request-changes
operations as bonus entries in the fallback case, alongside the static
both-direction bonus definitions (gitea-axi's search/diff/checks/checkout/
issue-dependency edges, and the not-applicable repository/release/milestone
operations).

self-review.ts adds probeSelfReview and detectSelfReviewSupport, the live
boundary that determines self-review support once per sweep; it reuses the
now-exported non-throwing request helper from seed.ts.
This commit was merged in pull request #29.
This commit is contained in:
2026-07-16 09:51:22 -04:00
parent c6a972734e
commit e8310fb616
7 changed files with 1076 additions and 8 deletions

View File

@@ -0,0 +1,40 @@
import { describe, expect, it } from "vitest";
import type { CliDeps } from "../src/deps.js";
import { resolveBenchAccess, type BenchAccess } from "./seed.js";
import { detectSelfReviewSupport } from "./self-review.js";
/**
* The self-review probe smoke tier: a single live validation that the capability
* probe runs end-to-end against a real host — provisioning a throwaway
* repository, seeding it, checking whether the authenticated user may approve
* their own pull request, and cleaning the repository up — and reaches a definite
* boolean verdict without throwing. The benchmark-harness spec designates the
* self-review probe, like seed provisioning, as validated by a smoke run against
* a real host rather than by mocks, since its value is the real API interaction.
* Like the seed smoke tier, this suite skips cleanly when GITEA_AXI_BENCH_LOGIN
* is unset, which counts as a pass.
*
* The verdict itself is host-configuration-dependent — some hosts forbid a user
* from approving their own pull request, some permit it — so the assertion is
* only that a definite boolean is reached, never which value it is.
*/
const login = process.env.GITEA_AXI_BENCH_LOGIN;
describe.skipIf(!login)("self-review smoke: capability probe", () => {
it(
"runs the self-review probe against the live host and returns a definite boolean verdict",
async () => {
const deps: CliDeps = {
env: process.env,
cwd: process.cwd(),
globals: { login },
};
const access: BenchAccess = await resolveBenchAccess(deps, login!);
const result = await detectSelfReviewSupport(access);
expect(typeof result).toBe("boolean");
},
180_000,
);
});