feat: add benchmark task suite (task 0028)
Add the full 20-task scored suite and the capability-asymmetric bonus definitions, plus the self-review capability probe that resolves the two review tasks. buildScoredSuite returns the shared-surface tasks weighted four read / six single-mutation / six find-then-act / four multi-step, each a natural-language intent parametrized against the seed and carrying a tier and a scoring spec keyed on the single user. The two find-then-act review tasks are approve/request-changes when the host permits self-review and comment reviews otherwise; buildBonusTasks emits the approve/request-changes operations as bonus entries in the fallback case, alongside the static both-direction bonus definitions (gitea-axi's search/diff/checks/checkout/ issue-dependency edges, and the not-applicable repository/release/milestone operations). self-review.ts adds probeSelfReview and detectSelfReviewSupport, the live boundary that determines self-review support once per sweep; it reuses the now-exported non-throwing request helper from seed.ts.
This commit was merged in pull request #29.
This commit is contained in:
40
bench/self-review.smoke.test.ts
Normal file
40
bench/self-review.smoke.test.ts
Normal file
@@ -0,0 +1,40 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import type { CliDeps } from "../src/deps.js";
|
||||
import { resolveBenchAccess, type BenchAccess } from "./seed.js";
|
||||
import { detectSelfReviewSupport } from "./self-review.js";
|
||||
|
||||
/**
|
||||
* The self-review probe smoke tier: a single live validation that the capability
|
||||
* probe runs end-to-end against a real host — provisioning a throwaway
|
||||
* repository, seeding it, checking whether the authenticated user may approve
|
||||
* their own pull request, and cleaning the repository up — and reaches a definite
|
||||
* boolean verdict without throwing. The benchmark-harness spec designates the
|
||||
* self-review probe, like seed provisioning, as validated by a smoke run against
|
||||
* a real host rather than by mocks, since its value is the real API interaction.
|
||||
* Like the seed smoke tier, this suite skips cleanly when GITEA_AXI_BENCH_LOGIN
|
||||
* is unset, which counts as a pass.
|
||||
*
|
||||
* The verdict itself is host-configuration-dependent — some hosts forbid a user
|
||||
* from approving their own pull request, some permit it — so the assertion is
|
||||
* only that a definite boolean is reached, never which value it is.
|
||||
*/
|
||||
const login = process.env.GITEA_AXI_BENCH_LOGIN;
|
||||
|
||||
describe.skipIf(!login)("self-review smoke: capability probe", () => {
|
||||
it(
|
||||
"runs the self-review probe against the live host and returns a definite boolean verdict",
|
||||
async () => {
|
||||
const deps: CliDeps = {
|
||||
env: process.env,
|
||||
cwd: process.cwd(),
|
||||
globals: { login },
|
||||
};
|
||||
const access: BenchAccess = await resolveBenchAccess(deps, login!);
|
||||
|
||||
const result = await detectSelfReviewSupport(access);
|
||||
|
||||
expect(typeof result).toBe("boolean");
|
||||
},
|
||||
180_000,
|
||||
);
|
||||
});
|
||||
Reference in New Issue
Block a user