alexion
e8310fb616
Add the full 20-task scored suite and the capability-asymmetric bonus definitions, plus the self-review capability probe that resolves the two review tasks. buildScoredSuite returns the shared-surface tasks weighted four read / six single-mutation / six find-then-act / four multi-step, each a natural-language intent parametrized against the seed and carrying a tier and a scoring spec keyed on the single user. The two find-then-act review tasks are approve/request-changes when the host permits self-review and comment reviews otherwise; buildBonusTasks emits the approve/request-changes operations as bonus entries in the fallback case, alongside the static both-direction bonus definitions (gitea-axi's search/diff/checks/checkout/ issue-dependency edges, and the not-applicable repository/release/milestone operations). self-review.ts adds probeSelfReview and detectSelfReviewSupport, the live boundary that determines self-review support once per sweep; it reuses the now-exported non-throwing request helper from seed.ts.
41 lines
1.7 KiB
TypeScript
41 lines
1.7 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
|
import type { CliDeps } from "../src/deps.js";
|
|
import { resolveBenchAccess, type BenchAccess } from "./seed.js";
|
|
import { detectSelfReviewSupport } from "./self-review.js";
|
|
|
|
/**
|
|
* The self-review probe smoke tier: a single live validation that the capability
|
|
* probe runs end-to-end against a real host — provisioning a throwaway
|
|
* repository, seeding it, checking whether the authenticated user may approve
|
|
* their own pull request, and cleaning the repository up — and reaches a definite
|
|
* boolean verdict without throwing. The benchmark-harness spec designates the
|
|
* self-review probe, like seed provisioning, as validated by a smoke run against
|
|
* a real host rather than by mocks, since its value is the real API interaction.
|
|
* Like the seed smoke tier, this suite skips cleanly when GITEA_AXI_BENCH_LOGIN
|
|
* is unset, which counts as a pass.
|
|
*
|
|
* The verdict itself is host-configuration-dependent — some hosts forbid a user
|
|
* from approving their own pull request, some permit it — so the assertion is
|
|
* only that a definite boolean is reached, never which value it is.
|
|
*/
|
|
const login = process.env.GITEA_AXI_BENCH_LOGIN;
|
|
|
|
describe.skipIf(!login)("self-review smoke: capability probe", () => {
|
|
it(
|
|
"runs the self-review probe against the live host and returns a definite boolean verdict",
|
|
async () => {
|
|
const deps: CliDeps = {
|
|
env: process.env,
|
|
cwd: process.cwd(),
|
|
globals: { login },
|
|
};
|
|
const access: BenchAccess = await resolveBenchAccess(deps, login!);
|
|
|
|
const result = await detectSelfReviewSupport(access);
|
|
|
|
expect(typeof result).toBe("boolean");
|
|
},
|
|
180_000,
|
|
);
|
|
});
|