Add the full 20-task scored suite and the capability-asymmetric bonus
definitions, plus the self-review capability probe that resolves the two
review tasks.
buildScoredSuite returns the shared-surface tasks weighted four read /
six single-mutation / six find-then-act / four multi-step, each a
natural-language intent parametrized against the seed and carrying a tier
and a scoring spec keyed on the single user. The two find-then-act review
tasks are approve/request-changes when the host permits self-review and
comment reviews otherwise; buildBonusTasks emits the approve/request-changes
operations as bonus entries in the fallback case, alongside the static
both-direction bonus definitions (gitea-axi's search/diff/checks/checkout/
issue-dependency edges, and the not-applicable repository/release/milestone
operations).
self-review.ts adds probeSelfReview and detectSelfReviewSupport, the live
boundary that determines self-review support once per sweep; it reuses the
now-exported non-throwing request helper from seed.ts.
Thread every benchmark layer to run one (arm, task, trial) cell end to
end: provision and seed a throwaway repository, run the agent under the
active arm bounded by a turn cap and a wall-clock backstop, audit the
transcript, capture and score the post-run state, append the sample, and
delete the repository.
- runner.ts: runCell orchestration behind the BenchHost and AgentDriver
seams, so the flow is unit-tested with fakes while the live wiring is
validated by a smoke run; turn-cap and wall-clock failures are tagged
confused-versus-hung, and a leaked transcript is flagged invalid.
- audit.ts: the post-run transcript audit plus the shared
foreignToolReason predicate both isolation enforcement points consume.
- task.ts: the runnable BenchTask wrapper and one sample single-mutation
task exercising the full path.
- snapshot.ts: captureRepoState, the seed's read-back counterpart, in the
RepoState shape the checker diffs against.
- host.ts / sdk-driver.ts: the live BenchHost and the Claude Agent SDK
driver (an optional peer, loaded via dynamic import) for real runs.
- runner.smoke.test.ts: the live tracer-bullet tier, skipping cleanly
when no host or SDK is configured.
Add the deterministic, idempotent seed that brings a freshly provisioned
throwaway repository to a known ground truth before a trial runs, scripted
over the live Gitea API.
- bench/seed-plan.ts: the pure ground truth (fixed labels, an issue spread
across the discriminating dimensions, and labelled/reviewed/real-branch
pull requests) plus groundTruth(user), realizing it into a RepoState.
- bench/seed.ts: idempotent seeding reconciled by natural key, reusing
gitea-axi's own tea-login credential discovery (no new secret handling).
- A live smoke tier (test:bench:smoke) validating the seed end-to-end and
skipping cleanly when no host is configured, kept out of the deterministic
bench tier.
Export selectLogin from src/context.ts so the bench reuses the exact
credential-selection path.
