feat: add benchmark single-cell runner (task 0027)
All checks were successful
CI / test (pull_request) Successful in 50s
CI / test (push) Successful in 52s

Thread every benchmark layer to run one (arm, task, trial) cell end to
end: provision and seed a throwaway repository, run the agent under the
active arm bounded by a turn cap and a wall-clock backstop, audit the
transcript, capture and score the post-run state, append the sample, and
delete the repository.

- runner.ts: runCell orchestration behind the BenchHost and AgentDriver
  seams, so the flow is unit-tested with fakes while the live wiring is
  validated by a smoke run; turn-cap and wall-clock failures are tagged
  confused-versus-hung, and a leaked transcript is flagged invalid.
- audit.ts: the post-run transcript audit plus the shared
  foreignToolReason predicate both isolation enforcement points consume.
- task.ts: the runnable BenchTask wrapper and one sample single-mutation
  task exercising the full path.
- snapshot.ts: captureRepoState, the seed's read-back counterpart, in the
  RepoState shape the checker diffs against.
- host.ts / sdk-driver.ts: the live BenchHost and the Claude Agent SDK
  driver (an optional peer, loaded via dynamic import) for real runs.
- runner.smoke.test.ts: the live tracer-bullet tier, skipping cleanly
  when no host or SDK is configured.
This commit was merged in pull request #28.
This commit is contained in:
2026-07-16 09:17:38 -04:00
parent 9a2ba40657
commit c6a972734e
13 changed files with 1529 additions and 11 deletions

86
bench/audit.ts Normal file
View File

@@ -0,0 +1,86 @@
// The post-run transcript audit: a defence-in-depth check that re-inspects a
// completed run's tool invocations and asserts no foreign tool was reached. The
// guard (guard.ts) is the primary, in-band enforcement — it denies a foreign
// shell command before it runs — but the audit is the independent backstop the
// benchmark trusts: if enforcement ever leaked, a run in which a foreign tool
// actually executed is flagged invalid rather than being scored (see the
// benchmark-harness spec's testing decisions).
//
// This module is pure — it re-runs the arm's own guard over the recorded shell
// commands and checks the arm's channel discipline (shell arms never reach MCP
// tools; the MCP arm never reaches the shell). It does not run the agent; the
// runner (runner.ts) drives the run and feeds the transcript here.
import type { ArmDefinition } from "./arm.js";
/**
* One tool invocation recorded in the agent's transcript, reduced to what the
* isolation audit needs. `shell` is a proposed shell command; `mcp` is a call to
* an attached MCP server's tool; `other` is a built-in, non-Gitea-reaching tool
* (file read/edit and the like) that carries no isolation risk.
*/
export type ToolUse =
| { kind: "shell"; command: string }
| { kind: "mcp"; server: string; tool: string }
| { kind: "other"; name: string };
/**
* The audit's verdict. On a leak it carries a human-readable reason per foreign
* tool that was reached, so an invalidated trial can be diagnosed from the record.
*/
export type AuditResult = { clean: true } | { clean: false; leaks: string[] };
/**
* The single source of truth for whether one tool is foreign to an arm: returns a
* human-readable reason it must not run, or `null` when it is permitted. A shell
* arm puts every Bash command through its own guard and admits every non-shell
* built-in, but has no MCP server; the MCP arm disables the shell entirely and
* admits its MCP tools. Built-in `other` tools reach no Gitea channel and are
* always permitted.
*
* Both isolation enforcement points share this predicate so they cannot drift: the
* agent driver (sdk-driver.ts) consults it in-band to deny a foreign tool before
* it runs, and `auditTranscript` re-applies it post-run as the independent backstop.
*/
export function foreignToolReason(arm: ArmDefinition, use: ToolUse): string | null {
if (use.kind === "shell") {
if (arm.shell === null) {
return `the ${arm.arm} arm runs with the shell disabled; only its MCP tools are available`;
}
const decision = arm.shell.guard(use.command);
return decision.allowed ? null : decision.reason;
}
if (use.kind === "mcp") {
return arm.mcp === null ? `the ${arm.arm} arm has no MCP server attached` : null;
}
return null;
}
/** A human-readable rendering of a leaked tool use, tagging it with the reason. */
function describeLeak(use: ToolUse, reason: string): string {
switch (use.kind) {
case "shell":
return `foreign shell command ${JSON.stringify(use.command)} reached: ${reason}`;
case "mcp":
return `MCP tool "${use.server}/${use.tool}" reached: ${reason}`;
case "other":
return `tool "${use.name}" reached: ${reason}`;
}
}
/**
* Re-check a completed run's transcript against the arm's isolation rules,
* re-applying `foreignToolReason` to every executed tool. A tool the arm should
* never have reached — a guard-denied shell command, a shell command on the MCP
* arm, an MCP call on a shell arm — is reported as a leak. Clean when nothing leaked.
*/
export function auditTranscript(arm: ArmDefinition, transcript: ToolUse[]): AuditResult {
const leaks: string[] = [];
for (const use of transcript) {
const reason = foreignToolReason(arm, use);
if (reason !== null) {
leaks.push(describeLeak(use, reason));
}
}
return leaks.length === 0 ? { clean: true } : { clean: false, leaks };
}