feat: add benchmark tool-isolation guard (task 0023)
Add the guard that confines each benchmark arm's agent to exactly one tool, so a result measures the tool rather than the agent's choice between tools. `guardCommand` inspects every binary a proposed shell command would reach — across pipelines, sequences, subshells, command and process substitutions, redirections, and leading environment assignments — and permits only the active arm's one allow-listed binary plus a curated set of harmless read-only utilities. Foreign binaries, absolute-path evasions (even of the arm's own binary), and interpreter-based fetch tricks are denied; the gitea-mcp arm runs with the shell disabled entirely. `provisionArmBin` produces a curated per-arm bin directory exposing only that arm's binary as the convenience layer behind the authoritative guard. Tests are colocated in bench/guard.test.ts and run via `npm run test:bench`.
This commit was merged in pull request #24.
This commit is contained in:
@@ -33,8 +33,9 @@ The raw component breakdown is retained on every sample so the data can be re-we
|
||||
|
||||
- `result.ts` — the immutable result-record shape and its tags (arm, task, tier, trial, timestamp).
|
||||
- `store.ts` — the append-only, per-cell sample store that accumulates result records.
|
||||
- `guard.ts` — the authoritative tool-isolation guard plus the curated per-arm bin directory that backs it.
|
||||
|
||||
Later slices add the tool-isolation guard, the seed provisioning, the arm scaffolding, the single-cell runner, the task suite, the run-loop CLI, and the aggregator.
|
||||
Later slices add the seed provisioning, the arm scaffolding, the single-cell runner, the task suite, the run-loop CLI, and the aggregator.
|
||||
|
||||
## Tests
|
||||
|
||||
|
||||
Reference in New Issue
Block a user