feat: add benchmark tool-isolation guard (task 0023)
All checks were successful
CI / test (pull_request) Successful in 50s
CI / test (push) Successful in 49s

Add the guard that confines each benchmark arm's agent to exactly one
tool, so a result measures the tool rather than the agent's choice
between tools.

`guardCommand` inspects every binary a proposed shell command would
reach — across pipelines, sequences, subshells, command and process
substitutions, redirections, and leading environment assignments — and
permits only the active arm's one allow-listed binary plus a curated set
of harmless read-only utilities. Foreign binaries, absolute-path
evasions (even of the arm's own binary), and interpreter-based fetch
tricks are denied; the gitea-mcp arm runs with the shell disabled
entirely. `provisionArmBin` produces a curated per-arm bin directory
exposing only that arm's binary as the convenience layer behind the
authoritative guard.

Tests are colocated in bench/guard.test.ts and run via `npm run
test:bench`.
This commit was merged in pull request #24.
This commit is contained in:
2026-07-15 22:26:55 -04:00
parent 9bf8c85dc3
commit 0436dc25fd
4 changed files with 565 additions and 6 deletions

View File

@@ -10,8 +10,22 @@ The guard is a callback that inspects every proposed shell command and permits o
## Acceptance criteria
- [ ] Each arm's allow-listed binary passes the guard; a foreign binary is denied.
- [ ] An absolute-path invocation of a foreign binary is denied.
- [ ] An interpreter-based fetch attempt (e.g. driving an HTTP request through a language runtime) is denied.
- [ ] A curated per-arm PATH is produced exposing only that arm's allowed binary.
- [ ] Unit tests cover the allowed-binary, foreign-binary, absolute-path, and interpreter-fetch cases per arm.
- [x] Each arm's allow-listed binary passes the guard; a foreign binary is denied.
- [x] An absolute-path invocation of a foreign binary is denied.
- [x] An interpreter-based fetch attempt (e.g. driving an HTTP request through a language runtime) is denied.
- [x] A curated per-arm PATH is produced exposing only that arm's allowed binary.
- [x] Unit tests cover the allowed-binary, foreign-binary, absolute-path, and interpreter-fetch cases per arm.
## Implementation Notes
The guard lives in `bench/guard.ts` and exposes a small interface over a deliberately deep implementation:
- `guardCommand(arm, command)` — the authoritative guard, returning `{ allowed: true }` or `{ allowed: false, reason }`.
- `provisionArmBin(arm, binDir, locate?)` — populates a per-arm bin directory with a single symlink to the arm's binary (empty for `gitea-mcp`); `locate` is an injectable resolver so tests stay host-independent.
- `ARM_BINARY` and `HARMLESS_BINARIES` — the per-arm allow-listed binary (`null` for the shell-disabled `gitea-mcp` arm) and the curated set of harmless read/text/flow utilities.
Depth added beyond the literal criteria, invited by ADR 0016 ("isolation strength rests on the completeness of the guard's deny rules"): the guard checks *every* binary a command would reach, not just the leading token, via a hand-rolled shell command parser (`extractCommands`) that handles pipelines, `;`/`&&`/`||` sequences, subshells, `$(...)` and backtick substitutions, process substitutions, redirections (including `2>&1` and `&>` forms), and leading `NAME=value` assignments. This closes pipe-hiding and substitution-hiding evasions in addition to the named absolute-path and interpreter-fetch cases. Path-qualified invocations are refused even for the arm's own binary, since the curated PATH is meant to resolve it by name and a path-qualified form is a symlink/copy evasion vector.
The `gitea-mcp` arm runs with the shell disabled entirely (no allow-listed binary), so `guardCommand` denies every shell command for it with a shell-disabled reason, and `provisionArmBin` exposes nothing.
Tests are colocated in `bench/guard.test.ts` (31 tests) and run via `npm run test:bench`, kept out of the `src/` coverage tier.