alexion
d445b2bd2b
Add the benchmark-harness spec, three supporting ADRs (cost-equivalent token metric, single-user seed, guard-based tool isolation), and the 0022-0030 task breakdown that slices the harness into foundational seams, an integrating single-cell runner, and the reporting layer.
2.0 KiB
Guard-based tool isolation instead of containers
The benchmark's validity depends on each arm's agent reaching exactly one tool.
If the tea-arm agent could quietly call curl or gitea-axi, the comparison would be meaningless.
The benchmark machine has no container runtime available, so per-arm operating-system sandboxing is not an option.
Considered Options
Container per arm (rejected) — A container image carrying only the arm's binary would give hard, kernel-level isolation, but it requires installing and depending on a container runtime the maintainer does not have and declined to add for this benchmark.
Curated PATH alone (rejected) — Prepending a directory that exposes only the allowed binary is convenient but leaky: an agent can invoke another tool by absolute path, or reach the API through a language interpreter's fetch, bypassing PATH entirely.
Guard callback as the authority (chosen) — A callback inspects every proposed shell command and permits only the one binary allow-listed for the active arm plus harmless utilities, denying foreign binaries, absolute-path evasions, and interpreter-based fetch attempts. A curated per-arm PATH backs it as a convenience layer, and the gitea-mcp arm disables the shell tool entirely and attaches only the MCP tools, giving that arm no leakage surface at all.
Consequences
- The guard, not the PATH, is authoritative; the PATH is defense in depth.
- A blocked attempt is left in the transcript and counts as realistic wasted effort, reflecting an agent fumbling with a tool that cannot do the job; blocked calls are never silently retried or discarded.
- Every command is logged, and a post-run audit asserts no foreign tool was reached; a detected leak flags the trial invalid rather than letting it be scored.
- Isolation strength rests on the completeness of the guard's deny rules, so the guard is one of the harness's primary unit-tested seams, covering absolute-path and interpreter-fetch evasions explicitly.