alexion
d445b2bd2b
Add the benchmark-harness spec, three supporting ADRs (cost-equivalent token metric, single-user seed, guard-based tool isolation), and the 0022-0030 task breakdown that slices the harness into foundational seams, an integrating single-cell runner, and the reporting layer.
22 lines
2.0 KiB
Markdown
22 lines
2.0 KiB
Markdown
# Guard-based tool isolation instead of containers
|
|
|
|
The benchmark's validity depends on each arm's agent reaching exactly one tool.
|
|
If the tea-arm agent could quietly call `curl` or `gitea-axi`, the comparison would be meaningless.
|
|
The benchmark machine has no container runtime available, so per-arm operating-system sandboxing is not an option.
|
|
|
|
## Considered Options
|
|
|
|
**Container per arm** (rejected) — A container image carrying only the arm's binary would give hard, kernel-level isolation, but it requires installing and depending on a container runtime the maintainer does not have and declined to add for this benchmark.
|
|
|
|
**Curated PATH alone** (rejected) — Prepending a directory that exposes only the allowed binary is convenient but leaky: an agent can invoke another tool by absolute path, or reach the API through a language interpreter's fetch, bypassing PATH entirely.
|
|
|
|
**Guard callback as the authority** (chosen) — A callback inspects every proposed shell command and permits only the one binary allow-listed for the active arm plus harmless utilities, denying foreign binaries, absolute-path evasions, and interpreter-based fetch attempts.
|
|
A curated per-arm PATH backs it as a convenience layer, and the gitea-mcp arm disables the shell tool entirely and attaches only the MCP tools, giving that arm no leakage surface at all.
|
|
|
|
## Consequences
|
|
|
|
- The guard, not the PATH, is authoritative; the PATH is defense in depth.
|
|
- A blocked attempt is left in the transcript and counts as realistic wasted effort, reflecting an agent fumbling with a tool that cannot do the job; blocked calls are never silently retried or discarded.
|
|
- Every command is logged, and a post-run audit asserts no foreign tool was reached; a detected leak flags the trial invalid rather than letting it be scored.
|
|
- Isolation strength rests on the completeness of the guard's deny rules, so the guard is one of the harness's primary unit-tested seams, covering absolute-path and interpreter-fetch evasions explicitly.
|