docs: add benchmark harness spec, ADRs, and task breakdown
All checks were successful
CI / test (pull_request) Successful in 50s
CI / test (push) Successful in 49s

Add the benchmark-harness spec, three supporting ADRs (cost-equivalent
token metric, single-user seed, guard-based tool isolation), and the
0022-0030 task breakdown that slices the harness into foundational
seams, an integrating single-cell runner, and the reporting layer.
This commit was merged in pull request #22.
This commit is contained in:
2026-07-15 10:04:28 -04:00
parent b5955cd7b8
commit d445b2bd2b
13 changed files with 377 additions and 0 deletions

View File

@@ -0,0 +1,21 @@
# Guard-based tool isolation instead of containers
The benchmark's validity depends on each arm's agent reaching exactly one tool.
If the tea-arm agent could quietly call `curl` or `gitea-axi`, the comparison would be meaningless.
The benchmark machine has no container runtime available, so per-arm operating-system sandboxing is not an option.
## Considered Options
**Container per arm** (rejected) — A container image carrying only the arm's binary would give hard, kernel-level isolation, but it requires installing and depending on a container runtime the maintainer does not have and declined to add for this benchmark.
**Curated PATH alone** (rejected) — Prepending a directory that exposes only the allowed binary is convenient but leaky: an agent can invoke another tool by absolute path, or reach the API through a language interpreter's fetch, bypassing PATH entirely.
**Guard callback as the authority** (chosen) — A callback inspects every proposed shell command and permits only the one binary allow-listed for the active arm plus harmless utilities, denying foreign binaries, absolute-path evasions, and interpreter-based fetch attempts.
A curated per-arm PATH backs it as a convenience layer, and the gitea-mcp arm disables the shell tool entirely and attaches only the MCP tools, giving that arm no leakage surface at all.
## Consequences
- The guard, not the PATH, is authoritative; the PATH is defense in depth.
- A blocked attempt is left in the transcript and counts as realistic wasted effort, reflecting an agent fumbling with a tool that cannot do the job; blocked calls are never silently retried or discarded.
- Every command is logged, and a post-run audit asserts no foreign tool was reached; a detected leak flags the trial invalid rather than letting it be scored.
- Isolation strength rests on the completeness of the guard's deny rules, so the guard is one of the harness's primary unit-tested seams, covering absolute-path and interpreter-fetch evasions explicitly.