alexion
d445b2bd2b
Add the benchmark-harness spec, three supporting ADRs (cost-equivalent token metric, single-user seed, guard-based tool isolation), and the 0022-0030 task breakdown that slices the harness into foundational seams, an integrating single-cell runner, and the reporting layer.
1.2 KiB
1.2 KiB
spec
| spec |
|---|
| benchmark-harness |
What to build
The guard that keeps each arm's agent confined to exactly one tool, so a benchmark result measures the tool rather than the agent's choice between tools (see the guard-based-tool-isolation ADR).
The guard is a callback that inspects every proposed shell command and permits only the one binary allow-listed for the active arm plus a curated set of harmless utilities, denying everything else. It rejects foreign binaries, absolute-path evasions that sidestep the allow-list, and interpreter-based fetch tricks (reaching the API through a language runtime's HTTP client). A curated per-arm PATH backs the guard as a convenience layer, but the guard — not the PATH — is authoritative. A blocked attempt is surfaced, not silently retried.
Acceptance criteria
- Each arm's allow-listed binary passes the guard; a foreign binary is denied.
- An absolute-path invocation of a foreign binary is denied.
- An interpreter-based fetch attempt (e.g. driving an HTTP request through a language runtime) is denied.
- A curated per-arm PATH is produced exposing only that arm's allowed binary.
- Unit tests cover the allowed-binary, foreign-binary, absolute-path, and interpreter-fetch cases per arm.