Commit Graph

2 Commits

Author SHA1 Message Date
9ceadb3482 fix: make bench arm bin provisioning idempotent across trials
All checks were successful
CI / test (pull_request) Successful in 1m5s
provisionArmBin symlinks the shell arm's one allow-listed binary into a
per-sitting bin directory, but a sitting's trials share that directory and
re-provision it each trial, so the second trial's symlinkSync threw EEXIST.
This crashed every shell arm (gitea-axi, tea, raw-api) at more than one
trial per sitting; the gitea-mcp arm was unaffected because its shell is
disabled and it symlinks nothing.

Remove any existing link before re-creating it so provisioning is
idempotent. Add a regression test that provisions the same bin directory
twice and asserts it neither throws nor duplicates the link.
2026-07-16 22:13:38 -04:00
0436dc25fd feat: add benchmark tool-isolation guard (task 0023)
All checks were successful
CI / test (pull_request) Successful in 50s
CI / test (push) Successful in 49s
Add the guard that confines each benchmark arm's agent to exactly one
tool, so a result measures the tool rather than the agent's choice
between tools.

`guardCommand` inspects every binary a proposed shell command would
reach — across pipelines, sequences, subshells, command and process
substitutions, redirections, and leading environment assignments — and
permits only the active arm's one allow-listed binary plus a curated set
of harmless read-only utilities. Foreign binaries, absolute-path
evasions (even of the arm's own binary), and interpreter-based fetch
tricks are denied; the gitea-mcp arm runs with the shell disabled
entirely. `provisionArmBin` produces a curated per-arm bin directory
exposing only that arm's binary as the convenience layer behind the
authoritative guard.

Tests are colocated in bench/guard.test.ts and run via `npm run
test:bench`.
2026-07-15 22:26:55 -04:00