fix: make bench arm bin provisioning idempotent across trials #34

alexion · 2026-07-16T22:13:56-04:00

alexion commented

2026-07-16 22:13:56 -04:00

Fixes a benchmark-harness bug that crashed every shell arm (gitea-axi, tea, raw-api) at more than one trial per sitting.

Root cause

provisionArmBin symlinks the shell arm's one allow-listed binary into a per-sitting bin directory. But a sitting's trials share that directory (runCells creates one binRoot and passes it to every trial), and each trial re-runs provisionArmBin, so the second trial's symlinkSync threw EEXIST: file already exists. The gitea-mcp arm was unaffected because its shell is disabled (ARM_BINARY is null), so it symlinks nothing — which is why the earlier 60-sample gitea-mcp run never hit this.

Fix

Remove any existing link before re-creating it, so provisioning is idempotent across a sitting's trials. Add a regression test that provisions the same bin directory twice and asserts it neither throws nor duplicates the link.

Verification

Full bench tier green (107 tests, incl. the new idempotency guard); typecheck clean.
Verified live: a 3-trial gitea-axi cell (which previously crashed at trial 2) now records all 3 trials with 0 invalids.

Fixes a benchmark-harness bug that crashed **every shell arm** (`gitea-axi`, `tea`, `raw-api`) at more than one trial per sitting. ## Root cause `provisionArmBin` symlinks the shell arm's one allow-listed binary into a per-sitting bin directory. But a sitting's trials **share** that directory (`runCells` creates one `binRoot` and passes it to every trial), and each trial re-runs `provisionArmBin`, so the second trial's `symlinkSync` threw `EEXIST: file already exists`. The `gitea-mcp` arm was unaffected because its shell is disabled (`ARM_BINARY` is `null`), so it symlinks nothing — which is why the earlier 60-sample gitea-mcp run never hit this. ## Fix Remove any existing link before re-creating it, so provisioning is idempotent across a sitting's trials. Add a regression test that provisions the same bin directory twice and asserts it neither throws nor duplicates the link. ## Verification - Full bench tier green (107 tests, incl. the new idempotency guard); typecheck clean. - Verified live: a 3-trial `gitea-axi` cell (which previously crashed at trial 2) now records all 3 trials with 0 invalids.

alexion added 1 commit 2026-07-16 22:13:57 -04:00

fix: make bench arm bin provisioning idempotent across trials

CI / test (pull_request) Successful in 1m5s

Details

9ceadb3482

provisionArmBin symlinks the shell arm's one allow-listed binary into a
per-sitting bin directory, but a sitting's trials share that directory and
re-provision it each trial, so the second trial's symlinkSync threw EEXIST.
This crashed every shell arm (gitea-axi, tea, raw-api) at more than one
trial per sitting; the gitea-mcp arm was unaffected because its shell is
disabled and it symlinks nothing.

Remove any existing link before re-creating it so provisioning is
idempotent. Add a regression test that provisions the same bin directory
twice and asserts it neither throws nor duplicates the link.

alexion merged commit de75ac9479 into main

2026-07-16 22:14:08 -04:00

alexion deleted branch fix-bench-shell-arm-reprovision

2026-07-16 22:14:09 -04:00

alexion referenced this issue from a commit

2026-07-16 22:14:10 -04:00

fix: make bench arm bin provisioning idempotent across trials (#34)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: alexion/gitea-axi#34