The gitea-axi arm was the only shell arm handed no credentials: the
runner set only PATH, so the agent had to reverse-engineer the tea-login
system — guessing a profile name and hunting for a config file — before
any real work, burning ~4 turns per task. Since turns drive cache-read,
the benchmark's dominant cost metric, this scaffolding gap alone inflated
gitea-axi's cost-equivalent tokens above every other arm.
Hand the arm its host and token through gitea-axi's own env interface
(GITEA_AXI_API_URL / GITEA_AXI_TOKEN), the symmetric counterpart to the
gitea-mcp server's GITEA_HOST / GITEA_ACCESS_TOKEN env: both name the
same two facts, and both still leave the agent to name the repository per
call. A shell arm now carries a credential env (empty for tea and
raw-api, which need none), merged under PATH in the driver.
Also strengthen SKILL.md so a cold agent targets and authenticates on the
first call: an explicit "Targeting and authentication" section replaces
the buried, optional-looking one-liner, spelling out that outside a
checkout `-R OWNER/NAME` plus the environment's token is all that is
needed — do not go hunting for a config file or login profile.
Verified live: create-memory-leak-issue dropped from 10 turns to 3 and
its cache-read fell ~3.8x, with the auth flailing gone from the transcript.
The SDK driver ran the agent with no explicit cwd, so its shell inherited
the harness's own checkout. When the agent omitted `-R OWNER/NAME`, the
gitea-axi (and tea) CLI defaulted the repository from that local checkout —
silently resolving the harness repo instead of the seeded throwaway — and
returned a plausible but wrong result (e.g. `count: 0 open of 0 total` for a
repo with no issues). This contaminated read-tier scoring for the checkout-
defaulting arms and was surfaced by the newly persisted read reports.
Give each run a fresh, empty working directory outside any checkout, so a
forgotten `-R` errors instead of hitting the wrong repository, and delete it
when the run ends.
sumTokens read the Agent SDK's per-model `modelUsage` entries with
snake_case field names, but the SDK reports those per-model entries in
camelCase (`inputTokens`, `cacheReadInputTokens`, ...). Every token
component therefore fell through to zero, silently zeroing the
cost-equivalent-token headline metric — while `total_cost_usd` and
`num_turns` (top-level snake_case) kept working and masked it.
Read `modelUsage` with the correct camelCase fields, keeping the
snake_case aggregate `usage` as the fallback. Export `sumTokens` and add
a regression test covering both the per-model camelCase sum (folding in
the auxiliary model) and the snake_case fallback, so a future SDK
field-casing drift fails a test instead of producing zero-token samples.
Thread every benchmark layer to run one (arm, task, trial) cell end to
end: provision and seed a throwaway repository, run the agent under the
active arm bounded by a turn cap and a wall-clock backstop, audit the
transcript, capture and score the post-run state, append the sample, and
delete the repository.
- runner.ts: runCell orchestration behind the BenchHost and AgentDriver
seams, so the flow is unit-tested with fakes while the live wiring is
validated by a smoke run; turn-cap and wall-clock failures are tagged
confused-versus-hung, and a leaked transcript is flagged invalid.
- audit.ts: the post-run transcript audit plus the shared
foreignToolReason predicate both isolation enforcement points consume.
- task.ts: the runnable BenchTask wrapper and one sample single-mutation
task exercising the full path.
- snapshot.ts: captureRepoState, the seed's read-back counterpart, in the
RepoState shape the checker diffs against.
- host.ts / sdk-driver.ts: the live BenchHost and the Claude Agent SDK
driver (an optional peer, loaded via dynamic import) for real runs.
- runner.smoke.test.ts: the live tracer-bullet tier, skipping cleanly
when no host or SDK is configured.
