Commit Graph

2 Commits

Author SHA1 Message Date
a6ab749211 fix: run the bench agent in a neutral working directory
Some checks failed
CI / test (pull_request) Failing after 52s
The SDK driver ran the agent with no explicit cwd, so its shell inherited
the harness's own checkout. When the agent omitted `-R OWNER/NAME`, the
gitea-axi (and tea) CLI defaulted the repository from that local checkout —
silently resolving the harness repo instead of the seeded throwaway — and
returned a plausible but wrong result (e.g. `count: 0 open of 0 total` for a
repo with no issues). This contaminated read-tier scoring for the checkout-
defaulting arms and was surfaced by the newly persisted read reports.

Give each run a fresh, empty working directory outside any checkout, so a
forgotten `-R` errors instead of hitting the wrong repository, and delete it
when the run ends.
2026-07-17 10:46:45 -04:00
1c92a389dd fix: capture per-model token usage in bench SDK driver
All checks were successful
CI / test (push) Successful in 52s
sumTokens read the Agent SDK's per-model `modelUsage` entries with
snake_case field names, but the SDK reports those per-model entries in
camelCase (`inputTokens`, `cacheReadInputTokens`, ...). Every token
component therefore fell through to zero, silently zeroing the
cost-equivalent-token headline metric — while `total_cost_usd` and
`num_turns` (top-level snake_case) kept working and masked it.

Read `modelUsage` with the correct camelCase fields, keeping the
snake_case aggregate `usage` as the fallback. Export `sumTokens` and add
a regression test covering both the per-model camelCase sum (folding in
the auxiliary model) and the snake_case fallback, so a future SDK
field-casing drift fails a test instead of producing zero-token samples.
2026-07-16 22:01:14 -04:00