fix: capture per-model token usage in bench SDK driver #33

alexion · 2026-07-16T22:01:00-04:00

alexion commented

2026-07-16 22:01:00 -04:00

Fixes a benchmark-harness bug: the SDK driver recorded zero token components for every run, silently zeroing the cost-equivalent-token headline metric the benchmark exists to measure.

Root cause

sumTokens read the Agent SDK's per-model modelUsage entries with snake_case field names (input_tokens, cache_read_input_tokens, …), but the SDK reports those per-model entries in camelCase (inputTokens, cacheReadInputTokens, …). Every component fell through to ?? 0. Top-level total_cost_usd / num_turns are snake_case and kept working, which masked the problem (cost and turns looked fine while tokens were all zero).

Fix

Read modelUsage with the correct camelCase fields, keeping the snake_case aggregate usage as the fallback (the two SDK shapes genuinely use different casing). Export sumTokens and add bench/sdk-driver.test.ts (2 tests): the per-model camelCase sum folding in the auxiliary model, and the snake_case aggregate fallback — so a future SDK field-casing drift fails a test instead of silently producing zero-token samples.

Verification

Full bench tier green (106 tests, incl. the 2 new guards); typecheck clean.
Verified live against Agent SDK v0.3.211: a fresh 60-sample gitea-mcp run records realistic token breakdowns (mean cache-read ~51.7k/run, 0 of 60 samples with zero tokens) vs. all-zeros before the fix.

Fixes a benchmark-harness bug: the SDK driver recorded **zero token components for every run**, silently zeroing the cost-equivalent-token headline metric the benchmark exists to measure. ## Root cause `sumTokens` read the Agent SDK's per-model `modelUsage` entries with snake_case field names (`input_tokens`, `cache_read_input_tokens`, …), but the SDK reports those per-model entries in **camelCase** (`inputTokens`, `cacheReadInputTokens`, …). Every component fell through to `?? 0`. Top-level `total_cost_usd` / `num_turns` are snake_case and kept working, which masked the problem (cost and turns looked fine while tokens were all zero). ## Fix Read `modelUsage` with the correct camelCase fields, keeping the snake_case aggregate `usage` as the fallback (the two SDK shapes genuinely use different casing). Export `sumTokens` and add `bench/sdk-driver.test.ts` (2 tests): the per-model camelCase sum folding in the auxiliary model, and the snake_case aggregate fallback — so a future SDK field-casing drift fails a test instead of silently producing zero-token samples. ## Verification - Full bench tier green (106 tests, incl. the 2 new guards); typecheck clean. - Verified live against Agent SDK **v0.3.211**: a fresh 60-sample gitea-mcp run records realistic token breakdowns (mean cache-read ~51.7k/run, 0 of 60 samples with zero tokens) vs. all-zeros before the fix.

alexion added 1 commit 2026-07-16 22:01:00 -04:00

fix: capture per-model token usage in bench SDK driver

CI / test (pull_request) Successful in 55s

Details

690456d21e

sumTokens read the Agent SDK's per-model `modelUsage` entries with
snake_case field names, but the SDK reports those per-model entries in
camelCase (`inputTokens`, `cacheReadInputTokens`, ...). Every token
component therefore fell through to zero, silently zeroing the
cost-equivalent-token headline metric — while `total_cost_usd` and
`num_turns` (top-level snake_case) kept working and masked it.

Read `modelUsage` with the correct camelCase fields, keeping the
snake_case aggregate `usage` as the fallback. Export `sumTokens` and add
a regression test covering both the per-model camelCase sum (folding in
the auxiliary model) and the snake_case fallback, so a future SDK
field-casing drift fails a test instead of producing zero-token samples.

alexion merged commit 1c92a389dd into main

2026-07-16 22:01:15 -04:00

alexion deleted branch fix-bench-token-extraction

2026-07-16 22:01:15 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: alexion/gitea-axi#33