The state-aware count line from task 0033 (1166a48) renders
`count: N open of M total`, and the unit tests were updated to match, but
the three e2e tracer assertions still expected the old bare
`count: N of M total`. They are skipped without GITEA_AXI_E2E_URL, so the
staleness only surfaced in CI, where the e2e tier runs. Update them to the
state-qualified form the shipped code already produces — default `open`,
`--state closed` → `closed`. The code was correct; the tests were stale.
The prior results table and narrative claimed gitea-axi posts the lowest
cost-equivalent tokens. That snapshot predated the neutral-working-dir
isolation fix, when the checkout-defaulting arms (gitea-axi, tea) drew
repo and login for free from the harness's own checkout — so gitea-axi
was implicitly pre-authenticated and looked like the winner.
On a clean run with every arm fairly credentialed and executed together,
raw REST is the cheapest on cost-equivalent tokens and leads every tier —
terse HTTP is the token floor no wrapper undercuts. gitea-axi is a clear
second overall and the lowest-cost structured interface, beating gitea-mcp
and tea on every tier at 100% success. Keep the raw-REST arm and state
this plainly rather than crown the wrapper by omitting the floor.
Numbers regenerated from bench:report over the 240-sample clean snapshot.
`issue view` rendered state but never labels, and offered no way to add
them — so reading one issue's labels forced a detour through
`issue list --fields labels` and hunting the matching row. The benchmark
transcripts showed agents paying this round-trip on every labels/state
read.
Show labels by default in the detail view (a detail view should be
complete), and add a `--fields` flag mirroring `issue list` / `search`
to append assignees, closedAt, milestone, updatedAt, url on request.
Also strengthen SKILL.md against the two command-discovery round-trips
the transcripts exposed: name the required `search issues` / `search prs`
subcommand form (a bare `search "<query>"` is invalid), and point agents
straight at `issue view <n>` for a single issue's fields.
Verified live: read-issue-labels-and-state dropped from 10 turns to 4
(cache-read ~3.3x lower), the transcript reduced to three clean commands
with the search-help and issue-list round-trips gone.
The gitea-axi arm was the only shell arm handed no credentials: the
runner set only PATH, so the agent had to reverse-engineer the tea-login
system — guessing a profile name and hunting for a config file — before
any real work, burning ~4 turns per task. Since turns drive cache-read,
the benchmark's dominant cost metric, this scaffolding gap alone inflated
gitea-axi's cost-equivalent tokens above every other arm.
Hand the arm its host and token through gitea-axi's own env interface
(GITEA_AXI_API_URL / GITEA_AXI_TOKEN), the symmetric counterpart to the
gitea-mcp server's GITEA_HOST / GITEA_ACCESS_TOKEN env: both name the
same two facts, and both still leave the agent to name the repository per
call. A shell arm now carries a credential env (empty for tea and
raw-api, which need none), merged under PATH in the driver.
Also strengthen SKILL.md so a cold agent targets and authenticates on the
first call: an explicit "Targeting and authentication" section replaces
the buried, optional-looking one-liner, spelling out that outside a
checkout `-R OWNER/NAME` plus the environment's token is all that is
needed — do not go hunting for a config file or login profile.
Verified live: create-memory-leak-issue dropped from 10 turns to 3 and
its cache-read fell ~3.8x, with the auth flailing gone from the transcript.
Records stored only a run's token/turn totals, so an arm's turn cost —
the dominant driver of cache-read tokens — could not be diagnosed from
the store. Retain the ordered transcript of tool invocations (the exact
shell commands, MCP calls, and built-in tools the run made) on every
scored record, absent only for a hung run that produced no transcript.
The canonical TranscriptEntry shape lives on the record (result.ts); the
isolation audit's ToolUse now aliases it so the persisted and audited
shapes cannot drift.
The read checker matched a task's required-fact phrasings as plain
substrings of the agent's report after only lowercasing and collapsing
whitespace. An answer that was substantively correct but wrapped a value in
markdown (e.g. `**5**`) failed the match, because the emphasis markers broke
the phrase adjacency (`**5** open` does not contain `5 open`) — a correct
answer scored incorrect on formatting alone.
Strip markdown emphasis/code markers (`*`, `_`, backtick) during
normalization so the match is on substance, not presentation. A guard test
confirms a wrong value still fails after stripping.
The SDK driver ran the agent with no explicit cwd, so its shell inherited
the harness's own checkout. When the agent omitted `-R OWNER/NAME`, the
gitea-axi (and tea) CLI defaulted the repository from that local checkout —
silently resolving the harness repo instead of the seeded throwaway — and
returned a plausible but wrong result (e.g. `count: 0 open of 0 total` for a
repo with no issues). This contaminated read-tier scoring for the checkout-
defaulting arms and was surfaced by the newly persisted read reports.
Give each run a fresh, empty working directory outside any checkout, so a
forgotten `-R` errors instead of hitting the wrong repository, and delete it
when the run ends.
Make the `issue list` count line name the state it filtered on, so the
answer to "how many issues are open?" is present on the summary line
rather than only inferable from each row. The count line now renders
`count: 5 open of 5 total`; the command composes the state into a generic
optional qualifier while `formatCountLine` stays state-agnostic, so
`pr list`, `search`, and `dashboard` are unaffected. `--state all` imposes
no narrowing and stays unqualified.
The wording is chosen so an agent quoting the summary lands on a phrase the
benchmark read-checker already accepts (`5 open`), closing the accuracy gap
this feature targets. Pairs with the report persistence in task 0032.
Retain the agent's final report on the benchmark result record for read
tasks, so a failed read is diagnosable directly from the stored record
instead of only carrying an opaque `incorrect` tag. The runner resolves
the scoring spec once and records `run.finalReport` when the spec is a
read; mutation records omit the field entirely. The sample store needs no
change — it serializes whatever record it is handed.
This is the prerequisite for confirming the read-open-issue-count failure
from real report text before the state-aware count-line change (task 0033).
