bench:run gains a --skill <path> flag, threaded into the existing
BuildArmOptions.skillPath, so a skill variant can be A/B'd against the
shipped SKILL.md with the same binary and harness — the mechanism used to
validate the find-then-act rewrite — without mutating the shipped file.
Two benchmark-harness races were being scored as agent failures. A freshly
seeded throwaway repo could 404 for an independent reader (the agent, a fresh
process) before Gitea made it consistent, and the async issue indexer lagged so
`search issues`/`search prs` returned nothing right after seeding — leaving the
agent unable to find the target it was asked to act on.
seedRepo now blocks until an independent read confirms the repo is reachable,
its full seeded issue and pull spread is visible, and a seeded issue and pull
are returned by the search index, before releasing the agent. It polls up to
60s and throws a loud harness error on timeout rather than letting a
propagation delay become a scored agent failure.
Also add an optional `pattern` regex to RequiredFact so a read answer's count is
recognised semantically rather than as a fixed phrase: "5 issues are currently
open" no longer fails against the literal "5 issues are open", while the
alphabetic-only filler run keeps a wrong count beside the right number (e.g.
"5 issues ... 3 open") failing.
The prior results table and narrative claimed gitea-axi posts the lowest
cost-equivalent tokens. That snapshot predated the neutral-working-dir
isolation fix, when the checkout-defaulting arms (gitea-axi, tea) drew
repo and login for free from the harness's own checkout — so gitea-axi
was implicitly pre-authenticated and looked like the winner.
On a clean run with every arm fairly credentialed and executed together,
raw REST is the cheapest on cost-equivalent tokens and leads every tier —
terse HTTP is the token floor no wrapper undercuts. gitea-axi is a clear
second overall and the lowest-cost structured interface, beating gitea-mcp
and tea on every tier at 100% success. Keep the raw-REST arm and state
this plainly rather than crown the wrapper by omitting the floor.
Numbers regenerated from bench:report over the 240-sample clean snapshot.
The gitea-axi arm was the only shell arm handed no credentials: the
runner set only PATH, so the agent had to reverse-engineer the tea-login
system — guessing a profile name and hunting for a config file — before
any real work, burning ~4 turns per task. Since turns drive cache-read,
the benchmark's dominant cost metric, this scaffolding gap alone inflated
gitea-axi's cost-equivalent tokens above every other arm.
Hand the arm its host and token through gitea-axi's own env interface
(GITEA_AXI_API_URL / GITEA_AXI_TOKEN), the symmetric counterpart to the
gitea-mcp server's GITEA_HOST / GITEA_ACCESS_TOKEN env: both name the
same two facts, and both still leave the agent to name the repository per
call. A shell arm now carries a credential env (empty for tea and
raw-api, which need none), merged under PATH in the driver.
Also strengthen SKILL.md so a cold agent targets and authenticates on the
first call: an explicit "Targeting and authentication" section replaces
the buried, optional-looking one-liner, spelling out that outside a
checkout `-R OWNER/NAME` plus the environment's token is all that is
needed — do not go hunting for a config file or login profile.
Verified live: create-memory-leak-issue dropped from 10 turns to 3 and
its cache-read fell ~3.8x, with the auth flailing gone from the transcript.
Records stored only a run's token/turn totals, so an arm's turn cost —
the dominant driver of cache-read tokens — could not be diagnosed from
the store. Retain the ordered transcript of tool invocations (the exact
shell commands, MCP calls, and built-in tools the run made) on every
scored record, absent only for a hung run that produced no transcript.
The canonical TranscriptEntry shape lives on the record (result.ts); the
isolation audit's ToolUse now aliases it so the persisted and audited
shapes cannot drift.
The read checker matched a task's required-fact phrasings as plain
substrings of the agent's report after only lowercasing and collapsing
whitespace. An answer that was substantively correct but wrapped a value in
markdown (e.g. `**5**`) failed the match, because the emphasis markers broke
the phrase adjacency (`**5** open` does not contain `5 open`) — a correct
answer scored incorrect on formatting alone.
Strip markdown emphasis/code markers (`*`, `_`, backtick) during
normalization so the match is on substance, not presentation. A guard test
confirms a wrong value still fails after stripping.
The SDK driver ran the agent with no explicit cwd, so its shell inherited
the harness's own checkout. When the agent omitted `-R OWNER/NAME`, the
gitea-axi (and tea) CLI defaulted the repository from that local checkout —
silently resolving the harness repo instead of the seeded throwaway — and
returned a plausible but wrong result (e.g. `count: 0 open of 0 total` for a
repo with no issues). This contaminated read-tier scoring for the checkout-
defaulting arms and was surfaced by the newly persisted read reports.
Give each run a fresh, empty working directory outside any checkout, so a
forgotten `-R` errors instead of hitting the wrong repository, and delete it
when the run ends.
Retain the agent's final report on the benchmark result record for read
tasks, so a failed read is diagnosable directly from the stored record
instead of only carrying an opaque `incorrect` tag. The runner resolves
the scoring spec once and records `run.finalReport` when the spec is a
read; mutation records omit the field entirely. The sample store needs no
change — it serializes whatever record it is handed.
This is the prerequisite for confirming the read-open-issue-count failure
from real report text before the state-aware count-line change (task 0033).
Reframe bench/README.md from maintainer-facing internals to a reader-facing
overview: what the benchmark measures, how it works at a high level, and what
it found. Add the completed 4-arm results (240 samples, 2026-07-17).
Remove the per-file layout, the vocabulary glossary, the packaging note, the
spec/ADR pointer, and the run/report/test command sections — that detail lives
in the code and the spec, and it buried the point. The result is a one-screen
doc: pitch and headline finding, a one-paragraph methodology, and the results
table sorted by cost-equivalent tokens.
sumTokens read the Agent SDK's per-model `modelUsage` entries with
snake_case field names, but the SDK reports those per-model entries in
camelCase (`inputTokens`, `cacheReadInputTokens`, ...). Every token
component therefore fell through to zero, silently zeroing the
cost-equivalent-token headline metric — while `total_cost_usd` and
`num_turns` (top-level snake_case) kept working and masked it.
Read `modelUsage` with the correct camelCase fields, keeping the
snake_case aggregate `usage` as the fallback. Export `sumTokens` and add
a regression test covering both the per-model camelCase sum (folding in
the auxiliary model) and the snake_case fallback, so a future SDK
field-casing drift fails a test instead of producing zero-token samples.
Add the maintainer-facing `bench:report` command, the offline counterpart
to `bench:run`: it opens the sample store, drains it, aggregates against the
scored suite and bonus definitions, and prints the aggregator's comparison to
stdout. It renders whatever has accumulated, annotating incomplete coverage
rather than blocking on a complete matrix.
`parseReportArgs` is the pure argument seam (`--store`, `--help`, plus a
`--self-review` / `--no-self-review` variant selector). Unlike `bench:run` the
boundary is offline — it reads only the local store, no host or Agent SDK — so
the whole `runReportCommand` is deterministic and unit-tested, not smoke-run.
Move `DEFAULT_STORE_ROOT` to `store.ts` as the single source of truth,
re-exported from `run.ts` for its existing importers.
Add bench/aggregate.ts: the pure aggregator seam that rolls the
accumulated sample store into a readable comparison. aggregate produces
a headline table (one row per arm, cost-equivalent tokens as the
headline plus raw tokens, turns, duration, success rate, coverage, and
imputed cost as a de-emphasized secondary column), per-tier and
per-token-component breakdowns, and the separate bonus table; renderReport
renders it as stable text. Cost-equivalent tokens are weighted at render
time from the four retained components per ADR 0014, and incomplete
coverage is annotated rather than hidden. Unit-tested against synthetic
append-only sample stores.
Add the maintainer-facing command that runs one chosen benchmark cell on
demand, so only the token budget available at that moment is spent.
runCells (bench/run-loop.ts) runs one (arm, task) cell for a batch of
trials — defaulting to five with a reporting floor of three — by driving
the existing single-cell runner and the append-only sample store rather
than reimplementing orchestration. Re-running a cell deepens it: trial
numbering continues past the highest trial the cell already holds and the
new samples append, so a cell's sample size grows across sittings without
overwriting prior runs.
bench/run.ts is the command: parseRunArgs is the pure, unit-tested
argument seam, and runBenchCommand is the live boundary that resolves host
access, resolves the scored suite against the host's self-review support,
selects the task, and drives the run loop. It is invoked via the new
bench:run npm script, run under tsx (a new devDependency) because the
harness's .js-specifier imports need a TypeScript-aware runner. The Claude
Agent SDK is now declared as an optional peerDependency — documented but
neither installed for package consumers nor pulled into CI.
Every arm runs on the driver's single fixed model; the command exposes no
per-cell model override that could break cross-arm comparability. The
default store root bench/results/ is gitignored.
Add the full 20-task scored suite and the capability-asymmetric bonus
definitions, plus the self-review capability probe that resolves the two
review tasks.
buildScoredSuite returns the shared-surface tasks weighted four read /
six single-mutation / six find-then-act / four multi-step, each a
natural-language intent parametrized against the seed and carrying a tier
and a scoring spec keyed on the single user. The two find-then-act review
tasks are approve/request-changes when the host permits self-review and
comment reviews otherwise; buildBonusTasks emits the approve/request-changes
operations as bonus entries in the fallback case, alongside the static
both-direction bonus definitions (gitea-axi's search/diff/checks/checkout/
issue-dependency edges, and the not-applicable repository/release/milestone
operations).
self-review.ts adds probeSelfReview and detectSelfReviewSupport, the live
boundary that determines self-review support once per sweep; it reuses the
now-exported non-throwing request helper from seed.ts.
Thread every benchmark layer to run one (arm, task, trial) cell end to
end: provision and seed a throwaway repository, run the agent under the
active arm bounded by a turn cap and a wall-clock backstop, audit the
transcript, capture and score the post-run state, append the sample, and
delete the repository.
- runner.ts: runCell orchestration behind the BenchHost and AgentDriver
seams, so the flow is unit-tested with fakes while the live wiring is
validated by a smoke run; turn-cap and wall-clock failures are tagged
confused-versus-hung, and a leaked transcript is flagged invalid.
- audit.ts: the post-run transcript audit plus the shared
foreignToolReason predicate both isolation enforcement points consume.
- task.ts: the runnable BenchTask wrapper and one sample single-mutation
task exercising the full path.
- snapshot.ts: captureRepoState, the seed's read-back counterpart, in the
RepoState shape the checker diffs against.
- host.ts / sdk-driver.ts: the live BenchHost and the Claude Agent SDK
driver (an optional peer, loaded via dynamic import) for real runs.
- runner.smoke.test.ts: the live tracer-bullet tier, skipping cleanly
when no host or SDK is configured.
Add bench/arm.ts, the per-arm scaffolding that produces the single arm
definition the runner consumes. Every arm shares an identical task-agnostic
base prompt and the same repository coordinates and token; each arm then
receives a minimal, symmetric bootstrap.
The deliberate asymmetries follow the shipped products: the gitea-axi arm
embeds the bundled Agent Skill (its body, charging its ambient cost to
gitea-axi); the tea and raw-api arms get a one-line native-discovery pointer;
the gitea-mcp arm runs with the shell disabled and only the MCP server
attached, its dispatcher schemas loading eagerly. Shell arms' PATH and guard
come from bench/guard.ts.
Add the deterministic, idempotent seed that brings a freshly provisioned
throwaway repository to a known ground truth before a trial runs, scripted
over the live Gitea API.
- bench/seed-plan.ts: the pure ground truth (fixed labels, an issue spread
across the discriminating dimensions, and labelled/reviewed/real-branch
pull requests) plus groundTruth(user), realizing it into a RepoState.
- bench/seed.ts: idempotent seeding reconciled by natural key, reusing
gitea-axi's own tea-login credential discovery (no new secret handling).
- A live smoke tier (test:bench:smoke) validating the seed end-to-end and
skipping cleanly when no host is configured, kept out of the deterministic
bench tier.
Export selectLogin from src/context.ts so the bench reuses the exact
credential-selection path.
Add the guard that confines each benchmark arm's agent to exactly one
tool, so a result measures the tool rather than the agent's choice
between tools.
`guardCommand` inspects every binary a proposed shell command would
reach — across pipelines, sequences, subshells, command and process
substitutions, redirections, and leading environment assignments — and
permits only the active arm's one allow-listed binary plus a curated set
of harmless read-only utilities. Foreign binaries, absolute-path
evasions (even of the arm's own binary), and interpreter-based fetch
tricks are denied; the gitea-mcp arm runs with the shell disabled
entirely. `provisionArmBin` produces a curated per-arm bin directory
exposing only that arm's binary as the convenience layer behind the
authoritative guard.
Tests are colocated in bench/guard.test.ts and run via `npm run
test:bench`.
Lay the foundation the benchmark harness reads and writes: a bench/
directory (excluded from the published npm package), the immutable
result-record shape, and an append-only per-cell sample store.
- bench/result.ts: the ResultRecord shape — four token components,
turns, duration, imputed cost, tagged pass/fail outcome, and the
arm/task/tier/trial/timestamp tags. Arm and Tier are typed unions.
- bench/store.ts: append-only sample store, one JSONL file per cell at
<root>/<arm>/<taskId>.jsonl; deepening a cell only ever adds samples.
- bench/README.md: harness working docs and benchmark vocabulary, kept
out of the tool's domain glossary per the spec.
- Dedicated bench test tier (vitest.bench.config.ts, npm run test:bench)
kept out of the fast tier; tsconfig typechecks bench.
- Packaging tier asserts bench/ never ships in the tarball.
