Refresh the benchmark README from a clean co-temporal 4-arm run taken after
the find-then-act skill rewrite, the /api/v1 tolerance, and the search
next-step fix. gitea-axi drops from most-expensive arm to co-leader: within
~1% of raw REST overall, cheapest structured interface by ~23%, cheapest
arm outright on the read and find-then-act tiers, at 100% success and the
fewest turns of any arm.
The next-step suggestion on `search issues`/`search prs` is now conditioned
on the in-repo match count. On a miss it pointed the agent at `view <number>`,
which is nonsensical when nothing matched; it now suggests the non-indexed
`issue list --state all` / `pr list --state all` fallback, which recovers from
both an over-narrow query and issue-indexer lag without naming the cause. On
exactly one match it fills the real number (`issue view 2`), applying AXI
Principle 9's single-id fill. Two or more matches keep the parameterized
placeholder.
Search stays a locator — it never auto-loads the detail even on a single
match; ADR 0017 records that decision (a deliberate narrowing of Principle 4)
and the CONTEXT.md search term is updated to match.
bench:run gains a --skill <path> flag, threaded into the existing
BuildArmOptions.skillPath, so a skill variant can be A/B'd against the
shipped SKILL.md with the same binary and harness — the mechanism used to
validate the find-then-act rewrite — without mutating the shipped file.
The bundled skill's Discovery section told the agent to run the bare
dashboard and reach for --help proactively, and advertised overlapping
find-paths — inducing exploratory commands that made the gitea-axi arm the
most expensive of the benchmark's four. Replace it with a "find the target,
then act" section, name the non-obvious mutation flags so common edits do
not need --help, and drop the setup line and the over-tea/raw/git bullets
that only duplicated the description. A same-time A/B cut cost-equivalent
tokens ~10% and collapsed bare-dashboard use from 60% to 7% with no loss of
success.
The client (gitea-js) appends /api/v1 to the base URL itself, so a
GITEA_AXI_API_URL that already carries it — a natural guess given the
variable's name — doubled the segment and failed as a spurious
REPO_NOT_FOUND. Normalize the base URL by stripping a trailing /api/v1
(and any trailing slashes) on both the env-URL and tea-login paths, so the
host base and the /api/v1 endpoint both resolve.
The tea login store here holds `alexion` and `csv-reviewer`, not `axi`;
`selectLogin` matches `--login` by exact name, so `--login alexion` works and an
unknown `--login axi` fails with VALIDATION_ERROR — the prior note had this
backwards. Also record that bench/ unit tests run only under
vitest.bench.config.ts (plain `vitest run` matches test/** and finds none).
Two benchmark-harness races were being scored as agent failures. A freshly
seeded throwaway repo could 404 for an independent reader (the agent, a fresh
process) before Gitea made it consistent, and the async issue indexer lagged so
`search issues`/`search prs` returned nothing right after seeding — leaving the
agent unable to find the target it was asked to act on.
seedRepo now blocks until an independent read confirms the repo is reachable,
its full seeded issue and pull spread is visible, and a seeded issue and pull
are returned by the search index, before releasing the agent. It polls up to
60s and throws a loud harness error on timeout rather than letting a
propagation delay become a scored agent failure.
Also add an optional `pattern` regex to RequiredFact so a read answer's count is
recognised semantically rather than as a fixed phrase: "5 issues are currently
open" no longer fails against the literal "5 issues are open", while the
alphabetic-only filler run keeps a wrong count beside the right number (e.g.
"5 issues ... 3 open") failing.
