Harden the benchmark and cut gitea-axi's agent cost #44

alexion · 2026-07-19T12:36:42-04:00

alexion commented

2026-07-19 12:36:42 -04:00

Summary

A session of benchmark-harness and agent-ergonomics work on top of main (the task 0034/0035 pr-review commits this was cut over are already merged, so the diff is only the changes below). It hardens the benchmark, fixes two agent-facing CLI issues, and rewrites the bundled Agent Skill — moving gitea-axi from the most-expensive benchmark arm to a co-leader at the raw-REST token floor.

Headline result

A clean co-temporal 4-arm run (post-change) puts gitea-axi at −34% cost-equivalent tokens (27,002 → 17,815), turns 8.0 → 4.8, and success 86% → 100% — from most-expensive arm to within ~1% of hand-rolled raw REST, and cheapest structured interface by ~23%. The other three arms moved ±2–6%, so the gain is attributable to the changes rather than environment drift.

What's in it

Agent Skill

refactor(skill): rewrite for predictability — replace the open-ended "Discovery" section (bare dashboard, proactive --help) with a "find the target, then act" steer plus named mutation flags. A same-time A/B cut cost ~10% and collapsed bare-dashboard use 60% → 7%.

CLI

fix(context): tolerate a trailing /api/v1 on GITEA_AXI_API_URL. Agents habitually append it (the standard REST convention), which double-appended and 404'd as a spurious REPO_NOT_FOUND — present in 8/8 failures of an earlier run. Now harmless.
feat(search): on a 0-result search, point at the non-indexed list fallback (recovers from an over-narrow query or index lag) instead of a nonsensical view <number> hint; on a single match, fill the real number. Search stays a locator — no auto-collapse (ADR 0017).

Benchmark harness

fix(bench): gate seeding on repo + issue-index readiness (closes a provisioning/index-lag race), and match read-answer counts semantically rather than by brittle substring.
feat(bench): --skill <path> to A/B a skill variant against the shipped one with the same binary and harness.

Docs

docs(bench): rewrite the benchmark README from the post-fix snapshot.
docs: correct the bench/CLI login-name gotcha (alexion, not axi) and record the bench test-config gotcha.

Testing

Full unit suite green (410 tests), with new coverage for the /api/v1 normalization and the count-conditional search suggestion. Benchmark validated by live runs against git.alexion.dev.

## Summary A session of benchmark-harness and agent-ergonomics work on top of `main` (the task 0034/0035 pr-review commits this was cut over are already merged, so the diff is only the changes below). It hardens the benchmark, fixes two agent-facing CLI issues, and rewrites the bundled Agent Skill — moving gitea-axi from the most-expensive benchmark arm to a co-leader at the raw-REST token floor. ## Headline result A clean co-temporal 4-arm run (post-change) puts gitea-axi at **−34% cost-equivalent tokens** (27,002 → 17,815), **turns 8.0 → 4.8**, and **success 86% → 100%** — from most-expensive arm to within ~1% of hand-rolled raw REST, and cheapest structured interface by ~23%. The other three arms moved ±2–6%, so the gain is attributable to the changes rather than environment drift. ## What's in it **Agent Skill** - `refactor(skill)`: rewrite for predictability — replace the open-ended "Discovery" section (bare dashboard, proactive `--help`) with a "find the target, then act" steer plus named mutation flags. A same-time A/B cut cost ~10% and collapsed bare-dashboard use 60% → 7%. **CLI** - `fix(context)`: tolerate a trailing `/api/v1` on `GITEA_AXI_API_URL`. Agents habitually append it (the standard REST convention), which double-appended and 404'd as a spurious `REPO_NOT_FOUND` — present in 8/8 failures of an earlier run. Now harmless. - `feat(search)`: on a 0-result search, point at the non-indexed `list` fallback (recovers from an over-narrow query or index lag) instead of a nonsensical `view <number>` hint; on a single match, fill the real number. Search stays a locator — no auto-collapse (ADR 0017). **Benchmark harness** - `fix(bench)`: gate seeding on repo + issue-index readiness (closes a provisioning/index-lag race), and match read-answer counts semantically rather than by brittle substring. - `feat(bench)`: `--skill <path>` to A/B a skill variant against the shipped one with the same binary and harness. **Docs** - `docs(bench)`: rewrite the benchmark README from the post-fix snapshot. - `docs`: correct the bench/CLI login-name gotcha (`alexion`, not `axi`) and record the bench test-config gotcha. ## Testing Full unit suite green (410 tests), with new coverage for the `/api/v1` normalization and the count-conditional search suggestion. Benchmark validated by live runs against `git.alexion.dev`.

alexion added 7 commits 2026-07-19 12:36:42 -04:00

fix(bench): gate seeding on repo+index readiness and match read counts semantically b0a0ffc1ab

Two benchmark-harness races were being scored as agent failures. A freshly
seeded throwaway repo could 404 for an independent reader (the agent, a fresh
process) before Gitea made it consistent, and the async issue indexer lagged so
`search issues`/`search prs` returned nothing right after seeding — leaving the
agent unable to find the target it was asked to act on.

seedRepo now blocks until an independent read confirms the repo is reachable,
its full seeded issue and pull spread is visible, and a seeded issue and pull
are returned by the search index, before releasing the agent. It polls up to
60s and throws a loud harness error on timeout rather than letting a
propagation delay become a scored agent failure.

Also add an optional `pattern` regex to RequiredFact so a read answer's count is
recognised semantically rather than as a fixed phrase: "5 issues are currently
open" no longer fails against the literal "5 issues are open", while the
alphabetic-only filler run keeps a wrong count beside the right number (e.g.
"5 issues ... 3 open") failing.

docs: correct bench/CLI login name and record bench test-config gotcha f9125e099d

The tea login store here holds `alexion` and `csv-reviewer`, not `axi`;
`selectLogin` matches `--login` by exact name, so `--login alexion` works and an
unknown `--login axi` fails with VALIDATION_ERROR — the prior note had this
backwards. Also record that bench/ unit tests run only under
vitest.bench.config.ts (plain `vitest run` matches test/** and finds none).

fix(context): tolerate a trailing /api/v1 in the base URL dd58cd9dad

The client (gitea-js) appends /api/v1 to the base URL itself, so a
GITEA_AXI_API_URL that already carries it — a natural guess given the
variable's name — doubled the segment and failed as a spurious
REPO_NOT_FOUND. Normalize the base URL by stripping a trailing /api/v1
(and any trailing slashes) on both the env-URL and tea-login paths, so the
host base and the /api/v1 endpoint both resolve.

refactor(skill): steer find-then-act instead of open-ended discovery d56b1d0707

The bundled skill's Discovery section told the agent to run the bare
dashboard and reach for --help proactively, and advertised overlapping
find-paths — inducing exploratory commands that made the gitea-axi arm the
most expensive of the benchmark's four. Replace it with a "find the target,
then act" section, name the non-obvious mutation flags so common edits do
not need --help, and drop the setup line and the over-tea/raw/git bullets
that only duplicated the description. A same-time A/B cut cost-equivalent
tokens ~10% and collapsed bare-dashboard use from 60% to 7% with no loss of
success.

feat(bench): add --skill to override the gitea-axi arm's bundled skill 0b0186674e

bench:run gains a --skill <path> flag, threaded into the existing
BuildArmOptions.skillPath, so a skill variant can be A/B'd against the
shipped SKILL.md with the same binary and harness — the mechanism used to
validate the find-then-act rewrite — without mutating the shipped file.

feat(search): guide a 0-result search and fill the single-match number 5422f49f38

The next-step suggestion on `search issues`/`search prs` is now conditioned
on the in-repo match count. On a miss it pointed the agent at `view <number>`,
which is nonsensical when nothing matched; it now suggests the non-indexed
`issue list --state all` / `pr list --state all` fallback, which recovers from
both an over-narrow query and issue-indexer lag without naming the cause. On
exactly one match it fills the real number (`issue view 2`), applying AXI
Principle 9's single-id fill. Two or more matches keep the parameterized
placeholder.

Search stays a locator — it never auto-loads the detail even on a single
match; ADR 0017 records that decision (a deliberate narrowing of Principle 4)
and the CONTEXT.md search term is updated to match.

docs: rewrite bench results for the post-fix 4-arm snapshot

CI / test (pull_request) Successful in 55s

Details

CI / test (push) Successful in 52s

Details

408956cf32

Refresh the benchmark README from a clean co-temporal 4-arm run taken after
the find-then-act skill rewrite, the /api/v1 tolerance, and the search
next-step fix. gitea-axi drops from most-expensive arm to co-leader: within
~1% of raw REST overall, cheapest structured interface by ~23%, cheapest
arm outright on the read and find-then-act tiers, at 100% success and the
fewest turns of any arm.

alexion merged commit 408956cf32 into main

2026-07-19 12:40:57 -04:00

alexion deleted branch bench-skill-and-cli-cost-fixes

2026-07-19 12:40:57 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: alexion/gitea-axi#44