alexion/gitea-axi
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: benchmark credential parity, transcript, honest results + read-tier accuracy (tasks 0032, 0033) #36

Merged
alexion merged 10 commits from task-0032-bench-read-report-persistence into main 2026-07-18 10:28:33 -04:00
Conversation 0 Commits 10 Files Changed 23 +22 -13
1 changed files with 22 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit 040daed39d - Show all commits

35
bench/README.md
View File

@@ -1,28 +1,37 @@
# Benchmark harness # Benchmark harness
This directory holds the benchmark that tests gitea-axi's central claim — that it is an agent-ergonomic, low-token interface to Gitea — against the `tea` CLI, the official `gitea-mcp` server, and raw Gitea REST calls. This directory holds the benchmark that tests gitea-axi's central claim — that it is an agent-ergonomic, low-token interface to Gitea — against the `tea` CLI, the official `gitea-mcp` server, and raw Gitea REST calls.
This run bears that out on cost: gitea-axi posts the lowest cost-equivalent tokens and the lowest imputed cost of the four tools, though `gitea-mcp` edges it slightly on accuracy. The result is honest rather than flattering: gitea-axi is the lowest-cost of the *structured* interfaces — it beats both `tea` and `gitea-mcp` on every tier at 100% task success — but hand-rolled raw REST is cheaper still, because terse HTTP is the token floor no wrapper undercuts.
Keeping the raw-REST arm in the comparison is deliberate: a benchmark of agent-CLIs that omits it will always crown the wrapper, and this one refuses to.
## How it works ## How it works
Each arm is an agent given exactly one of the four tools and nothing else, run on the same fixed model at temperature zero, so the comparison measures the tool rather than the model. Each arm is an agent given exactly one of the four tools and nothing else, run on the same fixed model at temperature zero, so the comparison measures the tool rather than the model.
The suite is 20 tasks across four tiers — read, single-mutation, find-then-act, and multi-step — each run against a freshly seeded throwaway repository and scored deterministically by diffing the resulting repository state (or matching required facts in the agent's answer) against the seeded ground truth. The suite is 20 tasks across four tiers — read, single-mutation, find-then-act, and multi-step — each run against a freshly seeded throwaway repository and scored deterministically by diffing the resulting repository state (or matching required facts in the agent's answer) against the seeded ground truth.
The headline metric is cost-equivalent tokens: the four token components (fresh input, cache write, cache read, output) weighted by Anthropic's published API pricing ratios, which is why an arm can spend more raw tokens yet cost less. The headline metric is cost-equivalent tokens: the four token components (fresh input, cache write, cache read, output) weighted by Anthropic's published API pricing ratios, which is why an arm can spend more raw tokens yet cost less.
Every arm is credentialed the way its product is really configured — the token in its environment (`gitea-axi`, `gitea-mcp`) or in its prompt (`raw-api`), and a `tea` login for `tea` — so no arm pays a turn tax rediscovering how to authenticate.
## Results ## Results
| arm | cost-equivalent tokens | raw tokens | success | imputed cost | | arm | cost-equivalent tokens | raw tokens | turns | success | imputed cost |
| --- | ---: | ---: | ---: | ---: | | --- | ---: | ---: | ---: | ---: | ---: |
| gitea-axi | 16,921 | 68,093 | 95% | $6.20 | | raw-api | 16,971 | 52,586 | 4.3 | 100% | ~$0.11 |
| raw-api | 17,773 | 55,631 | 95% | $6.64 | | gitea-axi | 19,240 | 81,067 | 6.0 | 100% | ~$0.12 |
| gitea-mcp | 17,898 | 60,028 | 97% | $6.82 | | tea | 20,568 | 82,188 | 6.2 | 97% | ~$0.12 |
| tea | 20,505 | 80,702 | 90% | $7.25 | | gitea-mcp | 21,803 | 79,961 | 5.7 | 100% | ~$0.14 |
All four arms completed the full matrix — 20 of 20 tasks each, at the reporting floor. All four arms completed the full matrix — 20 of 20 tasks each, at the reporting floor — and success is near-perfect: only `tea` slips, to 89% on find-then-act, while the other three pass every run.
gitea-axi wins on cost-equivalent tokens and on real imputed cost even though it does not use the fewest raw tokens: its interactions are output-light, and output is the most expensive component (weighted 5×), so its compact answers beat arms that emit more.
gitea-mcp is the most accurate at 97% against gitea-axi's 95%, so the two leaders trade a small accuracy edge for a clear cost lead.
By tier, the read tasks are the hardest for every arm (75–83% success) — exact-answer reads, not mutations, are where correctness slips. Raw REST posts the lowest cost-equivalent tokens and leads every tier.
tea is the outlier on find-then-act, dropping to 78% success at about 1.7× the cost-equivalent tokens of the other three arms. It is direct HTTP with the token in the request header, so it takes the fewest turns (4.3) and reads the least cached context, and no higher-level tool beats that on tokens alone.
This is the honest ceiling, and the reason gitea-axi does not claim the cost crown outright.
_Snapshot: 2026-07-17 — 4 arms × 20 tasks × 3 trials each (240 samples), a single run against one live Gitea host; imputed cost is Anthropic API-priced._ gitea-axi is a clear second overall and the cheapest of the structured tools: it undercuts the official `gitea-mcp` server and the `tea` CLI on every tier, at 100% success, with the lowest output-token count of any arm.
Note the split between raw and cost-equivalent tokens — gitea-axi spends more raw tokens than `gitea-mcp` yet costs less, because output is weighted 5× and gitea-axi's answers are compact.
By tier, raw REST's edge is widest on reads (10,921 vs gitea-axi's 14,415) — a read is one HTTP request for curl, where a CLI still spends a turn or two — and narrows on multi-step (24,348 vs 26,963), where the work itself dominates and interface overhead matters less.
Cost parity on the scored suite also understates gitea-axi, because the suite is the subset every arm can do at all.
The bonus table records capability-asymmetric operations — full-text issue search, rendering a PR's diff and checks, issue dependencies — that gitea-axi handles directly and raw REST has no first-class equivalent for.
_Snapshot: 2026-07-17 — 4 arms × 20 tasks × 3 trials each (240 samples), a single clean run with all four arms executed together against one live Gitea host; imputed cost is the mean per-task Anthropic-API-priced dollar cost._