docs: rewrite bench results for the clean 4-arm snapshot
Some checks failed
CI / test (pull_request) Failing after 54s

The prior results table and narrative claimed gitea-axi posts the lowest
cost-equivalent tokens. That snapshot predated the neutral-working-dir
isolation fix, when the checkout-defaulting arms (gitea-axi, tea) drew
repo and login for free from the harness's own checkout — so gitea-axi
was implicitly pre-authenticated and looked like the winner.

On a clean run with every arm fairly credentialed and executed together,
raw REST is the cheapest on cost-equivalent tokens and leads every tier —
terse HTTP is the token floor no wrapper undercuts. gitea-axi is a clear
second overall and the lowest-cost structured interface, beating gitea-mcp
and tea on every tier at 100% success. Keep the raw-REST arm and state
this plainly rather than crown the wrapper by omitting the floor.

Numbers regenerated from bench:report over the 240-sample clean snapshot.
This commit is contained in:
2026-07-17 22:35:40 -04:00
parent 6e65032944
commit 040daed39d

View File

@@ -1,28 +1,37 @@
# Benchmark harness
This directory holds the benchmark that tests gitea-axi's central claim — that it is an agent-ergonomic, low-token interface to Gitea — against the `tea` CLI, the official `gitea-mcp` server, and raw Gitea REST calls.
This run bears that out on cost: gitea-axi posts the lowest cost-equivalent tokens and the lowest imputed cost of the four tools, though `gitea-mcp` edges it slightly on accuracy.
The result is honest rather than flattering: gitea-axi is the lowest-cost of the *structured* interfaces — it beats both `tea` and `gitea-mcp` on every tier at 100% task success — but hand-rolled raw REST is cheaper still, because terse HTTP is the token floor no wrapper undercuts.
Keeping the raw-REST arm in the comparison is deliberate: a benchmark of agent-CLIs that omits it will always crown the wrapper, and this one refuses to.
## How it works
Each arm is an agent given exactly one of the four tools and nothing else, run on the same fixed model at temperature zero, so the comparison measures the tool rather than the model.
The suite is 20 tasks across four tiers — read, single-mutation, find-then-act, and multi-step — each run against a freshly seeded throwaway repository and scored deterministically by diffing the resulting repository state (or matching required facts in the agent's answer) against the seeded ground truth.
The headline metric is cost-equivalent tokens: the four token components (fresh input, cache write, cache read, output) weighted by Anthropic's published API pricing ratios, which is why an arm can spend more raw tokens yet cost less.
Every arm is credentialed the way its product is really configured — the token in its environment (`gitea-axi`, `gitea-mcp`) or in its prompt (`raw-api`), and a `tea` login for `tea` — so no arm pays a turn tax rediscovering how to authenticate.
## Results
| arm | cost-equivalent tokens | raw tokens | success | imputed cost |
| --- | ---: | ---: | ---: | ---: |
| gitea-axi | 16,921 | 68,093 | 95% | $6.20 |
| raw-api | 17,773 | 55,631 | 95% | $6.64 |
| gitea-mcp | 17,898 | 60,028 | 97% | $6.82 |
| tea | 20,505 | 80,702 | 90% | $7.25 |
| arm | cost-equivalent tokens | raw tokens | turns | success | imputed cost |
| --- | ---: | ---: | ---: | ---: | ---: |
| raw-api | 16,971 | 52,586 | 4.3 | 100% | ~$0.11 |
| gitea-axi | 19,240 | 81,067 | 6.0 | 100% | ~$0.12 |
| tea | 20,568 | 82,188 | 6.2 | 97% | ~$0.12 |
| gitea-mcp | 21,803 | 79,961 | 5.7 | 100% | ~$0.14 |
All four arms completed the full matrix — 20 of 20 tasks each, at the reporting floor.
gitea-axi wins on cost-equivalent tokens and on real imputed cost even though it does not use the fewest raw tokens: its interactions are output-light, and output is the most expensive component (weighted 5×), so its compact answers beat arms that emit more.
gitea-mcp is the most accurate at 97% against gitea-axi's 95%, so the two leaders trade a small accuracy edge for a clear cost lead.
All four arms completed the full matrix — 20 of 20 tasks each, at the reporting floor — and success is near-perfect: only `tea` slips, to 89% on find-then-act, while the other three pass every run.
By tier, the read tasks are the hardest for every arm (7583% success) — exact-answer reads, not mutations, are where correctness slips.
tea is the outlier on find-then-act, dropping to 78% success at about 1.7× the cost-equivalent tokens of the other three arms.
Raw REST posts the lowest cost-equivalent tokens and leads every tier.
It is direct HTTP with the token in the request header, so it takes the fewest turns (4.3) and reads the least cached context, and no higher-level tool beats that on tokens alone.
This is the honest ceiling, and the reason gitea-axi does not claim the cost crown outright.
_Snapshot: 2026-07-17 — 4 arms × 20 tasks × 3 trials each (240 samples), a single run against one live Gitea host; imputed cost is Anthropic API-priced._
gitea-axi is a clear second overall and the cheapest of the structured tools: it undercuts the official `gitea-mcp` server and the `tea` CLI on every tier, at 100% success, with the lowest output-token count of any arm.
Note the split between raw and cost-equivalent tokens — gitea-axi spends more raw tokens than `gitea-mcp` yet costs less, because output is weighted 5× and gitea-axi's answers are compact.
By tier, raw REST's edge is widest on reads (10,921 vs gitea-axi's 14,415) — a read is one HTTP request for curl, where a CLI still spends a turn or two — and narrows on multi-step (24,348 vs 26,963), where the work itself dominates and interface overhead matters less.
Cost parity on the scored suite also understates gitea-axi, because the suite is the subset every arm can do at all.
The bonus table records capability-asymmetric operations — full-text issue search, rendering a PR's diff and checks, issue dependencies — that gitea-axi handles directly and raw REST has no first-class equivalent for.
_Snapshot: 2026-07-17 — 4 arms × 20 tasks × 3 trials each (240 samples), a single clean run with all four arms executed together against one live Gitea host; imputed cost is the mean per-task Anthropic-API-priced dollar cost._