From 408956cf328737e7b42546c57913f15c8ccb024a Mon Sep 17 00:00:00 2001 From: alexion Date: Sun, 19 Jul 2026 12:31:05 -0400 Subject: [PATCH] docs: rewrite bench results for the post-fix 4-arm snapshot Refresh the benchmark README from a clean co-temporal 4-arm run taken after the find-then-act skill rewrite, the /api/v1 tolerance, and the search next-step fix. gitea-axi drops from most-expensive arm to co-leader: within ~1% of raw REST overall, cheapest structured interface by ~23%, cheapest arm outright on the read and find-then-act tiers, at 100% success and the fewest turns of any arm. --- bench/README.md | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/bench/README.md b/bench/README.md index b9f1341..97b0d02 100644 --- a/bench/README.md +++ b/bench/README.md @@ -1,8 +1,8 @@ # Benchmark harness This directory holds the benchmark that tests gitea-axi's central claim — that it is an agent-ergonomic, low-token interface to Gitea — against the `tea` CLI, the official `gitea-mcp` server, and raw Gitea REST calls. -The result is honest rather than flattering: gitea-axi is the lowest-cost of the *structured* interfaces — it beats both `tea` and `gitea-mcp` on every tier at 100% task success — but hand-rolled raw REST is cheaper still, because terse HTTP is the token floor no wrapper undercuts. -Keeping the raw-REST arm in the comparison is deliberate: a benchmark of agent-CLIs that omits it will always crown the wrapper, and this one refuses to. +The result is that gitea-axi has reached the token floor: it is the cheapest of the structured interfaces by a wide margin and now runs neck-and-neck with hand-rolled raw REST — within ~1% overall — at 100% task success and the fewest turns of any arm. +Keeping the raw-REST arm in the comparison is deliberate: a benchmark of agent-CLIs that omits the hand-rolled baseline will always flatter the wrapper, and this one refuses to — which is exactly what makes gitea-axi matching that baseline meaningful. ## How it works @@ -15,23 +15,25 @@ Every arm is credentialed the way its product is really configured — the token | arm | cost-equivalent tokens | raw tokens | turns | success | imputed cost | | --- | ---: | ---: | ---: | ---: | ---: | -| raw-api | 16,971 | 52,586 | 4.3 | 100% | ~$0.11 | -| gitea-axi | 19,240 | 81,067 | 6.0 | 100% | ~$0.12 | -| tea | 20,568 | 82,188 | 6.2 | 97% | ~$0.12 | -| gitea-mcp | 21,803 | 79,961 | 5.7 | 100% | ~$0.14 | +| raw-api | 17,613 | 60,384 | 5.0 | 100% | ~$0.11 | +| gitea-axi | 17,815 | 62,705 | 4.8 | 100% | ~$0.11 | +| gitea-mcp | 23,198 | 76,378 | 5.2 | 100% | ~$0.15 | +| tea | 23,210 | 94,600 | 7.2 | 85% | ~$0.14 | -All four arms completed the full matrix — 20 of 20 tasks each, at the reporting floor — and success is near-perfect: only `tea` slips, to 89% on find-then-act, while the other three pass every run. +All four arms completed the full matrix — 20 of 20 tasks each — and success is near-perfect: only `tea` slips, to 85% overall (67% on find-then-act), while the other three pass every run. -Raw REST posts the lowest cost-equivalent tokens and leads every tier. -It is direct HTTP with the token in the request header, so it takes the fewest turns (4.3) and reads the least cached context, and no higher-level tool beats that on tokens alone. -This is the honest ceiling, and the reason gitea-axi does not claim the cost crown outright. +Raw REST posts the lowest cost-equivalent tokens, but only barely: gitea-axi lands within ~1% of it (17,815 vs 17,613), a gap well inside the noise of three trials. +Direct HTTP with the token in the request header is the token floor no wrapper is supposed to undercut — and a structured tool drawing level with it is the headline of this snapshot. -gitea-axi is a clear second overall and the cheapest of the structured tools: it undercuts the official `gitea-mcp` server and the `tea` CLI on every tier, at 100% success, with the lowest output-token count of any arm. -Note the split between raw and cost-equivalent tokens — gitea-axi spends more raw tokens than `gitea-mcp` yet costs less, because output is weighted 5× and gitea-axi's answers are compact. +gitea-axi is the cheapest structured interface by a wide margin — roughly 23% under both the official `gitea-mcp` server and the `tea` CLI — and it beats both of them on every tier, at 100% success. +It also takes the fewest turns of any arm (4.8, raw REST included) and the lowest output-token count of the shell arms; its compact TOON answers are what let a structured tool run this close to the floor. -By tier, raw REST's edge is widest on reads (10,921 vs gitea-axi's 14,415) — a read is one HTTP request for curl, where a CLI still spends a turn or two — and narrows on multi-step (24,348 vs 26,963), where the work itself dominates and interface overhead matters less. +By tier the picture is sharper than the overall total. +gitea-axi is the *cheapest arm outright* on the two discovery-heavy tiers — reads (13,294 vs raw's 14,656) and find-then-act (17,585 vs 18,631) — where finding the right entity is the work, and its compact search and list output beats reconstructing and parsing raw JSON. +Raw REST reclaims the lead on the mutation-heavy tiers — narrowly on single-mutation (15,212 vs 15,552), more clearly on multi-step (22,643 vs 26,079) — where the task is a handful of terse POSTs that no wrapper undercuts. +So the two trade tiers: gitea-axi wins where an interface earns its keep, raw REST wins where the request was already minimal. Cost parity on the scored suite also understates gitea-axi, because the suite is the subset every arm can do at all. The bonus table records capability-asymmetric operations — full-text issue search, rendering a PR's diff and checks, issue dependencies — that gitea-axi handles directly and raw REST has no first-class equivalent for. -_Snapshot: 2026-07-17 — 4 arms × 20 tasks × 3 trials each (240 samples), a single clean run with all four arms executed together against one live Gitea host; imputed cost is the mean per-task Anthropic-API-priced dollar cost._ +_Snapshot: 2026-07-19 — 4 arms × 20 tasks × 3 trials each (240 samples), a single clean run with all four arms executed together against one live Gitea host; imputed cost is the mean per-task Anthropic-API-priced dollar cost._