docs: rewrite bench README as reader-first with results #35

alexion · 2026-07-17T08:47:31-04:00

alexion commented

2026-07-17 08:47:31 -04:00

Rewrites bench/README.md from maintainer-facing internal documentation into a reader-facing overview, and adds the completed benchmark results.

Why

The old README led with a per-file ## Layout of every module and a vocabulary glossary, then run/report/test instructions — optimized for someone modifying the harness, not for a reader asking "what is this and what did it find." It also had no results.

What changed

The README is now a one-screen doc:

Pitch + headline finding — what the benchmark measures, and the punchline (gitea-axi posts the lowest cost-equivalent tokens and imputed cost; gitea-mcp edges it on accuracy).
How it works — one paragraph: arms are one-tool agents on a fixed model, 20 tasks across four tiers on seeded throwaway repos scored deterministically, and the cost-equivalent-token metric (why raw ≠ cost).
Results — the completed 4-arm matrix (240 samples, 2026-07-17), a table sorted by cost-equivalent tokens, the raw-vs-cost-eq finding, and per-tier notes.

Removed: the per-file layout, the vocabulary glossary, the packaging note, the spec/ADR pointer, and the ## Running a cell / ## Reading the results / ## Tests sections. That detail lives in the code and the spec; here it buried the point.

Docs-only change — no code touched.

Rewrites `bench/README.md` from maintainer-facing internal documentation into a reader-facing overview, and adds the completed benchmark results. ## Why The old README led with a per-file `## Layout` of every module and a vocabulary glossary, then run/report/test instructions — optimized for someone modifying the harness, not for a reader asking "what is this and what did it find." It also had no results. ## What changed The README is now a one-screen doc: - **Pitch + headline finding** — what the benchmark measures, and the punchline (gitea-axi posts the lowest cost-equivalent tokens and imputed cost; `gitea-mcp` edges it on accuracy). - **How it works** — one paragraph: arms are one-tool agents on a fixed model, 20 tasks across four tiers on seeded throwaway repos scored deterministically, and the cost-equivalent-token metric (why raw ≠ cost). - **Results** — the completed 4-arm matrix (240 samples, 2026-07-17), a table sorted by cost-equivalent tokens, the raw-vs-cost-eq finding, and per-tier notes. Removed: the per-file layout, the vocabulary glossary, the packaging note, the spec/ADR pointer, and the `## Running a cell` / `## Reading the results` / `## Tests` sections. That detail lives in the code and the spec; here it buried the point. Docs-only change — no code touched.

alexion added 1 commit 2026-07-17 08:47:31 -04:00

docs: rewrite bench README as reader-first with results

CI / test (pull_request) Successful in 53s

Details

CI / test (push) Successful in 51s

Details

0fdc2bed9e

Reframe bench/README.md from maintainer-facing internals to a reader-facing
overview: what the benchmark measures, how it works at a high level, and what
it found. Add the completed 4-arm results (240 samples, 2026-07-17).

Remove the per-file layout, the vocabulary glossary, the packaging note, the
spec/ADR pointer, and the run/report/test command sections — that detail lives
in the code and the spec, and it buried the point. The result is a one-screen
doc: pitch and headline finding, a one-paragraph methodology, and the results
table sorted by cost-equivalent tokens.

alexion merged commit 0fdc2bed9e into main

2026-07-17 08:48:21 -04:00

alexion deleted branch docs-bench-readme

2026-07-17 08:48:21 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: alexion/gitea-axi#35