alexion/gitea-axi
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add benchmark harness spec, ADRs, and task breakdown
All checks were successful
CI / test (pull_request) Successful in 50s
Details
CI / test (push) Successful in 49s
Details

Browse Source 
Add the benchmark-harness spec, three supporting ADRs (cost-equivalent
token metric, single-user seed, guard-based tool isolation), and the
0022-0030 task breakdown that slices the harness into foundational
seams, an integrating single-cell runner, and the reporting layer.
This commit was merged in pull request #22.
This commit is contained in:
alexion
2026-07-15 10:04:28 -04:00
parent b5955cd7b8
commit d445b2bd2b
13 changed files with 377 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
.claude/tasks/0030-bench-aggregator-and-reporting.md Normal file
View File

@@ -0,0 +1,20 @@
---
spec: benchmark-harness
blocked-by: [0022-bench-scaffold-and-result-store, 0028-bench-task-suite]
---
## What to build
The aggregator that renders the accumulated sample store into a readable comparison, rendering whatever exists and annotating incomplete coverage rather than blocking on a complete matrix.
The headline table has one row per arm: cost-equivalent tokens as the headline, then raw tokens, turns, duration, success rate, and a coverage figure, with imputed cost shown as a de-emphasized secondary column. Cost-equivalent tokens are computed at render time by weighting each run's four retained components by Anthropic's published API pricing ratios (see the cost-equivalent-token-metric ADR), so the stored records can be re-weighted without re-running if the subscription's accounting is ever documented. Partially-run cells are annotated rather than hidden.
Supporting views derived from the same records include a per-tier breakdown, a per-token-component breakdown, and the separate bonus table for the capability-asymmetric operations. The aggregator is a pure seam, unit-tested against synthetic sample stores.
## Acceptance criteria
- [ ] The headline table renders one row per arm with cost-equivalent tokens as the headline, plus raw tokens, turns, duration, success rate, coverage, and imputed cost as a de-emphasized secondary column.
- [ ] Cost-equivalent tokens are computed from the retained four components at render time using the documented pricing-ratio weights.
- [ ] A partial matrix renders without error and incomplete coverage is annotated rather than hidden or treated as complete.
- [ ] Per-tier and per-token-component breakdowns and the separate bonus table are rendered from the same records.
- [ ] Rendering is stable and unit-tested against a synthetic append-only sample store.