docs: benchmark harness spec, ADRs, and task breakdown #22

alexion · 2026-07-15T10:05:13-04:00

alexion commented

2026-07-15 10:05:13 -04:00

Summary

Adds the planning artifacts for the benchmark harness: the spec, three supporting ADRs, and the task breakdown that slices the work.

The harness will drive a Claude agent against a suite of realistic Gitea tasks under four tool conditions (gitea-axi, tea, gitea-mcp, raw REST) and record correctness and token consumption, rendering a headline table analogous to the gh-axi benchmark. Runs are incremental — one (arm, task, trial) cell at a time — so the maintainer spends only the token budget available at that moment.

What's here

Spec — .claude/spec/benchmark-harness.md
ADRs
- 0014 cost-equivalent tokens as the headline metric
- 0015 single-user seed and its constraints on the task surface
- 0016 guard-based tool isolation instead of containers
Task breakdown (0022–0030) — four foundational seams (store, guard, checker, seed), the integrating single-cell runner tracer bullet, then the full task suite, on-demand run loop, and aggregator/reporting.

Docs only — no code or behavior changes.

Slice map

#	Slice	Blocked by
0022	bench scaffold + result record & append-only store	—
0023	tool-isolation guard	—
0024	checker + scoring-spec contract	—
0025	seed provisioning (live host)	—
0026	arm scaffolding	0023
0027	single-cell runner (tracer bullet)	0022–0026
0028	full task suite (20 scored + bonus)	0024, 0025, 0027
0029	run loop / on-demand cell command	0027, 0028
0030	aggregator + reporting	0022, 0028

## Summary Adds the planning artifacts for the benchmark harness: the spec, three supporting ADRs, and the task breakdown that slices the work. The harness will drive a Claude agent against a suite of realistic Gitea tasks under four tool conditions (gitea-axi, tea, gitea-mcp, raw REST) and record correctness and token consumption, rendering a headline table analogous to the gh-axi benchmark. Runs are incremental — one `(arm, task, trial)` cell at a time — so the maintainer spends only the token budget available at that moment. ## What's here - **Spec** — `.claude/spec/benchmark-harness.md` - **ADRs** - `0014` cost-equivalent tokens as the headline metric - `0015` single-user seed and its constraints on the task surface - `0016` guard-based tool isolation instead of containers - **Task breakdown (0022–0030)** — four foundational seams (store, guard, checker, seed), the integrating single-cell runner tracer bullet, then the full task suite, on-demand run loop, and aggregator/reporting. Docs only — no code or behavior changes. ## Slice map | # | Slice | Blocked by | |---|-------|-----------| | 0022 | bench scaffold + result record & append-only store | — | | 0023 | tool-isolation guard | — | | 0024 | checker + scoring-spec contract | — | | 0025 | seed provisioning (live host) | — | | 0026 | arm scaffolding | 0023 | | 0027 | single-cell runner (tracer bullet) | 0022–0026 | | 0028 | full task suite (20 scored + bonus) | 0024, 0025, 0027 | | 0029 | run loop / on-demand cell command | 0027, 0028 | | 0030 | aggregator + reporting | 0022, 0028 |

alexion added 1 commit 2026-07-15 10:05:14 -04:00

docs: add benchmark harness spec, ADRs, and task breakdown

CI / test (pull_request) Successful in 50s

Details

CI / test (push) Successful in 49s

Details

d445b2bd2b

Add the benchmark-harness spec, three supporting ADRs (cost-equivalent
token metric, single-user seed, guard-based tool isolation), and the
0022-0030 task breakdown that slices the harness into foundational
seams, an integrating single-cell runner, and the reporting layer.

alexion merged commit d445b2bd2b into main

2026-07-15 10:07:34 -04:00

alexion deleted branch bench/harness-plan

2026-07-15 10:07:34 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: alexion/gitea-axi#22