docs: add benchmark harness spec, ADRs, and task breakdown

Add the benchmark-harness spec, three supporting ADRs (cost-equivalent
token metric, single-user seed, guard-based tool isolation), and the
0022-0030 task breakdown that slices the harness into foundational
seams, an integrating single-cell runner, and the reporting layer.

.claude/tasks/0022-bench-scaffold-and-result-store.md

@@ -0,0 +1,18 @@
+---
+spec: benchmark-harness
+---
+
+## What to build
+
+The foundation the whole benchmark harness reads and writes: a `bench/` directory (excluded from the published npm package, alongside the existing `dist`/`skills` allow-list), the immutable result-record shape, and an append-only per-cell sample store.
+
+A result record captures one completed `(arm, task, trial)` run: the four token components (fresh input, cache-creation, cache-read, output), the turn count, the wall-clock duration, the imputed cost, and the pass/fail outcome with a failure tag distinguishing a confused agent from a hung one. It also carries the tags later views group by — arm, task id, tier, trial, and a timestamp.
+
+The store appends records as immutable, timestamped samples to a per-cell location. Deepening a cell's sample size adds samples rather than overwriting any prior run, and reading a cell returns every accumulated sample.
+
+## Acceptance criteria
+
+- [ ] A `bench/` directory exists and is excluded from the npm package (verified by the packaging tier or an equivalent `files` check).
+- [ ] The result-record shape records the four token components, turns, duration, imputed cost, outcome, failure tag, and the arm/task/tier/trial/timestamp tags.
+- [ ] Appending a sample to a cell that already has samples leaves the prior samples intact; reading the cell returns all of them.
+- [ ] A round-trip test writes several samples across cells and reads back exactly what was written.

.claude/tasks/0023-bench-tool-isolation-guard.md

@@ -0,0 +1,17 @@
+---
+spec: benchmark-harness
+---
+
+## What to build
+
+The guard that keeps each arm's agent confined to exactly one tool, so a benchmark result measures the tool rather than the agent's choice between tools (see the guard-based-tool-isolation ADR).
+
+The guard is a callback that inspects every proposed shell command and permits only the one binary allow-listed for the active arm plus a curated set of harmless utilities, denying everything else. It rejects foreign binaries, absolute-path evasions that sidestep the allow-list, and interpreter-based fetch tricks (reaching the API through a language runtime's HTTP client). A curated per-arm PATH backs the guard as a convenience layer, but the guard — not the PATH — is authoritative. A blocked attempt is surfaced, not silently retried.
+
+## Acceptance criteria
+
+- [ ] Each arm's allow-listed binary passes the guard; a foreign binary is denied.
+- [ ] An absolute-path invocation of a foreign binary is denied.
+- [ ] An interpreter-based fetch attempt (e.g. driving an HTTP request through a language runtime) is denied.
+- [ ] A curated per-arm PATH is produced exposing only that arm's allowed binary.
+- [ ] Unit tests cover the allowed-binary, foreign-binary, absolute-path, and interpreter-fetch cases per arm.

.claude/tasks/0024-bench-checker-and-scoring-spec.md

@@ -0,0 +1,19 @@
+---
+spec: benchmark-harness
+---
+
+## What to build
+
+The pure scoring seam that turns a completed run into a deterministic pass/fail, plus the scoring-spec contract each task pairs with.
+
+For a mutation task, the checker diffs the entire post-run repository state against the expected end state, so both the intended change and any collateral damage are caught. Comparison runs after normalization: volatile identifiers and timestamps are dropped, comments are matched by author and body, and labels are compared as sets. For a read task, the checker matches required facts in the agent's final report against the seeded ground truth, with no LLM judge.
+
+This slice also defines the scoring-spec contract — a task's expected end state (for mutations) or its required answer facts (for reads) — that the checker consumes and that the runner and task suite will produce. The checker is fed synthetic state snapshots and expected states; capturing live state from a real repository is the runner's job.
+
+## Acceptance criteria
+
+- [ ] Given synthetic actual and expected state snapshots, the full-state diff passes when they match after normalization and fails when the actual state is missing the intended change.
+- [ ] The diff fails when the actual state carries collateral change beyond the intended mutation.
+- [ ] Normalization drops volatile identifiers and timestamps, matches comments by author and body, and compares label sets order-independently.
+- [ ] The read-task answer-match passes when the required facts are present in the final report and fails when a required fact is missing.
+- [ ] The scoring-spec contract expresses both a mutation's expected end state and a read's required answer facts.

.claude/tasks/0025-bench-seed-provisioning.md

@@ -0,0 +1,16 @@
+---
+spec: benchmark-harness
+---
+
+## What to build
+
+The deterministic, idempotent seed that brings a freshly provisioned throwaway repository to a known ground truth before a trial runs, scripted entirely over the Gitea API against the live host (see the single-user-seed ADR). Authentication reuses gitea-axi's existing credential discovery path rather than introducing new secret handling.
+
+The seed establishes a fixed set of labels with fixed colors; a spread of open and closed issues varying by label, assignee presence (assigned-to-self versus unassigned), title keyword, and pre-existing comments; and a handful of pull requests including one labeled, one carrying an existing review, and one backed by a real pushed feature branch. All content is authored by the single available user, so the discriminating dimensions are label, state, assignee presence, and title keyword — not author.
+
+## Acceptance criteria
+
+- [ ] Provisioning a fresh repository and seeding it produces the fixed labels, the open/closed issue spread across the discriminating dimensions, and the pull requests (labeled, reviewed, and real-branch-backed).
+- [ ] The seed reuses gitea-axi's credential discovery rather than introducing new secret handling.
+- [ ] Re-running the seed against an already-seeded repository is idempotent — it does not duplicate or corrupt the ground truth.
+- [ ] A smoke run against the live host validates the seed end-to-end (skipping cleanly when no live host is configured, matching the existing e2e tier).

.claude/tasks/0026-bench-arm-scaffolding.md

@@ -0,0 +1,18 @@
+---
+spec: benchmark-harness
+blocked-by: 0023-bench-tool-isolation-guard
+---
+
+## What to build
+
+The per-arm scaffolding that charges each tool's real ambient-context cost honestly. All arms share one task-agnostic base prompt and the same repository coordinates and token; each arm then receives a minimal, symmetric bootstrap naming its tool and pointing at that tool's own native discovery affordance.
+
+The deliberate asymmetries follow the shipped products: the gitea-axi arm loads the bundled Agent Skill, because the Skill is part of what ships and its token cost belongs to gitea-axi. The tea and raw-API arms receive a one-line pointer to their native discovery affordance. The gitea-mcp arm's dispatcher schemas load eagerly as its ambient cost, and that arm disables the shell tool entirely, attaching only the MCP tools. Each arm's assembled prompt plus tool/PATH configuration (from the guard) is produced as a single arm definition the runner consumes.
+
+## Acceptance criteria
+
+- [ ] All arms share the identical base prompt and are handed the same repository coordinates and token.
+- [ ] The gitea-axi arm's assembled context carries the bundled Agent Skill.
+- [ ] The tea and raw-API arms each receive only a one-line native-discovery pointer.
+- [ ] The gitea-mcp arm loads its dispatcher schemas eagerly, has the shell tool disabled, and is attached only the MCP tools.
+- [ ] Each non-MCP arm's tool/PATH configuration comes from the guard and exposes only that arm's allowed binary.

.claude/tasks/0027-bench-single-cell-runner.md

@@ -0,0 +1,21 @@
+---
+spec: benchmark-harness
+blocked-by: [0022-bench-scaffold-and-result-store, 0023-bench-tool-isolation-guard, 0024-bench-checker-and-scoring-spec, 0025-bench-seed-provisioning, 0026-bench-arm-scaffolding]
+---
+
+## What to build
+
+The tracer bullet that threads every layer: run a single `(arm, task, trial)` cell end-to-end and record an immutable result. This is the walking skeleton — one arm against one sample task, one trial — that proves seed, arm scaffolding, guard, runner, checker, and store all connect.
+
+The runner provisions and seeds a fresh throwaway repository, runs the agent via the Claude Agent SDK on a single fixed model at temperature zero with exactly the active arm's tool and guard enforced, and bounds the run by a turn cap and a wall-clock backstop — exceeding either records a failure tagged to distinguish a confused agent from a hung one. It captures the four token components (including the auxiliary small model the runtime invokes, since that is real consumption), the turn count, the duration, and the imputed cost. After the run it captures the entire post-run repository state as a snapshot, scores it with the checker against the task's scoring spec, appends the result sample to the store, and deletes the throwaway repository. A post-run transcript audit asserts no foreign tool was reached; a detected leak flags the trial invalid rather than letting it be scored.
+
+This slice also defines the runnable Task wrapper (natural-language intent, parameters, tier, and scoring spec) and includes one sample task to exercise the path; the full suite is authored in a later slice.
+
+## Acceptance criteria
+
+- [ ] Running one cell provisions and seeds a fresh repository, runs the agent under its arm's tool with the guard active, and deletes the repository afterward.
+- [ ] The run is bounded by both a turn cap and a wall-clock backstop; exceeding either records a failure tagged confused-versus-hung.
+- [ ] The recorded sample carries the four token components (including the auxiliary small model), turns, duration, imputed cost, and the checker's pass/fail outcome.
+- [ ] The post-run repository state is captured as a snapshot and scored by the checker against the task's scoring spec.
+- [ ] A transcript audit runs after each cell; a run in which a foreign tool was reached is flagged invalid instead of scored.
+- [ ] A runnable Task wrapper is defined and one sample task runs the full path against the live host.

.claude/tasks/0028-bench-task-suite.md

@@ -0,0 +1,21 @@
+---
+spec: benchmark-harness
+blocked-by: [0024-bench-checker-and-scoring-spec, 0025-bench-seed-provisioning, 0027-bench-single-cell-runner]
+---
+
+## What to build
+
+The full scored task suite plus the capability-asymmetric bonus definitions, authored against the runnable Task wrapper and scored against the seed's ground truth.
+
+The scored suite is 20 tasks drawn only from the capability surface shared by all four arms — issue and pull-request listing, viewing, creation, editing, closing and reopening, commenting and comment retrieval, label management and application, review comments, merge, and assignee changes. Tasks are phrased as natural-language intents, not command invocations, parametrized against the seed, each carrying its tier tag and scoring spec. The suite is weighted toward discovery and multi-step work: roughly four read tasks, six single-mutation tasks, six find-then-act tasks, and four multi-step workflows.
+
+Review tasks default to comment-type reviews, which a single user can leave on their own pull request. Whether the host permits a user to approve or request changes on their own pull request is probed during implementation; if permitted, the two review tasks are promoted from comment reviews to approve and request-changes, otherwise those move to the bonus table.
+
+The bonus definitions cover capability-asymmetric operations in both directions: where tea, gitea-mcp, or raw API fall short of gitea-axi (full-text search, diff, checks, checkout, issue dependencies), and operations outside gitea-axi's scope (repository, release, and milestone management), for which gitea-axi is reported not-applicable. These are kept out of the scored suite.
+
+## Acceptance criteria
+
+- [ ] The scored suite has 20 tasks confined to the shared capability surface, phrased as natural-language intents parametrized against the seed.
+- [ ] The suite is weighted roughly four read / six single-mutation / six find-then-act / four multi-step, with each task carrying a tier tag and a scoring spec.
+- [ ] A self-review capability probe determines whether the two review tasks run as approve/request-changes or as comment reviews (falling back to the bonus table if self-review is not permitted).
+- [ ] Bonus task definitions cover the asymmetries in both directions, including the gitea-axi not-applicable operations, and are kept separate from the scored suite.

.claude/tasks/0029-bench-run-loop-cli.md

@@ -0,0 +1,17 @@
+---
+spec: benchmark-harness
+blocked-by: [0027-bench-single-cell-runner, 0028-bench-task-suite]
+---
+
+## What to build
+
+The maintainer-facing command that runs a chosen benchmark cell on demand, so only the token budget available at that moment is spent. The maintainer selects an arm and a task; the command runs that cell and accumulates results.
+
+Each cell defaults to five trials with a reporting floor of three. Because results are immutable timestamped samples, running a cell that already has samples deepens it — the new trials append rather than overwrite, so a cell's sample size can be grown opportunistically across separate sittings.
+
+## Acceptance criteria
+
+- [ ] The command runs a single selected `(arm, task)` cell on demand.
+- [ ] A cell defaults to five trials, and the reporting floor of three is respected.
+- [ ] Re-running an already-sampled cell appends new trials rather than overwriting prior samples.
+- [ ] The command drives the runner and store built in earlier slices rather than reimplementing orchestration.

.claude/tasks/0030-bench-aggregator-and-reporting.md

@@ -0,0 +1,20 @@
+---
+spec: benchmark-harness
+blocked-by: [0022-bench-scaffold-and-result-store, 0028-bench-task-suite]
+---
+
+## What to build
+
+The aggregator that renders the accumulated sample store into a readable comparison, rendering whatever exists and annotating incomplete coverage rather than blocking on a complete matrix.
+
+The headline table has one row per arm: cost-equivalent tokens as the headline, then raw tokens, turns, duration, success rate, and a coverage figure, with imputed cost shown as a de-emphasized secondary column. Cost-equivalent tokens are computed at render time by weighting each run's four retained components by Anthropic's published API pricing ratios (see the cost-equivalent-token-metric ADR), so the stored records can be re-weighted without re-running if the subscription's accounting is ever documented. Partially-run cells are annotated rather than hidden.
+
+Supporting views derived from the same records include a per-tier breakdown, a per-token-component breakdown, and the separate bonus table for the capability-asymmetric operations. The aggregator is a pure seam, unit-tested against synthetic sample stores.
+
+## Acceptance criteria
+
+- [ ] The headline table renders one row per arm with cost-equivalent tokens as the headline, plus raw tokens, turns, duration, success rate, coverage, and imputed cost as a de-emphasized secondary column.
+- [ ] Cost-equivalent tokens are computed from the retained four components at render time using the documented pricing-ratio weights.
+- [ ] A partial matrix renders without error and incomplete coverage is annotated rather than hidden or treated as complete.
+- [ ] Per-tier and per-token-component breakdowns and the separate bonus table are rendered from the same records.
+- [ ] Rendering is stable and unit-tested against a synthetic append-only sample store.