.claude/adr/0014-bench-cost-equivalent-token-metric.md

@@ -0,0 +1,26 @@
+# Cost-equivalent tokens as the benchmark's headline metric
+
+The benchmark compares how much each arm costs to drive.
+The maintainer runs on a Claude subscription with a fixed weekly token allowance, so the scarce resource is token consumption against that allowance, not dollars.
+The question is how to reduce each run's four token components — fresh input, cache-creation, cache-read, and output — into a single headline number that reflects weekly-budget burn.
+
+Research into Anthropic's documentation established that the exact unit and per-component weighting of the subscription weekly limit are not publicly documented.
+The one anchoring signal is that overage past the included allowance is billed at standard API rates, which points toward cost-weighted accounting rather than a flat token count.
+
+## Considered Options
+
+**Raw summed tokens as headline** (rejected) — Summing all four components at 1× is transparent and assumption-free, but cache-read routinely dominates the total, and Anthropic's API prices cache-reads at roughly a tenth of fresh input.
+A raw sum therefore overstates the burn of arms whose context is largely cached (notably the eager-schema MCP arm) by up to an order of magnitude, which would misrank the arms on the very axis the benchmark exists to measure.
+
+**Imputed dollars as headline** (rejected) — The runtime already reports an imputed cost that folds in every component at the correct weights.
+It is an accurate comparative number, but it is expressed in a unit the maintainer does not spend; on a subscription no dollars leave the account, and the mental model is weekly tokens.
+
+**Cost-equivalent tokens as headline** (chosen) — Weight the four components by Anthropic's published API pricing ratios (fresh input 1×, cache-write 1.25× or 2× by TTL, cache-read 0.1×, output 5×) and express the result as a token count.
+This is tokens — the maintainer's unit — weighted the way their budget most plausibly burns, and it is the same ranking as the imputed dollar figure.
+
+## Consequences
+
+- The headline is cost-equivalent tokens; the raw summed tokens and the full four-component breakdown are recorded alongside every run, so the data can be re-weighted without re-running if the subscription's real accounting is ever documented.
+- Imputed dollars are retained as a de-emphasized secondary column, portable for readers who are on the API rather than a subscription.
+- The weighting is an explicit, documented assumption grounded in the overage-pricing signal, not a measured fact; an optional later validation could pin the real weekly weighting empirically by burning a known token mix.
+- The auxiliary small model invoked by the runtime is counted rather than suppressed, since it is real consumption against the same allowance.

.claude/adr/0015-bench-single-user-seed.md

@@ -0,0 +1,22 @@
+# Single-user seed and its constraints on the task surface
+
+The benchmark seeds a throwaway repository to a known state before each trial and scores tasks against that ground truth.
+Seeding realistic author and assignee variety would require several Gitea accounts.
+The maintainer prefers to run the benchmark under their single existing account rather than provision additional accounts.
+
+## Considered Options
+
+**Provision throwaway collaborator accounts** (rejected) — Multiple accounts would restore the author and assignee dimensions and enable non-self pull-request approvals, but they add account lifecycle and credential handling that the maintainer explicitly declined for this benchmark.
+
+**Keep multi-user tasks and let arms fail** (rejected) — Retaining tasks that need distinct authors or a non-self reviewer would make those tasks impossible under one account for every arm uniformly, producing no comparative signal while consuming budget.
+
+**Single-user seed with a redesigned surface** (chosen) — All seed content is authored by the one account, and the discriminating dimensions become label, state, assignee presence (assigned-to-self versus unassigned), and title keyword instead of author.
+Tasks that assumed author or assignee variety are recast onto these axes: reassignment becomes assign-to-self or unassign, and author-filtered bulk mutation becomes a filter on assignee presence.
+
+## Consequences
+
+- Author filtering leaves the scored suite; it carries little signal with one author anyway.
+- Review tasks in the scored suite use comment-type reviews, which a user may leave on their own pull request.
+  Whether the host permits self-approval and self-request-changes is verified during implementation; if permitted, those two tasks are promoted from comment reviews to approve and request-changes, otherwise approve and request-changes move to the bonus table as two-account scenarios.
+- Non-self approval, distinct-author filtering, and distinct-assignee tasks are out of scope for the scored suite and belong to the multi-account bonus scenarios.
+- The seed stays small and fully deterministic, which keeps the full-state diff used for collateral-damage checking cheap to compute and reason about.

.claude/adr/0016-bench-guard-based-tool-isolation.md

@@ -0,0 +1,21 @@
+# Guard-based tool isolation instead of containers
+
+The benchmark's validity depends on each arm's agent reaching exactly one tool.
+If the tea-arm agent could quietly call `curl` or `gitea-axi`, the comparison would be meaningless.
+The benchmark machine has no container runtime available, so per-arm operating-system sandboxing is not an option.
+
+## Considered Options
+
+**Container per arm** (rejected) — A container image carrying only the arm's binary would give hard, kernel-level isolation, but it requires installing and depending on a container runtime the maintainer does not have and declined to add for this benchmark.
+
+**Curated PATH alone** (rejected) — Prepending a directory that exposes only the allowed binary is convenient but leaky: an agent can invoke another tool by absolute path, or reach the API through a language interpreter's fetch, bypassing PATH entirely.
+
+**Guard callback as the authority** (chosen) — A callback inspects every proposed shell command and permits only the one binary allow-listed for the active arm plus harmless utilities, denying foreign binaries, absolute-path evasions, and interpreter-based fetch attempts.
+A curated per-arm PATH backs it as a convenience layer, and the gitea-mcp arm disables the shell tool entirely and attaches only the MCP tools, giving that arm no leakage surface at all.
+
+## Consequences
+
+- The guard, not the PATH, is authoritative; the PATH is defense in depth.
+- A blocked attempt is left in the transcript and counts as realistic wasted effort, reflecting an agent fumbling with a tool that cannot do the job; blocked calls are never silently retried or discarded.
+- Every command is logged, and a post-run audit asserts no foreign tool was reached; a detected leak flags the trial invalid rather than letting it be scored.
+- Isolation strength rests on the completeness of the guard's deny rules, so the guard is one of the harness's primary unit-tested seams, covering absolute-path and interpreter-fetch evasions explicitly.

.claude/spec/benchmark-harness.md

@@ -0,0 +1,141 @@
+## Problem Statement
+
+gitea-axi claims to be an agent-ergonomic, low-token interface to Gitea issues and pull requests, positioned against the `tea` CLI, the official `gitea-mcp` server, and raw Gitea REST calls.
+That claim is currently unmeasured.
+The maintainer wants evidence — a reproducible comparison of correctness and token consumption across those four ways of driving Gitea from a coding agent, presented as a table analogous to the published gh-axi benchmark.
+
+Because the maintainer runs on a Claude subscription with a fixed weekly token allowance (not API credits), the resource that actually matters is token usage, not dollars.
+And because a full sweep would be expensive to run in one sitting, the benchmark must be runnable incrementally — one arm-and-task cell at a time, whenever spare budget is available — accumulating results rather than requiring a single monolithic run.
+
+## Solution
+
+A benchmark harness, living in a `bench/` directory in this repository (excluded from the npm package), that drives a Claude agent against a suite of realistic Gitea tasks under four tool conditions and records how each performs.
+
+For each **cell** — one `(arm, task, trial)` combination — the harness provisions a fresh throwaway repository on the live Gitea host, seeds it to a known state, runs the agent with access to exactly one arm's tool, scores the outcome deterministically against the seeded ground truth, and appends an immutable result record.
+An aggregator renders the accumulated records into a headline table (one row per arm) plus supporting views, annotating any partially-run cells rather than blocking on a complete matrix.
+
+The four arms are **gitea-axi**, **tea** (native structured commands only), **gitea-mcp** (eager schemas), and **raw Gitea REST API** (curl).
+The headline metric is **cost-equivalent tokens** — token components weighted by Anthropic's published API pricing ratios — reported alongside the raw token sum and the full component breakdown.
+
+## User Stories
+
+1. As the maintainer, I want to run a single benchmark cell on demand by selecting an arm and a task, so that I can spend only the token budget I have available at that moment.
+2. As the maintainer, I want each cell to run against a freshly provisioned and seeded throwaway repository, so that trials are isolated and correctness is scored against a known ground truth.
+3. As the maintainer, I want the agent in each arm to have access to exactly one tool, so that the comparison measures the tool rather than the agent's choice between tools.
+4. As the maintainer, I want the gitea-axi arm to carry the product's bundled Agent Skill and the other arms to receive only a minimal pointer to their tool's native discovery affordance, so that each tool's real ambient-context cost is charged honestly.
+5. As the maintainer, I want mutation tasks scored by diffing the entire post-run repository state against the expected end state, so that both the intended change and any collateral damage are caught.
+6. As the maintainer, I want read tasks scored by matching required facts in the agent's final report against the seeded ground truth, so that correctness is judged without an LLM-judge.
+7. As the maintainer, I want the headline metric to be cost-equivalent tokens with the raw sum and per-component breakdown recorded alongside, so that I can gauge weekly-budget burn today and re-weight the data if Anthropic's accounting is ever documented.
+8. As the maintainer, I want each run bounded by a turn cap and a wall-clock backstop, so that a confused or hung agent cannot drain my budget.
+9. As the maintainer, I want results appended as immutable timestamped samples, so that I can deepen any cell's sample size opportunistically without overwriting prior runs.
+10. As the maintainer, I want the aggregator to render whatever results exist and annotate incomplete coverage, so that a half-run matrix still produces a readable, non-misleading table.
+11. As the maintainer, I want tasks confined to the capability surface shared by all four arms, so that success differences reflect ergonomics rather than scope.
+12. As the maintainer, I want capability-asymmetric operations reported in a separate bonus table, so that each tool's distinctive edges and gaps are visible without contaminating the headline comparison.
+13. As the maintainer, I want a per-tier and per-token-component view derived from the same records, so that I can see where an arm wins or loses and what drives its cost.
+14. As the maintainer, I want the harness to reject or flag any run in which the agent reached a tool outside its arm, so that a leak invalidates the trial instead of silently corrupting the metrics.
+
+## Implementation Decisions
+
+### Arms
+
+Four arms are compared:
+
+- **gitea-axi** — the hero arm; its bundled Agent Skill is loaded into the agent's context, matching how the product ships.
+- **tea** — restricted to its native structured subcommands.
+  The `tea api` escape hatch is excluded, because allowing it would collapse the tea arm into the raw-API arm.
+- **gitea-mcp** — the official server (v1.3.0 at design time), with its full set of dispatcher tools loaded eagerly.
+- **raw Gitea REST API** — the agent issues `curl` calls against `HOST/api/v1` with a bearer token.
+
+### Environment
+
+Every cell runs against the live Gitea host, but isolation comes from a per-trial throwaway repository the harness creates, seeds, runs against, and deletes.
+The harness authenticates by reusing gitea-axi's existing credential discovery path rather than introducing new secret handling.
+
+### Seed
+
+The seed is scripted through the Gitea API and is deterministic and idempotent.
+It establishes a fixed set of labels with fixed colors, a spread of open and closed issues varying by label, assignee state, title keyword, and pre-existing comments, and a handful of pull requests including one labeled, one carrying an existing review, and one backed by a real pushed feature branch.
+All content is authored by the single available user.
+
+Because only one Gitea account is available, the seed carries no author or assignee variety across users.
+Discriminating dimensions are label, state, assignee presence (assigned-to-self versus unassigned), and title keyword.
+
+### Task suite
+
+The scored suite is 20 tasks drawn only from the capability surface shared by all four arms — issue and pull-request listing, viewing, creation, editing, closing and reopening, commenting and comment retrieval, label management and application, review comments, merge, and assignee changes.
+Tasks are phrased as natural-language intents, not command invocations, and are parametrized against the seed.
+The suite is weighted toward discovery and multi-step work, where tool ergonomics diverge: roughly four read tasks, six single-mutation tasks, six find-then-act tasks, and four multi-step workflows.
+
+Reviews in the scored suite use comment-type reviews, which a single user can leave on their own pull request.
+Whether the host permits a user to approve or request changes on their own pull request is verified during implementation; if permitted, the two review tasks are promoted from comment reviews to approve and request-changes.
+
+Capability-asymmetric operations are excluded from the scored suite and reported in a separate bonus table.
+These fall in both directions: operations where tea, gitea-mcp, or raw API fall short of gitea-axi (full-text search, diff, checks, checkout, issue dependencies), and operations outside gitea-axi's scope entirely (repository, release, and milestone management), for which gitea-axi is reported as not-applicable.
+
+### Scaffolding
+
+All arms share an identical task-agnostic base prompt and the same repository coordinates and token.
+Each arm then receives a minimal, symmetric bootstrap naming its tool and pointing at that tool's own native discovery affordance — with the deliberate exception that the gitea-axi arm loads the bundled Agent Skill, because the Skill is part of the shipped product and its token cost should be charged to gitea-axi.
+The tea and raw-API arms receive a one-line pointer; the gitea-mcp arm's schemas load eagerly as its ambient cost.
+
+### Tool isolation
+
+Enforcement is a guard callback that inspects every proposed shell command and permits only the one binary allow-listed for the active arm plus harmless utilities, denying foreign binaries, absolute-path evasions, and fetch-via-interpreter tricks.
+A curated per-arm PATH backs the guard as a convenience layer.
+The gitea-mcp arm disables the shell tool entirely and attaches only the MCP tools.
+Blocked attempts are left in the transcript and count as realistic wasted effort; they are not silently retried.
+
+### Runner and metrics
+
+The runner is the Claude Agent SDK on the maintainer's subscription, using a single fixed model at temperature zero across all arms.
+The auxiliary small model that the agent runtime invokes for internal chores is included in metrics rather than suppressed, because it is real consumption.
+
+Each completed run records the four token components (fresh input, cache-creation, cache-read, output), the turn count, the wall-clock duration, the imputed cost, and the pass/fail outcome.
+The headline metric, **cost-equivalent tokens**, weights the components by Anthropic's published API pricing ratios (see the cost-equivalent-token-metric ADR).
+The raw token sum and the component breakdown are retained so the data can be re-weighted if the subscription's weekly accounting is ever documented.
+
+### Run loop and results
+
+Each cell defaults to five trials, with a reporting floor of three.
+Each run is bounded by a turn cap and a wall-clock backstop; exceeding either records a failure, tagged to distinguish a confused agent from a hung one.
+Results are appended as immutable, timestamped samples to a per-cell store; deepening a cell adds samples rather than overwriting slots.
+
+### Reporting
+
+The aggregator reads the accumulated results and renders a headline table with one row per arm — cost-equivalent tokens, raw tokens, turns, duration, success rate, and a coverage figure — with imputed cost shown as a de-emphasized secondary column.
+It renders whatever exists and annotates incomplete coverage rather than blocking on a full matrix.
+Supporting views derived from the same records include a per-tier breakdown, a per-token-component breakdown, and the separate bonus table.
+
+## Testing Decisions
+
+A good test here exercises external behavior at a seam, not internal wiring, mirroring the repository's existing split between a deterministic fixture tier and a live end-to-end tier.
+
+Three pure seams are unit-tested:
+
+- The **checker**, fed synthetic state snapshots and expected states, covering the normalization rules (dropping volatile identifiers and timestamps, matching comments by author and body, comparing label sets), the full-state diff that catches both missing intent and collateral change, and the deterministic answer-match for read tasks.
+- The **guard**, covering that each arm's allow-listed binary passes and that foreign binaries, absolute-path evasions, and interpreter-based fetch attempts are denied.
+- The **aggregator**, covering partial-matrix annotation, coverage reporting, per-tier and per-component rollups, and stable rendering from an append-only sample store.
+
+Two boundaries are validated by integration and audit rather than unit tests:
+
+- **Seed provisioning** against the live host, validated by a smoke run rather than mocked, since its value is the real API interaction.
+- **Run orchestration** against the real model, validated by the post-run transcript audit that asserts no foreign tool was reached; a detected leak flags the trial invalid.
+
+Prior art for the deterministic seams is the project's existing fixture-server tier; prior art for the live boundary is the existing end-to-end tier, including its use of polling for the eventually-consistent issue indexer.
+
+## Out of Scope
+
+- The numbered task file that schedules this work is authored separately by the maintainer.
+- Repository, release, milestone, and other operations outside gitea-axi's command surface are not scored; they appear only in the bonus table.
+- Multi-account scenarios (distinct authors and assignees, and non-self approvals) are out of scope for the scored suite under the single-user constraint.
+- Empirically pinning the subscription's exact weekly-budget weighting is a possible later validation, not part of this harness.
+- Container-based operating-system isolation is not used; the guard is the enforcement mechanism.
+
+## Further Notes
+
+The gitea-mcp server uses a compact read/write dispatcher design rather than one eager schema per operation, so the MCP arm may not reproduce the dramatic token inflation seen for heavier MCP servers in the reference gh-axi benchmark.
+That is a legitimate result about dispatcher-style MCP design, not a defect in the harness, and the expectation is set here so the outcome is not read as a bug.
+
+Duration is treated as a soft metric throughout, because every arm runs against the live host and inherits its network variance.
+
+The benchmark's own vocabulary (arm, cell, shared surface, cost-equivalent tokens, seed, checker) is intentionally kept in this spec and the `bench/` documentation rather than in the tool's domain glossary, which describes gitea-axi's own language and should not be diluted by harness terms.

.claude/tasks/0022-bench-scaffold-and-result-store.md

@@ -0,0 +1,18 @@
+---
+spec: benchmark-harness
+---
+
+## What to build
+
+The foundation the whole benchmark harness reads and writes: a `bench/` directory (excluded from the published npm package, alongside the existing `dist`/`skills` allow-list), the immutable result-record shape, and an append-only per-cell sample store.
+
+A result record captures one completed `(arm, task, trial)` run: the four token components (fresh input, cache-creation, cache-read, output), the turn count, the wall-clock duration, the imputed cost, and the pass/fail outcome with a failure tag distinguishing a confused agent from a hung one. It also carries the tags later views group by — arm, task id, tier, trial, and a timestamp.
+
+The store appends records as immutable, timestamped samples to a per-cell location. Deepening a cell's sample size adds samples rather than overwriting any prior run, and reading a cell returns every accumulated sample.
+
+## Acceptance criteria
+
+- [ ] A `bench/` directory exists and is excluded from the npm package (verified by the packaging tier or an equivalent `files` check).
+- [ ] The result-record shape records the four token components, turns, duration, imputed cost, outcome, failure tag, and the arm/task/tier/trial/timestamp tags.
+- [ ] Appending a sample to a cell that already has samples leaves the prior samples intact; reading the cell returns all of them.
+- [ ] A round-trip test writes several samples across cells and reads back exactly what was written.

.claude/tasks/0023-bench-tool-isolation-guard.md

@@ -0,0 +1,17 @@
+---
+spec: benchmark-harness
+---
+
+## What to build
+
+The guard that keeps each arm's agent confined to exactly one tool, so a benchmark result measures the tool rather than the agent's choice between tools (see the guard-based-tool-isolation ADR).
+
+The guard is a callback that inspects every proposed shell command and permits only the one binary allow-listed for the active arm plus a curated set of harmless utilities, denying everything else. It rejects foreign binaries, absolute-path evasions that sidestep the allow-list, and interpreter-based fetch tricks (reaching the API through a language runtime's HTTP client). A curated per-arm PATH backs the guard as a convenience layer, but the guard — not the PATH — is authoritative. A blocked attempt is surfaced, not silently retried.
+
+## Acceptance criteria
+
+- [ ] Each arm's allow-listed binary passes the guard; a foreign binary is denied.
+- [ ] An absolute-path invocation of a foreign binary is denied.
+- [ ] An interpreter-based fetch attempt (e.g. driving an HTTP request through a language runtime) is denied.
+- [ ] A curated per-arm PATH is produced exposing only that arm's allowed binary.
+- [ ] Unit tests cover the allowed-binary, foreign-binary, absolute-path, and interpreter-fetch cases per arm.

.claude/tasks/0024-bench-checker-and-scoring-spec.md

@@ -0,0 +1,19 @@
+---
+spec: benchmark-harness
+---
+
+## What to build
+
+The pure scoring seam that turns a completed run into a deterministic pass/fail, plus the scoring-spec contract each task pairs with.
+
+For a mutation task, the checker diffs the entire post-run repository state against the expected end state, so both the intended change and any collateral damage are caught. Comparison runs after normalization: volatile identifiers and timestamps are dropped, comments are matched by author and body, and labels are compared as sets. For a read task, the checker matches required facts in the agent's final report against the seeded ground truth, with no LLM judge.
+
+This slice also defines the scoring-spec contract — a task's expected end state (for mutations) or its required answer facts (for reads) — that the checker consumes and that the runner and task suite will produce. The checker is fed synthetic state snapshots and expected states; capturing live state from a real repository is the runner's job.
+
+## Acceptance criteria
+
+- [ ] Given synthetic actual and expected state snapshots, the full-state diff passes when they match after normalization and fails when the actual state is missing the intended change.
+- [ ] The diff fails when the actual state carries collateral change beyond the intended mutation.
+- [ ] Normalization drops volatile identifiers and timestamps, matches comments by author and body, and compares label sets order-independently.
+- [ ] The read-task answer-match passes when the required facts are present in the final report and fails when a required fact is missing.
+- [ ] The scoring-spec contract expresses both a mutation's expected end state and a read's required answer facts.

.claude/tasks/0025-bench-seed-provisioning.md

@@ -0,0 +1,16 @@
+---
+spec: benchmark-harness
+---
+
+## What to build
+
+The deterministic, idempotent seed that brings a freshly provisioned throwaway repository to a known ground truth before a trial runs, scripted entirely over the Gitea API against the live host (see the single-user-seed ADR). Authentication reuses gitea-axi's existing credential discovery path rather than introducing new secret handling.
+
+The seed establishes a fixed set of labels with fixed colors; a spread of open and closed issues varying by label, assignee presence (assigned-to-self versus unassigned), title keyword, and pre-existing comments; and a handful of pull requests including one labeled, one carrying an existing review, and one backed by a real pushed feature branch. All content is authored by the single available user, so the discriminating dimensions are label, state, assignee presence, and title keyword — not author.
+
+## Acceptance criteria
+
+- [ ] Provisioning a fresh repository and seeding it produces the fixed labels, the open/closed issue spread across the discriminating dimensions, and the pull requests (labeled, reviewed, and real-branch-backed).
+- [ ] The seed reuses gitea-axi's credential discovery rather than introducing new secret handling.
+- [ ] Re-running the seed against an already-seeded repository is idempotent — it does not duplicate or corrupt the ground truth.
+- [ ] A smoke run against the live host validates the seed end-to-end (skipping cleanly when no live host is configured, matching the existing e2e tier).

.claude/tasks/0026-bench-arm-scaffolding.md

@@ -0,0 +1,18 @@
+---
+spec: benchmark-harness
+blocked-by: 0023-bench-tool-isolation-guard
+---
+
+## What to build
+
+The per-arm scaffolding that charges each tool's real ambient-context cost honestly. All arms share one task-agnostic base prompt and the same repository coordinates and token; each arm then receives a minimal, symmetric bootstrap naming its tool and pointing at that tool's own native discovery affordance.
+
+The deliberate asymmetries follow the shipped products: the gitea-axi arm loads the bundled Agent Skill, because the Skill is part of what ships and its token cost belongs to gitea-axi. The tea and raw-API arms receive a one-line pointer to their native discovery affordance. The gitea-mcp arm's dispatcher schemas load eagerly as its ambient cost, and that arm disables the shell tool entirely, attaching only the MCP tools. Each arm's assembled prompt plus tool/PATH configuration (from the guard) is produced as a single arm definition the runner consumes.
+
+## Acceptance criteria
+
+- [ ] All arms share the identical base prompt and are handed the same repository coordinates and token.
+- [ ] The gitea-axi arm's assembled context carries the bundled Agent Skill.
+- [ ] The tea and raw-API arms each receive only a one-line native-discovery pointer.
+- [ ] The gitea-mcp arm loads its dispatcher schemas eagerly, has the shell tool disabled, and is attached only the MCP tools.
+- [ ] Each non-MCP arm's tool/PATH configuration comes from the guard and exposes only that arm's allowed binary.

.claude/tasks/0027-bench-single-cell-runner.md

@@ -0,0 +1,21 @@
+---
+spec: benchmark-harness
+blocked-by: [0022-bench-scaffold-and-result-store, 0023-bench-tool-isolation-guard, 0024-bench-checker-and-scoring-spec, 0025-bench-seed-provisioning, 0026-bench-arm-scaffolding]
+---
+
+## What to build
+
+The tracer bullet that threads every layer: run a single `(arm, task, trial)` cell end-to-end and record an immutable result. This is the walking skeleton — one arm against one sample task, one trial — that proves seed, arm scaffolding, guard, runner, checker, and store all connect.
+
+The runner provisions and seeds a fresh throwaway repository, runs the agent via the Claude Agent SDK on a single fixed model at temperature zero with exactly the active arm's tool and guard enforced, and bounds the run by a turn cap and a wall-clock backstop — exceeding either records a failure tagged to distinguish a confused agent from a hung one. It captures the four token components (including the auxiliary small model the runtime invokes, since that is real consumption), the turn count, the duration, and the imputed cost. After the run it captures the entire post-run repository state as a snapshot, scores it with the checker against the task's scoring spec, appends the result sample to the store, and deletes the throwaway repository. A post-run transcript audit asserts no foreign tool was reached; a detected leak flags the trial invalid rather than letting it be scored.
+
+This slice also defines the runnable Task wrapper (natural-language intent, parameters, tier, and scoring spec) and includes one sample task to exercise the path; the full suite is authored in a later slice.
+
+## Acceptance criteria
+
+- [ ] Running one cell provisions and seeds a fresh repository, runs the agent under its arm's tool with the guard active, and deletes the repository afterward.
+- [ ] The run is bounded by both a turn cap and a wall-clock backstop; exceeding either records a failure tagged confused-versus-hung.
+- [ ] The recorded sample carries the four token components (including the auxiliary small model), turns, duration, imputed cost, and the checker's pass/fail outcome.
+- [ ] The post-run repository state is captured as a snapshot and scored by the checker against the task's scoring spec.
+- [ ] A transcript audit runs after each cell; a run in which a foreign tool was reached is flagged invalid instead of scored.
+- [ ] A runnable Task wrapper is defined and one sample task runs the full path against the live host.

.claude/tasks/0028-bench-task-suite.md

@@ -0,0 +1,21 @@
+---
+spec: benchmark-harness
+blocked-by: [0024-bench-checker-and-scoring-spec, 0025-bench-seed-provisioning, 0027-bench-single-cell-runner]
+---
+
+## What to build
+
+The full scored task suite plus the capability-asymmetric bonus definitions, authored against the runnable Task wrapper and scored against the seed's ground truth.
+
+The scored suite is 20 tasks drawn only from the capability surface shared by all four arms — issue and pull-request listing, viewing, creation, editing, closing and reopening, commenting and comment retrieval, label management and application, review comments, merge, and assignee changes. Tasks are phrased as natural-language intents, not command invocations, parametrized against the seed, each carrying its tier tag and scoring spec. The suite is weighted toward discovery and multi-step work: roughly four read tasks, six single-mutation tasks, six find-then-act tasks, and four multi-step workflows.
+
+Review tasks default to comment-type reviews, which a single user can leave on their own pull request. Whether the host permits a user to approve or request changes on their own pull request is probed during implementation; if permitted, the two review tasks are promoted from comment reviews to approve and request-changes, otherwise those move to the bonus table.
+
+The bonus definitions cover capability-asymmetric operations in both directions: where tea, gitea-mcp, or raw API fall short of gitea-axi (full-text search, diff, checks, checkout, issue dependencies), and operations outside gitea-axi's scope (repository, release, and milestone management), for which gitea-axi is reported not-applicable. These are kept out of the scored suite.
+
+## Acceptance criteria
+
+- [ ] The scored suite has 20 tasks confined to the shared capability surface, phrased as natural-language intents parametrized against the seed.
+- [ ] The suite is weighted roughly four read / six single-mutation / six find-then-act / four multi-step, with each task carrying a tier tag and a scoring spec.
+- [ ] A self-review capability probe determines whether the two review tasks run as approve/request-changes or as comment reviews (falling back to the bonus table if self-review is not permitted).
+- [ ] Bonus task definitions cover the asymmetries in both directions, including the gitea-axi not-applicable operations, and are kept separate from the scored suite.

.claude/tasks/0029-bench-run-loop-cli.md

@@ -0,0 +1,17 @@
+---
+spec: benchmark-harness
+blocked-by: [0027-bench-single-cell-runner, 0028-bench-task-suite]
+---
+
+## What to build
+
+The maintainer-facing command that runs a chosen benchmark cell on demand, so only the token budget available at that moment is spent. The maintainer selects an arm and a task; the command runs that cell and accumulates results.
+
+Each cell defaults to five trials with a reporting floor of three. Because results are immutable timestamped samples, running a cell that already has samples deepens it — the new trials append rather than overwrite, so a cell's sample size can be grown opportunistically across separate sittings.
+
+## Acceptance criteria
+
+- [ ] The command runs a single selected `(arm, task)` cell on demand.
+- [ ] A cell defaults to five trials, and the reporting floor of three is respected.
+- [ ] Re-running an already-sampled cell appends new trials rather than overwriting prior samples.
+- [ ] The command drives the runner and store built in earlier slices rather than reimplementing orchestration.

.claude/tasks/0030-bench-aggregator-and-reporting.md

@@ -0,0 +1,20 @@
+---
+spec: benchmark-harness
+blocked-by: [0022-bench-scaffold-and-result-store, 0028-bench-task-suite]
+---
+
+## What to build
+
+The aggregator that renders the accumulated sample store into a readable comparison, rendering whatever exists and annotating incomplete coverage rather than blocking on a complete matrix.
+
+The headline table has one row per arm: cost-equivalent tokens as the headline, then raw tokens, turns, duration, success rate, and a coverage figure, with imputed cost shown as a de-emphasized secondary column. Cost-equivalent tokens are computed at render time by weighting each run's four retained components by Anthropic's published API pricing ratios (see the cost-equivalent-token-metric ADR), so the stored records can be re-weighted without re-running if the subscription's accounting is ever documented. Partially-run cells are annotated rather than hidden.
+
+Supporting views derived from the same records include a per-tier breakdown, a per-token-component breakdown, and the separate bonus table for the capability-asymmetric operations. The aggregator is a pure seam, unit-tested against synthetic sample stores.
+
+## Acceptance criteria
+
+- [ ] The headline table renders one row per arm with cost-equivalent tokens as the headline, plus raw tokens, turns, duration, success rate, coverage, and imputed cost as a de-emphasized secondary column.
+- [ ] Cost-equivalent tokens are computed from the retained four components at render time using the documented pricing-ratio weights.
+- [ ] A partial matrix renders without error and incomplete coverage is annotated rather than hidden or treated as complete.
+- [ ] Per-tier and per-token-component breakdowns and the separate bonus table are rendered from the same records.
+- [ ] Rendering is stable and unit-tested against a synthetic append-only sample store.