From 4cbed21ff3d23fcf43c5d4f9f983cbe254aa98c1 Mon Sep 17 00:00:00 2001 From: alexion Date: Fri, 17 Jul 2026 09:50:38 -0400 Subject: [PATCH 01/10] feat: persist agent report on read result records (task 0032) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Retain the agent's final report on the benchmark result record for read tasks, so a failed read is diagnosable directly from the stored record instead of only carrying an opaque `incorrect` tag. The runner resolves the scoring spec once and records `run.finalReport` when the spec is a read; mutation records omit the field entirely. The sample store needs no change — it serializes whatever record it is handed. This is the prerequisite for confirming the read-open-issue-count failure from real report text before the state-aware count-line change (task 0033). --- .claude/spec/read-tier-accuracy.md | 78 +++++++++++++ .../0032-bench-read-report-persistence.md | 26 +++++ bench/result.ts | 9 ++ bench/runner.test.ts | 107 ++++++++++++++++++ bench/runner.ts | 24 +++- bench/store.test.ts | 20 ++++ 6 files changed, 258 insertions(+), 6 deletions(-) create mode 100644 .claude/spec/read-tier-accuracy.md create mode 100644 .claude/tasks/0032-bench-read-report-persistence.md diff --git a/.claude/spec/read-tier-accuracy.md b/.claude/spec/read-tier-accuracy.md new file mode 100644 index 0000000..7f79b71 --- /dev/null +++ b/.claude/spec/read-tier-accuracy.md @@ -0,0 +1,78 @@ +# Read-tier accuracy + +## Problem Statement + +The benchmark ranks gitea-axi first on both cost metrics — lowest cost-equivalent tokens and lowest imputed cost of the four arms — but second on accuracy, 95% against gitea-mcp's 97%. +That entire two-point deficit is a single task: `read-open-issue-count`, which gitea-axi fails on all three trials while gitea-mcp passes one of three. +The read tier is the weakest tier for every arm, and on it gitea-axi spends more turns and more output than gitea-mcp yet scores lower, so agents are working harder to answer count questions and still getting them wrong. + +Two things stand in the way of closing this gap. +First, the `issue list` summary reports `count: N of M total` and never names the state it filtered on, so an agent asking "how many issues are open?" has to infer the answer rather than read it off the summary line. +Second, the benchmark records only tokens and a `failure: "incorrect"` tag for a failed read — never the agent's actual report — so a maintainer cannot tell whether the agent reported a wrong number or reported the right number in wording the checker's phrase list did not accept. +Without the report text the root cause of the 3/3 failure cannot be confirmed, so the product fix would be a guess. + +## Solution + +Make list count summaries self-answering, and make read failures diagnosable. + +For the product, the count line of a filtered list names the state it counted. +When a maintainer or an agent runs `issue list` on the default open filter, the summary states that the count is a count of open issues, so the answer to "how many are open?" is present in the summary rather than only inferable from each row's state field. + +For the harness, every result record carries the agent's final report for read tasks. +A failed read is then diagnosable directly from the stored record: the maintainer can see the exact text the agent submitted and tell a wrong count apart from a right count phrased in words the checker did not list. +This is the prerequisite that turns the `read-open-issue-count` failure from an opaque tag into evidence, and it is the data that decides whether the checker's accepted-phrasing list is later too strict. + +## User Stories + +1. As an agent answering a count question, I want the list summary to state which state it counted, so that I can report "5 open issues" straight from the summary line without inferring it from the rows. +2. As a maintainer reading `issue list` output, I want the count line to say what it counted, so that a bare `count: 5` is never ambiguous about whether those five are open, closed, or all issues. +3. As a maintainer auditing a failed read trial, I want the stored result record to include the agent's final report, so that I can see exactly what the agent said instead of only that it was "incorrect". +4. As a maintainer deciding whether the read checker is too strict, I want the persisted reports across trials, so that I can judge from evidence whether correct answers are being rejected on wording rather than substance. +5. As a maintainer re-running the benchmark after the count-line change, I want `read-open-issue-count` to move from a consistent failure toward a pass, so that gitea-axi's accuracy stops trailing on the one task that accounts for the whole gap. + +## Implementation Decisions + +- Two modules change: the `issue list` command in the product, and the benchmark runner's result-record assembly in the harness. + They serve one goal but are testable at different seams and can land independently, with the harness change first so the product fix can be confirmed against real report text. + +- **State-aware count line (product).** + The list command composes the state into its own count line; the generic render helper that formats count lines stays generic. + The command already resolves the effective state filter (defaulting to open), so it passes that filter descriptor into the summary rather than the helper learning about issue state. + This keeps a single render seam and lets each list command opt in on its own terms; `pr list`, `search`, and `dashboard` are unaffected unless they choose to opt in the same way. + The existing count-line invariants are preserved: a total is always reported, and the bare `count: N` form must never appear. + +- **Report persistence (harness).** + The agent's final report is already in hand at the point the runner scores a read task, so persisting it is threading that value into the record the runner assembles rather than plumbing it up from a new source. + The report is added to the result record for read tasks; the store serializes whatever record it is handed, so it needs no change of its own. + Mutation tasks are scored by diffing repository state and have no agent report to record, so the field is populated for read tasks and absent otherwise. + +- **Ordering.** + Persist the report first and re-run enough of the read tier to capture real reports, then confirm from those reports whether the failure is a wrong count or a rejected phrasing before finalizing the count-line wording. + The count-line wording should be chosen so that an agent quoting the summary lands on an answer the read checker already accepts. + +## Testing Decisions + +- A good test here asserts external behavior: the text a user or agent sees on stdout, and the shape of the record the harness writes — not the internal composition of the count string. + +- **Count line** is tested at the command's behavioral seam — the fixture-server CLI harness that runs the real command against a stubbed Gitea and asserts on rendered stdout. + Prior art is the existing `issue list` command test, which already asserts exact count-line strings such as `count: 3 of 17 total` and guards that the bare `count: N` form never appears. + New assertions extend that file: the count line names the state for the default open filter and for an explicit state filter, and the existing total-and-cap invariants still hold. + +- **Report persistence** is tested at the runner's record-assembly seam, where the existing runner test already drives a cell to a recorded outcome and asserts on the produced record. + A completed read cell yields a record carrying the agent's final report; a mutation cell yields a record without one. + +## Out of Scope + +- Trimming gitea-axi's input and cache-read footprint. + gitea-axi replays the largest context of the efficient arms and wins cost only on the 5×-weighted output component, so reducing its input footprint would harden the cost lead — but it is a separate optimization touching many commands and is not part of closing the accuracy gap. + +- Relaxing the read checker's accepted-phrasing list. + Whether the checker is too strict is a benchmark-validity decision that must be made from the persisted report evidence this spec produces, not pre-judged; changing accepted phrasings before seeing the reports risks tuning the benchmark to the tool rather than fixing the tool. + +- Any change to how mutation tasks are scored, and any change to the cost-equivalent-token metric or its weighting. + +## Further Notes + +The two leaders trade a narrow accuracy edge for a clear cost lead, and this one task is the whole of that edge, so the count-line change is the single highest-leverage move on the accuracy axis. +The report-persistence change also has standing value beyond this task: it makes every future read failure diagnosable rather than opaque, which is the harness's current blind spot and the reason the root cause could not be confirmed from the existing records. +The state-explicit summary is squarely on gitea-axi's central thesis — an agent-ergonomic, low-token interface — because it hands the answer to a count question directly on the summary line instead of forcing the agent into extra turns to derive it. diff --git a/.claude/tasks/0032-bench-read-report-persistence.md b/.claude/tasks/0032-bench-read-report-persistence.md new file mode 100644 index 0000000..6684007 --- /dev/null +++ b/.claude/tasks/0032-bench-read-report-persistence.md @@ -0,0 +1,26 @@ +--- +spec: read-tier-accuracy +--- + +## What to build + +Persist the agent's final report on the benchmark result record for read tasks, so a failed read is diagnosable directly from the stored record instead of only carrying an opaque `incorrect` tag. + +The report is already in hand where the runner scores a read task — threading it into the assembled record is all that is required; the sample store serializes whatever record it is handed and needs no change of its own. + +Mutation tasks are scored by diffing repository state and have no agent report, so the field is populated for read tasks and absent otherwise. + +## Acceptance criteria + +- [x] The result record carries the agent's final report for read tasks. +- [x] The field is absent on records for mutation tasks. +- [x] A completed read cell produces a record whose report is the agent's final report; a mutation cell produces a record without one — asserted at the runner's record-assembly seam. +- [x] The sample store round-trips the report-bearing record without any change to the store itself. +- [x] Existing runner and store tests still pass. + +## Implementation Notes + +- `ResultRecord` gained an optional `report?: string`. The runner resolves the scoring spec once in `runCell` and records `run.finalReport` only when `spec.kind === "read"`, so the field is populated for read tasks and absent otherwise. `makeRecord` conditionally spreads the key (`...(report !== undefined ? { report } : {})`) so a mutation (or hung) record genuinely omits it rather than carrying `report: undefined`; verified by `expect(sample).not.toHaveProperty("report")` after the JSON round-trip. +- `scoreRun` was refactored to take a pre-resolved `ScoringSpec` instead of a `BenchTask`. This is marginally more than "thread the value into the record," but it is the minimal clean way to branch on `spec.kind` in `runCell` without calling `task.scoringSpec(owner)` twice. +- The store needed no change, as the spec predicted: it serializes whatever record it is handed, so the round-trip test passed on first run. +- Two review findings were left as deliberate judgement calls (both non-blocking baseline smells, no documented-standard breach): the `makeRecord` positional parameter list (extended by one param in the module's pre-existing positional style, kept consistent with surrounding code rather than refactored to an options object), and the two `spec.kind` branches a few lines apart in `runCell` and `scoreRun` (they select different things — the report vs. the scorer — and read clearer inline). diff --git a/bench/result.ts b/bench/result.ts index 323678e..6327320 100644 --- a/bench/result.ts +++ b/bench/result.ts @@ -72,6 +72,15 @@ export interface ResultRecord { /** Pass/fail outcome with a failure tag. */ outcome: Outcome; + + /** + * The agent's final report, retained for read tasks so a failed read is + * diagnosable directly from the record — the exact text the agent submitted, + * distinguishing a wrong answer from a right one phrased in words the checker's + * accepted-phrasing list did not match. Absent for mutation tasks, which are + * scored by diffing repository state and have no agent report to record. + */ + report?: string; } /** diff --git a/bench/runner.test.ts b/bench/runner.test.ts index 9f47520..4747354 100644 --- a/bench/runner.test.ts +++ b/bench/runner.test.ts @@ -6,6 +6,7 @@ import type { RepoState } from "./scoring-spec.js"; import type { BenchAccess, RepoCoords } from "./seed.js"; import { groundTruth } from "./seed-plan.js"; import { createSampleStore } from "./store.js"; +import type { BenchTask } from "./task.js"; import { SAMPLE_TASK } from "./task.js"; import type { AgentDriver, BenchHost } from "./runner.js"; import { runCell } from "./runner.js"; @@ -397,4 +398,110 @@ describe("runCell", () => { if (sample === undefined) return; expect(sample.outcome).toEqual({ pass: false, failure: "incorrect" }); }); + + // Behavior: a completed read cell produces a record whose report is the agent's + // final report (benchmark-harness spec, record-assembly seam). A read task is + // scored by matching the agent's final report against required facts; the + // recorded ResultRecord must carry that final report in its `report` field so a + // read is diagnosable from the record alone. Here an inline read task's fact is + // satisfied by the driver's finalReport (it contains "5 open"), so the cell + // completes as a pass; the recorded sample's `report` must equal the exact + // planted finalReport literal — an independent literal, read back through the + // store, not recomputed from runner.ts. + it("records the agent's final report on a completed read cell", async () => { + const READ_TASK: BenchTask = { + id: "count-open-issues", + tier: "read", + intent: "How many issues are open?", + scoringSpec: () => ({ + kind: "read", + facts: [{ description: "open-issue count", anyOf: ["5 open"] }], + }), + }; + + // The independent literal we plant as the driver's final report. It contains + // the fact's rendering ("5 open"), so the read scorer scores it a pass. + const FINAL_REPORT = "There are 5 open issues."; + + const readDriver: AgentDriver = { + async run() { + return { + tokens: { freshInput: 50, cacheCreation: 0, cacheRead: 0, output: 10 }, + turns: 2, + imputedCostUsd: 0.03, + transcript: [{ kind: "mcp", server: "gitea-mcp", tool: "list_repo_issues" }], + finalReport: FINAL_REPORT, + stoppedByTurnCap: false, + }; + }, + }; + + const { host } = createFakeHost(); + const store = createSampleStore(storeRoot); + + const outcome = await runCell({ + arm: "gitea-mcp", + task: READ_TASK, + trial: 1, + access: ACCESS, + host, + driver: readDriver, + store, + bounds: { turnCap: 10, wallClockMs: 60_000 }, + build: { binRoot }, + }); + + expect(outcome.kind).toBe("recorded"); + if (outcome.kind !== "recorded") return; + + const samples = store.read({ arm: "gitea-mcp", taskId: READ_TASK.id }); + expect(samples).toHaveLength(1); + const [sample] = samples; + expect(sample).toBeDefined(); + if (sample === undefined) return; + + // A completed read cell: the checker scored the final report a pass. + expect(sample.outcome).toEqual({ pass: true }); + // The record carries the agent's final report verbatim. + expect(sample.report).toBe(FINAL_REPORT); + }); + + // Behavior: a completed mutation cell produces a record WITHOUT an agent report + // (benchmark-harness spec, record-assembly seam). A mutation task is scored by + // diffing repository state, not by reading the agent's words, so the recorded + // ResultRecord must carry no `report` field at all — absent, not present-but- + // undefined. SAMPLE_TASK is a mutation task; paired with the passing fake host + // and driver it drives a completed mutation PASS. The absence is the independent + // source of truth (the acceptance criterion), read back through the store after + // the JSON round-trip so an absent field is genuinely not a key. + it("records no agent report on a completed mutation cell", async () => { + const { host } = createFakeHost(); + const store = createSampleStore(storeRoot); + + const outcome = await runCell({ + arm: "gitea-mcp", + task: SAMPLE_TASK, + trial: 1, + access: ACCESS, + host, + driver, + store, + bounds: { turnCap: 10, wallClockMs: 60_000 }, + build: { binRoot }, + }); + + expect(outcome.kind).toBe("recorded"); + if (outcome.kind !== "recorded") return; + + const samples = store.read({ arm: "gitea-mcp", taskId: SAMPLE_TASK.id }); + expect(samples).toHaveLength(1); + const [sample] = samples; + expect(sample).toBeDefined(); + if (sample === undefined) return; + + // A completed mutation cell: the state diff scored a pass. + expect(sample.outcome).toEqual({ pass: true }); + // A mutation record carries no agent report — the key is absent entirely. + expect(sample).not.toHaveProperty("report"); + }); }); diff --git a/bench/runner.ts b/bench/runner.ts index afa3096..25caa3f 100644 --- a/bench/runner.ts +++ b/bench/runner.ts @@ -15,7 +15,7 @@ import { buildArm, type ArmDefinition, type BuildArmOptions, type SharedContext import { auditTranscript, type ToolUse } from "./audit.js"; import { score } from "./checker.js"; import type { Arm, Outcome, ResultRecord, TokenComponents } from "./result.js"; -import type { RepoState } from "./scoring-spec.js"; +import type { RepoState, ScoringSpec } from "./scoring-spec.js"; import type { BenchAccess, RepoCoords } from "./seed.js"; import type { SampleStore } from "./store.js"; import type { BenchTask } from "./task.js"; @@ -149,7 +149,10 @@ export async function runCell(input: RunCellInput): Promise { // A hung run produced no completed transcript to audit or score; record it // as a failure with no measured consumption. if (result.kind === "hung") { - return recorded(store, makeRecord(input, NO_TOKENS, 0, 0, durationMs, { pass: false, failure: "hung" }, clock)); + return recorded( + store, + makeRecord(input, NO_TOKENS, 0, 0, durationMs, { pass: false, failure: "hung" }, undefined, clock), + ); } const run = result.run; @@ -161,13 +164,19 @@ export async function runCell(input: RunCellInput): Promise { return { kind: "invalid", leaks: audit.leaks }; } + const spec = task.scoringSpec(coords.owner); const outcome = run.stoppedByTurnCap ? ({ pass: false, failure: "confused" } as const) - : await scoreRun(host, coords, task, run); + : await scoreRun(host, coords, spec, run); + + // Retain the agent's final report for read tasks so a failed read is + // diagnosable directly from the record; mutation tasks are scored by diffing + // repository state and have no agent report to record. + const report = spec.kind === "read" ? run.finalReport : undefined; return recorded( store, - makeRecord(input, run.tokens, run.turns, run.imputedCostUsd, durationMs, outcome, clock), + makeRecord(input, run.tokens, run.turns, run.imputedCostUsd, durationMs, outcome, report, clock), ); } finally { await host.delete(coords); @@ -214,8 +223,7 @@ async function runBounded( * final report against the required facts for a read. A pass is a pass; anything * the checker rejects is an incorrect failure. */ -async function scoreRun(host: BenchHost, coords: RepoCoords, task: BenchTask, run: AgentRun): Promise { - const spec = task.scoringSpec(coords.owner); +async function scoreRun(host: BenchHost, coords: RepoCoords, spec: ScoringSpec, run: AgentRun): Promise { const snapshot = await host.capture(coords); const check = spec.kind === "mutation" @@ -232,6 +240,7 @@ function makeRecord( imputedCostUsd: number, durationMs: number, outcome: Outcome, + report: string | undefined, clock: RunnerClock, ): ResultRecord { return { @@ -245,6 +254,9 @@ function makeRecord( durationMs, imputedCostUsd, outcome, + // Absent for mutation runs and runs with no completed report (hung); JSON + // serialization drops the key when undefined. + ...(report !== undefined ? { report } : {}), }; } diff --git a/bench/store.test.ts b/bench/store.test.ts index d6e6ada..b5b365a 100644 --- a/bench/store.test.ts +++ b/bench/store.test.ts @@ -102,4 +102,24 @@ describe("SampleStore", () => { secondRun.append(second); expect(secondRun.read(cell)).toEqual([first, second]); }); + + // Behavior: the store round-trips a report-bearing record without any change to + // the store itself. A read task's record carries the agent's final report in the + // `report` field; appending it and reading it back must return it equal, `report` + // and all, proving the store persists the field with no store change. The planted + // report string is an independent literal, not anything production code computes. + it("round-trips a record carrying a report field, preserving it unchanged", () => { + const store = createSampleStore(root); + const report = "There are 5 open issues in the repository."; + const record = sample({ tier: "read", report }); + + store.append(record); + + const readBack = store.read({ arm: record.arm, taskId: record.taskId }); + expect(readBack).toEqual([record]); + const [only] = readBack; + expect(only).toBeDefined(); + if (only === undefined) return; + expect(only.report).toBe(report); + }); }); -- 2.47.3 From 1166a481303714e16e66d445ffd36e65d2bf0dfa Mon Sep 17 00:00:00 2001 From: alexion Date: Fri, 17 Jul 2026 10:09:50 -0400 Subject: [PATCH 02/10] feat: name filtered state in issue list count line (task 0033) Make the `issue list` count line name the state it filtered on, so the answer to "how many issues are open?" is present on the summary line rather than only inferable from each row. The count line now renders `count: 5 open of 5 total`; the command composes the state into a generic optional qualifier while `formatCountLine` stays state-agnostic, so `pr list`, `search`, and `dashboard` are unaffected. `--state all` imposes no narrowing and stays unqualified. The wording is chosen so an agent quoting the summary lands on a phrase the benchmark read-checker already accepts (`5 open`), closing the accuracy gap this feature targets. Pairs with the report persistence in task 0032. --- .../0033-state-aware-issue-list-count-line.md | 29 ++++++++++++++++ src/commands/issue.ts | 10 +++++- src/render.ts | 12 +++++-- test/detection.test.ts | 4 +-- test/issue-list.test.ts | 34 +++++++++++++++---- 5 files changed, 77 insertions(+), 12 deletions(-) create mode 100644 .claude/tasks/0033-state-aware-issue-list-count-line.md diff --git a/.claude/tasks/0033-state-aware-issue-list-count-line.md b/.claude/tasks/0033-state-aware-issue-list-count-line.md new file mode 100644 index 0000000..ecc7dd6 --- /dev/null +++ b/.claude/tasks/0033-state-aware-issue-list-count-line.md @@ -0,0 +1,29 @@ +--- +spec: read-tier-accuracy +blocked-by: 0032-bench-read-report-persistence +--- + +## What to build + +Make the `issue list` count line name the state it filtered on, so the answer to "how many issues are open?" is present in the summary rather than only inferable from each row's state field. + +The command already resolves the effective state filter (defaulting to open), so it composes that state into its own count line and passes the composed line down; the generic count-line render helper stays generic and unaware of issue state. This keeps a single render seam and lets other list commands (`pr list`, `search`, `dashboard`) opt in on their own terms rather than inheriting the behavior. + +The count-line wording is chosen so that an agent quoting the summary lands on an answer the read checker already accepts (its accepted renderings include forms like `5 open`). After the change lands, a re-run of the read tier confirms `read-open-issue-count` moves from a consistent failure toward a pass — using the reports persisted by [[0032-bench-read-report-persistence]] to confirm the failure was a wrong/inferred count rather than a rejected phrasing before finalizing the wording. + +## Acceptance criteria + +- [x] The `issue list` count line names the state it counted for the default open filter. +- [x] The count line names the state for an explicit `--state` filter. +- [x] The generic count-line render helper is unchanged and remains state-agnostic; `pr list`, `search`, and `dashboard` output is unaffected. +- [x] Existing count-line invariants hold: a total is always reported, and the bare `count: N` form never appears. +- [x] New assertions extend the existing `issue list` command test at the fixture-server CLI seam, asserting exact rendered count-line strings. +- [-] A re-run of the read tier shows `read-open-issue-count` moving toward a pass, and the chosen wording contains a rendering the read checker already accepts. + +## Implementation Notes + +- The `issue list` count line now renders `count: of total` — e.g. `count: 5 open of 5 total`, `count: 1 closed of 1 total`. The command composes the state into the count line via a local `countStateQualifier(state)` helper; the generic `formatCountLine` gained only an optional, domain-agnostic `qualifier?: string` and never learns about issue state. Every other caller (`pr list`, `search`, `dashboard`, `label`, `issue`'s blocks/blocked-by list) passes no qualifier, so their output is byte-identical — the untouched command tests still pass, which is what criterion 3 really guards. +- **Criterion 3 wording.** Read literally, `formatCountLine` is not "unchanged" — it gained a parameter. But it stays *state-agnostic* (the qualifier is a bare string; state-to-qualifier mapping lives in the command), which is the spec's actual intent ("the generic render helper that formats count lines stays generic … rather than the helper learning about issue state"). The single render seam is preserved. Marked satisfied on that reading; the spec author may wish to reword the criterion from "unchanged" to "state-agnostic". +- **`--state all`.** Deliberately renders with no state word (`count: N of M total`): `all` imposes no narrowing filter and has no natural one-word name, so naming it adds no disambiguation. Open and closed — the filters where a bare count could mislead — are named, which is what resolves User Story 2's ambiguity. Pinned by a dedicated `--state all` test. +- **Criterion 6 (`[-]`, deferred not dropped).** The controllable half is done and verified: the chosen wording contains a checker-accepted rendering — running the real `checkReadAnswer` against the real `formatCountLine(5, 5, false, "open")` output (`count: 5 open of 5 total`) scores a pass on the `read-open-issue-count` fact, so an agent that merely echoes the summary now passes. The live read-tier re-run itself needs the benchmark environment (a live Gitea host + the Claude Agent SDK) and is left as a follow-up to run when the harness is next exercised; the report-persistence from [[0032-bench-read-report-persistence]] is now in place to confirm the movement from real report text. +- Reconciled the pre-existing count-line assertions that the format change made stale (five in `test/issue-list.test.ts`, two in `test/detection.test.ts`) to the state-named form; added a `--state all` guard test. diff --git a/src/commands/issue.ts b/src/commands/issue.ts index ebfe046..beacd9d 100644 --- a/src/commands/issue.ts +++ b/src/commands/issue.ts @@ -460,11 +460,19 @@ async function issueList(deps: CliDeps, args: string[]): Promise { return renderList({ noun: "issues", rows, - countLine: formatCountLine(rows.length, total, rows.length >= limit), + countLine: formatCountLine(rows.length, total, rows.length >= limit, countStateQualifier(state)), help: issueListSuggestions(context, state, rows.length, total), }); } +// The count line names the state the list was filtered to, so the answer to +// "how many are open?" is on the summary line rather than only inferable from +// each row. `all` imposes no narrowing and has no natural one-word name, so it +// adds no qualifier and the generic count line stands. +function countStateQualifier(state: IssueState): string | undefined { + return state === "all" ? undefined : state; +} + // The default detail fields reuse the same declarative extraction as the list // path; only `body` (truncation) and `comment_count` need bespoke handling. const ISSUE_VIEW_FIELDS: FieldDef[] = [ diff --git a/src/render.ts b/src/render.ts index ce2d5bf..6bc70f6 100644 --- a/src/render.ts +++ b/src/render.ts @@ -18,14 +18,20 @@ export function formatCountLine( shown: number, total: number | undefined, atLimit: boolean, + qualifier?: string, ): string { + // A generic, caller-supplied qualifier names what was counted (e.g. the state + // a list was filtered to) right after the count, so `count: 5 open of 5 total` + // answers "how many are open?" off the summary line. The helper stays unaware + // of any specific domain concept — callers that pass none render as before. + const counted = qualifier === undefined ? `${shown}` : `${shown} ${qualifier}`; if (total === undefined) { if (atLimit) { - return `count: ${shown} (showing first ${shown})`; + return `count: ${counted} (showing first ${shown})`; } - return `count: ${shown} of ${shown} total`; + return `count: ${counted} of ${shown} total`; } - return `count: ${shown} of ${total} total`; + return `count: ${counted} of ${total} total`; } /** Encode a named list block, with an explicit empty-state line when there are no rows. */ diff --git a/test/detection.test.ts b/test/detection.test.ts index 05018ff..dafdcb5 100644 --- a/test/detection.test.ts +++ b/test/detection.test.ts @@ -122,7 +122,7 @@ describe("repository context detection", () => { }); expect(exitCode).toBe(0); - expect(stdout).toContain("count: 3 of 3 total"); + expect(stdout).toContain("count: 3 open of 3 total"); expect(server!.requests[0]!.headers.authorization).toBe("Bearer detected-token"); // Auto-detected context: suggestions must not carry override flags. expect(stdout).not.toContain("-R testowner/testrepo"); @@ -143,7 +143,7 @@ describe("repository context detection", () => { }); expect(exitCode).toBe(0); - expect(stdout).toContain("count: 3 of 3 total"); + expect(stdout).toContain("count: 3 open of 3 total"); }); it("fails with REPO_NOT_FOUND when there is no recognizable origin remote", async () => { diff --git a/test/issue-list.test.ts b/test/issue-list.test.ts index d8ef559..edebb11 100644 --- a/test/issue-list.test.ts +++ b/test/issue-list.test.ts @@ -62,7 +62,7 @@ describe("issue list", () => { expect(exitCode).toBe(0); const lines = stdout.split("\n"); - expect(lines[0]).toBe("count: 3 of 17 total"); + expect(lines[0]).toBe("count: 3 open of 17 total"); expect(lines[1]).toBe("issues[3]{number,title,state,author,created}:"); expect(lines[2]).toMatch(/^ {2}42,"Fix login redirect loop, please",open,alexion,\d+(mo|[smhdy]) ago$/); expect(lines[3]).toMatch(/^ {2}41,Add dark mode,open,contributor,\d+(mo|[smhdy]) ago$/); @@ -105,7 +105,7 @@ describe("issue list", () => { ); expect(exitCode).toBe(0); - expect(stdout).toContain("count: 1 of 1 total"); + expect(stdout).toContain("count: 1 closed of 1 total"); expect(stdout).toContain("37,Crash on empty config,closed,contributor"); }); @@ -133,7 +133,7 @@ describe("issue list", () => { }); expect(exitCode).toBe(0); - expect(stdout).toContain("count: 0 of 0 total"); + expect(stdout).toContain("count: 0 open of 0 total"); expect(stdout).toContain("issues[0]: (none)"); expect(stdout).toMatch(/^help\[\d+\]:/m); }); @@ -209,6 +209,28 @@ describe("issue list", () => { expect(stdout).toContain("issues[3]{number,title,state,author,created}:"); expect(stdout).not.toContain("type"); }); + + it("names no state in the count line for --state all", async () => { + // `all` imposes no narrowing filter, so the count line stays the plain + // `count: of total` form with no state qualifier. + const issues = Array.from({ length: 9 }, (_, i) => issueOf(i + 1)); + server = await startFixtureServer([ + { + method: "GET", + path: ISSUES_PATH, + query: { state: "all" }, + headers: { "X-Total-Count": "9" }, + body: issues, + }, + ]); + const { stdout, exitCode } = await runCliTest( + ["issue", "list", "--state", "all"], + { env: testModeEnv(server.url) }, + ); + + expect(exitCode).toBe(0); + expect(stdout.split("\n")[0]).toBe("count: 9 of 9 total"); + }); }); describe("issue list filters", () => { @@ -375,7 +397,7 @@ describe("issue list --sort", () => { expect(renderedNumbers(stdout)[0]).toBe(7); // The count line keeps T from X-Total-Count: sorting reorders without // changing membership, so the unfiltered total stays accurate (ADR 0005). - expect(stdout).toContain("count: 30 of 52 total"); + expect(stdout).toContain("count: 30 open of 52 total"); }); it("applies --limit to the sorted order, not to the fetched pages", async () => { @@ -393,7 +415,7 @@ describe("issue list --sort", () => { ); expect(renderedNumbers(stdout)).toEqual([38, 42]); - expect(stdout).toContain("count: 2 of 17 total"); + expect(stdout).toContain("count: 2 open of 17 total"); // Pagination reads full pages regardless of --limit; the cap is applied after sorting. expect(server.requests[0]!.query.limit).toBe("50"); }); @@ -408,7 +430,7 @@ describe("issue list --sort", () => { env: testModeEnv(server.url), }); - expect(stdout).toContain("count: 2 of 3 total"); + expect(stdout).toContain("count: 2 open of 3 total"); }); it("stops at the page cap when a server keeps returning full pages", async () => { -- 2.47.3 From a6ab749211060ba567562d31cf825fae8a632275 Mon Sep 17 00:00:00 2001 From: alexion Date: Fri, 17 Jul 2026 10:46:45 -0400 Subject: [PATCH 03/10] fix: run the bench agent in a neutral working directory MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The SDK driver ran the agent with no explicit cwd, so its shell inherited the harness's own checkout. When the agent omitted `-R OWNER/NAME`, the gitea-axi (and tea) CLI defaulted the repository from that local checkout — silently resolving the harness repo instead of the seeded throwaway — and returned a plausible but wrong result (e.g. `count: 0 open of 0 total` for a repo with no issues). This contaminated read-tier scoring for the checkout- defaulting arms and was surfaced by the newly persisted read reports. Give each run a fresh, empty working directory outside any checkout, so a forgotten `-R` errors instead of hitting the wrong repository, and delete it when the run ends. --- bench/sdk-driver.test.ts | 32 ++++++++++++++++++++- bench/sdk-driver.ts | 60 ++++++++++++++++++++++++++++------------ 2 files changed, 74 insertions(+), 18 deletions(-) diff --git a/bench/sdk-driver.test.ts b/bench/sdk-driver.test.ts index 32170ed..f8521f8 100644 --- a/bench/sdk-driver.test.ts +++ b/bench/sdk-driver.test.ts @@ -1,5 +1,7 @@ +import { readdirSync, rmSync, statSync } from "node:fs"; +import path from "node:path"; import { describe, expect, it } from "vitest"; -import { sumTokens } from "./sdk-driver.js"; +import { createAgentWorkdir, sumTokens } from "./sdk-driver.js"; import type { SdkResultMessage } from "./sdk-driver.js"; describe("sumTokens", () => { @@ -87,3 +89,31 @@ describe("sumTokens", () => { }); }); }); + +describe("createAgentWorkdir", () => { + // Behavior: each agent run must operate in a fresh, empty directory located + // OUTSIDE the harness's own checkout. This isolation is what stops an agent + // that forgets an explicit `-R OWNER/NAME` from having its `gitea-axi`/`tea` + // tools silently default their target repo to the harness's own git checkout: + // a directory that is empty (no `.git`) and outside the current checkout gives + // those tools nothing local to resolve. + // + // The three assertions are independent anti-bug properties drawn directly from + // the requirement, not recomputed from the implementation: + // 1. the path exists and is a directory, + // 2. it is empty (zero entries — in particular no `.git`), + // 3. it sits outside the current working directory (relative path escapes + // upward with ".."). + it("returns a fresh, empty directory located outside the current checkout", () => { + const dir = createAgentWorkdir(); + try { + expect(statSync(dir).isDirectory()).toBe(true); + expect(readdirSync(dir)).toHaveLength(0); + expect(path.relative(process.cwd(), dir).startsWith("..")).toBe(true); + } finally { + // Safe: `dir` is a fresh throwaway temp dir we just received from + // createAgentWorkdir(); never a delete of cwd or any pre-existing path. + rmSync(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/bench/sdk-driver.ts b/bench/sdk-driver.ts index 5734e26..13c2702 100644 --- a/bench/sdk-driver.ts +++ b/bench/sdk-driver.ts @@ -17,6 +17,9 @@ // recorded in the transcript, so the runner's post-run audit sees what actually // executed — a blocked attempt is realistic wasted effort, not a leak. +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; import type { ArmDefinition } from "./arm.js"; import { foreignToolReason, type ToolUse } from "./audit.js"; import type { TokenComponents } from "./result.js"; @@ -104,6 +107,8 @@ interface SdkQueryOptions { abortController: AbortController; canUseTool: (toolName: string, input: Record) => Promise; settingSources: string[]; + /** The agent's shell working directory: a fresh empty dir outside any checkout. */ + cwd: string; env?: Record; mcpServers?: Record; disallowedTools?: string[]; @@ -197,30 +202,47 @@ export function sdkAgentDriver(config: SdkDriverConfig = {}): AgentDriver { return { behavior: "allow", updatedInput: toolInput }; }; - const options = buildOptions(input.arm, model, input.turnCap, controller, canUseTool); + const workdir = createAgentWorkdir(); + try { + const options = buildOptions(input.arm, model, input.turnCap, controller, canUseTool, workdir); - let result: SdkResultMessage | undefined; - for await (const message of query({ prompt: input.intent, options })) { - if (message.type === "result") { - result = message as SdkResultMessage; + let result: SdkResultMessage | undefined; + for await (const message of query({ prompt: input.intent, options })) { + if (message.type === "result") { + result = message as SdkResultMessage; + } + } + if (result === undefined) { + throw new Error("the Agent SDK produced no result message"); } - } - if (result === undefined) { - throw new Error("the Agent SDK produced no result message"); - } - return { - tokens: sumTokens(result), - turns: result.num_turns ?? 0, - imputedCostUsd: result.total_cost_usd ?? 0, - transcript, - finalReport: result.result ?? "", - stoppedByTurnCap: result.subtype === "error_max_turns", - }; + return { + tokens: sumTokens(result), + turns: result.num_turns ?? 0, + imputedCostUsd: result.total_cost_usd ?? 0, + transcript, + finalReport: result.result ?? "", + stoppedByTurnCap: result.subtype === "error_max_turns", + }; + } finally { + rmSync(workdir, { recursive: true, force: true }); + } }, }; } +/** + * Create a fresh, empty working directory for one agent run, outside any git + * checkout. The agent's shell runs here so a shell tool that defaults its target + * repository from the local checkout (gitea-axi, tea) cannot silently resolve the + * harness's own repository when the agent omits an explicit `-R`; with no ambient + * checkout the agent must target the repository named in its prompt. The caller + * deletes it when the run ends. + */ +export function createAgentWorkdir(): string { + return mkdtempSync(join(tmpdir(), "bench-agent-cwd-")); +} + /** Assemble the SDK query options for an arm's tool configuration. */ function buildOptions( arm: ArmDefinition, @@ -228,6 +250,7 @@ function buildOptions( turnCap: number, controller: AbortController, canUseTool: SdkQueryOptions["canUseTool"], + cwd: string, ): SdkQueryOptions { const options: SdkQueryOptions = { model, @@ -241,6 +264,9 @@ function buildOptions( // Start from a clean slate: no user/project settings leak tools or config // into the measured run. settingSources: [], + // Run outside any checkout so a forgotten -R cannot resolve the harness's own + // repository instead of the seeded throwaway (see createAgentWorkdir). + cwd, }; if (arm.shell !== null) { // Lead the agent's PATH with the arm's curated bin directory so only its one -- 2.47.3 From 05b72f69879780e3b5c8cfcdd9470f5e5507ae4a Mon Sep 17 00:00:00 2001 From: alexion Date: Fri, 17 Jul 2026 11:55:53 -0400 Subject: [PATCH 04/10] fix: ignore markdown emphasis when matching read answers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The read checker matched a task's required-fact phrasings as plain substrings of the agent's report after only lowercasing and collapsing whitespace. An answer that was substantively correct but wrapped a value in markdown (e.g. `**5**`) failed the match, because the emphasis markers broke the phrase adjacency (`**5** open` does not contain `5 open`) — a correct answer scored incorrect on formatting alone. Strip markdown emphasis/code markers (`*`, `_`, backtick) during normalization so the match is on substance, not presentation. A guard test confirms a wrong value still fails after stripping. --- bench/checker.test.ts | 30 ++++++++++++++++++++++++++++++ bench/checker.ts | 13 +++++++++++-- 2 files changed, 41 insertions(+), 2 deletions(-) diff --git a/bench/checker.test.ts b/bench/checker.test.ts index e6c3ed3..ddaebe7 100644 --- a/bench/checker.test.ts +++ b/bench/checker.test.ts @@ -310,6 +310,36 @@ describe("checkReadAnswer", () => { expect(checkReadAnswer(facts, report)).toEqual({ pass: true }); }); + + it("passes when the report wraps a fact's phrasing in markdown emphasis", () => { + // The fact's acceptable phrasing is the bare substring "5 open". + const facts: RequiredFact[] = [ + { description: "open-issue count", anyOf: ["5 open"] }, + ]; + + // The report is correct but renders the number in markdown bold. Emphasis + // markers (`*`, `_`, backtick) are formatting, not substance, so "**5**" is + // equivalent to "5" and the phrasing "5 open" is present. + const report = "There are **5** open issues in the repository."; + + expect(checkReadAnswer(facts, report)).toEqual({ pass: true }); + }); + + it("fails when a markdown-formatted report states the wrong value", () => { + // The only acceptable phrasing counts five open issues. + const facts: RequiredFact[] = [ + { description: "open-issue count", anyOf: ["5 open"] }, + ]; + + // The report is markdown-formatted but substantively wrong: it counts three, + // not five. Stripping emphasis must only remove formatting, so after stripping + // ("there are 3 open issues") the phrasing "5 open" is still absent. + const report = "There are **3** open issues in the repository."; + + const result = checkReadAnswer(facts, report); + + expect(result.pass).toBe(false); + }); }); describe("score", () => { diff --git a/bench/checker.ts b/bench/checker.ts index 082d5e2..c51a217 100644 --- a/bench/checker.ts +++ b/bench/checker.ts @@ -253,9 +253,18 @@ export function checkReadAnswer(facts: RequiredFact[], report: string): CheckRes }; } -/** Lower-case and collapse runs of whitespace so incidental phrasing does not matter. */ +/** + * Lower-case, drop markdown emphasis/code markers, and collapse runs of + * whitespace so incidental phrasing and formatting do not matter — a report that + * bolds a value (`**5**`) matches a phrasing that does not (`5 open`), since the + * emphasis is presentation, not substance. + */ function normalizeText(text: string): string { - return text.toLowerCase().replace(/\s+/g, " ").trim(); + return text + .toLowerCase() + .replace(/[*_`]/g, "") + .replace(/\s+/g, " ") + .trim(); } /** -- 2.47.3 From 14d494afa2dc6ccda8e5f9aa81e91f69b748f0dc Mon Sep 17 00:00:00 2001 From: alexion Date: Fri, 17 Jul 2026 14:43:43 -0400 Subject: [PATCH 05/10] feat: persist the tool transcript on every result record MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records stored only a run's token/turn totals, so an arm's turn cost — the dominant driver of cache-read tokens — could not be diagnosed from the store. Retain the ordered transcript of tool invocations (the exact shell commands, MCP calls, and built-in tools the run made) on every scored record, absent only for a hung run that produced no transcript. The canonical TranscriptEntry shape lives on the record (result.ts); the isolation audit's ToolUse now aliases it so the persisted and audited shapes cannot drift. --- bench/audit.ts | 10 +++++----- bench/result.ts | 22 ++++++++++++++++++++++ bench/runner.test.ts | 6 ++++++ bench/runner.ts | 10 +++++++--- 4 files changed, 40 insertions(+), 8 deletions(-) diff --git a/bench/audit.ts b/bench/audit.ts index c2e86f1..7e28876 100644 --- a/bench/audit.ts +++ b/bench/audit.ts @@ -12,17 +12,17 @@ // runner (runner.ts) drives the run and feeds the transcript here. import type { ArmDefinition } from "./arm.js"; +import type { TranscriptEntry } from "./result.js"; /** * One tool invocation recorded in the agent's transcript, reduced to what the * isolation audit needs. `shell` is a proposed shell command; `mcp` is a call to * an attached MCP server's tool; `other` is a built-in, non-Gitea-reaching tool - * (file read/edit and the like) that carries no isolation risk. + * (file read/edit and the like) that carries no isolation risk. This is the same + * shape the record persists ({@link TranscriptEntry}); the audit and the record + * share one type so they cannot drift. */ -export type ToolUse = - | { kind: "shell"; command: string } - | { kind: "mcp"; server: string; tool: string } - | { kind: "other"; name: string }; +export type ToolUse = TranscriptEntry; /** * The audit's verdict. On a leak it carries a human-readable reason per foreign diff --git a/bench/result.ts b/bench/result.ts index 6327320..9f0639d 100644 --- a/bench/result.ts +++ b/bench/result.ts @@ -45,6 +45,19 @@ export type FailureTag = "incorrect" | "confused" | "hung"; /** The pass/fail outcome of a run, tagged with the failure mode when it fails. */ export type Outcome = { pass: true } | { pass: false; failure: FailureTag }; +/** + * One tool invocation as it is recorded in a run's transcript, in the order it + * executed. This is the canonical shape the harness both audits for isolation + * (see audit.ts, whose `ToolUse` aliases this) and persists on the record for + * diagnosis. A `shell` entry keeps the exact command line the agent ran; an `mcp` + * entry names the server and tool it called; `other` names a built-in tool that + * reaches no Gitea channel. + */ +export type TranscriptEntry = + | { kind: "shell"; command: string } + | { kind: "mcp"; server: string; tool: string } + | { kind: "other"; name: string }; + /** * One completed `(arm, task, trial)` run. Carries the metrics the headline and * supporting views are computed from, plus the tags those views group by. @@ -81,6 +94,15 @@ export interface ResultRecord { * scored by diffing repository state and have no agent report to record. */ report?: string; + + /** + * The ordered transcript of tool invocations the run made, retained on every + * scored run so its turn cost is diagnosable directly from the record — the + * exact command sequence, which is how an arm's turn count (the dominant driver + * of cache-read tokens) is explained. Absent only for a hung run, which + * produced no completed transcript to record. + */ + transcript?: TranscriptEntry[]; } /** diff --git a/bench/runner.test.ts b/bench/runner.test.ts index 4747354..f4cda06 100644 --- a/bench/runner.test.ts +++ b/bench/runner.test.ts @@ -241,6 +241,12 @@ describe("runCell", () => { expect(sample.imputedCostUsd).toBe(DRIVER_COST); expect(sample.outcome).toEqual({ pass: true }); + // The recorded sample carries the run's tool transcript — the exact ordered + // sequence of tool invocations the driver reported — so the turn's cost is + // diagnosable directly from the record. The expected value is the literal the + // fake driver planted, deep-equal and in order, not recomputed from runner.ts. + expect(sample.transcript).toEqual([{ kind: "mcp", server: "gitea-mcp", tool: "edit_issue" }]); + // The sample carries the run's wall-clock duration; a completed run takes // non-negative time. expect(typeof sample.durationMs).toBe("number"); diff --git a/bench/runner.ts b/bench/runner.ts index 25caa3f..3e7ff81 100644 --- a/bench/runner.ts +++ b/bench/runner.ts @@ -14,7 +14,7 @@ import { buildArm, type ArmDefinition, type BuildArmOptions, type SharedContext } from "./arm.js"; import { auditTranscript, type ToolUse } from "./audit.js"; import { score } from "./checker.js"; -import type { Arm, Outcome, ResultRecord, TokenComponents } from "./result.js"; +import type { Arm, Outcome, ResultRecord, TokenComponents, TranscriptEntry } from "./result.js"; import type { RepoState, ScoringSpec } from "./scoring-spec.js"; import type { BenchAccess, RepoCoords } from "./seed.js"; import type { SampleStore } from "./store.js"; @@ -151,7 +151,7 @@ export async function runCell(input: RunCellInput): Promise { if (result.kind === "hung") { return recorded( store, - makeRecord(input, NO_TOKENS, 0, 0, durationMs, { pass: false, failure: "hung" }, undefined, clock), + makeRecord(input, NO_TOKENS, 0, 0, durationMs, { pass: false, failure: "hung" }, undefined, undefined, clock), ); } @@ -176,7 +176,7 @@ export async function runCell(input: RunCellInput): Promise { return recorded( store, - makeRecord(input, run.tokens, run.turns, run.imputedCostUsd, durationMs, outcome, report, clock), + makeRecord(input, run.tokens, run.turns, run.imputedCostUsd, durationMs, outcome, report, run.transcript, clock), ); } finally { await host.delete(coords); @@ -241,6 +241,7 @@ function makeRecord( durationMs: number, outcome: Outcome, report: string | undefined, + transcript: TranscriptEntry[] | undefined, clock: RunnerClock, ): ResultRecord { return { @@ -257,6 +258,9 @@ function makeRecord( // Absent for mutation runs and runs with no completed report (hung); JSON // serialization drops the key when undefined. ...(report !== undefined ? { report } : {}), + // Absent only for a hung run, which produced no transcript; JSON + // serialization drops the key when undefined. + ...(transcript !== undefined ? { transcript } : {}), }; } -- 2.47.3 From ab59e699c10bf7d1fba483b74a766450ffe01b1a Mon Sep 17 00:00:00 2001 From: alexion Date: Fri, 17 Jul 2026 15:10:48 -0400 Subject: [PATCH 06/10] fix: pre-authenticate the gitea-axi bench arm via its env interface MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gitea-axi arm was the only shell arm handed no credentials: the runner set only PATH, so the agent had to reverse-engineer the tea-login system — guessing a profile name and hunting for a config file — before any real work, burning ~4 turns per task. Since turns drive cache-read, the benchmark's dominant cost metric, this scaffolding gap alone inflated gitea-axi's cost-equivalent tokens above every other arm. Hand the arm its host and token through gitea-axi's own env interface (GITEA_AXI_API_URL / GITEA_AXI_TOKEN), the symmetric counterpart to the gitea-mcp server's GITEA_HOST / GITEA_ACCESS_TOKEN env: both name the same two facts, and both still leave the agent to name the repository per call. A shell arm now carries a credential env (empty for tea and raw-api, which need none), merged under PATH in the driver. Also strengthen SKILL.md so a cold agent targets and authenticates on the first call: an explicit "Targeting and authentication" section replaces the buried, optional-looking one-liner, spelling out that outside a checkout `-R OWNER/NAME` plus the environment's token is all that is needed — do not go hunting for a config file or login profile. Verified live: create-memory-leak-issue dropped from 10 turns to 3 and its cache-read fell ~3.8x, with the auth flailing gone from the transcript. --- bench/arm.test.ts | 26 ++++++++++++++++++++++++++ bench/arm.ts | 36 ++++++++++++++++++++++++++++++++++-- bench/sdk-driver.ts | 5 ++++- skills/gitea-axi/SKILL.md | 12 +++++++++++- 4 files changed, 75 insertions(+), 4 deletions(-) diff --git a/bench/arm.test.ts b/bench/arm.test.ts index c397ec8..b932b82 100644 --- a/bench/arm.test.ts +++ b/bench/arm.test.ts @@ -112,6 +112,32 @@ describe("buildArm", () => { expect(envValues).toContain("s3cr3t-token"); }); + // Behavior: the gitea-axi arm's shell is handed a credential environment + // carrying the host and token from the shared access, so its tool is + // pre-authenticated without the agent having to discover credentials + // (benchmark-harness spec, "Scaffolding"). The access here is built from + // independent literals; shell.env must deep-equal exactly the two facts echoed + // back under their env-var names (host→GITEA_AXI_API_URL, token→GITEA_AXI_TOKEN), + // and nothing else. The keys and mapping are fixed by gitea-axi's own env + // contract, not recomputed from arm.ts. + it("gives the gitea-axi arm's shell a credential env with the shared host URL and token", () => { + const preAuthed: SharedContext = { + coords: { owner: "acme", repo: "bench-xyz" }, + access: { apiUrl: "https://git.example.test", token: "tok-abc123" }, + }; + + const definition = buildArm("gitea-axi", preAuthed, { binRoot, locate }); + + const shell = definition.shell; + expect(shell).not.toBeNull(); + if (shell === null) return; + + expect(shell.env).toEqual({ + GITEA_AXI_API_URL: "https://git.example.test", + GITEA_AXI_TOKEN: "tok-abc123", + }); + }); + // Behavior: each non-MCP arm's tool/PATH configuration comes from the guard and // exposes only that arm's allowed binary (benchmark-harness spec, "Tool // isolation" / ADR 0016). The (arm, binary) pairs are independent literals — diff --git a/bench/arm.ts b/bench/arm.ts index a952803..9b0507d 100644 --- a/bench/arm.ts +++ b/bench/arm.ts @@ -40,6 +40,17 @@ export interface ArmShell { path: string; /** The authoritative tool-isolation guard, bound to this arm. */ guard: (command: string) => GuardDecision; + /** + * Credential environment the arm's tool is pre-configured with, merged into + * the agent's shell environment on top of {@link path}. This keeps the arms + * symmetric on authentication: every arm is handed its host and token the way + * its product is really configured, so none pays a turn tax rediscovering how + * to authenticate. The gitea-mcp arm gets the equivalent through its MCP + * server's env; raw-api uses the token stated in its prompt directly; the + * gitea-axi arm is configured through its own env interface here. Empty for an + * arm that needs no ambient credentials. + */ + env: Record; } /** @@ -172,7 +183,7 @@ function mcpAttachment(context: SharedContext): ArmMcp { * gitea-mcp arm has no shell binary (`provisionArmBin` exposes nothing for it), * so this returns null there and the arm reaches Gitea through its MCP tools. */ -function buildShell(arm: Arm, options: BuildArmOptions): ArmShell | null { +function buildShell(arm: Arm, context: SharedContext, options: BuildArmOptions): ArmShell | null { if (arm === "gitea-mcp") { return null; } @@ -183,16 +194,37 @@ function buildShell(arm: Arm, options: BuildArmOptions): ArmShell | null { binDir, path: ambient === "" ? binDir : `${binDir}${delimiter}${ambient}`, guard: (command) => guardCommand(arm, command), + env: shellEnv(arm, context), }; } +/** + * The credential environment a shell arm's tool is pre-configured with. The + * gitea-axi arm is handed its host and token through its own env interface + * (`GITEA_AXI_API_URL` / `GITEA_AXI_TOKEN`), the symmetric counterpart to the + * gitea-mcp arm's server env: both name the same host and token, and both leave + * the agent to name the repository per call (gitea-axi via `-R`, gitea-mcp via + * each tool's arguments). The tea and raw-api arms need no ambient credentials — + * raw-api uses the token stated in its prompt directly in each request, and tea + * resolves its own login store — so their env is empty. + */ +function shellEnv(arm: Arm, context: SharedContext): Record { + if (arm === "gitea-axi") { + return { + GITEA_AXI_API_URL: context.access.apiUrl, + GITEA_AXI_TOKEN: context.access.token, + }; + } + return {}; +} + /** Assemble the single arm definition the runner consumes for the given arm. */ export function buildArm(arm: Arm, context: SharedContext, options: BuildArmOptions): ArmDefinition { const systemPrompt = `${basePrompt(context)}\n\n${armBootstrap(arm, context, options)}`; return { arm, systemPrompt, - shell: buildShell(arm, options), + shell: buildShell(arm, context, options), mcp: arm === "gitea-mcp" ? mcpAttachment(context) : null, }; } diff --git a/bench/sdk-driver.ts b/bench/sdk-driver.ts index 13c2702..30ddb6d 100644 --- a/bench/sdk-driver.ts +++ b/bench/sdk-driver.ts @@ -271,7 +271,10 @@ function buildOptions( if (arm.shell !== null) { // Lead the agent's PATH with the arm's curated bin directory so only its one // allowed binary resolves by name; the guard on canUseTool is the authority. - options.env = { ...process.env, PATH: arm.shell.path }; + // Layer the arm's credential env underneath so its tool is pre-authenticated + // the way its product is really configured, symmetric to the gitea-mcp + // server's env (see ArmShell.env); PATH stays last so it is never overridden. + options.env = { ...process.env, ...arm.shell.env, PATH: arm.shell.path }; } if (arm.mcp !== null) { options.mcpServers = { [arm.arm]: { type: "stdio", ...arm.mcp.server } }; diff --git a/skills/gitea-axi/SKILL.md b/skills/gitea-axi/SKILL.md index fcbf6a9..085086d 100644 --- a/skills/gitea-axi/SKILL.md +++ b/skills/gitea-axi/SKILL.md @@ -16,7 +16,17 @@ Reach for `gitea-axi` whenever a task touches a Gitea repository's issues, pull - **Over raw Gitea API calls:** it handles auth, pagination, name-to-ID resolution, and review-decision aggregation for you, so you do not hand-roll HTTP. - **Over improvised `git`:** for anything about issues or pull requests as entities (state, reviews, labels, comments) rather than local commits and branches. -Run it inside a Gitea checkout, or pass `-R OWNER/NAME` (and `--login `) to target a repository explicitly. +## Targeting and authentication + +Every command resolves two things: which repository to act on, and which credentials to authenticate with. +Get both right on the first call — they are the usual reason a command fails and has to be retried. + +- **Repository.** Inside a Gitea checkout it is taken from the `origin` remote automatically. + Outside a checkout you must name it: pass `-R OWNER/NAME` on every command (or set `GITEA_AXI_REPO=OWNER/NAME` once for the session). +- **Credentials.** When the environment is pre-configured — `GITEA_AXI_TOKEN` together with `GITEA_AXI_API_URL` — authentication is automatic and you need nothing more. + Otherwise credentials come from a `tea` login: pass `--login ` (or set `GITEA_AXI_LOGIN=`) unless the checkout's remote already selects one. + +So outside a checkout with the token in the environment, `gitea-axi -R OWNER/NAME …` is all you need; do not go hunting for a config file or a login profile. ## Command groups -- 2.47.3 From 8653b896128fe3e638b742a25e6db854c0f39879 Mon Sep 17 00:00:00 2001 From: alexion Date: Fri, 17 Jul 2026 18:59:16 -0400 Subject: [PATCH 07/10] feat: show issue labels in `issue view` and add its --fields flag MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `issue view` rendered state but never labels, and offered no way to add them — so reading one issue's labels forced a detour through `issue list --fields labels` and hunting the matching row. The benchmark transcripts showed agents paying this round-trip on every labels/state read. Show labels by default in the detail view (a detail view should be complete), and add a `--fields` flag mirroring `issue list` / `search` to append assignees, closedAt, milestone, updatedAt, url on request. Also strengthen SKILL.md against the two command-discovery round-trips the transcripts exposed: name the required `search issues` / `search prs` subcommand form (a bare `search ""` is invalid), and point agents straight at `issue view ` for a single issue's fields. Verified live: read-issue-labels-and-state dropped from 10 turns to 4 (cache-read ~3.3x lower), the transcript reduced to three clean commands with the search-help and issue-list round-trips gone. --- skills/gitea-axi/SKILL.md | 5 ++++- src/commands/issue.ts | 31 +++++++++++++++++++++++++++--- test/issue-view.test.ts | 40 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 72 insertions(+), 4 deletions(-) diff --git a/skills/gitea-axi/SKILL.md b/skills/gitea-axi/SKILL.md index 085086d..5e30a48 100644 --- a/skills/gitea-axi/SKILL.md +++ b/skills/gitea-axi/SKILL.md @@ -33,9 +33,12 @@ So outside a checkout with the token in the environment, `gitea-axi -R - `issue` — list, view, create, comment on, edit, close/reopen, pin, and link issues. - `pr` — create, view, comment on, edit, review, merge, check out, diff, and inspect the checks of pull requests. - `label` — list, create, edit, and delete labels. -- `search` — full-text search across issues and pull requests. +- `search` — full-text search; it takes a subcommand, so search issues with `search issues ""` and pull requests with `search prs ""` (a bare `search ""` is not valid). - `setup` — install this skill (`setup`) and, opt-in, the SessionStart dashboard hook (`setup hooks`). +To read one issue's fields, reach straight for `issue view `: it shows labels and state by default, and takes `--fields assignees,milestone,…` for the rest. +You rarely need `issue list` to answer a question about a single issue. + ## Discovery This skill is a pointer, not a command reference — the CLI is the single source of truth for its own interface. diff --git a/src/commands/issue.ts b/src/commands/issue.ts index beacd9d..680f9f4 100644 --- a/src/commands/issue.ts +++ b/src/commands/issue.ts @@ -222,8 +222,11 @@ Show a single issue. Pull request numbers are rejected — use \`pr view\` inste flags: --comments Render every comment in full (bodies truncated at 800 chars) --full Suppress all truncation of the issue body and comment bodies + --fields Append extra fields: assignees, closedAt, milestone, updatedAt, url --help Show this help +Labels are shown by default; use --fields to add assignees, milestone, and more. + global flags: -R, --repo Override the repository detected from the git origin remote --login Select a tea login profile by name @@ -479,19 +482,32 @@ const ISSUE_VIEW_FIELDS: FieldDef[] = [ pluck("number"), pluck("title"), lowercased("state"), + joined("labels", "labels", "name"), pluck("author", "user.login"), relativeTimeField("created", "created_at"), ]; +// Appended to the default view fields on request via `--fields`, never replacing +// them. Labels and body are shown by default, so they are not offered here. +const ISSUE_VIEW_EXTRA_FIELDS: Record> = { + assignees: joined("assignees", "assignees", "login"), + closedAt: relativeTimeField("closedAt", "closed_at"), + milestone: pluck("milestone", "milestone.title"), + updatedAt: relativeTimeField("updatedAt", "updated_at"), + url: pluck("url", "html_url"), +}; + interface IssueDetailOptions { host: string; full: boolean; withComments: boolean; now: Date; + /** Extra fields selected via `--fields`, appended after the defaults. */ + extraFields: FieldDef[]; } function buildIssueDetail(issue: Issue, options: IssueDetailOptions): Record { - const row = extractRow(issue, ISSUE_VIEW_FIELDS, { + const row = extractRow(issue, [...ISSUE_VIEW_FIELDS, ...options.extraFields], { now: options.now, host: options.host, full: options.full, @@ -539,12 +555,21 @@ async function issueView(deps: CliDeps, args: string[]): Promise { } const { flags, positionals } = parseFlags( args, - { "--comments": { takesValue: false }, "--full": { takesValue: false } }, + { + "--comments": { takesValue: false }, + "--full": { takesValue: false }, + "--fields": { takesValue: true }, + }, "issue view", ); const number = parsePositionalNumber(positionals, "issue view", "issue"); const full = flags["--full"] === true; const withComments = flags["--comments"] === true; + const extraFields = selectExtraFields( + flagValue(flags, "--fields"), + ISSUE_VIEW_EXTRA_FIELDS, + "issue view", + ); const context = await resolveRepoContext(deps); const api = createClient(context); @@ -557,7 +582,7 @@ async function issueView(deps: CliDeps, args: string[]): Promise { } const now = new Date(); - const item = buildIssueDetail(issue, { host: context.host, full, withComments, now }); + const item = buildIssueDetail(issue, { host: context.host, full, withComments, now, extraFields }); const blocks: DetailBlock[] = []; if (withComments) { diff --git a/test/issue-view.test.ts b/test/issue-view.test.ts index afdb614..36eed88 100644 --- a/test/issue-view.test.ts +++ b/test/issue-view.test.ts @@ -47,6 +47,46 @@ describe("issue view", () => { expect(stdout).toContain("comment_count: 3 — use --comments to see full comments"); }); + it("renders the issue's labels comma-joined by default, with no flag", async () => { + server = await startFixtureServer([ + { + method: "GET", + path: ISSUE_PATH, + body: issueBody({ labels: [{ name: "bug" }, { name: "regression" }] }), + }, + ]); + const { stdout, exitCode } = await runCliTest(["issue", "view", "42"], { + env: testModeEnv(server.url), + }); + + expect(exitCode).toBe(0); + // TOON-quoted because the joined value contains a comma. + expect(stdout).toContain('labels: "bug, regression"'); + }); + + it("appends named extra fields with --fields on top of the default fields", async () => { + server = await startFixtureServer([ + { + method: "GET", + path: ISSUE_PATH, + body: issueBody({ + assignees: [{ login: "alexion" }], + milestone: { title: "v2.0" }, + }), + }, + ]); + const { stdout, exitCode } = await runCliTest( + ["issue", "view", "42", "--fields", "assignees,milestone"], + { env: testModeEnv(server.url) }, + ); + + expect(exitCode).toBe(0); + // Default fields are still present; the extra fields are appended. + expect(stdout).toContain("state: open"); + expect(stdout).toContain("assignees: alexion"); + expect(stdout).toContain("milestone: v2.0"); + }); + it("renders comment_count: 0 when there are no comments", async () => { server = await startFixtureServer([ { method: "GET", path: ISSUE_PATH, body: issueBody({ comments: 0 }) }, -- 2.47.3 From 6e6503294499160d42cb4a9d22de2a89e6ed85db Mon Sep 17 00:00:00 2001 From: alexion Date: Fri, 17 Jul 2026 18:59:48 -0400 Subject: [PATCH 08/10] docs: note the bench runs built dist, not src, in CLAUDE.md gotchas --- CLAUDE.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index 7ee913c..0dce006 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -9,6 +9,9 @@ Any commit message you write must follow the Conventional Commits specification The `origin` remote is a self-hosted **Gitea** instance (`git.alexion.dev`), not GitHub. The `gh` CLI does not work here. +The benchmark arms invoke the **built `dist/main.js`** (the `gitea-axi` binary on `PATH`), not the TypeScript source. +Run `npm run build` before any live `bench:run` if you want `src/` changes reflected; the bench does not run from source. + Prefer this project's own CLI for pull requests — it is the tool being built, so opening its PRs with it is the dogfood path: `npm run build && node dist/main.js pr create --login alexion --base main --head --title --body-file `. It reuses the `tea` login profiles, so it needs no separate credentials. -- 2.47.3 From 040daed39ddca386e4eadad59269e79a844afa75 Mon Sep 17 00:00:00 2001 From: alexion Date: Fri, 17 Jul 2026 22:35:40 -0400 Subject: [PATCH 09/10] docs: rewrite bench results for the clean 4-arm snapshot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The prior results table and narrative claimed gitea-axi posts the lowest cost-equivalent tokens. That snapshot predated the neutral-working-dir isolation fix, when the checkout-defaulting arms (gitea-axi, tea) drew repo and login for free from the harness's own checkout — so gitea-axi was implicitly pre-authenticated and looked like the winner. On a clean run with every arm fairly credentialed and executed together, raw REST is the cheapest on cost-equivalent tokens and leads every tier — terse HTTP is the token floor no wrapper undercuts. gitea-axi is a clear second overall and the lowest-cost structured interface, beating gitea-mcp and tea on every tier at 100% success. Keep the raw-REST arm and state this plainly rather than crown the wrapper by omitting the floor. Numbers regenerated from bench:report over the 240-sample clean snapshot. --- bench/README.md | 35 ++++++++++++++++++++++------------- 1 file changed, 22 insertions(+), 13 deletions(-) diff --git a/bench/README.md b/bench/README.md index 6fc5c87..b9f1341 100644 --- a/bench/README.md +++ b/bench/README.md @@ -1,28 +1,37 @@ # Benchmark harness This directory holds the benchmark that tests gitea-axi's central claim — that it is an agent-ergonomic, low-token interface to Gitea — against the `tea` CLI, the official `gitea-mcp` server, and raw Gitea REST calls. -This run bears that out on cost: gitea-axi posts the lowest cost-equivalent tokens and the lowest imputed cost of the four tools, though `gitea-mcp` edges it slightly on accuracy. +The result is honest rather than flattering: gitea-axi is the lowest-cost of the *structured* interfaces — it beats both `tea` and `gitea-mcp` on every tier at 100% task success — but hand-rolled raw REST is cheaper still, because terse HTTP is the token floor no wrapper undercuts. +Keeping the raw-REST arm in the comparison is deliberate: a benchmark of agent-CLIs that omits it will always crown the wrapper, and this one refuses to. ## How it works Each arm is an agent given exactly one of the four tools and nothing else, run on the same fixed model at temperature zero, so the comparison measures the tool rather than the model. The suite is 20 tasks across four tiers — read, single-mutation, find-then-act, and multi-step — each run against a freshly seeded throwaway repository and scored deterministically by diffing the resulting repository state (or matching required facts in the agent's answer) against the seeded ground truth. The headline metric is cost-equivalent tokens: the four token components (fresh input, cache write, cache read, output) weighted by Anthropic's published API pricing ratios, which is why an arm can spend more raw tokens yet cost less. +Every arm is credentialed the way its product is really configured — the token in its environment (`gitea-axi`, `gitea-mcp`) or in its prompt (`raw-api`), and a `tea` login for `tea` — so no arm pays a turn tax rediscovering how to authenticate. ## Results -| arm | cost-equivalent tokens | raw tokens | success | imputed cost | -| --- | ---: | ---: | ---: | ---: | -| gitea-axi | 16,921 | 68,093 | 95% | $6.20 | -| raw-api | 17,773 | 55,631 | 95% | $6.64 | -| gitea-mcp | 17,898 | 60,028 | 97% | $6.82 | -| tea | 20,505 | 80,702 | 90% | $7.25 | +| arm | cost-equivalent tokens | raw tokens | turns | success | imputed cost | +| --- | ---: | ---: | ---: | ---: | ---: | +| raw-api | 16,971 | 52,586 | 4.3 | 100% | ~$0.11 | +| gitea-axi | 19,240 | 81,067 | 6.0 | 100% | ~$0.12 | +| tea | 20,568 | 82,188 | 6.2 | 97% | ~$0.12 | +| gitea-mcp | 21,803 | 79,961 | 5.7 | 100% | ~$0.14 | -All four arms completed the full matrix — 20 of 20 tasks each, at the reporting floor. -gitea-axi wins on cost-equivalent tokens and on real imputed cost even though it does not use the fewest raw tokens: its interactions are output-light, and output is the most expensive component (weighted 5×), so its compact answers beat arms that emit more. -gitea-mcp is the most accurate at 97% against gitea-axi's 95%, so the two leaders trade a small accuracy edge for a clear cost lead. +All four arms completed the full matrix — 20 of 20 tasks each, at the reporting floor — and success is near-perfect: only `tea` slips, to 89% on find-then-act, while the other three pass every run. -By tier, the read tasks are the hardest for every arm (75–83% success) — exact-answer reads, not mutations, are where correctness slips. -tea is the outlier on find-then-act, dropping to 78% success at about 1.7× the cost-equivalent tokens of the other three arms. +Raw REST posts the lowest cost-equivalent tokens and leads every tier. +It is direct HTTP with the token in the request header, so it takes the fewest turns (4.3) and reads the least cached context, and no higher-level tool beats that on tokens alone. +This is the honest ceiling, and the reason gitea-axi does not claim the cost crown outright. -_Snapshot: 2026-07-17 — 4 arms × 20 tasks × 3 trials each (240 samples), a single run against one live Gitea host; imputed cost is Anthropic API-priced._ +gitea-axi is a clear second overall and the cheapest of the structured tools: it undercuts the official `gitea-mcp` server and the `tea` CLI on every tier, at 100% success, with the lowest output-token count of any arm. +Note the split between raw and cost-equivalent tokens — gitea-axi spends more raw tokens than `gitea-mcp` yet costs less, because output is weighted 5× and gitea-axi's answers are compact. + +By tier, raw REST's edge is widest on reads (10,921 vs gitea-axi's 14,415) — a read is one HTTP request for curl, where a CLI still spends a turn or two — and narrows on multi-step (24,348 vs 26,963), where the work itself dominates and interface overhead matters less. + +Cost parity on the scored suite also understates gitea-axi, because the suite is the subset every arm can do at all. +The bonus table records capability-asymmetric operations — full-text issue search, rendering a PR's diff and checks, issue dependencies — that gitea-axi handles directly and raw REST has no first-class equivalent for. + +_Snapshot: 2026-07-17 — 4 arms × 20 tasks × 3 trials each (240 samples), a single clean run with all four arms executed together against one live Gitea host; imputed cost is the mean per-task Anthropic-API-priced dollar cost._ -- 2.47.3 From a557745e59ce9ef106fb62529a9fb1503663da78 Mon Sep 17 00:00:00 2001 From: alexion Date: Fri, 17 Jul 2026 22:52:13 -0400 Subject: [PATCH 10/10] test: update e2e tracer count-line assertions for task 0033 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The state-aware count line from task 0033 (1166a48) renders `count: N open of M total`, and the unit tests were updated to match, but the three e2e tracer assertions still expected the old bare `count: N of M total`. They are skipped without GITEA_AXI_E2E_URL, so the staleness only surfaced in CI, where the e2e tier runs. Update them to the state-qualified form the shipped code already produces — default `open`, `--state closed` → `closed`. The code was correct; the tests were stale. --- test/e2e/tracer.test.ts | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/test/e2e/tracer.test.ts b/test/e2e/tracer.test.ts index 5add944..edd1aac 100644 --- a/test/e2e/tracer.test.ts +++ b/test/e2e/tracer.test.ts @@ -40,7 +40,7 @@ describe.skipIf(!E2E_URL)("end-to-end: tracer command set", () => { expect(exitCode).toBe(0); const lines = stdout.split("\n"); - expect(lines[0]).toBe("count: 3 of 3 total"); + expect(lines[0]).toBe("count: 3 open of 3 total"); expect(lines[1]).toBe("issues[3]{number,title,state,author,created}:"); for (const title of instance.openTitles) { expect(stdout).toContain(title); @@ -58,7 +58,7 @@ describe.skipIf(!E2E_URL)("end-to-end: tracer command set", () => { }); expect(exitCode).toBe(0); - expect(stdout).toContain("count: 1 of 1 total"); + expect(stdout).toContain("count: 1 closed of 1 total"); expect(stdout).toContain(instance.closedTitle); expect(stdout).toContain(",closed,"); }); @@ -69,7 +69,7 @@ describe.skipIf(!E2E_URL)("end-to-end: tracer command set", () => { }); expect(exitCode).toBe(0); - expect(stdout).toContain("count: 1 of 3 total"); + expect(stdout).toContain("count: 1 open of 3 total"); expect(stdout).toContain("issues[1]{number,title,state,author,created}:"); expect(stdout).toContain("issue list --limit "); }); -- 2.47.3