feat: benchmark credential parity, transcript, honest results + read-tier accuracy (tasks 0032, 0033) #36

Merged
alexion merged 10 commits from task-0032-bench-read-report-persistence into main 2026-07-18 10:28:33 -04:00
23 changed files with 663 additions and 68 deletions

View File

@@ -0,0 +1,78 @@
# Read-tier accuracy
## Problem Statement
The benchmark ranks gitea-axi first on both cost metrics — lowest cost-equivalent tokens and lowest imputed cost of the four arms — but second on accuracy, 95% against gitea-mcp's 97%.
That entire two-point deficit is a single task: `read-open-issue-count`, which gitea-axi fails on all three trials while gitea-mcp passes one of three.
The read tier is the weakest tier for every arm, and on it gitea-axi spends more turns and more output than gitea-mcp yet scores lower, so agents are working harder to answer count questions and still getting them wrong.
Two things stand in the way of closing this gap.
First, the `issue list` summary reports `count: N of M total` and never names the state it filtered on, so an agent asking "how many issues are open?" has to infer the answer rather than read it off the summary line.
Second, the benchmark records only tokens and a `failure: "incorrect"` tag for a failed read — never the agent's actual report — so a maintainer cannot tell whether the agent reported a wrong number or reported the right number in wording the checker's phrase list did not accept.
Without the report text the root cause of the 3/3 failure cannot be confirmed, so the product fix would be a guess.
## Solution
Make list count summaries self-answering, and make read failures diagnosable.
For the product, the count line of a filtered list names the state it counted.
When a maintainer or an agent runs `issue list` on the default open filter, the summary states that the count is a count of open issues, so the answer to "how many are open?" is present in the summary rather than only inferable from each row's state field.
For the harness, every result record carries the agent's final report for read tasks.
A failed read is then diagnosable directly from the stored record: the maintainer can see the exact text the agent submitted and tell a wrong count apart from a right count phrased in words the checker did not list.
This is the prerequisite that turns the `read-open-issue-count` failure from an opaque tag into evidence, and it is the data that decides whether the checker's accepted-phrasing list is later too strict.
## User Stories
1. As an agent answering a count question, I want the list summary to state which state it counted, so that I can report "5 open issues" straight from the summary line without inferring it from the rows.
2. As a maintainer reading `issue list` output, I want the count line to say what it counted, so that a bare `count: 5` is never ambiguous about whether those five are open, closed, or all issues.
3. As a maintainer auditing a failed read trial, I want the stored result record to include the agent's final report, so that I can see exactly what the agent said instead of only that it was "incorrect".
4. As a maintainer deciding whether the read checker is too strict, I want the persisted reports across trials, so that I can judge from evidence whether correct answers are being rejected on wording rather than substance.
5. As a maintainer re-running the benchmark after the count-line change, I want `read-open-issue-count` to move from a consistent failure toward a pass, so that gitea-axi's accuracy stops trailing on the one task that accounts for the whole gap.
## Implementation Decisions
- Two modules change: the `issue list` command in the product, and the benchmark runner's result-record assembly in the harness.
They serve one goal but are testable at different seams and can land independently, with the harness change first so the product fix can be confirmed against real report text.
- **State-aware count line (product).**
The list command composes the state into its own count line; the generic render helper that formats count lines stays generic.
The command already resolves the effective state filter (defaulting to open), so it passes that filter descriptor into the summary rather than the helper learning about issue state.
This keeps a single render seam and lets each list command opt in on its own terms; `pr list`, `search`, and `dashboard` are unaffected unless they choose to opt in the same way.
The existing count-line invariants are preserved: a total is always reported, and the bare `count: N` form must never appear.
- **Report persistence (harness).**
The agent's final report is already in hand at the point the runner scores a read task, so persisting it is threading that value into the record the runner assembles rather than plumbing it up from a new source.
The report is added to the result record for read tasks; the store serializes whatever record it is handed, so it needs no change of its own.
Mutation tasks are scored by diffing repository state and have no agent report to record, so the field is populated for read tasks and absent otherwise.
- **Ordering.**
Persist the report first and re-run enough of the read tier to capture real reports, then confirm from those reports whether the failure is a wrong count or a rejected phrasing before finalizing the count-line wording.
The count-line wording should be chosen so that an agent quoting the summary lands on an answer the read checker already accepts.
## Testing Decisions
- A good test here asserts external behavior: the text a user or agent sees on stdout, and the shape of the record the harness writes — not the internal composition of the count string.
- **Count line** is tested at the command's behavioral seam — the fixture-server CLI harness that runs the real command against a stubbed Gitea and asserts on rendered stdout.
Prior art is the existing `issue list` command test, which already asserts exact count-line strings such as `count: 3 of 17 total` and guards that the bare `count: N` form never appears.
New assertions extend that file: the count line names the state for the default open filter and for an explicit state filter, and the existing total-and-cap invariants still hold.
- **Report persistence** is tested at the runner's record-assembly seam, where the existing runner test already drives a cell to a recorded outcome and asserts on the produced record.
A completed read cell yields a record carrying the agent's final report; a mutation cell yields a record without one.
## Out of Scope
- Trimming gitea-axi's input and cache-read footprint.
gitea-axi replays the largest context of the efficient arms and wins cost only on the 5×-weighted output component, so reducing its input footprint would harden the cost lead — but it is a separate optimization touching many commands and is not part of closing the accuracy gap.
- Relaxing the read checker's accepted-phrasing list.
Whether the checker is too strict is a benchmark-validity decision that must be made from the persisted report evidence this spec produces, not pre-judged; changing accepted phrasings before seeing the reports risks tuning the benchmark to the tool rather than fixing the tool.
- Any change to how mutation tasks are scored, and any change to the cost-equivalent-token metric or its weighting.
## Further Notes
The two leaders trade a narrow accuracy edge for a clear cost lead, and this one task is the whole of that edge, so the count-line change is the single highest-leverage move on the accuracy axis.
The report-persistence change also has standing value beyond this task: it makes every future read failure diagnosable rather than opaque, which is the harness's current blind spot and the reason the root cause could not be confirmed from the existing records.
The state-explicit summary is squarely on gitea-axi's central thesis — an agent-ergonomic, low-token interface — because it hands the answer to a count question directly on the summary line instead of forcing the agent into extra turns to derive it.

View File

@@ -0,0 +1,26 @@
---
spec: read-tier-accuracy
---
## What to build
Persist the agent's final report on the benchmark result record for read tasks, so a failed read is diagnosable directly from the stored record instead of only carrying an opaque `incorrect` tag.
The report is already in hand where the runner scores a read task — threading it into the assembled record is all that is required; the sample store serializes whatever record it is handed and needs no change of its own.
Mutation tasks are scored by diffing repository state and have no agent report, so the field is populated for read tasks and absent otherwise.
## Acceptance criteria
- [x] The result record carries the agent's final report for read tasks.
- [x] The field is absent on records for mutation tasks.
- [x] A completed read cell produces a record whose report is the agent's final report; a mutation cell produces a record without one — asserted at the runner's record-assembly seam.
- [x] The sample store round-trips the report-bearing record without any change to the store itself.
- [x] Existing runner and store tests still pass.
## Implementation Notes
- `ResultRecord` gained an optional `report?: string`. The runner resolves the scoring spec once in `runCell` and records `run.finalReport` only when `spec.kind === "read"`, so the field is populated for read tasks and absent otherwise. `makeRecord` conditionally spreads the key (`...(report !== undefined ? { report } : {})`) so a mutation (or hung) record genuinely omits it rather than carrying `report: undefined`; verified by `expect(sample).not.toHaveProperty("report")` after the JSON round-trip.
- `scoreRun` was refactored to take a pre-resolved `ScoringSpec` instead of a `BenchTask`. This is marginally more than "thread the value into the record," but it is the minimal clean way to branch on `spec.kind` in `runCell` without calling `task.scoringSpec(owner)` twice.
- The store needed no change, as the spec predicted: it serializes whatever record it is handed, so the round-trip test passed on first run.
- Two review findings were left as deliberate judgement calls (both non-blocking baseline smells, no documented-standard breach): the `makeRecord` positional parameter list (extended by one param in the module's pre-existing positional style, kept consistent with surrounding code rather than refactored to an options object), and the two `spec.kind` branches a few lines apart in `runCell` and `scoreRun` (they select different things — the report vs. the scorer — and read clearer inline).

View File

@@ -0,0 +1,29 @@
---
spec: read-tier-accuracy
blocked-by: 0032-bench-read-report-persistence
---
## What to build
Make the `issue list` count line name the state it filtered on, so the answer to "how many issues are open?" is present in the summary rather than only inferable from each row's state field.
The command already resolves the effective state filter (defaulting to open), so it composes that state into its own count line and passes the composed line down; the generic count-line render helper stays generic and unaware of issue state. This keeps a single render seam and lets other list commands (`pr list`, `search`, `dashboard`) opt in on their own terms rather than inheriting the behavior.
The count-line wording is chosen so that an agent quoting the summary lands on an answer the read checker already accepts (its accepted renderings include forms like `5 open`). After the change lands, a re-run of the read tier confirms `read-open-issue-count` moves from a consistent failure toward a pass — using the reports persisted by [[0032-bench-read-report-persistence]] to confirm the failure was a wrong/inferred count rather than a rejected phrasing before finalizing the wording.
## Acceptance criteria
- [x] The `issue list` count line names the state it counted for the default open filter.
- [x] The count line names the state for an explicit `--state` filter.
- [x] The generic count-line render helper is unchanged and remains state-agnostic; `pr list`, `search`, and `dashboard` output is unaffected.
- [x] Existing count-line invariants hold: a total is always reported, and the bare `count: N` form never appears.
- [x] New assertions extend the existing `issue list` command test at the fixture-server CLI seam, asserting exact rendered count-line strings.
- [-] A re-run of the read tier shows `read-open-issue-count` moving toward a pass, and the chosen wording contains a rendering the read checker already accepts.
## Implementation Notes
- The `issue list` count line now renders `count: <shown> <state> of <total> total` — e.g. `count: 5 open of 5 total`, `count: 1 closed of 1 total`. The command composes the state into the count line via a local `countStateQualifier(state)` helper; the generic `formatCountLine` gained only an optional, domain-agnostic `qualifier?: string` and never learns about issue state. Every other caller (`pr list`, `search`, `dashboard`, `label`, `issue`'s blocks/blocked-by list) passes no qualifier, so their output is byte-identical — the untouched command tests still pass, which is what criterion 3 really guards.
- **Criterion 3 wording.** Read literally, `formatCountLine` is not "unchanged" — it gained a parameter. But it stays *state-agnostic* (the qualifier is a bare string; state-to-qualifier mapping lives in the command), which is the spec's actual intent ("the generic render helper that formats count lines stays generic … rather than the helper learning about issue state"). The single render seam is preserved. Marked satisfied on that reading; the spec author may wish to reword the criterion from "unchanged" to "state-agnostic".
- **`--state all`.** Deliberately renders with no state word (`count: N of M total`): `all` imposes no narrowing filter and has no natural one-word name, so naming it adds no disambiguation. Open and closed — the filters where a bare count could mislead — are named, which is what resolves User Story 2's ambiguity. Pinned by a dedicated `--state all` test.
- **Criterion 6 (`[-]`, deferred not dropped).** The controllable half is done and verified: the chosen wording contains a checker-accepted rendering — running the real `checkReadAnswer` against the real `formatCountLine(5, 5, false, "open")` output (`count: 5 open of 5 total`) scores a pass on the `read-open-issue-count` fact, so an agent that merely echoes the summary now passes. The live read-tier re-run itself needs the benchmark environment (a live Gitea host + the Claude Agent SDK) and is left as a follow-up to run when the harness is next exercised; the report-persistence from [[0032-bench-read-report-persistence]] is now in place to confirm the movement from real report text.
- Reconciled the pre-existing count-line assertions that the format change made stale (five in `test/issue-list.test.ts`, two in `test/detection.test.ts`) to the state-named form; added a `--state all` guard test.

View File

@@ -9,6 +9,9 @@ Any commit message you write must follow the Conventional Commits specification
The `origin` remote is a self-hosted **Gitea** instance (`git.alexion.dev`), not GitHub.
The `gh` CLI does not work here.
The benchmark arms invoke the **built `dist/main.js`** (the `gitea-axi` binary on `PATH`), not the TypeScript source.
Run `npm run build` before any live `bench:run` if you want `src/` changes reflected; the bench does not run from source.
Prefer this project's own CLI for pull requests — it is the tool being built, so opening its PRs with it is the dogfood path:
`npm run build && node dist/main.js pr create --login alexion --base main --head <branch> --title <text> --body-file <path>`.
It reuses the `tea` login profiles, so it needs no separate credentials.

View File

@@ -1,28 +1,37 @@
# Benchmark harness
This directory holds the benchmark that tests gitea-axi's central claim — that it is an agent-ergonomic, low-token interface to Gitea — against the `tea` CLI, the official `gitea-mcp` server, and raw Gitea REST calls.
This run bears that out on cost: gitea-axi posts the lowest cost-equivalent tokens and the lowest imputed cost of the four tools, though `gitea-mcp` edges it slightly on accuracy.
The result is honest rather than flattering: gitea-axi is the lowest-cost of the *structured* interfaces — it beats both `tea` and `gitea-mcp` on every tier at 100% task success — but hand-rolled raw REST is cheaper still, because terse HTTP is the token floor no wrapper undercuts.
Keeping the raw-REST arm in the comparison is deliberate: a benchmark of agent-CLIs that omits it will always crown the wrapper, and this one refuses to.
## How it works
Each arm is an agent given exactly one of the four tools and nothing else, run on the same fixed model at temperature zero, so the comparison measures the tool rather than the model.
The suite is 20 tasks across four tiers — read, single-mutation, find-then-act, and multi-step — each run against a freshly seeded throwaway repository and scored deterministically by diffing the resulting repository state (or matching required facts in the agent's answer) against the seeded ground truth.
The headline metric is cost-equivalent tokens: the four token components (fresh input, cache write, cache read, output) weighted by Anthropic's published API pricing ratios, which is why an arm can spend more raw tokens yet cost less.
Every arm is credentialed the way its product is really configured — the token in its environment (`gitea-axi`, `gitea-mcp`) or in its prompt (`raw-api`), and a `tea` login for `tea` — so no arm pays a turn tax rediscovering how to authenticate.
## Results
| arm | cost-equivalent tokens | raw tokens | success | imputed cost |
| --- | ---: | ---: | ---: | ---: |
| gitea-axi | 16,921 | 68,093 | 95% | $6.20 |
| raw-api | 17,773 | 55,631 | 95% | $6.64 |
| gitea-mcp | 17,898 | 60,028 | 97% | $6.82 |
| tea | 20,505 | 80,702 | 90% | $7.25 |
| arm | cost-equivalent tokens | raw tokens | turns | success | imputed cost |
| --- | ---: | ---: | ---: | ---: | ---: |
| raw-api | 16,971 | 52,586 | 4.3 | 100% | ~$0.11 |
| gitea-axi | 19,240 | 81,067 | 6.0 | 100% | ~$0.12 |
| tea | 20,568 | 82,188 | 6.2 | 97% | ~$0.12 |
| gitea-mcp | 21,803 | 79,961 | 5.7 | 100% | ~$0.14 |
All four arms completed the full matrix — 20 of 20 tasks each, at the reporting floor.
gitea-axi wins on cost-equivalent tokens and on real imputed cost even though it does not use the fewest raw tokens: its interactions are output-light, and output is the most expensive component (weighted 5×), so its compact answers beat arms that emit more.
gitea-mcp is the most accurate at 97% against gitea-axi's 95%, so the two leaders trade a small accuracy edge for a clear cost lead.
All four arms completed the full matrix — 20 of 20 tasks each, at the reporting floor — and success is near-perfect: only `tea` slips, to 89% on find-then-act, while the other three pass every run.
By tier, the read tasks are the hardest for every arm (7583% success) — exact-answer reads, not mutations, are where correctness slips.
tea is the outlier on find-then-act, dropping to 78% success at about 1.7× the cost-equivalent tokens of the other three arms.
Raw REST posts the lowest cost-equivalent tokens and leads every tier.
It is direct HTTP with the token in the request header, so it takes the fewest turns (4.3) and reads the least cached context, and no higher-level tool beats that on tokens alone.
This is the honest ceiling, and the reason gitea-axi does not claim the cost crown outright.
_Snapshot: 2026-07-17 — 4 arms × 20 tasks × 3 trials each (240 samples), a single run against one live Gitea host; imputed cost is Anthropic API-priced._
gitea-axi is a clear second overall and the cheapest of the structured tools: it undercuts the official `gitea-mcp` server and the `tea` CLI on every tier, at 100% success, with the lowest output-token count of any arm.
Note the split between raw and cost-equivalent tokens — gitea-axi spends more raw tokens than `gitea-mcp` yet costs less, because output is weighted 5× and gitea-axi's answers are compact.
By tier, raw REST's edge is widest on reads (10,921 vs gitea-axi's 14,415) — a read is one HTTP request for curl, where a CLI still spends a turn or two — and narrows on multi-step (24,348 vs 26,963), where the work itself dominates and interface overhead matters less.
Cost parity on the scored suite also understates gitea-axi, because the suite is the subset every arm can do at all.
The bonus table records capability-asymmetric operations — full-text issue search, rendering a PR's diff and checks, issue dependencies — that gitea-axi handles directly and raw REST has no first-class equivalent for.
_Snapshot: 2026-07-17 — 4 arms × 20 tasks × 3 trials each (240 samples), a single clean run with all four arms executed together against one live Gitea host; imputed cost is the mean per-task Anthropic-API-priced dollar cost._

View File

@@ -112,6 +112,32 @@ describe("buildArm", () => {
expect(envValues).toContain("s3cr3t-token");
});
// Behavior: the gitea-axi arm's shell is handed a credential environment
// carrying the host and token from the shared access, so its tool is
// pre-authenticated without the agent having to discover credentials
// (benchmark-harness spec, "Scaffolding"). The access here is built from
// independent literals; shell.env must deep-equal exactly the two facts echoed
// back under their env-var names (host→GITEA_AXI_API_URL, token→GITEA_AXI_TOKEN),
// and nothing else. The keys and mapping are fixed by gitea-axi's own env
// contract, not recomputed from arm.ts.
it("gives the gitea-axi arm's shell a credential env with the shared host URL and token", () => {
const preAuthed: SharedContext = {
coords: { owner: "acme", repo: "bench-xyz" },
access: { apiUrl: "https://git.example.test", token: "tok-abc123" },
};
const definition = buildArm("gitea-axi", preAuthed, { binRoot, locate });
const shell = definition.shell;
expect(shell).not.toBeNull();
if (shell === null) return;
expect(shell.env).toEqual({
GITEA_AXI_API_URL: "https://git.example.test",
GITEA_AXI_TOKEN: "tok-abc123",
});
});
// Behavior: each non-MCP arm's tool/PATH configuration comes from the guard and
// exposes only that arm's allowed binary (benchmark-harness spec, "Tool
// isolation" / ADR 0016). The (arm, binary) pairs are independent literals —

View File

@@ -40,6 +40,17 @@ export interface ArmShell {
path: string;
/** The authoritative tool-isolation guard, bound to this arm. */
guard: (command: string) => GuardDecision;
/**
* Credential environment the arm's tool is pre-configured with, merged into
* the agent's shell environment on top of {@link path}. This keeps the arms
* symmetric on authentication: every arm is handed its host and token the way
* its product is really configured, so none pays a turn tax rediscovering how
* to authenticate. The gitea-mcp arm gets the equivalent through its MCP
* server's env; raw-api uses the token stated in its prompt directly; the
* gitea-axi arm is configured through its own env interface here. Empty for an
* arm that needs no ambient credentials.
*/
env: Record<string, string>;
}
/**
@@ -172,7 +183,7 @@ function mcpAttachment(context: SharedContext): ArmMcp {
* gitea-mcp arm has no shell binary (`provisionArmBin` exposes nothing for it),
* so this returns null there and the arm reaches Gitea through its MCP tools.
*/
function buildShell(arm: Arm, options: BuildArmOptions): ArmShell | null {
function buildShell(arm: Arm, context: SharedContext, options: BuildArmOptions): ArmShell | null {
if (arm === "gitea-mcp") {
return null;
}
@@ -183,16 +194,37 @@ function buildShell(arm: Arm, options: BuildArmOptions): ArmShell | null {
binDir,
path: ambient === "" ? binDir : `${binDir}${delimiter}${ambient}`,
guard: (command) => guardCommand(arm, command),
env: shellEnv(arm, context),
};
}
/**
* The credential environment a shell arm's tool is pre-configured with. The
* gitea-axi arm is handed its host and token through its own env interface
* (`GITEA_AXI_API_URL` / `GITEA_AXI_TOKEN`), the symmetric counterpart to the
* gitea-mcp arm's server env: both name the same host and token, and both leave
* the agent to name the repository per call (gitea-axi via `-R`, gitea-mcp via
* each tool's arguments). The tea and raw-api arms need no ambient credentials —
* raw-api uses the token stated in its prompt directly in each request, and tea
* resolves its own login store — so their env is empty.
*/
function shellEnv(arm: Arm, context: SharedContext): Record<string, string> {
if (arm === "gitea-axi") {
return {
GITEA_AXI_API_URL: context.access.apiUrl,
GITEA_AXI_TOKEN: context.access.token,
};
}
return {};
}
/** Assemble the single arm definition the runner consumes for the given arm. */
export function buildArm(arm: Arm, context: SharedContext, options: BuildArmOptions): ArmDefinition {
const systemPrompt = `${basePrompt(context)}\n\n${armBootstrap(arm, context, options)}`;
return {
arm,
systemPrompt,
shell: buildShell(arm, options),
shell: buildShell(arm, context, options),
mcp: arm === "gitea-mcp" ? mcpAttachment(context) : null,
};
}

View File

@@ -12,17 +12,17 @@
// runner (runner.ts) drives the run and feeds the transcript here.
import type { ArmDefinition } from "./arm.js";
import type { TranscriptEntry } from "./result.js";
/**
* One tool invocation recorded in the agent's transcript, reduced to what the
* isolation audit needs. `shell` is a proposed shell command; `mcp` is a call to
* an attached MCP server's tool; `other` is a built-in, non-Gitea-reaching tool
* (file read/edit and the like) that carries no isolation risk.
* (file read/edit and the like) that carries no isolation risk. This is the same
* shape the record persists ({@link TranscriptEntry}); the audit and the record
* share one type so they cannot drift.
*/
export type ToolUse =
| { kind: "shell"; command: string }
| { kind: "mcp"; server: string; tool: string }
| { kind: "other"; name: string };
export type ToolUse = TranscriptEntry;
/**
* The audit's verdict. On a leak it carries a human-readable reason per foreign

View File

@@ -310,6 +310,36 @@ describe("checkReadAnswer", () => {
expect(checkReadAnswer(facts, report)).toEqual({ pass: true });
});
it("passes when the report wraps a fact's phrasing in markdown emphasis", () => {
// The fact's acceptable phrasing is the bare substring "5 open".
const facts: RequiredFact[] = [
{ description: "open-issue count", anyOf: ["5 open"] },
];
// The report is correct but renders the number in markdown bold. Emphasis
// markers (`*`, `_`, backtick) are formatting, not substance, so "**5**" is
// equivalent to "5" and the phrasing "5 open" is present.
const report = "There are **5** open issues in the repository.";
expect(checkReadAnswer(facts, report)).toEqual({ pass: true });
});
it("fails when a markdown-formatted report states the wrong value", () => {
// The only acceptable phrasing counts five open issues.
const facts: RequiredFact[] = [
{ description: "open-issue count", anyOf: ["5 open"] },
];
// The report is markdown-formatted but substantively wrong: it counts three,
// not five. Stripping emphasis must only remove formatting, so after stripping
// ("there are 3 open issues") the phrasing "5 open" is still absent.
const report = "There are **3** open issues in the repository.";
const result = checkReadAnswer(facts, report);
expect(result.pass).toBe(false);
});
});
describe("score", () => {

View File

@@ -253,9 +253,18 @@ export function checkReadAnswer(facts: RequiredFact[], report: string): CheckRes
};
}
/** Lower-case and collapse runs of whitespace so incidental phrasing does not matter. */
/**
* Lower-case, drop markdown emphasis/code markers, and collapse runs of
* whitespace so incidental phrasing and formatting do not matter — a report that
* bolds a value (`**5**`) matches a phrasing that does not (`5 open`), since the
* emphasis is presentation, not substance.
*/
function normalizeText(text: string): string {
return text.toLowerCase().replace(/\s+/g, " ").trim();
return text
.toLowerCase()
.replace(/[*_`]/g, "")
.replace(/\s+/g, " ")
.trim();
}
/**

View File

@@ -45,6 +45,19 @@ export type FailureTag = "incorrect" | "confused" | "hung";
/** The pass/fail outcome of a run, tagged with the failure mode when it fails. */
export type Outcome = { pass: true } | { pass: false; failure: FailureTag };
/**
* One tool invocation as it is recorded in a run's transcript, in the order it
* executed. This is the canonical shape the harness both audits for isolation
* (see audit.ts, whose `ToolUse` aliases this) and persists on the record for
* diagnosis. A `shell` entry keeps the exact command line the agent ran; an `mcp`
* entry names the server and tool it called; `other` names a built-in tool that
* reaches no Gitea channel.
*/
export type TranscriptEntry =
| { kind: "shell"; command: string }
| { kind: "mcp"; server: string; tool: string }
| { kind: "other"; name: string };
/**
* One completed `(arm, task, trial)` run. Carries the metrics the headline and
* supporting views are computed from, plus the tags those views group by.
@@ -72,6 +85,24 @@ export interface ResultRecord {
/** Pass/fail outcome with a failure tag. */
outcome: Outcome;
/**
* The agent's final report, retained for read tasks so a failed read is
* diagnosable directly from the record — the exact text the agent submitted,
* distinguishing a wrong answer from a right one phrased in words the checker's
* accepted-phrasing list did not match. Absent for mutation tasks, which are
* scored by diffing repository state and have no agent report to record.
*/
report?: string;
/**
* The ordered transcript of tool invocations the run made, retained on every
* scored run so its turn cost is diagnosable directly from the record — the
* exact command sequence, which is how an arm's turn count (the dominant driver
* of cache-read tokens) is explained. Absent only for a hung run, which
* produced no completed transcript to record.
*/
transcript?: TranscriptEntry[];
}
/**

View File

@@ -6,6 +6,7 @@ import type { RepoState } from "./scoring-spec.js";
import type { BenchAccess, RepoCoords } from "./seed.js";
import { groundTruth } from "./seed-plan.js";
import { createSampleStore } from "./store.js";
import type { BenchTask } from "./task.js";
import { SAMPLE_TASK } from "./task.js";
import type { AgentDriver, BenchHost } from "./runner.js";
import { runCell } from "./runner.js";
@@ -240,6 +241,12 @@ describe("runCell", () => {
expect(sample.imputedCostUsd).toBe(DRIVER_COST);
expect(sample.outcome).toEqual({ pass: true });
// The recorded sample carries the run's tool transcript — the exact ordered
// sequence of tool invocations the driver reported — so the turn's cost is
// diagnosable directly from the record. The expected value is the literal the
// fake driver planted, deep-equal and in order, not recomputed from runner.ts.
expect(sample.transcript).toEqual([{ kind: "mcp", server: "gitea-mcp", tool: "edit_issue" }]);
// The sample carries the run's wall-clock duration; a completed run takes
// non-negative time.
expect(typeof sample.durationMs).toBe("number");
@@ -397,4 +404,110 @@ describe("runCell", () => {
if (sample === undefined) return;
expect(sample.outcome).toEqual({ pass: false, failure: "incorrect" });
});
// Behavior: a completed read cell produces a record whose report is the agent's
// final report (benchmark-harness spec, record-assembly seam). A read task is
// scored by matching the agent's final report against required facts; the
// recorded ResultRecord must carry that final report in its `report` field so a
// read is diagnosable from the record alone. Here an inline read task's fact is
// satisfied by the driver's finalReport (it contains "5 open"), so the cell
// completes as a pass; the recorded sample's `report` must equal the exact
// planted finalReport literal — an independent literal, read back through the
// store, not recomputed from runner.ts.
it("records the agent's final report on a completed read cell", async () => {
const READ_TASK: BenchTask = {
id: "count-open-issues",
tier: "read",
intent: "How many issues are open?",
scoringSpec: () => ({
kind: "read",
facts: [{ description: "open-issue count", anyOf: ["5 open"] }],
}),
};
// The independent literal we plant as the driver's final report. It contains
// the fact's rendering ("5 open"), so the read scorer scores it a pass.
const FINAL_REPORT = "There are 5 open issues.";
const readDriver: AgentDriver = {
async run() {
return {
tokens: { freshInput: 50, cacheCreation: 0, cacheRead: 0, output: 10 },
turns: 2,
imputedCostUsd: 0.03,
transcript: [{ kind: "mcp", server: "gitea-mcp", tool: "list_repo_issues" }],
finalReport: FINAL_REPORT,
stoppedByTurnCap: false,
};
},
};
const { host } = createFakeHost();
const store = createSampleStore(storeRoot);
const outcome = await runCell({
arm: "gitea-mcp",
task: READ_TASK,
trial: 1,
access: ACCESS,
host,
driver: readDriver,
store,
bounds: { turnCap: 10, wallClockMs: 60_000 },
build: { binRoot },
});
expect(outcome.kind).toBe("recorded");
if (outcome.kind !== "recorded") return;
const samples = store.read({ arm: "gitea-mcp", taskId: READ_TASK.id });
expect(samples).toHaveLength(1);
const [sample] = samples;
expect(sample).toBeDefined();
if (sample === undefined) return;
// A completed read cell: the checker scored the final report a pass.
expect(sample.outcome).toEqual({ pass: true });
// The record carries the agent's final report verbatim.
expect(sample.report).toBe(FINAL_REPORT);
});
// Behavior: a completed mutation cell produces a record WITHOUT an agent report
// (benchmark-harness spec, record-assembly seam). A mutation task is scored by
// diffing repository state, not by reading the agent's words, so the recorded
// ResultRecord must carry no `report` field at all — absent, not present-but-
// undefined. SAMPLE_TASK is a mutation task; paired with the passing fake host
// and driver it drives a completed mutation PASS. The absence is the independent
// source of truth (the acceptance criterion), read back through the store after
// the JSON round-trip so an absent field is genuinely not a key.
it("records no agent report on a completed mutation cell", async () => {
const { host } = createFakeHost();
const store = createSampleStore(storeRoot);
const outcome = await runCell({
arm: "gitea-mcp",
task: SAMPLE_TASK,
trial: 1,
access: ACCESS,
host,
driver,
store,
bounds: { turnCap: 10, wallClockMs: 60_000 },
build: { binRoot },
});
expect(outcome.kind).toBe("recorded");
if (outcome.kind !== "recorded") return;
const samples = store.read({ arm: "gitea-mcp", taskId: SAMPLE_TASK.id });
expect(samples).toHaveLength(1);
const [sample] = samples;
expect(sample).toBeDefined();
if (sample === undefined) return;
// A completed mutation cell: the state diff scored a pass.
expect(sample.outcome).toEqual({ pass: true });
// A mutation record carries no agent report — the key is absent entirely.
expect(sample).not.toHaveProperty("report");
});
});

View File

@@ -14,8 +14,8 @@
import { buildArm, type ArmDefinition, type BuildArmOptions, type SharedContext } from "./arm.js";
import { auditTranscript, type ToolUse } from "./audit.js";
import { score } from "./checker.js";
import type { Arm, Outcome, ResultRecord, TokenComponents } from "./result.js";
import type { RepoState } from "./scoring-spec.js";
import type { Arm, Outcome, ResultRecord, TokenComponents, TranscriptEntry } from "./result.js";
import type { RepoState, ScoringSpec } from "./scoring-spec.js";
import type { BenchAccess, RepoCoords } from "./seed.js";
import type { SampleStore } from "./store.js";
import type { BenchTask } from "./task.js";
@@ -149,7 +149,10 @@ export async function runCell(input: RunCellInput): Promise<CellOutcome> {
// A hung run produced no completed transcript to audit or score; record it
// as a failure with no measured consumption.
if (result.kind === "hung") {
return recorded(store, makeRecord(input, NO_TOKENS, 0, 0, durationMs, { pass: false, failure: "hung" }, clock));
return recorded(
store,
makeRecord(input, NO_TOKENS, 0, 0, durationMs, { pass: false, failure: "hung" }, undefined, undefined, clock),
);
}
const run = result.run;
@@ -161,13 +164,19 @@ export async function runCell(input: RunCellInput): Promise<CellOutcome> {
return { kind: "invalid", leaks: audit.leaks };
}
const spec = task.scoringSpec(coords.owner);
const outcome = run.stoppedByTurnCap
? ({ pass: false, failure: "confused" } as const)
: await scoreRun(host, coords, task, run);
: await scoreRun(host, coords, spec, run);
// Retain the agent's final report for read tasks so a failed read is
// diagnosable directly from the record; mutation tasks are scored by diffing
// repository state and have no agent report to record.
const report = spec.kind === "read" ? run.finalReport : undefined;
return recorded(
store,
makeRecord(input, run.tokens, run.turns, run.imputedCostUsd, durationMs, outcome, clock),
makeRecord(input, run.tokens, run.turns, run.imputedCostUsd, durationMs, outcome, report, run.transcript, clock),
);
} finally {
await host.delete(coords);
@@ -214,8 +223,7 @@ async function runBounded(
* final report against the required facts for a read. A pass is a pass; anything
* the checker rejects is an incorrect failure.
*/
async function scoreRun(host: BenchHost, coords: RepoCoords, task: BenchTask, run: AgentRun): Promise<Outcome> {
const spec = task.scoringSpec(coords.owner);
async function scoreRun(host: BenchHost, coords: RepoCoords, spec: ScoringSpec, run: AgentRun): Promise<Outcome> {
const snapshot = await host.capture(coords);
const check =
spec.kind === "mutation"
@@ -232,6 +240,8 @@ function makeRecord(
imputedCostUsd: number,
durationMs: number,
outcome: Outcome,
report: string | undefined,
transcript: TranscriptEntry[] | undefined,
clock: RunnerClock,
): ResultRecord {
return {
@@ -245,6 +255,12 @@ function makeRecord(
durationMs,
imputedCostUsd,
outcome,
// Absent for mutation runs and runs with no completed report (hung); JSON
// serialization drops the key when undefined.
...(report !== undefined ? { report } : {}),
// Absent only for a hung run, which produced no transcript; JSON
// serialization drops the key when undefined.
...(transcript !== undefined ? { transcript } : {}),
};
}

View File

@@ -1,5 +1,7 @@
import { readdirSync, rmSync, statSync } from "node:fs";
import path from "node:path";
import { describe, expect, it } from "vitest";
import { sumTokens } from "./sdk-driver.js";
import { createAgentWorkdir, sumTokens } from "./sdk-driver.js";
import type { SdkResultMessage } from "./sdk-driver.js";
describe("sumTokens", () => {
@@ -87,3 +89,31 @@ describe("sumTokens", () => {
});
});
});
describe("createAgentWorkdir", () => {
// Behavior: each agent run must operate in a fresh, empty directory located
// OUTSIDE the harness's own checkout. This isolation is what stops an agent
// that forgets an explicit `-R OWNER/NAME` from having its `gitea-axi`/`tea`
// tools silently default their target repo to the harness's own git checkout:
// a directory that is empty (no `.git`) and outside the current checkout gives
// those tools nothing local to resolve.
//
// The three assertions are independent anti-bug properties drawn directly from
// the requirement, not recomputed from the implementation:
// 1. the path exists and is a directory,
// 2. it is empty (zero entries — in particular no `.git`),
// 3. it sits outside the current working directory (relative path escapes
// upward with "..").
it("returns a fresh, empty directory located outside the current checkout", () => {
const dir = createAgentWorkdir();
try {
expect(statSync(dir).isDirectory()).toBe(true);
expect(readdirSync(dir)).toHaveLength(0);
expect(path.relative(process.cwd(), dir).startsWith("..")).toBe(true);
} finally {
// Safe: `dir` is a fresh throwaway temp dir we just received from
// createAgentWorkdir(); never a delete of cwd or any pre-existing path.
rmSync(dir, { recursive: true, force: true });
}
});
});

View File

@@ -17,6 +17,9 @@
// recorded in the transcript, so the runner's post-run audit sees what actually
// executed — a blocked attempt is realistic wasted effort, not a leak.
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { ArmDefinition } from "./arm.js";
import { foreignToolReason, type ToolUse } from "./audit.js";
import type { TokenComponents } from "./result.js";
@@ -104,6 +107,8 @@ interface SdkQueryOptions {
abortController: AbortController;
canUseTool: (toolName: string, input: Record<string, unknown>) => Promise<SdkPermissionResult>;
settingSources: string[];
/** The agent's shell working directory: a fresh empty dir outside any checkout. */
cwd: string;
env?: Record<string, string | undefined>;
mcpServers?: Record<string, SdkStdioServer>;
disallowedTools?: string[];
@@ -197,30 +202,47 @@ export function sdkAgentDriver(config: SdkDriverConfig = {}): AgentDriver {
return { behavior: "allow", updatedInput: toolInput };
};
const options = buildOptions(input.arm, model, input.turnCap, controller, canUseTool);
const workdir = createAgentWorkdir();
try {
const options = buildOptions(input.arm, model, input.turnCap, controller, canUseTool, workdir);
let result: SdkResultMessage | undefined;
for await (const message of query({ prompt: input.intent, options })) {
if (message.type === "result") {
result = message as SdkResultMessage;
let result: SdkResultMessage | undefined;
for await (const message of query({ prompt: input.intent, options })) {
if (message.type === "result") {
result = message as SdkResultMessage;
}
}
if (result === undefined) {
throw new Error("the Agent SDK produced no result message");
}
}
if (result === undefined) {
throw new Error("the Agent SDK produced no result message");
}
return {
tokens: sumTokens(result),
turns: result.num_turns ?? 0,
imputedCostUsd: result.total_cost_usd ?? 0,
transcript,
finalReport: result.result ?? "",
stoppedByTurnCap: result.subtype === "error_max_turns",
};
return {
tokens: sumTokens(result),
turns: result.num_turns ?? 0,
imputedCostUsd: result.total_cost_usd ?? 0,
transcript,
finalReport: result.result ?? "",
stoppedByTurnCap: result.subtype === "error_max_turns",
};
} finally {
rmSync(workdir, { recursive: true, force: true });
}
},
};
}
/**
* Create a fresh, empty working directory for one agent run, outside any git
* checkout. The agent's shell runs here so a shell tool that defaults its target
* repository from the local checkout (gitea-axi, tea) cannot silently resolve the
* harness's own repository when the agent omits an explicit `-R`; with no ambient
* checkout the agent must target the repository named in its prompt. The caller
* deletes it when the run ends.
*/
export function createAgentWorkdir(): string {
return mkdtempSync(join(tmpdir(), "bench-agent-cwd-"));
}
/** Assemble the SDK query options for an arm's tool configuration. */
function buildOptions(
arm: ArmDefinition,
@@ -228,6 +250,7 @@ function buildOptions(
turnCap: number,
controller: AbortController,
canUseTool: SdkQueryOptions["canUseTool"],
cwd: string,
): SdkQueryOptions {
const options: SdkQueryOptions = {
model,
@@ -241,11 +264,17 @@ function buildOptions(
// Start from a clean slate: no user/project settings leak tools or config
// into the measured run.
settingSources: [],
// Run outside any checkout so a forgotten -R cannot resolve the harness's own
// repository instead of the seeded throwaway (see createAgentWorkdir).
cwd,
};
if (arm.shell !== null) {
// Lead the agent's PATH with the arm's curated bin directory so only its one
// allowed binary resolves by name; the guard on canUseTool is the authority.
options.env = { ...process.env, PATH: arm.shell.path };
// Layer the arm's credential env underneath so its tool is pre-authenticated
// the way its product is really configured, symmetric to the gitea-mcp
// server's env (see ArmShell.env); PATH stays last so it is never overridden.
options.env = { ...process.env, ...arm.shell.env, PATH: arm.shell.path };
}
if (arm.mcp !== null) {
options.mcpServers = { [arm.arm]: { type: "stdio", ...arm.mcp.server } };

View File

@@ -102,4 +102,24 @@ describe("SampleStore", () => {
secondRun.append(second);
expect(secondRun.read(cell)).toEqual([first, second]);
});
// Behavior: the store round-trips a report-bearing record without any change to
// the store itself. A read task's record carries the agent's final report in the
// `report` field; appending it and reading it back must return it equal, `report`
// and all, proving the store persists the field with no store change. The planted
// report string is an independent literal, not anything production code computes.
it("round-trips a record carrying a report field, preserving it unchanged", () => {
const store = createSampleStore(root);
const report = "There are 5 open issues in the repository.";
const record = sample({ tier: "read", report });
store.append(record);
const readBack = store.read({ arm: record.arm, taskId: record.taskId });
expect(readBack).toEqual([record]);
const [only] = readBack;
expect(only).toBeDefined();
if (only === undefined) return;
expect(only.report).toBe(report);
});
});

View File

@@ -16,16 +16,29 @@ Reach for `gitea-axi` whenever a task touches a Gitea repository's issues, pull
- **Over raw Gitea API calls:** it handles auth, pagination, name-to-ID resolution, and review-decision aggregation for you, so you do not hand-roll HTTP.
- **Over improvised `git`:** for anything about issues or pull requests as entities (state, reviews, labels, comments) rather than local commits and branches.
Run it inside a Gitea checkout, or pass `-R OWNER/NAME` (and `--login <name>`) to target a repository explicitly.
## Targeting and authentication
Every command resolves two things: which repository to act on, and which credentials to authenticate with.
Get both right on the first call — they are the usual reason a command fails and has to be retried.
- **Repository.** Inside a Gitea checkout it is taken from the `origin` remote automatically.
Outside a checkout you must name it: pass `-R OWNER/NAME` on every command (or set `GITEA_AXI_REPO=OWNER/NAME` once for the session).
- **Credentials.** When the environment is pre-configured — `GITEA_AXI_TOKEN` together with `GITEA_AXI_API_URL` — authentication is automatic and you need nothing more.
Otherwise credentials come from a `tea` login: pass `--login <name>` (or set `GITEA_AXI_LOGIN=<name>`) unless the checkout's remote already selects one.
So outside a checkout with the token in the environment, `gitea-axi <command> -R OWNER/NAME …` is all you need; do not go hunting for a config file or a login profile.
## Command groups
- `issue` — list, view, create, comment on, edit, close/reopen, pin, and link issues.
- `pr` — create, view, comment on, edit, review, merge, check out, diff, and inspect the checks of pull requests.
- `label` — list, create, edit, and delete labels.
- `search` — full-text search across issues and pull requests.
- `search` — full-text search; it takes a subcommand, so search issues with `search issues "<query>"` and pull requests with `search prs "<query>"` (a bare `search "<query>"` is not valid).
- `setup` — install this skill (`setup`) and, opt-in, the SessionStart dashboard hook (`setup hooks`).
To read one issue's fields, reach straight for `issue view <number>`: it shows labels and state by default, and takes `--fields assignees,milestone,…` for the rest.
You rarely need `issue list` to answer a question about a single issue.
## Discovery
This skill is a pointer, not a command reference — the CLI is the single source of truth for its own interface.

View File

@@ -222,8 +222,11 @@ Show a single issue. Pull request numbers are rejected — use \`pr view\` inste
flags:
--comments Render every comment in full (bodies truncated at 800 chars)
--full Suppress all truncation of the issue body and comment bodies
--fields <a,b,c> Append extra fields: assignees, closedAt, milestone, updatedAt, url
--help Show this help
Labels are shown by default; use --fields to add assignees, milestone, and more.
global flags:
-R, --repo <OWNER/NAME> Override the repository detected from the git origin remote
--login <name> Select a tea login profile by name
@@ -460,30 +463,51 @@ async function issueList(deps: CliDeps, args: string[]): Promise<string> {
return renderList({
noun: "issues",
rows,
countLine: formatCountLine(rows.length, total, rows.length >= limit),
countLine: formatCountLine(rows.length, total, rows.length >= limit, countStateQualifier(state)),
help: issueListSuggestions(context, state, rows.length, total),
});
}
// The count line names the state the list was filtered to, so the answer to
// "how many are open?" is on the summary line rather than only inferable from
// each row. `all` imposes no narrowing and has no natural one-word name, so it
// adds no qualifier and the generic count line stands.
function countStateQualifier(state: IssueState): string | undefined {
return state === "all" ? undefined : state;
}
// The default detail fields reuse the same declarative extraction as the list
// path; only `body` (truncation) and `comment_count` need bespoke handling.
const ISSUE_VIEW_FIELDS: FieldDef<Issue>[] = [
pluck("number"),
pluck("title"),
lowercased("state"),
joined("labels", "labels", "name"),
pluck("author", "user.login"),
relativeTimeField("created", "created_at"),
];
// Appended to the default view fields on request via `--fields`, never replacing
// them. Labels and body are shown by default, so they are not offered here.
const ISSUE_VIEW_EXTRA_FIELDS: Record<string, FieldDef<Issue>> = {
assignees: joined("assignees", "assignees", "login"),
closedAt: relativeTimeField("closedAt", "closed_at"),
milestone: pluck("milestone", "milestone.title"),
updatedAt: relativeTimeField("updatedAt", "updated_at"),
url: pluck("url", "html_url"),
};
interface IssueDetailOptions {
host: string;
full: boolean;
withComments: boolean;
now: Date;
/** Extra fields selected via `--fields`, appended after the defaults. */
extraFields: FieldDef<Issue>[];
}
function buildIssueDetail(issue: Issue, options: IssueDetailOptions): Record<string, unknown> {
const row = extractRow(issue, ISSUE_VIEW_FIELDS, {
const row = extractRow(issue, [...ISSUE_VIEW_FIELDS, ...options.extraFields], {
now: options.now,
host: options.host,
full: options.full,
@@ -531,12 +555,21 @@ async function issueView(deps: CliDeps, args: string[]): Promise<string> {
}
const { flags, positionals } = parseFlags(
args,
{ "--comments": { takesValue: false }, "--full": { takesValue: false } },
{
"--comments": { takesValue: false },
"--full": { takesValue: false },
"--fields": { takesValue: true },
},
"issue view",
);
const number = parsePositionalNumber(positionals, "issue view", "issue");
const full = flags["--full"] === true;
const withComments = flags["--comments"] === true;
const extraFields = selectExtraFields(
flagValue(flags, "--fields"),
ISSUE_VIEW_EXTRA_FIELDS,
"issue view",
);
const context = await resolveRepoContext(deps);
const api = createClient(context);
@@ -549,7 +582,7 @@ async function issueView(deps: CliDeps, args: string[]): Promise<string> {
}
const now = new Date();
const item = buildIssueDetail(issue, { host: context.host, full, withComments, now });
const item = buildIssueDetail(issue, { host: context.host, full, withComments, now, extraFields });
const blocks: DetailBlock[] = [];
if (withComments) {

View File

@@ -18,14 +18,20 @@ export function formatCountLine(
shown: number,
total: number | undefined,
atLimit: boolean,
qualifier?: string,
): string {
// A generic, caller-supplied qualifier names what was counted (e.g. the state
// a list was filtered to) right after the count, so `count: 5 open of 5 total`
// answers "how many are open?" off the summary line. The helper stays unaware
// of any specific domain concept — callers that pass none render as before.
const counted = qualifier === undefined ? `${shown}` : `${shown} ${qualifier}`;
if (total === undefined) {
if (atLimit) {
return `count: ${shown} (showing first ${shown})`;
return `count: ${counted} (showing first ${shown})`;
}
return `count: ${shown} of ${shown} total`;
return `count: ${counted} of ${shown} total`;
}
return `count: ${shown} of ${total} total`;
return `count: ${counted} of ${total} total`;
}
/** Encode a named list block, with an explicit empty-state line when there are no rows. */

View File

@@ -122,7 +122,7 @@ describe("repository context detection", () => {
});
expect(exitCode).toBe(0);
expect(stdout).toContain("count: 3 of 3 total");
expect(stdout).toContain("count: 3 open of 3 total");
expect(server!.requests[0]!.headers.authorization).toBe("Bearer detected-token");
// Auto-detected context: suggestions must not carry override flags.
expect(stdout).not.toContain("-R testowner/testrepo");
@@ -143,7 +143,7 @@ describe("repository context detection", () => {
});
expect(exitCode).toBe(0);
expect(stdout).toContain("count: 3 of 3 total");
expect(stdout).toContain("count: 3 open of 3 total");
});
it("fails with REPO_NOT_FOUND when there is no recognizable origin remote", async () => {

View File

@@ -40,7 +40,7 @@ describe.skipIf(!E2E_URL)("end-to-end: tracer command set", () => {
expect(exitCode).toBe(0);
const lines = stdout.split("\n");
expect(lines[0]).toBe("count: 3 of 3 total");
expect(lines[0]).toBe("count: 3 open of 3 total");
expect(lines[1]).toBe("issues[3]{number,title,state,author,created}:");
for (const title of instance.openTitles) {
expect(stdout).toContain(title);
@@ -58,7 +58,7 @@ describe.skipIf(!E2E_URL)("end-to-end: tracer command set", () => {
});
expect(exitCode).toBe(0);
expect(stdout).toContain("count: 1 of 1 total");
expect(stdout).toContain("count: 1 closed of 1 total");
expect(stdout).toContain(instance.closedTitle);
expect(stdout).toContain(",closed,");
});
@@ -69,7 +69,7 @@ describe.skipIf(!E2E_URL)("end-to-end: tracer command set", () => {
});
expect(exitCode).toBe(0);
expect(stdout).toContain("count: 1 of 3 total");
expect(stdout).toContain("count: 1 open of 3 total");
expect(stdout).toContain("issues[1]{number,title,state,author,created}:");
expect(stdout).toContain("issue list --limit <n>");
});

View File

@@ -62,7 +62,7 @@ describe("issue list", () => {
expect(exitCode).toBe(0);
const lines = stdout.split("\n");
expect(lines[0]).toBe("count: 3 of 17 total");
expect(lines[0]).toBe("count: 3 open of 17 total");
expect(lines[1]).toBe("issues[3]{number,title,state,author,created}:");
expect(lines[2]).toMatch(/^ {2}42,"Fix login redirect loop, please",open,alexion,\d+(mo|[smhdy]) ago$/);
expect(lines[3]).toMatch(/^ {2}41,Add dark mode,open,contributor,\d+(mo|[smhdy]) ago$/);
@@ -105,7 +105,7 @@ describe("issue list", () => {
);
expect(exitCode).toBe(0);
expect(stdout).toContain("count: 1 of 1 total");
expect(stdout).toContain("count: 1 closed of 1 total");
expect(stdout).toContain("37,Crash on empty config,closed,contributor");
});
@@ -133,7 +133,7 @@ describe("issue list", () => {
});
expect(exitCode).toBe(0);
expect(stdout).toContain("count: 0 of 0 total");
expect(stdout).toContain("count: 0 open of 0 total");
expect(stdout).toContain("issues[0]: (none)");
expect(stdout).toMatch(/^help\[\d+\]:/m);
});
@@ -209,6 +209,28 @@ describe("issue list", () => {
expect(stdout).toContain("issues[3]{number,title,state,author,created}:");
expect(stdout).not.toContain("type");
});
it("names no state in the count line for --state all", async () => {
// `all` imposes no narrowing filter, so the count line stays the plain
// `count: <shown> of <total> total` form with no state qualifier.
const issues = Array.from({ length: 9 }, (_, i) => issueOf(i + 1));
server = await startFixtureServer([
{
method: "GET",
path: ISSUES_PATH,
query: { state: "all" },
headers: { "X-Total-Count": "9" },
body: issues,
},
]);
const { stdout, exitCode } = await runCliTest(
["issue", "list", "--state", "all"],
{ env: testModeEnv(server.url) },
);
expect(exitCode).toBe(0);
expect(stdout.split("\n")[0]).toBe("count: 9 of 9 total");
});
});
describe("issue list filters", () => {
@@ -375,7 +397,7 @@ describe("issue list --sort", () => {
expect(renderedNumbers(stdout)[0]).toBe(7);
// The count line keeps T from X-Total-Count: sorting reorders without
// changing membership, so the unfiltered total stays accurate (ADR 0005).
expect(stdout).toContain("count: 30 of 52 total");
expect(stdout).toContain("count: 30 open of 52 total");
});
it("applies --limit to the sorted order, not to the fetched pages", async () => {
@@ -393,7 +415,7 @@ describe("issue list --sort", () => {
);
expect(renderedNumbers(stdout)).toEqual([38, 42]);
expect(stdout).toContain("count: 2 of 17 total");
expect(stdout).toContain("count: 2 open of 17 total");
// Pagination reads full pages regardless of --limit; the cap is applied after sorting.
expect(server.requests[0]!.query.limit).toBe("50");
});
@@ -408,7 +430,7 @@ describe("issue list --sort", () => {
env: testModeEnv(server.url),
});
expect(stdout).toContain("count: 2 of 3 total");
expect(stdout).toContain("count: 2 open of 3 total");
});
it("stops at the page cap when a server keeps returning full pages", async () => {

View File

@@ -47,6 +47,46 @@ describe("issue view", () => {
expect(stdout).toContain("comment_count: 3 — use --comments to see full comments");
});
it("renders the issue's labels comma-joined by default, with no flag", async () => {
server = await startFixtureServer([
{
method: "GET",
path: ISSUE_PATH,
body: issueBody({ labels: [{ name: "bug" }, { name: "regression" }] }),
},
]);
const { stdout, exitCode } = await runCliTest(["issue", "view", "42"], {
env: testModeEnv(server.url),
});
expect(exitCode).toBe(0);
// TOON-quoted because the joined value contains a comma.
expect(stdout).toContain('labels: "bug, regression"');
});
it("appends named extra fields with --fields on top of the default fields", async () => {
server = await startFixtureServer([
{
method: "GET",
path: ISSUE_PATH,
body: issueBody({
assignees: [{ login: "alexion" }],
milestone: { title: "v2.0" },
}),
},
]);
const { stdout, exitCode } = await runCliTest(
["issue", "view", "42", "--fields", "assignees,milestone"],
{ env: testModeEnv(server.url) },
);
expect(exitCode).toBe(0);
// Default fields are still present; the extra fields are appended.
expect(stdout).toContain("state: open");
expect(stdout).toContain("assignees: alexion");
expect(stdout).toContain("milestone: v2.0");
});
it("renders comment_count: 0 when there are no comments", async () => {
server = await startFixtureServer([
{ method: "GET", path: ISSUE_PATH, body: issueBody({ comments: 0 }) },