Harden the benchmark and cut gitea-axi's agent cost #44

Merged
alexion merged 7 commits from bench-skill-and-cli-cost-fixes into main 2026-07-19 12:40:57 -04:00
4 changed files with 196 additions and 1 deletions
Showing only changes of commit 5422f49f38 - Show all commits

View File

@@ -99,6 +99,8 @@ _Avoid_: depends, depends-on, dependencies
**search**: The full-text query commands (`search issues <query>`, `search prs <query>`), repo-scoped via `owner` param plus [[client-side filtering]] by repository (Gitea's `/repos/issues/search` has no repo-name filter). **search**: The full-text query commands (`search issues <query>`, `search prs <query>`), repo-scoped via `owner` param plus [[client-side filtering]] by repository (Gitea's `/repos/issues/search` has no repo-name filter).
Results use a locator schema (`number`, `title`, `state`, `author`, `created`) — search finds the number; `issue view` / `pr view` load the detail. Results use a locator schema (`number`, `title`, `state`, `author`, `created`) — search finds the number; `issue view` / `pr view` load the detail.
The [[next-step suggestion]] is conditioned on the in-repo match count: zero matches point at the non-indexed `issue list --state all` / `pr list --state all` fallback ("to list all … instead"), which recovers from both an over-narrow query and issue-indexer lag; exactly one match fills the real number (`issue view <n>`, Principle 9's single-id fill); two or more keep the parameterized `<number>` placeholder.
Search never auto-loads the detail even on a single match — it stays a locator (see ADR 0017).
The forbidden `--search` flag on the list commands redirects here. The forbidden `--search` flag on the list commands redirects here.
_Avoid_: query command, find _Avoid_: query command, find

View File

@@ -0,0 +1,26 @@
# Search stays a locator; a single match does not auto-load its detail
`search issues` / `search prs` always return a locator list (`number`, `title`, `state`, `author`, `created`) plus a next-step suggestion — never the full detail, even when exactly one result matches.
This deliberately narrows a literal reading of AXI Principle 4 ("eliminate round trips").
## Considered Options
**Auto-collapse to `view` on a single match** (rejected) — On exactly one result, run `issue view` / `pr view` and return the detail record, sparing the agent a second command.
It reads as the purest Principle 4 outcome, and it is what prompted this decision.
But the agent that searches most often wants the *number* to feed a mutation (`edit`, `close`, `comment`), not the body — so auto-loading the detail spends exactly the body tokens Principle 3's truncation exists to avoid, taxing the common find-then-act path to save a step on the rarer find-then-read one.
It also makes the output shape non-uniform — a list for zero and 2+ matches, a detail record for one — which the agent can no longer rely on.
**Stay a locator, suggest the next step** (chosen) — Search's job is finding the number to feed into `view` / `edit` (the spec's locator-schema rationale).
On a single match the next-step suggestion fills the real number (`issue view 2`), applying Principle 9's single-id fill; the agent decides whether that number feeds a `view`, an `edit`, or a `close`.
## The dividing line
Principle 4 eliminates a *redundant* round trip — a mutation returns the entity it just wrote, so no follow-up `view` is needed (ADR 0008).
`search``view` is not redundant: the follow-up is optional and its intent (read vs. act) is the agent's to choose, so collapsing it means guessing intent and over-fetching when the guess is wrong.
## Consequences
- `search` output shape is uniform across all match counts: always a locator list with a `help[N]:` next step.
- The next-step suggestion is conditioned on the in-repo match count: 0 → `list --state all` fallback ("to list all … instead"); 1 → `view <n>` with the real number; 2+ → `view <number>` placeholder.
- The zero-match fallback points at the non-indexed list, so it recovers from both an over-narrow query and issue-indexer lag without the command having to tell the two apart.
- An agent that does want the detail spends one more command (`view <n>`) by design — the number is already in hand, and it pays only for the detail it actually asks for.

View File

@@ -107,6 +107,10 @@ interface SearchKind {
noun: string; noun: string;
/** The command a matched number feeds into. */ /** The command a matched number feeds into. */
viewCommand: string; viewCommand: string;
/** The list command suggested as the fallback when a search finds nothing. */
listCommand: string;
/** Human plural for the fallback note, e.g. "issues" or "pull requests". */
things: string;
/** The `--help` text for this variant. */ /** The `--help` text for this variant. */
help: string; help: string;
} }
@@ -116,6 +120,8 @@ const SEARCH_ISSUES: SearchKind = {
type: "issues", type: "issues",
noun: "issues", noun: "issues",
viewCommand: "issue view", viewCommand: "issue view",
listCommand: "issue list",
things: "issues",
help: SEARCH_ISSUES_HELP, help: SEARCH_ISSUES_HELP,
}; };
@@ -124,6 +130,8 @@ const SEARCH_PRS: SearchKind = {
type: "pulls", type: "pulls",
noun: "pull_requests", noun: "pull_requests",
viewCommand: "pr view", viewCommand: "pr view",
listCommand: "pr list",
things: "pull requests",
help: SEARCH_PRS_HELP, help: SEARCH_PRS_HELP,
}; };
@@ -217,11 +225,24 @@ async function runSearch(deps: CliDeps, args: string[], kind: SearchKind): Promi
extractRow(issue, [...SEARCH_FIELDS, ...extraFields], { now, host: context.host, full }), extractRow(issue, [...SEARCH_FIELDS, ...extraFields], { now, host: context.host, full }),
); );
// The next-step suggestion is conditioned on the match count. On a miss the
// `view` hint is nonsensical, so point at the non-indexed list as a fallback
// (it recovers from both an over-narrow query and index lag); on a single match
// fill the real number (Principle 9's single-id fill); otherwise leave the
// number parameterized. Search stays a locator either way — it never auto-loads
// the detail (see ADR 0017).
const suggestion =
total === 0
? suggestCommand(context, `${kind.listCommand} --state all`, `to list all ${kind.things} instead`)
: total === 1
? suggestCommand(context, `${kind.viewCommand} ${matches[0]!.number}`, "to see it in full")
: suggestCommand(context, `${kind.viewCommand} <number>`, "to see a match in full");
return renderList({ return renderList({
noun: kind.noun, noun: kind.noun,
rows, rows,
countLine: formatCountLine(rows.length, total, false), countLine: formatCountLine(rows.length, total, false),
help: [suggestCommand(context, `${kind.viewCommand} <number>`, "to see a match in full")], help: [suggestion],
}); });
} }

View File

@@ -402,3 +402,149 @@ describe("search empty results", () => {
}); });
} }
}); });
/**
* The `help[1]:` next-step line is emitted by `suggestCommand`, so it appears
* wrapped in a `Run \`gitea-axi …\`` line with `-R`/`--login` normalization.
* These tests pull that help block out of the output and assert on the
* count-conditional suggestion within it.
*/
function helpBlock(stdout: string): string {
const lines = stdout.split("\n");
const start = lines.findIndex((line) => /^help\[\d+\]:/.test(line));
expect(start).toBeGreaterThanOrEqual(0);
// The block runs from the help[N]: header to the next blank line / EOF.
const rest = lines.slice(start);
const end = rest.findIndex((line, i) => i > 0 && line.trim() === "");
return (end === -1 ? rest : rest.slice(0, end)).join("\n");
}
describe("search issues count-conditional next-step suggestion", () => {
it("suggests the list fallback when there are zero in-repo matches", async () => {
server = await startFixtureServer([
{ method: "GET", path: SEARCH_PATH, headers: { "X-Total-Count": "0" }, body: [] },
]);
const { stdout, exitCode } = await runCliTest(
["search", "issues", "login bug"],
{ env: testModeEnv(server.url) },
);
expect(exitCode).toBe(0);
const help = helpBlock(stdout);
expect(help).toContain("issue list --state all");
expect(help).toContain("to list all issues instead");
expect(help).not.toContain("issue view");
});
it("fills the real number when there is exactly one in-repo match", async () => {
server = await startFixtureServer([
{
method: "GET",
path: SEARCH_PATH,
headers: { "X-Total-Count": "1" },
body: [searchIssueOf(2, { title: "Fix login redirect loop" })],
},
]);
const { stdout, exitCode } = await runCliTest(
["search", "issues", "login bug"],
{ env: testModeEnv(server.url) },
);
expect(exitCode).toBe(0);
const help = helpBlock(stdout);
expect(help).toContain("issue view 2");
expect(help).toContain("to see it in full");
expect(help).not.toContain("issue view <number>");
});
it("keeps the <number> placeholder when there are two or more in-repo matches", async () => {
server = await startFixtureServer([
{
method: "GET",
path: SEARCH_PATH,
headers: { "X-Total-Count": "2" },
body: [
searchIssueOf(42, { title: "Fix login redirect loop" }),
searchIssueOf(41, { title: "Login button unresponsive" }),
],
},
]);
const { stdout, exitCode } = await runCliTest(
["search", "issues", "login bug"],
{ env: testModeEnv(server.url) },
);
expect(exitCode).toBe(0);
const help = helpBlock(stdout);
expect(help).toContain("issue view <number>");
expect(help).toContain("to see a match in full");
});
});
describe("search prs count-conditional next-step suggestion", () => {
it("suggests the list fallback when there are zero in-repo matches", async () => {
server = await startFixtureServer([
{ method: "GET", path: SEARCH_PATH, headers: { "X-Total-Count": "0" }, body: [] },
]);
const { stdout, exitCode } = await runCliTest(
["search", "prs", "flaky ci"],
{ env: testModeEnv(server.url) },
);
expect(exitCode).toBe(0);
const help = helpBlock(stdout);
expect(help).toContain("pr list --state all");
expect(help).toContain("to list all pull requests instead");
expect(help).not.toContain("pr view");
});
it("fills the real number when there is exactly one in-repo match", async () => {
server = await startFixtureServer([
{
method: "GET",
path: SEARCH_PATH,
headers: { "X-Total-Count": "1" },
body: [searchIssueOf(2, { title: "Retry flaky CI jobs" })],
},
]);
const { stdout, exitCode } = await runCliTest(
["search", "prs", "flaky ci"],
{ env: testModeEnv(server.url) },
);
expect(exitCode).toBe(0);
const help = helpBlock(stdout);
expect(help).toContain("pr view 2");
expect(help).toContain("to see it in full");
expect(help).not.toContain("pr view <number>");
});
it("keeps the <number> placeholder when there are two or more in-repo matches", async () => {
server = await startFixtureServer([
{
method: "GET",
path: SEARCH_PATH,
headers: { "X-Total-Count": "2" },
body: [
searchIssueOf(73, { title: "Retry flaky CI jobs" }),
searchIssueOf(72, { title: "Stabilize CI runners" }),
],
},
]);
const { stdout, exitCode } = await runCliTest(
["search", "prs", "flaky ci"],
{ env: testModeEnv(server.url) },
);
expect(exitCode).toBe(0);
const help = helpBlock(stdout);
expect(help).toContain("pr view <number>");
expect(help).toContain("to see a match in full");
});
});