Harden the benchmark and cut gitea-axi's agent cost #44

Merged
alexion merged 7 commits from bench-skill-and-cli-cost-fixes into main 2026-07-19 12:40:57 -04:00
16 changed files with 522 additions and 63 deletions

View File

@@ -99,6 +99,8 @@ _Avoid_: depends, depends-on, dependencies
**search**: The full-text query commands (`search issues <query>`, `search prs <query>`), repo-scoped via `owner` param plus [[client-side filtering]] by repository (Gitea's `/repos/issues/search` has no repo-name filter). **search**: The full-text query commands (`search issues <query>`, `search prs <query>`), repo-scoped via `owner` param plus [[client-side filtering]] by repository (Gitea's `/repos/issues/search` has no repo-name filter).
Results use a locator schema (`number`, `title`, `state`, `author`, `created`) — search finds the number; `issue view` / `pr view` load the detail. Results use a locator schema (`number`, `title`, `state`, `author`, `created`) — search finds the number; `issue view` / `pr view` load the detail.
The [[next-step suggestion]] is conditioned on the in-repo match count: zero matches point at the non-indexed `issue list --state all` / `pr list --state all` fallback ("to list all … instead"), which recovers from both an over-narrow query and issue-indexer lag; exactly one match fills the real number (`issue view <n>`, Principle 9's single-id fill); two or more keep the parameterized `<number>` placeholder.
Search never auto-loads the detail even on a single match — it stays a locator (see ADR 0017).
The forbidden `--search` flag on the list commands redirects here. The forbidden `--search` flag on the list commands redirects here.
_Avoid_: query command, find _Avoid_: query command, find

View File

@@ -0,0 +1,26 @@
# Search stays a locator; a single match does not auto-load its detail
`search issues` / `search prs` always return a locator list (`number`, `title`, `state`, `author`, `created`) plus a next-step suggestion — never the full detail, even when exactly one result matches.
This deliberately narrows a literal reading of AXI Principle 4 ("eliminate round trips").
## Considered Options
**Auto-collapse to `view` on a single match** (rejected) — On exactly one result, run `issue view` / `pr view` and return the detail record, sparing the agent a second command.
It reads as the purest Principle 4 outcome, and it is what prompted this decision.
But the agent that searches most often wants the *number* to feed a mutation (`edit`, `close`, `comment`), not the body — so auto-loading the detail spends exactly the body tokens Principle 3's truncation exists to avoid, taxing the common find-then-act path to save a step on the rarer find-then-read one.
It also makes the output shape non-uniform — a list for zero and 2+ matches, a detail record for one — which the agent can no longer rely on.
**Stay a locator, suggest the next step** (chosen) — Search's job is finding the number to feed into `view` / `edit` (the spec's locator-schema rationale).
On a single match the next-step suggestion fills the real number (`issue view 2`), applying Principle 9's single-id fill; the agent decides whether that number feeds a `view`, an `edit`, or a `close`.
## The dividing line
Principle 4 eliminates a *redundant* round trip — a mutation returns the entity it just wrote, so no follow-up `view` is needed (ADR 0008).
`search``view` is not redundant: the follow-up is optional and its intent (read vs. act) is the agent's to choose, so collapsing it means guessing intent and over-fetching when the guess is wrong.
## Consequences
- `search` output shape is uniform across all match counts: always a locator list with a `help[N]:` next step.
- The next-step suggestion is conditioned on the in-repo match count: 0 → `list --state all` fallback ("to list all … instead"); 1 → `view <n>` with the real number; 2+ → `view <number>` placeholder.
- The zero-match fallback points at the non-indexed list, so it recovers from both an over-narrow query and issue-indexer lag without the command having to tell the two apart.
- An agent that does want the detail spends one more command (`view <n>`) by design — the number is already in hand, and it pays only for the detail it actually asks for.

View File

@@ -13,10 +13,15 @@ The benchmark arms invoke the **built `dist/main.js`** (the `gitea-axi` binary o
Run `npm run build` before any live `bench:run` if you want `src/` changes reflected; the bench does not run from source. Run `npm run build` before any live `bench:run` if you want `src/` changes reflected; the bench does not run from source.
Prefer this project's own CLI for pull requests — it is the tool being built, so opening its PRs with it is the dogfood path: Prefer this project's own CLI for pull requests — it is the tool being built, so opening its PRs with it is the dogfood path:
`npm run build && node dist/main.js pr create --login axi --base main --head <branch> --title <text> --body-file <path>`. `npm run build && node dist/main.js pr create --login alexion --base main --head <branch> --title <text> --body-file <path>`.
The login profile is named `axi`, not `alexion` — passing `--login alexion` fails with `VALIDATION_ERROR`. It reuses the `tea` login store, which here holds exactly `alexion` and `csv-reviewer` — there is no `axi` profile.
It reuses the `tea` login profiles, so it needs no separate credentials. `selectLogin` matches the `--login` value against those names exactly, so `--login alexion` works and an unknown name like `--login axi` fails with `VALIDATION_ERROR` ("Login profile "axi" not found").
Fall back to `tea pr create --login axi --base main --head <branch>` only for what gitea-axi cannot do yet; `tea pr` still lists PRs until `pr list` lands (task 0008). Fall back to `tea pr create --login alexion --base main --head <branch>` only for what gitea-axi cannot do yet; `tea pr` still lists PRs until `pr list` lands (task 0008).
The same login store backs the benchmark: `npm run bench:run -- --arm <arm> --login alexion --task <id>` (or set `GITEA_AXI_BENCH_LOGIN=alexion`).
The `bench/` unit tests only run under their own Vitest project config: `npx vitest run --config vitest.bench.config.ts bench/<file>.test.ts`.
Plain `npx vitest run bench/<file>.test.ts` reports "no tests" because the default `vitest.config.ts` includes only `test/**`.
Task branches are merged into `main` on the remote, so the local `main` goes stale. Task branches are merged into `main` on the remote, so the local `main` goes stale.
Always `git fetch origin` and cut a task branch from `origin/main`, not from whatever local `main` happens to point at. Always `git fetch origin` and cut a task branch from `origin/main`, not from whatever local `main` happens to point at.

View File

@@ -1,8 +1,8 @@
# Benchmark harness # Benchmark harness
This directory holds the benchmark that tests gitea-axi's central claim — that it is an agent-ergonomic, low-token interface to Gitea — against the `tea` CLI, the official `gitea-mcp` server, and raw Gitea REST calls. This directory holds the benchmark that tests gitea-axi's central claim — that it is an agent-ergonomic, low-token interface to Gitea — against the `tea` CLI, the official `gitea-mcp` server, and raw Gitea REST calls.
The result is honest rather than flattering: gitea-axi is the lowest-cost of the *structured* interfaces — it beats both `tea` and `gitea-mcp` on every tier at 100% task success — but hand-rolled raw REST is cheaper still, because terse HTTP is the token floor no wrapper undercuts. The result is that gitea-axi has reached the token floor: it is the cheapest of the structured interfaces by a wide margin and now runs neck-and-neck with hand-rolled raw REST — within ~1% overall — at 100% task success and the fewest turns of any arm.
Keeping the raw-REST arm in the comparison is deliberate: a benchmark of agent-CLIs that omits it will always crown the wrapper, and this one refuses to. Keeping the raw-REST arm in the comparison is deliberate: a benchmark of agent-CLIs that omits the hand-rolled baseline will always flatter the wrapper, and this one refuses to — which is exactly what makes gitea-axi matching that baseline meaningful.
## How it works ## How it works
@@ -15,23 +15,25 @@ Every arm is credentialed the way its product is really configured — the token
| arm | cost-equivalent tokens | raw tokens | turns | success | imputed cost | | arm | cost-equivalent tokens | raw tokens | turns | success | imputed cost |
| --- | ---: | ---: | ---: | ---: | ---: | | --- | ---: | ---: | ---: | ---: | ---: |
| raw-api | 16,971 | 52,586 | 4.3 | 100% | ~$0.11 | | raw-api | 17,613 | 60,384 | 5.0 | 100% | ~$0.11 |
| gitea-axi | 19,240 | 81,067 | 6.0 | 100% | ~$0.12 | | gitea-axi | 17,815 | 62,705 | 4.8 | 100% | ~$0.11 |
| tea | 20,568 | 82,188 | 6.2 | 97% | ~$0.12 | | gitea-mcp | 23,198 | 76,378 | 5.2 | 100% | ~$0.15 |
| gitea-mcp | 21,803 | 79,961 | 5.7 | 100% | ~$0.14 | | tea | 23,210 | 94,600 | 7.2 | 85% | ~$0.14 |
All four arms completed the full matrix — 20 of 20 tasks each, at the reporting floor — and success is near-perfect: only `tea` slips, to 89% on find-then-act, while the other three pass every run. All four arms completed the full matrix — 20 of 20 tasks each — and success is near-perfect: only `tea` slips, to 85% overall (67% on find-then-act), while the other three pass every run.
Raw REST posts the lowest cost-equivalent tokens and leads every tier. Raw REST posts the lowest cost-equivalent tokens, but only barely: gitea-axi lands within ~1% of it (17,815 vs 17,613), a gap well inside the noise of three trials.
It is direct HTTP with the token in the request header, so it takes the fewest turns (4.3) and reads the least cached context, and no higher-level tool beats that on tokens alone. Direct HTTP with the token in the request header is the token floor no wrapper is supposed to undercut — and a structured tool drawing level with it is the headline of this snapshot.
This is the honest ceiling, and the reason gitea-axi does not claim the cost crown outright.
gitea-axi is a clear second overall and the cheapest of the structured tools: it undercuts the official `gitea-mcp` server and the `tea` CLI on every tier, at 100% success, with the lowest output-token count of any arm. gitea-axi is the cheapest structured interface by a wide margin — roughly 23% under both the official `gitea-mcp` server and the `tea` CLI — and it beats both of them on every tier, at 100% success.
Note the split between raw and cost-equivalent tokens — gitea-axi spends more raw tokens than `gitea-mcp` yet costs less, because output is weighted 5× and gitea-axi's answers are compact. It also takes the fewest turns of any arm (4.8, raw REST included) and the lowest output-token count of the shell arms; its compact TOON answers are what let a structured tool run this close to the floor.
By tier, raw REST's edge is widest on reads (10,921 vs gitea-axi's 14,415) — a read is one HTTP request for curl, where a CLI still spends a turn or two — and narrows on multi-step (24,348 vs 26,963), where the work itself dominates and interface overhead matters less. By tier the picture is sharper than the overall total.
gitea-axi is the *cheapest arm outright* on the two discovery-heavy tiers — reads (13,294 vs raw's 14,656) and find-then-act (17,585 vs 18,631) — where finding the right entity is the work, and its compact search and list output beats reconstructing and parsing raw JSON.
Raw REST reclaims the lead on the mutation-heavy tiers — narrowly on single-mutation (15,212 vs 15,552), more clearly on multi-step (22,643 vs 26,079) — where the task is a handful of terse POSTs that no wrapper undercuts.
So the two trade tiers: gitea-axi wins where an interface earns its keep, raw REST wins where the request was already minimal.
Cost parity on the scored suite also understates gitea-axi, because the suite is the subset every arm can do at all. Cost parity on the scored suite also understates gitea-axi, because the suite is the subset every arm can do at all.
The bonus table records capability-asymmetric operations — full-text issue search, rendering a PR's diff and checks, issue dependencies — that gitea-axi handles directly and raw REST has no first-class equivalent for. The bonus table records capability-asymmetric operations — full-text issue search, rendering a PR's diff and checks, issue dependencies — that gitea-axi handles directly and raw REST has no first-class equivalent for.
_Snapshot: 2026-07-17 — 4 arms × 20 tasks × 3 trials each (240 samples), a single clean run with all four arms executed together against one live Gitea host; imputed cost is the mean per-task Anthropic-API-priced dollar cost._ _Snapshot: 2026-07-19 — 4 arms × 20 tasks × 3 trials each (240 samples), a single clean run with all four arms executed together against one live Gitea host; imputed cost is the mean per-task Anthropic-API-priced dollar cost._

View File

@@ -340,6 +340,64 @@ describe("checkReadAnswer", () => {
expect(result.pass).toBe(false); expect(result.pass).toBe(false);
}); });
it("passes via pattern when filler words break every contiguous anyOf phrase", () => {
// A count fact is brittle when pinned to fixed phrases: a human padding the
// answer with filler words ("issues are currently") splits any contiguous
// rendering. The optional `pattern` — a case-insensitive regex source — spans
// that filler as a bounded run of alphabetic words between the number and
// "open", so the fact is satisfied even though no `anyOf` phrase appears verbatim.
const facts: RequiredFact[] = [
{
description: "count of open issues",
anyOf: ["5 open", "5 issues are open"],
pattern: "\\b5(?: [a-z]+){0,4} open\\b",
},
];
// "issues are currently" is inserted between "5" and "open", so neither
// contiguous anyOf phrase matches — but the pattern's alphabetic filler run does.
const report = "5 issues are currently open (5 of 5 total).";
expect(checkReadAnswer(facts, report)).toEqual({ pass: true });
});
it("fails when a digit interrupts the number-to-open run, so a wrong count cannot slip through", () => {
// The same pattern-bearing fact as above. Here the report states a DIFFERENT
// open count (3), merely mentioning the number 5 elsewhere. The pattern's
// filler run is alphabetic only, so the digit "3" between the matched "5" and
// "open" is not spanned, and no anyOf phrase matches either — the fact fails.
const facts: RequiredFact[] = [
{
description: "count of open issues",
anyOf: ["5 open", "5 issues are open"],
pattern: "\\b5(?: [a-z]+){0,4} open\\b",
},
];
// "5 issues in total, and 3 open" — the 5 is a total, the open count is 3.
const report = "There are 5 issues in total, and 3 open.";
const result = checkReadAnswer(facts, report);
expect(result.pass).toBe(false);
// The unmet fact must remain identifiable by its own description.
if (result.pass === false) {
expect(result.differences.some((d) => d.includes("count of open issues"))).toBe(true);
}
});
it("still matches on anyOf alone when a fact carries no pattern", () => {
// A fact without `pattern` behaves exactly as before: only the contiguous
// anyOf renderings are consulted, unaffected by the new pattern support.
const facts: RequiredFact[] = [
{ description: "count of open issues", anyOf: ["7 open issues", "seven open issues"] },
];
const report = "The board shows 7 open issues right now.";
expect(checkReadAnswer(facts, report)).toEqual({ pass: true });
});
}); });
describe("score", () => { describe("score", () => {

View File

@@ -241,9 +241,7 @@ function matchByKey<T>(
*/ */
export function checkReadAnswer(facts: RequiredFact[], report: string): CheckResult { export function checkReadAnswer(facts: RequiredFact[], report: string): CheckResult {
const haystack = normalizeText(report); const haystack = normalizeText(report);
const missing = facts.filter( const missing = facts.filter((fact) => !factPresent(fact, haystack));
(fact) => !fact.anyOf.some((rendering) => haystack.includes(normalizeText(rendering))),
);
if (missing.length === 0) { if (missing.length === 0) {
return { pass: true }; return { pass: true };
} }
@@ -253,6 +251,20 @@ export function checkReadAnswer(facts: RequiredFact[], report: string): CheckRes
}; };
} }
/**
* Whether a required fact is present in the normalized report: any `anyOf`
* rendering as a contiguous substring, or the optional `pattern` regex matching.
* The pattern is compiled case-insensitively over the already-normalized text, so
* it recognises a count padded with filler ("5 issues are currently open") that no
* fixed phrase would.
*/
function factPresent(fact: RequiredFact, haystack: string): boolean {
if (fact.anyOf.some((rendering) => haystack.includes(normalizeText(rendering)))) {
return true;
}
return fact.pattern !== undefined && new RegExp(fact.pattern, "i").test(haystack);
}
/** /**
* Lower-case, drop markdown emphasis/code markers, and collapse runs of * Lower-case, drop markdown emphasis/code markers, and collapse runs of
* whitespace so incidental phrasing and formatting do not matter — a report that * whitespace so incidental phrasing and formatting do not matter — a report that

View File

@@ -57,6 +57,8 @@ export interface RunArgs {
turnCap: number; turnCap: number;
wallClockMs: number; wallClockMs: number;
storeRoot: string; storeRoot: string;
/** Optional bundled-skill override for the gitea-axi arm; defaults to the shipped SKILL.md. */
skillPath?: string;
} }
/** The parse outcome: a request for help, or a resolved configuration to run. */ /** The parse outcome: a request for help, or a resolved configuration to run. */
@@ -71,6 +73,7 @@ const KNOWN_FLAGS = new Set([
"turn-cap", "turn-cap",
"wall-clock-ms", "wall-clock-ms",
"store", "store",
"skill",
]); ]);
/** A usage error, surfaced to the maintainer with the offending detail. */ /** A usage error, surfaced to the maintainer with the offending detail. */
@@ -157,6 +160,7 @@ export function parseRunArgs(
turnCap, turnCap,
wallClockMs, wallClockMs,
storeRoot: flags.get("store") ?? DEFAULT_STORE_ROOT, storeRoot: flags.get("store") ?? DEFAULT_STORE_ROOT,
...(flags.has("skill") ? { skillPath: flags.get("skill") } : {}),
}; };
} }
@@ -180,6 +184,7 @@ Options:
--turn-cap <n> Per-run turn cap (default: ${DEFAULT_TURN_CAP}) --turn-cap <n> Per-run turn cap (default: ${DEFAULT_TURN_CAP})
--wall-clock-ms <n> Per-run wall-clock backstop in ms (default: ${DEFAULT_WALL_CLOCK_MS}) --wall-clock-ms <n> Per-run wall-clock backstop in ms (default: ${DEFAULT_WALL_CLOCK_MS})
--store <dir> Sample store root (default: ${DEFAULT_STORE_ROOT}) --store <dir> Sample store root (default: ${DEFAULT_STORE_ROOT})
--skill <path> Override the gitea-axi arm's bundled skill (default: shipped SKILL.md)
-h, --help Show this help`; -h, --help Show this help`;
/** Render the run-loop tally into the lines printed after a sitting. */ /** Render the run-loop tally into the lines printed after a sitting. */
@@ -246,7 +251,7 @@ export async function runBenchCommand(
driver: sdkAgentDriver(), driver: sdkAgentDriver(),
store, store,
bounds: { turnCap: parsed.turnCap, wallClockMs: parsed.wallClockMs }, bounds: { turnCap: parsed.turnCap, wallClockMs: parsed.wallClockMs },
build: { binRoot }, build: { binRoot, ...(parsed.skillPath !== undefined ? { skillPath: parsed.skillPath } : {}) },
}); });
for (const line of summarize(result, parsed.storeRoot)) { for (const line of summarize(result, parsed.storeRoot)) {
out(line); out(line);

View File

@@ -131,10 +131,21 @@ export interface RepoState {
* whitespace and case normalization), so a count or a name can be phrased * whitespace and case normalization), so a count or a name can be phrased
* variously without resorting to an LLM judge. `description` names the fact in * variously without resorting to an LLM judge. `description` names the fact in
* diagnostics when it is missing. * diagnostics when it is missing.
*
* `anyOf` matches a *contiguous* substring, which is brittle for facts a human
* naturally pads with filler — "5 issues are currently open" does not contain
* the fixed phrase "5 issues are open". For those, supply `pattern`: a regular
* expression (matched against the same normalized report) that satisfies the
* fact when it matches, so the count itself can be recognised rather than one
* exact wording. A fact is present when any `anyOf` rendering *or* `pattern`
* matches; `anyOf` stays the human-readable renderings even when `pattern`
* carries the real matcher.
*/ */
export interface RequiredFact { export interface RequiredFact {
description: string; description: string;
anyOf: string[]; anyOf: string[];
/** Optional regex (source, matched case-insensitively against the normalized report). */
pattern?: string;
} }
/** /**

View File

@@ -359,12 +359,122 @@ async function ensurePullRequest(
await ensureReviews(access, coords, number, pr.reviews); await ensureReviews(access, coords, number, pr.reviews);
} }
/** How long the readiness gate polls before giving up, and how often it re-checks. */
const READINESS_TIMEOUT_MS = 60_000;
const READINESS_INTERVAL_MS = 750;
/** Resolve after `ms` milliseconds. */
function delay(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
/** A search hit's minimal shape: the repository the matched issue/pull belongs to. */
interface SearchHit {
repository?: { name?: string } | null;
}
/**
* Whether the issue indexer has caught up: a full-text search for `title` of the
* given kind returns a hit in this repository. The endpoint spans every repo the
* owner can access (mirroring `gitea-axi search`), so a hit only counts when its
* repository matches. A non-2xx or a not-yet-indexed title reads as not-ready.
*/
async function searchIndexed(
access: BenchAccess,
coords: RepoCoords,
type: "issues" | "pulls",
title: string,
): Promise<boolean> {
const query = new URLSearchParams({ q: title, type, owner: coords.owner, state: "all", limit: "50" });
const res = await request(access, "GET", `/repos/issues/search?${query.toString()}`);
if (!res.ok) {
return false;
}
const hits = (await res.json()) as SearchHit[];
return hits.some((hit) => hit.repository?.name === coords.repo);
}
/**
* The reason a freshly seeded repository is not yet ready for the agent, or `null`
* when it is. Ready means an independent read sees the repository and its full
* seeded issue and pull spread, and the issue indexer returns a seeded issue and
* pull — the two consistency windows (repo visibility and index lag) an agent
* would otherwise race and fail against. Every read is non-throwing, so a
* transient error reads as not-ready and is retried rather than propagated.
*/
async function readinessGap(access: BenchAccess, coords: RepoCoords): Promise<string | null> {
const repoPath = `/repos/${coords.owner}/${coords.repo}`;
const repoRes = await request(access, "GET", repoPath);
if (!repoRes.ok) {
return `repository read returned ${repoRes.status}`;
}
const issuesRes = await request(access, "GET", `${repoPath}/issues?type=issues&state=all&limit=100`);
if (!issuesRes.ok) {
return `issue list returned ${issuesRes.status}`;
}
const issues = (await issuesRes.json()) as GiteaIssue[];
if (issues.length < SEED_PLAN.issues.length) {
return `only ${issues.length}/${SEED_PLAN.issues.length} issues visible`;
}
const pullsRes = await request(access, "GET", `${repoPath}/pulls?state=all&limit=100`);
if (!pullsRes.ok) {
return `pull list returned ${pullsRes.status}`;
}
const pulls = (await pullsRes.json()) as GiteaPull[];
if (pulls.length < SEED_PLAN.pullRequests.length) {
return `only ${pulls.length}/${SEED_PLAN.pullRequests.length} pull requests visible`;
}
const sampleIssue = SEED_PLAN.issues[0]!.title;
if (!(await searchIndexed(access, coords, "issues", sampleIssue))) {
return `issue "${sampleIssue}" not yet indexed for search`;
}
const samplePull = SEED_PLAN.pullRequests[0]!.title;
if (!(await searchIndexed(access, coords, "pulls", samplePull))) {
return `pull request "${samplePull}" not yet indexed for search`;
}
return null;
}
/**
* Block until a freshly seeded repository is fully consistent for the agent, or
* fail loudly if it never settles within the timeout. Gitea makes a just-created
* repository and its just-written issues and pulls visible to the seed's own
* writes immediately, but an independent reader — the agent, a fresh process
* moments later — can hit a brief 404 window on the repository and a longer lag on
* the async issue indexer. Both are the benchmark's races, not the tool's, so
* closing them here keeps a scored run measuring gitea-axi rather than host
* propagation. A timeout is a harness failure surfaced to the maintainer, never a
* scored agent failure.
*/
export async function waitForSeedReady(
access: BenchAccess,
coords: RepoCoords,
timeoutMs = READINESS_TIMEOUT_MS,
intervalMs = READINESS_INTERVAL_MS,
): Promise<void> {
const deadline = Date.now() + timeoutMs;
let gap = await readinessGap(access, coords);
while (gap !== null) {
if (Date.now() >= deadline) {
throw new Error(
`seeded repository ${coords.owner}/${coords.repo} not ready after ${timeoutMs}ms: ${gap}`,
);
}
await delay(intervalMs);
gap = await readinessGap(access, coords);
}
}
/** /**
* Seed a freshly provisioned repository to the ground truth, idempotently. Labels * Seed a freshly provisioned repository to the ground truth, idempotently. Labels
* come first (so issues and pull requests can apply them), then the issue spread, * come first (so issues and pull requests can apply them), then the issue spread,
* then the pull requests. Returns the deterministic ground-truth RepoState the * then the pull requests. Returns the deterministic ground-truth RepoState the
* checker scores against; on a fresh repository the created numbers match it, * checker scores against; on a fresh repository the created numbers match it,
* and a re-run leaves them unchanged. * and a re-run leaves them unchanged.
*
* Before returning, it waits out the repository-visibility and issue-indexer
* consistency windows (see waitForSeedReady), so the agent that runs next never
* races a repository that is not yet readable or searchable.
*/ */
export async function seedRepo(access: BenchAccess, coords: RepoCoords): Promise<RepoState> { export async function seedRepo(access: BenchAccess, coords: RepoCoords): Promise<RepoState> {
const user = await currentUser(access); const user = await currentUser(access);
@@ -384,6 +494,7 @@ export async function seedRepo(access: BenchAccess, coords: RepoCoords): Promise
await ensurePullRequest(access, coords, pr, labelIds, pullsByTitle); await ensurePullRequest(access, coords, pr, labelIds, pullsByTitle);
} }
await waitForSeedReady(access, coords);
return groundTruth(user); return groundTruth(user);
} }

View File

@@ -92,6 +92,11 @@ function readTasks(): BenchTask[] {
{ {
description: "the repository has 5 open issues", description: "the repository has 5 open issues",
anyOf: ["5 open", "five open", "open issues: 5", "open: 5", "5 issues are open"], anyOf: ["5 open", "five open", "open issues: 5", "open: 5", "5 issues are open"],
// "5" followed by "open" across up to four alphabetic filler words, so
// natural padding ("5 issues are currently open") is recognised while a
// digit between them (a wrong "5 issues, 3 open") is not: the filler run
// is alphabetic only, so it cannot span another count.
pattern: "\\b5(?: [a-z]+){0,4} open\\b",
}, },
]), ]),
}, },

View File

@@ -6,43 +6,34 @@ description: Use when working with a Gitea repository's issues, pull requests, l
# gitea-axi # gitea-axi
`gitea-axi` is an agent-ergonomic CLI for a Gitea repository's issues and pull requests. `gitea-axi` is an agent-ergonomic CLI for a Gitea repository's issues and pull requests.
Its output is compact TOON built for another program to read, and its errors are structured with actionable suggestions. Its output is compact TOON meant to be read directly, and a failed command's error names the fix — follow that suggestion rather than guessing at another command.
## When to use it
Reach for `gitea-axi` whenever a task touches a Gitea repository's issues, pull requests, labels, or reviews.
- **Over `tea`:** `gitea-axi` returns structured output and typed errors instead of human-formatted tables, and it defaults the repository and login from the local checkout.
- **Over raw Gitea API calls:** it handles auth, pagination, name-to-ID resolution, and review-decision aggregation for you, so you do not hand-roll HTTP.
- **Over improvised `git`:** for anything about issues or pull requests as entities (state, reviews, labels, comments) rather than local commits and branches.
## Targeting and authentication ## Targeting and authentication
Every command resolves two things: which repository to act on, and which credentials to authenticate with. Every command resolves a repository and credentials; getting both right on the first call is the difference between one command and a retry.
Get both right on the first call — they are the usual reason a command fails and has to be retried.
- **Repository.** Inside a Gitea checkout it is taken from the `origin` remote automatically. - **Repository.** Inside a Gitea checkout it comes from the `origin` remote.
Outside a checkout you must name it: pass `-R OWNER/NAME` on every command (or set `GITEA_AXI_REPO=OWNER/NAME` once for the session). Outside one, pass `-R OWNER/NAME` on every command, or set `GITEA_AXI_REPO=OWNER/NAME` once for the session.
- **Credentials.** When the environment is pre-configured — `GITEA_AXI_TOKEN` together with `GITEA_AXI_API_URL` authentication is automatic and you need nothing more. - **Credentials.** With `GITEA_AXI_TOKEN` and `GITEA_AXI_API_URL` set, authentication is automatic.
Otherwise credentials come from a `tea` login: pass `--login <name>` (or set `GITEA_AXI_LOGIN=<name>`) unless the checkout's remote already selects one. Otherwise pass `--login <name>`, or set `GITEA_AXI_LOGIN`.
So outside a checkout with the token in the environment, `gitea-axi <command> -R OWNER/NAME …` is all you need; do not go hunting for a config file or a login profile. Outside a checkout with the token in the environment, `gitea-axi <command> -R OWNER/NAME …` is the whole invocation — don't look for a config file or a login profile.
## Command groups ## Commands
- `issue` — list, view, create, comment on, edit, close/reopen, pin, and link issues. - `issue` — list, view, create, comment, edit, close, reopen, pin, and link (blocks / blocked-by).
- `pr`create, view, comment on, edit, review, merge, check out, diff, and inspect the checks of pull requests. - `pr`list, view, create, comment, edit, review, merge, close, reopen, diff, checks, and checkout.
- `label` — list, create, edit, and delete labels. - `label` — list, create, edit, delete.
- `search` — full-text search; it takes a subcommand, so search issues with `search issues "<query>"` and pull requests with `search prs "<query>"` (a bare `search "<query>"` is not valid). - `search issues "<query>"` and `search prs "<query>"` — full-text search (a bare `search "<query>"` is not valid).
- `setup` — install this skill (`setup`) and, opt-in, the SessionStart dashboard hook (`setup hooks`).
To read one issue's fields, reach straight for `issue view <number>`: it shows labels and state by default, and takes `--fields assignees,milestone,…` for the rest. ## Finding and acting
You rarely need `issue list` to answer a question about a single issue.
## Discovery Find the target, then act on it — two commands, not a survey of the repository.
This skill is a pointer, not a command reference — the CLI is the single source of truth for its own interface. - **Find it.** If you already know the number, act on it directly.
Otherwise reach for one command — `search issues "<query>"` for a title or keyword, or `issue list --state all --label <name>` to narrow by a property — not both.
- Run `gitea-axi` with no arguments for the repository dashboard (open issues and pull requests). - **Read one issue or PR.** `issue view <number>` (or `pr view <number>`) shows labels and state by default, and takes `--fields assignees,milestone,…` for the rest.
Add `--full` for the open-PR table and issue counts by label. You do not need `issue list` to answer a question about a single known issue.
- Run `gitea-axi <command> --help` (or `gitea-axi <group> <command> --help`) for the exact flags of any command. - **Act on it.** `issue edit <number>` and `pr edit <number>` change fields with repeatable `--add-label` / `--remove-label` and `--add-assignee` / `--remove-assignee`, plus `--title`, `--body`, and `--milestone`.
Reviewing is `pr review <number>` with exactly one of `--approve`, `--request-changes`, or `--comment`, and an optional `--body`.
A comment is `issue comment <number> --body <text>`; a new label is `label create --name <text> --color <hex-without-#>`.

View File

@@ -107,6 +107,10 @@ interface SearchKind {
noun: string; noun: string;
/** The command a matched number feeds into. */ /** The command a matched number feeds into. */
viewCommand: string; viewCommand: string;
/** The list command suggested as the fallback when a search finds nothing. */
listCommand: string;
/** Human plural for the fallback note, e.g. "issues" or "pull requests". */
things: string;
/** The `--help` text for this variant. */ /** The `--help` text for this variant. */
help: string; help: string;
} }
@@ -116,6 +120,8 @@ const SEARCH_ISSUES: SearchKind = {
type: "issues", type: "issues",
noun: "issues", noun: "issues",
viewCommand: "issue view", viewCommand: "issue view",
listCommand: "issue list",
things: "issues",
help: SEARCH_ISSUES_HELP, help: SEARCH_ISSUES_HELP,
}; };
@@ -124,6 +130,8 @@ const SEARCH_PRS: SearchKind = {
type: "pulls", type: "pulls",
noun: "pull_requests", noun: "pull_requests",
viewCommand: "pr view", viewCommand: "pr view",
listCommand: "pr list",
things: "pull requests",
help: SEARCH_PRS_HELP, help: SEARCH_PRS_HELP,
}; };
@@ -217,11 +225,24 @@ async function runSearch(deps: CliDeps, args: string[], kind: SearchKind): Promi
extractRow(issue, [...SEARCH_FIELDS, ...extraFields], { now, host: context.host, full }), extractRow(issue, [...SEARCH_FIELDS, ...extraFields], { now, host: context.host, full }),
); );
// The next-step suggestion is conditioned on the match count. On a miss the
// `view` hint is nonsensical, so point at the non-indexed list as a fallback
// (it recovers from both an over-narrow query and index lag); on a single match
// fill the real number (Principle 9's single-id fill); otherwise leave the
// number parameterized. Search stays a locator either way — it never auto-loads
// the detail (see ADR 0017).
const suggestion =
total === 0
? suggestCommand(context, `${kind.listCommand} --state all`, `to list all ${kind.things} instead`)
: total === 1
? suggestCommand(context, `${kind.viewCommand} ${matches[0]!.number}`, "to see it in full")
: suggestCommand(context, `${kind.viewCommand} <number>`, "to see a match in full");
return renderList({ return renderList({
noun: kind.noun, noun: kind.noun,
rows, rows,
countLine: formatCountLine(rows.length, total, false), countLine: formatCountLine(rows.length, total, false),
help: [suggestCommand(context, `${kind.viewCommand} <number>`, "to see a match in full")], help: [suggestion],
}); });
} }

View File

@@ -60,6 +60,17 @@ function hostnameOf(url: string, origin: string): string {
} }
} }
/**
* Normalize a Gitea base URL to the host root the client expects. The client
* (gitea-js) appends `/api/v1` itself, so a value that already carries it — a
* natural guess when the variable is literally named `..._API_URL` — would double
* the segment and 404 as a spurious `REPO_NOT_FOUND`. Strip a trailing `/api/v1`
* (with any trailing slashes) so the host base and the API endpoint both work.
*/
function normalizeApiBase(url: string): string {
return url.replace(/\/+$/, "").replace(/\/api\/v1$/, "");
}
function resolveTestModeContext( function resolveTestModeContext(
deps: CliDeps, deps: CliDeps,
apiUrl: string, apiUrl: string,
@@ -72,10 +83,11 @@ function resolveTestModeContext(
["Set `GITEA_AXI_REPO=OWNER/NAME` or pass `-R OWNER/NAME`"], ["Set `GITEA_AXI_REPO=OWNER/NAME` or pass `-R OWNER/NAME`"],
); );
} }
const base = normalizeApiBase(apiUrl);
return { return {
...parseRepoSpec(overrides.repoSpec, overrides.repoOrigin), ...parseRepoSpec(overrides.repoSpec, overrides.repoOrigin),
host: hostnameOf(apiUrl, "`GITEA_AXI_API_URL`"), host: hostnameOf(base, "`GITEA_AXI_API_URL`"),
apiUrl: apiUrl.replace(/\/+$/, ""), apiUrl: base,
token: deps.env.GITEA_AXI_TOKEN ?? "", token: deps.env.GITEA_AXI_TOKEN ?? "",
repoSource: overrides.repoSource, repoSource: overrides.repoSource,
loginSource: overrides.loginSource, loginSource: overrides.loginSource,
@@ -184,7 +196,7 @@ export async function resolveRepoContext(deps: CliDeps): Promise<RepoContext> {
owner, owner,
name, name,
host, host,
apiUrl: login.url.replace(/\/+$/, ""), apiUrl: normalizeApiBase(login.url),
token, token,
repoSource: overrides.repoSource, repoSource: overrides.repoSource,
loginSource: overrides.loginSource, loginSource: overrides.loginSource,

View File

@@ -1,4 +1,6 @@
import { afterEach, describe, expect, it } from "vitest"; import { afterEach, describe, expect, it } from "vitest";
import { resolveRepoContext } from "../src/context.js";
import type { CliDeps } from "../src/deps.js";
import { startFixtureServer, type FixtureServer } from "./fixture-server.js"; import { startFixtureServer, type FixtureServer } from "./fixture-server.js";
import { runCliTest, testModeEnv } from "./harness.js"; import { runCliTest, testModeEnv } from "./harness.js";
@@ -117,3 +119,49 @@ describe("context overrides", () => {
expect(exitCode).toBe(0); expect(exitCode).toBe(0);
}); });
}); });
describe("apiUrl normalization", () => {
function depsWithApiUrl(apiUrl: string): CliDeps {
return {
env: {
GITEA_AXI_API_URL: apiUrl,
GITEA_AXI_REPO: "acme/widgets",
GITEA_AXI_TOKEN: "test-token",
},
cwd: process.cwd(),
globals: {},
};
}
it("strips a trailing /api/v1 suffix from the host base", async () => {
const context = await resolveRepoContext(
depsWithApiUrl("https://git.example.com/api/v1"),
);
expect(context.apiUrl).toBe("https://git.example.com");
});
it("strips a trailing /api/v1/ with a trailing slash", async () => {
const context = await resolveRepoContext(
depsWithApiUrl("https://git.example.com/api/v1/"),
);
expect(context.apiUrl).toBe("https://git.example.com");
});
it("leaves a host base without an /api/v1 suffix unchanged", async () => {
const context = await resolveRepoContext(
depsWithApiUrl("https://git.example.com"),
);
expect(context.apiUrl).toBe("https://git.example.com");
});
it("strips a lone trailing slash from the host base", async () => {
const context = await resolveRepoContext(
depsWithApiUrl("https://git.example.com/"),
);
expect(context.apiUrl).toBe("https://git.example.com");
});
});

View File

@@ -402,3 +402,149 @@ describe("search empty results", () => {
}); });
} }
}); });
/**
* The `help[1]:` next-step line is emitted by `suggestCommand`, so it appears
* wrapped in a `Run \`gitea-axi …\`` line with `-R`/`--login` normalization.
* These tests pull that help block out of the output and assert on the
* count-conditional suggestion within it.
*/
function helpBlock(stdout: string): string {
const lines = stdout.split("\n");
const start = lines.findIndex((line) => /^help\[\d+\]:/.test(line));
expect(start).toBeGreaterThanOrEqual(0);
// The block runs from the help[N]: header to the next blank line / EOF.
const rest = lines.slice(start);
const end = rest.findIndex((line, i) => i > 0 && line.trim() === "");
return (end === -1 ? rest : rest.slice(0, end)).join("\n");
}
describe("search issues count-conditional next-step suggestion", () => {
it("suggests the list fallback when there are zero in-repo matches", async () => {
server = await startFixtureServer([
{ method: "GET", path: SEARCH_PATH, headers: { "X-Total-Count": "0" }, body: [] },
]);
const { stdout, exitCode } = await runCliTest(
["search", "issues", "login bug"],
{ env: testModeEnv(server.url) },
);
expect(exitCode).toBe(0);
const help = helpBlock(stdout);
expect(help).toContain("issue list --state all");
expect(help).toContain("to list all issues instead");
expect(help).not.toContain("issue view");
});
it("fills the real number when there is exactly one in-repo match", async () => {
server = await startFixtureServer([
{
method: "GET",
path: SEARCH_PATH,
headers: { "X-Total-Count": "1" },
body: [searchIssueOf(2, { title: "Fix login redirect loop" })],
},
]);
const { stdout, exitCode } = await runCliTest(
["search", "issues", "login bug"],
{ env: testModeEnv(server.url) },
);
expect(exitCode).toBe(0);
const help = helpBlock(stdout);
expect(help).toContain("issue view 2");
expect(help).toContain("to see it in full");
expect(help).not.toContain("issue view <number>");
});
it("keeps the <number> placeholder when there are two or more in-repo matches", async () => {
server = await startFixtureServer([
{
method: "GET",
path: SEARCH_PATH,
headers: { "X-Total-Count": "2" },
body: [
searchIssueOf(42, { title: "Fix login redirect loop" }),
searchIssueOf(41, { title: "Login button unresponsive" }),
],
},
]);
const { stdout, exitCode } = await runCliTest(
["search", "issues", "login bug"],
{ env: testModeEnv(server.url) },
);
expect(exitCode).toBe(0);
const help = helpBlock(stdout);
expect(help).toContain("issue view <number>");
expect(help).toContain("to see a match in full");
});
});
describe("search prs count-conditional next-step suggestion", () => {
it("suggests the list fallback when there are zero in-repo matches", async () => {
server = await startFixtureServer([
{ method: "GET", path: SEARCH_PATH, headers: { "X-Total-Count": "0" }, body: [] },
]);
const { stdout, exitCode } = await runCliTest(
["search", "prs", "flaky ci"],
{ env: testModeEnv(server.url) },
);
expect(exitCode).toBe(0);
const help = helpBlock(stdout);
expect(help).toContain("pr list --state all");
expect(help).toContain("to list all pull requests instead");
expect(help).not.toContain("pr view");
});
it("fills the real number when there is exactly one in-repo match", async () => {
server = await startFixtureServer([
{
method: "GET",
path: SEARCH_PATH,
headers: { "X-Total-Count": "1" },
body: [searchIssueOf(2, { title: "Retry flaky CI jobs" })],
},
]);
const { stdout, exitCode } = await runCliTest(
["search", "prs", "flaky ci"],
{ env: testModeEnv(server.url) },
);
expect(exitCode).toBe(0);
const help = helpBlock(stdout);
expect(help).toContain("pr view 2");
expect(help).toContain("to see it in full");
expect(help).not.toContain("pr view <number>");
});
it("keeps the <number> placeholder when there are two or more in-repo matches", async () => {
server = await startFixtureServer([
{
method: "GET",
path: SEARCH_PATH,
headers: { "X-Total-Count": "2" },
body: [
searchIssueOf(73, { title: "Retry flaky CI jobs" }),
searchIssueOf(72, { title: "Stabilize CI runners" }),
],
},
]);
const { stdout, exitCode } = await runCliTest(
["search", "prs", "flaky ci"],
{ env: testModeEnv(server.url) },
);
expect(exitCode).toBe(0);
const help = helpBlock(stdout);
expect(help).toContain("pr view <number>");
expect(help).toContain("to see a match in full");
});
});

View File

@@ -36,19 +36,23 @@ describe("bundled Agent Skill markdown", () => {
it("references each command group as a one-liner", () => { it("references each command group as a one-liner", () => {
const body = skill.toLowerCase(); const body = skill.toLowerCase();
for (const group of ["issue", "pr", "label", "search", "setup"]) { for (const group of ["issue", "pr", "label", "search"]) {
expect(body, `expected the skill to mention the ${group} command group`).toContain( expect(body, `expected the skill to mention the ${group} command group`).toContain(
group, group,
); );
} }
}); });
it("points at the bare dashboard and per-command help for discovery", () => { it("steers find-then-act and does not push exploratory discovery", () => {
const body = skill.toLowerCase(); const body = skill.toLowerCase();
// Bare dashboard: running the binary with no arguments. // The intended steering: find the target, then act on it.
expect(body).toContain("no argument"); expect(body).toContain("find the target");
expect(body).toContain("dashboard"); expect(body).toContain("act on it");
// Per-command help. // The find/act discipline: not a full survey, and not both find commands at once.
expect(body).toContain("--help"); expect(body).toContain("not a survey");
expect(body).toContain("not both");
// The removed exploratory anti-pattern must be gone.
expect(body).not.toContain("dashboard");
expect(body).not.toContain("no argument");
}); });
}); });