fix: pre-authenticate the gitea-axi bench arm via its env interface

The gitea-axi arm was the only shell arm handed no credentials: the
runner set only PATH, so the agent had to reverse-engineer the tea-login
system — guessing a profile name and hunting for a config file — before
any real work, burning ~4 turns per task. Since turns drive cache-read,
the benchmark's dominant cost metric, this scaffolding gap alone inflated
gitea-axi's cost-equivalent tokens above every other arm.

Hand the arm its host and token through gitea-axi's own env interface
(GITEA_AXI_API_URL / GITEA_AXI_TOKEN), the symmetric counterpart to the
gitea-mcp server's GITEA_HOST / GITEA_ACCESS_TOKEN env: both name the
same two facts, and both still leave the agent to name the repository per
call. A shell arm now carries a credential env (empty for tea and
raw-api, which need none), merged under PATH in the driver.

Also strengthen SKILL.md so a cold agent targets and authenticates on the
first call: an explicit "Targeting and authentication" section replaces
the buried, optional-looking one-liner, spelling out that outside a
checkout `-R OWNER/NAME` plus the environment's token is all that is
needed — do not go hunting for a config file or login profile.

Verified live: create-memory-leak-issue dropped from 10 turns to 3 and
its cache-read fell ~3.8x, with the auth flailing gone from the transcript.
This commit is contained in:
2026-07-17 15:10:48 -04:00
parent 14d494afa2
commit ab59e699c1
4 changed files with 75 additions and 4 deletions

View File

@@ -112,6 +112,32 @@ describe("buildArm", () => {
expect(envValues).toContain("s3cr3t-token");
});
// Behavior: the gitea-axi arm's shell is handed a credential environment
// carrying the host and token from the shared access, so its tool is
// pre-authenticated without the agent having to discover credentials
// (benchmark-harness spec, "Scaffolding"). The access here is built from
// independent literals; shell.env must deep-equal exactly the two facts echoed
// back under their env-var names (host→GITEA_AXI_API_URL, token→GITEA_AXI_TOKEN),
// and nothing else. The keys and mapping are fixed by gitea-axi's own env
// contract, not recomputed from arm.ts.
it("gives the gitea-axi arm's shell a credential env with the shared host URL and token", () => {
const preAuthed: SharedContext = {
coords: { owner: "acme", repo: "bench-xyz" },
access: { apiUrl: "https://git.example.test", token: "tok-abc123" },
};
const definition = buildArm("gitea-axi", preAuthed, { binRoot, locate });
const shell = definition.shell;
expect(shell).not.toBeNull();
if (shell === null) return;
expect(shell.env).toEqual({
GITEA_AXI_API_URL: "https://git.example.test",
GITEA_AXI_TOKEN: "tok-abc123",
});
});
// Behavior: each non-MCP arm's tool/PATH configuration comes from the guard and
// exposes only that arm's allowed binary (benchmark-harness spec, "Tool
// isolation" / ADR 0016). The (arm, binary) pairs are independent literals —