fix: pre-authenticate the gitea-axi bench arm via its env interface
The gitea-axi arm was the only shell arm handed no credentials: the runner set only PATH, so the agent had to reverse-engineer the tea-login system — guessing a profile name and hunting for a config file — before any real work, burning ~4 turns per task. Since turns drive cache-read, the benchmark's dominant cost metric, this scaffolding gap alone inflated gitea-axi's cost-equivalent tokens above every other arm. Hand the arm its host and token through gitea-axi's own env interface (GITEA_AXI_API_URL / GITEA_AXI_TOKEN), the symmetric counterpart to the gitea-mcp server's GITEA_HOST / GITEA_ACCESS_TOKEN env: both name the same two facts, and both still leave the agent to name the repository per call. A shell arm now carries a credential env (empty for tea and raw-api, which need none), merged under PATH in the driver. Also strengthen SKILL.md so a cold agent targets and authenticates on the first call: an explicit "Targeting and authentication" section replaces the buried, optional-looking one-liner, spelling out that outside a checkout `-R OWNER/NAME` plus the environment's token is all that is needed — do not go hunting for a config file or login profile. Verified live: create-memory-leak-issue dropped from 10 turns to 3 and its cache-read fell ~3.8x, with the auth flailing gone from the transcript.
This commit is contained in:
@@ -112,6 +112,32 @@ describe("buildArm", () => {
|
||||
expect(envValues).toContain("s3cr3t-token");
|
||||
});
|
||||
|
||||
// Behavior: the gitea-axi arm's shell is handed a credential environment
|
||||
// carrying the host and token from the shared access, so its tool is
|
||||
// pre-authenticated without the agent having to discover credentials
|
||||
// (benchmark-harness spec, "Scaffolding"). The access here is built from
|
||||
// independent literals; shell.env must deep-equal exactly the two facts echoed
|
||||
// back under their env-var names (host→GITEA_AXI_API_URL, token→GITEA_AXI_TOKEN),
|
||||
// and nothing else. The keys and mapping are fixed by gitea-axi's own env
|
||||
// contract, not recomputed from arm.ts.
|
||||
it("gives the gitea-axi arm's shell a credential env with the shared host URL and token", () => {
|
||||
const preAuthed: SharedContext = {
|
||||
coords: { owner: "acme", repo: "bench-xyz" },
|
||||
access: { apiUrl: "https://git.example.test", token: "tok-abc123" },
|
||||
};
|
||||
|
||||
const definition = buildArm("gitea-axi", preAuthed, { binRoot, locate });
|
||||
|
||||
const shell = definition.shell;
|
||||
expect(shell).not.toBeNull();
|
||||
if (shell === null) return;
|
||||
|
||||
expect(shell.env).toEqual({
|
||||
GITEA_AXI_API_URL: "https://git.example.test",
|
||||
GITEA_AXI_TOKEN: "tok-abc123",
|
||||
});
|
||||
});
|
||||
|
||||
// Behavior: each non-MCP arm's tool/PATH configuration comes from the guard and
|
||||
// exposes only that arm's allowed binary (benchmark-harness spec, "Tool
|
||||
// isolation" / ADR 0016). The (arm, binary) pairs are independent literals —
|
||||
|
||||
Reference in New Issue
Block a user