feat: add benchmark arm scaffolding (task 0026) #27

alexion · 2026-07-16T08:39:58-04:00

alexion commented

2026-07-16 08:39:58 -04:00

Task: .claude/tasks/0026-bench-arm-scaffolding.md

Summary

Adds bench/arm.ts, the per-arm scaffolding that produces the single ArmDefinition the runner (task 0027) consumes for one cell. Two exports:

basePrompt(context) — the identical, task-agnostic base prompt every arm shares, carrying only the repository coordinates, host URL, and token, so it is byte-for-byte identical across arms.
buildArm(arm, context, options) — assembles the arm definition: the fully assembled system prompt plus the tool configuration (shell xor mcp).

The deliberate asymmetries follow the shipped products: the gitea-axi arm embeds the bundled Agent Skill; the tea and raw-api arms get a one-line native-discovery pointer; the gitea-mcp arm runs with the shell disabled and only the MCP server attached (its dispatcher schemas load eagerly). Shell arms' PATH and guard come from bench/guard.ts (provisionArmBin + guardCommand).

Built test-first; bench/arm.test.ts covers all five acceptance criteria (12 tests). Full suite green (387), bench tier green (63), typecheck clean.

Deviations (see the task's Implementation Notes)

AC4's "eager schemas" / "only the MCP tools" are conveyed structurally, not enforced here: eager loading is inherent to attaching the MCP server, and granting only those tools is the runner's SDK wiring (task 0027). The arm definition expresses the intent via shell: null + a populated mcp.
loadSkillBody strips the skill's YAML frontmatter, embedding only the instructional body, so gitea-axi is not overcharged for the description metadata Claude Code loads ambiently for every skill (honest ambient-cost charging, User Story 4).

Review

Risk

Overall: LOW

Blast radius: Low — two new dev-only bench/ files plus one README line; nothing outside the excluded harness imports arm.ts.
Reversibility: Low — brand-new unpublished files, no migrations or API/schema changes.
Test coverage: Low (well-covered) — base prompt, per-arm bootstraps, MCP-only wiring, and shell PATH/guard isolation all exercised with independent literal anchors.
Sensitive domain: Low — a throwaway bench-repo token placed in prompt/env in dev tooling, not production auth.
Size & complexity: Low — ~190 lines of straight-line assembly with a single switch.
Runtime criticality: Low — bench/ is dev-only and excluded from the published npm package.

Standards

No hard violations of a documented standard. The one actionable judgement call — using import.meta.dirname instead of the product's house style (new URL(..., import.meta.url), as in src/commands/setup.ts) — was addressed in this branch. Remaining items are accepted judgement calls (deliberate independent-oracle duplication in tests; the MCP env map matches the official server's own env contract; skillPath/locate are justified testability seams mirroring guard.ts).

Spec

Faithful; AC1/AC2/AC3/AC5 directly implemented. The only remaining gaps are the intentional AC4 deferrals noted above (eager-schema loading and "only the MCP tools" are the runner's SDK responsibility in task 0027) — deferred by design, not missing.

Task: `.claude/tasks/0026-bench-arm-scaffolding.md` ## Summary Adds `bench/arm.ts`, the per-arm scaffolding that produces the single `ArmDefinition` the runner (task 0027) consumes for one cell. Two exports: - `basePrompt(context)` — the identical, task-agnostic base prompt every arm shares, carrying only the repository coordinates, host URL, and token, so it is byte-for-byte identical across arms. - `buildArm(arm, context, options)` — assembles the arm definition: the fully assembled system prompt plus the tool configuration (`shell` xor `mcp`). The deliberate asymmetries follow the shipped products: the **gitea-axi** arm embeds the bundled Agent Skill; the **tea** and **raw-api** arms get a one-line native-discovery pointer; the **gitea-mcp** arm runs with the shell disabled and only the MCP server attached (its dispatcher schemas load eagerly). Shell arms' PATH and guard come from `bench/guard.ts` (`provisionArmBin` + `guardCommand`). Built test-first; `bench/arm.test.ts` covers all five acceptance criteria (12 tests). Full suite green (387), bench tier green (63), typecheck clean. ### Deviations (see the task's Implementation Notes) - **AC4's "eager schemas" / "only the MCP tools" are conveyed structurally**, not enforced here: eager loading is inherent to attaching the MCP server, and granting *only* those tools is the runner's SDK wiring (task 0027). The arm definition expresses the intent via `shell: null` + a populated `mcp`. - **`loadSkillBody` strips the skill's YAML frontmatter**, embedding only the instructional body, so gitea-axi is not overcharged for the `description` metadata Claude Code loads ambiently for every skill (honest ambient-cost charging, User Story 4). ## Review ### Risk **Overall: LOW** - Blast radius: Low — two new dev-only `bench/` files plus one README line; nothing outside the excluded harness imports `arm.ts`. - Reversibility: Low — brand-new unpublished files, no migrations or API/schema changes. - Test coverage: Low (well-covered) — base prompt, per-arm bootstraps, MCP-only wiring, and shell PATH/guard isolation all exercised with independent literal anchors. - Sensitive domain: Low — a throwaway bench-repo token placed in prompt/env in dev tooling, not production auth. - Size & complexity: Low — ~190 lines of straight-line assembly with a single switch. - Runtime criticality: Low — `bench/` is dev-only and excluded from the published npm package. ### Standards No hard violations of a documented standard. The one actionable judgement call — using `import.meta.dirname` instead of the product's house style (`new URL(..., import.meta.url)`, as in `src/commands/setup.ts`) — **was addressed** in this branch. Remaining items are accepted judgement calls (deliberate independent-oracle duplication in tests; the MCP env map matches the official server's own env contract; `skillPath`/`locate` are justified testability seams mirroring `guard.ts`). ### Spec Faithful; AC1/AC2/AC3/AC5 directly implemented. The only remaining gaps are the intentional AC4 deferrals noted above (eager-schema loading and "only the MCP tools" are the runner's SDK responsibility in task 0027) — deferred by design, not missing.

alexion added 1 commit 2026-07-16 08:39:59 -04:00

feat: add benchmark arm scaffolding (task 0026)

CI / test (pull_request) Successful in 50s

Details

CI / test (push) Successful in 52s

Details

9a2ba40657

Add bench/arm.ts, the per-arm scaffolding that produces the single arm
definition the runner consumes. Every arm shares an identical task-agnostic
base prompt and the same repository coordinates and token; each arm then
receives a minimal, symmetric bootstrap.

The deliberate asymmetries follow the shipped products: the gitea-axi arm
embeds the bundled Agent Skill (its body, charging its ambient cost to
gitea-axi); the tea and raw-api arms get a one-line native-discovery pointer;
the gitea-mcp arm runs with the shell disabled and only the MCP server
attached, its dispatcher schemas loading eagerly. Shell arms' PATH and guard
come from bench/guard.ts.

alexion merged commit 9a2ba40657 into main

2026-07-16 08:40:57 -04:00

alexion deleted branch task-0026-bench-arm-scaffolding

2026-07-16 08:40:57 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: alexion/gitea-axi#27