feat: add benchmark run-loop command (task 0029)

Add the maintainer-facing command that runs one chosen benchmark cell on
demand, so only the token budget available at that moment is spent.

runCells (bench/run-loop.ts) runs one (arm, task) cell for a batch of
trials — defaulting to five with a reporting floor of three — by driving
the existing single-cell runner and the append-only sample store rather
than reimplementing orchestration. Re-running a cell deepens it: trial
numbering continues past the highest trial the cell already holds and the
new samples append, so a cell's sample size grows across sittings without
overwriting prior runs.

bench/run.ts is the command: parseRunArgs is the pure, unit-tested
argument seam, and runBenchCommand is the live boundary that resolves host
access, resolves the scored suite against the host's self-review support,
selects the task, and drives the run loop. It is invoked via the new
bench:run npm script, run under tsx (a new devDependency) because the
harness's .js-specifier imports need a TypeScript-aware runner. The Claude
Agent SDK is now declared as an optional peerDependency — documented but
neither installed for package consumers nor pulled into CI.

Every arm runs on the driver's single fixed model; the command exposes no
per-cell model override that could break cross-arm comparability. The
default store root bench/results/ is gitignored.

.claude/tasks/0029-bench-run-loop-cli.md

@@ -11,7 +11,34 @@ Each cell defaults to five trials with a reporting floor of three. Because resul
 
 ## Acceptance criteria
 
-- [ ] The command runs a single selected `(arm, task)` cell on demand.
-- [ ] A cell defaults to five trials, and the reporting floor of three is respected.
-- [ ] Re-running an already-sampled cell appends new trials rather than overwriting prior samples.
-- [ ] The command drives the runner and store built in earlier slices rather than reimplementing orchestration.
+- [x] The command runs a single selected `(arm, task)` cell on demand.
+- [x] A cell defaults to five trials, and the reporting floor of three is respected.
+- [x] Re-running an already-sampled cell appends new trials rather than overwriting prior samples.
+- [x] The command drives the runner and store built in earlier slices rather than reimplementing orchestration.
+
+## Implementation Notes
+
+**Two seams, matching the harness's established split.**
+The pure orchestration is `runCells` in `bench/run-loop.ts`: it decides only how many trials to run and at what trial numbers, then delegates provision/run/score/append to `runCell` and the sample store — it reimplements no orchestration (criterion 4).
+The maintainer command is `bench/run.ts`: `parseRunArgs` is the pure, unit-tested argument seam, and `runBenchCommand`/`main` are the live boundary (resolve credentials, probe self-review, select the task, drive `runCells`).
+Following `bench/README.md`'s convention, the live boundary is validated by running it rather than by mocked unit tests — and, deliberately, by reusing pieces already smoke-covered (`runCell` via the runner smoke, `detectSelfReviewSupport` via the self-review smoke, `liveBenchHost` via the seed smoke) rather than adding a new smoke test that would spend real tokens on every invocation.
+
+**Deepening by highest trial number, not sample count.**
+`runCells` continues numbering from `max(existing trial) + 1`, not from the sample count, so an earlier invalid attempt (which records no sample and leaves a gap) can never cause a later sitting to reuse a trial number.
+
+**TDD sequencing deviation.**
+The run loop is a small cohesive unit, so its first GREEN already carried the trial-numbering and floor logic; tests 2 and 3 (deepening, invalid/floor) and the parser override/reject/help tests are therefore passing characterization/regression tests rather than red-first cycles.
+Each was still written test-first by the test-writer sub-agent, from the public interface only, with independent expected literals — they remain discriminating guards.
+
+**Running the harness's TypeScript.**
+Node's native type stripping does not rewrite `.js` import specifiers to `.ts`, which the whole `bench/` tree relies on, so the command runs under `tsx` (added as a devDependency) via `npm run bench:run`.
+The Claude Agent SDK is now formally declared — as an *optional* `peerDependency` (`@anthropic-ai/claude-agent-sdk`) — so it is documented but neither installed for package consumers nor pulled into CI's `npm ci`; the maintainer installs it for live runs, matching how the driver already treats it as an optional peer.
+The default store root `bench/results/` is gitignored.
+
+**Review finding addressed — `--model` dropped.**
+The spec fixes a single model across all arms so the comparison measures the tool, not the model.
+A per-cell `--model` override (flagged as scope creep by the spec-fidelity review) would let arms drift onto different models, so it was removed.
+`--turn-cap` and `--wall-clock-ms` were kept: they are safe-defaulted bounds a maintainer may legitimately need to raise for a heavier task, and they do not affect cross-arm comparability.
+
+**Deferred to the aggregator (slice 0030).**
+The CLI tally reports recorded/invalid counts and reporting-floor status, but does not break failures out by tag (incorrect/confused/hung); those tags are on every stored sample and are the reporting slice's job to surface.

.gitignore

@@ -1,3 +1,4 @@
 node_modules/
 dist/
 coverage/
+bench/results/

bench/README.md

@@ -47,8 +47,24 @@ The raw component breakdown is retained on every sample so the data can be re-we
 - `runner.ts` — the single-cell runner: `runCell` threads every layer to run one `(arm, task, trial)` cell end to end — provision, seed, run the agent bounded by a turn cap and a wall-clock backstop, audit the transcript, capture and score the post-run state, append the sample, and delete the repository. The live host and the Agent SDK are factored behind the `BenchHost` and `AgentDriver` seams, so the orchestration is unit-tested with fakes while the live wiring is validated by the smoke run.
 - `host.ts` — `liveBenchHost`, the production `BenchHost`: a thin composition of `seed.ts` (provision, seed, delete) and `snapshot.ts` (capture) bound to one set of host credentials.
 - `sdk-driver.ts` — `sdkAgentDriver`, the production `AgentDriver`: it runs one arm through the Claude Agent SDK on the maintainer's subscription, enforcing isolation in-band via the SDK's permission callback (the arm's guard on every Bash command; the shell disabled on the MCP arm) and reporting the four token components (folding in the auxiliary small model), the turn count, the imputed cost, the transcript, and the final report. The SDK is loaded through a computed dynamic import so it is an optional peer needed only for live runs.
+- `run-loop.ts` — `runCells`, the run loop: it runs one chosen `(arm, task)` cell for a batch of trials (defaulting to five, with a reporting floor of three) by driving `runCell` and the sample store, deciding only how many trials to run and at what trial numbers. Deepening a cell continues numbering past the highest trial it already holds and appends, so a cell's sample size grows across sittings rather than being overwritten. It reimplements no orchestration — provision, run, score, and append stay in `runCell`.
+- `run.ts` — the maintainer-facing run-loop command. `parseRunArgs` is the pure, unit-tested argument seam; `runBenchCommand` is the live boundary that resolves host access, resolves the scored suite against the host's self-review support, selects the task, and drives `runCells`. Invoked via `npm run bench:run` (see below).
 
-Later slices add the run-loop CLI and the aggregator.
+Later slices add the aggregator.
+
+## Running a cell
+
+The run-loop command runs one chosen cell on demand, so only the token budget available at that moment is spent:
+
+```
+GITEA_AXI_BENCH_LOGIN=<tea-login-name> npm run bench:run -- --arm gitea-axi --task close-csv-export-issue
+```
+
+It defaults to five trials per sitting with a reporting floor of three; pass `--trials <n>` to run a different batch, and `--help` for the full flag list.
+Re-running the same cell deepens it — the new trials append rather than overwrite — so a cell's sample size can be grown opportunistically across separate sittings.
+
+The command is executed with [`tsx`](https://tsx.is) (a devDependency) so the harness's TypeScript runs directly.
+Like the runner smoke tier it drives the live host and the Claude Agent SDK, so a real run needs a configured login, the `gitea-axi` CLI on `PATH`, the Agent SDK installed (`npm install @anthropic-ai/claude-agent-sdk` — an optional peer, declared but not installed by default), and a Claude subscription.
 
 ## Tests
 

bench/run-loop.test.ts

@@ -0,0 +1,188 @@
+import { mkdtempSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { afterEach, beforeEach, describe, expect, it } from "vitest";
+import type { ResultRecord } from "./result.js";
+import { runCells } from "./run-loop.js";
+import type {
+  AgentDriver,
+  BenchHost,
+  RunBounds,
+  RunCellInput,
+} from "./runner.js";
+import type { BenchAccess } from "./seed.js";
+import { createSampleStore } from "./store.js";
+import { SAMPLE_TASK } from "./task.js";
+
+// Trivial stubs for the collaborators the run loop merely forwards to the
+// injected single-cell runner. Because runOne is faked below, none of these is
+// ever touched, so bare casts are enough to satisfy the input shape.
+const ACCESS: BenchAccess = { apiUrl: "https://git.example.test", token: "tok" };
+const HOST = {} as BenchHost;
+const DRIVER = {} as AgentDriver;
+const BOUNDS: RunBounds = { turnCap: 10, wallClockMs: 60_000 };
+const BUILD = { binRoot: "/nonexistent" };
+
+/** A minimal recorded ResultRecord the fake runOne returns per trial. */
+function record(overrides: Partial<ResultRecord> = {}): ResultRecord {
+  return {
+    arm: "gitea-axi",
+    taskId: SAMPLE_TASK.id,
+    tier: SAMPLE_TASK.tier,
+    trial: 1,
+    timestamp: "2026-07-16T00:00:00Z",
+    tokens: { freshInput: 1, cacheCreation: 0, cacheRead: 0, output: 1 },
+    turns: 1,
+    durationMs: 1,
+    imputedCostUsd: 0.01,
+    outcome: { pass: true },
+    ...overrides,
+  };
+}
+
+describe("runCells", () => {
+  let storeRoot: string;
+
+  beforeEach(() => {
+    storeRoot = mkdtempSync(join(tmpdir(), "bench-run-loop-"));
+  });
+
+  afterEach(() => {
+    rmSync(storeRoot, { recursive: true, force: true });
+  });
+
+  // Behavior: with no trials count given, the run loop runs the selected
+  // (arm, task) cell for the default of five trials, invoking the injected
+  // single-cell runner once per trial with the cell's arm and task. The default
+  // of five is an independent literal fixed by the benchmark-harness spec / task
+  // 0029 ("Each cell defaults to five trials"), not recomputed from run-loop.ts.
+  it("runs the default of five trials when no trials count is given, once per trial with the cell's arm and task", async () => {
+    const store = createSampleStore(storeRoot);
+    const calls: RunCellInput[] = [];
+
+    const runOne = async (input: RunCellInput) => {
+      calls.push(input);
+      const rec = record({ trial: input.trial });
+      input.store.append(rec);
+      return { kind: "recorded", record: rec } as const;
+    };
+
+    await runCells({
+      arm: "gitea-axi",
+      task: SAMPLE_TASK,
+      access: ACCESS,
+      host: HOST,
+      driver: DRIVER,
+      store,
+      bounds: BOUNDS,
+      build: BUILD,
+      runOne,
+    });
+
+    // Exactly five invocations — the spec's default cell depth.
+    expect(calls).toHaveLength(5);
+
+    // Every invocation ran the selected cell's arm and task.
+    for (const call of calls) {
+      expect(call.arm).toBe("gitea-axi");
+      expect(call.task.id).toBe(SAMPLE_TASK.id);
+    }
+  });
+
+  // Behavior: re-running an already-sampled cell deepens it — the new trials are
+  // numbered past the highest trial the cell already holds and are appended, so
+  // prior samples are never overwritten (benchmark-harness spec / task 0029). A
+  // cell holding 2 samples at trials 1 and 2, run for 3 more trials, must end with
+  // 5 samples at trials [1,2,3,4,5]: the first two unchanged, three appended at
+  // 3, 4, 5. The trial sequence is an independent literal, not recomputed.
+  it("deepens an already-sampled cell, appending new trials past the highest without overwriting priors", async () => {
+    const store = createSampleStore(storeRoot);
+
+    // A prior sitting: two samples already accumulated in the cell.
+    const prior1 = record({ trial: 1 });
+    const prior2 = record({ trial: 2 });
+    store.append(prior1);
+    store.append(prior2);
+
+    const runOne = async (input: RunCellInput) => {
+      const rec = record({ trial: input.trial });
+      input.store.append(rec);
+      return { kind: "recorded", record: rec } as const;
+    };
+
+    const result = await runCells({
+      arm: "gitea-axi",
+      task: SAMPLE_TASK,
+      trials: 3,
+      access: ACCESS,
+      host: HOST,
+      driver: DRIVER,
+      store,
+      bounds: BOUNDS,
+      build: BUILD,
+      runOne,
+    });
+
+    const samples = store.read({ arm: "gitea-axi", taskId: SAMPLE_TASK.id });
+
+    // The cell deepened from 2 to 5 samples, numbered 1..5 in append order.
+    expect(samples).toHaveLength(5);
+    expect(samples.map((s) => s.trial)).toEqual([1, 2, 3, 4, 5]);
+
+    // The two prior samples were preserved byte-for-byte, not overwritten.
+    expect(samples[0]).toEqual(prior1);
+    expect(samples[1]).toEqual(prior2);
+
+    // The result reports how many samples the cell held before and after.
+    expect(result.priorSamples).toBe(2);
+    expect(result.totalSamples).toBe(5);
+  });
+
+  // Behavior: an attempt the single-cell runner flags invalid produces no sample
+  // and is tallied separately; a cell only meets the reporting floor once it holds
+  // at least three samples (benchmark-harness spec / task 0029, "the reporting
+  // floor of three"). Here 2 of 5 attempts record and 3 are flagged invalid (a
+  // foreign tool was reached), so the store gains only the 2 recorded samples, the
+  // invalid count is tracked apart, and 2 < 3 leaves the cell below the floor. The
+  // literals 2, 3, and false come from this worked example, not from run-loop.ts.
+  it("tallies invalid attempts apart from recorded samples and stays below the reporting floor at two samples", async () => {
+    const store = createSampleStore(storeRoot);
+
+    // Record on the first two attempts, flag the rest invalid without appending.
+    let call = 0;
+    const runOne = async (input: RunCellInput) => {
+      call += 1;
+      if (call <= 2) {
+        const rec = record({ trial: input.trial });
+        input.store.append(rec);
+        return { kind: "recorded", record: rec } as const;
+      }
+      const leaks: string[] = ["curl"];
+      return { kind: "invalid" as const, leaks };
+    };
+
+    const result = await runCells({
+      arm: "gitea-axi",
+      task: SAMPLE_TASK,
+      trials: 5,
+      access: ACCESS,
+      host: HOST,
+      driver: DRIVER,
+      store,
+      bounds: BOUNDS,
+      build: BUILD,
+      runOne,
+    });
+
+    // Recorded and invalid attempts are tallied separately.
+    expect(result.recorded).toBe(2);
+    expect(result.invalid).toBe(3);
+
+    // Only the two recorded attempts became samples; invalid attempts left none.
+    expect(result.totalSamples).toBe(2);
+    expect(store.read({ arm: "gitea-axi", taskId: SAMPLE_TASK.id })).toHaveLength(2);
+
+    // Two samples is below the reporting floor of three.
+    expect(result.meetsFloor).toBe(false);
+  });
+});

bench/run-loop.ts

@@ -0,0 +1,109 @@
+// The run loop: the maintainer-facing orchestration that runs one chosen
+// `(arm, task)` cell for a batch of trials on demand, so only the token budget
+// available at that moment is spent. It drives the single-cell runner (runner.ts)
+// and the append-only sample store (store.ts) built in earlier slices rather than
+// reimplementing any orchestration — its whole job is to decide how many trials to
+// run and at what trial numbers, then leave provisioning, running, scoring, and
+// appending to `runCell`.
+//
+// Because results are immutable timestamped samples, running a cell that already
+// has samples deepens it: the new trials continue past the highest trial the cell
+// holds and append, so a cell's sample size can be grown opportunistically across
+// separate sittings. Each cell defaults to five trials with a reporting floor of
+// three; the loop reports whether the cell now meets that floor.
+
+import type { BuildArmOptions } from "./arm.js";
+import type { Arm } from "./result.js";
+import { runCell, type CellOutcome, type RunBounds, type RunCellInput, type RunnerClock } from "./runner.js";
+import type { BenchAccess } from "./seed.js";
+import type { SampleStore } from "./store.js";
+import type { BenchTask } from "./task.js";
+
+/** A cell defaults to five trials per sitting. */
+export const DEFAULT_TRIALS = 5;
+
+/** A cell is only reported once it holds at least this many samples. */
+export const REPORTING_FLOOR = 3;
+
+/** Everything needed to run a batch of trials for one `(arm, task)` cell. */
+export interface RunCellsInput {
+  arm: Arm;
+  task: BenchTask;
+  /** Trials to run this sitting; defaults to {@link DEFAULT_TRIALS}. */
+  trials?: number;
+  access: BenchAccess;
+  host: RunCellInput["host"];
+  driver: RunCellInput["driver"];
+  store: SampleStore;
+  bounds: RunBounds;
+  build: BuildArmOptions;
+  clock?: Partial<RunnerClock>;
+  /** The single-cell runner; injectable for tests. Defaults to {@link runCell}. */
+  runOne?: (input: RunCellInput) => Promise<CellOutcome>;
+}
+
+/** The tally of running one batch of trials for a cell. */
+export interface RunCellsResult {
+  arm: Arm;
+  taskId: string;
+  /** Per-attempt outcomes in run order. */
+  outcomes: CellOutcome[];
+  /** Attempts that produced a scored sample this sitting. */
+  recorded: number;
+  /** Attempts flagged invalid (a foreign tool was reached) this sitting; not sampled. */
+  invalid: number;
+  /** Samples the cell held before this sitting. */
+  priorSamples: number;
+  /** Samples the cell holds after this sitting. */
+  totalSamples: number;
+  /** Whether the cell now meets the reporting floor of {@link REPORTING_FLOOR} samples. */
+  meetsFloor: boolean;
+}
+
+/**
+ * Run a batch of trials for one cell. Deepens the cell if it already has samples:
+ * trial numbering continues past the highest existing trial, and every scored
+ * sample is appended by `runCell` rather than overwriting a slot.
+ */
+export async function runCells(input: RunCellsInput): Promise<RunCellsResult> {
+  const { arm, task, access, host, driver, store, bounds, build, clock } = input;
+  const runOne = input.runOne ?? runCell;
+  const trials = input.trials ?? DEFAULT_TRIALS;
+  const cell = { arm, taskId: task.id };
+
+  const prior = store.read(cell);
+  // Continue numbering past the highest trial the cell already holds so a
+  // deepening sitting never reuses a trial number, even if earlier attempts were
+  // flagged invalid and left gaps (an invalid attempt records no sample).
+  const highestTrial = prior.reduce((max, sample) => Math.max(max, sample.trial), 0);
+
+  const outcomes: CellOutcome[] = [];
+  for (let offset = 0; offset < trials; offset += 1) {
+    outcomes.push(
+      await runOne({
+        arm,
+        task,
+        trial: highestTrial + offset + 1,
+        access,
+        host,
+        driver,
+        store,
+        bounds,
+        build,
+        clock,
+      }),
+    );
+  }
+
+  const totalSamples = store.read(cell).length;
+  return {
+    arm,
+    taskId: task.id,
+    outcomes,
+    recorded: outcomes.filter((outcome) => outcome.kind === "recorded").length,
+    invalid: outcomes.filter((outcome) => outcome.kind === "invalid").length,
+    priorSamples: prior.length,
+    totalSamples,
+    meetsFloor: totalSamples >= REPORTING_FLOOR,
+  };
+}

bench/run.test.ts

@@ -0,0 +1,121 @@
+import { describe, expect, it } from "vitest";
+import {
+  DEFAULT_STORE_ROOT,
+  DEFAULT_TURN_CAP,
+  DEFAULT_WALL_CLOCK_MS,
+  parseRunArgs,
+} from "./run.js";
+
+describe("parseRunArgs", () => {
+  // Behavior: parsing the required --arm and --task selection yields the resolved
+  // cell with defaults applied for everything else — five trials (spec: "Each cell
+  // defaults to five trials"), the documented turn-cap and wall-clock backstop, the
+  // default store root, and the login taken from the environment when --login is
+  // omitted. Trials is the independent literal 5; the other defaults are asserted
+  // against the module's documented default constants (the single source of truth
+  // for each default), which checks the parser wires them through on omission.
+  it("applies the documented defaults when only the required arm and task are given, taking the login from the environment", () => {
+    const result = parseRunArgs(
+      ["--arm", "gitea-axi", "--task", "close-csv-export-issue"],
+      { GITEA_AXI_BENCH_LOGIN: "alexion" },
+    );
+
+    expect(result.help).toBe(false);
+    if (result.help) return;
+
+    // The required selection resolves to the chosen cell.
+    expect(result.arm).toBe("gitea-axi");
+    expect(result.taskId).toBe("close-csv-export-issue");
+
+    // Trials default to five (independent literal from the spec).
+    expect(result.trials).toBe(5);
+
+    // The remaining bounds and store root fall back to the documented defaults.
+    expect(result.turnCap).toBe(DEFAULT_TURN_CAP);
+    expect(result.wallClockMs).toBe(DEFAULT_WALL_CLOCK_MS);
+    expect(result.storeRoot).toBe(DEFAULT_STORE_ROOT);
+
+    // The login comes from the environment.
+    expect(result.login).toBe("alexion");
+  });
+
+  // Behavior: every optional flag overrides its default, and an explicit --login
+  // takes precedence over the environment. The parser passes through whatever the
+  // maintainer supplies. All expected values are independent literals chosen apart
+  // from the code; login must be the explicit "explicit-login" even though the
+  // environment also sets GITEA_AXI_BENCH_LOGIN ("env-login").
+  it("passes every supplied flag through, with an explicit login overriding the environment", () => {
+    const result = parseRunArgs(
+      [
+        "--arm",
+        "tea",
+        "--task",
+        "read-open-issue-count",
+        "--trials",
+        "3",
+        "--turn-cap",
+        "12",
+        "--wall-clock-ms",
+        "90000",
+        "--store",
+        "/tmp/bench-out",
+        "--login",
+        "explicit-login",
+      ],
+      { GITEA_AXI_BENCH_LOGIN: "env-login" },
+    );
+
+    expect(result.help).toBe(false);
+    if (result.help) return;
+
+    expect(result.arm).toBe("tea");
+    expect(result.taskId).toBe("read-open-issue-count");
+    expect(result.trials).toBe(3);
+    expect(result.turnCap).toBe(12);
+    expect(result.wallClockMs).toBe(90000);
+    expect(result.storeRoot).toBe("/tmp/bench-out");
+
+    // The explicit --login beats the env-provided login.
+    expect(result.login).toBe("explicit-login");
+  });
+
+  // Behavior: malformed or incomplete input is rejected with a usage error. The
+  // four arms are exactly gitea-axi, tea, gitea-mcp, raw-api; --arm and --task are
+  // required; --trials must be a positive integer; unknown flags are not accepted;
+  // and a login must be resolvable (from --login or the environment). A non-empty
+  // env login is supplied where the tested defect is elsewhere, so the throw is the
+  // intended one rather than a missing login.
+  it("rejects malformed or incomplete input with a usage error", () => {
+    const env = { GITEA_AXI_BENCH_LOGIN: "alexion" };
+
+    // Unknown arm (not one of the four).
+    expect(() => parseRunArgs(["--arm", "github", "--task", "t"], env)).toThrow();
+    // Missing required --arm.
+    expect(() => parseRunArgs(["--task", "t"], env)).toThrow();
+    // Missing required --task.
+    expect(() => parseRunArgs(["--arm", "tea"], env)).toThrow();
+    // Non-numeric trials.
+    expect(() =>
+      parseRunArgs(["--arm", "tea", "--task", "t", "--trials", "abc"], env),
+    ).toThrow();
+    // Unknown flag.
+    expect(() =>
+      parseRunArgs(["--arm", "tea", "--task", "t", "--frobnicate", "x"], env),
+    ).toThrow();
+    // The removed --model flag is now unknown and rejected.
+    expect(() =>
+      parseRunArgs(["--arm", "tea", "--task", "t", "--model", "x"], env),
+    ).toThrow();
+
+    // Login is required and here is resolvable from neither --login nor the env.
+    expect(() => parseRunArgs(["--arm", "tea", "--task", "t"], {})).toThrow();
+  });
+
+  // Behavior: --help short-circuits parsing and reports a help request, winning
+  // even alongside other arguments and via the -h alias.
+  it("short-circuits to a help request for --help and -h, even alongside other args", () => {
+    expect(parseRunArgs(["--help"], {}).help).toBe(true);
+    expect(parseRunArgs(["-h"], {}).help).toBe(true);
+    expect(parseRunArgs(["--arm", "tea", "--help"], {}).help).toBe(true);
+  });
+});

bench/run.ts

@@ -0,0 +1,278 @@
+// The maintainer-facing run-loop command: run one chosen benchmark cell on demand.
+//
+// This is the entry point the maintainer invokes to spend the token budget
+// available at a given moment on exactly one `(arm, task)` cell. It parses the
+// selection, resolves live host access through gitea-axi's own credential path,
+// resolves the scored suite against the host's self-review support, and drives the
+// run loop (run-loop.ts) — which in turn drives the single-cell runner and the
+// append-only sample store built in earlier slices. No orchestration is
+// reimplemented here.
+//
+// The command is bench-internal (bench/ is excluded from the published package)
+// and is executed with a TypeScript-aware runner; see `npm run bench:run`.
+//
+// The argument parser (`parseRunArgs`) is the pure, unit-tested seam. The live
+// wiring in `runBenchCommand` is a live boundary — it resolves real credentials
+// and drives the real host and Agent SDK — so, like the seed and runner smoke
+// tiers, it is validated by running it rather than by mocked unit tests.
+
+import { mkdtempSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { pathToFileURL } from "node:url";
+import type { CliDeps } from "../src/deps.js";
+import { liveBenchHost } from "./host.js";
+import type { Arm } from "./result.js";
+import { DEFAULT_TRIALS, REPORTING_FLOOR, runCells, type RunCellsResult } from "./run-loop.js";
+import { resolveBenchAccess } from "./seed.js";
+import { detectSelfReviewSupport } from "./self-review.js";
+import { sdkAgentDriver } from "./sdk-driver.js";
+import { createSampleStore } from "./store.js";
+import { buildScoredSuite } from "./task-suite.js";
+
+/** The default turn cap a run is bounded by when not overridden. */
+export const DEFAULT_TURN_CAP = 40;
+
+/** The default wall-clock backstop (ms) a run is bounded by when not overridden. */
+export const DEFAULT_WALL_CLOCK_MS = 300_000;
+
+/** Where accumulated samples are stored when `--store` is not given. */
+export const DEFAULT_STORE_ROOT = "bench/results";
+
+/** Environment variable naming the tea login the benchmark authenticates through. */
+export const LOGIN_ENV = "GITEA_AXI_BENCH_LOGIN";
+
+/** The four arms a cell may select. */
+export const ARMS: readonly Arm[] = ["gitea-axi", "tea", "gitea-mcp", "raw-api"];
+
+/** A fully-resolved cell selection and run configuration. */
+export interface RunArgs {
+  arm: Arm;
+  taskId: string;
+  /** Trials to run this sitting; defaults to {@link DEFAULT_TRIALS}. */
+  trials: number;
+  /** The tea login the benchmark authenticates through. */
+  login: string;
+  turnCap: number;
+  wallClockMs: number;
+  storeRoot: string;
+}
+
+/** The parse outcome: a request for help, or a resolved configuration to run. */
+export type ParsedRunArgs = { help: true } | ({ help: false } & RunArgs);
+
+/** The value-taking flags the command understands; anything else is rejected. */
+const KNOWN_FLAGS = new Set([
+  "arm",
+  "task",
+  "trials",
+  "login",
+  "turn-cap",
+  "wall-clock-ms",
+  "store",
+]);
+
+/** A usage error, surfaced to the maintainer with the offending detail. */
+function usage(detail: string): Error {
+  return new Error(`${detail}\n\nUsage: bench:run --arm <arm> --task <task-id> [--login <name>] [--trials <n>]`);
+}
+
+/** Parse a flag's value as a positive integer, rejecting anything else. */
+function positiveInt(value: string, flag: string): number {
+  if (!/^\d+$/.test(value) || Number(value) < 1) {
+    throw usage(`--${flag} must be a positive integer, got "${value}"`);
+  }
+  return Number(value);
+}
+
+/**
+ * Parse the run-loop command's argv into a resolved configuration, applying
+ * defaults ({@link DEFAULT_TRIALS} trials, {@link DEFAULT_TURN_CAP} turn cap,
+ * {@link DEFAULT_WALL_CLOCK_MS} backstop, {@link DEFAULT_STORE_ROOT} store, and the
+ * login from {@link LOGIN_ENV}). Throws a usage error when a required selection is
+ * missing or a value is malformed.
+ */
+export function parseRunArgs(
+  argv: string[],
+  env: Record<string, string | undefined>,
+): ParsedRunArgs {
+  const flags = new Map<string, string>();
+  for (let index = 0; index < argv.length; index += 1) {
+    const token = argv[index] as string;
+    if (token === "--help" || token === "-h") {
+      return { help: true };
+    }
+    if (!token.startsWith("--")) {
+      throw usage(`unexpected argument "${token}"`);
+    }
+    const equals = token.indexOf("=");
+    const name = equals === -1 ? token.slice(2) : token.slice(2, equals);
+    if (!KNOWN_FLAGS.has(name)) {
+      throw usage(`unknown flag "--${name}"`);
+    }
+    let value: string | undefined;
+    if (equals === -1) {
+      value = argv[index + 1];
+      if (value === undefined || value.startsWith("--")) {
+        throw usage(`flag --${name} needs a value`);
+      }
+      index += 1;
+    } else {
+      value = token.slice(equals + 1);
+    }
+    flags.set(name, value);
+  }
+
+  const arm = flags.get("arm");
+  if (arm === undefined) {
+    throw usage("--arm <arm> is required");
+  }
+  if (!ARMS.includes(arm as Arm)) {
+    throw usage(`--arm must be one of ${ARMS.join(", ")}, got "${arm}"`);
+  }
+  const taskId = flags.get("task");
+  if (taskId === undefined) {
+    throw usage("--task <task-id> is required");
+  }
+  const login = flags.get("login") ?? env[LOGIN_ENV];
+  if (login === undefined || login.length === 0) {
+    throw usage(`--login <name> is required (or set ${LOGIN_ENV})`);
+  }
+
+  const trials = flags.has("trials") ? positiveInt(flags.get("trials") as string, "trials") : DEFAULT_TRIALS;
+  const turnCap = flags.has("turn-cap")
+    ? positiveInt(flags.get("turn-cap") as string, "turn-cap")
+    : DEFAULT_TURN_CAP;
+  const wallClockMs = flags.has("wall-clock-ms")
+    ? positiveInt(flags.get("wall-clock-ms") as string, "wall-clock-ms")
+    : DEFAULT_WALL_CLOCK_MS;
+
+  return {
+    help: false,
+    arm: arm as Arm,
+    taskId,
+    trials,
+    login,
+    turnCap,
+    wallClockMs,
+    storeRoot: flags.get("store") ?? DEFAULT_STORE_ROOT,
+  };
+}
+
+/** The help text printed for `--help` / `-h`. */
+const HELP_TEXT = `bench:run — run one benchmark cell on demand.
+
+Runs a single (arm, task) cell for a batch of trials against the live Gitea host,
+appending each scored sample to the store. Re-running a cell deepens it: new trials
+append rather than overwrite, so a cell's sample size can be grown across sittings.
+
+Usage:
+  npm run bench:run -- --arm <arm> --task <task-id> [options]
+
+Required:
+  --arm <arm>          One of: ${ARMS.join(", ")}
+  --task <task-id>     A scored-suite task id (an unknown id prints the available ids)
+
+Options:
+  --login <name>       tea login to authenticate through (default: $${LOGIN_ENV})
+  --trials <n>         Trials to run this sitting (default: ${DEFAULT_TRIALS})
+  --turn-cap <n>       Per-run turn cap (default: ${DEFAULT_TURN_CAP})
+  --wall-clock-ms <n>  Per-run wall-clock backstop in ms (default: ${DEFAULT_WALL_CLOCK_MS})
+  --store <dir>        Sample store root (default: ${DEFAULT_STORE_ROOT})
+  -h, --help           Show this help`;
+
+/** Render the run-loop tally into the lines printed after a sitting. */
+function summarize(result: RunCellsResult, storeRoot: string): string[] {
+  const floorNote = result.meetsFloor
+    ? `meets the reporting floor of ${REPORTING_FLOOR}`
+    : `below the reporting floor of ${REPORTING_FLOOR} — deepen this cell before reporting`;
+  return [
+    `Cell (${result.arm}, ${result.taskId}): ${result.recorded} recorded, ${result.invalid} invalid this sitting.`,
+    `Samples: ${result.priorSamples} → ${result.totalSamples} (${floorNote}).`,
+    `Store: ${storeRoot}`,
+  ];
+}
+
+/**
+ * Run one chosen cell on demand: resolve live host access, resolve the scored
+ * suite against the host's self-review support, select the task, and drive the run
+ * loop. This is the command's live boundary — it authenticates and drives the real
+ * host and Agent SDK — so it is validated by running it, not by mocked unit tests
+ * (the pure `parseRunArgs` seam is the unit-tested part). Returns a process exit
+ * code and prints progress and the final tally through `out`.
+ */
+export async function runBenchCommand(
+  argv: string[],
+  deps: CliDeps,
+  out: (line: string) => void,
+): Promise<number> {
+  const parsed = parseRunArgs(argv, deps.env);
+  if (parsed.help) {
+    out(HELP_TEXT);
+    return 0;
+  }
+
+  const access = await resolveBenchAccess(deps, parsed.login);
+
+  // The two review tasks are approve/request-changes or comment reviews depending
+  // on what the host permits, so the suite is resolved against a live probe once
+  // before selecting the task (see task-suite.ts and self-review.ts).
+  out(`Probing self-review support on ${new URL(access.apiUrl).host}…`);
+  const selfReviewPermitted = await detectSelfReviewSupport(access);
+  const suite = buildScoredSuite({ selfReviewPermitted });
+  const task = suite.find((candidate) => candidate.id === parsed.taskId);
+  if (task === undefined) {
+    out(`No scored task with id "${parsed.taskId}". Available task ids:`);
+    for (const candidate of suite) {
+      out(`  ${candidate.id}  (${candidate.tier})`);
+    }
+    return 1;
+  }
+
+  const store = createSampleStore(parsed.storeRoot);
+  const binRoot = mkdtempSync(join(tmpdir(), "bench-run-bin-"));
+  out(`Running ${parsed.trials} trial(s) of cell (${parsed.arm}, ${task.id})…`);
+  try {
+    const result = await runCells({
+      arm: parsed.arm,
+      task,
+      trials: parsed.trials,
+      access,
+      host: liveBenchHost(access),
+      // Every arm runs on the driver's single fixed model per the spec, so the
+      // comparison measures the tool rather than the model; the command exposes
+      // no per-cell model override that could break that invariant.
+      driver: sdkAgentDriver(),
+      store,
+      bounds: { turnCap: parsed.turnCap, wallClockMs: parsed.wallClockMs },
+      build: { binRoot },
+    });
+    for (const line of summarize(result, parsed.storeRoot)) {
+      out(line);
+    }
+    return 0;
+  } finally {
+    rmSync(binRoot, { recursive: true, force: true });
+  }
+}
+
+/** Entry point: parse argv, run the command, and set the process exit code. */
+export async function main(): Promise<void> {
+  const deps: CliDeps = { env: process.env, cwd: process.cwd(), globals: {} };
+  try {
+    process.exitCode = await runBenchCommand(
+      process.argv.slice(2),
+      deps,
+      (line) => process.stdout.write(`${line}\n`),
+    );
+  } catch (error) {
+    process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
+    process.exitCode = 1;
+  }
+}
+
+// Run only when executed directly (e.g. `tsx bench/run.ts`), not when imported by
+// a test. Under a TypeScript runner argv[1] is this file's own path.
+if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
+  await main();
+}

package-lock.json

@@ -19,11 +19,20 @@
       "devDependencies": {
         "@types/node": "^20.19.0",
         "@vitest/coverage-v8": "^3.2.7",
+        "tsx": "^4.23.1",
         "typescript": "^5.8.0",
         "vitest": "^3.2.0"
       },
       "engines": {
         "node": ">=20"
+      },
+      "peerDependencies": {
+        "@anthropic-ai/claude-agent-sdk": ">=0.3.0"
+      },
+      "peerDependenciesMeta": {
+        "@anthropic-ai/claude-agent-sdk": {
+          "optional": true
+        }
       }
     },
     "node_modules/@ampproject/remapping": {
@@ -2229,6 +2238,25 @@
         "node": ">=14.0.0"
       }
     },
+    "node_modules/tsx": {
+      "version": "4.23.1",
+      "resolved": "https://registry.npmjs.org/tsx/-/tsx-4.23.1.tgz",
+      "integrity": "sha512-GQHnkIfxyx1wYCOS/wonik5MVRZU9hi1TEZmzGZSCJB1y9YgoZ8H6itNE/u4suE+yLmOzuE4E5S4TZ/ZX2wcWQ==",
+      "dev": true,
+      "license": "MIT",
+      "dependencies": {
+        "esbuild": "~0.28.0"
+      },
+      "bin": {
+        "tsx": "dist/cli.mjs"
+      },
+      "engines": {
+        "node": ">=18.0.0"
+      },
+      "optionalDependencies": {
+        "fsevents": "~2.3.3"
+      }
+    },
     "node_modules/typescript": {
       "version": "5.9.3",
       "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",

package.json

@@ -36,7 +36,8 @@
     "test:e2e": "vitest run --config vitest.e2e.config.ts",
     "test:pack": "vitest run --config vitest.packaging.config.ts",
     "test:bench": "vitest run --config vitest.bench.config.ts",
-    "test:bench:smoke": "vitest run --config vitest.bench-smoke.config.ts"
+    "test:bench:smoke": "vitest run --config vitest.bench-smoke.config.ts",
+    "bench:run": "tsx bench/run.ts"
   },
   "dependencies": {
     "@toon-format/toon": "^2.3.0",
@@ -46,7 +47,16 @@
   "devDependencies": {
     "@types/node": "^20.19.0",
     "@vitest/coverage-v8": "^3.2.7",
+    "tsx": "^4.23.1",
     "typescript": "^5.8.0",
     "vitest": "^3.2.0"
+  },
+  "peerDependencies": {
+    "@anthropic-ai/claude-agent-sdk": ">=0.3.0"
+  },
+  "peerDependenciesMeta": {
+    "@anthropic-ai/claude-agent-sdk": {
+      "optional": true
+    }
   }
 }