Add the maintainer-facing command that runs one chosen benchmark cell on
demand, so only the token budget available at that moment is spent.
runCells (bench/run-loop.ts) runs one (arm, task) cell for a batch of
trials — defaulting to five with a reporting floor of three — by driving
the existing single-cell runner and the append-only sample store rather
than reimplementing orchestration. Re-running a cell deepens it: trial
numbering continues past the highest trial the cell already holds and the
new samples append, so a cell's sample size grows across sittings without
overwriting prior runs.
bench/run.ts is the command: parseRunArgs is the pure, unit-tested
argument seam, and runBenchCommand is the live boundary that resolves host
access, resolves the scored suite against the host's self-review support,
selects the task, and drives the run loop. It is invoked via the new
bench:run npm script, run under tsx (a new devDependency) because the
harness's .js-specifier imports need a TypeScript-aware runner. The Claude
Agent SDK is now declared as an optional peerDependency — documented but
neither installed for package consumers nor pulled into CI.
Every arm runs on the driver's single fixed model; the command exposes no
per-cell model override that could break cross-arm comparability. The
default store root bench/results/ is gitignored.
Add the deterministic, idempotent seed that brings a freshly provisioned
throwaway repository to a known ground truth before a trial runs, scripted
over the live Gitea API.
- bench/seed-plan.ts: the pure ground truth (fixed labels, an issue spread
across the discriminating dimensions, and labelled/reviewed/real-branch
pull requests) plus groundTruth(user), realizing it into a RepoState.
- bench/seed.ts: idempotent seeding reconciled by natural key, reusing
gitea-axi's own tea-login credential discovery (no new secret handling).
- A live smoke tier (test:bench:smoke) validating the seed end-to-end and
skipping cleanly when no host is configured, kept out of the deterministic
bench tier.
Export selectLogin from src/context.ts so the bench reuses the exact
credential-selection path.
Lay the foundation the benchmark harness reads and writes: a bench/
directory (excluded from the published npm package), the immutable
result-record shape, and an append-only per-cell sample store.
- bench/result.ts: the ResultRecord shape — four token components,
turns, duration, imputed cost, tagged pass/fail outcome, and the
arm/task/tier/trial/timestamp tags. Arm and Tier are typed unions.
- bench/store.ts: append-only sample store, one JSONL file per cell at
<root>/<arm>/<taskId>.jsonl; deepening a cell only ever adds samples.
- bench/README.md: harness working docs and benchmark vocabulary, kept
out of the tool's domain glossary per the spec.
- Dedicated bench test tier (vitest.bench.config.ts, npm run test:bench)
kept out of the fast tier; tsconfig typechecks bench.
- Packaging tier asserts bench/ never ships in the tarball.
Complete the distribution metadata and publish flow for the unscoped
`gitea-axi` package:
- add `repository`, `homepage`, and `bugs` to package.json
- add `publishConfig` (public access, npmjs registry) so `npm publish`
needs no extra flags
- replace `prepublishOnly` with `prepack: npm run build`, so both
`npm pack` and `npm publish` rebuild `dist/` first
- document the single-command flow in PUBLISHING.md
- add a packaging smoke-test tier (`test:pack`) that packs the real
tarball, installs it globally, and drives the installed binary
(`--help`, dashboard header, and `setup` finding the bundled skill)
Distribute gitea-axi's ambient context via explicit user actions (ADRs
0009, 0013), with no postinstall script:
- Bundle the Agent Skill markdown at skills/gitea-axi/SKILL.md (a
minimal pointer, not a command reference) and ship it via package.json
files.
- Add `setup`, which installs the skill into ~/.claude/skills/
idempotently (installed/updated/unchanged).
- Add `setup hooks`, which registers a SessionStart hook running the
bare dashboard for Claude Code, Codex, and OpenCode via the SDK's
installSessionStartHooks(), updating managed entries in place.
- Shadow the SDK's built-in `update` so it fails with VALIDATION_ERROR
and points at the npm update command, keeping the ten-code error list
intact.
Integration tests drive all three at the CLI seam against a temporary
HOME; these commands make no Gitea API calls, so there is no live-Gitea
e2e case.
Add v8 coverage via a test:coverage script and vitest config: measure src
only, exclude the types-only deps.ts and the bin entrypoint main.ts, and
gate on ratcheted thresholds (statements 85, branches 78, functions 90,
lines 85) set just under current coverage. CI runs the unit+integration
tier through coverage so a real drop fails the build.
Add the end-to-end test tier: a Gitea Actions workflow (GitHub-compatible)
that runs the unit+integration tiers plus e2e tests against a pinned
disposable Gitea service container. The e2e suite provisions its own repo,
token, and seed data over HTTP and drives the real CLI seam, gated on
GITEA_AXI_E2E_URL so the default suite stays dependency-free.
A fixture-vs-live shape guard anchors the issue-list FieldDef paths against
both the recorded fixture and the live response, failing on divergence.
Tracer bullet for gitea-axi: runnable npm package on axi-sdk-js with
gitea-js as the sole HTTP layer, ESM on Node 20+.
- issue list with --state/--limit, default fields, count line from
X-Total-Count, type=issues guard, explicit empty state, and next-step
suggestions
- repo context detection from the git origin remote (SSH/scp/HTTPS),
tea credential discovery with the three-way login-matching split, and
-R/--repo and --login overrides (flag > env > auto)
- token retrieval via tea login helper get: tea's login list JSON
carries no token (ADR 0001 amended)
- full AxiError classification table with path-based 404 split, TOON
errors on stdout, exit codes 0/1/2
- test mode (GITEA_AXI_API_URL/TOKEN/REPO) suppressing subprocesses,
fixture server, and vitest suites driving the CLI seam (50 tests)
