Add a `home-manager` flake input, its nixpkgs following this flake's, and a
`home-manager-module` check that evaluates the real module through
home-manager's standalone entry point and builds the home files derivation
under four operator configurations, asserting on the tree each produces:
the Skill coexisting with an operator's own skills in both the attribute-set
and whole-directory forms, the sibling-enable gate leaving no Skill when
Claude Code is off, and the hook merging into an operator's own SessionStart
list. This is the first automated proof of the module's composition (ADR 0021),
replacing verification by maintainer rebuild.
Reshape the home-manager module so `programs.gitea-axi.enable` installs the
binary always, and the Claude Code context follows the harness. The two
per-artefact toggles and their assertion are replaced by one per-harness
toggle, `enableClaudeCodeIntegration` (default true); its artefacts land only
when `programs.claude-code.enable` is also on, silently absent otherwise.
The Agent Skill is now written through home.file into Claude Code's skills
directory, rather than contributed to `programs.claude-code.skills`, so it
composes with both the attribute-set and whole-directory forms of an operator's
own skills option. The Skill write is gated on `claude-code.enable` explicitly
(keeping package realisation lazy); the hook keeps its sibling-module gate for
free, and the asymmetry is commented. Supersedes three decisions of ADR 0020;
recorded in ADR 0021.
INSTALL.md is updated to the new option surface and the path-form limitation
paragraph removed. Adds the parent spec and the follow-up task 0047 (the flake
check proving composition, implemented separately).
Let a Nix configuration declare gitea-axi's ambient context instead of
running a command that writes it. `setup` and `setup hooks` are write-only
against files the operator is assumed to own, so an operator whose agent
configuration is generated cannot use them at all.
The package installs the bundled Agent Skill to share/gitea-axi/skills and
publishes it as `passthru.skill`, alongside `passthru.sessionStartHook` read
from `session-start-hook.json` — a committed declaration the fast tier reads
too, so a test drives `setup hooks` and asserts the two agree.
On top of that, `homeModules.gitea-axi` declares both from those attributes
through home-manager's own Claude Code options, so an operator's existing
skills and SessionStart hooks compose rather than collide. Importing it
without enabling it yields a byte-identical generation.
The spec's Out of Scope entry deferring a home-manager module is deleted;
ADR 0020 records the reversal, and INSTALL.md describes both paths.
Both halves of `setup` assumed the files they manage are writable. A
declaratively managed target — read-only because a configuration manager
owns it, because a file is flagged immutable, or because the path is
root-owned — made the skill install raise a raw filesystem exception and
the hook install surface the underlying message with no guidance.
Both now fail with `TARGET_NOT_WRITABLE`, naming the file and pointing at
the general remedy: it appears to be managed by another tool, so declare
the skill or hook through that configuration instead. The error names no
particular manager, because read-only is not diagnostic of one.
A target already byte-identical to the bundled copy still succeeds —
nothing needs writing, so its being read-only is beside the point.
The SDK records a bare, upgrade-stable binary name only when a PATH entry
realpath-matches the execPath it is handed. From the module-relative
entrypoint that can only succeed under npm, which symlinks its bin entry
straight at it; a wrapper-based install never can, because a script that
invokes a file does not resolve to that file. So every wrapper install --
Nix, a shim, a generated .cmd -- recorded an absolute path that moves on
upgrade, and a session-start hook that cannot execute fails silently.
`setup hooks` now resolves gitea-axi on PATH itself and hands that
location to the SDK, so the bare name is recorded. A candidate qualifies
only if it resolves to the running entrypoint -- by realpath for a
symlink, or by naming it in its text for a wrapper, following the chain,
since a Nix install is two hops. A same-named binary that is some other
program does not qualify, and the absolute entrypoint path stands as the
fallback exactly as before.
The related defect on the same line goes too: the SDK recognises its hook
by finding the marker inside the recorded command, so an entrypoint path
without "gitea-axi" in it made a re-run append a duplicate rather than
update in place. `setup hooks` now prunes duplicates by matching the
exact command it records, which is independent of that command's shape
and cannot mistake another tool's hook for its own.
With the coupling gone, package.nix no longer renames its build tree; the
fast tier runs from /build/source and its idempotency test passes there.
The help text's instruction to re-run hooks after an upgrade is deleted,
having become false.
Verified against the built Nix binary and a globally npm-installed pack:
both record the bare name, both fall back to the absolute path when the
name is absent, an impostor on PATH is refused, and re-runs leave one
entry. Decision recorded as ADR 0019; ADR 0009's addendum is amended.
Grilling task 0042's mitigation found its framing too narrow. Recording an
absolute entrypoint path is one defect; the deeper one is that `setup` is
write-only, and an operator whose agent configuration is generated
declaratively cannot let it write at all. On such a machine both halves fail —
`setup hooks` against a read-only settings file, and `setup` on an unhandled
filesystem error — and the Agent Skill gets hand-copied into the operator's own
configuration, where it silently drifts from the package that ships it.
Three tasks follow:
- 0043 records the bare binary name, resolved through PATH, so the hook
survives an upgrade on any wrapper-based install. Fixes the marker-substring
coupling with it and drops the derivation's build-tree rename.
- 0044 reports an unwritable target as a structured error naming no cause,
rather than crashing.
- 0045 adds the declarative install path: a stable Skill location, the Skill
and hook specification exposed as Nix-consumable attributes, one committed
hook specification read by both the expression and the test suite, and a
home-manager module wiring them. Blocked by 0043, whose bare name the
specification declares.
CONTEXT.md gains the four terms this settled and amends `setup` and
`SessionStart hook`, which described the imperative path as the only one.
Entries for unbuilt work name the task that lands them, so the glossary does
not assert behaviour the code lacks.
The re-run-after-upgrade help text this branch added stays as it is: accurate
until 0043 removes it, which that task carries as a criterion.
Resolve the nix-flake-packaging spec's open verification item by observation
rather than inference, and act on the unfavourable answer.
`resolvePortableHookCommand` in axi-sdk-js returns the bare binary name only
when a PATH entry realpath-matches the entrypoint. npm symlinks its bin entry
straight at dist/main.js and satisfies that; Nix installs a generated wrapper
script whose realpath is the wrapper, so the absolute store path is recorded
instead. Verified by probing the SDK with both install shapes and by driving
the flake-built binary against a temporary HOME.
The path is content-addressed, so it moves on every rebuild and is eventually
collected, and a session-start hook that cannot execute fails silently. The
mitigation is documentation, per the decision recorded when the item was
opened: the setup help text now says to re-run `setup hooks` after an upgrade.
How the setup command constructs the hook is deliberately unchanged. Preferring
the bare name is the right answer for every wrapper-based install, not a Nix
special case, so it belongs in a successor task with its own ADR alongside the
related `isManagedHook` substring defect. package.nix's postUnpack rename
therefore stays; its comment no longer promises this task will remove it.
A distinct `flake` job runs `nix flake check` on push and pull request,
catching flake rot — most concretely a build-relevant file left out of
package.nix's source allowlist — at the commit that causes it rather than
weeks later at the maintainer's next system rebuild.
The job is non-gating by two mechanisms: no `needs` edge, so it neither
waits on the test job nor is waited on, and `continue-on-error` so a red
build does not fail the run. That flag sits on the steps rather than the
job because Gitea's `act` fork declares it on its Step struct only and
silently ignores the job-level key; recorded as a Gotcha.
The workflow pinned Node 20, which reached end-of-life in April 2026, while
the manifest promised support down to it — so the entire claimed range below
the single tested version went unverified and its floor was unsupported.
Matrix over the two supported long-term-support majors and narrow the engine
range to `^22 || ^24`, naming exactly what is tested. Narrowing is free now
because nothing has been published and no tags exist.
The benchmark harness tier joins every leg: it is deterministic and needs no
network, and its non-default runner configuration made it easy to believe it
was running when it was not. The end-to-end and packaging tiers run on the
highest leg only, conditioned on a flag attached to that leg through a matrix
`include` entry rather than on a version number restated at each site. The
benchmark smoke tier stays out, since it would pass by skipping.
`@types/node` follows the new floor; it was the last Node 20 reference in the
manifest, and the typecheck runs on every leg.
`nix develop` now yields the toolchain the repository actually needs — Node,
`git`, `tea`, and `curl` — giving a declarative answer to "what do I need to
work on this", which the repository previously specified nowhere.
The shell takes its Node from the package's `passthru` rather than naming
`pkgs.nodejs` a second time, so development and the shipped artifact cannot
drift onto different majors and cannot be set independently. `package.nix`
declares that `passthru` as an interface rather than leaving the shell to read
an incidental build attribute; it does not enter the derivation, so the store
path is unchanged.
The checks output aliases the package, so `nix flake check` builds it and
thereby runs both its verification phases instead of being a silent no-op. No
per-stage checks: the only stage adding coverage is the full typecheck, which
spans `test/` and `bench/` and would drag the benchmark harness into the
derivation's inputs, undoing the source filtering. It stays in CI.
The shell deliberately omits `gitea-axi` itself. The benchmark's arm resolves
that binary by name off PATH and must get the locally built `dist/main.js`, so
supplying the packaged one would silently substitute the wrong artifact.
The Nix build now runs the installed-binary facet of the packaging tier
against the binary it has just produced, via an `installCheckPhase` that
sets `GITEA_AXI_INSTALLED_BIN` to `$out/bin/gitea-axi` and runs the new
`test:installed` script. Running after `fixupPhase` means the binary
under test is the wrapped one an operator actually gets, and naming it is
all the phase does — the assertions stay in the shared tier, so the npm
and Nix installation paths cannot drift apart in what they guarantee.
`npmInstallHook` prunes dev dependencies out of the build tree before the
check runs, so `preInstall` snapshots `node_modules` with `cp -al` and the
check restores it by copying, leaving the snapshot intact for a replayed
phase.
Two defects fixed in passing, both surfaced by review:
`test:installed` pins a test file by path while the packaging runner sets
`passWithNoTests: true`, so moving that file would have taken the build
green having asserted nothing — the same silently-inert trap `doCheck`
sprang in task 0037. The script now passes `--passWithNoTests=false`.
`checkPhase`'s vitest left a timestamped run cache under
`node_modules/.vite`, which `npmInstallHook` copied into `$out`, shipping
a stray cache and making the derivation non-reproducible. It is now
removed before the install phase; `nix build --rebuild` passes.
Add a flake at the repository root exposing gitea-axi as a package, with the
derivation in its own callable expression so it stays buildable outside a flake
context and usable in an overlay unchanged.
Dependencies are fetched from the lockfile's integrity fields via importNpmLock
rather than a committed fixed-output hash, and the version is read from the
package manifest, so neither a dependency bump nor a release edits any Nix
expression. The source is an explicit allowlist, keeping ADR, spec, task and
bench churn out of the derivation's inputs.
The installed binary is wrapped with --suffix PATH per ADR 0018: the operator's
own git and tea win, and the closure's are a fresh-machine fallback.
Two things differ from the plan. Systems coverage is three targets, not four:
nixpkgs 26.11 dropped x86_64-darwin and now throws on evaluating it, which
would break nix flake show and nix flake check for every system at once. And
buildNpmPackage supplies no check hook, so doCheck alone was silently inert and
produced a green build whose tests never ran; running the fast tier needs an
explicit checkPhase.
The store holds only `alexion` now; `csv-reviewer` is gone for good.
Also records that node/npm/tea are absent from a non-interactive shell on
this host and that git carries no configured identity, both of which cost
a full session of rediscovery.
The packaging tier held two kinds of assertion joined only by an
expensive shared setup: the shape of the packed tarball and its
manifest, and the behaviour of the resulting installed binary. Split
them, and teach the second to take the binary it drives from
GITEA_AXI_INSTALLED_BIN.
When that variable names an existing binary the tier drives it and skips
pack-and-install entirely; unset, it packs and installs exactly as
before. The installed-binary facet becomes one seam with two callers —
the npm distribution path today, the Nix installation path in task 0038
— so the two cannot drift apart in what they guarantee about an
installed gitea-axi. Nothing in it may assert on how the binary came to
exist, since store paths, wrapper internals, and the arrangement of the
installed tree are implementation detail of the installation method.
The tarball assertions stay npm-only: no other distribution method
produces a tarball or a packed manifest.
Add the design record for distributing gitea-axi as a Nix flake: a
package, a development shell, and a checks output, plus the continuous
integration changes that come with it.
ADR 0018 records the wrapper's deference to the operator's own `git` and
`tea` — the reverse of the hermetic instinct, chosen because `tea`
refreshes OAuth tokens in place and so must not have two versions
mutating one credential store.
Also records the `tea`-is-still-a-runtime-dependency gotcha, which ADR
0002's title obscures.
Refresh the benchmark README from a clean co-temporal 4-arm run taken after
the find-then-act skill rewrite, the /api/v1 tolerance, and the search
next-step fix. gitea-axi drops from most-expensive arm to co-leader: within
~1% of raw REST overall, cheapest structured interface by ~23%, cheapest
arm outright on the read and find-then-act tiers, at 100% success and the
fewest turns of any arm.
The next-step suggestion on `search issues`/`search prs` is now conditioned
on the in-repo match count. On a miss it pointed the agent at `view <number>`,
which is nonsensical when nothing matched; it now suggests the non-indexed
`issue list --state all` / `pr list --state all` fallback, which recovers from
both an over-narrow query and issue-indexer lag without naming the cause. On
exactly one match it fills the real number (`issue view 2`), applying AXI
Principle 9's single-id fill. Two or more matches keep the parameterized
placeholder.
Search stays a locator — it never auto-loads the detail even on a single
match; ADR 0017 records that decision (a deliberate narrowing of Principle 4)
and the CONTEXT.md search term is updated to match.
bench:run gains a --skill <path> flag, threaded into the existing
BuildArmOptions.skillPath, so a skill variant can be A/B'd against the
shipped SKILL.md with the same binary and harness — the mechanism used to
validate the find-then-act rewrite — without mutating the shipped file.
The bundled skill's Discovery section told the agent to run the bare
dashboard and reach for --help proactively, and advertised overlapping
find-paths — inducing exploratory commands that made the gitea-axi arm the
most expensive of the benchmark's four. Replace it with a "find the target,
then act" section, name the non-obvious mutation flags so common edits do
not need --help, and drop the setup line and the over-tea/raw/git bullets
that only duplicated the description. A same-time A/B cut cost-equivalent
tokens ~10% and collapsed bare-dashboard use from 60% to 7% with no loss of
success.
The client (gitea-js) appends /api/v1 to the base URL itself, so a
GITEA_AXI_API_URL that already carries it — a natural guess given the
variable's name — doubled the segment and failed as a spurious
REPO_NOT_FOUND. Normalize the base URL by stripping a trailing /api/v1
(and any trailing slashes) on both the env-URL and tea-login paths, so the
host base and the /api/v1 endpoint both resolve.
The tea login store here holds `alexion` and `csv-reviewer`, not `axi`;
`selectLogin` matches `--login` by exact name, so `--login alexion` works and an
unknown `--login axi` fails with VALIDATION_ERROR — the prior note had this
backwards. Also record that bench/ unit tests run only under
vitest.bench.config.ts (plain `vitest run` matches test/** and finds none).
Two benchmark-harness races were being scored as agent failures. A freshly
seeded throwaway repo could 404 for an independent reader (the agent, a fresh
process) before Gitea made it consistent, and the async issue indexer lagged so
`search issues`/`search prs` returned nothing right after seeding — leaving the
agent unable to find the target it was asked to act on.
seedRepo now blocks until an independent read confirms the repo is reachable,
its full seeded issue and pull spread is visible, and a seeded issue and pull
are returned by the search index, before releasing the agent. It polls up to
60s and throws a loud harness error on timeout rather than letting a
propagation delay become a scored agent failure.
Also add an optional `pattern` regex to RequiredFact so a read answer's count is
recognised semantically rather than as a fixed phrase: "5 issues are currently
open" no longer fails against the literal "5 issues are open", while the
alphabetic-only filler run keeps a wrong count beside the right number (e.g.
"5 issues ... 3 open") failing.
`pr review <n>` gains `--comments-file <path>`, a JSON array of inline
comments submitted with the review. Each entry is one of two exclusive
shapes: a new comment `{ path, line, body }` (mapped to `new_position`,
always the new side) or a reply `{ reply_to, body }`. A reply carries no
line or side — gitea-axi finds the target via the reviews-plus-comments
fan-out (there is no get-comment-by-id endpoint), reconstructs its anchor
from the target's own `diff_hunk`, and infers old/new side from it, so a
same-line post threads with the existing conversation. All entries map
onto the review-submission payload's `comments[]`; no new HTTP layer is
added. An unknown `reply_to` is a VALIDATION_ERROR raised before the POST,
and the submitted inline-comment count rides the action block.
The shared path-resolve-and-read behind --body-file and --comments-file is
extracted into src/flag-file.ts.
Each inline review comment under `pr view <n> --reviews` now renders its
`id` (the handle a reply targets), `resolved` (`yes`/`no`, from whether
Gitea populated the comment's `resolver`), and `diff_hunk`. The hunk is
structurally trimmed to its `@@` header line plus its last two lines by
default (hunks of three lines or fewer are left whole) and emitted
verbatim under `--full`, so the trim never touches the char-based body
truncation path. The raw `position`/`original_position` diff offsets stay
unsurfaced. The fields ride the existing reviews-plus-per-review-comments
fetch — no extra API calls.
Nine capabilities the web UI exposes that the REST API does not, in the
issue/PR/review domain, analysed against go-gitea/gitea at e8befe0.
Each finding cites the web route, the service function its handler
reaches, the swagger evidence, upstream prior art, and a patch sketch.
Ranked by impact and tractability, with issue drafts for the five to
file first.
Also records that resolve/unresolve review conversations and threaded
reply endpoints already exist upstream, which supersedes the premises of
issues #38 and #39.
The state-aware count line from task 0033 (1166a48) renders
`count: N open of M total`, and the unit tests were updated to match, but
the three e2e tracer assertions still expected the old bare
`count: N of M total`. They are skipped without GITEA_AXI_E2E_URL, so the
staleness only surfaced in CI, where the e2e tier runs. Update them to the
state-qualified form the shipped code already produces — default `open`,
`--state closed` → `closed`. The code was correct; the tests were stale.
The prior results table and narrative claimed gitea-axi posts the lowest
cost-equivalent tokens. That snapshot predated the neutral-working-dir
isolation fix, when the checkout-defaulting arms (gitea-axi, tea) drew
repo and login for free from the harness's own checkout — so gitea-axi
was implicitly pre-authenticated and looked like the winner.
On a clean run with every arm fairly credentialed and executed together,
raw REST is the cheapest on cost-equivalent tokens and leads every tier —
terse HTTP is the token floor no wrapper undercuts. gitea-axi is a clear
second overall and the lowest-cost structured interface, beating gitea-mcp
and tea on every tier at 100% success. Keep the raw-REST arm and state
this plainly rather than crown the wrapper by omitting the floor.
Numbers regenerated from bench:report over the 240-sample clean snapshot.
`issue view` rendered state but never labels, and offered no way to add
them — so reading one issue's labels forced a detour through
`issue list --fields labels` and hunting the matching row. The benchmark
transcripts showed agents paying this round-trip on every labels/state
read.
Show labels by default in the detail view (a detail view should be
complete), and add a `--fields` flag mirroring `issue list` / `search`
to append assignees, closedAt, milestone, updatedAt, url on request.
Also strengthen SKILL.md against the two command-discovery round-trips
the transcripts exposed: name the required `search issues` / `search prs`
subcommand form (a bare `search "<query>"` is invalid), and point agents
straight at `issue view <n>` for a single issue's fields.
Verified live: read-issue-labels-and-state dropped from 10 turns to 4
(cache-read ~3.3x lower), the transcript reduced to three clean commands
with the search-help and issue-list round-trips gone.
The gitea-axi arm was the only shell arm handed no credentials: the
runner set only PATH, so the agent had to reverse-engineer the tea-login
system — guessing a profile name and hunting for a config file — before
any real work, burning ~4 turns per task. Since turns drive cache-read,
the benchmark's dominant cost metric, this scaffolding gap alone inflated
gitea-axi's cost-equivalent tokens above every other arm.
Hand the arm its host and token through gitea-axi's own env interface
(GITEA_AXI_API_URL / GITEA_AXI_TOKEN), the symmetric counterpart to the
gitea-mcp server's GITEA_HOST / GITEA_ACCESS_TOKEN env: both name the
same two facts, and both still leave the agent to name the repository per
call. A shell arm now carries a credential env (empty for tea and
raw-api, which need none), merged under PATH in the driver.
Also strengthen SKILL.md so a cold agent targets and authenticates on the
first call: an explicit "Targeting and authentication" section replaces
the buried, optional-looking one-liner, spelling out that outside a
checkout `-R OWNER/NAME` plus the environment's token is all that is
needed — do not go hunting for a config file or login profile.
Verified live: create-memory-leak-issue dropped from 10 turns to 3 and
its cache-read fell ~3.8x, with the auth flailing gone from the transcript.
Records stored only a run's token/turn totals, so an arm's turn cost —
the dominant driver of cache-read tokens — could not be diagnosed from
the store. Retain the ordered transcript of tool invocations (the exact
shell commands, MCP calls, and built-in tools the run made) on every
scored record, absent only for a hung run that produced no transcript.
The canonical TranscriptEntry shape lives on the record (result.ts); the
isolation audit's ToolUse now aliases it so the persisted and audited
shapes cannot drift.
The read checker matched a task's required-fact phrasings as plain
substrings of the agent's report after only lowercasing and collapsing
whitespace. An answer that was substantively correct but wrapped a value in
markdown (e.g. `**5**`) failed the match, because the emphasis markers broke
the phrase adjacency (`**5** open` does not contain `5 open`) — a correct
answer scored incorrect on formatting alone.
Strip markdown emphasis/code markers (`*`, `_`, backtick) during
normalization so the match is on substance, not presentation. A guard test
confirms a wrong value still fails after stripping.
The SDK driver ran the agent with no explicit cwd, so its shell inherited
the harness's own checkout. When the agent omitted `-R OWNER/NAME`, the
gitea-axi (and tea) CLI defaulted the repository from that local checkout —
silently resolving the harness repo instead of the seeded throwaway — and
returned a plausible but wrong result (e.g. `count: 0 open of 0 total` for a
repo with no issues). This contaminated read-tier scoring for the checkout-
defaulting arms and was surfaced by the newly persisted read reports.
Give each run a fresh, empty working directory outside any checkout, so a
forgotten `-R` errors instead of hitting the wrong repository, and delete it
when the run ends.
Make the `issue list` count line name the state it filtered on, so the
answer to "how many issues are open?" is present on the summary line
rather than only inferable from each row. The count line now renders
`count: 5 open of 5 total`; the command composes the state into a generic
optional qualifier while `formatCountLine` stays state-agnostic, so
`pr list`, `search`, and `dashboard` are unaffected. `--state all` imposes
no narrowing and stays unqualified.
The wording is chosen so an agent quoting the summary lands on a phrase the
benchmark read-checker already accepts (`5 open`), closing the accuracy gap
this feature targets. Pairs with the report persistence in task 0032.
Retain the agent's final report on the benchmark result record for read
tasks, so a failed read is diagnosable directly from the stored record
instead of only carrying an opaque `incorrect` tag. The runner resolves
the scoring spec once and records `run.finalReport` when the spec is a
read; mutation records omit the field entirely. The sample store needs no
change — it serializes whatever record it is handed.
This is the prerequisite for confirming the read-open-issue-count failure
from real report text before the state-aware count-line change (task 0033).
Reframe bench/README.md from maintainer-facing internals to a reader-facing
overview: what the benchmark measures, how it works at a high level, and what
it found. Add the completed 4-arm results (240 samples, 2026-07-17).
Remove the per-file layout, the vocabulary glossary, the packaging note, the
spec/ADR pointer, and the run/report/test command sections — that detail lives
in the code and the spec, and it buried the point. The result is a one-screen
doc: pitch and headline finding, a one-paragraph methodology, and the results
table sorted by cost-equivalent tokens.
sumTokens read the Agent SDK's per-model `modelUsage` entries with
snake_case field names, but the SDK reports those per-model entries in
camelCase (`inputTokens`, `cacheReadInputTokens`, ...). Every token
component therefore fell through to zero, silently zeroing the
cost-equivalent-token headline metric — while `total_cost_usd` and
`num_turns` (top-level snake_case) kept working and masked it.
Read `modelUsage` with the correct camelCase fields, keeping the
snake_case aggregate `usage` as the fallback. Export `sumTokens` and add
a regression test covering both the per-model camelCase sum (folding in
the auxiliary model) and the snake_case fallback, so a future SDK
field-casing drift fails a test instead of producing zero-token samples.
Add the maintainer-facing `bench:report` command, the offline counterpart
to `bench:run`: it opens the sample store, drains it, aggregates against the
scored suite and bonus definitions, and prints the aggregator's comparison to
stdout. It renders whatever has accumulated, annotating incomplete coverage
rather than blocking on a complete matrix.
`parseReportArgs` is the pure argument seam (`--store`, `--help`, plus a
`--self-review` / `--no-self-review` variant selector). Unlike `bench:run` the
boundary is offline — it reads only the local store, no host or Agent SDK — so
the whole `runReportCommand` is deterministic and unit-tested, not smoke-run.
Move `DEFAULT_STORE_ROOT` to `store.ts` as the single source of truth,
re-exported from `run.ts` for its existing importers.
Add bench/aggregate.ts: the pure aggregator seam that rolls the
accumulated sample store into a readable comparison. aggregate produces
a headline table (one row per arm, cost-equivalent tokens as the
headline plus raw tokens, turns, duration, success rate, coverage, and
imputed cost as a de-emphasized secondary column), per-tier and
per-token-component breakdowns, and the separate bonus table; renderReport
renders it as stable text. Cost-equivalent tokens are weighted at render
time from the four retained components per ADR 0014, and incomplete
coverage is annotated rather than hidden. Unit-tested against synthetic
append-only sample stores.
Add the maintainer-facing command that runs one chosen benchmark cell on
demand, so only the token budget available at that moment is spent.
runCells (bench/run-loop.ts) runs one (arm, task) cell for a batch of
trials — defaulting to five with a reporting floor of three — by driving
the existing single-cell runner and the append-only sample store rather
than reimplementing orchestration. Re-running a cell deepens it: trial
numbering continues past the highest trial the cell already holds and the
new samples append, so a cell's sample size grows across sittings without
overwriting prior runs.
bench/run.ts is the command: parseRunArgs is the pure, unit-tested
argument seam, and runBenchCommand is the live boundary that resolves host
access, resolves the scored suite against the host's self-review support,
selects the task, and drives the run loop. It is invoked via the new
bench:run npm script, run under tsx (a new devDependency) because the
harness's .js-specifier imports need a TypeScript-aware runner. The Claude
Agent SDK is now declared as an optional peerDependency — documented but
neither installed for package consumers nor pulled into CI.
Every arm runs on the driver's single fixed model; the command exposes no
per-cell model override that could break cross-arm comparability. The
default store root bench/results/ is gitignored.
Add the full 20-task scored suite and the capability-asymmetric bonus
definitions, plus the self-review capability probe that resolves the two
review tasks.
buildScoredSuite returns the shared-surface tasks weighted four read /
six single-mutation / six find-then-act / four multi-step, each a
natural-language intent parametrized against the seed and carrying a tier
and a scoring spec keyed on the single user. The two find-then-act review
tasks are approve/request-changes when the host permits self-review and
comment reviews otherwise; buildBonusTasks emits the approve/request-changes
operations as bonus entries in the fallback case, alongside the static
both-direction bonus definitions (gitea-axi's search/diff/checks/checkout/
issue-dependency edges, and the not-applicable repository/release/milestone
operations).
self-review.ts adds probeSelfReview and detectSelfReviewSupport, the live
boundary that determines self-review support once per sweep; it reuses the
now-exported non-throwing request helper from seed.ts.
Thread every benchmark layer to run one (arm, task, trial) cell end to
end: provision and seed a throwaway repository, run the agent under the
active arm bounded by a turn cap and a wall-clock backstop, audit the
transcript, capture and score the post-run state, append the sample, and
delete the repository.
- runner.ts: runCell orchestration behind the BenchHost and AgentDriver
seams, so the flow is unit-tested with fakes while the live wiring is
validated by a smoke run; turn-cap and wall-clock failures are tagged
confused-versus-hung, and a leaked transcript is flagged invalid.
- audit.ts: the post-run transcript audit plus the shared
foreignToolReason predicate both isolation enforcement points consume.
- task.ts: the runnable BenchTask wrapper and one sample single-mutation
task exercising the full path.
- snapshot.ts: captureRepoState, the seed's read-back counterpart, in the
RepoState shape the checker diffs against.
- host.ts / sdk-driver.ts: the live BenchHost and the Claude Agent SDK
driver (an optional peer, loaded via dynamic import) for real runs.
- runner.smoke.test.ts: the live tracer-bullet tier, skipping cleanly
when no host or SDK is configured.
Add bench/arm.ts, the per-arm scaffolding that produces the single arm
definition the runner consumes. Every arm shares an identical task-agnostic
base prompt and the same repository coordinates and token; each arm then
receives a minimal, symmetric bootstrap.
The deliberate asymmetries follow the shipped products: the gitea-axi arm
embeds the bundled Agent Skill (its body, charging its ambient cost to
gitea-axi); the tea and raw-api arms get a one-line native-discovery pointer;
the gitea-mcp arm runs with the shell disabled and only the MCP server
attached, its dispatcher schemas loading eagerly. Shell arms' PATH and guard
come from bench/guard.ts.
Add the deterministic, idempotent seed that brings a freshly provisioned
throwaway repository to a known ground truth before a trial runs, scripted
over the live Gitea API.
- bench/seed-plan.ts: the pure ground truth (fixed labels, an issue spread
across the discriminating dimensions, and labelled/reviewed/real-branch
pull requests) plus groundTruth(user), realizing it into a RepoState.
- bench/seed.ts: idempotent seeding reconciled by natural key, reusing
gitea-axi's own tea-login credential discovery (no new secret handling).
- A live smoke tier (test:bench:smoke) validating the seed end-to-end and
skipping cleanly when no host is configured, kept out of the deterministic
bench tier.
Export selectLogin from src/context.ts so the bench reuses the exact
credential-selection path.
Add the guard that confines each benchmark arm's agent to exactly one
tool, so a result measures the tool rather than the agent's choice
between tools.
`guardCommand` inspects every binary a proposed shell command would
reach — across pipelines, sequences, subshells, command and process
substitutions, redirections, and leading environment assignments — and
permits only the active arm's one allow-listed binary plus a curated set
of harmless read-only utilities. Foreign binaries, absolute-path
evasions (even of the arm's own binary), and interpreter-based fetch
tricks are denied; the gitea-mcp arm runs with the shell disabled
entirely. `provisionArmBin` produces a curated per-arm bin directory
exposing only that arm's binary as the convenience layer behind the
authoritative guard.
Tests are colocated in bench/guard.test.ts and run via `npm run
test:bench`.
Lay the foundation the benchmark harness reads and writes: a bench/
directory (excluded from the published npm package), the immutable
result-record shape, and an append-only per-cell sample store.
- bench/result.ts: the ResultRecord shape — four token components,
turns, duration, imputed cost, tagged pass/fail outcome, and the
arm/task/tier/trial/timestamp tags. Arm and Tier are typed unions.
- bench/store.ts: append-only sample store, one JSONL file per cell at
<root>/<arm>/<taskId>.jsonl; deepening a cell only ever adds samples.
- bench/README.md: harness working docs and benchmark vocabulary, kept
out of the tool's domain glossary per the spec.
- Dedicated bench test tier (vitest.bench.config.ts, npm run test:bench)
kept out of the fast tier; tsconfig typechecks bench.
- Packaging tier asserts bench/ never ships in the tarball.
Add the benchmark-harness spec, three supporting ADRs (cost-equivalent
token metric, single-user seed, guard-based tool isolation), and the
0022-0030 task breakdown that slices the harness into foundational
seams, an integrating single-cell runner, and the reporting layer.
