Make the `issue list` count line name the state it filtered on, so the
answer to "how many issues are open?" is present on the summary line
rather than only inferable from each row. The count line now renders
`count: 5 open of 5 total`; the command composes the state into a generic
optional qualifier while `formatCountLine` stays state-agnostic, so
`pr list`, `search`, and `dashboard` are unaffected. `--state all` imposes
no narrowing and stays unqualified.
The wording is chosen so an agent quoting the summary lands on a phrase the
benchmark read-checker already accepts (`5 open`), closing the accuracy gap
this feature targets. Pairs with the report persistence in task 0032.
Retain the agent's final report on the benchmark result record for read
tasks, so a failed read is diagnosable directly from the stored record
instead of only carrying an opaque `incorrect` tag. The runner resolves
the scoring spec once and records `run.finalReport` when the spec is a
read; mutation records omit the field entirely. The sample store needs no
change — it serializes whatever record it is handed.
This is the prerequisite for confirming the read-open-issue-count failure
from real report text before the state-aware count-line change (task 0033).
Reframe bench/README.md from maintainer-facing internals to a reader-facing
overview: what the benchmark measures, how it works at a high level, and what
it found. Add the completed 4-arm results (240 samples, 2026-07-17).
Remove the per-file layout, the vocabulary glossary, the packaging note, the
spec/ADR pointer, and the run/report/test command sections — that detail lives
in the code and the spec, and it buried the point. The result is a one-screen
doc: pitch and headline finding, a one-paragraph methodology, and the results
table sorted by cost-equivalent tokens.
sumTokens read the Agent SDK's per-model `modelUsage` entries with
snake_case field names, but the SDK reports those per-model entries in
camelCase (`inputTokens`, `cacheReadInputTokens`, ...). Every token
component therefore fell through to zero, silently zeroing the
cost-equivalent-token headline metric — while `total_cost_usd` and
`num_turns` (top-level snake_case) kept working and masked it.
Read `modelUsage` with the correct camelCase fields, keeping the
snake_case aggregate `usage` as the fallback. Export `sumTokens` and add
a regression test covering both the per-model camelCase sum (folding in
the auxiliary model) and the snake_case fallback, so a future SDK
field-casing drift fails a test instead of producing zero-token samples.
Add the maintainer-facing `bench:report` command, the offline counterpart
to `bench:run`: it opens the sample store, drains it, aggregates against the
scored suite and bonus definitions, and prints the aggregator's comparison to
stdout. It renders whatever has accumulated, annotating incomplete coverage
rather than blocking on a complete matrix.
`parseReportArgs` is the pure argument seam (`--store`, `--help`, plus a
`--self-review` / `--no-self-review` variant selector). Unlike `bench:run` the
boundary is offline — it reads only the local store, no host or Agent SDK — so
the whole `runReportCommand` is deterministic and unit-tested, not smoke-run.
Move `DEFAULT_STORE_ROOT` to `store.ts` as the single source of truth,
re-exported from `run.ts` for its existing importers.
Add bench/aggregate.ts: the pure aggregator seam that rolls the
accumulated sample store into a readable comparison. aggregate produces
a headline table (one row per arm, cost-equivalent tokens as the
headline plus raw tokens, turns, duration, success rate, coverage, and
imputed cost as a de-emphasized secondary column), per-tier and
per-token-component breakdowns, and the separate bonus table; renderReport
renders it as stable text. Cost-equivalent tokens are weighted at render
time from the four retained components per ADR 0014, and incomplete
coverage is annotated rather than hidden. Unit-tested against synthetic
append-only sample stores.
Add the maintainer-facing command that runs one chosen benchmark cell on
demand, so only the token budget available at that moment is spent.
runCells (bench/run-loop.ts) runs one (arm, task) cell for a batch of
trials — defaulting to five with a reporting floor of three — by driving
the existing single-cell runner and the append-only sample store rather
than reimplementing orchestration. Re-running a cell deepens it: trial
numbering continues past the highest trial the cell already holds and the
new samples append, so a cell's sample size grows across sittings without
overwriting prior runs.
bench/run.ts is the command: parseRunArgs is the pure, unit-tested
argument seam, and runBenchCommand is the live boundary that resolves host
access, resolves the scored suite against the host's self-review support,
selects the task, and drives the run loop. It is invoked via the new
bench:run npm script, run under tsx (a new devDependency) because the
harness's .js-specifier imports need a TypeScript-aware runner. The Claude
Agent SDK is now declared as an optional peerDependency — documented but
neither installed for package consumers nor pulled into CI.
Every arm runs on the driver's single fixed model; the command exposes no
per-cell model override that could break cross-arm comparability. The
default store root bench/results/ is gitignored.
Add the full 20-task scored suite and the capability-asymmetric bonus
definitions, plus the self-review capability probe that resolves the two
review tasks.
buildScoredSuite returns the shared-surface tasks weighted four read /
six single-mutation / six find-then-act / four multi-step, each a
natural-language intent parametrized against the seed and carrying a tier
and a scoring spec keyed on the single user. The two find-then-act review
tasks are approve/request-changes when the host permits self-review and
comment reviews otherwise; buildBonusTasks emits the approve/request-changes
operations as bonus entries in the fallback case, alongside the static
both-direction bonus definitions (gitea-axi's search/diff/checks/checkout/
issue-dependency edges, and the not-applicable repository/release/milestone
operations).
self-review.ts adds probeSelfReview and detectSelfReviewSupport, the live
boundary that determines self-review support once per sweep; it reuses the
now-exported non-throwing request helper from seed.ts.
Thread every benchmark layer to run one (arm, task, trial) cell end to
end: provision and seed a throwaway repository, run the agent under the
active arm bounded by a turn cap and a wall-clock backstop, audit the
transcript, capture and score the post-run state, append the sample, and
delete the repository.
- runner.ts: runCell orchestration behind the BenchHost and AgentDriver
seams, so the flow is unit-tested with fakes while the live wiring is
validated by a smoke run; turn-cap and wall-clock failures are tagged
confused-versus-hung, and a leaked transcript is flagged invalid.
- audit.ts: the post-run transcript audit plus the shared
foreignToolReason predicate both isolation enforcement points consume.
- task.ts: the runnable BenchTask wrapper and one sample single-mutation
task exercising the full path.
- snapshot.ts: captureRepoState, the seed's read-back counterpart, in the
RepoState shape the checker diffs against.
- host.ts / sdk-driver.ts: the live BenchHost and the Claude Agent SDK
driver (an optional peer, loaded via dynamic import) for real runs.
- runner.smoke.test.ts: the live tracer-bullet tier, skipping cleanly
when no host or SDK is configured.
Add bench/arm.ts, the per-arm scaffolding that produces the single arm
definition the runner consumes. Every arm shares an identical task-agnostic
base prompt and the same repository coordinates and token; each arm then
receives a minimal, symmetric bootstrap.
The deliberate asymmetries follow the shipped products: the gitea-axi arm
embeds the bundled Agent Skill (its body, charging its ambient cost to
gitea-axi); the tea and raw-api arms get a one-line native-discovery pointer;
the gitea-mcp arm runs with the shell disabled and only the MCP server
attached, its dispatcher schemas loading eagerly. Shell arms' PATH and guard
come from bench/guard.ts.
Add the deterministic, idempotent seed that brings a freshly provisioned
throwaway repository to a known ground truth before a trial runs, scripted
over the live Gitea API.
- bench/seed-plan.ts: the pure ground truth (fixed labels, an issue spread
across the discriminating dimensions, and labelled/reviewed/real-branch
pull requests) plus groundTruth(user), realizing it into a RepoState.
- bench/seed.ts: idempotent seeding reconciled by natural key, reusing
gitea-axi's own tea-login credential discovery (no new secret handling).
- A live smoke tier (test:bench:smoke) validating the seed end-to-end and
skipping cleanly when no host is configured, kept out of the deterministic
bench tier.
Export selectLogin from src/context.ts so the bench reuses the exact
credential-selection path.
Add the guard that confines each benchmark arm's agent to exactly one
tool, so a result measures the tool rather than the agent's choice
between tools.
`guardCommand` inspects every binary a proposed shell command would
reach — across pipelines, sequences, subshells, command and process
substitutions, redirections, and leading environment assignments — and
permits only the active arm's one allow-listed binary plus a curated set
of harmless read-only utilities. Foreign binaries, absolute-path
evasions (even of the arm's own binary), and interpreter-based fetch
tricks are denied; the gitea-mcp arm runs with the shell disabled
entirely. `provisionArmBin` produces a curated per-arm bin directory
exposing only that arm's binary as the convenience layer behind the
authoritative guard.
Tests are colocated in bench/guard.test.ts and run via `npm run
test:bench`.
Lay the foundation the benchmark harness reads and writes: a bench/
directory (excluded from the published npm package), the immutable
result-record shape, and an append-only per-cell sample store.
- bench/result.ts: the ResultRecord shape — four token components,
turns, duration, imputed cost, tagged pass/fail outcome, and the
arm/task/tier/trial/timestamp tags. Arm and Tier are typed unions.
- bench/store.ts: append-only sample store, one JSONL file per cell at
<root>/<arm>/<taskId>.jsonl; deepening a cell only ever adds samples.
- bench/README.md: harness working docs and benchmark vocabulary, kept
out of the tool's domain glossary per the spec.
- Dedicated bench test tier (vitest.bench.config.ts, npm run test:bench)
kept out of the fast tier; tsconfig typechecks bench.
- Packaging tier asserts bench/ never ships in the tarball.
Add the benchmark-harness spec, three supporting ADRs (cost-equivalent
token metric, single-user seed, guard-based tool isolation), and the
0022-0030 task breakdown that slices the harness into foundational
seams, an integrating single-cell runner, and the reporting layer.
Rule that the `body` extra field truncates at 500 chars like `issue view`,
resolving the spec's Principle 3 / Command Surface contradiction. Route the
`body` extractor through a new `truncatedBody()` FieldDef so `issue list`,
`issue create`, `pr list`, and `search` all present it exactly as the detail
views do, and add a `--full` flag to each to suppress truncation, keeping the
inline hint's "use --full" promise honest.
Complete the distribution metadata and publish flow for the unscoped
`gitea-axi` package:
- add `repository`, `homepage`, and `bugs` to package.json
- add `publishConfig` (public access, npmjs registry) so `npm publish`
needs no extra flags
- replace `prepublishOnly` with `prepack: npm run build`, so both
`npm pack` and `npm publish` rebuild `dist/` first
- document the single-command flow in PUBLISHING.md
- add a packaging smoke-test tier (`test:pack`) that packs the real
tarball, installs it globally, and drives the installed binary
(`--help`, dashboard header, and `setup` finding the bundled skill)
Distribute gitea-axi's ambient context via explicit user actions (ADRs
0009, 0013), with no postinstall script:
- Bundle the Agent Skill markdown at skills/gitea-axi/SKILL.md (a
minimal pointer, not a command reference) and ship it via package.json
files.
- Add `setup`, which installs the skill into ~/.claude/skills/
idempotently (installed/updated/unchanged).
- Add `setup hooks`, which registers a SessionStart hook running the
bare dashboard for Claude Code, Codex, and OpenCode via the SDK's
installSessionStartHooks(), updating managed entries in place.
- Shadow the SDK's built-in `update` so it fails with VALIDATION_ERROR
and points at the npm update command, keeping the ten-code error list
intact.
Integration tests drive all three at the CLI seam against a temporary
HOME; these commands make no Gitea API calls, so there is no live-Gitea
e2e case.
Wire the bare `gitea-axi` home command to a two-tier repository
dashboard (ADR 0012): the short tier shows up to 3 open issues and 3
open PRs with the client-side `review` decision, and `--full` shows the
20-row open-PR table plus open issue counts grouped by label, aggregated
across every page of open issues up to the 1000-issue cap with a `+`
suffix when capped. Empty states render the raw `prs: 0 open` /
`issues: 0 open` strings; issue fetches pass `type=issues`; outside a
Gitea repo the dashboard errors with REPO_NOT_FOUND.
`paginate.ts` now reports whether pagination stopped at the cap, which
drives the label-count `+` suffix.
Gitea's issue/PR search endpoint is backed by an asynchronous, eventually-
consistent indexer (bleve by default), so a PR opened moments earlier in the
e2e `beforeAll` was not yet searchable when `search prs` ran, and the live
assertion saw zero matches. Both live-search tests now poll the search via
`expect.poll` until the freshly-created content is indexed before asserting on
the exact locator-schema output. Record the eventual-consistency behaviour as
a project gotcha.
Add `search issues <query>` and `search prs <query>`, the full-text escape
hatch the forbidden `--search` flag on the list commands redirects to.
Both hit Gitea's cross-repo issue-search endpoint with the query, a `type`
of issues or pulls, and the owner param, then filter results to the current
repository client-side via each result's `repository` field — the endpoint
has no repo-name filter. The count line reports `count: N of T total` with
`T` from the filtered set (ADR 0005), never the endpoint's cross-repo
`X-Total-Count`.
The positional query is required (VALIDATION_ERROR if missing). Flags:
`--state` (default open), `--label` (comma-separated names passed straight
through as the API `labels` param), `--limit` (default 30), and `--fields`.
Default output is the locator schema (`number`, `title`, `state`, `author`,
`created`) under `issues:` / `pull_requests:` blocks matching the list
commands — search finds the number, `issue view` / `pr view` load the detail.
Covered by fixture-server tests for both types, cross-repo filtering, the
count rule, each flag, the empty state, and missing-query validation, plus
end-to-end tests against a live Gitea instance.
Add the two PR commands that touch content and the local worktree:
- `pr diff <n>` fetches the raw diff from the `.diff` endpoint (forcing a
text response, which the JSON-defaulting client would otherwise discard),
truncates at 4000 chars with separate `truncated`/`original_length` fields
and a prepended `--full` suggestion; `--full` returns the raw diff.
- `pr checkout <n>` reads the PR head branch from the PR fetch and fetches
`refs/pull/{n}/head` from origin (uniform for same-repo and fork PRs, ADR
0011), three-cased on local branch state so re-checkout is idempotent and
divergent local commits fail with `GIT_ERROR` rather than being discarded.
Introduces the `GIT_ERROR` mapping (`runGit`) carrying git's first stderr
line plus a remediation help line, and widens the PR 404 classifier so a
`.diff` path still resolves to `PR_NOT_FOUND`.
Add `pr review <n>` with the three action switches --approve,
--request-changes, and --comment, plus --body/--body-file. Exactly one
action flag is required; zero or multiple raise VALIDATION_ERROR before
any API call, mirroring the merge shorthand-conflict rule. Body
requirements are left to Gitea: a body-less event it rejects surfaces its
422 as VALIDATION_ERROR carrying the server's message. Output is
`review: { number, action }`.
Add `pr merge` with all six Gitea methods, the three common-method
shorthands, `--auto`, `--delete-branch`, `--subject`, `--body`/`--body-file`,
and `--merge-commit-id` (required with and only valid for
`--method manually-merged`). Conflicting method flags and the
manually-merged/commit-id pairing are rejected as VALIDATION_ERROR before
any API call. An already-merged PR short-circuits to a `pull_request` entity
block with `merged_by`/`merged_at`; merge-blocked 405/409 responses surface
as VALIDATION_ERROR with update-branch/checkout remediation.
Add `pr update-branch` merging the base into the head via the update
endpoint with `--style <merge|rebase>` (default merge).
Add the PR-side state mutations mirroring the issue-side slice:
- `pr edit` applies title/body/base/milestone and the recomputed assignee
list in one PATCH, with additive label endpoints and (per the ADR 0007
amendment) the dedicated requested-reviewers POST/DELETE endpoints for
`--add-reviewer`/`--remove-reviewer`.
- `pr close --comment` posts the comment after the PATCH and surfaces a
comment-post failure; an already-closed or merged PR is an `already: true`
no-op reporting the actual state.
- `pr reopen` is an `already: true` no-op when already open.
Extract the fetch-then-patch assignee merge into a shared `src/assignees.ts`
(`mergeAssignees` + `assigneeLogins`), now used by both `issue edit` and
`pr edit`.
Add `pr view <n>` and `pr checks <n>`, built on the truncation machinery
and review fetches from earlier slices.
`pr view` uses the three-call fetch pattern (PR + reviews in parallel, then
the head commit's combined status) so `checks`, `comment_count`, and
`review_count` are in the default output; `--comments`, `--reviews`, and
`--full` behave as on `issue view`, with `--reviews` exposing Gitea's
`official`/`stale` fields plus per-review inline comments.
`pr checks <n>` renders the checks summary line and the `{ name, conclusion }`
rows, or the scalar no-CI message when no statuses exist.
The state→conclusion mapping and summary live in a new `src/checks.ts`;
`commentRows` is extracted into `src/comment.ts` and shared with `issue view`.
Implements `pr list` with the two policies later PR slices reuse:
client-side filtering with the filtered-set count line (ADR 0005) and
the official-first reviewDecision via parallel per-PR review fetches
(ADR 0006), extracted into src/review.ts.
API-supported flags map to their params (--state, --author→poster,
--label name→ID, --label-id, --sort, --limit, --fields); --assignee,
--base, --head, and --draft filter in-process after full pagination,
with the count line's total taken from the filtered set. --search is
refused with a redirect to `search prs`.
Adds a boolText field extractor and a shared parsePositiveInt helper,
the latter also adopted by issue list's --limit parsing.
Add the remaining simple issue mutations. `issue delete` hard-deletes via
the DELETE endpoint and is deliberately not idempotent — a nonexistent
issue yields ISSUE_NOT_FOUND rather than reporting success. `issue pin`
and `issue unpin` read the current pin state (Gitea's pin_order field) and
short-circuit to an idempotent no-op with an Already pinned/unpinned
message when there is nothing to do.
Implement the issue state-transition mutations:
- issue edit: --title, --body/--body-file, --add-label/--remove-label,
--add-assignee/--remove-assignee, --milestone. Label mutations use
Gitea's dedicated additive/removal endpoints (names added directly,
removals resolved to IDs, unapplied-label 404 as silent success);
assignees use fetch-then-patch (ADR 0007); title/body/milestone and the
recomputed assignee list travel in one PATCH. Outputs edited: {number,
status: ok}.
- issue close: PATCHes state closed, optional --comment posted after with
its failure surfaced; already-closed short-circuits with Already closed.
- issue reopen: PATCHes state open; already-open short-circuits with
Already open.
Extract a shared getIssue helper used by view/edit/close/reopen.
Finish the `issue list` flag surface left minimal by the tracer slice:
- Server-side filters `--label`, `--assignee`, `--author`, `--milestone`,
mapped to Gitea's `labels`, `assigned_by`, `created_by`, `milestones`.
- Client-side `--sort <created|updated|comments>`, always descending, over
the fully paginated set (ADR 0005); `--limit` caps the sorted result.
- `--fields` exposing body, closedAt, labels, milestone, updatedAt, url.
- `--search` refused with a VALIDATION_ERROR redirecting to `search issues`.
Exhaustive pagination lands as a shared `paginate.ts`, since ADR 0005 makes
it a policy later slices reuse; `lookup.ts` drops its hand-rolled copy of the
same loop. The helper carries the 20-page cap that copy encoded, matching the
1000-item ceiling Principle 8 sets.
`pr create` takes --title (required), --body/--body-file, --base, --head,
--assignee, --reviewer, repeatable name-resolved --label, and --milestone.
An omitted --head defaults to the current local branch; an omitted --base to
the repository's default branch. Before creating, an existing open PR for the
same base/head pair short-circuits to `pull_request: { number, url, already:
true }` rather than opening a duplicate; only an open PR does, since a closed
one's branches are free to be proposed again.
`pr comment <n>` posts through the shared issue-comment endpoint and returns
the created comment as `comment: { number, author, created, body }` (ADR 0008),
reporting a 404 as PR_NOT_FOUND since the caller asked about a pull request.
That comment block is now built in one place (src/comment.ts) for both issue
and pr comment, as ADR 0008 requires them to stay identical.
Introduce the first mutations, along with the shared machinery the later
issue and PR mutation slices reuse.
- `issue create` with --title/--body/--body-file/--assignee/--label/
--milestone/--fields, emitting `issue: { number, title, state, url }`
- `issue comment <n>`, echoing the created comment from the POST response
with the body cleaned and truncated at 800 chars
- body-source resolution (--body vs --body-file), label and milestone
name->ID lookup, repeatable flags, and the `joined`/`selectExtraFields`
field extractors
Label lookup pages until exhausted, since a repo with more labels than one
page would otherwise fail to resolve a valid name. The end-to-end tier seeds
a mixed-case label and milestone and passes both in a different case, so the
case-insensitive lookup is verified against live Gitea rather than only
against fixtures.
Add `issue view <n>` as the first detail command, introducing the
content-cleaning and truncation machinery (src/body.ts) that later issue
and PR slices reuse.
- Default output: number, title, state, author, created, body (truncated
at 500), plus comment_count; --comments expands every comment (bodies
truncated at 800); --full suppresses all truncation.
- cleanBody runs only when a body exceeds its limit: normalizes Gitea
issue/PR URLs on the detected host to Issue#N/PR#N, strips image embeds
and long URLs, and collapses quoted blocks.
- Type guard: a PR number fails with VALIDATION_ERROR and a `pr view <n>`
hint, detected via the fetched object's pull_request field.
- renderDetail joins the detail entity, optional sub-blocks, and help.
Cover the previously untested credential and error-classification paths:
- tea.ts: failing/invalid/non-array `tea login list`, a failing token
helper, and an empty token, all driven through the CLI seam via the fake
tea harness (now parametrizable for exit codes and output).
- errors.ts: a unit-tier test of classifyHttpError for the pure-classifier
branches no shipped command reaches yet (issue/pull 404s, the
already-classified passthrough, transport failures, no-body defaults).
errors.ts is now fully covered and tea.ts is at 96% (the remainder is
defensive code unreachable through the seam). Overall coverage rose from
~87% to ~93% statements / ~89% branches; the ratchet is raised to
92/87/95/92 accordingly.
Add v8 coverage via a test:coverage script and vitest config: measure src
only, exclude the types-only deps.ts and the bin entrypoint main.ts, and
gate on ratcheted thresholds (statements 85, branches 78, functions 90,
lines 85) set just under current coverage. CI runs the unit+integration
tier through coverage so a real drop fails the build.
Add the end-to-end test tier: a Gitea Actions workflow (GitHub-compatible)
that runs the unit+integration tiers plus e2e tests against a pinned
disposable Gitea service container. The e2e suite provisions its own repo,
token, and seed data over HTTP and drives the real CLI seam, gated on
GITEA_AXI_E2E_URL so the default suite stays dependency-free.
A fixture-vs-live shape guard anchors the issue-list FieldDef paths against
both the recorded fixture and the live response, failing on divergence.
- CONVENTIONAL-COMMITS.md: vendored spec (v1.0.0); CLAUDE.md requires
agent-written commits to follow it
- tasks 0002-0020: remaining task breakdown of the gitea-axi spec
(0019 reframed around the three-tier test taxonomy: unit,
integration, end-to-end)
- spec, CONTEXT.md, and ADRs 0006/0007/0011: pending design
refinements from the planning sessions
Tracer bullet for gitea-axi: runnable npm package on axi-sdk-js with
gitea-js as the sole HTTP layer, ESM on Node 20+.
- issue list with --state/--limit, default fields, count line from
X-Total-Count, type=issues guard, explicit empty state, and next-step
suggestions
- repo context detection from the git origin remote (SSH/scp/HTTPS),
tea credential discovery with the three-way login-matching split, and
-R/--repo and --login overrides (flag > env > auto)
- token retrieval via tea login helper get: tea's login list JSON
carries no token (ADR 0001 amended)
- full AxiError classification table with path-based 404 split, TOON
errors on stdout, exit codes 0/1/2
- test mode (GITEA_AXI_API_URL/TOKEN/REPO) suppressing subprocesses,
fixture server, and vitest suites driving the CLI seam (50 tests)
