diff --git a/skills/axi-review/AXI-PRINCIPLES.md b/skills/axi-review/AXI-PRINCIPLES.md new file mode 100644 index 0000000..f5f2ef4 --- /dev/null +++ b/skills/axi-review/AXI-PRINCIPLES.md @@ -0,0 +1,141 @@ +# The 10 AXI Principles — review rubric + +The canonical statements come from axi.md, which is the authority for the wording. +Each entry distils the canonical statement, gives the black-box probe to run, and the criteria that map observed output to a verdict. +Judge against the canonical statement, not against any single tool's interpretation. + +Verdict states are defined in [`REPORT-FORMAT.md`](REPORT-FORMAT.md). +A deviation that the tool's own skill file documents is still judged by the canonical bar, with the documentation noted on the row. + +## Efficiency + +### Principle 1 — Token-efficient output + +Canonical: structured output uses TOON (Token-Optimized Object Notation), not JSON, omitting braces, quotes, and commas while staying unambiguously parseable. + +Probe: run a list command and a detail command, and inspect stdout. + +- PASS: structured output is TOON — tabular `name[N]{f1,f2}:` blocks with no per-field JSON braces or quotes. +- PARTIAL: some commands emit TOON and others emit JSON or ad-hoc text. +- FAIL: the structured commands emit JSON or free-form prose. + +Failure modes: emitting raw API JSON, pretty-printed JSON, or a human-oriented box-drawing table. + +### Principle 2 — Minimal default schemas + +Canonical: return 3–4 fields per list row by default, with more available through an explicit `--fields` flag. + +Probe: run a list command with no `--fields` and count the fields per row, then confirm `--fields` adds fields. + +- PASS: default list rows carry 3–4 fields and `--fields` opts into more. +- PARTIAL: the default carries 5–6 fields, or `--fields` is absent while the default stays lean. +- FAIL: the default carries 7 or more fields, or every field is always present with no minimal default. + +Failure modes: returning the full API object by default, or offering no `--fields` mechanism. + +### Principle 3 — Content truncation + +Canonical: large text fields are truncated to a limit with a size hint appended, for example `(truncated, 2847 chars total — use --full to see complete body)`. + +Probe: view an entity with a long body, then check that a list command carrying a body field truncates it too rather than leaking it raw, and confirm `--full` (or its equivalent) suppresses truncation on both. +The body-leak check belongs to the list command, not the detail view, where the body is always shown. + +- PASS: long bodies truncate with an inline size hint that names how to get the full text, and `--full` returns the untruncated body. +- PARTIAL: it truncates without a hint, or hints without a `--full` escape, or truncates on detail while leaking raw bodies on a list path. +- FAIL: full bodies are emitted untruncated, or they truncate with no indication at all. + +Failure modes: an unbounded body on a list path that spills many full bodies into context, or silent truncation with no hint. + +## Robustness + +### Principle 4 — Pre-computed aggregates + +Canonical: include derived fields that eliminate round trips, always report total item counts, and provide inline summaries such as `27 passed, 0 failed, 10 skipped`. + +Probe: run a list command and look for a leading count that reports a total, then run a command whose result has a natural rollup (status or checks) and look for an inline summary. + +- PASS: list output leads with a total count such as `count: N of T total`, and derived summaries appear inline where relevant. +- PARTIAL: the count is bare (`count: N` with no total), or some aggregates are present while obvious round trips remain. +- FAIL: no counts and no derived fields, so the agent must issue follow-up calls for basics. + +When the target has no data that exercises the rollup facet, for example no PRs or CI for a checks summary, judge on the count facet alone and note that the rollup was not observable. + +Failure modes: a bare count with no total, or requiring a second call for a review decision or a status rollup. + +### Principle 5 — Definitive empty states + +Canonical: a query that returns nothing emits an explicit zero-result message rather than empty output, because an agent cannot tell a silent failure from a genuinely empty result. + +Probe: run a query guaranteed to return nothing — a nonsense search or a no-match filter — and inspect stdout. + +- PASS: empty results emit an explicit marker such as `[0]: (none)` or `0 open`, never silence. +- PARTIAL: some empty paths are marked while others print nothing, for example a dashboard that is blank when empty. +- FAIL: empty output on no results, indistinguishable from a silent failure. + +Failure modes: printing nothing, or exiting 0 with blank stdout. + +### Principle 6 — Structured errors and exit codes + +Canonical: mutations are idempotent, errors are structured and written to stdout rather than stderr, commands never prompt for interactive input, exit code is 0 for success and 1 for errors and 2 for unknown flags, and unknown flags fail loudly rather than being silently ignored. + +Probe, read facet (safe): trigger a not-found id, an unknown flag, and a bad target, then check for a structured error block on stdout with a code field and the right exit codes, with 2 for the unknown flag. +Probe, write facet: run a state-toggling mutation and then repeat it — one whose second run is a no-op, such as closing an already-closed entity or pinning an already-pinned one, not a create, which duplicates by design. +Observe an idempotent result on the repeat, an early return or an `already:` marker, and confirm no command prompts. +If the target rejects the write probes, the write facet is BLOCKED. + +- PASS: errors are a structured block on stdout with a code, exit codes are 0/1/2 as canonical, unknown flags exit 2, no command prompts, and a repeated mutation is idempotent. +- PARTIAL: errors are structured but on stderr, or exit codes are collapsed to a single value, or an unknown flag is silently ignored, or some mutations are not idempotent. +- FAIL: errors are free-form text on stderr, or a command prompts interactively, or a repeated mutation duplicates its effect. +- BLOCKED: the write target rejected the mutation probes, so the idempotency facet could not be observed — record the reason. + +Failure modes: writing errors to stderr, prompting for confirmation, or exiting 1 for every outcome. + +## Discoverability + +### Principle 7 — Ambient context + +Canonical: install into the agent's session from an explicit setup command, so every conversation starts with relevant state visible, and ship an installable skill for on-demand guidance. + +Probe: locate the tool's bundled Agent Skill file, under `~/.claude/skills//` or in the installed package or repo, and check the surface for an explicit setup command. + +- PASS: the tool ships a discoverable Agent Skill installed by an explicit setup command, optionally alongside a hook that injects a dashboard as initial context. +- PARTIAL: a setup command exists but installs no skill, or a skill exists with no explicit setup path, for example one installed by a postinstall script. +- FAIL: no discoverable skill file at all. + +Failure modes: shipping no skill, or auto-installing via a postinstall script instead of an explicit setup command. + +### Principle 8 — Content first + +Canonical: running the command with no arguments displays live, actionable data rather than help text, preceded by the executable path (with `~` for home) and a one-sentence description of the tool's purpose. + +Probe: run the bare invocation with no arguments. + +- PASS: no-arg output is live tool state, preceded by a header naming the executable path and a one-sentence purpose. +- PARTIAL: it shows live data without the path-and-description header, or mixes help text with data. +- FAIL: no-arg output is a help or usage screen, or it errors. + +Failure modes: printing usage text on no args, or showing no live data. + +### Principle 9 — Contextual disclosure + +Canonical: append `help[]` lines after output suggesting logical next steps as concrete command templates, keeping fixed disambiguating flags but leaving runtime values parameterized as placeholders like `` rather than guessing them. + +Probe: inspect the list and detail outputs for a `help[]` block, checking placeholders on list output and a filled id on unambiguous single-entity output. + +- PASS: commands append a `help[]` block of concrete next-step templates, parameterizing runtime values as placeholders, and may fill a known id where a single entity makes it unambiguous. +- PARTIAL: suggestions appear on some commands but not others, or they hardcode a guessed runtime value, or the `help[]` block is empty. +- FAIL: no next-step suggestions anywhere. + +Failure modes: guessing an id in a suggestion, or omitting suggestions. + +### Principle 10 — Consistent way to get help + +Canonical: each subcommand offers a concise `--help` flag as a fallback when contextual hints are insufficient. + +Probe: run `--help` on the root and on at least one subcommand. + +- PASS: every subcommand responds to `--help` with a concise flag reference. +- PARTIAL: `--help` works on some subcommands but not all, or its output is inconsistent or verbose. +- FAIL: no `--help`, or it errors or prompts. + +Failure modes: `--help` missing on subcommands, or help that prompts interactively. diff --git a/skills/axi-review/REPORT-FORMAT.md b/skills/axi-review/REPORT-FORMAT.md new file mode 100644 index 0000000..d589ee1 --- /dev/null +++ b/skills/axi-review/REPORT-FORMAT.md @@ -0,0 +1,54 @@ +# Report card format + +The card is rendered in-session as terminal markdown. +It reports all 10 principles, in canonical order grouped by category, so the whole rubric is visible rather than only the failures. + +## Verdict states + +- **PASS** — observed behaviour meets the canonical bar. +- **PARTIAL** — the principle is partly met, for example TOON output with no truncation hint. +- **FAIL** — observed behaviour violates the principle. +- **N/A** — the principle genuinely does not apply, for example an aggregate count on a tool with no list commands. +- **BLOCKED** — the probe could not be run, so behaviour was not observed. This is not a FAIL. It is reserved for the write facet of Principle 6 when the target rejects the mutation probes. + +## Structure + +Lead with a header naming the reviewed invocation and a one-line tally, then the skill-file status, then one table of all 10 principles in canonical order, then the artifacts manifest. +Each principle is one row. +The `Evidence` cell is the command that was run plus a short inline snippet of its output. +The `Fix / Note` cell carries a concrete fix when the verdict is not PASS, or the documented-deviation note when the skill file documents the deviation, and is left empty for a clean PASS. + +``` +# AXI review — + +** PASS · PARTIAL · FAIL · N/A · BLOCKED** + +Skill file: + +| # | Principle | Verdict | Evidence | Fix / Note | +|---|-----------|---------|----------|------------| +| 1 | Token-efficient output | | `` → | | +| 2 | Minimal default schemas | | `` → | | +| 3 | Content truncation | | `` → | | +| 4 | Pre-computed aggregates | | `` → | | +| 5 | Definitive empty states | | `` → | | +| 6 | Structured errors and exit codes | | `` → | | +| 7 | Ambient context | | `` → | | +| 8 | Content first | | `` → | | +| 9 | Contextual disclosure | | `` → | | +| 10 | Consistent way to get help | | `` → | | + +## Artifacts created + + +``` + +## Rules + +Every non-PASS row carries a concrete fix in `Fix / Note`, a real remedy rather than a restatement of the principle. + +The `Evidence` cell is a terse one-liner — the command plus a short inline snippet, never a multi-line dump. +When one command's output is the evidence for several principles, cite the same command in each row. + +The artifacts manifest lists every artifact the write battery created, with the identifier needed to find it, so the user can remove anything they do not want. +When writes were BLOCKED and nothing was created, the manifest says so explicitly. diff --git a/skills/axi-review/SKILL.md b/skills/axi-review/SKILL.md new file mode 100644 index 0000000..5fc9db1 --- /dev/null +++ b/skills/axi-review/SKILL.md @@ -0,0 +1,65 @@ +--- +name: axi-review +description: Review a CLI for conformance to the 10 AXI (Agent eXperience Interface) principles by running it black-box and scoring each principle on a report card. Use when asked to review, audit, check, or measure a CLI's AXI conformance, or when "AXI principles", "Agent eXperience Interface", or axi.md is named. +--- + +# axi-review + +Review a CLI against the 10 canonical AXI principles by running it and judging what it emits. +The review is black-box: the tool's observed output is the evidence, and source is consulted only where a facet cannot be triggered by running the tool. +It writes by default, so it creates real artifacts in a target you provide. + +The rubric lives in [`AXI-PRINCIPLES.md`](AXI-PRINCIPLES.md) — one entry per principle, each with the probe to run and its PASS/PARTIAL/FAIL criteria. +The card's shape lives in [`REPORT-FORMAT.md`](REPORT-FORMAT.md). + +## 1. Establish what you are reviewing + +Fix the invocation string first — how this CLI is run (`gitea-axi`, a binary path, `npx -y foo`). + +Locate the tool's own Agent Skill file, since it is the authoritative description of the command surface and is itself the evidence for Principle 7. +Look under `~/.claude/skills//SKILL.md`, then in the installed package or the repo. +A missing skill file is a Principle 7 FAIL that you carry into the judging step. +When it is missing, enumerate the surface from `--help` and no-arg output instead. + +Run the bare no-arg invocation to confirm the tool executes. +If nothing runs at all — binary absent, `npx` fails — stop and report that a behavioural review is impossible. +Emit no card in that case. + +Done when the tool has run, you have its subcommand surface, and you know whether a skill file exists. + +## 2. Get the write target + +The write battery creates real artifacts, so it needs a target you are cleared to mutate — the repo, folder, or destination this CLI acts on. +How the tool takes that target comes from its skill file, for example gitea-axi's `-R owner/name`. + +If the invoking prompt did not name a target, ask the user for one in prose before running any command. + +Done when you have a write target you may create artifacts in. + +## 3. Run the battery once, capturing a transcript + +Read [`AXI-PRINCIPLES.md`](AXI-PRINCIPLES.md) and run every probe it lists, capturing each command's stdout and exit code into one transcript. +The battery is staged, not a single command: run the empty-state reads first, then the write probes, then the reads that depend on what the writes created. +Run it once in that sense — one transcript reused across principles, not a fresh run per principle — since one command's output is evidence for several. + +The write probes include creating any fixture a read probe needs, such as the long-body entity Principle 3 inspects. +The read probes create nothing. +The write probes create real artifacts — record each one in the artifacts manifest as you create it, so nothing is orphaned silently. +Capture stdout and stderr separately, since Principle 6 turns on which stream an error lands in. +If the target rejects the write probes, mark the write facet of Principle 6 BLOCKED with the reason and continue with the read probes. + +Done when every probe has a captured result or a recorded BLOCKED reason, and every created artifact is in the manifest. + +## 4. Judge all 10 principles + +For each principle in [`AXI-PRINCIPLES.md`](AXI-PRINCIPLES.md), apply its criteria to the transcript and assign PASS, PARTIAL, FAIL, N/A, or BLOCKED. +Judge strictly against the canonical statement. +When the tool's skill file documents a deviation, note it on the row but keep the canonical verdict. + +Done when all 10 principles carry a verdict backed by a cited command and a short output excerpt from the transcript. + +## 5. Emit the report card + +Render the card in-session following [`REPORT-FORMAT.md`](REPORT-FORMAT.md). + +Done when the card matches `REPORT-FORMAT.md` and every non-PASS row carries a concrete fix.