Compare commits

..

42 Commits

Author SHA1 Message Date
9b2a6bcd58 Clear reconciled wayfinder subagents 2026-08-02 14:23:44 -04:00
13cf92f627 fix(slice): write implementation tickets 2026-08-01 22:10:14 -04:00
40b16b8796 fix(skills): require isolated review and test workers 2026-08-01 18:19:32 -04:00
0f2b13c5f9 feat(wayfinder): default to isolated frontier workers 2026-08-01 17:52:09 -04:00
d758f722f9 feat(skills): add isolated worker workflow guidance 2026-08-01 16:37:09 -04:00
346413cd7d Add implementation workflow skills 2026-08-01 12:12:07 -04:00
f283646124 Add slice skill 2026-07-31 23:36:43 -04:00
99df169fbd Relax Wayfinder session ticket limit 2026-07-31 20:46:14 -04:00
8035e0e436 Allow AFK prototype tickets 2026-07-31 20:43:18 -04:00
012a4bfd1e Unify Wayfinder ticket artifacts 2026-07-31 20:38:27 -04:00
b5c6e33730 Tighten grill turn-ending protocol 2026-07-31 19:36:03 -04:00
778e726d4c Trim redundant skill-build fixture 2026-07-31 18:51:34 -04:00
d3bc9a5760 Remove benchmark skill 2026-07-31 18:50:42 -04:00
8479ad096c Remove obsolete skills placeholder 2026-07-31 18:48:54 -04:00
d0c9f94713 Remove consume and wiki skills 2026-07-31 18:45:55 -04:00
942d32dcba Make artifact naming destination-driven 2026-07-31 18:45:11 -04:00
96f795a63a fix(wayfinder): simplify map references 2026-07-31 16:03:00 -04:00
80cecf3350 feat(skills): adapt planning skills to artifact workflow 2026-07-31 15:13:31 -04:00
181dcc7a9e docs: record gitea-axi view limitation 2026-07-31 10:41:45 -04:00
7d22ff0c02 fix(skills): align invocation and prose conventions 2026-07-31 10:40:56 -04:00
4da2452084 feat(wayfinder): add artifact-backed planning skill 2026-07-31 10:35:25 -04:00
7550096968 feat(prototype): add throwaway prototype skill 2026-07-31 10:28:50 -04:00
cbc03650d1 feat(research): add durable research skill 2026-07-31 10:05:24 -04:00
8ad93c7617 feat(skills): support flat numbered artifacts 2026-07-31 09:23:13 -04:00
43280e4d4a fix(design-skill): tighten predictability checks 2026-07-30 19:34:41 -04:00
f72c262229 Add design-skill skill 2026-07-30 19:03:29 -04:00
fbd009337d Remove obsolete project skill placeholder 2026-07-30 00:30:19 -04:00
920517e623 Add project skills to dev shell 2026-07-30 00:28:43 -04:00
6bb3242e7f Move consume artifacts to artifact store 2026-07-30 00:24:52 -04:00
fb16c891b2 Move wiki source to artifact store 2026-07-30 00:15:15 -04:00
5886017976 Move grill artifacts outside project repos 2026-07-30 00:05:59 -04:00
b541251f98 feat(agents): make skill metadata agent-agnostic 2026-07-29 22:39:02 -04:00
40b37281d5 fix(shell): keep generated ignores visible to Pi 2026-07-29 17:01:22 -04:00
02f2017b40 feat(shell): deliver project skills through .agents 2026-07-29 11:54:36 -04:00
c6beafc53e fix(consume): quote Pi skill description 2026-07-29 09:13:42 -04:00
a4b88f8f42 feat(home-manager): place skills for Pi 2026-07-29 08:07:09 -04:00
ea8e2c50bc feat: add grill skill merging domain-modeling
Combine the interview-driven grill and the domain-modeling discipline into a
single model-invokable grill skill, so other skills can reach it as a backbone.

- One question at a time, each with options plus a recommendation and rationale,
  and a context quote when it turns on existing material.
- Agent-agnostic mode selection: discover .agents/CONTEXT.md then
  .claude/CONTEXT.md, and drive Domain Modeling, Free, or Clarification.
- Domain Modeling Mode maintains the glossary inline (CONTEXT-FORMAT.md) and
  offers ADRs under the three-gate test.
- Never begin implementation without an explicit user instruction, even when a
  caller skill invoked grill.
2026-07-25 16:10:40 -04:00
1e9d19fe3d chore: clear consumed spec and task scaffolding
Both spec docs and tasks 0001-0008 were mined into the personal wiki by
/consume; delete the scaffolding it consumed, leaving the now-empty spec/
and tasks/ directories in place.
2026-07-25 11:19:47 -04:00
52fc449824 feat: route consume knowledge to two channels (#9-#14)
Reworks the consume skill around two acceptance axes and two destinations,
implementing the six issues filed from the first real consume run as one design.

- Add trigger-ability as a second acceptance axis alongside generalizes, with
  its two premises; a pull-channel note now requires both (#9).
- Give consume a push channel — the target's own CLAUDE.md — for knowledge that
  fails an axis but is still useful: specific residue or proactive rules (#10).
- Name the general-lesson/specific-residue split and route each half (#12).
- Make the step-5 plan report a complete ledger with a per-note trigger line,
  push write-set, ref-fixes, flagged-not-authored, and drops (#13).
- Extend project cleanup to reconcile CLAUDE.md (capture-gated removal/trim) and
  fix references to deleted files wherever they land (#11).
- Encode a branch-owned output location: project notes flat under
  01 sources/claude/projects/, honoring any pre-existing grouping (#14).

Records the resolved vocabulary in .claude/CONTEXT.md and the two-channel write
decision in .claude/adr/0001.

Closes #9
Closes #10
Closes #11
Closes #12
Closes #13
Closes #14
2026-07-24 22:22:28 -04:00
8c17f3aeb8 feat: add consume skill for mining knowledge into the vault
/consume is the write-side counterpart to the read-only /wiki: it mines a
target for generalized, transferable knowledge and files atomic concept
notes into 01 sources/claude/, the agent-owned area of the Obsidian vault.

It is user-invoked/deliberate. SKILL.md holds the branch-detection ladder
plus the common writing logic — read existing knowledge (via /wiki and a
direct scan), plan atomic generalized notes with a file-to-section-to-note
contribution map, present an HTML plan and wait for explicit approval, then
write on the vault's note template and clear the consumed scaffolding.

v1 ships one branch, project.md: source code is truth for current state,
while .claude/spec and .claude/tasks are the record of why (mined, then
deleted). The ladder is extensible for future source types.
2026-07-24 20:03:44 -04:00
0a8b5c2096 feat: add wiki skill for read-only vault lookup
A model-invoked skill that answers questions from the user's Obsidian
vault, read-only, over the filesystem. Enforces the vault's own
settings.json deny-Read globs itself, since the harness does not apply
them when running from another directory. Uses the 02 tags/ hubs as a
recall backstop and reads to saturation, then answers with per-note
provenance.
2026-07-24 20:03:44 -04:00
97c0eeb055 feat: add all-skills leaderboard (task 0007)
Add a second pure transform to the benchmark core: a `--leaderboard` mode
that ranks per-skill results models into an index leaderboard, one row per
skill carrying both verdicts, links out to each per-skill report, and sorts
any red first (regressions above efficacy failures), then fragile-but-passing,
then clean green.

The fragile tier reuses the 0006 per-badge chips: the efficacy chip now
carries its win count so the leaderboard reads it against the pass floor
without recomputing margins, and fragility is scoped to the passing tier so a
red row never carries a chip. The runner's SKILL.md gains the no-argument
batch flow and the index invocation. The fixture test covers the tiered sort,
the not-applicable Regression cell, and the per-skill links.
2026-07-24 16:58:02 -04:00
57 changed files with 2328 additions and 5071 deletions

1
.claude/skills Symbolic link
View File

@@ -0,0 +1 @@
../.agents/skills

View File

@@ -1,115 +0,0 @@
## Problem Statement
I manage a collection of Claude Code agent skills and need a way to select which ones apply globally (on every project) versus per-project.
Today that per-project selection is driven by three agent-run skills — `setup-skills`, `update-skills`, `remove-skills` — backed by a `.claude/skills-lock.yaml` lockfile and a content-hashing script.
Setup copies a skill out of a shared library into a project's tree; update re-hashes to detect drift and applies safe upstream changes; remove deletes it.
This is a lot of hand-maintained machinery: an LLM chore to run, a lockfile to keep clean, and a hash-based drift classifier to reason about.
I want skills packaged and versioned through Nix instead, so that selection is declarative, updates are a pinned-input bump, and the whole copy-and-hash apparatus disappears.
## Solution
A standalone Nix flake (`alexion/skills`) that packages each skill as an individually addressable derivation and exposes optional integration outputs for placing selected skills where Claude Code discovers them.
- Adding a skill is creating a directory containing a `SKILL.md`; the flake auto-discovers it.
- A home-manager config selects skills it wants **globally** (into `~/.claude/skills/`) via a home-manager module.
- A project's own flake selects skills it wants **for that project** (into `<project>/.claude/skills/`) via a dev-shell helper, pinning this flake as an input.
- "Update" becomes `nix flake update`; "remove" becomes deleting a line from the selecting config; "setup" becomes adding one.
- The three management skills, the lockfile, and the hashing script are retired — Nix subsumes all of their responsibilities.
Nix is accepted as a hard requirement: every machine and every project is Nix/dev-shell based, so there is no need to keep skills as portable, committed-into-the-tree files for non-Nix consumers.
## User Stories
1. As a skill author, I want to add a new skill by creating a directory with a `SKILL.md` and nothing else, so that adding a skill requires no edit to the flake.
2. As a skill author, I want to organize skills into arbitrarily nested subfolders (e.g. group related workflow skills together), so that the source tree stays browsable as the collection grows.
3. As a skill author, I want that grouping to be purely cosmetic, so that where a skill's folder sits never changes how it is selected, placed, or named.
4. As an operator, I want to select a set of skills to be active in every project via my home-manager configuration, so that my always-on skills are declared in one place and installed reproducibly.
5. As a project maintainer, I want to select a set of skills for a single project via that project's flake, so that only the skills relevant to that project are active there.
6. As a project maintainer, I want the selected skills to appear under the project's `.claude/skills/` automatically when I enter its dev shell, so that I never run a manual install step.
7. As a project maintainer, I want the committed record of which skills a project uses to be the project's `flake.nix` selection plus its pinned input, so that the selection diffs cleanly in git and reproduces exactly on any machine.
8. As a project maintainer, I want the generated skill symlinks kept out of git, so that machine-specific store paths never get committed.
9. As a project maintainer, I want to keep my own hand-authored, project-private skills in `.claude/skills/` alongside the Nix-delivered ones, so that a project can have bespoke skills without adding them to the shared repo.
10. As a project maintainer, I want the dev shell to only ever touch the skills it manages, so that my hand-authored skills are never removed or clobbered.
11. As an operator, I want to update all pinned skills with `nix flake update`, so that picking up upstream skill changes is a single, reviewable, atomic operation.
12. As an operator, I want a skill I previously selected to disappear when I remove it from my selection and re-enter the shell (or rebuild), so that removal needs no cleanup command.
13. As an operator, I want to promote a skill from per-project to global (or vice versa) by moving one line between a project flake and my home-manager config, so that changing a skill's reach never moves files.
14. As a skill author, I want two skills that would resolve to the same name to fail the build with a clear error, so that a silent clobber can never happen.
15. As a skill author, I want the packaging to know nothing about `.claude/skills` or Claude Code, so that when a skill later needs to feed a different agent/tool the content is not welded to one placement convention.
16. As an operator, I want my global skill placement to compose with other modules that write into the same skills directory (e.g. a tool that self-installs its own skill), so that independent sources coexist without collisions.
17. As a maintainer, I want `nix flake check` to fail if a skill is malformed, if the home-manager placement regresses, or if the dev-shell helper stops producing a valid hook, so that breakage is caught before it ships.
## Implementation Decisions
### Overall structure
- The flake has two clearly separated tiers: a **content tier** (tool-agnostic skill packaging) and an **integration tier** (optional, Claude-Code-specific placement). A skill derivation in the content tier must never reference `.claude/skills` or anything Claude-Code-specific.
- Flake scaffolding uses `flake-utils.lib.eachSystem` over `x86_64-linux`, `aarch64-linux`, `aarch64-darwin`, with `nixpkgs` tracking unstable.
An explicit system list is passed rather than `eachDefaultSystem`, because that default set includes `x86_64-darwin`, which recent nixpkgs dropped and whose `legacyPackages` now throws.
### Content tier
- Skills are **auto-discovered** by recursively walking the `skills/` tree; any directory containing a `SKILL.md` is a skill, at any depth. Once a `SKILL.md` is found, that directory is a skill and its own subfolders are its assets, not further skills.
- Directories that do not contain a `SKILL.md` are organizational containers only; the walk descends through them. This nesting is **cosmetic**: it never reaches the placement location and is not a selectable unit (no "select a whole group" — that is a purely additive future change if ever wanted).
- Each skill becomes an **individually addressable derivation**, exposed at `packages.<system>.<skill-name>`. There is no `packages.<system>.default` — a catalog has no single default skill.
- A skill is addressed by its **name** (leaf directory / frontmatter `name`), independent of its source path. Placement flattens the source path away — a skill nested at `skills/workflow/to-spec/` is placed as a direct child `.../skills/to-spec/`, because Claude Code only discovers direct children of the skills root.
- **Skill-derivation contract** (the interchange primitive, shared with self-packaging tool repos like `gitea-axi`):
- `$out` contains `SKILL.md` at its root, plus any assets.
- The derivation carries its **name as an eval-time attribute** (e.g. `pname` / a passthru), so placement can form `.../skills/<name>` without reading `$out` (no import-from-derivation).
- **Names must be globally unique** across the whole tree. A collision detected during the discovery walk is a **hard build error** with a clear message, not a warning or a lint.
- A `lib.mkSkill` builder is exposed (turns a skill source directory into a conforming derivation). Exposing it costs nothing and lets a tool repo reuse it, even though tool repos are expected to self-package today.
### Cross-skill references
- There is **no dependency/closure machinery.** Skills interact by **name-based invocation** (the Skill tool), not by reading each other's files. A skill that references another skill relies on that skill being loaded (globally or in the same project), not on any relative path resolving on disk.
- Design convention, enforced by convention only (no lint): **skills are self-contained** — a skill never reaches into another skill's files; anything it needs at a path, it carries itself. The retired management skills (which read siblings' `LOCKFILE.md` and executed a sibling's script) are exactly the anti-pattern this convention forbids.
### Integration tier — selection
- Selection is **by derivation**, not by name string. Consumers pass skill derivations pulled from `packages.<system>` (e.g. bind `packages.<system>` to a short local name and list the skills off it).
- Rationale: skills are derivations in this ecosystem, so a derivation-based API composes with any skill satisfying the contract, from any repo, and keeps this flake ignorant of skills it does not own (respecting the boundary that tool-specific skills live in and are packaged by their own repos, not registered here).
### Integration tier — global placement (home-manager)
- Exposed as `homeModules.default`.
- The operator-facing option is **`programs.agents.skills`**, a `listOf package` with default `[]`. `agents` is deliberately an umbrella namespace (room for future `programs.agents.<other>`), and it has **no shared `programs.agents.enable`** — each sub-feature self-gates, so the namespace stays a clean, mergeable surface a different repo could also extend without an ownership conflict. An empty list is a no-op.
- For each selected skill the module writes an individual home file at `${claude-code.configDir}/skills/<name>` with `source = <skill derivation>` and **`recursive = true`**, gated on `programs.claude-code.enable`, reading `configDir` from the claude-code module.
- `recursive = true` is a **hard requirement, not a style choice**: it forces home-manager to materialize `.../skills/<name>/` as a real directory of per-file symlinks rather than claiming the directory as one opaque symlink. That is what allows this module, the operator's own skills declarations, and self-placing tool modules to coexist under one `skills/` tree.
- The module deliberately does **not** feed `programs.claude-code.skills`. That option is single-valued and would collide with an operator already setting it; per-skill `home.file` composes where the option does not.
### Integration tier — per-project placement (dev shell)
- Exposed as `lib.mkSkillsShellHook`, which takes a list of selected skill derivations and returns a shellHook string a project drops into its dev shell.
- On shell entry the hook symlinks each selected skill as a **direct child** of `<project>/.claude/skills/`, pointing into the Nix store.
- **Stateless lifecycle:** each entry first removes only the symlinks under `.claude/skills/` that point into the store (unambiguously "ours"), then recreates the current selection. This yields free stale-removal (deselect a skill → its symlink is gone next entry) and never touches real directories. No manifest or state file.
- **Coexistence with hand-authored skills:** real (non-symlink) skill directories under `.claude/skills/` are left untouched. The hook maintains a generated, self-ignoring `.claude/skills/.gitignore` that lists the names it manages (and ignores itself), so Nix-delivered symlinks stay out of git while hand-authored skills remain tracked.
- The committed record of a project's selection is its **`flake.nix`** (the selection list) plus its pinned `flake.lock`; the generated symlinks are gitignored.
- The hook is a plain shell string relying on ambient POSIX tools; it does not need a home-manager context (projects are not home-manager-managed).
### Retired
- `setup-skills`, `update-skills`, `remove-skills`, the `skills-lock.yaml` lockfile concept, and the directory-hashing script are all replaced by Nix and are **not** carried into this repo. Their responsibilities map to: selection in a flake (setup / remove) and `nix flake update` (update).
## Testing Decisions
- A good test here exercises the flake's **public outputs** as a consumer would observe them — a built skill, an evaluated-and-built module, an instantiated helper — not the internal shape of the discovery walk or builder. The outputs are the surface; there is no separate production code path needing its own seam.
- The single seam is **`nix flake check`**, with three focused checks beneath it (prior art: the `gitea-axi` flake's package-build and `checks/home-manager-module.nix` checks):
1. **Skill-build check** — builds every auto-discovered skill derivation. This one check transitively exercises the recursive discovery walk, the `mkSkill` builder, the `SKILL.md`-at-`$out`-root contract, and the name-uniqueness hard error (which fires at eval and so also surfaces here).
2. **Home-manager-module composition check** — instantiates `homeModules.default` under a sample home-manager configuration that selects a couple of skills with `programs.claude-code` enabled, and builds the resulting home-files derivation. This is the load-bearing check: it proves per-skill `home.file`, `recursive = true`, the `claude-code.enable` gate, and `configDir` sourcing compose as intended. Direct analogue of gitea-axi's home-manager-module check.
3. **shellHook assertion** — instantiates `lib.mkSkillsShellHook` with sample skills and asserts the produced hook is non-empty and references the expected store paths / skill names.
- Prefer the highest seam: none of these introduce a bespoke test hook into production logic; they evaluate/realize the flake outputs directly.
## Out of Scope
- **Dotfiles and any migration of existing skills.** This repo owns only the packaging and integration outputs. Moving current skills out of the dotfiles tree, rewiring the dotfiles home-manager configuration to consume this flake, deleting the retired management skills from wherever they currently live, and reviewing/rewriting individual skills are all handled separately and by hand. The repo starts with flake machinery and no skill content; skills are added incrementally, "as needed."
- **Integration with tool-specific skills.** Skills that ship inside a tool's own repo (e.g. `gitea-axi`) self-package and self-place; they are not listed in or delivered by this flake. This flake stays ignorant of them. The shared skill-derivation contract is the only thing in common.
- **Selectable groups.** Nested folders are cosmetic; selecting a whole group as a unit is not built (additive later if wanted).
- **A formal cross-skill dependency system.** Not built; name-based invocation plus the self-containment convention is the whole mechanism.
- **Non-Nix portability.** Skills are not required to work on a machine without Nix; there is no committed-into-the-tree copy for non-Nix consumers.
- **Multiple harnesses.** Placement targets Claude Code only. The `programs.agents` namespace is chosen to leave room for other harnesses later, but no second harness is implemented now.
## Further Notes
- The global-vs-per-project distinction is no longer a directory (`library/` is gone) or a file-copy state; it is purely **which config selects a skill**. The same per-skill derivation is selected by home-manager for the global set and by a project's dev shell for that project's set.
- The two integration outputs place skills through different mechanisms because they run in different contexts: home-manager (`home.file`, `recursive = true`) for the global set, and a dev-shell shellHook (store symlinks) for the per-project set. Both operate on the same by-derivation selection and the same skill-derivation contract.
- `programs.agents` is a generic namespace; if this repo is ever made public it is a mild land-grab worth revisiting, but it is appropriate for a personal ecosystem.
- The design intentionally mirrors `gitea-axi` (flake scaffolding, per-skill `home.file` with `recursive = true`, module shape, check style) so the two repos stay consistent and the placement lessons already encoded in gitea-axi's ADRs carry over.

View File

@@ -1,288 +0,0 @@
## Problem Statement
I write agent skills, but I have no way to know whether a skill is actually pulling its weight.
A skill can read well, be faithfully followed, and still make the agent's output no better than it would have been with no skill at all.
That skill is worthless, yet nothing in my current setup would catch it.
I want a repeatable way to prove that a given skill genuinely improves the agent's work before I trust it.
I also edit skills over time, and an edit can quietly make a skill worse.
A skill that still beats no-skill but has degraded from its released version is a regression I currently cannot see, because measuring only against no-skill sets too low a bar to catch it.
I want the same harness to tell me, the moment I edit a released skill, whether my change held quality steady or regressed against the version that is already trusted.
## Solution
A skill-benchmarking convention plus an agent-run harness that measures a skill's **efficacy against a no-skill baseline** and, whenever a released version exists, its **regression against that previous version**.
For each skill I author a set of test cases under a top-level `tests/` tree.
Running the harness on a skill executes every case as a controlled experiment: the same realistic prompt is given to several arms, several times, and a blind judge decides which arm's output better satisfies the case's author-written expectations.
Every run has a **new-skill arm** (my working tree) and a **no-skill baseline arm**.
Whenever the skill already exists on the main branch, the run adds a third **previous-version arm** materialized from main's `HEAD`.
Two blind head-to-head comparisons fall out of this: **efficacy** (new-vs-no-skill) tells me whether the skill earns its keep, and **regression** (new-vs-old) tells me whether my edit made it worse.
A skill "passes" efficacy only when the new arm reliably beats the no-skill baseline — a skill the agent follows faithfully but that does not beat the baseline is reported as adding nothing.
A skill "passes" regression when the new arm reliably does *not* lose to the previous version — a clean edit that holds quality steady passes, and only a real degradation fails.
The two are reported as **separate verdicts**, because "this skill never worked" and "I just broke a skill that did work" are different problems.
The harness runs in-session as an "AI script": it orchestrates the arms and the judge as subagents, and it hands the boring mechanical work — counting results, summing token usage, applying the pass rules, rendering the report — to a small committed program so the numbers are exact and reproducible.
It produces a self-contained HTML report with the two verdicts, a per-arm cost table, trend lines across recent runs, and drill-down evidence for anything that failed or regressed.
The whole thing is a distributed skill named `benchmark-skill`, invoked as `/benchmark-skill <name>`, so any repo that follows the `tests/` convention can benchmark its own skills.
## User Stories
1. As a skill author, I want to prove a skill improves the agent's output versus doing the task with no skill, so that I do not keep a skill that reads well but adds nothing.
2. As a skill author, I want each test case to pin down explicit expectations I write myself, so that the judge measures against my intent rather than inventing its own bar.
3. As a skill author, I want the same realistic prompt given to every arm, so that the only difference measured is which version of the skill (or no skill) produced the output.
4. As a skill author, I want the with-skill arms to actually load and follow the skill, so that I am measuring the skill's effect and not whether it happened to trigger.
5. As a skill author, I want each case to carry a hermetic, committed fixture, so that a run is reproducible and needs no live external state.
6. As a skill author who writes conversation-driven skills, I want a case to seed a prior conversation, so that I can benchmark skills whose input is a discussion rather than a file tree.
7. As a skill author, I want a mechanical, deterministic gate (hard assertions) separate from the judged comparison, so that a malformed output fails immediately without spending judgment on it.
8. As a skill author, I want a run repeated several times at a realistic temperature, so that the result reflects whether the skill reliably helps rather than helping once by luck.
9. As a developer iterating on a skill, I want trend lines across recent runs, so that I can tell a real change from run-to-run noise after I edit the skill.
10. As a developer, I want a per-arm table of turns, tokens, and imputed cost, so that I can see what the skill's quality gain costs and whether my edit made it more expensive.
11. As a developer, I want the report to open on the verdicts and expand only what failed or regressed, so that a clean run is calm and a problem run puts the evidence in front of me.
12. As a developer, I want a green run to still flag its most fragile case on each axis, so that a skill that barely passed cannot masquerade as robust.
13. As a developer, I want to benchmark a single skill quickly, so that my edit-and-recheck loop is cheap.
14. As a developer, I want to benchmark all skills at once and see a leaderboard, so that I know at a glance which of my skills are green and which regressed.
15. As an operator, I want the whole battery to run on my normal interactive subscription, so that a benchmark does not draw down a separate metered automation credit pool.
16. As a maintainer, I want the mechanical scoring and rendering to be committed, tested code, so that the report's numbers are exact and reproducible rather than re-derived by the agent each run.
17. As a maintainer, I want the test tree kept out of the packaged skills, so that installing a skill never drags its fixtures along.
18. As a maintainer, I want reports and history kept out of git, so that machine-specific run artifacts never get committed.
19. As a developer updating a skill, I want the new version compared against the released version on main and not only against no-skill, so that a skill that silently got worse but still beats no-skill is caught rather than passing green.
20. As a developer, I want efficacy and regression reported as two independent verdicts, so that I can tell "this skill never worked" apart from "I just broke a skill that did work."
21. As a developer, I want a clean refactor that holds output quality steady to pass the regression check, so that a deliberate no-op change is not failed merely for tying the previous version.
22. As a developer, I want the regression comparison to appear automatically the moment I edit a released skill, so that I never have to remember to opt into it.
## Implementation Decisions
### What is measured
- The benchmark measures **efficacy against a no-skill baseline** and, when a previous version exists, **regression against that previous version**.
Trigger-correctness (does a model-invokable skill fire on a realistic prompt) and format-conformance are explicitly not measured here.
They are separable evals that can be added later without disturbing this one.
- Both dimensions are judged on **output quality**, with cost reported alongside but **never gating**.
A skill that improves quality is worth keeping even when it costs more tokens, and the cost columns are there to inform, not to fail.
### The arms
- Every case is a controlled experiment with a **new-skill arm** (my working tree) and a **no-skill baseline arm**, and — whenever the skill already exists on the main branch — a third **previous-version arm** materialized from main's `HEAD`.
- Two blind head-to-head comparisons are drawn from the arms per trial: **efficacy** pairs the new arm against the no-skill arm, and **regression** pairs the new arm against the previous-version arm.
- Every arm is given the **identical realistic prompt**, authored once and arm-agnostically, so the arms differ only in which skill (or no skill) is present.
- The new arm and the previous-version arm are both **force-invoked**: each subagent is pointed at its own isolated skill materialization and told to use it, reading that skill's own file and any assets it references, so the real skill machinery is exercised rather than a reconstruction.
The previous-version arm is force-invoked identically to the new arm, pointed at the materialization of the old skill instead of the new arm's copy of the working-tree directory, so the skill version is the only difference between them.
- The baseline arm receives the bare prompt with the skill **absent from its context**, which is the honest counterfactual of the skill not existing.
- Force-invoking uniformly is a deliberate consequence of running in-session.
An earlier design varied arm construction by whether a skill was model-invokable, but in-session subagents cannot faithfully reproduce skill auto-discovery, so the harness force-invokes every with-skill arm and leaves triggering to a future, separate eval.
### The regression comparison
- The previous version is **always main's `HEAD`** — the repo's default branch, resolved rather than hardcoded to the literal name "main" — and the new version is **the working tree**.
Anchoring to the released state on main, rather than to whatever was last committed on the working branch, means the regression question is always "did my in-progress edit degrade from the trusted version," which is the moment of real risk.
- The third arm runs **only when the skill directory differs from main's `HEAD`**.
A brand-new skill that does not yet exist on main, or a skill whose directory is unchanged from main, has no meaningful previous version, so the run degrades to the two-arm efficacy-only shape.
- **Cases are held fixed to the working tree.**
All arms run against today's prompt, today's fixture, today's expectations, so the skill version is the only variable differing between the new and previous-version arms.
Letting a case drift with the version would change two things at once and rob the new-vs-old verdict of meaning.
The conservative consequence is accepted: if a case's expectations were rewritten alongside the skill, the old version is judged against a bar it was never written for, which surfaces a possible regression for a human to eyeball rather than silently excusing it.
- The **hard-assertion gate applies to the new arm only**, exactly as before.
The gate exists to catch the current skill emitting malformed output.
Running it against the previous version would manufacture spurious failures for any assertion introduced in the very edit under test, and the no-skill arm was already never gated.
- The **judge and the soft criteria are reused unchanged** for the regression comparison.
A case's author-written soft criteria describe what a good answer looks like, which does not depend on the opponent, so the same blind judge grounds both head-to-heads and simply receives a different pair of outputs.
### Expectations per case
- A case's expectations come in two tiers.
- **Hard assertions** are executable shell predicates that form a deterministic, LLM-free gate.
They run against two provided values: `$OUTPUT`, the path to the arm's captured final message, and `$WORLD`, the path to that arm's fresh fixture copy.
A non-zero exit fails the assertion.
A case may have zero hard assertions.
- **Soft criteria** are natural-language statements the judge grounds the head-to-heads on.
A case has at least one.
- This snippet, from the case-format decision, encodes the authoring contract more precisely than prose:
```
---
description: <one-line scenario>
---
## Prompt
<the realistic user request, given identically to every arm>
## Seed (optional; conversation-based skills only)
**User:** ...
**Assistant:** ...
## Hard assertions (executable; $OUTPUT = arm's final message, $WORLD = its fixture copy)
```sh
test -f "$WORLD/review.md"
grep -qE '<pattern>' "$OUTPUT"
```
## Soft criteria
- <a statement the judge grounds the comparisons on>
```
### The judge
- The judge is an **LLM subagent**, one **blind judge per trial per comparison**, shown both arms' outputs as unlabelled A and B with the **order randomized** per trial.
- The judge is grounded on the case's soft criteria rather than free-forming its own standard, and its prompt explicitly instructs it to discount mere length and formatting differences, since a skill can otherwise "win" by being more verbose.
- The same judge machinery serves efficacy and regression.
Only the pair of outputs it is handed differs.
- Variance is handled by repeating trials rather than by a per-trial panel.
A panel is escalated to only for a comparison whose trials come back consistently split.
### Trials, temperature, and the pass rules
- Each case runs **5 paired trials** at a **realistic temperature**, configurable per skill, with **no reuse of arms or judgments across arms or runs**.
Trial *i*'s new-skill output is judged against trial *i*'s no-skill output for efficacy and against trial *i*'s previous-version output for regression.
A realistic temperature is chosen because most skills guide open-ended reasoning, and a big part of a skill's value is making a good outcome reliable across that variance, which temperature zero would hide.
A per-skill temperature-zero override remains available for a skill that genuinely wraps a mechanical task.
- The **efficacy pass rule**:
- Per trial, the graded head-to-head collapses to **WIN, TIE, or LOSS** for the new arm, and a tie counts as a non-win.
- Per case, efficacy **passes** when **wins ≥ 3 and losses ≤ 1** across the 5 trials, and **any** loss is flagged in the report for human review.
A failed hard assertion fails the case outright regardless of the head-to-head.
- The **regression pass rule** inverts, because holding quality steady is the goal:
- Per trial, the new-vs-old head-to-head collapses to **WIN, TIE, or LOSS** for the new arm, where a TIE means "as good as the previous version" (a success), a LOSS is the regression being hunted, and a WIN is a bonus improvement.
- Per case, regression **passes** when **losses ≤ 1** across the 5 trials, with **no wins floor** — ties and wins both count as non-regressions — and **any** loss is flagged for human review.
- The wins floor is dropped deliberately, so a deliberate no-op edit that ties the previous version is a pass rather than a failure.
- Tolerating one loss on each axis, rather than demanding zero, keeps a single spurious judge miscall at temperature above zero from reddening a genuinely good result, while still surfacing every regression and failing on a second loss.
### The two verdicts
- A skill carries **two independent verdicts**: an **Efficacy** verdict, green when every case passes efficacy, and a **Regression** verdict, green when no case regressed.
- The two are reported side by side rather than collapsed into one, because their four crossings carry genuinely different meanings:
- efficacy-green + regression-green → the edit is safe and the skill earns its keep.
- efficacy-green + **regression-red** → the skill still beats no-skill but got worse than the released version — the exact regression this feature exists to catch.
- **efficacy-red** + regression-green → the skill does not beat no-skill, but the edit did not make it worse (it was already dead weight).
- efficacy-red + regression-red → the skill does not beat baseline and the edit made it worse.
- The report headline reads the two verdicts together into one "so what" (for example, "Still valuable, but this edit regressed two cases").
- On a two-arm run with no previous version, only the Efficacy verdict is meaningful and the Regression verdict reads not-applicable.
### Fixtures and isolation
- Every case ships its own **hermetic, committed fixture**, and each arm-and-trial combination runs against a **fresh copy** of it, so writes from one run never leak into another.
- File-and-tree skills get a fixture directory.
Conversation-driven skills get a **seed transcript** in role-tagged form, injected as the subagent's prior context before the prompt.
- Each arm-and-trial runs in a **fixture-only world**: the arm subagent's working directory is its fresh fixture copy rather than the repo root, and nothing under `skills/` or the grading `tests/` tree sits on any path it explores from there.
This is what makes the no-skill baseline an honest counterfactual, because it cannot discover and read the skill's assets off disk, and it also stops any arm from reading its own case's soft criteria or hard assertions and tuning its answer to the bar it will be judged against.
- A with-skill arm is handed its skill as an **isolated temp materialization** placed outside the fixture, and is pointed there to force-invoke it.
The new-skill arm's materialization is a copy of the working-tree skill directory, so it reflects uncommitted edits.
The previous-version arm's is the same skill directory checked out at main's `HEAD`, since each skill is packaged as its own self-contained derivation and does not need the rest of the repo.
The no-skill arm is handed nothing.
Each temp materialization is cleaned up after the run, like the fresh fixture copies.
- Isolation here is deliberately **soft**: an in-session subagent shares the machine and could in principle reach the repo by absolute path.
Relocating each arm's working directory to its fixture, and instructing the baseline to stay within it, moves contamination from near-certain to requiring an arm to deliberately wander outside its world.
A hard filesystem guarantee would require an OS sandbox, which is out of scope.
- A live-target fixture that seeds and tears down an external resource per run is out of scope for now.
### Location and the test tree
- Tests live in a top-level **`tests/` tree that mirrors `skills/` by full path**, so a skill's cosmetic nesting is mirrored and its tests sit at the same relative path.
- Placing tests outside `skills/` is deliberate: each skill is packaged as its own derivation and placed into a project's skills directory, so bundling fixtures inside a skill would bloat every installation.
The test tree is neither content nor integration output — it is repo infrastructure that is never packaged or placed.
- A skill's tests sit as case directories at the mirror point, each holding a `case.md` and an optional fixture.
### The harness
- The harness runs **in-session as an AI script**, orchestrating the arms and judges as subagents via the workflow mechanism.
- Running in-session is a billing decision: it stays on my **interactive subscription quota**, avoiding the separate metered automation credit pool that headless `claude -p` and the Agent SDK draw from since mid-2026.
- The full per-arm metric set is recovered by **reading each arm-subagent's transcript file**, which records per-message token usage.
Summing that usage yields raw tokens across the four components, the pricing-weighted cost-equivalent-token figure that the gitea-axi bench also reports, and an imputed dollar cost, while the turn count comes from the transcript's tool-call rounds.
- The **mechanical work is done by a small committed program, not by agent reasoning** (see the deterministic core below).
### The deterministic core
- A committed program owns every mechanical, non-judgment step so the results are exact and reproducible: summing token usage across every arm and computing the cost figures, collapsing each trial of each comparison to WIN/TIE/LOSS, applying the two pass rules and the two verdicts, parsing `case.md`, rendering the HTML report, and appending to and trimming the run history.
- It is **parameterized over the number of arms and the two comparisons**, so it handles both the three-arm and the degenerate two-arm run through one path.
- It is a pure transform: given the collected run data (per-arm transcript usage, hard-assertion results, and the per-trial judge verdicts for both comparisons) it returns the results model, the rendered HTML, and the updated history.
- It ships with the `benchmark-skill` skill and is invoked by that skill's prose, which keeps the agent confined to the judgment work — running arms and judging — that it is actually good at.
### The runner skill
- The runner is a **distributed skill** named `benchmark-skill`, auto-discovered and packaged like any other skill in this repo, and marked non-model-invocable so it is only ever run deliberately.
- It is invoked as `/benchmark-skill <name>` to benchmark one skill, and with no argument to benchmark every skill.
- Choosing the two-arm or three-arm shape is **fully automatic and carries no new invocation surface**: the harness diffs the skill directory against main's `HEAD` and adds the previous-version arm exactly when a released version exists to compare against.
- Because it is distributed and relies only on the `tests/` convention, any repo following that convention can benchmark its own skills.
- Validating the shape of the `tests/` tree (that a `case.md` parses, that a fixture referenced exists, that a test directory maps to a real skill, and reporting skills that have no tests) is the runner's own responsibility at run time, not a separate build-time check.
### The report
- The report uses a **layered shape**: a verdict scorecard on top with progressive disclosure beneath.
A failed or regressed case auto-expands with its losing-trial evidence, while passing cases stay collapsed but can be opened.
- Its sections run top to bottom.
First a header with the **two verdict badges** (Efficacy and Regression), run metadata, and a jump-to-failure link shown only when something failed or regressed.
Then the **trend ribbons**.
Then the **verdict headline** reading both badges together.
Then a **per-arm cost table**.
And last the **cases** in stable authored order.
- There are **two stacked trend ribbons** — an **Efficacy** ribbon and a **Regression** ribbon — each a net-margin (wins minus losses) sparkline over roughly the last seven runs, with verdict-colored per-run dots, the current run ringed, and a readout of the current net, the change versus the previous run, and how many recent runs were green.
The Regression ribbon's dot is **absent for any run that was two-arm**.
The Regression series is comparable across runs only while main is unchanged.
Because the previous-version arm is always main's `HEAD`, merging the branch moves main and resets the meaningful regression history, which is accepted rather than normalized.
The ribbons are what let an edit's real change be told apart from run-to-run noise.
- The **cost table** has **three rows — new-skill, previous-version, no-skill** — each with turns, raw tokens, cost-equivalent tokens, and imputed cost, with a footnote that absolute cost is inflated by shared-context cache overhead and that the trustworthy signals are two ratios: **new-vs-no-skill** (what the skill costs over nothing) and **new-vs-old** (what this edit added or saved).
The previous-version row drops out on a two-arm run.
- Each case panel shows **both comparisons side by side**, each with its per-trial WIN/TIE/LOSS strip and its pass result.
A case **auto-expands when either comparison fails or carries a flagged loss**, so a regression opens the panel even when efficacy is green, and the auto-expanded evidence shows the losing-trial output pair for whichever comparison failed.
- **Cases render in stable authored order and are never reshuffled by verdict**, so their positions are learnable and diff cleanly across runs, with failures reached via auto-expansion and the header's jump link rather than by sorting them to the top.
- A **clean run rests fully collapsed**, except that fragility is flagged **per badge**: a green Efficacy badge carries a "narrowest margin" chip on the case fewest trial-flips from failing efficacy, and a green Regression badge independently carries one on the case sitting at exactly one loss.
A case that is fragile on both axes shows both chips, so a barely-green skill cannot look as safe as a clean sweep on either axis.
- The report carries a **thin, constrained narrative layer** — a headline "so what" and per-case one-liners — under the rule that every sentence it writes must be backed by a value or judge quote visible on the same screen.
The per-trial judge rationale and the mechanical "why it failed" are inherent evidence, not part of that narrative layer.
### All-skills output and files
- Benchmarking all skills produces an **index leaderboard** where each skill row carries **both badges (Efficacy and Regression)**.
- The sort promotes **any red first**, with **regressions ordered above efficacy failures** — a regression means "you just broke something that was working," the more urgent signal while iterating — then fragile-but-passing skills, then clean green.
A skill with no previous version reads not-applicable in its Regression cell.
- The leaderboard links out to separate per-skill report files.
- All report artifacts live under a git-ignored `tests/.reports/` directory, flat and keyed by unique skill name, since skill names are globally unique in this repo.
- Per-skill HTML is **latest-only and overwritten each run**, because the longitudinal data lives in a per-skill history file to which the harness appends one summary line per run, capped at roughly the last fifty and trimmed oldest-first.
- Each history line records **both the efficacy and the regression net-margin and pass/fail**, plus a flag for whether the run was two-arm or three-arm, so both ribbons can plot their own sparkline and correctly show a gap for any two-arm run.
- That history is **local and ephemeral** — it lives inside the ignored reports directory and resets if that directory is cleaned — consistent with reports being transient artifacts.
Promoting it to durable, committed history is a later, additive change.
## Testing Decisions
- The judgment half of this feature — the arms and the blind judges — is inherently non-deterministic and is **not unit-tested**.
Its correctness is established by running the harness on a real skill and reading the report.
- The one deterministic, testable seam is the **deterministic core**.
A good test here exercises that core's external behavior as its caller observes it: fed **committed fixtures** — sample arm transcripts for all three arms, hard-assertion results, and the per-trial judge verdicts for both comparisons — it must produce the expected per-arm metrics, the expected per-case and per-skill verdicts on both axes, and a correct report including the two ribbons and the three-row cost table.
The **two-arm run is retained as the degenerate "no previous version" case** in the same test, so both shapes are covered.
It is tested at that single seam, not through the internals of the token-summing or the rendering.
- This isolates exactly the error-prone arithmetic, counting, pass-rules, and templating that must not be re-derived by the agent, and it needs no LLM to run.
- Prior art is `checks/shell-hook.nix`, which runs a produced artifact against a fixture project and asserts on the observed result.
The `home-manager-module` check, which builds an output and inspects it, is the same style.
- Whether this fixture test is also wired into `nix flake check` is left open and deferred, in keeping with the earlier decision not to add a build-time check for this feature.
The test can exist and run without being a flake gate.
## Out of Scope
- **Trigger-correctness** — whether a model-invokable skill fires on a realistic prompt.
It is a distinct dimension the in-session harness cannot cleanly measure, and it is deferred to a separate eval.
- **Format-conformance** as its own measure, which the efficacy comparison largely subsumes.
- **Live-target fixtures** that seed and tear down an external resource per run.
Only hermetic committed fixtures are supported for now.
- **Headless or API execution** of the battery, which would draw the metered automation credit pool.
A headless run of a single skill remains only a fallback for the unlikely case that transcript files lack usage data.
- **Cost as a gate.** Cost is always reported and never fails a skill on its own.
- **A pinned or blessed baseline, and comparison against arbitrary refs.** The previous version is always main's `HEAD`.
Comparing against a marked version-to-beat or two chosen historical refs is a later, additive refinement.
- **A `--no-regression` fast-path** for skipping the previous-version arm mid-edit.
The shape is fully automatic for now.
An opt-out flag can be added later.
- **Durable or normalized trend history.** History is local and ephemeral, and the regression series is not normalized across a main move, until a later opt-in.
- **Selectable, per-skill-kind arm construction.** The harness force-invokes every with-skill arm uniformly.
- **Hard filesystem isolation of the arms.** Each arm runs in a fixture-only world with its working directory relocated off the repo root, which is soft isolation that an arm could defeat only by deliberately reaching for an absolute repo path.
An OS sandbox that makes the skill genuinely unreachable is a later hardening.
## Further Notes
- The gitea-axi benchmark is the **methodology source**, not a template: its honest-baseline discipline, its per-arm comparison table, and its pricing-weighted cost-equivalent-token unit carry over, while its temperature-zero, headless, standalone-script shape does not, because a reasoning skill needs a realistic temperature and the billing reality pushes the run in-session.
- The shared-context cache overhead means a per-arm absolute cost reads higher than a standalone run would.
The inter-arm ratios are what stay trustworthy, and the report says so.
- A prototype of the report was built during design and is the visual reference for the layered shape, the trend ribbons, the fragility chips, and the clean-versus-failing resting states.
- "Fragile," for a green-run chip, means the passing case that is the fewest trial-flips from failing on that badge's axis — closest to dropping under three wins for efficacy, or over one loss for regression.
- Because the previous version is always main's `HEAD`, a change committed directly to main stops being regression-testable once the working tree matches `HEAD`.
Regression is designed for the branch-in-progress workflow (working tree versus released main).
The deferred pinned-baseline feature would cover the commit-to-main case.

View File

@@ -1,53 +0,0 @@
---
spec: nix-skill-packaging
---
## What to build
The foundational tracer bullet: a standalone Nix flake that packages each skill as an individually addressable derivation, plus the check that proves it.
The flake scaffolding uses `flake-utils.lib.eachSystem` over `x86_64-linux`, `aarch64-linux`, `aarch64-darwin`, with `nixpkgs` tracking unstable.
An explicit system list is passed rather than `eachDefaultSystem`, whose default set includes the now-dropped `x86_64-darwin`.
The content tier recursively walks the `skills/` tree: any directory containing a `SKILL.md` is a skill (at any depth), and once found, its own subfolders are its assets rather than further skills. Directories without a `SKILL.md` are cosmetic organizational containers the walk descends through — the nesting never reaches the placement name and is not a selectable unit.
Each discovered skill becomes an individually addressable derivation exposed at `packages.<system>.<skill-name>`, built by an exposed `lib.mkSkill` builder. There is no `packages.<system>.default`. A skill is addressed by its name (leaf directory / frontmatter `name`), independent of its source path. The derivation satisfies the skill-derivation contract: `$out` contains `SKILL.md` at its root plus any assets, and the derivation carries its name as an eval-time attribute (e.g. `pname` / a passthru) so placement can form `.../skills/<name>` without import-from-derivation. Nothing in this tier references `.claude/skills` or anything Claude-Code-specific.
Names must be globally unique across the whole tree; a collision detected during the discovery walk is a hard eval-time build error with a clear message, not a warning.
Verified by the skill-build check under `nix flake check`, which builds every auto-discovered skill derivation and thereby transitively exercises the recursive walk, the `mkSkill` builder, the `SKILL.md`-at-`$out`-root contract, and the name-uniqueness hard error. Because the repo ships with no real skill content, this check drives fixture skills (à la gitea-axi's `runCommandLocal` fixtures), including a collision fixture that must fail the build.
## Acceptance criteria
- [x] `flake.nix` iterates the three systems via `flake-utils.lib.eachSystem` (explicit list, not `eachDefaultSystem`) with `nixpkgs` unstable.
- [x] The `skills/` tree is auto-discovered recursively: any directory containing a `SKILL.md` is a skill at any depth; its subfolders become its assets, not further skills.
- [x] Directories without a `SKILL.md` are traversed as cosmetic containers only and never affect a skill's placement name.
- [x] Each discovered skill is exposed at `packages.<system>.<skill-name>`; there is no `packages.<system>.default`.
- [x] `lib.mkSkill` is exposed and turns a skill source directory into a conforming derivation.
- [x] A built skill's `$out` contains `SKILL.md` at its root, and the derivation carries its name as an eval-time attribute (no import-from-derivation needed to read it).
- [x] No content-tier derivation references `.claude/skills` or anything Claude-Code-specific.
- [x] Two skills that resolve to the same name fail the build at eval with a clear collision message.
- [x] `nix flake check` includes a skill-build check that builds every auto-discovered skill; a name-collision fixture makes it fail.
## Implementation Notes
Files: `flake.nix` (scaffolding + wiring), `lib/mk-skill.nix` (builder), `lib/discover.nix` (recursive walk + uniqueness), `checks/skill-build.nix` with fixtures under `checks/fixtures/`, and an empty `skills/.gitkeep` root.
- **Name source — leaf directory name, not frontmatter.**
The spec's "What to build" phrases the name as "leaf directory / frontmatter `name`", but the acceptance criteria only ever require the leaf directory name (criteria 2 and 6).
Discovery and `mkSkill` derive the name purely from the leaf directory (`mkSkill`'s `name` defaults to `builtins.baseNameOf src`).
The two are equal by Claude Code convention; parsing `SKILL.md` YAML frontmatter at eval would add real complexity for no behavioural gain when they agree, so the frontmatter branch is deliberately not implemented.
If divergence between directory name and frontmatter name ever needs enforcing, that is an additive validation for later.
- **Eval-time name attribute.**
Carried as `passthru.skillName` (in addition to the derivation's own `name`), so the integration tier can form `.../skills/<name>` at eval time without import-from-derivation.
- **Empty `skills/` root.**
The repo ships no real skill content, so `skills/` holds only a `.gitkeep` and `packages.<system>` is an empty set today.
The `skill-build` check therefore drives fixture skills (a top-level skill, one under a cosmetic container with an asset subfolder, and one nested several containers deep) plus a separate two-skills-one-name collision fixture that `builtins.tryEval` confirms fails discovery at eval.
- **Real skills also folded into `checks` (beyond the literal criteria).**
Each real discovered skill is added to `checks.<system>` alongside `skill-build`, so once skills land `nix flake check` builds each one and fails on a malformed skill (spec user story 17).
This is empty today and is a natural extension of criterion 9's "builds every auto-discovered skill", not new scope.
- **Integration tier (home-manager module, dev-shell shellHook) is intentionally absent** — it belongs to tasks 0002 and 0003. No `home-manager` flake input is added yet for that reason.

View File

@@ -1,48 +0,0 @@
---
spec: nix-skill-packaging
blocked-by: 0001-content-tier-skill-packaging
---
## What to build
The global (every-project) placement output: a home-manager module that installs an operator's selected skills into `~/.claude/skills/`, plus the check that proves its composition.
Exposed as `homeModules.default`. The operator-facing option is `programs.agents.skills`, a `listOf package` with default `[]` (an empty list is a no-op). `agents` is deliberately an umbrella namespace with no shared `programs.agents.enable` — each sub-feature self-gates so the namespace stays a clean, mergeable surface another repo could extend. Selection is by derivation: consumers pass skill derivations pulled from `packages.<system>`, not name strings.
For each selected skill the module writes an individual `home.file` at `${claude-code.configDir}/skills/<name>` with `source = <skill derivation>` and `recursive = true`, gated on `programs.claude-code.enable`, reading `configDir` from the claude-code module. `recursive = true` is a hard requirement: it materializes `.../skills/<name>/` as a real directory of per-file symlinks so this module, the operator's own skills declarations, and self-placing tool modules coexist under one `skills/` tree. The module deliberately does not feed the single-valued `programs.claude-code.skills` option, which would collide with an operator already setting it.
Verified by the home-manager-module composition check under `nix flake check`: it instantiates `homeModules.default` under a sample home-manager configuration selecting a couple of skills with `programs.claude-code` enabled, builds the resulting home-files derivation, and asserts the per-skill `home.file`, `recursive = true`, the `claude-code.enable` gate, and `configDir` sourcing all compose as intended (direct analogue of gitea-axi's home-manager-module check).
## Acceptance criteria
- [x] `homeModules.default` is exposed.
- [x] It defines `programs.agents.skills` as `listOf package` with default `[]`, and an empty list installs nothing.
- [x] There is no shared `programs.agents.enable`; the skills feature self-gates.
- [x] Selection is by derivation (skills pulled from `packages.<system>`), not by name string.
- [x] Each selected skill is written as an individual `home.file` at `${claude-code.configDir}/skills/<name>` with `recursive = true`, sourcing `configDir` from the claude-code module.
- [x] Placement is gated on `programs.claude-code.enable`; with it off, no skill files are written.
- [x] The module does not set `programs.claude-code.skills`.
- [x] The module composes with an operator's own skills declarations and self-placing tool modules under one `skills/` tree without collision.
- [x] `nix flake check` includes a home-manager-module composition check that builds the home-files derivation for a sample selection and asserts the above.
## Implementation Notes
Files: `home-manager-module.nix` (the module, at the repo root, mirroring gitea-axi's placement), `checks/home-manager-module.nix` (the composition check), and `flake.nix` wiring (a `homeModules` output plus the check registration and the new `home-manager` input).
- **Placement name comes from `skillName`.**
The module reads each derivation's `passthru.skillName` — the eval-time name attribute task 0001 established for exactly this purpose — to form `.../skills/<name>` without import-from-derivation.
- **`homeModules` exposes an `agents-skills` alias beside `default`.**
The criteria only require `homeModules.default`; the named alias is an additive convenience mirroring gitea-axi's `homeModules` shape, and `default` points at it.
- **`home-manager` flake input follows this flake's nixpkgs.**
It exists solely so `nix flake check` can evaluate the module against real home-manager; a consumer importing the module supplies their own home-manager and pkgs, so the input has no bearing on what they get.
- **Recursive placement is proven by the entry's type, not by file existence.**
A `test -f skills/<name>/SKILL.md` follows symlinks and cannot distinguish a recursive per-file tree from one opaque symlink over the whole skill, so the check asserts the entry is a real directory (`test -d` and `! -L`).
A mutation to `recursive = false` fails the check.
- **A non-default `configDir` scenario was added during review.**
The other scenarios all run at the default `configDir`, so a module hardcoding `.claude/skills/` would have passed them identically.
One configuration now sets a custom `configDir` and asserts placement follows it, closing the "configDir sourcing" leg of the check.
A mutation hardcoding `.claude` fails the check.

View File

@@ -1,55 +0,0 @@
---
spec: nix-skill-packaging
blocked-by: 0001-content-tier-skill-packaging
---
## What to build
The per-project placement output: a dev-shell helper that drops a project's selected skills into its own `.claude/skills/`, plus the check that proves it.
Exposed as `lib.mkSkillsShellHook`, which takes a list of selected skill derivations and returns a shellHook string a project drops into its dev shell. On shell entry the hook symlinks each selected skill as a direct child of `<project>/.claude/skills/`, pointing into the Nix store. Selection is by derivation (skills pulled from `packages.<system>`), and the hook is a plain shell string relying on ambient POSIX tools — it needs no home-manager context.
Lifecycle is stateless with no manifest or state file: each entry first removes only the symlinks under `.claude/skills/` that point into the store (unambiguously "ours"), then recreates the current selection. This gives free stale-removal (deselect a skill → its symlink is gone next entry) and never touches real directories. Real (non-symlink) skill directories under `.claude/skills/` — hand-authored, project-private skills — are left untouched. The hook maintains a generated, self-ignoring `.claude/skills/.gitignore` listing the names it manages (and ignoring itself), so Nix-delivered symlinks stay out of git while hand-authored skills remain tracked. The committed record of a project's selection is its `flake.nix` selection list plus its pinned `flake.lock`.
Verified by the shellHook assertion check under `nix flake check`: it instantiates `lib.mkSkillsShellHook` with sample skills and asserts the produced hook is non-empty and references the expected store paths / skill names.
## Acceptance criteria
- [x] `lib.mkSkillsShellHook` is exposed, takes a list of selected skill derivations, and returns a shellHook string.
- [x] The hook is a plain shell string using ambient POSIX tools and needs no home-manager context.
- [x] On entry, each selected skill is symlinked as a direct child of `<project>/.claude/skills/` pointing into the store.
- [x] Each entry first removes only store-pointing symlinks under `.claude/skills/`, then recreates the current selection — deselecting a skill removes its symlink next entry.
- [x] Real (non-symlink) skill directories under `.claude/skills/` are never touched.
- [x] The hook maintains a generated, self-ignoring `.claude/skills/.gitignore` listing the names it manages, keeping store symlinks out of git while hand-authored skills stay tracked.
- [x] No manifest or state file is used.
- [x] `nix flake check` includes a shellHook assertion that instantiates the helper with sample skills and asserts the hook is non-empty and references the expected store paths / skill names.
## Implementation Notes
Files: `lib/mk-skills-shell-hook.nix` (the helper), `checks/shell-hook.nix` (the check), and the `flake.nix` wiring (a `mkSkillsShellHook` binding in the top-level `lib` output, plus the check registration).
- **Placement name comes from `skillName`.**
The hook reads each derivation's `passthru.skillName` (from task 0001) to name its symlink, so the selection stays by-derivation and the name needs no `$out` read.
- **The whole hook runs in a subshell.**
It defines helper variables and a function, so wrapping it in `( … )` keeps those from leaking into the operator's interactive shell.
Verified by a real `bash` run: after sourcing, `_mkskills_managed` and the link function are absent from the parent shell.
- **Name collision with a hand-authored skill: the real directory wins.**
If a selected skill's name already exists as a real (non-symlink) directory, the hook skips it with a stderr warning and leaves the directory in place.
A skipped skill is deliberately kept out of the managed set, so it is never added to the `.gitignore` and the hand-authored directory stays tracked.
Only skills the hook actually links are listed in the `.gitignore`.
- **Cleanup is precise to store-pointing symlinks.**
The stale-removal sweep removes a direct-child symlink only when `readlink` shows it targets the Nix store, so a symlink a project points elsewhere is left alone.
The check's fixture includes such a non-store symlink and asserts it survives.
- **The check executes the hook, beyond the literal testing decision.**
The task's testing decision only requires asserting the hook string is non-empty and references the expected store paths and names.
The check does that, then additionally sources the hook against a fixture project across two successive selections and asserts placement, stateless stale-removal, the collision guard, the non-store-symlink survival, the `.gitignore` contents, and that no state file appears.
This is a strict superset that proves the spec-mandated lifecycle (criteria 37) rather than trusting a string match, and each added assertion maps to a stated requirement.
- **POSIX-tool notes.**
The hook relies on `readlink` to classify symlinks; it is not in POSIX but is present on every Nix-based machine this targets, and there is no clean substitute.
The link step avoids the non-POSIX `ln -n` overwrite flag by removing any residual symlink first, then using a plain `ln -s`.
A single quote in a skill name would break the generated shell string; skill names are kebab-case by convention, so hardening against that is left out as out of scope.

View File

@@ -1,117 +0,0 @@
---
spec: skill-benchmarking
blocked-by: 0001-content-tier-skill-packaging
---
## What to build
The foundational tracer bullet: the thinnest complete path that benchmarks one skill's **efficacy against a no-skill baseline** and produces a report.
Running `/benchmark-skill <name>` on a real skill executes its cases and returns an HTML report carrying a single **Efficacy verdict**.
This slice establishes three things end to end.
**The `tests/` convention.**
Tests live in a top-level `tests/` tree that mirrors `skills/` by full path, so a skill's cosmetic nesting is mirrored and its tests sit at the same relative path.
Tests are deliberately outside `skills/` so they are never packaged or placed when a skill is installed.
A skill's tests are case directories at the mirror point, each holding a `case.md` and an optional fixture.
A `case.md` carries a one-line `description`, a `## Prompt` given identically to every arm, an optional `## Seed` transcript in role-tagged form for conversation-driven skills, an optional `## Hard assertions` block of executable shell predicates, and a `## Soft criteria` list with at least one entry.
Every case ships a hermetic, committed fixture, and each arm-and-trial combination runs against a fresh copy of it so writes never leak between runs.
The `case.md` shape (from the case-format decision) is the authoring contract:
```
---
description: <one-line scenario>
---
## Prompt
<the realistic user request, given identically to every arm>
## Seed (optional; conversation-based skills only)
**User:** ...
**Assistant:** ...
## Hard assertions (executable; $OUTPUT = arm's final message, $WORLD = its fixture copy)
```sh
test -f "$WORLD/review.md"
grep -qE '<pattern>' "$OUTPUT"
```
## Soft criteria
- <a statement the judge grounds the comparisons on>
```
**The `benchmark-skill` runner skill.**
A distributed skill, auto-discovered and packaged like any other skill in this repo, marked non-model-invocable so it only ever runs deliberately, invoked `/benchmark-skill <name>`.
It runs in-session as an AI script, orchestrating the arms and the judge as subagents via the workflow mechanism, so the whole battery stays on the interactive subscription quota rather than the metered automation credit pool.
It validates the shape of the target skill's `tests/` tree at run time — that each `case.md` parses, that a referenced fixture exists, that the test directory maps to a real skill — and reports a skill that has no tests.
For each case it runs a **new-skill arm** (the working tree) and a **no-skill baseline arm**.
The new arm is force-invoked: its subagent is pointed at the skill's directory and told to use it, reading that skill's own file and any assets it references, so the real skill machinery is exercised.
The baseline arm receives the bare prompt with the skill absent from its context, the honest counterfactual of the skill not existing.
Every arm is given the identical realistic prompt authored once and arm-agnostically.
Each case runs 5 paired trials at a realistic temperature, configurable per skill, with a per-skill temperature-zero override for a skill that wraps a mechanical task, and with no reuse of arms or judgments across arms or runs.
Per trial, one blind judge subagent decides between the two arms' outputs shown as unlabelled A and B with the order randomized, grounded on the case's soft criteria rather than free-forming its own standard, and instructed to discount mere length and formatting differences.
The hard-assertion gate runs against the new arm only, with `$OUTPUT` the path to the arm's captured final message and `$WORLD` the path to its fresh fixture copy; a non-zero exit fails the assertion, and a failed hard assertion fails the case outright regardless of the head-to-head.
**The deterministic core.**
A small committed program, shipped with the `benchmark-skill` skill and invoked by its prose, owns every mechanical, non-judgment step so the numbers are exact and reproducible rather than re-derived by the agent each run.
It is a pure transform: given the collected run data (per-arm transcript usage, hard-assertion results, and the per-trial judge verdicts) it returns the results model and the rendered HTML.
The run-history file and the trend ribbons that consume it are deferred to a later slice (0006); this slice's report is latest-only.
It is structured parameterized over the number of arms and the comparisons from the start, but this slice exercises only the two-arm efficacy shape.
It parses `case.md`, sums each arm's transcript token usage into the metric set — raw tokens across the four components, the pricing-weighted cost-equivalent-token figure, an imputed dollar cost, and the turn count from the transcript's tool-call rounds — collapses each efficacy trial to WIN, TIE, or LOSS for the new arm (a tie is a non-win), and applies the efficacy pass rule.
Efficacy passes for a case when wins ≥ 3 and losses ≤ 1 across the 5 trials, and any loss is flagged for human review.
The **Efficacy verdict** for the skill is green when every case passes efficacy.
It renders a self-contained HTML report: an Efficacy verdict badge and run metadata at the top, a per-arm cost table (new-skill and no-skill rows, each with turns, raw tokens, cost-equivalent tokens, and imputed cost, footnoted that absolute cost is inflated by shared-context cache overhead so the trustworthy signal is the new-vs-no-skill ratio), and the cases in stable authored order.
Cost is reported alongside quality but never gates a verdict.
The deterministic core is unit-tested at its single external seam, in the style of `checks/shell-hook.nix` and `checks/home-manager-module.nix`: fed committed fixtures (sample new-skill and no-skill arm transcripts, hard-assertion results, and per-trial judge verdicts) it must produce the expected per-arm metrics, the expected per-case and skill-level Efficacy verdict, and a correct report.
The test needs no LLM to run.
All report artifacts live under a git-ignored `tests/.reports/` directory.
## Acceptance criteria
- [x] A `tests/` tree at the repo top level mirrors `skills/` by full path; a skill's tests sit as case directories at the mirror point, each with a `case.md` and optional committed fixture.
- [x] `case.md` parses its `description`, `## Prompt`, optional `## Seed`, optional `## Hard assertions`, and `## Soft criteria` (at least one) per the authoring contract.
- [x] `benchmark-skill` is an auto-discovered, packaged, non-model-invocable distributed skill invoked as `/benchmark-skill <name>`.
- [x] The runner validates the target's `tests/` tree at run time (each `case.md` parses, referenced fixtures exist, the directory maps to a real skill) and reports a skill with no tests.
- [x] For each case the runner orchestrates a force-invoked new-skill arm and a skill-absent no-skill baseline arm as subagents via the workflow mechanism, entirely in-session.
- [x] Every arm receives the identical prompt; each arm-and-trial runs against a fresh copy of the case fixture; a conversation-driven case injects its `## Seed` transcript as the subagent's prior context.
- [x] Each case runs 5 paired trials at a per-skill-configurable realistic temperature, with a per-skill temperature-zero override, and no reuse of arms or judgments across arms or runs.
- [x] Per trial, one blind judge subagent compares the two arms as randomized unlabelled A/B, grounded on the case's soft criteria and instructed to discount length and formatting.
- [x] The hard-assertion gate runs against the new arm only with `$OUTPUT` and `$WORLD` provided; a non-zero exit fails the assertion and a failed assertion fails the case outright.
- [x] The deterministic core is a committed, pure-transform program shipped with the skill, structured parameterized over arms and comparisons though exercised here two-arm.
- [x] The core sums per-arm transcript usage into raw tokens, cost-equivalent tokens, imputed dollar cost, and turn count, and collapses each efficacy trial to WIN/TIE/LOSS (tie = non-win).
- [x] The core applies the efficacy pass rule (wins ≥ 3 and losses ≤ 1), flags any loss, and yields a skill-level Efficacy verdict that is green when every case passes.
- [x] The core renders a self-contained HTML report with the Efficacy badge, run metadata, a two-row per-arm cost table with the cache-overhead footnote, and cases in stable authored order; cost never gates.
- [x] A fixture-driven unit test (no LLM), in the style of the existing `checks/`, feeds sample transcripts, hard-assertion results, and judge verdicts to the core and asserts the metrics, the Efficacy verdict, and the report.
- [x] Report artifacts are written under a git-ignored `tests/.reports/` directory.
- [x] At least one existing real skill carries an authored `tests/` tree (a `case.md` with a fixture), and a live `/benchmark-skill <name>` run exercises it end to end and produces its efficacy report — proving the tracer bullet actually fires.
## Implementation Notes
Files: `skills/benchmark-skill/SKILL.md` (the in-session runner prose), `skills/benchmark-skill/CASE-FORMAT.md` (the `case.md` authoring contract), `skills/benchmark-skill/core/benchmark_core.py` (the deterministic core), `checks/benchmark-core.nix` + `checks/fixtures/benchmark/run-bundle.json` (the fixture-driven no-LLM check, wired into `flake.nix`), `tests/skills/axi-review/basic-cli-review/` (the real authored test tree with a committed `fixture/greet`), and a `tests/.reports/` `.gitignore` rule.
- **Core language — Python 3.**
The deterministic core is Python stdlib only, chosen because it renders HTML and parses JSON far more cleanly than the repo's shell/jq idiom, keeping the error-prone arithmetic and templating in committed, tested code rather than re-derived by the agent.
This machine has no `python3` on its global PATH, so the check pulls `pkgs.python3` as a build input and the live run invokes the core via `nix shell nixpkgs#python3`.
A distributed consumer repo is expected to provide `python3` at run time.
- **Live run — baseline contamination is a real harness limitation.**
The live `/benchmark-skill axi-review` ran all 5 paired trials end to end and produced `tests/.reports/axi-review.html` with real per-arm token metrics recovered from the arm transcripts, proving the tracer bullet fires.
The Efficacy verdict came out **red** (new arm won 2/5, baseline won 3/5).
That red is substantially an artifact: the no-skill baseline subagent, running in this repo on disk, discovered and used the skill's `AXI-PRINCIPLES.md` rubric (its own output states it "judged each principle against the canonical rubric in `AXI-PRINCIPLES.md`"), so it was not a clean no-skill counterfactual.
This is a limitation of in-session benchmarking — a subagent can read repo-resident skill assets — not a defect in the core, the pass rule, or the skill's design.
Hardening the baseline arm's filesystem isolation is a follow-up.
- **Trial temperature.**
Subagent temperature is not directly settable through the workflow's `agent()` surface, so the arms ran at the session's default (realistic) temperature.
The per-skill temperature and the temperature-zero override remain documented knobs in `SKILL.md` for a future headless path.
- **Parameterization kept per spec.**
The core carries the arms list and comparisons list (each with a `rule`) from the start, per "structured parameterized over the number of arms and the comparisons from the start", though this slice exercises only the two-arm efficacy shape and the single efficacy pass rule.
The regression arm and its inverted rule arrive in task 0005.
- **Review follow-through.**
A per-run pricing override was removed as speculative config.
Documented-standard breaches (semicolons in authored prose/comments, a comment sentence-per-line) were fixed.
`CASE-FORMAT.md`'s all-caps name is retained to match the repo's existing skill-companion-doc convention (`AXI-PRINCIPLES.md`, `REPORT-FORMAT.md`).

View File

@@ -1,77 +0,0 @@
---
spec: skill-benchmarking
blocked-by: 0008-arm-isolation-hardening
---
## What to build
The second head-to-head: a **previous-version arm** that turns a run into a three-arm experiment and adds a **Regression verdict** beside the Efficacy one, so an edit that silently made a trusted skill worse is caught rather than passing green.
The previous version is always main's `HEAD` — the repo's default branch, resolved rather than hardcoded to the literal name "main" — and the new version is the working tree.
Anchoring to the released state on main makes the regression question always "did my in-progress edit degrade from the trusted version," which is the moment of real risk.
The third arm is added **fully automatically and carries no new invocation surface**: the runner diffs the skill's directory against main's `HEAD` and adds the previous-version arm exactly when the two differ.
A brand-new skill absent from main, or a skill unchanged from `HEAD`, has no meaningful previous version, so the run degrades to the two-arm efficacy-only shape.
The previous-version arm's skill is materialized by checking out just the skill's directory at main's `HEAD` into a temp path, since each skill is a self-contained derivation that does not need the rest of the repo.
That temp materialization is cleaned up after the run, like the fresh fixture copies.
The previous-version arm is force-invoked identically to the new arm, pointed at this temp checkout instead of the new arm's copy of the working-tree directory, so the skill version is the only difference between them.
Cases are held fixed to the working tree — all arms run against today's prompt, fixture, and expectations — so the skill version is the only variable differing between the new and previous-version arms.
The hard-assertion gate stays on the new arm only, and the blind judge and the case's soft criteria are reused unchanged for the regression comparison; only the pair of outputs handed to the judge differs.
Two blind head-to-heads are now drawn per trial: **efficacy** pairs the new arm against the no-skill arm, and **regression** pairs the new arm against the previous-version arm.
Trial *i*'s new-skill output is judged against trial *i*'s no-skill output for efficacy and against trial *i*'s previous-version output for regression.
The regression pass rule inverts, because holding quality steady is the goal.
Per trial, the new-vs-old head-to-head collapses to WIN, TIE, or LOSS for the new arm, where a TIE means "as good as the previous version" (a success), a LOSS is the regression being hunted, and a WIN is a bonus improvement.
Per case, regression passes when losses ≤ 1 across the 5 trials, with no wins floor — ties and wins both count as non-regressions — and any loss is flagged for human review.
Dropping the wins floor means a deliberate no-op edit that ties the previous version passes rather than failing merely for tying.
The deterministic core is exercised across its full three-arm, two-comparison path (the same parameterized program from 0004, now run with both comparisons rather than re-architected).
It gains the regression pass rule and a skill-level **Regression verdict**, green when no case regressed.
The per-run result now carries the regression net-margin and a two-arm/three-arm flag, but persisting those into the durable per-run history line is deferred to the trend-history slice (0006), which owns history persistence; this slice's report remains latest-only with no ribbons.
The two verdicts are reported side by side as independent badges, because their four crossings carry genuinely different meanings, and the report headline reads both badges together into one "so what."
On a two-arm run with no previous version the Regression verdict reads not-applicable.
The cost table gains a **previous-version row** (turns, raw tokens, cost-equivalent tokens, imputed cost), which drops out on a two-arm run, and the footnote names the two trustworthy ratios: new-vs-no-skill and new-vs-old.
Each case panel shows both comparisons side by side, each with its per-trial WIN/TIE/LOSS strip and its pass result, and auto-expands when either comparison fails or carries a flagged loss — so a regression opens the panel even when efficacy is green — surfacing the losing-trial output pair for whichever comparison failed.
The core's fixture unit test is extended so the same test covers both shapes: the three-arm run (sample transcripts for all three arms and per-trial judge verdicts for both comparisons) asserting the per-arm metrics, both per-case and skill-level verdicts, and a report with the three-row cost table, and the two-arm run retained as the degenerate "no previous version" case.
## Acceptance criteria
- [x] The previous version resolves to the default branch's `HEAD` dynamically, never a hardcoded "main" literal, and the new version is the working tree.
- [x] The third arm is added automatically, with no new flag or argument, exactly when the skill directory differs from `HEAD`; otherwise the run stays two-arm efficacy-only.
- [x] The previous-version arm materializes by checking out just the skill's directory at `HEAD` into a temp path, force-invoked against it identically to the new arm, and the temp path is cleaned up after the run.
- [x] All arms run against the working-tree case (prompt, fixture, expectations), so the skill version is the only variable between the new and previous-version arms.
- [x] The hard-assertion gate remains new-arm-only; the same blind judge and soft criteria are reused for the regression comparison, differing only in the output pair.
- [x] Per trial, efficacy is judged new-vs-no-skill and regression new-vs-previous, using that trial's own outputs with no cross-arm reuse.
- [x] The regression pass rule passes a case when losses ≤ 1 across 5 trials with no wins floor (ties and wins are non-regressions), flagging any loss.
- [x] The core, unchanged in structure, runs the full three-arm two-comparison path and yields a skill-level Regression verdict green when no case regressed.
- [x] The report shows Efficacy and Regression as two independent badges with a headline reading both together; Regression reads not-applicable on a two-arm run.
- [x] The cost table gains a previous-version row that drops out on a two-arm run, with a footnote naming the new-vs-no-skill and new-vs-old ratios.
- [x] Each case panel shows both comparisons side by side with per-trial WIN/TIE/LOSS strips, auto-expanding on any fail or flagged loss and showing the losing-trial output pair for the failed comparison.
- [x] The core's fixture unit test covers both the three-arm and the degenerate two-arm shapes in the same test, asserting metrics, both verdicts, and the three-row cost table.
## Implementation Notes
The deterministic half — the pass rules, the two verdicts, the net margins, and the report — lives in `core/benchmark_core.py` and is fully covered by the extended `checks/benchmark-core.nix` fixture test.
The orchestration half — resolving `HEAD`, diff-gating the third arm, materializing the previous-version checkout, and drawing the two head-to-heads (criteria 16) — is specified in the runner's `SKILL.md` prose, since that half runs in-session as an AI script and, per the spec's Testing Decisions, is not unit-tested but established by running the harness.
- **Branching deviated from the standard `/implement` flow, at the user's direction.**
Task 0005 is blocked by 0008, which stacks on 0004, and none of that had been merged to `main` — so branching 0005 off a fresh `main` would have lost the entire benchmark foundation.
`main` was fast-forwarded to the `task-0004` stack tip (a clean linear superset) and pushed, then `task-0005` was branched off it.
- **The hard-assertion gate feeds Efficacy only, not Regression.**
The spec defines the regression pass rule purely as losses ≤ 1 and never mentions the gate, and the gate exists to catch the *current* skill emitting malformed output — an absolute property of the new arm, whereas regression is relative to the previous version.
So a hard failure fails the case's Efficacy axis outright while its Regression axis is judged purely on the head-to-head.
The three-arm fixture's `hard-gate-fail` case pins this: Efficacy fails on the gate, Regression passes.
- **The run bundle gained a per-trial `outputs` map (arm id → final message).**
0004 carried no output text; the losing-trial evidence pair this slice requires needs it, so the core reads `outputs` to surface the new arm's output beside the one it lost to, only for flagged-loss trials of a failed or flagged comparison.
- **Arm order is canonical: new-skill, previous-version, no-skill.**
The cost table renders arms in bundle order, and a Spec-axis review caught that the initial new/no-skill/previous order put the previous-version row last rather than in the spec's stated middle position.
The fixtures and the `SKILL.md` example now emit arms in the spec order.
- **Comparisons render side by side via a CSS grid.**
A Spec-axis review caught that the two comparisons were stacked vertically rather than laid out side by side as the report shape requires; a `.cmps` grid container now places them beside each other, wrapping to one column on a narrow viewport.
- **Arm shape is derived from the presence of a regression comparison, not the arm count**, so a run is two-arm exactly when no regression comparison was drawn.
- **The two committed fixtures replace 0004's single `run-bundle.json`.**
`three-arm-bundle.json` is the primary shape and `two-arm-bundle.json` is retained as the degenerate "no previous version" case, both exercised by the one test.
They are machine-generated for accuracy; the generator is not committed.
- **A live three-arm `/benchmark-skill` run is the natural follow-up validation** (as 0008 did for isolation) but is not gated by this slice's acceptance criteria, so it was not run here.

View File

@@ -1,59 +0,0 @@
---
spec: skill-benchmarking
blocked-by: 0005-regression-arm-and-two-verdicts
---
## What to build
The longitudinal layer that lets a real change be told apart from run-to-run noise, plus the fragility flags that stop a barely-green skill from masquerading as robust.
**History.**
The deterministic core appends one summary line per run to a per-skill history file and trims it, oldest-first, at roughly the last fifty.
Each line records both the efficacy and the regression net-margin (wins minus losses) and pass/fail, plus a flag for whether the run was two-arm or three-arm, so both ribbons can plot their own sparkline and correctly show a gap for any two-arm run.
The history is local and ephemeral: it lives inside the git-ignored reports directory and resets if that directory is cleaned, consistent with reports being transient artifacts.
**Trend ribbons.**
The report grows two stacked trend ribbons — an Efficacy ribbon and a Regression ribbon — each a net-margin sparkline over roughly the last seven runs, with verdict-colored per-run dots, the current run ringed, and a readout of the current net, the change versus the previous run, and how many recent runs were green.
The Regression ribbon's dot is absent for any run that was two-arm.
The Regression series is comparable across runs only while main is unchanged; because the previous-version arm is always main's `HEAD`, merging the branch moves main and resets the meaningful regression history, which is accepted rather than normalized.
**Fragility chips and the clean resting state.**
A clean run rests fully collapsed, and fragility is flagged per badge.
A green Efficacy badge carries a "narrowest margin" chip on the case fewest trial-flips from failing efficacy — closest to dropping under three wins.
A green Regression badge independently carries a chip on the case sitting at exactly one loss.
A case fragile on both axes shows both chips, so a barely-green skill cannot look as safe as a clean sweep on either axis.
The core's fixture unit test is extended to cover the history append-and-trim, the two ribbons' rendering including the two-arm gap in the regression series, and the per-badge fragility chips.
## Acceptance criteria
- [x] The core appends one summary line per run to a per-skill history file and trims oldest-first at roughly fifty lines.
- [x] Each history line records the efficacy and regression net-margins and pass/fail plus a two-arm/three-arm flag.
- [x] The history file lives inside the git-ignored reports directory and is treated as ephemeral (resets when that directory is cleaned).
- [x] The report renders two stacked ribbons (Efficacy and Regression), each a net-margin sparkline over ~7 runs with verdict-colored dots, the current run ringed, and a net/delta/green-count readout.
- [x] The Regression ribbon shows no dot for any run that was two-arm.
- [x] A clean run rests fully collapsed.
- [x] A green Efficacy badge carries a narrowest-margin chip on the case fewest trial-flips from failing efficacy; a green Regression badge independently carries a chip on the case at exactly one loss; a both-axes-fragile case shows both.
- [x] The core's fixture unit test covers the history append-and-trim, both ribbons (including the two-arm gap), and the per-badge fragility chips.
## Implementation Notes
The whole slice lives in the deterministic core, `core/benchmark_core.py`, and is fully covered by the extended `checks/benchmark-core.nix` fixture test.
The runner's `SKILL.md` step 6 gains the `--history tests/.reports/<name>.history.jsonl` argument and a description of the append-trim and the ribbons, since the orchestration half runs in-session and is established by running the harness rather than unit-tested.
- **The standard `/implement` branching flow was followed this time.**
Task 0005's PR had merged to `main`, so the blocker was reachable: `main` was fast-forwarded to `origin/main` and `task-0006` branched off it, without the manual stack-tip fast-forward 0005 needed.
- **History persistence is opt-in via `--history` and is the core's only side effect.**
Without the flag the core stays a pure transform and renders no ribbons, so the 0005-era report shape is unchanged.
The history is JSON-lines — one run summary object per line — read, appended, trimmed to fifty, and rewritten.
- **The ribbon delta compares against the previous run that carries a value on that axis, not the literal previous line.**
On the regression axis this skips a two-arm gap rather than blanking the change readout, consistent with the spec accepting regression discontinuity across a main move.
- **The green-count readout shows `green/applicable`, not a bare green count.**
The denominator is the number of runs in the window that have a value on that axis, so a regression readout excludes two-arm runs it cannot score, which reads more honestly than counting them against the total.
- **The sparkline draws a faint dashed zero baseline** when the plotted window straddles zero, as a readability aid for telling a positive net-margin run from a negative one.
- **Two committed fixtures were added, machine-generated for accuracy with the generator not committed** (the 0005 precedent).
The existing red fixtures carry no green axis and so exercise no chips, so `clean-bundle.json` (fully clean, green/green) pins the fully-collapsed resting state and the lone narrowest-margin efficacy chip, and `fragile-bundle.json` (green but fragile) pins the independent per-badge chips and the both-axes case carrying both.
- **The efficacy chip is a single case, the regression chip is every case at exactly one loss.**
This asymmetry follows the wording: "narrowest margin" is comparative and picks one case (ties broken by authored order), while "the case sitting at exactly one loss" is an absolute condition any number of cases can meet.
- **A case at exactly one regression loss both auto-expands and carries its chip.**
"A clean run rests fully collapsed" holds because a clean run has zero losses, so nothing trips the 0005 flagged-loss auto-expand — the chip is what still flags fragility on a case that has no loss to open it (a green efficacy case sitting at exactly three wins).

View File

@@ -1,66 +0,0 @@
---
spec: skill-benchmarking
blocked-by: 0004-efficacy-benchmark-tracer-bullet
---
## What to build
The isolation fix the tracer bullet flagged as a follow-up: give every arm a hermetic **fixture-only world** so the no-skill baseline stops discovering the skill on disk, turning that run's contaminated red into an honest counterfactual.
In 0004 the arms ran with the repo as their working directory.
The no-skill baseline, solving its task inside the real repo, grepped up `skills/axi-review/AXI-PRINCIPLES.md` and graded against it — its own output admitted as much — so the measured "no-skill" arm was not the counterfactual of the skill not existing.
The Efficacy verdict came out red as a direct artifact of that contamination rather than a real result.
Each arm-and-trial now runs in a fixture-only world.
The arm subagent's working directory is its fresh fixture copy rather than the repo root, and nothing under `skills/` or the grading `tests/` tree sits on any path it explores from there.
This is what makes the no-skill baseline honest, because it can no longer find and read the skill's assets, and it also stops any arm from reading its own case's soft criteria or hard assertions and tuning its answer to the bar it will be judged against.
A with-skill arm is handed its skill as an **isolated temp materialization** placed outside the fixture, and the subagent is pointed there to force-invoke it.
For this slice's two-arm shape only the new-skill arm has one: a copy of the working-tree skill directory, so it reflects uncommitted edits and carries none of its repo surroundings.
The no-skill arm is handed nothing.
Each temp materialization is cleaned up after the run, like the fresh fixture copies.
Isolation here is deliberately soft.
An in-session subagent shares the machine and could in principle reach the repo by absolute path, so the baseline is additionally instructed to stay within its working directory.
Relocating each arm's world and giving that instruction moves contamination from near-certain to requiring an arm to deliberately wander outside its world.
A hard filesystem guarantee would require an OS sandbox and is out of scope.
`$WORLD` binds to the arm's fixture copy — now also its working directory — and `$OUTPUT` to its captured final message, with the hard-assertion gate unchanged from 0004.
The deterministic core and its fixture unit test are untouched, because isolation is a runner orchestration concern rather than a scoring one.
Correctness is established by re-running the harness on the real skill and reading the result.
A live `/benchmark-skill axi-review` run confirms the baseline arm's transcript no longer references `AXI-PRINCIPLES.md` or any `skills/` or `tests/` path, and reports the now-uncontaminated Efficacy verdict that replaces 0004's documented-artifact red.
This slice establishes the "materialize a with-skill arm's skill into an isolated temp path" pattern that the previous-version arm reuses for the old skill version.
## Acceptance criteria
- [x] Each arm-and-trial subagent runs with its working directory set to its fresh fixture copy, never the repo root.
- [x] Nothing under `skills/` or the `tests/` tree is present on any relative path an arm explores from its world.
- [x] The new-skill arm's skill is materialized into an isolated temp path outside the fixture — a copy of the working-tree directory that reflects uncommitted edits — and the subagent is pointed there to force-invoke it, with the temp materialization cleaned up after the run.
- [x] The no-skill baseline is handed no skill materialization and is instructed to stay within its working directory.
- [x] `$WORLD` binds to the arm's fixture copy and `$OUTPUT` to its captured final message, with the hard-assertion gate semantics unchanged from 0004.
- [x] The deterministic core and its fixture unit test are unchanged, since isolation is an orchestration concern and not a scoring one.
- [x] A live `/benchmark-skill axi-review` run confirms the baseline arm's transcript references no `AXI-PRINCIPLES.md`, `skills/`, or `tests/` path, and its Efficacy verdict reflects the honest counterfactual rather than the earlier contamination artifact.
## Implementation Notes
Changed `skills/benchmark-skill/SKILL.md` only: the world/materialization steps in §3, the `$WORLD` binding in §4, and the scratch cleanup in §6.
`core/benchmark_core.py` and its check were left untouched, and the unchanged core was confirmed to still run clean on its committed fixture.
- **Worlds are materialized outside the repo, not merely relocated within it.**
The spec calls for a "fixture-only world", and the sharp form that actually kills contamination is a world under a system temp path (`mktemp -d`), because a world still under `tests/.reports/` leaves `skills/` and the grading `tests/` tree reachable by upward navigation.
The prose now says so explicitly.
- **Live verification passed and reversed 0004's contaminated result.**
A real `/benchmark-skill axi-review`-shaped run of 5 paired trials, orchestrated through the workflow mechanism with each arm's world under a temp path, came back with the new arm winning 5/5 and the Efficacy verdict **green** — where 0004 had scored red at 2/5 purely because the baseline had read the skill's `AXI-PRINCIPLES.md`.
All five no-skill-baseline transcripts were clean of any `AXI-PRINCIPLES.md`, `skills/`, or `tests/` reference, and the baselines instead invented their own generic CLI rubric — the honest counterfactual of the skill not existing.
- **Soft isolation is realized by instruction, since a subagent's working directory is not hard-settable through the workflow surface.**
Each arm is told to `cd` into its world first and work only there, and the baseline is additionally told to stay within it.
This is the achievable in-session bar, and hard OS-sandbox isolation stays deferred per the spec's "Out of Scope".
- **Review follow-through.**
A spec-axis review caught that "copy the case fixture into the world" could be read as nesting the fixture one level down, which would break `$WORLD/<file>` predicates.
The wording now states that the world *is* the fresh fixture copy, so a `$WORLD/<file>` predicate resolves against the fixture root.
A semicolon in one acceptance-criterion line was recast.
- **The feature spec is not part of this commit.**
`.claude/spec/skill-benchmarking.md` carries the isolation decisions but remains untracked, matching how 0004 was committed (code and task file, without the spec), and committing it here would pull in prose-standard breaches from sections written earlier.
- The materialization pattern this establishes for the new arm is reused by 0005's previous-version arm, which points at a temp checkout of the skill at main's `HEAD` instead of a copy of the working-tree directory.

1
.envrc Normal file
View File

@@ -0,0 +1 @@
use flake

8
.gitignore vendored
View File

@@ -6,6 +6,8 @@ result-*
# Ignore automatically generated direnv output
.direnv
# Benchmark run artifacts: reports and local history, keyed by skill name.
tests/.reports/
# BEGIN mkSkillsShellHook
# Generated by mkSkillsShellHook. Nix-delivered skill symlinks, kept out of git.
.claude/skills
.agents/skills/gitea-axi
# END mkSkillsShellHook

36
AGENTS.md Normal file
View File

@@ -0,0 +1,36 @@
# skills
Personal agent skills packaged through Nix.
The domain glossary lives at `$(xdg-user-dir DOCUMENTS)/ai-artifacts/projects/skills/011-skills-context.md`.
## Conventions
- `AGENTS.md` is canonical for project instructions.
Context, ADRs, specs, and tasks live under `$(xdg-user-dir DOCUMENTS)/ai-artifacts/projects/skills/`.
- Skill source lives under `skills/<name>/`.
Do not edit generated links under `.agents/skills/` or `.claude/skills/` as source.
- Run `nix flake check` before considering a repository-wide change complete.
The check builds every packaged skill plus the Home Manager module and shell hook.
- The dev shell may place Nix-delivered skills into `.agents/skills/` and maintain `.claude/skills` as a compatibility symlink.
Generated ignore entries belong in the root `.gitignore`, not inside `.agents/skills/`, because skill scanners honor ignore files inside scanned skill directories.
## Gotchas
- A skill-local `GLOSSARY.md` is runtime reference for that skill and has no relationship to the project's AI-artifacts-vault context glossary.
- This repository owns packaged skill sources only.
Enabling, replacing, or removing them in a consumer repository such as dotfiles is outside its scope.
- `nix flake check` evaluates the Git snapshot and omits untracked skill files.
Use `nix flake check "path:$PWD"` to include a newly created skill before staging it.
- `$(xdg-user-dir DOCUMENTS)/ai-artifacts` is itself an Obsidian vault for AI-generated artifacts, distinct from the personal vault where AI-generated notes are forbidden.
- Project artifact filenames carry globally allocated identifiers.
Discover context and ADR artifacts through the AI artifacts vault convention rather than assuming fixed names or type directories.
- Do not turn opportunistic use of another skill into a declared dependency.
Invocation of an unrelated skill remains the agent's discretion unless the current skill directly requires its contract.
- Do not instruct a skill to execute synchronously.
In-process execution is the baseline, while isolation and parallelism belong to an external caller or runtime.
- When adapting an upstream skill, preserve mechanics the upstream skill leaves unspecified.
Customize only the surfaces required to fit the local workflow rather than turning incidental choices into new contracts.
- The installed `gitea-axi pr view` does not support `--fields`.
Use plain `gitea-axi pr view <number>`.
- Wayfinder `ticket/task/human` tickets can still be implementation work.
They are blocked only at the point where human input is required.

5
CLAUDE.md Normal file
View File

@@ -0,0 +1,5 @@
# Claude Code compatibility
You MUST read and follow [`AGENTS.md`](AGENTS.md) before doing any work in this repository.
`AGENTS.md` is the canonical project instruction file.
This file exists only so Claude Code discovers that canonical instruction file.

View File

@@ -1,3 +1,3 @@
# skills
My personal skills packaged through Nix.
Personal agent skills packaged through Nix.

View File

@@ -1,281 +0,0 @@
# Feeds the committed run-bundle fixtures through the benchmark skill's
# deterministic core and asserts the per-arm metrics, the per-case and
# skill-level Efficacy and Regression verdicts, and the rendered report — for
# both the three-arm shape and the degenerate two-arm "no previous version"
# shape.
# It also drives the longitudinal layer: the per-skill history append-and-trim,
# the two trend ribbons including the two-arm gap in the regression series, and
# the per-badge fragility chips on a clean run and a green-but-fragile one.
# No LLM runs.
{
pkgs,
}:
let
core = ../skills/benchmark-skill/core/benchmark_core.py;
threeArm = ./fixtures/benchmark/three-arm-bundle.json;
twoArm = ./fixtures/benchmark/two-arm-bundle.json;
clean = ./fixtures/benchmark/clean-bundle.json;
fragile = ./fixtures/benchmark/fragile-bundle.json;
in
pkgs.runCommandLocal "benchmark-core-check"
{
nativeBuildInputs = [ pkgs.python3 ];
inherit core threeArm twoArm clean fragile;
}
''
fail() { echo "FAIL: $1" >&2; exit 1; }
# --- three-arm run --------------------------------------------------------
echo "the core scores a three-arm run and renders its report"
python3 "$core" "$threeArm" --json three.json --html three.html \
|| fail "the core exited non-zero on the three-arm bundle"
echo "the three-arm results model carries both verdicts, net margins, and metrics"
python3 - three.json <<'PY' || fail "a three-arm results-model assertion failed"
import json, sys
r = json.load(open(sys.argv[1]))
assert r["armShape"] == "three-arm", r["armShape"]
# Efficacy is red: dead-weight fails the wins floor and hard-gate-fail trips
# the gate. Regression is red: regressed-still-valuable loses twice to the
# previous version.
assert r["efficacyVerdict"] == "red", r["efficacyVerdict"]
assert r["regressionVerdict"] == "red", r["regressionVerdict"]
assert r["efficacyNetMargin"] == 12, r["efficacyNetMargin"]
assert r["regressionNetMargin"] == 3, r["regressionNetMargin"]
new = r["armMetrics"]["new"]
assert new["turns"] == 60, new["turns"]
assert new["rawTokens"] == 228000, new["rawTokens"]
assert new["costEquivalentTokens"] == 246000, new["costEquivalentTokens"]
assert abs(new["imputedCost"] - 1.23) < 1e-9, new["imputedCost"]
base = r["armMetrics"]["baseline"]
assert base["turns"] == 40, base["turns"]
assert base["rawTokens"] == 114000, base["rawTokens"]
assert base["costEquivalentTokens"] == 123000, base["costEquivalentTokens"]
prev = r["armMetrics"]["previous"]
assert prev["turns"] == 60, prev["turns"]
assert prev["rawTokens"] == 174000, prev["rawTokens"]
assert prev["costEquivalentTokens"] == 195500, prev["costEquivalentTokens"]
assert abs(prev["imputedCost"] - 0.9775) < 1e-9, prev["imputedCost"]
# Cost-table rows render in arm order: new-skill, previous-version, no-skill.
assert [a["id"] for a in r["arms"]] == ["new", "previous", "baseline"], \
[a["id"] for a in r["arms"]]
cases = {c["name"]: c for c in r["cases"]}
# Authored order is preserved, never reshuffled by verdict.
assert [c["name"] for c in r["cases"]] == [
"clean-both", "regressed-still-valuable", "dead-weight", "hard-gate-fail"
]
a = cases["clean-both"]
assert a["efficacyPassed"] is True
assert a["regressionPassed"] is True
assert a["comparisons"]["efficacy"]["trials"] == ["WIN", "WIN", "WIN", "WIN", "TIE"]
assert a["comparisons"]["regression"]["trials"] == ["TIE", "TIE", "WIN", "TIE", "TIE"]
# Efficacy green but regression red the exact crossing this feature exists
# to catch: still beats no-skill, yet degraded from the released version.
b = cases["regressed-still-valuable"]
assert b["efficacyPassed"] is True
assert b["regressionPassed"] is False
assert b["comparisons"]["regression"]["losses"] == 2
assert b["comparisons"]["regression"]["flaggedLosses"] == [0, 1]
# Efficacy red but regression green already dead weight, but the edit did
# not make it worse.
c = cases["dead-weight"]
assert c["efficacyPassed"] is False
assert c["regressionPassed"] is True
assert c["comparisons"]["efficacy"]["wins"] == 0
assert c["comparisons"]["efficacy"]["losses"] == 1
assert c["comparisons"]["efficacy"]["flaggedLosses"] == [2]
assert c["comparisons"]["regression"]["wins"] == 3
# The hard gate fails efficacy outright but does not gate regression, which
# is judged purely on losses.
d = cases["hard-gate-fail"]
assert d["hardFailed"] is True
assert d["efficacyPassed"] is False
assert d["regressionPassed"] is True
assert d["comparisons"]["efficacy"]["wins"] == 5
print("three-arm results-model assertions passed")
PY
echo "the three-arm report shows both badges, the three-row table, and evidence"
grep -q '<!doctype html>' three.html || fail "three-arm report is not self-contained"
grep -q 'Efficacy: RED' three.html || fail "three-arm report is missing the red Efficacy badge"
grep -q 'Regression: RED' three.html || fail "three-arm report is missing the red Regression badge"
grep -qi 'regress' three.html || fail "three-arm report is missing the verdict headline"
grep -q 'Cost-equiv tokens' three.html || fail "three-arm report is missing the cost table"
grep -q 'Previous version' three.html || fail "three-arm report is missing the previous-version row"
grep -q 'shared-context cache' three.html || fail "three-arm report is missing the cache footnote"
grep -q 'new-vs-no-skill' three.html || fail "three-arm footnote omits the new-vs-no-skill ratio"
grep -q 'new-vs-old' three.html || fail "three-arm footnote omits the new-vs-old ratio"
for name in clean-both regressed-still-valuable dead-weight hard-gate-fail; do
grep -q "$name" three.html || fail "three-arm report omits case $name"
done
grep -q 'class="cell LOSS"' three.html || fail "three-arm report is missing a LOSS cell"
grep -q 'Hard assertion failed' three.html || fail "three-arm report does not flag the hard failure"
grep -q ' open>' three.html || fail "three-arm report does not auto-expand a failing case"
grep -q 'class="cmps"' three.html || fail "three-arm report does not lay comparisons side by side"
# Evidence for the failed regression comparison: new vs previous on trial 1.
grep -q 'OUT-new-regressed-still-valuable-t0' three.html || fail "missing new evidence for the regression loss"
grep -q 'OUT-previous-regressed-still-valuable-t0' three.html || fail "missing previous evidence for the regression loss"
# Evidence for the failed efficacy comparison: new vs baseline on trial 3.
grep -q 'OUT-new-dead-weight-t2' three.html || fail "missing new evidence for the efficacy loss"
grep -q 'OUT-baseline-dead-weight-t2' three.html || fail "missing baseline evidence for the efficacy loss"
# A clean, collapsed case emits no losing-trial evidence.
if grep -q 'OUT-new-clean-both' three.html; then fail "a clean case leaked losing-trial evidence"; fi
# --- two-arm run ----------------------------------------------------------
echo "the core scores the degenerate two-arm run"
python3 "$core" "$twoArm" --json two.json --html two.html \
|| fail "the core exited non-zero on the two-arm bundle"
echo "the two-arm results model reads regression as not-applicable"
python3 - two.json <<'PY' || fail "a two-arm results-model assertion failed"
import json, sys
r = json.load(open(sys.argv[1]))
assert r["armShape"] == "two-arm", r["armShape"]
assert r["efficacyVerdict"] == "red", r["efficacyVerdict"]
assert r["regressionVerdict"] == "not-applicable", r["regressionVerdict"]
assert r["regressionNetMargin"] is None, r["regressionNetMargin"]
assert r["efficacyNetMargin"] == 9, r["efficacyNetMargin"]
new = r["armMetrics"]["new"]
assert new["turns"] == 45, new["turns"]
assert new["rawTokens"] == 171000, new["rawTokens"]
assert new["costEquivalentTokens"] == 184500, new["costEquivalentTokens"]
assert abs(new["imputedCost"] - 0.9225) < 1e-9, new["imputedCost"]
base = r["armMetrics"]["baseline"]
assert base["turns"] == 30, base["turns"]
assert base["rawTokens"] == 85500, base["rawTokens"]
assert base["costEquivalentTokens"] == 92250, base["costEquivalentTokens"]
assert "previous" not in r["armMetrics"], "two-arm run has no previous arm"
assert [c["name"] for c in r["cases"]] == [
"clean-pass", "regresses-baseline", "hard-gate-fail"
]
for c in r["cases"]:
assert c["regressionPassed"] is None, (c["name"], c["regressionPassed"])
b = {c["name"]: c for c in r["cases"]}["regresses-baseline"]
assert b["efficacyPassed"] is False
assert b["comparisons"]["efficacy"]["losses"] == 2
assert b["comparisons"]["efficacy"]["flaggedLosses"] == [3, 4]
h = {c["name"]: c for c in r["cases"]}["hard-gate-fail"]
assert h["hardFailed"] is True
print("two-arm results-model assertions passed")
PY
echo "the two-arm report reads Regression as not-applicable and drops the previous row"
grep -q 'Efficacy: RED' two.html || fail "two-arm report is missing the Efficacy badge"
grep -q 'Regression: N/A' two.html || fail "two-arm report does not read Regression as not-applicable"
grep -q 'new-vs-no-skill' two.html || fail "two-arm footnote omits the new-vs-no-skill ratio"
if grep -q 'new-vs-old' two.html; then fail "two-arm footnote names a new-vs-old ratio that does not apply"; fi
if grep -q 'Previous version' two.html; then fail "two-arm report shows a previous-version row"; fi
grep -q 'OUT-new-regresses-baseline-t3' two.html || fail "missing new evidence for the two-arm efficacy loss"
grep -q 'OUT-baseline-regresses-baseline-t3' two.html || fail "missing baseline evidence for the two-arm efficacy loss"
# --- clean run: fully collapsed, one narrowest-margin chip, ribbons ---------
echo "the core rests a clean run fully collapsed with a lone narrowest-margin chip"
python3 "$core" "$clean" --json clean.json --html clean.html --history clean.history.jsonl \
|| fail "the core exited non-zero on the clean bundle"
python3 - clean.json <<'PY' || fail "a clean-run assertion failed"
import json, sys
r = json.load(open(sys.argv[1]))
assert r["efficacyVerdict"] == "green", r["efficacyVerdict"]
assert r["regressionVerdict"] == "green", r["regressionVerdict"]
cases = {c["name"]: c for c in r["cases"]}
# The narrowest efficacy case (fewest wins) carries the chip, the roomy one does not.
assert [ch["axis"] for ch in cases["narrowest"]["chips"]] == ["efficacy"], cases["narrowest"]["chips"]
assert cases["roomy"]["chips"] == [], cases["roomy"]["chips"]
print("clean-run assertions passed")
PY
if grep -q ' open>' clean.html; then fail "a clean run did not rest fully collapsed"; fi
grep -q 'class="chip efficacy"' clean.html || fail "clean run omits the narrowest-margin efficacy chip"
if grep -q 'class="chip regression"' clean.html; then fail "clean run shows a spurious regression chip"; fi
grep -q 'class="ribbons"' clean.html || fail "clean run omits the trend ribbons"
grep -q 'class="spark"' clean.html || fail "clean run omits a sparkline"
grep -q 'class="ring"' clean.html || fail "clean run does not ring the current run"
# --- history append-and-trim and the two-ribbon window with a two-arm gap ---
echo "the core appends to and trims the per-skill history and plots the ribbon window"
python3 - seed.history.jsonl <<'PY' || fail "seeding the history fixture failed"
import json, sys
# 55 prior runs, all three-arm except the most recent, which is two-arm and so
# must plot a gap on the regression axis once it lands inside the 7-run window.
lines = []
for i in range(55):
two = i == 54
lines.append({
"generatedAt": f"seed-{i}",
"armShape": "two-arm" if two else "three-arm",
"efficacyNet": i % 5,
"efficacyPass": True,
"regressionNet": None if two else i % 3,
"regressionPass": None if two else True,
})
open(sys.argv[1], "w").write("".join(json.dumps(l) + "\n" for l in lines))
PY
python3 "$core" "$clean" --json trend.json --history seed.history.jsonl \
|| fail "the core exited non-zero on the seeded-history run"
python3 - seed.history.jsonl trend.json <<'PY' || fail "a history/trend assertion failed"
import json, sys
lines = [json.loads(l) for l in open(sys.argv[1]) if l.strip()]
# 55 prior + this run = 56, trimmed oldest-first back to the cap of 50.
assert len(lines) == 50, len(lines)
gens = [l["generatedAt"] for l in lines]
assert "seed-0" not in gens and "seed-5" not in gens, "oldest runs were not trimmed"
assert "seed-6" in gens and "seed-54" in gens, "recent runs were wrongly trimmed"
assert lines[-1]["armShape"] == "three-arm", lines[-1]
assert lines[-1]["efficacyNet"] == 8 and lines[-1]["regressionNet"] == 1, lines[-1]
t = json.load(open(sys.argv[2]))["trend"]
eff, reg = t["efficacy"]["points"], t["regression"]["points"]
assert len(eff) == 7 and len(reg) == 7, (len(eff), len(reg))
# Efficacy has a value every run, while regression breaks at the two-arm run.
assert all(p["net"] is not None for p in eff), "efficacy series should have no gaps"
assert eff[-1]["current"] and reg[-1]["current"], "the current run is ringed on both axes"
assert reg[5]["net"] is None, [p["net"] for p in reg]
assert reg[-1]["net"] is not None, "the current three-arm run has a regression dot"
assert t["regression"]["applicable"] == 6, t["regression"]["applicable"]
print("history/trend assertions passed")
PY
# --- green-but-fragile run: independent per-badge chips, both on one case ----
echo "the core chips each fragility axis independently, and both on a doubly-fragile case"
python3 "$core" "$fragile" --json fragile.json --html fragile.html \
|| fail "the core exited non-zero on the fragile bundle"
python3 - fragile.json <<'PY' || fail "a fragility-chip assertion failed"
import json, sys
r = json.load(open(sys.argv[1]))
assert r["efficacyVerdict"] == "green" and r["regressionVerdict"] == "green"
cases = {c["name"]: c for c in r["cases"]}
# One case is both the narrowest efficacy margin and at exactly one regression
# loss, so it carries both chips.
# The others carry at most their own axis.
assert {ch["axis"] for ch in cases["both"]["chips"]} == {"efficacy", "regression"}, cases["both"]["chips"]
assert [ch["axis"] for ch in cases["reg-only"]["chips"]] == ["regression"], cases["reg-only"]["chips"]
assert cases["none"]["chips"] == [], cases["none"]["chips"]
print("fragility-chip assertions passed")
PY
grep -q 'class="chip efficacy"' fragile.html || fail "fragile run omits the efficacy chip"
grep -q 'class="chip regression"' fragile.html || fail "fragile run omits the regression chip"
touch "$out"
''

View File

@@ -1,496 +0,0 @@
{
"skill": "clean-skill",
"generatedAt": "2026-07-24T10:00:00Z",
"temperature": 1.0,
"trialsPerCase": 5,
"arms": [
{
"id": "new",
"label": "New skill"
},
{
"id": "previous",
"label": "Previous version"
},
{
"id": "baseline",
"label": "No skill"
}
],
"comparisons": [
{
"id": "efficacy",
"label": "Efficacy",
"new": "new",
"against": "baseline",
"rule": "efficacy"
},
{
"id": "regression",
"label": "Regression",
"new": "new",
"against": "previous",
"rule": "regression"
}
],
"cases": [
{
"name": "narrowest",
"description": "narrowest scenario",
"softCriteria": [
"a good answer for narrowest"
],
"trials": [
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-narrowest-t0",
"previous": "OUT-previous-narrowest-t0",
"baseline": "OUT-baseline-narrowest-t0"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
},
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-narrowest-t1",
"previous": "OUT-previous-narrowest-t1",
"baseline": "OUT-baseline-narrowest-t1"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
},
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-narrowest-t2",
"previous": "OUT-previous-narrowest-t2",
"baseline": "OUT-baseline-narrowest-t2"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
},
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "tie",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-narrowest-t3",
"previous": "OUT-previous-narrowest-t3",
"baseline": "OUT-baseline-narrowest-t3"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
},
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "tie",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-narrowest-t4",
"previous": "OUT-previous-narrowest-t4",
"baseline": "OUT-baseline-narrowest-t4"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
}
]
},
{
"name": "roomy",
"description": "roomy scenario",
"softCriteria": [
"a good answer for roomy"
],
"trials": [
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "fixture"
},
"regression": {
"winner": "new",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-roomy-t0",
"previous": "OUT-previous-roomy-t0",
"baseline": "OUT-baseline-roomy-t0"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
},
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-roomy-t1",
"previous": "OUT-previous-roomy-t1",
"baseline": "OUT-baseline-roomy-t1"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
},
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-roomy-t2",
"previous": "OUT-previous-roomy-t2",
"baseline": "OUT-baseline-roomy-t2"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
},
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-roomy-t3",
"previous": "OUT-previous-roomy-t3",
"baseline": "OUT-baseline-roomy-t3"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
},
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-roomy-t4",
"previous": "OUT-previous-roomy-t4",
"baseline": "OUT-baseline-roomy-t4"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
}
]
}
]
}

View File

@@ -1,725 +0,0 @@
{
"skill": "fragile-skill",
"generatedAt": "2026-07-24T11:00:00Z",
"temperature": 1.0,
"trialsPerCase": 5,
"arms": [
{
"id": "new",
"label": "New skill"
},
{
"id": "previous",
"label": "Previous version"
},
{
"id": "baseline",
"label": "No skill"
}
],
"comparisons": [
{
"id": "efficacy",
"label": "Efficacy",
"new": "new",
"against": "baseline",
"rule": "efficacy"
},
{
"id": "regression",
"label": "Regression",
"new": "new",
"against": "previous",
"rule": "regression"
}
],
"cases": [
{
"name": "both",
"description": "both scenario",
"softCriteria": [
"a good answer for both"
],
"trials": [
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "fixture"
},
"regression": {
"winner": "new",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-both-t0",
"previous": "OUT-previous-both-t0",
"baseline": "OUT-baseline-both-t0"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
},
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-both-t1",
"previous": "OUT-previous-both-t1",
"baseline": "OUT-baseline-both-t1"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
},
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-both-t2",
"previous": "OUT-previous-both-t2",
"baseline": "OUT-baseline-both-t2"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
},
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "tie",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-both-t3",
"previous": "OUT-previous-both-t3",
"baseline": "OUT-baseline-both-t3"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
},
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "tie",
"rationale": "fixture"
},
"regression": {
"winner": "previous",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-both-t4",
"previous": "OUT-previous-both-t4",
"baseline": "OUT-baseline-both-t4"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
}
]
},
{
"name": "reg-only",
"description": "reg-only scenario",
"softCriteria": [
"a good answer for reg-only"
],
"trials": [
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-reg-only-t0",
"previous": "OUT-previous-reg-only-t0",
"baseline": "OUT-baseline-reg-only-t0"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
},
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-reg-only-t1",
"previous": "OUT-previous-reg-only-t1",
"baseline": "OUT-baseline-reg-only-t1"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
},
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-reg-only-t2",
"previous": "OUT-previous-reg-only-t2",
"baseline": "OUT-baseline-reg-only-t2"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
},
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-reg-only-t3",
"previous": "OUT-previous-reg-only-t3",
"baseline": "OUT-baseline-reg-only-t3"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
},
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "fixture"
},
"regression": {
"winner": "previous",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-reg-only-t4",
"previous": "OUT-previous-reg-only-t4",
"baseline": "OUT-baseline-reg-only-t4"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
}
]
},
{
"name": "none",
"description": "none scenario",
"softCriteria": [
"a good answer for none"
],
"trials": [
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-none-t0",
"previous": "OUT-previous-none-t0",
"baseline": "OUT-baseline-none-t0"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
},
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-none-t1",
"previous": "OUT-previous-none-t1",
"baseline": "OUT-baseline-none-t1"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
},
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-none-t2",
"previous": "OUT-previous-none-t2",
"baseline": "OUT-baseline-none-t2"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
},
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-none-t3",
"previous": "OUT-previous-none-t3",
"baseline": "OUT-baseline-none-t3"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
},
{
"hard": {
"ran": true,
"pass": true
},
"comparisons": {
"efficacy": {
"winner": "tie",
"rationale": "fixture"
},
"regression": {
"winner": "tie",
"rationale": "fixture"
}
},
"outputs": {
"new": "OUT-new-none-t4",
"previous": "OUT-previous-none-t4",
"baseline": "OUT-baseline-none-t4"
},
"usage": {
"new": {
"input": 1000,
"output": 500,
"cacheCreation": 200,
"cacheRead": 300,
"turns": 3
},
"previous": {
"input": 900,
"output": 450,
"cacheCreation": 180,
"cacheRead": 270,
"turns": 3
},
"baseline": {
"input": 600,
"output": 300,
"cacheCreation": 0,
"cacheRead": 0,
"turns": 2
}
}
}
]
}
]
}

View File

@@ -1,954 +0,0 @@
{
"skill": "sample-skill",
"generatedAt": "2026-07-24T12:00:00Z",
"temperature": 1.0,
"trialsPerCase": 5,
"arms": [
{
"id": "new",
"label": "New skill"
},
{
"id": "previous",
"label": "Previous version"
},
{
"id": "baseline",
"label": "No skill"
}
],
"comparisons": [
{
"id": "efficacy",
"label": "Efficacy",
"new": "new",
"against": "baseline",
"rule": "efficacy"
},
{
"id": "regression",
"label": "Regression",
"new": "new",
"against": "previous",
"rule": "regression"
}
],
"cases": [
{
"name": "clean-both",
"description": "New beats baseline and holds steady versus the previous version.",
"softCriteria": [
"The answer is grounded in the fixture."
],
"trials": [
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
},
"regression": {
"winner": "tie",
"rationale": "judge: regression tie"
}
},
"outputs": {
"new": "OUT-new-clean-both-t0",
"previous": "OUT-previous-clean-both-t0",
"baseline": "OUT-baseline-clean-both-t0"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
},
"previous": {
"input": 800,
"output": 1600,
"cacheCreation": 300,
"cacheRead": 6000,
"turns": 3
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
},
"regression": {
"winner": "tie",
"rationale": "judge: regression tie"
}
},
"outputs": {
"new": "OUT-new-clean-both-t1",
"previous": "OUT-previous-clean-both-t1",
"baseline": "OUT-baseline-clean-both-t1"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
},
"previous": {
"input": 800,
"output": 1600,
"cacheCreation": 300,
"cacheRead": 6000,
"turns": 3
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
},
"regression": {
"winner": "new",
"rationale": "judge: regression new"
}
},
"outputs": {
"new": "OUT-new-clean-both-t2",
"previous": "OUT-previous-clean-both-t2",
"baseline": "OUT-baseline-clean-both-t2"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
},
"previous": {
"input": 800,
"output": 1600,
"cacheCreation": 300,
"cacheRead": 6000,
"turns": 3
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
},
"regression": {
"winner": "tie",
"rationale": "judge: regression tie"
}
},
"outputs": {
"new": "OUT-new-clean-both-t3",
"previous": "OUT-previous-clean-both-t3",
"baseline": "OUT-baseline-clean-both-t3"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
},
"previous": {
"input": 800,
"output": 1600,
"cacheCreation": 300,
"cacheRead": 6000,
"turns": 3
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "tie",
"rationale": "judge: efficacy tie"
},
"regression": {
"winner": "tie",
"rationale": "judge: regression tie"
}
},
"outputs": {
"new": "OUT-new-clean-both-t4",
"previous": "OUT-previous-clean-both-t4",
"baseline": "OUT-baseline-clean-both-t4"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
},
"previous": {
"input": 800,
"output": 1600,
"cacheCreation": 300,
"cacheRead": 6000,
"turns": 3
}
},
"hard": {
"ran": true,
"pass": true
}
}
]
},
{
"name": "regressed-still-valuable",
"description": "New still beats baseline but has degraded from the previous version.",
"softCriteria": [
"The answer resolves the user's request."
],
"trials": [
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
},
"regression": {
"winner": "previous",
"rationale": "judge: regression previous"
}
},
"outputs": {
"new": "OUT-new-regressed-still-valuable-t0",
"previous": "OUT-previous-regressed-still-valuable-t0",
"baseline": "OUT-baseline-regressed-still-valuable-t0"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
},
"previous": {
"input": 800,
"output": 1600,
"cacheCreation": 300,
"cacheRead": 6000,
"turns": 3
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
},
"regression": {
"winner": "previous",
"rationale": "judge: regression previous"
}
},
"outputs": {
"new": "OUT-new-regressed-still-valuable-t1",
"previous": "OUT-previous-regressed-still-valuable-t1",
"baseline": "OUT-baseline-regressed-still-valuable-t1"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
},
"previous": {
"input": 800,
"output": 1600,
"cacheCreation": 300,
"cacheRead": 6000,
"turns": 3
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
},
"regression": {
"winner": "tie",
"rationale": "judge: regression tie"
}
},
"outputs": {
"new": "OUT-new-regressed-still-valuable-t2",
"previous": "OUT-previous-regressed-still-valuable-t2",
"baseline": "OUT-baseline-regressed-still-valuable-t2"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
},
"previous": {
"input": 800,
"output": 1600,
"cacheCreation": 300,
"cacheRead": 6000,
"turns": 3
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "tie",
"rationale": "judge: efficacy tie"
},
"regression": {
"winner": "new",
"rationale": "judge: regression new"
}
},
"outputs": {
"new": "OUT-new-regressed-still-valuable-t3",
"previous": "OUT-previous-regressed-still-valuable-t3",
"baseline": "OUT-baseline-regressed-still-valuable-t3"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
},
"previous": {
"input": 800,
"output": 1600,
"cacheCreation": 300,
"cacheRead": 6000,
"turns": 3
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
},
"regression": {
"winner": "tie",
"rationale": "judge: regression tie"
}
},
"outputs": {
"new": "OUT-new-regressed-still-valuable-t4",
"previous": "OUT-previous-regressed-still-valuable-t4",
"baseline": "OUT-baseline-regressed-still-valuable-t4"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
},
"previous": {
"input": 800,
"output": 1600,
"cacheCreation": 300,
"cacheRead": 6000,
"turns": 3
}
},
"hard": {
"ran": true,
"pass": true
}
}
]
},
{
"name": "dead-weight",
"description": "New does not beat baseline, but the edit did not make it worse.",
"softCriteria": [
"The answer is correct."
],
"trials": [
{
"comparisons": {
"efficacy": {
"winner": "tie",
"rationale": "judge: efficacy tie"
},
"regression": {
"winner": "new",
"rationale": "judge: regression new"
}
},
"outputs": {
"new": "OUT-new-dead-weight-t0",
"previous": "OUT-previous-dead-weight-t0",
"baseline": "OUT-baseline-dead-weight-t0"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
},
"previous": {
"input": 800,
"output": 1600,
"cacheCreation": 300,
"cacheRead": 6000,
"turns": 3
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "tie",
"rationale": "judge: efficacy tie"
},
"regression": {
"winner": "new",
"rationale": "judge: regression new"
}
},
"outputs": {
"new": "OUT-new-dead-weight-t1",
"previous": "OUT-previous-dead-weight-t1",
"baseline": "OUT-baseline-dead-weight-t1"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
},
"previous": {
"input": 800,
"output": 1600,
"cacheCreation": 300,
"cacheRead": 6000,
"turns": 3
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "baseline",
"rationale": "judge: efficacy baseline"
},
"regression": {
"winner": "tie",
"rationale": "judge: regression tie"
}
},
"outputs": {
"new": "OUT-new-dead-weight-t2",
"previous": "OUT-previous-dead-weight-t2",
"baseline": "OUT-baseline-dead-weight-t2"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
},
"previous": {
"input": 800,
"output": 1600,
"cacheCreation": 300,
"cacheRead": 6000,
"turns": 3
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "tie",
"rationale": "judge: efficacy tie"
},
"regression": {
"winner": "new",
"rationale": "judge: regression new"
}
},
"outputs": {
"new": "OUT-new-dead-weight-t3",
"previous": "OUT-previous-dead-weight-t3",
"baseline": "OUT-baseline-dead-weight-t3"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
},
"previous": {
"input": 800,
"output": 1600,
"cacheCreation": 300,
"cacheRead": 6000,
"turns": 3
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "tie",
"rationale": "judge: efficacy tie"
},
"regression": {
"winner": "tie",
"rationale": "judge: regression tie"
}
},
"outputs": {
"new": "OUT-new-dead-weight-t4",
"previous": "OUT-previous-dead-weight-t4",
"baseline": "OUT-baseline-dead-weight-t4"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
},
"previous": {
"input": 800,
"output": 1600,
"cacheCreation": 300,
"cacheRead": 6000,
"turns": 3
}
},
"hard": {
"ran": true,
"pass": true
}
}
]
},
{
"name": "hard-gate-fail",
"description": "New sweeps both head-to-heads but violates a hard assertion.",
"softCriteria": [
"The output is well-formed."
],
"trials": [
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
},
"regression": {
"winner": "tie",
"rationale": "judge: regression tie"
}
},
"outputs": {
"new": "OUT-new-hard-gate-fail-t0",
"previous": "OUT-previous-hard-gate-fail-t0",
"baseline": "OUT-baseline-hard-gate-fail-t0"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
},
"previous": {
"input": 800,
"output": 1600,
"cacheCreation": 300,
"cacheRead": 6000,
"turns": 3
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
},
"regression": {
"winner": "tie",
"rationale": "judge: regression tie"
}
},
"outputs": {
"new": "OUT-new-hard-gate-fail-t1",
"previous": "OUT-previous-hard-gate-fail-t1",
"baseline": "OUT-baseline-hard-gate-fail-t1"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
},
"previous": {
"input": 800,
"output": 1600,
"cacheCreation": 300,
"cacheRead": 6000,
"turns": 3
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
},
"regression": {
"winner": "tie",
"rationale": "judge: regression tie"
}
},
"outputs": {
"new": "OUT-new-hard-gate-fail-t2",
"previous": "OUT-previous-hard-gate-fail-t2",
"baseline": "OUT-baseline-hard-gate-fail-t2"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
},
"previous": {
"input": 800,
"output": 1600,
"cacheCreation": 300,
"cacheRead": 6000,
"turns": 3
}
},
"hard": {
"ran": true,
"pass": false
}
},
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
},
"regression": {
"winner": "tie",
"rationale": "judge: regression tie"
}
},
"outputs": {
"new": "OUT-new-hard-gate-fail-t3",
"previous": "OUT-previous-hard-gate-fail-t3",
"baseline": "OUT-baseline-hard-gate-fail-t3"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
},
"previous": {
"input": 800,
"output": 1600,
"cacheCreation": 300,
"cacheRead": 6000,
"turns": 3
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
},
"regression": {
"winner": "tie",
"rationale": "judge: regression tie"
}
},
"outputs": {
"new": "OUT-new-hard-gate-fail-t4",
"previous": "OUT-previous-hard-gate-fail-t4",
"baseline": "OUT-baseline-hard-gate-fail-t4"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
},
"previous": {
"input": 800,
"output": 1600,
"cacheCreation": 300,
"cacheRead": 6000,
"turns": 3
}
},
"hard": {
"ran": true,
"pass": true
}
}
]
}
]
}

View File

@@ -1,515 +0,0 @@
{
"skill": "sample-skill-2arm",
"generatedAt": "2026-07-24T12:00:00Z",
"temperature": 1.0,
"trialsPerCase": 5,
"arms": [
{
"id": "new",
"label": "New skill"
},
{
"id": "baseline",
"label": "No skill"
}
],
"comparisons": [
{
"id": "efficacy",
"label": "Efficacy",
"new": "new",
"against": "baseline",
"rule": "efficacy"
}
],
"cases": [
{
"name": "clean-pass",
"description": "New skill reliably beats the no-skill baseline.",
"softCriteria": [
"The answer is grounded in the fixture.",
"The answer follows the skill's format."
],
"trials": [
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
}
},
"outputs": {
"new": "OUT-new-clean-pass-t0",
"baseline": "OUT-baseline-clean-pass-t0"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
}
},
"outputs": {
"new": "OUT-new-clean-pass-t1",
"baseline": "OUT-baseline-clean-pass-t1"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
}
},
"outputs": {
"new": "OUT-new-clean-pass-t2",
"baseline": "OUT-baseline-clean-pass-t2"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
}
},
"outputs": {
"new": "OUT-new-clean-pass-t3",
"baseline": "OUT-baseline-clean-pass-t3"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "tie",
"rationale": "judge: efficacy tie"
}
},
"outputs": {
"new": "OUT-new-clean-pass-t4",
"baseline": "OUT-baseline-clean-pass-t4"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
}
},
"hard": {
"ran": true,
"pass": true
}
}
]
},
{
"name": "regresses-baseline",
"description": "New skill does not reliably beat the baseline.",
"softCriteria": [
"The answer resolves the user's request."
],
"trials": [
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
}
},
"outputs": {
"new": "OUT-new-regresses-baseline-t0",
"baseline": "OUT-baseline-regresses-baseline-t0"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
}
}
},
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
}
},
"outputs": {
"new": "OUT-new-regresses-baseline-t1",
"baseline": "OUT-baseline-regresses-baseline-t1"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
}
}
},
{
"comparisons": {
"efficacy": {
"winner": "tie",
"rationale": "judge: efficacy tie"
}
},
"outputs": {
"new": "OUT-new-regresses-baseline-t2",
"baseline": "OUT-baseline-regresses-baseline-t2"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
}
}
},
{
"comparisons": {
"efficacy": {
"winner": "baseline",
"rationale": "judge: efficacy baseline"
}
},
"outputs": {
"new": "OUT-new-regresses-baseline-t3",
"baseline": "OUT-baseline-regresses-baseline-t3"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
}
}
},
{
"comparisons": {
"efficacy": {
"winner": "baseline",
"rationale": "judge: efficacy baseline"
}
},
"outputs": {
"new": "OUT-new-regresses-baseline-t4",
"baseline": "OUT-baseline-regresses-baseline-t4"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
}
}
}
]
},
{
"name": "hard-gate-fail",
"description": "Output wins the head-to-head but violates a hard assertion.",
"softCriteria": [
"The output is correct."
],
"trials": [
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
}
},
"outputs": {
"new": "OUT-new-hard-gate-fail-t0",
"baseline": "OUT-baseline-hard-gate-fail-t0"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
}
},
"outputs": {
"new": "OUT-new-hard-gate-fail-t1",
"baseline": "OUT-baseline-hard-gate-fail-t1"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
}
},
"outputs": {
"new": "OUT-new-hard-gate-fail-t2",
"baseline": "OUT-baseline-hard-gate-fail-t2"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
}
},
"hard": {
"ran": true,
"pass": false
}
},
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
}
},
"outputs": {
"new": "OUT-new-hard-gate-fail-t3",
"baseline": "OUT-baseline-hard-gate-fail-t3"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
}
},
"hard": {
"ran": true,
"pass": true
}
},
{
"comparisons": {
"efficacy": {
"winner": "new",
"rationale": "judge: efficacy new"
}
},
"outputs": {
"new": "OUT-new-hard-gate-fail-t4",
"baseline": "OUT-baseline-hard-gate-fail-t4"
},
"usage": {
"new": {
"input": 1000,
"output": 2000,
"cacheCreation": 400,
"cacheRead": 8000,
"turns": 3
},
"baseline": {
"input": 500,
"output": 1000,
"cacheCreation": 200,
"cacheRead": 4000,
"turns": 2
}
},
"hard": {
"ran": true,
"pass": true
}
}
]
}
]
}

View File

@@ -1 +0,0 @@
# delta — another top-level skill

View File

@@ -87,15 +87,38 @@ let
programs.agents.skills = [ skillAlpha ];
};
# Claude Code disabled, with a skill still selected.
claudeCodeOff = homeFiles {
programs.claude-code.enable = false;
# Pi enabled and Claude Code disabled.
piOnly = homeFiles {
programs.pi-coding-agent.enable = true;
programs.agents.skills = [ skillAlpha ];
};
# An empty selection with Claude Code enabled.
# Both supported harnesses enabled.
bothHarnesses = homeFiles {
programs.claude-code.enable = true;
programs.pi-coding-agent.enable = true;
programs.agents.skills = [ skillAlpha ];
};
# A non-default Pi `configDir`, so the assertions can prove placement follows it
# rather than a hardcoded `.pi/agent`.
customPiConfigDir = homeFiles {
programs.pi-coding-agent.enable = true;
programs.pi-coding-agent.configDir = "/home/operator/.config/pi/agent";
programs.agents.skills = [ skillAlpha ];
};
# No compatible harness enabled, with a skill still selected.
agentsOff = homeFiles {
programs.claude-code.enable = false;
programs.pi-coding-agent.enable = false;
programs.agents.skills = [ skillAlpha ];
};
# An empty selection with both supported harnesses enabled.
emptySelection = homeFiles {
programs.claude-code.enable = true;
programs.pi-coding-agent.enable = true;
programs.agents.skills = [ ];
};
in
@@ -107,7 +130,10 @@ pkgs.runCommandLocal "skills-home-manager-module-check"
attrSetSkills
wholeDirSkills
customConfigDir
claudeCodeOff
piOnly
bothHarnesses
customPiConfigDir
agentsOff
emptySelection
;
}
@@ -142,13 +168,35 @@ pkgs.runCommandLocal "skills-home-manager-module-check"
test ! -e "$customConfigDir/.claude/skills/sample-alpha" \
|| fail "skill was placed under a hardcoded .claude, ignoring configDir"
echo "Claude Code disabled: no skill entry is written despite a selection"
test ! -e "$claudeCodeOff/.claude/skills/sample-alpha" \
|| fail "a skill was placed with claude-code disabled"
echo "Pi enabled: selected skills land in Pi's global skills directory"
test -f "$piOnly/.pi/agent/skills/sample-alpha/SKILL.md" \
|| fail "selected skill did not land under Pi's configDir"
test ! -e "$piOnly/.claude/skills/sample-alpha" \
|| fail "a skill was placed for Claude Code when only Pi was enabled"
echo "both harnesses enabled: selected skills land in both global skills directories"
test -f "$bothHarnesses/.claude/skills/sample-alpha/SKILL.md" \
|| fail "selected skill did not land under Claude Code when both harnesses were enabled"
test -f "$bothHarnesses/.pi/agent/skills/sample-alpha/SKILL.md" \
|| fail "selected skill did not land under Pi when both harnesses were enabled"
echo "custom Pi configDir: placement follows configDir rather than a hardcoded .pi/agent"
test -f "$customPiConfigDir/.config/pi/agent/skills/sample-alpha/SKILL.md" \
|| fail "skill was not placed under Pi's configured configDir"
test ! -e "$customPiConfigDir/.pi/agent/skills/sample-alpha" \
|| fail "skill was placed under a hardcoded .pi/agent, ignoring configDir"
echo "agents disabled: no skill entry is written despite a selection"
test ! -e "$agentsOff/.claude/skills/sample-alpha" \
|| fail "a skill was placed with Claude Code disabled"
test ! -e "$agentsOff/.pi/agent/skills/sample-alpha" \
|| fail "a skill was placed with Pi disabled"
echo "empty selection: nothing is installed"
test ! -e "$emptySelection/.claude/skills/sample-alpha" \
|| fail "an unselected skill was placed"
|| fail "an unselected skill was placed for Claude Code"
test ! -e "$emptySelection/.pi/agent/skills/sample-alpha" \
|| fail "an unselected skill was placed for Pi"
touch "$out"
''

View File

@@ -53,69 +53,84 @@ pkgs.runCommandLocal "skills-shell-hook-check"
# The fixture project carries three things the hook must not disturb: a
# hand-authored skill under its own name, a hand-authored skill whose name
# collides with a selected one, and a symlink pointing outside the store.
mkdir -p proj/.claude/skills/handmade
printf '%s\n' "a hand-authored skill" > proj/.claude/skills/handmade/SKILL.md
mkdir -p proj/.claude/skills/sample-gamma
printf '%s\n' "a hand-authored gamma" > proj/.claude/skills/sample-gamma/SKILL.md
ln -s /nonexistent/external-target proj/.claude/skills/external
mkdir -p proj/.agents/skills/handmade
printf '%s\n' "a hand-authored skill" > proj/.agents/skills/handmade/SKILL.md
printf '%s\n' "# fixture ignore" "result" > proj/.gitignore
mkdir -p proj/.agents/skills/sample-gamma
printf '%s\n' "a hand-authored gamma" > proj/.agents/skills/sample-gamma/SKILL.md
ln -s /nonexistent/external-target proj/.agents/skills/external
cd proj
echo "first entry: the two free skills link as store symlinks that resolve"
printf '%s\n' "$hookBoth" > ../hook-both.sh
. ../hook-both.sh
for name in sample-alpha sample-beta; do
[ -L ".claude/skills/$name" ] || fail "$name was not linked as a symlink"
case "$(readlink ".claude/skills/$name")" in
[ -L ".agents/skills/$name" ] || fail "$name was not linked as a symlink"
case "$(readlink ".agents/skills/$name")" in
${builtins.storeDir}/*) : ;;
*) fail "$name's symlink does not point into the store" ;;
esac
test -f ".claude/skills/$name/SKILL.md" \
test -f ".agents/skills/$name/SKILL.md" \
|| fail "$name's link does not resolve to its SKILL.md"
done
echo "a selected skill colliding with a hand-authored one leaves the real directory"
{ [ -d .claude/skills/sample-gamma ] && [ ! -L .claude/skills/sample-gamma ]; } \
{ [ -d .agents/skills/sample-gamma ] && [ ! -L .agents/skills/sample-gamma ]; } \
|| fail "the colliding hand-authored skill was replaced by a symlink"
grep -qx "a hand-authored gamma" .claude/skills/sample-gamma/SKILL.md \
grep -qx "a hand-authored gamma" .agents/skills/sample-gamma/SKILL.md \
|| fail "the colliding hand-authored skill's content was overwritten"
echo "the non-colliding hand-authored skill is an untouched real directory"
{ [ -d .claude/skills/handmade ] && [ ! -L .claude/skills/handmade ]; } \
{ [ -d .agents/skills/handmade ] && [ ! -L .agents/skills/handmade ]; } \
|| fail "the hand-authored skill was replaced or removed"
test -f .claude/skills/handmade/SKILL.md || fail "the hand-authored skill lost its content"
test -f .agents/skills/handmade/SKILL.md || fail "the hand-authored skill lost its content"
echo "a symlink pointing outside the store is not ours, so it is left alone"
[ -L .claude/skills/external ] || fail "the non-store symlink was removed"
[ -L .agents/skills/external ] || fail "the non-store symlink was removed"
echo "the .gitignore lists only the skills the hook actually linked, plus itself"
grep -qx '/.gitignore' .claude/skills/.gitignore || fail ".gitignore does not ignore itself"
grep -qx '/sample-alpha' .claude/skills/.gitignore || fail ".gitignore omits sample-alpha"
grep -qx '/sample-beta' .claude/skills/.gitignore || fail ".gitignore omits sample-beta"
for tracked in /handmade /sample-gamma /external; do
grep -qx "$tracked" .claude/skills/.gitignore \
echo "Claude Code compatibility points at the canonical skill directory"
[ -L .claude/skills ] || fail ".claude/skills is not a compatibility symlink"
[ "$(readlink .claude/skills)" = "../.agents/skills" ] \
|| fail ".claude/skills points somewhere other than ../.agents/skills"
test -f .claude/skills/sample-alpha/SKILL.md \
|| fail "Claude Code compatibility link does not resolve selected skills"
echo "the project .gitignore lists only generated links and keeps skill scanners clear"
test ! -e .agents/skills/.gitignore || fail "the hook wrote an ignore file where skill scanners read it"
grep -qx '# fixture ignore' .gitignore || fail "the existing .gitignore content was lost"
grep -qx 'result' .gitignore || fail "the existing .gitignore pattern was lost"
grep -qx '# BEGIN mkSkillsShellHook' .gitignore || fail ".gitignore lacks the managed block start"
grep -qx '# END mkSkillsShellHook' .gitignore || fail ".gitignore lacks the managed block end"
grep -qx '.claude/skills' .gitignore || fail ".gitignore omits the compatibility symlink"
grep -qx '.agents/skills/sample-alpha' .gitignore || fail ".gitignore omits sample-alpha"
grep -qx '.agents/skills/sample-beta' .gitignore || fail ".gitignore omits sample-beta"
for tracked in .agents/skills/handmade .agents/skills/sample-gamma .agents/skills/external; do
grep -qx "$tracked" .gitignore \
&& fail ".gitignore lists $tracked, which would untrack a real path"
done
echo "no manifest or state file is written: only the fixtures and links exist"
entries=$(ls -A .claude/skills | sort | tr '\n' ' ')
[ "$entries" = ".gitignore external handmade sample-alpha sample-beta sample-gamma " ] \
|| fail "unexpected entries under .claude/skills: [$entries]"
entries=$(ls -A .agents/skills | sort | tr '\n' ' ')
[ "$entries" = "external handmade sample-alpha sample-beta sample-gamma " ] \
|| fail "unexpected entries under .agents/skills: [$entries]"
echo "second entry with sample-beta deselected: its symlink is removed, the rest stay"
printf '%s\n' "$hookAlpha" > ../hook-alpha.sh
. ../hook-alpha.sh
test ! -e .claude/skills/sample-beta || fail "the deselected skill's symlink was not removed"
[ -L .claude/skills/sample-alpha ] || fail "the still-selected skill was dropped"
{ [ -d .claude/skills/handmade ] && [ ! -L .claude/skills/handmade ]; } \
test ! -e .agents/skills/sample-beta || fail "the deselected skill's symlink was not removed"
[ -L .agents/skills/sample-alpha ] || fail "the still-selected skill was dropped"
{ [ -d .agents/skills/handmade ] && [ ! -L .agents/skills/handmade ]; } \
|| fail "the hand-authored skill was disturbed on re-entry"
{ [ -d .claude/skills/sample-gamma ] && [ ! -L .claude/skills/sample-gamma ]; } \
{ [ -d .agents/skills/sample-gamma ] && [ ! -L .agents/skills/sample-gamma ]; } \
|| fail "the colliding hand-authored skill was disturbed on re-entry"
[ -L .claude/skills/external ] || fail "the non-store symlink was disturbed on re-entry"
[ -L .agents/skills/external ] || fail "the non-store symlink was disturbed on re-entry"
echo "the .gitignore tracks the new selection"
grep -qx '/sample-alpha' .claude/skills/.gitignore || fail ".gitignore lost sample-alpha"
grep -qx '/sample-beta' .claude/skills/.gitignore \
echo "the .gitignore tracks the new selection without duplicating the managed block"
grep -qx '.agents/skills/sample-alpha' .gitignore || fail ".gitignore lost sample-alpha"
grep -qx '.agents/skills/sample-beta' .gitignore \
&& fail ".gitignore still lists the deselected sample-beta"
[ "$(grep -xc '# BEGIN mkSkillsShellHook' .gitignore)" = 1 ] \
|| fail ".gitignore has duplicate managed blocks"
touch "$out"
''

View File

@@ -34,8 +34,8 @@ pkgs.runCommandLocal "skill-build-check"
echo "content tier: discovery finds every skill regardless of nesting depth,"
echo " and stops at a skill rather than descending into its assets"
[ "$expectedNames" = "alpha beta delta gamma" ] \
|| fail "discovered names were [$expectedNames], expected [alpha beta delta gamma]"
[ "$expectedNames" = "alpha beta gamma" ] \
|| fail "discovered names were [$expectedNames], expected [alpha beta gamma]"
echo "content tier: each built skill has SKILL.md at its \$out root"
beta_path=

45
flake.lock generated
View File

@@ -18,7 +18,49 @@
"type": "github"
}
},
"gitea-axi": {
"inputs": {
"home-manager": "home-manager",
"nixpkgs": [
"nixpkgs"
]
},
"locked": {
"lastModified": 1785340481,
"narHash": "sha256-GSxdQ7w8yYfZnfkXUuZ2fYIKibe9ZU8xDGyqpeTb2tE=",
"ref": "refs/heads/main",
"rev": "627fc9a32bb80d97923540fc0d3e9661961462ba",
"revCount": 83,
"type": "git",
"url": "https://git.alexion.dev/alexion/gitea-axi"
},
"original": {
"type": "git",
"url": "https://git.alexion.dev/alexion/gitea-axi"
}
},
"home-manager": {
"inputs": {
"nixpkgs": [
"gitea-axi",
"nixpkgs"
]
},
"locked": {
"lastModified": 1784588016,
"narHash": "sha256-ouZe80aWEhMLVMkqICFDN+JUw+0FJtCr/bh+hHtRtMg=",
"owner": "nix-community",
"repo": "home-manager",
"rev": "deeb6b7eb7e0c44ae1819c051ce175bd92a85100",
"type": "github"
},
"original": {
"owner": "nix-community",
"repo": "home-manager",
"type": "github"
}
},
"home-manager_2": {
"inputs": {
"nixpkgs": [
"nixpkgs"
@@ -57,7 +99,8 @@
"root": {
"inputs": {
"flake-utils": "flake-utils",
"home-manager": "home-manager",
"gitea-axi": "gitea-axi",
"home-manager": "home-manager_2",
"nixpkgs": "nixpkgs"
}
},

View File

@@ -4,6 +4,11 @@
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
inputs.flake-utils.url = "github:numtide/flake-utils";
inputs.gitea-axi = {
url = "git+https://git.alexion.dev/alexion/gitea-axi";
inputs.nixpkgs.follows = "nixpkgs";
};
# Present only so `nix flake check` can evaluate `homeModules.default` against
# real home-manager (the home-manager-module check). Its nixpkgs follows this
# flake's, so the module is checked against the same nixpkgs-and-home-manager
@@ -14,7 +19,7 @@
inputs.home-manager.inputs.nixpkgs.follows = "nixpkgs";
outputs =
{ self, nixpkgs, flake-utils, home-manager }:
{ self, nixpkgs, flake-utils, home-manager, gitea-axi }:
let
mkSkill = import ./lib/mk-skill.nix;
mkSkillsShellHook = import ./lib/mk-skills-shell-hook.nix;
@@ -34,7 +39,7 @@
{
lib = { inherit mkSkill mkSkillsShellHook; };
# Global placement: an operator selects skills into `~/.claude/skills/`
# Global placement: an operator selects skills into enabled agent harnesses
# via `programs.agents.skills`. Not per-system — it is a module function,
# and the skills it places come from the importing configuration's own
# selection rather than this flake's nixpkgs.
@@ -85,21 +90,19 @@
inherit pkgs mkSkill mkSkillsShellHook;
};
# Proves the benchmark skill's deterministic core at its single seam:
# it feeds committed fixtures to the core and asserts the metrics, the
# Efficacy verdict, and the rendered report.
benchmark-core = import ./checks/benchmark-core.nix {
inherit pkgs;
};
};
# A working shell for developing skills in this repo.
# python3 runs the benchmark skill's deterministic core.
# The benchmark-skill is placed into ./.claude/skills/ on entry, so
# /benchmark-skill is available while working in this repo.
# The gitea-axi project skill is placed into ./.agents/skills/ on entry,
# with ./.claude/skills as a compatibility symlink.
devShells.default = pkgs.mkShell {
packages = [ pkgs.python3 ];
shellHook = mkSkillsShellHook [ (skillPackages pkgs).benchmark-skill ];
packages = [
gitea-axi.packages.${system}.gitea-axi
];
shellHook = mkSkillsShellHook [
gitea-axi.packages.${system}.gitea-axi-skill
];
};
}
);

View File

@@ -1,5 +1,5 @@
# Places operator-selected skills into Claude Code's skills directory, one per
# skill, for every project.
# Places operator-selected skills into each enabled agent's global skills
# directory, one per skill, for every project.
{
config,
lib,
@@ -8,6 +8,18 @@
let
cfg = config.programs.agents;
claudeCode = config.programs.claude-code;
pi = config.programs.pi-coding-agent;
skillFiles = configDir:
lib.listToAttrs (
map (skill: {
name = "${configDir}/skills/${skill.skillName}";
value = {
source = skill;
recursive = true;
};
}) cfg.skills
);
in
{
options.programs.agents.skills = lib.mkOption {
@@ -15,49 +27,42 @@ in
default = [ ];
example = lib.literalExpression "[ inputs.skills.packages.\${system}.some-skill ]";
description = ''
Skills to install globally, into the `skills/` subdirectory of
{option}`programs.claude-code.configDir`, so they are active in every
project.
Skills to install globally into every enabled agent harness that supports
Agent Skills.
Selection is by derivation: pass skill derivations from the skills flake's
`packages.<system>`, each of which carries its placement name as an
eval-time attribute. An empty list installs nothing.
Every skill is written as its own recursive {option}`home.file`, so this
option composes with an operator's own {option}`programs.claude-code.skills`
and with other modules placing skills under the same directory rather than
claiming the tree. Placement is gated on
{option}`programs.claude-code.enable`.
option composes with an operator's own harness-specific skills and with
other modules placing skills under the same directory rather than claiming
the tree. Placement is gated on each harness' enable option.
'';
};
# `agents` is an umbrella namespace with no shared enable, so the skills
# feature self-gates.
# An empty list writes nothing, and files land only when Claude Code is
# enabled.
# Reading each skill's source realises its derivation, so gating on
# `claude-code.enable` also keeps that realisation off a host that installs
# nothing.
config = lib.mkIf claudeCode.enable {
# `configDir` comes from the Claude Code module, so skills land beside its
# own wherever the operator points it.
# Each name is the derivation's eval-time `skillName`, so placement reads no
# `$out` and needs no import-from-derivation.
# `recursive = true` materialises `skills/<name>/` as a directory of per-file
# symlinks, which is what lets this module, the operator's own skills, and
# self-placing tool modules share one `skills/` tree.
#
# This deliberately does not feed the single-valued
# `programs.claude-code.skills`, which would collide with an operator already
# setting it.
home.file = lib.listToAttrs (
map (skill: {
name = "${claudeCode.configDir}/skills/${skill.skillName}";
value = {
source = skill;
recursive = true;
};
}) cfg.skills
);
};
# An empty list writes nothing, and files land only when a compatible harness
# is enabled.
# Reading each skill's source realises its derivation, so gating on harness
# enablement also keeps that realisation off a host that installs nothing.
config = lib.mkMerge [
(lib.mkIf claudeCode.enable {
# Each name is the derivation's eval-time `skillName`, so placement reads
# no `$out` and needs no import-from-derivation.
# `recursive = true` materialises `skills/<name>/` as a directory of
# per-file symlinks, which lets this module, the operator's own skills,
# and self-placing tool modules share one `skills/` tree.
#
# This deliberately does not feed the single-valued
# `programs.claude-code.skills`, which would collide with an operator
# already setting it.
home.file = skillFiles claudeCode.configDir;
})
(lib.mkIf pi.enable {
home.file = skillFiles pi.configDir;
})
];
}

View File

@@ -1,5 +1,8 @@
# Builds a dev-shell shellHook that links a project's selected skills into its
# own `.claude/skills/` as direct-child symlinks into the Nix store.
# own `.agents/skills/` as direct-child symlinks into the Nix store, with
# `.claude/skills` as a compatibility symlink when that path is free.
# Generated ignores live in the project root so agent skill scanners still see
# the linked skills.
skills:
let
storeDir = builtins.storeDir;
@@ -8,7 +11,7 @@ let
);
in
''
# Places this project's Nix-delivered skills into ./.claude/skills/ on shell
# Places this project's Nix-delivered skills into ./.agents/skills/ on shell
# entry.
# Each run first drops every direct-child symlink pointing into the store,
# then relinks the current selection.
@@ -16,12 +19,28 @@ in
# skills for free and needs no manifest.
# A real (non-symlink) skill directory is a hand-authored, project-private
# skill and is left untouched.
# When ./.claude/skills is free, it is kept as a compatibility symlink to the
# canonical ./.agents/skills directory so Claude Code sees the same skills.
# The whole hook runs in a subshell so it leaks no names into the interactive
# shell.
(
_mkskills_dir=".claude/skills"
_mkskills_dir=".agents/skills"
mkdir -p "$_mkskills_dir"
_mkskills_claude_skills_managed=0
mkdir -p .claude
if [ ! -e .claude/skills ] && [ ! -L .claude/skills ]; then
ln -s ../.agents/skills .claude/skills
_mkskills_claude_skills_managed=1
elif [ -L .claude/skills ]; then
case "$(readlink .claude/skills)" in
../.agents/skills) _mkskills_claude_skills_managed=1 ;;
${storeDir}/*) rm -f .claude/skills && ln -s ../.agents/skills .claude/skills && _mkskills_claude_skills_managed=1 ;;
esac
else
printf 'mkSkillsShellHook: leaving existing non-symlink .claude/skills in place; Claude Code compatibility link not installed\n' >&2
fi
for _mkskills_entry in "$_mkskills_dir"/*; do
[ -L "$_mkskills_entry" ] || continue
case "$(readlink "$_mkskills_entry")" in
@@ -30,8 +49,9 @@ in
done
# A real path at a selected skill's name is a hand-authored skill.
# It is left in place and left out of the managed set, so the .gitignore
# below keeps tracking it rather than the skill that would have been linked.
# It is left in place and left out of the managed set, so the generated
# ignore block keeps tracking it rather than the skill that would have been
# linked.
_mkskills_managed=""
_mkskills_link() {
_mkskills_dest="$_mkskills_dir/$1"
@@ -47,17 +67,46 @@ in
}
${links}
# Regenerate the self-ignoring .gitignore from the current selection, so the
# store symlinks stay out of git while hand-authored skills, absent from this
# list, remain tracked.
if [ -f "$_mkskills_dir/.gitignore" ] \
&& grep -qx '# Generated by mkSkillsShellHook. Nix-delivered skill symlinks, kept out of git.' "$_mkskills_dir/.gitignore"; then
rm -f "$_mkskills_dir/.gitignore"
fi
# Regenerate the project-root ignore block from the current selection.
# Skill scanners read ignore files under .agents/skills, so the block must
# live above the scanned tree.
_mkskills_gitignore=".gitignore"
_mkskills_gitignore_tmp=".gitignore.mkskills.$$"
if [ -f "$_mkskills_gitignore" ]; then
awk '
$0 == "# BEGIN mkSkillsShellHook" { skip = 1; next }
$0 == "# END mkSkillsShellHook" { skip = 0; next }
!skip { lines[++n] = $0 }
END {
while (n > 0 && lines[n] == "") n--
for (i = 1; i <= n; i++) print lines[i]
}
' "$_mkskills_gitignore" > "$_mkskills_gitignore_tmp"
else
: > "$_mkskills_gitignore_tmp"
fi
{
if [ -s "$_mkskills_gitignore_tmp" ]; then
cat "$_mkskills_gitignore_tmp"
printf '\n'
fi
printf '%s\n' "# BEGIN mkSkillsShellHook"
printf '%s\n' "# Generated by mkSkillsShellHook. Nix-delivered skill symlinks, kept out of git."
printf '%s\n' "/.gitignore"
if [ "$_mkskills_claude_skills_managed" = 1 ]; then
printf '%s\n' ".claude/skills"
fi
# Disable pathname expansion so a managed name is never glob-expanded.
set -f
for _mkskills_name in $_mkskills_managed; do
printf '/%s\n' "$_mkskills_name"
printf '.agents/skills/%s\n' "$_mkskills_name"
done
} > "$_mkskills_dir/.gitignore"
printf '%s\n' "# END mkSkillsShellHook"
} > "$_mkskills_gitignore"
rm -f "$_mkskills_gitignore_tmp"
)
''

View File

View File

@@ -96,7 +96,7 @@ Failure modes: writing errors to stderr, prompting for confirmation, or exiting
Canonical: install into the agent's session from an explicit setup command, so every conversation starts with relevant state visible, and ship an installable skill for on-demand guidance.
Probe: locate the tool's bundled Agent Skill file, under `~/.claude/skills/<tool>/` or in the installed package or repo, and check the surface for an explicit setup command.
Probe: locate the tool's bundled Agent Skill file, under `~/.pi/agent/skills/<tool>/`, `~/.claude/skills/<tool>/`, or in the installed package or repo, and check the surface for an explicit setup command.
- PASS: the tool ships a discoverable Agent Skill installed by an explicit setup command, optionally alongside a hook that injects a dashboard as initial context.
- PARTIAL: a setup command exists but installs no skill, or a skill exists with no explicit setup path, for example one installed by a postinstall script.

View File

@@ -17,7 +17,7 @@ The card's shape lives in [`REPORT-FORMAT.md`](REPORT-FORMAT.md).
Fix the invocation string first — how this CLI is run (`gitea-axi`, a binary path, `npx -y foo`).
Locate the tool's own Agent Skill file, since it is the authoritative description of the command surface and is itself the evidence for Principle 7.
Look under `~/.claude/skills/<tool>/SKILL.md`, then in the installed package or the repo.
Look under `~/.pi/agent/skills/<tool>/SKILL.md`, then `~/.claude/skills/<tool>/SKILL.md`, then in the installed package or the repo.
A missing skill file is a Principle 7 FAIL that you carry into the judging step.
When it is missing, enumerate the surface from `--help` and no-arg output instead.

View File

@@ -1,50 +0,0 @@
# The `case.md` authoring contract
Tests live in a top-level `tests/` tree that mirrors `skills/` by full path.
A skill at `skills/<maybe/cosmetic/nesting>/<skill>/` has its tests at `tests/skills/<maybe/cosmetic/nesting>/<skill>/`.
Under that mirror point, each **case** is its own directory holding a `case.md` and an optional fixture.
Tests sit outside `skills/` on purpose: a skill is packaged and placed on its own, and its tests must never ride along.
## `case.md`
```
---
description: <one-line scenario>
---
## Prompt
<the realistic user request, given identically to every arm>
## Seed (optional; conversation-based skills only)
**User:** ...
**Assistant:** ...
## Hard assertions (executable; $OUTPUT = arm's final message, $WORLD = its fixture copy)
```sh
test -f "$WORLD/review.md"
grep -qE '<pattern>' "$OUTPUT"
```
## Soft criteria
- <a statement the judge grounds the comparisons on>
```
- **`description`** — a one-line summary of the scenario, in the frontmatter.
- **`## Prompt`** — the realistic user request.
It is authored once and arm-agnostically, and every arm receives it verbatim.
Do not mention the skill by name or hint that a skill exists, or the baseline arm stops being an honest counterfactual.
- **`## Seed`** — optional, for conversation-driven skills whose input is a discussion rather than a file tree.
It is a role-tagged transcript (`**User:**` / `**Assistant:**`) injected as the subagent's prior context before the prompt.
- **`## Hard assertions`** — optional, a `sh` code block whose lines are each a separate predicate, together forming a deterministic gate.
Each predicate runs with `$OUTPUT` bound to the path of the arm's captured final message and `$WORLD` bound to the path of that arm's fresh fixture copy.
The gate passes only when **every** predicate exits zero — evaluate them so an earlier failure is never masked by a later success, not by a block's trailing exit code alone.
Any non-zero exit fails the gate, and a failed gate fails the case outright, against the new-skill arm only.
- **`## Soft criteria`** — at least one natural-language statement the blind judge grounds its comparisons on.
Describe what a good answer looks like, not which arm should win.
## Fixtures
A case ships a hermetic, committed fixture so a run is reproducible and needs no live external state.
File-and-tree skills get a fixture directory named `fixture/` beside `case.md`, whose contents populate each arm's world root — so a `$WORLD/<file>` predicate names a file directly under `fixture/`.
Conversation-driven skills use `## Seed` instead.
Every arm-and-trial combination runs against a **fresh copy** of the fixture, so writes never leak between runs.

View File

@@ -1,190 +0,0 @@
---
name: benchmark-skill
description: Benchmark a skill's efficacy against a no-skill baseline and, when a released version exists, its regression against that version, rendering an HTML report. Run deliberately as /benchmark-skill <name>, never automatically.
disable-model-invocation: true
---
# benchmark-skill
Prove that a skill genuinely improves the agent's work rather than reading well and adding nothing.
Running `/benchmark-skill <name>` executes that skill's authored test cases as a controlled experiment and produces a self-contained HTML report carrying two independent verdicts: an **Efficacy verdict** (does the skill beat no-skill) and a **Regression verdict** (did my in-progress edit degrade it from the released version).
Each case runs a **new-skill arm** (the working tree) and a **no-skill baseline arm**, and — whenever the skill already exists on main and its directory differs from that released state — a third **previous-version arm** materialized from main's `HEAD`.
Each arm runs several times, and a blind judge decides which arm's output better satisfies the case's author-written expectations.
Two head-to-heads fall out of the arms per trial: **efficacy** pairs the new arm against no-skill, and **regression** pairs it against the previous version.
When the skill is brand-new or unchanged from main, there is no previous version to compare against and the run degrades to the two-arm efficacy-only shape, with the Regression verdict reading not-applicable.
This skill runs **in-session as an AI script**: you orchestrate the arms and judges as subagents via the workflow mechanism, so the whole battery stays on the interactive subscription quota.
The mechanical, non-judgment work — summing token usage, collapsing verdicts, applying the pass rules, rendering the report — is done by a committed program, [`core/benchmark_core.py`](core/benchmark_core.py), so the numbers are exact and reproducible rather than re-derived each run.
You do the judgment work — running arms and judging — that the agent is actually good at.
The authoring contract for a test case is [`CASE-FORMAT.md`](CASE-FORMAT.md).
Read it before you read the target skill's tests.
Report artifacts live under a git-ignored `tests/.reports/` directory at the repo root, flat and keyed by skill name.
Run from the repo root: every path in the steps below is relative to it.
## 1. Resolve the target and validate its tests tree
The user passes the skill name as `<name>`.
Find the skill's directory by its leaf name under `skills/` (any depth), and find its tests at the mirror point under `tests/skills/…/<name>/`.
If no skill directory maps to `<name>`, stop and say so.
If the skill exists but has no tests directory or no case directories, report that the skill has no tests and stop — there is nothing to benchmark.
Validate the shape of the tests tree at run time:
- Each case directory has a `case.md` that parses per `CASE-FORMAT.md`.
- Any fixture a case references exists.
- The test directory maps to a real skill.
Report any malformed case and stop.
A benchmark on a broken tree would produce meaningless numbers.
Done when every case parses, its fixture exists, and you have the case list in stable authored order.
## 2. Fix the arms and per-run settings
Always run a **new-skill arm** and a **no-skill baseline arm**.
Add a third **previous-version arm** exactly when a released version exists to compare against, decided automatically with no flag or argument:
- Resolve the repo's default branch dynamically — `git symbolic-ref refs/remotes/origin/HEAD` (or `git remote show origin`), never a hardcoded `"main"` literal — and call its tip `HEAD`.
- Diff the skill's directory against `HEAD` (`git diff --quiet HEAD -- <skill-dir>`). Add the previous-version arm only when the directory both **exists at `HEAD`** and **differs** from it.
- A brand-new skill absent from `HEAD`, or a skill unchanged from it, has no meaningful previous version, so the run stays two-arm efficacy-only.
Arms run at the session's realistic default temperature (around 1.0), so the result reflects whether the skill *reliably* helps across variance rather than helping once by luck.
Per-arm temperature is not settable through the in-session workflow surface, so a skill's temperature-zero override for a genuinely mechanical task is a documented knob for the future headless path, not something to set here.
Each case runs **5 paired trials**.
For each trial, trial *i*'s new-skill output is judged against trial *i*'s no-skill output for efficacy and — on a three-arm run — against trial *i*'s previous-version output for regression.
There is no reuse of arms or judgments across arms or runs.
## 3. Run the arms and judges as subagents
Use the workflow mechanism (the `Workflow` tool) to fan out the arms and judges.
For each case, for each of the 5 trials, run every arm, then judge each comparison's pair.
**Every arm receives the identical `## Prompt`, authored once and arm-agnostically.**
For a conversation-driven case, inject the `## Seed` transcript as the subagent's prior context before the prompt.
**Give each arm-and-trial a hermetic fixture-only world.**
Make a fresh copy of the case fixture *outside the repo* — under a system temp path such as one from `mktemp -d`, never under `tests/.reports/`.
That copy is the **world**: it is the arm subagent's **working directory** and the `$WORLD` the hard-assertion gate reads, so a `$WORLD/<file>` predicate resolves against the fixture root.
Because the world lives outside the repo, nothing under `skills/` or the grading `tests/` tree sits on any path the arm reaches from there, so the baseline cannot discover the skill's assets and no arm can read its own case's soft criteria or hard assertions.
Each arm-and-trial gets its own fresh world, so writes never leak between them.
- **New-skill arm** — force-invoke the skill.
Materialize the skill into an **isolated temp path outside the world** — a copy of its working-tree directory, so it reflects uncommitted edits and carries none of its repo surroundings.
Point the subagent at that copy, tell it to use that skill, and have it read the skill's `SKILL.md` and any assets it references, so the real skill machinery is exercised.
Its working directory is the fresh world, and it also receives the prompt.
- **Previous-version arm** (three-arm runs only) — force-invoked identically to the new-skill arm, differing only in which skill it points at.
Materialize the skill's directory **at the default branch's `HEAD`** into its own isolated temp path (`git --work-tree=<temp> checkout HEAD -- <skill-dir>`, or `git archive HEAD <skill-dir>` piped into the temp path), since each skill is self-contained and needs none of the rest of the repo.
Point the subagent at that checkout and otherwise treat it exactly as the new-skill arm — same world, same prompt.
- **No-skill baseline arm** — the honest counterfactual of the skill not existing.
Materialize no skill for it: give it the bare prompt with the skill absent from its context, and do not mention the skill or hint that one exists.
Instruct it to stay within its working directory, since the isolation is soft and the subagent shares the machine.
The cases are held fixed to the working tree: all arms run against today's prompt, fixture, and expectations, so the skill version is the only variable between the new and previous-version arms.
Capture each arm's **final message** to a file — this is the `$OUTPUT` the hard-assertion gate reads and the text the judge compares.
**The judge** — one blind judge subagent per trial per comparison, so a three-arm trial draws two judgments (efficacy and regression) and a two-arm trial draws one.
For each comparison, show the judge the two arms' final messages as unlabelled **A** and **B**, with the order randomized per trial (record which of A/B is the new arm so you can map the verdict back).
The same judge machinery serves both comparisons; only the pair of outputs handed over differs — efficacy pairs new against no-skill, regression pairs new against the previous version.
Ground it on the case's `## Soft criteria` rather than letting it free-form its own standard, and instruct it explicitly to **discount mere length and formatting differences** — a skill must not win by being more verbose.
Have it return a single winner: A, B, or tie.
## 4. Run the hard-assertion gate
Run the case's `## Hard assertions` against the **new arm only**, once per trial, with `$OUTPUT` and `$WORLD` bound as `CASE-FORMAT.md` defines — that trial's new-arm final message and its world.
Record each trial's pass or fail into the bundle for the core to score.
A case with no `## Hard assertions` block simply has no gate.
## 5. Collect the run data
The deterministic core is a pure transform: it takes the collected data and returns the results model and HTML.
Assemble one **run bundle** JSON with this shape and write it under `tests/.reports/.work/<name>-bundle.json`:
The shape below is the **three-arm** run.
For a two-arm run, drop the `previous` arm, drop the `regression` comparison, and drop the `previous` key from each trial's `outputs` and `usage` — the core reads the arm shape off the presence of a regression comparison.
```json
{
"skill": "<name>",
"generatedAt": "<ISO-8601 timestamp>",
"temperature": 1.0,
"trialsPerCase": 5,
"arms": [
{"id": "new", "label": "New skill"},
{"id": "previous", "label": "Previous version"},
{"id": "baseline", "label": "No skill"}
],
"comparisons": [
{"id": "efficacy", "label": "Efficacy", "new": "new", "against": "baseline", "rule": "efficacy"},
{"id": "regression", "label": "Regression", "new": "new", "against": "previous", "rule": "regression"}
],
"cases": [
{
"name": "<case directory name>",
"description": "<from case.md frontmatter>",
"softCriteria": ["<each ## Soft criteria entry>"],
"trials": [
{
"hard": {"ran": true, "pass": true},
"comparisons": {
"efficacy": {"winner": "new", "rationale": "<judge's one line>"},
"regression": {"winner": "tie", "rationale": "<judge's one line>"}
},
"outputs": {
"new": "<new arm's final message>",
"previous": "<previous-version arm's final message>",
"baseline": "<no-skill arm's final message>"
},
"usage": {
"new": {"input": 0, "output": 0, "cacheCreation": 0, "cacheRead": 0, "turns": 0},
"previous": {"input": 0, "output": 0, "cacheCreation": 0, "cacheRead": 0, "turns": 0},
"baseline": {"input": 0, "output": 0, "cacheCreation": 0, "cacheRead": 0, "turns": 0}
}
}
]
}
]
}
```
- **`comparisons`** carries one entry per head-to-head, keyed by the comparison id.
`regression` is present only on a three-arm run.
Each **`winner`** is an arm id (`"new"`, `"baseline"`, or `"previous"`) or `"tie"`, mapped back from that comparison's blind A/B answer.
- **`hard`** carries the gate result for that trial's new arm.
Omit it or set `ran: false` when the case has no hard assertions.
The gate feeds the Efficacy axis only.
The Regression axis is judged purely on the head-to-head.
- **`outputs`** is each arm's captured final message, keyed by arm id.
The core surfaces the losing-trial pair as report evidence, so a failed comparison shows the new arm's output beside the one it lost to.
- **`usage`** is the per-arm-per-trial transcript usage.
Recover it by reading each arm-subagent's transcript file (the per-agent JSONL the workflow writes) and summing each message's token usage into the four components — `input`, `output`, `cacheCreation` (cache-creation input tokens), `cacheRead` (cache-read input tokens).
`turns` is the count of tool-call rounds in that transcript.
If a transcript genuinely lacks usage data, record zeros rather than guessing — cost never gates a verdict.
## 6. Score and render
Run the core over the bundle, writing the report and a machine-readable model:
```sh
python3 skills/benchmark-skill/core/benchmark_core.py \
tests/.reports/.work/<name>-bundle.json \
--json tests/.reports/<name>.results.json \
--html tests/.reports/<name>.html \
--history tests/.reports/<name>.history.jsonl
```
The core collapses each trial of each comparison to a WIN, TIE, or LOSS for the new arm, applies each comparison's pass rule — efficacy passes at wins ≥ 3 and losses ≤ 1, regression passes at losses ≤ 1 with no wins floor — flags any loss for human review, and yields the two skill-level verdicts: **Efficacy** (green when every case beats no-skill) and **Regression** (green when no case degraded, not-applicable on a two-arm run).
With `--history` it appends this run's summary line — each axis's net margin and pass/fail plus the two-arm/three-arm flag — to the per-skill history file, trimming it oldest-first at roughly the last fifty runs.
That history file lives in the git-ignored reports directory and is ephemeral: cleaning the directory resets it, consistent with reports being transient artifacts.
It renders the self-contained HTML report — the two verdict badges, two stacked trend ribbons plotting each axis's net-margin over the recent runs (the regression ribbon leaving a gap for any two-arm run), a headline reading the badges together, run metadata, the per-arm cost table (a previous-version row on a three-arm run), and the cases in stable authored order, each auto-expanding on any failure or flagged loss to show the losing-trial output pair — alongside the machine-readable model.
A clean run rests fully collapsed, and each green badge still flags its most fragile case with a chip so a barely-green skill cannot look robust.
Cost is reported but never gates a verdict.
Clean up the scratch when done: remove the `tests/.reports/.work/` directory and every per-arm world and skill materialization you created under the system temp path — including the previous-version checkout.
Done when `tests/.reports/<name>.html` exists.
Report both verdicts, the path to the report, and any flagged losses or regressions to the user.

View File

@@ -1,701 +0,0 @@
#!/usr/bin/env python3
"""Deterministic scoring and rendering for a skill benchmark run.
A pure transform: given a collected run bundle (per-arm transcript usage,
hard-assertion results, and the per-trial judge verdicts for each comparison) it
produces the results model and a self-contained HTML report. No LLM, no network,
no clock — the `generatedAt` stamp is supplied by the caller so the transform
stays pure.
The model is parameterized over the arms and the comparisons, so one path serves
both the three-arm run (efficacy against no-skill and regression against the
previous version) and the degenerate two-arm run with no previous version.
When `--history` names a per-skill history file, the run's summary line is
appended to it, the file is trimmed oldest-first, and the report grows two trend
ribbons plotted from the recent tail. That file read-and-rewrite is the only side
effect, opt-in and deterministic given the same file contents and inputs.
Usage:
benchmark_core.py <bundle.json> [--json <out.json>] [--html <out.html>]
[--history <hist.jsonl>]
With no output flags it writes the results model as JSON to stdout.
"""
import argparse
import html
import json
import sys
# Opus 4.8 per-million-token rates, in US dollars.
# Cache creation is the 5-minute-TTL write, priced at 1.25x input.
# Cache read is priced at 0.1x input.
# Cost is reported alongside quality but never gates a verdict.
DEFAULT_PRICING = {
"input": 5.0,
"output": 25.0,
"cacheCreation": 6.25,
"cacheRead": 0.5,
}
USAGE_COMPONENTS = ("input", "output", "cacheCreation", "cacheRead")
WIN, TIE, LOSS = "WIN", "TIE", "LOSS"
# The per-skill history is a JSON-lines tail of run summaries kept in the
# git-ignored reports directory.
# It is capped and trimmed oldest-first, and the ribbons plot only its most
# recent window.
HISTORY_CAP = 50
RIBBON_WINDOW = 7
# Efficacy passes at three wins, so this is the floor a green case sits above.
EFFICACY_WINS_FLOOR = 3
def _arm_usage_totals(bundle, arm_id):
"""Sum an arm's transcript usage across every case and trial."""
totals = {c: 0 for c in USAGE_COMPONENTS}
turns = 0
for case in bundle["cases"]:
for trial in case["trials"]:
usage = trial["usage"][arm_id]
for c in USAGE_COMPONENTS:
totals[c] += usage.get(c, 0)
turns += usage.get("turns", 0)
return totals, turns
def _arm_metrics(bundle, arm_id, pricing):
components, turns = _arm_usage_totals(bundle, arm_id)
raw_tokens = sum(components.values())
imputed_cost = sum(components[c] * pricing[c] for c in USAGE_COMPONENTS) / 1_000_000
# Cost-equivalent tokens: the dollar cost expressed in units of base input
# tokens, so a single figure captures the pricing-weighted total.
cost_equivalent_tokens = round(imputed_cost * 1_000_000 / pricing["input"])
return {
"turns": turns,
"rawTokens": raw_tokens,
"components": components,
"costEquivalentTokens": cost_equivalent_tokens,
"imputedCost": imputed_cost,
}
def _collapse_trial(verdict, comparison):
"""A trial's head-to-head becomes WIN, TIE, or LOSS for the new arm.
A tie is a non-win. Any winner that is neither the new nor the against arm
(a malformed verdict) is treated as a tie rather than crashing the run.
"""
winner = verdict.get("winner")
if winner == comparison["new"]:
return WIN
if winner == comparison["against"]:
return LOSS
return TIE
# Each comparison names a pass rule keyed here.
# Efficacy hunts for a reliable win, so it needs a wins floor.
# Regression hunts for a degradation, so a tie is already a success and only losses matter.
PASS_RULES = {
"efficacy": lambda wins, losses: wins >= 3 and losses <= 1,
"regression": lambda wins, losses: losses <= 1,
}
def _comparison_by_rule(comparisons, rule):
"""The comparison carrying a given pass rule, or None if absent."""
return next((c for c in comparisons if c["rule"] == rule), None)
def _score_case(case, comparisons):
hard_failed = any(
t.get("hard", {}).get("ran") and not t["hard"].get("pass", False)
for t in case["trials"]
)
per_comparison = {}
for comp in comparisons:
strip, flagged, evidence = [], [], []
for i, trial in enumerate(case["trials"]):
outcome = _collapse_trial(trial["comparisons"][comp["id"]], comp)
strip.append(outcome)
if outcome == LOSS:
flagged.append(i)
# The losing-trial output pair lets the report surface the new
# arm's output beside the one it lost to, for a human to eyeball.
outputs = trial.get("outputs", {})
evidence.append({
"trial": i,
"new": outputs.get(comp["new"], ""),
"against": outputs.get(comp["against"], ""),
})
wins = strip.count(WIN)
ties = strip.count(TIE)
losses = strip.count(LOSS)
head_to_head = PASS_RULES[comp["rule"]](wins, losses)
# The hard gate is new-arm-only and fails efficacy outright.
# It catches the current skill emitting malformed output.
# Regression is relative to the previous version, so it is judged purely on losses.
passed = head_to_head and not (hard_failed and comp["rule"] == "efficacy")
per_comparison[comp["id"]] = {
"trials": strip,
"wins": wins,
"ties": ties,
"losses": losses,
"passed": passed,
"flaggedLosses": flagged,
"evidence": evidence,
}
efficacy_id = _comparison_by_rule(comparisons, "efficacy")["id"]
regression = _comparison_by_rule(comparisons, "regression")
return {
"name": case["name"],
"description": case.get("description", ""),
"softCriteria": case.get("softCriteria", []),
"hardFailed": hard_failed,
"comparisons": per_comparison,
"efficacyPassed": per_comparison[efficacy_id]["passed"],
"regressionPassed": (
per_comparison[regression["id"]]["passed"] if regression else None
),
}
def _net_margin(cases, comparison_id):
"""Wins minus losses for a comparison, summed across every case."""
total = 0
for case in cases:
comp = case["comparisons"][comparison_id]
total += comp["wins"] - comp["losses"]
return total
def _flag_fragility(cases, efficacy_id, regression, efficacy_green, regression_green):
"""Chip the most fragile passing case on each green axis.
A green run still names its weakest case so a barely-green skill cannot pass
for robust. The efficacy chip lands on the single case closest to dropping
under the three-win floor, ties broken by authored order. The regression chip
lands on every case sitting at its one tolerated loss, so a case fragile on
both axes carries both.
"""
for case in cases:
case["chips"] = []
if efficacy_green and cases:
narrowest = min(cases, key=lambda c: c["comparisons"][efficacy_id]["wins"])
wins = narrowest["comparisons"][efficacy_id]["wins"]
narrowest["chips"].append({
"axis": "efficacy",
"label": f"narrowest efficacy margin · {wins}W",
})
if regression is not None and regression_green:
for case in cases:
if case["comparisons"][regression["id"]]["losses"] == 1:
case["chips"].append({
"axis": "regression",
"label": "one loss from regressing",
})
def build_results(bundle):
"""Turn a run bundle into the results model."""
pricing = DEFAULT_PRICING
comparisons = bundle["comparisons"]
arms = bundle["arms"]
regression = _comparison_by_rule(comparisons, "regression")
arm_metrics = {a["id"]: _arm_metrics(bundle, a["id"], pricing) for a in arms}
cases = [_score_case(c, comparisons) for c in bundle["cases"]]
efficacy_id = _comparison_by_rule(comparisons, "efficacy")["id"]
efficacy_green = all(c["efficacyPassed"] for c in cases)
# No previous-version arm means the regression axis has nothing to measure,
# so it reads not-applicable rather than green or red.
if regression is None:
regression_verdict = "not-applicable"
regression_net = None
else:
regression_verdict = (
"green" if all(c["regressionPassed"] for c in cases) else "red"
)
regression_net = _net_margin(cases, regression["id"])
_flag_fragility(
cases, efficacy_id, regression, efficacy_green, regression_verdict == "green"
)
return {
"skill": bundle["skill"],
"generatedAt": bundle.get("generatedAt", ""),
"temperature": bundle.get("temperature"),
"trialsPerCase": bundle.get("trialsPerCase"),
"armShape": "three-arm" if regression is not None else "two-arm",
"arms": arms,
"comparisons": comparisons,
"efficacyVerdict": "green" if efficacy_green else "red",
"regressionVerdict": regression_verdict,
"efficacyNetMargin": _net_margin(cases, efficacy_id),
"regressionNetMargin": regression_net,
"armMetrics": arm_metrics,
"cases": cases,
}
# --- History and trend --------------------------------------------------------
def summary_line(results):
"""The one-line run summary appended to the per-skill history.
It records each axis's net margin and pass/fail plus the arm shape — all the
ribbons need to redraw both sparklines and show a gap for any two-arm run. A
two-arm run has no regression measurement, so both its regression fields are
null.
"""
three_arm = results["armShape"] == "three-arm"
return {
"generatedAt": results.get("generatedAt", ""),
"armShape": results["armShape"],
"efficacyNet": results["efficacyNetMargin"],
"efficacyPass": results["efficacyVerdict"] == "green",
"regressionNet": results["regressionNetMargin"],
"regressionPass": (results["regressionVerdict"] == "green") if three_arm else None,
}
def load_history(path):
"""Read the JSON-lines history, tolerating a missing or empty file."""
try:
with open(path) as f:
return [json.loads(line) for line in f if line.strip()]
except FileNotFoundError:
return []
def trim_history(lines):
"""Keep the most recent runs, dropping the oldest past the cap."""
return lines[-HISTORY_CAP:]
def write_history(path, lines):
"""Rewrite the history file with the given lines, oldest first."""
with open(path, "w") as f:
for line in lines:
f.write(json.dumps(line) + "\n")
def build_trend(series):
"""Ribbon data for both axes over the most recent window of runs.
A run with no value on an axis — a two-arm run on the regression axis — plots
no dot and breaks the line, so the gap reads as absent rather than being
interpolated across. The delta compares against the previous run that actually
carries a value on that axis, so a two-arm gap does not blank the comparison.
"""
window = series[-RIBBON_WINDOW:]
def axis(net_key, pass_key):
points = [
{
"net": run.get(net_key),
"pass": run.get(pass_key),
"current": i == len(window) - 1,
}
for i, run in enumerate(window)
]
valued = [p for p in points if p["net"] is not None]
current_net = points[-1]["net"] if points else None
delta = None
if current_net is not None and len(valued) >= 2:
delta = valued[-1]["net"] - valued[-2]["net"]
return {
"points": points,
"current": current_net,
"delta": delta,
"green": sum(1 for p in points if p["pass"] is True),
"applicable": len(valued),
}
return {
"efficacy": axis("efficacyNet", "efficacyPass"),
"regression": axis("regressionNet", "regressionPass"),
}
# --- HTML rendering -----------------------------------------------------------
_STYLE = """
:root { color-scheme: light; }
body { font: 15px/1.5 -apple-system, system-ui, sans-serif; margin: 0; padding: 2rem;
color: #1a1a1a; background: #fafafa; }
.wrap { max-width: 60rem; margin: 0 auto; }
h1 { font-size: 1.4rem; margin: 0 0 .25rem; }
.badges { display: flex; gap: .5rem; margin: .25rem 0; flex-wrap: wrap; }
.badge { display: inline-block; padding: .2rem .7rem; border-radius: 999px;
font-weight: 600; font-size: .85rem; }
.badge.green { background: #d6f5df; color: #0f6b34; }
.badge.red { background: #fbdcdc; color: #9b1c1c; }
.badge.na { background: #eee; color: #666; }
.headline { font-size: 1.02rem; font-weight: 600; margin: .8rem 0 .2rem; }
.meta { color: #666; font-size: .85rem; margin: .5rem 0 1rem; }
.ribbons { display: grid; gap: .5rem; margin: .5rem 0 1.5rem; }
.ribbon { display: grid; grid-template-columns: 6.5rem 1fr auto; align-items: center;
gap: 1rem; border: 1px solid #e5e5e5; border-radius: 8px;
padding: .5rem .9rem; background: #fff; }
.rlabel { font-weight: 600; font-size: .85rem; }
.spark { width: 100%; height: 44px; display: block; }
.spark .line { fill: none; stroke: #c4c4c4; stroke-width: 1.5; }
.spark .zero { stroke: #e5e5e5; stroke-width: 1; stroke-dasharray: 2 3; }
.spark .dot.pass { fill: #2f9e5f; }
.spark .dot.fail { fill: #c0392b; }
.spark .ring { fill: none; stroke: #1a1a1a; stroke-width: 1.5; }
.readout { font-size: .8rem; color: #555; white-space: nowrap; text-align: right; }
.readout .up { color: #0f6b34; font-weight: 600; }
.readout .down { color: #9b1c1c; font-weight: 600; }
.chip { display: inline-block; margin-left: .5rem; padding: .05rem .55rem;
border-radius: 999px; font-size: .7rem; font-weight: 600;
vertical-align: middle; }
.chip.efficacy { background: #fff1cf; color: #8a5a00; }
.chip.regression { background: #e9e2fb; color: #5b3ea8; }
table { border-collapse: collapse; width: 100%; margin: .5rem 0; font-size: .9rem; }
th, td { text-align: right; padding: .4rem .6rem; border-bottom: 1px solid #e5e5e5; }
th:first-child, td:first-child { text-align: left; }
thead th { border-bottom: 2px solid #ccc; }
.footnote { color: #777; font-size: .8rem; margin: .3rem 0 1.5rem; }
.case { border: 1px solid #e5e5e5; border-radius: 8px; padding: 1rem 1.25rem;
margin: .75rem 0; background: #fff; }
.case.fail { border-color: #f0b6b6; }
.case .desc { color: #666; font-size: .85rem; margin: 0 0 .6rem; }
.strip { display: flex; gap: .3rem; margin: .3rem 0; flex-wrap: wrap; }
.cell { width: 2.6rem; text-align: center; padding: .2rem 0; border-radius: 4px;
font-size: .72rem; font-weight: 700; letter-spacing: .02em; }
.cell.WIN { background: #d6f5df; color: #0f6b34; }
.cell.TIE { background: #eee; color: #555; }
.cell.LOSS { background: #fbdcdc; color: #9b1c1c; }
.result { font-size: .85rem; font-weight: 600; }
.result.pass { color: #0f6b34; }
.result.fail { color: #9b1c1c; }
.tag { font-size: .75rem; color: #9b1c1c; font-weight: 600; }
.crit { color: #555; font-size: .82rem; margin: .5rem 0 0; padding-left: 1.1rem; }
.case > summary { cursor: pointer; font-weight: 600; font-size: 1rem;
list-style-position: outside; }
.case > summary::marker { color: #999; }
.cmps { display: grid; grid-template-columns: repeat(auto-fit, minmax(18rem, 1fr));
gap: 1rem; align-items: start; margin: .5rem 0 .2rem; }
.cmp { min-width: 0; }
.evidence { display: grid; grid-template-columns: repeat(auto-fit, minmax(12rem, 1fr));
gap: .6rem; margin: .4rem 0 0; }
.evidence figure { margin: 0; }
.evidence figcaption { font-size: .72rem; color: #666; font-weight: 600;
margin: 0 0 .2rem; }
.evidence pre { margin: 0; padding: .5rem .6rem; background: #f6f6f6;
border: 1px solid #e5e5e5; border-radius: 4px; font-size: .78rem;
white-space: pre-wrap; overflow-wrap: anywhere; }
.evidence .lead { grid-column: 1 / -1; font-size: .78rem; color: #9b1c1c;
font-weight: 600; margin: .3rem 0 0; }
"""
def _fmt_int(n):
return f"{n:,}"
def _fmt_cost(d):
return f"${d:,.4f}"
def _plural(n, noun):
return f"{n} {noun}" if n == 1 else f"{n} {noun}s"
def _headline(results):
"""One 'so what' sentence reading the two verdicts together.
Every clause it states is backed by a verdict or a case count rendered on the
same screen.
"""
eff = results["efficacyVerdict"]
reg = results["regressionVerdict"]
eff_fail = sum(1 for c in results["cases"] if not c["efficacyPassed"])
reg_fail = sum(1 for c in results["cases"] if c["regressionPassed"] is False)
if reg == "not-applicable":
if eff == "green":
return "This skill earns its keep against no-skill, with no previous version to regress against."
return (f"This skill does not beat no-skill on {_plural(eff_fail, 'case')}, "
"with no previous version to regress against.")
if eff == "green" and reg == "green":
return "This skill earns its keep and this edit held quality steady."
if eff == "green" and reg == "red":
return f"Still valuable, but this edit regressed {_plural(reg_fail, 'case')}."
if eff == "red" and reg == "green":
return (f"This skill does not beat no-skill on {_plural(eff_fail, 'case')}, "
"but this edit did not make it worse.")
return (f"This skill does not beat no-skill on {_plural(eff_fail, 'case')} "
f"and this edit regressed {_plural(reg_fail, 'case')}.")
def _badge(label, state):
text = {"green": "GREEN", "red": "RED", "not-applicable": "N/A"}[state]
cls = {"green": "green", "red": "red", "not-applicable": "na"}[state]
return f"<span class=\"badge {cls}\">{label}: {text}</span>"
def _sparkline(points, width=240, height=44):
"""An inline SVG net-margin sparkline where gaps break the line and drop no dot."""
nets = [p["net"] for p in points if p["net"] is not None]
if not nets:
return f"<svg viewBox=\"0 0 {width} {height}\" class=\"spark\"></svg>"
pad_x, pad_y = 10, 9
lo, hi = min(nets), max(nets)
n = len(points)
def x(i):
return width / 2 if n == 1 else pad_x + i * (width - 2 * pad_x) / (n - 1)
def y(v):
if hi == lo:
return height / 2
return height - pad_y - (v - lo) / (hi - lo) * (height - 2 * pad_y)
parts = [
f"<svg viewBox=\"0 0 {width} {height}\" class=\"spark\" "
"preserveAspectRatio=\"xMidYMid meet\" xmlns=\"http://www.w3.org/2000/svg\">"
]
if lo < 0 < hi:
zy = y(0)
parts.append(f"<line x1=\"0\" y1=\"{zy:.1f}\" x2=\"{width}\" y2=\"{zy:.1f}\" class=\"zero\"/>")
# Split the run into contiguous valued segments so a gap leaves a real break.
segments, current = [], []
for i, p in enumerate(points):
if p["net"] is None:
if current:
segments.append(current)
current = []
else:
current.append((x(i), y(p["net"])))
if current:
segments.append(current)
for seg in segments:
if len(seg) >= 2:
coords = " ".join(f"{px:.1f},{py:.1f}" for px, py in seg)
parts.append(f"<polyline points=\"{coords}\" class=\"line\"/>")
for i, p in enumerate(points):
if p["net"] is None:
continue
cx, cy = x(i), y(p["net"])
if p["current"]:
parts.append(f"<circle cx=\"{cx:.1f}\" cy=\"{cy:.1f}\" r=\"5.5\" class=\"ring\"/>")
cls = "dot pass" if p["pass"] else "dot fail"
parts.append(f"<circle cx=\"{cx:.1f}\" cy=\"{cy:.1f}\" r=\"3\" class=\"{cls}\"/>")
parts.append("</svg>")
return "".join(parts)
def _ribbon_readout(axis):
"""The net / delta-vs-previous / green-count line beneath a ribbon."""
net = "net —" if axis["current"] is None else f"net {axis['current']:+d}"
delta = axis["delta"]
if delta is None:
change = ""
elif delta > 0:
change = f"<span class=\"up\">▲ {delta}</span> vs prev"
elif delta < 0:
change = f"<span class=\"down\">▼ {abs(delta)}</span> vs prev"
else:
change = "±0 vs prev"
if axis["applicable"]:
green = f"{axis['green']}/{axis['applicable']} green"
else:
green = "no runs yet"
return " · ".join(bit for bit in (net, change, green) if bit)
def _render_ribbons(results):
"""Two stacked net-margin ribbons, empty when no history was supplied."""
trend = results.get("trend")
if not trend:
return ""
out = ["<div class=\"ribbons\">"]
for key, label in (("efficacy", "Efficacy"), ("regression", "Regression")):
axis = trend[key]
out.append("<div class=\"ribbon\">")
out.append(f"<div class=\"rlabel\">{label}</div>")
out.append(_sparkline(axis["points"]))
out.append(f"<div class=\"readout\">{_ribbon_readout(axis)}</div>")
out.append("</div>")
out.append("</div>")
return "".join(out)
def render_html(results):
e = html.escape
arm_labels = {a["id"]: a["label"] for a in results["arms"]}
three_arm = results["armShape"] == "three-arm"
out = []
out.append("<div class=\"wrap\">")
out.append(f"<h1>Benchmark — {e(results['skill'])}</h1>")
out.append("<div class=\"badges\">")
out.append(_badge("Efficacy", results["efficacyVerdict"]))
out.append(_badge("Regression", results["regressionVerdict"]))
out.append("</div>")
meta_bits = [f"arm shape: {e(results['armShape'])}"]
if results.get("trialsPerCase") is not None:
meta_bits.append(f"{results['trialsPerCase']} trials/case")
if results.get("temperature") is not None:
meta_bits.append(f"temperature {results['temperature']}")
if results.get("generatedAt"):
meta_bits.append(e(results["generatedAt"]))
out.append(f"<div class=\"meta\">{' · '.join(meta_bits)}</div>")
out.append(_render_ribbons(results))
out.append(f"<p class=\"headline\">{e(_headline(results))}</p>")
# Per-arm cost table.
out.append("<table><thead><tr><th>Arm</th><th>Turns</th><th>Raw tokens</th>"
"<th>Cost-equiv tokens</th><th>Imputed cost</th></tr></thead><tbody>")
for arm in results["arms"]:
m = results["armMetrics"][arm["id"]]
out.append(
f"<tr><td>{e(arm['label'])}</td><td>{_fmt_int(m['turns'])}</td>"
f"<td>{_fmt_int(m['rawTokens'])}</td>"
f"<td>{_fmt_int(m['costEquivalentTokens'])}</td>"
f"<td>{_fmt_cost(m['imputedCost'])}</td></tr>"
)
out.append("</tbody></table>")
if three_arm:
footnote = ("Absolute cost is inflated by shared-context cache overhead, "
"so the trustworthy signals are two ratios: new-vs-no-skill "
"(what the skill costs over nothing) and new-vs-old (what this "
"edit added or saved), not the absolute figures.")
else:
footnote = ("Absolute cost is inflated by shared-context cache overhead, "
"so the trustworthy signal is the new-vs-no-skill ratio, not "
"the absolute figures.")
out.append(f"<p class=\"footnote\">{footnote}</p>")
# Cases in stable authored order, each a collapsible panel that auto-expands
# when either comparison fails or carries a flagged loss.
for case in results["cases"]:
comps = case["comparisons"]
expanded = case["hardFailed"] or any(
c["flaggedLosses"] or not c["passed"] for c in comps.values()
)
failed = (not case["efficacyPassed"]) or case["regressionPassed"] is False
cls = "case fail" if failed else "case"
opened = " open" if expanded else ""
chips = "".join(
f"<span class=\"chip {ch['axis']}\">{e(ch['label'])}</span>"
for ch in case.get("chips", [])
)
out.append(f"<details class=\"{cls}\"{opened}>")
out.append(f"<summary>{e(case['name'])}{chips}</summary>")
if case["description"]:
out.append(f"<p class=\"desc\">{e(case['description'])}</p>")
if case["hardFailed"]:
out.append("<p class=\"tag\">Hard assertion failed — case fails outright.</p>")
out.append("<div class=\"cmps\">")
for comp in results["comparisons"]:
c = comps[comp["id"]]
out.append("<div class=\"cmp\">")
out.append(f"<div><strong>{e(comp['label'])}</strong></div>")
out.append("<div class=\"strip\">")
for outcome in c["trials"]:
out.append(f"<span class=\"cell {outcome}\">{outcome}</span>")
out.append("</div>")
rc = "pass" if c["passed"] else "fail"
summary = (f"{c['wins']}W / {c['ties']}T / {c['losses']}L — "
f"{'PASS' if c['passed'] else 'FAIL'}")
out.append(f"<div class=\"result {rc}\">{summary}</div>")
if c["flaggedLosses"]:
trials = ", ".join(f"#{i + 1}" for i in c["flaggedLosses"])
out.append(f"<div class=\"tag\">Loss flagged for review: trial {trials}</div>")
new_label = arm_labels.get(comp["new"], comp["new"])
against_label = arm_labels.get(comp["against"], comp["against"])
for ev in c["evidence"]:
out.append("<div class=\"evidence\">")
out.append(f"<p class=\"lead\">Trial #{ev['trial'] + 1} — losing output pair</p>")
out.append(
f"<figure><figcaption>{e(new_label)}</figcaption>"
f"<pre>{e(ev['new'])}</pre></figure>"
)
out.append(
f"<figure><figcaption>{e(against_label)}</figcaption>"
f"<pre>{e(ev['against'])}</pre></figure>"
)
out.append("</div>")
out.append("</div>")
out.append("</div>")
if case["softCriteria"]:
out.append("<ul class=\"crit\">")
for crit in case["softCriteria"]:
out.append(f"<li>{e(crit)}</li>")
out.append("</ul>")
out.append("</details>")
out.append("</div>")
title = f"Benchmark — {e(results['skill'])}"
return (
"<!doctype html><html><head><meta charset=\"utf-8\">"
f"<title>{title}</title><style>{_STYLE}</style></head><body>"
+ "".join(out)
+ "</body></html>"
)
def main(argv=None):
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("bundle", help="path to the run-bundle JSON")
parser.add_argument("--json", dest="json_out", help="write the results model here")
parser.add_argument("--html", dest="html_out", help="write the HTML report here")
parser.add_argument(
"--history",
dest="history",
help="append this run to the JSON-lines history file and draw the trend ribbons",
)
args = parser.parse_args(argv)
with open(args.bundle) as f:
bundle = json.load(f)
results = build_results(bundle)
if args.history:
series = load_history(args.history) + [summary_line(results)]
write_history(args.history, trim_history(series))
results["trend"] = build_trend(series)
if args.json_out:
with open(args.json_out, "w") as f:
json.dump(results, f, indent=2)
if args.html_out:
with open(args.html_out, "w") as f:
f.write(render_html(results))
if not args.json_out and not args.html_out:
json.dump(results, sys.stdout, indent=2)
sys.stdout.write("\n")
return 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -1,95 +0,0 @@
---
name: consume
description: Consume a project (or, later, another source) into the agent's personal wiki — distil generalized, transferable knowledge from it into the Obsidian vault's agent-owned area, then clear the consumed scaffolding. Deliberate: run as /consume [target], never automatically.
disable-model-invocation: true
---
# consume
Distil generalized, transferable knowledge from a source into the agent's own wiki, then clear the consumed scaffolding once the knowledge is safely captured.
`/consume [target]` ingests one target — a filesystem path, defaulting to the current directory — and writes what it mines into `01 sources/claude/`, the one area of the user's Obsidian vault the agent may write.
It is the write side of a pair with the read-only `/wiki`, which it invokes to read the rest of the vault.
The wiki holds reusable knowledge, never project documentation.
A fact earns a note only if it **generalizes** past the target it came from — strip the target's name and specifics, and if nothing teachable is left, it does not belong in the wiki.
Nothing is written and nothing is deleted until the user approves the plan.
## The write boundary
The vault lives at `$(xdg-user-dir DOCUMENTS)/notes`.
Resolve it; if it does not exist, report that the vault is unreachable and stop.
Write only under `01 sources/claude/`, and treat every other vault path as off-limits to writes.
The harness does not apply the vault's own permission rules when you run from outside it, so enforcing this boundary is the skill's job.
All reading of the wider vault goes through `/wiki` (step 3), which enforces the vault's read boundary itself.
## 1. Select the branch
Route the target through this ladder, first match wins:
- A directory containing `.claude/spec/` and/or `.claude/tasks/` → a project: follow [`project.md`](project.md).
- No rung matches → report that the target is not something consume knows how to read, write nothing, and stop.
Done when a branch file is selected, or consume has stopped on an unsupported target.
## 2. Read the target and establish ground truth
Follow the selected branch file to understand the target's *current* state.
The branch file names its source of truth and what within the target is merely stale intent; ground your understanding in the former, never the latter.
Done when you understand what the target actually is now, from its branch's source of truth.
## 3. Read what the wiki already knows
Two reads with different jobs:
- Invoke `/wiki` on the concepts the target raised, to learn what the whole vault already holds — so new notes link to existing notes and hubs, and you do not re-capture knowledge the vault already has.
- Scan `01 sources/claude/` directly for the concept notes you may need to enrich.
Done when you know which existing notes bear on what you are about to write.
## 4. Plan the notes
Distil the target's knowledge into generalized concept notes, each passing the **generalizes** test: it must teach something that holds on a *different* target, not a fact true only of this one.
Keep each note atomic — one transferable idea per note; when a note carries two independent lessons, split it.
Write as many notes as the knowledge warrants; the count follows from atomicity, not from a target number.
For each note, decide whether it is new or an update to an existing note whose knowledge is now stale or thinner than what you have learned, keeping the wiki accurate to your current understanding.
Then build the contribution map: for every file you consumed, the notes it feeds and the exact sections that contributed — including any file that fed no note, recorded as feeding nothing.
The mapping is many-to-many — one file may feed several notes, and one note may draw on several files.
Done when every planned note passes the generalizes test and is atomic, and every consumed file — including those feeding no note — is mapped, with its contributing sections named.
## 5. Present the plan and wait for approval
Write the plan as a self-contained HTML file to the session's scratchpad directory and give the user its path.
The report lists every file consumed, the notes each one contributes to, and the exact contributing sections.
It also flags any `[[hub]]` a note will link that does not yet exist under `02 tags/` or `03 index/`, as a suggested new hub for the user to create by hand.
Stop and wait for the user's explicit approval.
Write nothing and delete nothing until they approve.
Done when the user has approved the plan.
## 6. Write the notes
Write each new note and update each changed one under `01 sources/claude/`, using the vault's note template (`99 meta/templates/note.md`): fill its `Tags:` line with the `[[hub]]` wikilinks the note belongs under, and its body with the distilled knowledge.
Link a hub whether or not its note exists yet — a dangling `[[hub]]` is a valid link and still feeds `/wiki` recall.
Organize files by your own judgment: flat until a category has accumulated enough notes to warrant its own subfolder under `claude/`.
Done when every note in the plan has been written or updated.
## 7. Clear the consumed scaffolding
With the notes written, perform the branch's cleanup — the branch file names exactly what to remove.
Deletion never precedes the writes of step 6.
Done when the branch's cleanup has run.
## 8. Report
Tell the user what you created, updated, and left untouched, and repeat the suggested new hubs from the report so the vault's `02 tags/` and `03 index/` can be filled in by hand.
Done when the summary names every note written or updated and every suggested hub.

View File

@@ -1,23 +0,0 @@
# project branch
How consume reads a project target, and what it clears afterward.
Reached from step 1 of [`SKILL.md`](SKILL.md) when the target is a directory containing `.claude/spec/` and/or `.claude/tasks/`.
## Read the source, mine the specs
The source code is the truth for *what* the project is and does now.
Read it thoroughly enough to understand its current state and to surface the reusable ideas it embodies — techniques, patterns, decisions, gotchas.
`.claude/spec/` and `.claude/tasks/` are the record of *why* — the reasoning, trade-offs, and intent behind what the code became.
They are the richest source of the generalizable lessons, and cleanup destroys them, so mine their reasoning now or lose it.
But they are not current fact: where a spec or task disagrees with the source, the source wins, and where one describes work later abandoned or changed, the source is what actually happened.
Together these are the raw material the writing logic generalizes; extracting the generalized notes themselves is step 4 of SKILL.md, not this branch's job.
Done when you understand the project's current state from its source and have mined its specs and tasks for the reasoning behind it.
## Cleanup
After the notes are written (step 6 of SKILL.md), delete the spec and task files that were present under `.claude/spec/` and `.claude/tasks/` when you read the project in step 2 — the scaffolding this run consumed.
A file added after that read is not swept up.
Leave the now-empty `.claude/spec/` and `.claude/tasks/` directories in place, along with everything else under `.claude/`.

View File

@@ -0,0 +1,304 @@
# Glossary — Structural Predictability
The disclosed domain model for [`design-skill`](SKILL.md).
Every term below is a lever on **Predictability**.
The terms are grouped by axis: **Invocation** (how a skill is reached), **Information Hierarchy** (how its content is arranged), **Steering** (how the agent's runtime behaviour is shaped), and **Pruning** (how it is kept lean).
Each **failure mode** lives beside the lever that cures it, tagged _failure mode_.
**Bold terms** in any definition are themselves defined in this glossary.
Find them by their heading.
## Predictability
The degree to which a skill makes the agent behave the same _way_ on every run — the same process, not the same output.
A brainstorming skill should _predictably_ diverge because its tokens vary while its behaviour does not.
The root virtue every other term serves — cost and maintainability are symptoms of it, not rivals.
_Avoid_: consistency, reliability, robustness, output-determinism
## Invocation
How a skill is reached — and the two loads you pay for the choice.
### Model-Invoked
A skill exposed to the model's available-skills index so the agent can discover and load it autonomously.
The human can still invoke it directly, so model-invocation always includes user reach.
It pays permanent **context load** in exchange for discoverability.
A model-invoked skill whose content is all **reference** can provide shared reference to other discoverable skills.
Pick model-invocation only when the agent must reach the skill on its own.
_Avoid_: ability, tool, capability
### User-Invoked
A skill hidden from autonomous model discovery and reached deliberately by the human through the target harness's invocation mechanism.
It trades agent discoverability for zero **context load**.
Agent Skills still require a **description**, so use a documented harness control such as `disable-model-invocation` rather than deleting required metadata.
If the target harness cannot hide a skill from model discovery, report that user-only invocation is unsupported rather than inventing a mechanism.
_Avoid_: procedure, workflow, command
### Description
Required Agent Skills metadata stating what the skill does and when it applies.
For a **model-invoked** skill, it is the top-level **context pointer** kept in the model's available-skills index and the source of that skill's **context load**.
For a **user-invoked** skill, it remains valid metadata but the harness keeps it out of the model's index.
_Avoid_: frontmatter, summary
### Context Pointer
A reference held in the agent's context that names some out-of-context material and encodes the condition for reaching it.
The **description** is the top-level context pointer from the context window to the skill.
Pointers to disclosed files are the same object one level down.
Its wording, not the target, decides _when_ the agent reaches — and _how reliably_.
A must-have target behind a weakly worded pointer is a variance bug: fix the wording first, and inline the material only if sharpening fails.
_Avoid_: link, reference, import
### Context Load
The cost a **model-invoked** skill imposes on the agent's context window through its indexed **description**, spending both tokens and attention.
A **user-invoked** skill escapes this load because the harness does not expose its description to the model.
Context load is the brake on splitting behavior into more model-invoked skills.
_Avoid_: token cost, context bloat
### Cognitive Load
The cost a **user-invoked** skill imposes on the human — what they must hold in their head: which skills exist and when to reach for each (the human is the index).
What **model-invocation** removes by being agent-discoverable, and the brake on splitting into more user-invoked skills.
It is not a cost to minimise.
It is the price of human agency and the reason some skills stay user-invoked.
Spend it where human judgement matters.
Remove it where it does not.
_Avoid_: human index, burden, overhead
### Router Skill
A **user-invoked** skill whose job is to name other user-invoked skills and explain when the human should reach for each one.
It gives the human one entry point to remember instead of many.
It can only direct the human toward skills hidden from model discovery, not make those skills autonomously discoverable.
It is the cure for **cognitive load** when user-invoked skills multiply.
_Avoid_: dispatcher, menu, registry, index, router procedure
### Granularity
How finely you divide skills.
Finer division spends one of the two loads.
More **model-invoked** skills spend **context load** as more descriptions crowd the window and compete for attention.
More **user-invoked** skills spend **cognitive load** by giving the human more to remember and reach for.
Two cuts guide the division.
By **invocation**, split off a model-invoked skill where you have a distinct **leading word** to trigger it — a trigger word you actually use in your prompts.
By **sequence**, split a run of **steps** only when the hand-off creates a real context boundary that hides a step's **post-completion steps**.
Beware the reverse.
Merging sequences exposes each step's post-completion steps to what follows, inviting premature completion.
_Avoid_: chunking, modularity
## Information Hierarchy
How a skill's content is arranged, and how far down the ladder each piece sits.
### Information Hierarchy
A skill's content ranked by how immediately the agent needs it — a single ladder, produced by two cuts: in-file or behind a pointer, and step or reference.
The rungs:
- **Steps** — in-file, primary
- **Reference**, in-file — secondary
- **Reference**, disclosed — behind a **context pointer**
A skill with no **steps** uses just the bottom two rungs — often a legitimately flat peer-set (e.g. every rule of a review on one rung), which is a fine arrangement, not a smell.
The hierarchy is independent of invocation: a skill can be model- or user-invoked whether it is all steps, all reference, or both.
When a skill has steps, in-file reference that should be disclosed buries them and turns attending to them into a coin-flip — a variance lever, not just a legibility one.
Keep the top of the ladder legible.
Push down it whatever you can.
_Avoid_: structure, organization, layout
### Steps
The ordered actions the agent performs — when a skill has them, the primary tier of its content, and the part that earns its place in SKILL.md.
Not every skill has steps: a skill can be all steps (`test-driven-development`), all **reference** (a review), or both, independent of invocation.
Every step ends on a **completion criterion**, clear or vague.
_Avoid_: workflow, instructions, choreography
### Reference
Material the agent refers to on demand — definitions, facts, parameters, examples, conditional instructions.
When a skill has **steps**, reference is secondary to them.
When a skill has none, reference is the entire content.
Reference can also live outside any skill entirely under **External Reference**.
Reached via **context pointers**, and the prime candidate for **progressive disclosure**.
_Avoid_: supporting material, docs, background
### External Reference
**Reference** that lives outside the skill system as a plain, non-invocable file with no **description** or **steps**.
Any skill can point at it.
It is the home for shared reference that need not fire on its own and can be used by skills hidden from one another's discovery surface.
_Avoid_: doc, resource, knowledge base
### Progressive Disclosure
Moving **reference** down the ladder — out of SKILL.md and behind a **context pointer** — so the top stays legible.
It is not primarily a token optimisation.
It is how the **information hierarchy** is protected.
Licensed by **branching**: disclose what only some branches need, inline what every path needs, and if a pointer fires unreliably on must-have material, sharpen its wording, and pull it back inline only if that fails.
_Avoid_: lazy loading, chunking
### Co-location
Keeping the material an agent needs at once in one place — a concept's definition, rules, and caveats under a single heading, not scattered across the file — so reading one part brings its neighbours with it.
The within-file companion to the **Information Hierarchy**.
The hierarchy ranks _how far down_ a piece sits, while co-location decides _what sits beside it_ once there.
There is no formula for the right format of a body of **reference**.
The test is that a skill should read like documentation written for the agent, and grouped material reads that way where scattered material does not.
Distinct from **Duplication**: that repeats one meaning in two places, where scattering fragments a single meaning across many.
_Avoid_: grouping, clustering, cohesion
### Sprawl
_Failure mode._ A skill that is simply too long — too many lines in SKILL.md — independent of whether they are stale or repeated.
Even an all-live, all-unique skill can sprawl.
It costs readability (the agent wades through more before it can act, and attention thins across the excess), maintainability (every extra line is one more to keep **relevant**), and tokens.
The cure is the **information hierarchy**: push **reference** down behind **context pointers**, and split by **branch** or sequence so each path carries only what it needs.
Distinct from **sediment** (length from stale accumulation) and **duplication** (length from repeated meaning) — sprawl is length itself, whatever its cause.
_Avoid_: bloat, length, size, verbosity
## Steering
The levers that shape the agent's runtime behaviour toward **Predictability**.
### Branch
A distinct way a skill can be invoked — a case the skill handles — so different runs take different paths through it.
A skill with many steps may carry many branches.
A linear one has none.
_Avoid_: path, case, fork
### Leading Word
A compact concept — also called a _Leitwort_ — already living in the model's pretraining, that the agent thinks with while running the skill.
It encodes a behavioural principle in the fewest possible tokens by invoking priors the model already holds, such as _lesson_, _proximal zone of development_, _fog of war_, or _tracer bullets_.
Repeated as a token, never as a sentence, it accumulates a distributed definition across the skill and anchors a whole region of behaviour.
Coining your own works if you define it clearly, but a made-up word recruits no priors — you pay in definition tokens what a pretrained word gives free.
Reach for an existing word first.
A leading word serves **predictability** twice.
In the body it anchors **execution** — the agent reaches for the same behaviour every time the concept appears, and inside flat reference it focuses attention on a class of thing to look for, recruiting the right checks each run.
For a **model-invoked** skill, the **description** anchors **invocation**.
When the same word lives in prompts, documentation, and code, the agent links that shared language to the skill and fires it more reliably.
Word a model-invoked skill's description with the leading words actually used to request its behavior.
_Avoid_: keyword, term, motif
### Completion Criterion
The condition that tells the agent a unit of work is done — the target it judges against.
Two properties make it a lever, not just a quality.
Its **clarity** asks whether the agent can tell done from not done.
Clarity resists **premature completion** because a vague bound such as "understanding reached" lets the agent declare done and slip to the next step.
This axis needs _steps_ to bite because premature completion is a between-steps failure.
Its **demand** (how much it requires) sets **legwork** — "every modified model accounted for" forces thorough work where "produce a change list" does not — and this axis is _not_ step-bound: it can bind a body of flat reference too, which is how a skill with no steps still carries an exhaustiveness bar ("every rule applied").
The strongest criteria are both checkable and exhaustive.
_Avoid_: done condition, exit condition, stopping rule
### Legwork
The work an agent does behind the scenes within a single step — reading files, exploring the codebase, making changes, digging up what it needs rather than offloading to the user.
It lives below the step structure: never written as its own step, latent in the wording, controlled by the agent rather than the skill.
The within-step counterpart to **post-completion steps**' across-step pull.
Raised by a strong **leading word** or a **completion criterion** that demands exhaustive work.
The demand axis also applies to flat reference and drives a reference-only skill to cover every required check.
Goes thin either when that demand is missing or when **premature completion** cuts the step short.
_Avoid_: scope, effort, diligence, coverage
### Post-Completion Steps
The **steps** that follow the current step.
Visible post-completion steps pull the agent forward into **premature completion**.
The more it sees, the stronger the tug.
The defence is to hide them by splitting the sequence of steps into two.
_Avoid_: horizon, fog of war, lookahead
### Premature Completion
_Failure mode._ Ending the current step before it is genuinely done, because the agent's attention slips to being done rather than to the work.
A between-steps failure: it needs **steps** to occur — a skill with no steps that quits early isn't premature completion but thin **legwork** under an unmet demand.
It is a tug-of-war between visible **post-completion steps**, which pull forward, and the **completion criterion**'s clarity, which resists that pull.
A sharp, checkable bar holds while a vague one gives way.
Fuzziness is the necessary condition: a sharp bound resists the pull no matter how many later steps are visible, so a step that never rushes needs no defending.
Two levers hold a step that does, but reach for them in order: **sharpen the bound first** — it is local and cheap.
Only when the criterion is irreducibly fuzzy _and_ you actually observe the rush do you **hide the later steps**.
Hiding works only across a real context boundary such as a user-invoked hand-off or a subagent dispatch.
An inline model-invoked call leaves the later steps in context and clears nothing.
One cause of thin legwork, but distinct from it: legwork can be thin even when a step runs to full completion.
_Avoid_: premature closure, the rush, rushing, shortcutting
## Pruning
Keeping a skill lean — each remedy paired with the failure it cures.
### Single Source of Truth
The desired state where each meaning lives in exactly one authoritative place, so a change to the skill's behaviour is a change in one place.
**Duplication** is its violation.
_Avoid_: home, canonical location
### Duplication
_Failure mode._ The same meaning given more than one **single source of truth**.
It costs maintenance (change one place, you must change the others), costs tokens, and inflates prominence — repeating a meaning weights it on the ladder past its real rank.
The accidental inverse of a **leading word**, which raises attention on purpose by repeating a token, never the meaning.
_Avoid_: repetition, redundancy
### Relevance
Whether an instruction still bears on what the skill does — the lens for what to keep.
An instruction loses relevance either by never bearing on the task or by going stale as the behavior or world it describes changes.
Shorter skills are easier to keep relevant because each instruction is cheaper to check.
Distinct from **no-op**: relevance asks whether an instruction bears on the task, not whether it changes behavior.
_Avoid_: load-bearing, staleness, freshness
### Sediment
_Failure mode._ Layers of old content that settle in a skill and are never cleared, because adding feels safe and removing feels risky — so stale and irrelevant lines accumulate and you must core down through them to find what is still live.
This is the default fate of any skill without a pruning discipline.
It is the slow erosion of **relevance**, as opposed to **duplication**'s repeated meaning.
_Avoid_: accretion, bloat, cruft, rot
### No-Op
_Failure mode._ An instruction that changes nothing because the model already does it by default — you pay load to tell the agent what it would do anyway.
The test asks whether an instruction changes behavior versus the default.
An instruction can be perfectly **relevant** and still be a no-op.
The same priors that make a **leading word** free make a no-op worthless.
A leading word is a _technique_, while No-Op is a _verdict_ on an instruction.
The two concepts cross.
A leading word too weak to beat the default is a no-op (_be thorough_ when the agent is already thorough-ish), and the fix is a stronger word that passes the verdict (_relentless_), not a different technique.
So the No-Op test — does it change behaviour versus the default? — is also how you grade whether a leading word is earning its repetitions.
This is model-relative, not reader-relative: two people disagreeing over whether a line is a no-op disagree about the default, and settle it by running the skill, not by debate.
_Avoid_: redundant instruction, restating the obvious, belaboring

View File

@@ -0,0 +1,144 @@
---
name: design-skill
description: Draft a new Agent Skill, or audit and rewrite an existing one for structural predictability. Run deliberately as /skill:design-skill when designing or improving a skill, never automatically.
disable-model-invocation: true
---
# design-skill
Judge only structural **predictability** using the terms in [`GLOSSARY.md`](GLOSSARY.md).
Do not expand the work into general prose, safety, or technical review.
Read each relevant definition before applying it.
## Select the branch
Use **Draft a new skill** when the request describes a new capability or repeated process with no existing skill to modify.
Use **Audit an existing skill** when the request names an existing skill to review, fix, or improve.
Both branches end at **Verify**.
## Draft a new skill
### 1. Capture intent
Extract every answer already settled by the conversation or target project before asking anything.
Ask one question at a time only for missing decisions:
- Is the skill a **procedure**, **reference**, or both?
- Should it be model-invoked or user-invoked?
Walk the **context load** and **cognitive load** trade-off rather than defaulting.
- What distinct **branches** take genuinely different paths?
- Does project documentation already own any required material?
Point to that **external reference** instead of duplicating it.
- What checkable **completion criterion** ends each step?
- Does the project already use a **leading word** for the behavior?
Done when every item is settled or the user explicitly asks for a draft to iterate on.
### 2. Resolve the canonical source
Read the target project's instructions and inspect its repository structure before choosing a destination.
Prefer an existing Agent Skills source convention.
For a project-specific skill, use the project's canonical project-local skill source.
For a general skill, use the personal collection named by the user or project instructions.
Do not edit generated, installed, vendored, or compatibility surfaces as source.
When the discovered skill is a link or generated placement, trace it to its canonical source.
If multiple plausible sources remain, ask which one owns the skill.
Done when the destination is writable, authoritative, and inside the intended project or collection.
### 3. Write the skill
Use the Agent Skills standard as the baseline:
- Put the skill in a directory containing `SKILL.md`.
- Give the frontmatter a valid `name` and a specific `description`.
- Use documented harness extensions only when the target environment supports them.
- Keep paths relative to the skill directory.
Follow the **information hierarchy**.
Keep ordered **steps** in `SKILL.md`, keep reference needed by every branch nearby, and disclose branch-specific or extensive reference through precise pointers.
Reuse authoritative project documentation rather than restating it.
Every branch must have a complete path, every step must have a checkable completion criterion, and every sentence must pass the **no-op** test.
Done when the skill conforms to the target project's conventions and every captured decision has one authoritative expression.
## Audit an existing skill
### 1. Locate and read it
Read the target project's instructions and locate the named skill's canonical source.
Trace generated or linked placements rather than editing them.
Ask only when multiple plausible sources remain.
Read `SKILL.md`, every disclosed file, and any project documentation it treats as authoritative.
Done when the full design and every source-of-truth boundary are visible.
### 2. Apply the predictability checklist
Check every relevant definition in `GLOSSARY.md` and quote the exact offending passage for each finding:
- **Premature completion**: does each step end on a checkable criterion demanding all required work?
- **Duplication**: does any meaning have more than one source of truth?
- **Sediment**: does any sentence no longer bear on the skill's behavior?
- **Sprawl**: should reference be disclosed or a genuine branch be separated?
- **No-op**: does each sentence change model behavior from the default?
- **Invocation**: does the chosen invocation mode fit actual use and target-harness support?
- **Leading word**: can repeated explanation collapse into established vocabulary?
Judge sentence by sentence rather than line by line.
A line may contain both a load-bearing sentence and a no-op sentence.
Done when every glossary failure mode has been checked and every finding has quoted evidence.
### 3. Rewrite it
Rewrite the canonical source to address every finding.
Preserve intentional behavior that passed the checklist.
Follow target-project instructions for any change outside the skill directory.
Done when every finding is fixed or explicitly retained with a structural reason.
## Verify
### 1. Run deterministic checks
Run the target project's required checks and any available Agent Skills validator.
At minimum, verify frontmatter, naming, relative references, disclosed-file existence, and invocation compatibility with the target harness.
Done when every available deterministic check passes and every unavailable check is reported.
### 2. Smoke-test behavior
Derive one realistic task prompt from the skill's intended trigger phrasing.
Launch a fresh context through an actual subagent tool or documented headless harness command, load the skill there, and run the prompt.
The fresh context must attempt the target skill's task rather than design or audit that skill.
Save the exact prompt and fresh-context output in temporary files outside the target project.
Inspect that output for whether the agent followed the intended branch, hesitated, misread an instruction, or behaved unexpectedly.
Do not count the current design run, a self-review, or deterministic validation as behavioral evidence.
A smoke test does not require advance approval.
Done when the final report names the fresh-context mechanism and both evidence paths, and the captured output shows no structural confusion.
If no isolated execution mechanism exists, report that limitation instead.
### 3. Offer stronger verification when available
Detect whether the target project has an established behavioral test or benchmark convention for skills.
Use existing persistent tests when present.
Create a persistent test only when project instructions require one.
Before running a stochastic, repeated, paid, or otherwise substantial benchmark, show the user the proposed scenario and obtain approval.
If approval is declined, retain the smoke-test result as the weaker evidence level.
Done when an approved benchmark passes, the user declines it, or no established benchmark exists.
### 4. Re-audit and finish
Re-read the final skill against every applicable `GLOSSARY.md` failure mode in light of verification.
Fix every new finding and repeat affected checks after a substantial fix.
Follow explicit user or project instructions for staging or other version-control actions.
Otherwise leave changes unstaged and report every changed path.
Never commit unless explicitly instructed.
Done when no structural finding remains, required checks pass, behavioral confidence is stated accurately, and the applicable version-control policy was followed.

View File

@@ -0,0 +1,30 @@
# Context Artifact Format
## Structure
```md
# {Context Name}
{One or two sentence description of what this context is and why it exists.}
## Language
**Order**:
{A one or two sentence description of the term}
_Avoid_: Purchase, transaction
**Invoice**:
A request for payment sent to a customer after delivery.
_Avoid_: Bill, payment request
**Customer**:
A person or organization that places orders.
_Avoid_: Client, buyer, account
```
## Rules
- **Be opinionated.** When multiple words exist for the same concept, pick the best one and list the others under `_Avoid_`.
- **Keep definitions tight.** One or two sentences max. Define what it IS, not what it does.
- **Only include terms specific to this project's context.** General programming concepts (timeouts, error types, utility patterns) don't belong even if the project uses them extensively. Before adding a term, ask: is this a concept unique to this context, or a general programming concept? Only the former belongs.
- **Group terms under subheadings** when natural clusters emerge. If all terms belong to a single cohesive area, a flat list is fine.

94
skills/grill/SKILL.md Normal file
View File

@@ -0,0 +1,94 @@
---
name: grill
description: Interview the user relentlessly to stress-test a plan or design before building — one question at a time, each with options, a recommendation, and its rationale — walking the decision tree until every branch is decided. Folds in domain-model and ADR upkeep when the project has a context file. Model-invokable, including as a step inside other skills. Use when the user wants to pressure-test a plan, or uses any 'grill' trigger phrase.
---
# grill
Interview the user relentlessly about every aspect of a plan or design.
Keep going not until it feels like you understand each other, but until the decision tree is walked to exhaustion.
Ask one question at a time, in plain text, and wait for the answer before the next.
Never use an interactive question tool.
While the interview is active, every assistant turn must end with exactly one of: the next numbered question, the final decision summary, or a plain statement that grill is paused because the user explicitly changed tasks.
If the turn includes a correction, file edit, apology, or explanation, follow it with the next numbered question unless the user explicitly paused grill or changed tasks.
Do not use dangling transition phrases such as “continuing” unless the continuation is actually present in the same message.
Every question carries **options** — the concrete choices, each with its trade-offs — followed by your **recommendation**, the option you would pick and why.
When the question turns on an existing document, artifact, or piece of code, open it with a **context quote**: blockquote the exact passage, so the user sees what you mean without hunting for it.
Before asking, investigate.
When code, docs, or the conversation already settle a question, resolve it yourself instead of spending a question on it.
**Never begin implementation** — during the interview or after it — without an explicit instruction from the user.
This holds even when another skill invoked grill: grill delivers its summary and stops, and a caller that receives that summary is not itself the instruction to build.
## 1. Select the mode
Resolve the artifact destination before looking for or creating artifacts.
Default to `$(xdg-user-dir DOCUMENTS)/ai-artifacts/projects/<project>` when the AI artifacts vault is available, where `<project>` is the lowercase basename of the current working directory.
If that vault is unavailable, fall back to `./docs` in the current project.
Create the destination directory only when a write requires it.
Read the destination's `AGENTS.md` before any artifact write.
Inspect the destination directly for its context artifact, using the destination naming convention instead of assuming a filename.
- **Domain Modeling Mode** — exactly one context artifact was found.
Run the interview and maintain the project's domain model as terms settle (see [Domain Modeling Mode](#domain-modeling-mode)).
- **Free Mode** — no context artifact was found, and the plan raises no project-specific vocabulary worth pinning down.
Run the interview with no document side effects.
- **Clarification** — no context artifact was found, but the plan introduces terms specific to this project that later work will need to use consistently — the kind of terms [`CONTEXT-FORMAT.md`](CONTEXT-FORMAT.md) admits, not general programming concepts.
Before interviewing, ask whether to create a glossary.
If yes, choose the context artifact filename from the destination's `AGENTS.md` and nearby artifact convention, allocate any required identifier, and continue in Domain Modeling Mode.
If no, continue in Free Mode.
Stop and report the conflicting paths if more than one matching context artifact exists.
Done when the interview is running in Domain Modeling Mode or Free Mode.
## 2. Run the interview
Walk the decision tree one branch at a time, resolving each decision's dependencies before moving to the next.
Alongside the questions, challenge and stress-test:
- When the user's language is vague or overloaded, propose a precise canonical term in its place.
- Probe edge cases with concrete scenarios that force the boundaries between concepts to be made explicit.
- When a stated behavior contradicts what the code actually does, surface the contradiction with both sides quoted.
In Domain Modeling Mode, feed the settling terms and decisions into the glossary and ADRs below.
Done when every branch carries an explicit decision and no dependency between decisions is left open.
## 3. Close the interview
Once the interview is done:
1. Output a concise summary of every decision reached — and, in Domain Modeling Mode, every term crystallized.
2. Stop and wait for the user's next instruction.
Done when the summary is delivered and grill is waiting.
## Domain Modeling Mode
Active only when step 1 selected this mode.
### Glossary
As a term crystallizes, update the context artifact right then — do not batch these to the end.
When a settling term clashes with one already in the glossary, call it out and reconcile to a single canonical word.
Keep the artifact a glossary and nothing else: vocabulary and ubiquitous language, no implementation detail.
Write it in the format of [`CONTEXT-FORMAT.md`](CONTEXT-FORMAT.md).
### ADRs
Offer to record an architectural decision only when all three hold:
1. **Hard to reverse** — changing your mind later carries real cost.
2. **Surprising without context** — a future reader will wonder why it was done this way.
3. **The result of a real trade-off** — there were genuine alternatives and you picked one for specific reasons.
If any of the three is missing, skip it.
A recorded decision lives directly in the resolved artifact destination.
Choose its filename from the destination's `AGENTS.md` and nearby artifact convention, allocating an identifier only when that convention requires one.
It can be a single paragraph:
> # {Short title of the decision}
>
> {1-3 sentences: the context, what was decided, and why.}

122
skills/implement/SKILL.md Normal file
View File

@@ -0,0 +1,122 @@
---
name: implement
description: Implement scoped work using the local implementation workflow, following Wayfinder ticket protocol when an implementation ticket artifact is part of the selected work.
disable-model-invocation: true
---
# implement
Implement scoped work and hand off working code.
This skill defines execution protocol rather than deciding what work the agent is allowed to implement.
When selected work has a Wayfinder ticket artifact, follow the Wayfinder artifact lifecycle.
When selected work has no Wayfinder artifact, implement normally and use the same review, commit, and handoff discipline without creating a tracking artifact.
This skill delegates to other skills by name.
Before satisfying a delegated skill step, explicitly load that skill's `SKILL.md` with `read` unless the harness has already injected that skill's full content into the current context.
Do not satisfy a delegated skill step by imitating its title from memory.
## 1. Establish the selected work
Identify the work to implement from the user's request and current context.
Use a Wayfinder implementation ticket artifact, issue, spec, direct request, branch context, or other clear source when that is what the agent has selected.
Do not require a Wayfinder artifact and do not create one just to satisfy this skill.
Old `.claude/tasks` files may be ordinary context, but their lifecycle semantics are not part of this workflow.
When the selected work has a Wayfinder ticket artifact, re-read it before claiming.
Verify that blockers are satisfied by the artifact metadata and that no other session has claimed it.
Claim it before changing code by setting `status: claimed`, `claimed-by` to the current session identifier, and `claimed-at` to the current timestamp.
Use `PI_SESSION_ID` when available.
Done when the selected work is clear, and any Wayfinder ticket artifact is freshly read and claimed when applicable.
## 2. Verify the execution checkout
Assume the current checkout is the assigned execution checkout.
Do not create, lease, clean up, or switch worktrees.
Report the checkout path and branch when useful for handoff.
Check the working tree before changing code.
Stop on unrelated or ambiguous uncommitted changes.
Continue only when dirty state is clearly already part of the selected work.
Never auto-stash.
For source-changing implementation work that will be committed, work on a non-default branch unless repository context explicitly directs otherwise.
If an appropriate non-default branch is already prepared, continue there.
If you are on the default branch, fast-forward the default branch first and create a task branch.
Stop if the default branch cannot fast-forward cleanly.
Follow repository, user, or orchestrator branch naming conventions.
If no convention is discoverable and you must create a branch, choose a clear short descriptive name and report it.
Check declared implementation blockers where they exist.
Stop when prerequisite implementation work is not reachable from the current base, and report the likely unmerged prerequisite.
Do not automatically branch from, merge, or cherry-pick sibling task work.
Follow an explicit integration-branch or wide-refactor plan only when the selected work names that exception.
Done when the checkout, branch, dirty state, and reachable prerequisites are safe for the selected implementation work.
## 3. Implement the work
Use `test-driven-development` as a strong default when behavior can usefully be specified and tested before implementation.
Load `test-driven-development` before deciding whether it applies.
Skip it only when the work is mechanical, documentation-only, exploratory, or when test-first would not add value.
When skipping it, state the reason before implementation.
Skipping test-driven development is an agent judgment, not an omission.
Build the selected work in the assigned checkout.
Run focused verification while working and run broader verification when the repository or change calls for it.
Treat working code and tests as the implementation artifact.
Do not create a separate process report for test-driven development.
Record only durable outcomes such as tests added or changed, verification run, and deviations worth noting.
Stage created and modified files intentionally.
Do not use broad staging that sweeps unrelated files into the change.
Done when the selected work is implemented as working code and locally verified to the level the change warrants.
## 4. Review the final intended handoff state
Load and run `review` before reporting final handoff or resolving a Wayfinder ticket.
Choose review timing by judgment.
Review may happen before commit, after commit, or both.
The review must cover the final intended handoff state, including a local uncommitted handoff when no commit is made.
Do not report final handoff until review has run or you have explained why review could not cover that state.
Fix blocking review findings unless the agent or user explicitly accepts them.
When code changes are made in response to review, rerun `review` on the relevant change set.
Record accepted blocking findings in the final handoff or Wayfinder closeout when they affect a future reader's decision to trust or continue the work.
Handle non-blocking findings by judgment.
Fix them when cheap or high-value, and otherwise report or record only when useful.
Done when review has covered the final intended handoff state and blocking findings are fixed or explicitly accepted.
## 5. Commit and hand off
Follow the repository's commit and PR conventions.
Discover those conventions from repository instructions or recent history rather than assuming them.
Commit only the selected work.
Use one commit when that is the natural shape, but follow the repository's convention when it expects a different history shape.
Push and open a PR when the repository and selected workflow call for that handoff.
If there is no remote or no supported forge workflow, stop after the local handoff point and report what remains for a human.
Keep PR and handoff text focused on what was built, verification, review disposition, and any deviations that matter.
Done when the implementation has a concrete handoff state: committed locally, pushed, opened as a PR, or stopped with the exact remaining human action reported.
## 6. Close out Wayfinder artifacts when present
When the selected work has a Wayfinder ticket artifact, re-read it before closeout.
Record durable coordination facts only.
Wayfinder `status` is the lifecycle authority.
Acceptance criteria checkboxes may be updated as useful detail with `[x]` for satisfied criteria and `[-]` for deliberately dropped criteria when practical.
Checkbox state is not the source of truth.
Keep Implementation Notes concise.
Capture only what future agents or humans need: deviations, dropped or changed scope, verification, branch, commit, PR, or why the ticket remains unresolved.
Use judgment for partial or ambiguous outcomes, but keep lifecycle recording honest.
Do not imply completion when the work is not complete.
When resolving a Wayfinder ticket, set `status: resolved` and repair the owning map's Frontier according to Wayfinder artifact rules.
Put detailed implementation records in the ticket, commit, PR, and working code rather than in the map.
The parent map should receive only the concise outcome summary required by Wayfinder.
Done when any Wayfinder ticket and parent map agree with the implementation's actual lifecycle state.

100
skills/prototype/LOGIC.md Normal file
View File

@@ -0,0 +1,100 @@
# Logic Prototype
A tiny interactive terminal app that lets the user drive a state model by hand. Use this when the question is about **business logic, state transitions, or data shape** — the kind of thing that looks reasonable on paper but only feels wrong once you push it through real cases.
## When this is the right shape
- "I'm not sure if this state machine handles the edge case where X then Y."
- "Does this data model actually let me represent the case where..."
- "I want to feel out what the API should look like before writing it."
- Anything where the user wants to **press buttons and watch state change**.
If the question is "what should this look like" — wrong branch. Use [UI.md](UI.md).
## Process
### 1. State the question
Before writing code, write down what state model and what question you're prototyping.
Use one paragraph in the prototype's README or a comment at the top of the file.
A logic prototype that answers the wrong question is pure waste, so make the question explicit enough to check later whether the user is watching now or returning to it AFK.
Done when the prototype states one concrete logic question and the model being tested.
### 2. Pick the language
Use whatever the host project uses. If the project has no obvious runtime (e.g. a docs repo), ask.
Match the project's existing conventions for tooling.
Don't add a new package manager or runtime just for the prototype.
Done when the prototype has a runnable host-project language and toolchain without introducing a new runtime convention.
### 3. Isolate the logic in a portable module
Put the actual logic — the bit that's answering the question — behind a small, pure interface that could be lifted out and dropped into the real codebase later. The TUI around it is throwaway; the logic module shouldn't be.
The right shape depends on the question:
- **A pure reducer** — `(state, action) => state`. Good when actions are discrete events and state is a single value.
- **A state machine** — explicit states and transitions. Good when "which actions are even legal right now" is part of the question.
- **A small set of pure functions** over a plain data type. Good when there's no implicit current state — just transformations.
- **A class or module with a clear method surface** when the logic genuinely owns ongoing internal state.
Pick whichever shape best fits the question being asked, *not* whichever is easiest to wire to a TUI. Keep it pure: no I/O, no terminal code, no `console.log` for control flow. The TUI imports it and calls into it; nothing flows the other direction.
This is what makes the prototype useful past its own lifetime.
When the question is answered, the validated reducer, machine, or function set can be lifted into the real module on its own.
Done when all tested logic lives behind one portable, pure interface and the TUI depends on it in only one direction.
### 4. Build the smallest TUI that exposes the state
Build it as a **lightweight TUI** — on every tick, clear the screen (`console.clear()` / `print("\033[2J\033[H")` / equivalent) and re-render the whole frame. The user should always see one stable view, not an ever-growing scrollback.
Each frame has two parts, in this order:
1. **Current state**, pretty-printed and diff-friendly (one field per line, or formatted JSON). Use **bold** for field names or section headers and **dim** for less important context (timestamps, IDs, derived values). Native ANSI escape codes are fine — `\x1b[1m` bold, `\x1b[2m` dim, `\x1b[0m` reset. No need to pull in a styling library unless one is already in the project.
2. **Keyboard shortcuts**, listed at the bottom: `[a] add user [d] delete user [t] tick clock [q] quit`. Bold the key, dim the description, or vice-versa — whatever reads cleanly.
Behaviour:
1. **Initialise state** — a single in-memory object/struct. Render the first frame on start.
2. **Read one keystroke (or one line)** at a time, dispatch to a handler that mutates state.
3. **Re-render** the full frame after every action — don't append, replace.
4. **Loop until quit.**
The whole frame should fit on one screen.
Done when every available action re-renders a complete one-screen view of the current state and shortcuts.
### 5. Make it runnable in one command
Add a script to the project's existing task runner (`package.json` scripts, `Makefile`, `justfile`, `pyproject.toml`). The user should run `pnpm run <prototype-name>` or equivalent — never need to remember a path.
If the host project has no task runner, put the command at the top of the prototype's README.
Done when a fresh user can launch the prototype with one documented command.
### 6. Evaluate it
In HITL mode, give the user the run command.
They drive it themselves.
The interesting moments are when they say "wait, that shouldn't be possible" or "huh, I assumed X would be different" because those expose bugs in the idea.
Add actions when the feedback needs them.
In AFK mode, run the command yourself and drive the hard cases that answer the stated question.
Record the observations that support the verdict.
Done when the evaluator can exercise the model and the prototype exposes every state transition needed to reach a verdict.
## Production mapping
When the shared [SKILL](SKILL.md) permits production work, lift the validated reducer, machine, or function set into the real module.
Keep the TUI shell on the throwaway branch.
## Anti-patterns
- **Don't generalise.** No "what if we wanted to support X later." The prototype answers one question.
- **Don't blur the logic and the TUI together.** If the reducer / state machine references `console.log`, prompts, or terminal escape codes, it's no longer portable. Keep the TUI as a thin shell over a pure module.
- **Don't ship the TUI shell into production.** The shell is optimised for being driven by hand from a terminal. The logic module behind it is the bit worth keeping.

88
skills/prototype/SKILL.md Normal file
View File

@@ -0,0 +1,88 @@
---
name: prototype
description: Build a throwaway prototype to answer a design question. Use when the user wants to sanity-check whether a state model or logic feels right, or explore what a UI should look like.
---
# Prototype
A prototype is **throwaway code that answers one question**.
The question decides the branch.
The interaction mode decides who reaches the verdict.
Use AFK mode when the question has objective observable criteria the agent can test end to end.
Use HITL mode when the verdict depends on human judgment, taste, or UX feel.
## 1. Pick a branch
Identify the question from the user's prompt and surrounding code.
Ask when it remains genuinely ambiguous and the user is reachable.
- **Does this logic or state model feel right?**
Follow [`LOGIC.md`](LOGIC.md) to build a tiny interactive terminal app that pushes the model through hard-to-reason-about cases.
- **What should this look like?**
Follow [`UI.md`](UI.md) to build several radically different UI variants on one route with a URL-controlled switcher.
When the user is unavailable, default to Logic for a backend module and UI for a page or component, then state the assumption in the prototype.
Done when exactly one branch and one design question govern the prototype.
## Common rules
- **Throwaway from day one.**
Locate the code close to where it would be used, but name it so nobody mistakes it for production.
Follow the project's routing and source-layout conventions rather than inventing a new top-level structure.
- **One command to run.**
Use the project's existing task runner so the user does not need to remember a path or setup sequence.
- **No persistence by default.**
Keep state in memory unless persistence is the question being tested.
Use an unmistakably disposable database or local file when that question requires one.
- **Skip polish.**
Add no tests, production-grade error handling, speculative abstractions, or unrelated cleanup.
- **Surface state.**
Show the full relevant state after every Logic action or UI variant switch.
## 2. Build and reach a verdict
Follow the selected branch through its evaluation step.
In HITL mode, hand the prototype to the user and iterate in response to their feedback.
In AFK mode, run the prototype yourself and evaluate the objective observations against the question.
Do not treat a runnable prototype as the result.
The result is the verdict that answers the design question.
Done when the verdict is explicit, or when the prototype establishes that it did not resolve the question.
## 3. Capture the primary source
Commit the complete prototype to a throwaway branch outside main.
The branch is the primary source.
Resolve the artifact destination before naming the file.
When the caller provides a destination directory, use it exactly.
Otherwise, default to `$(xdg-user-dir DOCUMENTS)/ai-artifacts/projects/<project>` when the AI artifacts vault is available, where `<project>` is the lowercase basename of the current working directory.
If that vault is unavailable, fall back to `./docs` in the current project.
Create the destination directory only when needed.
Before choosing a filename, read the destination's `AGENTS.md`.
Use the naming convention declared there, or infer it from nearby artifacts when the file delegates naming to local context.
Do not assume a counter, numeric prefix, slug shape, or artifact-type suffix unless the destination convention requires it.
If no convention can be determined, choose the least surprising lowercase descriptive Markdown filename and state that the destination did not define a naming convention.
When the caller provides an existing ticket artifact path, use it exactly, do not allocate a filename, and complete that artifact in place.
Otherwise, allocate any identifier required by the destination convention and include `parent` only when an earlier artifact directly caused the prototype.
The Prototype artifact links the throwaway branch and preserves the question, run instructions, verdict, and branch-appropriate evidence.
When the resolved file is an existing ticket artifact, preserve its workflow metadata and complete the result in that file.
The branch-appropriate evidence is:
- UI evidence uses screenshots.
- Logic evidence uses useful code snippets and, where needed, a short interaction transcript.
Done when the complete prototype is committed outside main and exactly one Prototype artifact preserves the result according to the destination convention.
## 4. Fold in the decision when permitted
A planning-only caller such as Wayfinder stops after the verdict and leaves production code unchanged.
Otherwise, fold the validated decision into production only when the caller permits implementation.
Follow the selected branch's **Production mapping** and keep all other throwaway code out of main.
Done when production is unchanged for a planning-only run, or contains only the permitted validated decision for an implementation run.

124
skills/prototype/UI.md Normal file
View File

@@ -0,0 +1,124 @@
# UI Prototype
Generate **several radically different UI variations** on a single route, switchable from a floating bottom bar. The user flips between variants in the browser, picks one (or steals bits from each), then throws the rest away.
If the question is about logic/state rather than what something looks like — wrong branch. Use [LOGIC.md](LOGIC.md).
## When this is the right shape
- "What should this page look like?"
- "I want to see a few options for this dashboard before committing."
- "Try a different layout for the settings screen."
- Any time the user would otherwise spend a day picking between three vague mockups in their head.
## Two sub-shapes — strongly prefer sub-shape A
A UI prototype is much easier to judge when it's **butting up against the rest of the app** — real header, real sidebar, real data, real density. A throwaway route on its own is a vacuum: every variant looks fine in isolation. Default to sub-shape A whenever there's a plausible existing page to host the variants. Only reach for sub-shape B if the prototype genuinely has no nearby home.
### Sub-shape A — adjustment to an existing page (preferred)
The route already exists. Variants are rendered **on the same route**, gated by a `?variant=` URL search param. The existing data fetching, params, and auth all stay — only the rendering swaps. This is the default; pick it unless there's a specific reason not to.
If the prototype is for something that doesn't yet have a page but *would naturally live inside one* (a new section of the dashboard, a new card on the settings screen, a new step in an existing flow) — that's still sub-shape A. Mount the variants inside the host page.
### Sub-shape B — a new page (last resort)
Only use this when the thing being prototyped genuinely has no existing page to live inside — e.g. an entirely new top-level surface, or a flow that can't be embedded anywhere sensible.
Create a **throwaway route** following whatever routing convention the project already uses — don't invent a new top-level structure. Name it so it's obviously a prototype (e.g. include the word `prototype` in the path or filename). Same `?variant=` pattern.
Before committing to sub-shape B, sanity-check: is there really no existing page this could be embedded in? An empty route hides design problems that a populated one would expose.
In both sub-shapes the floating bottom bar is identical.
## Process
### 1. State the question and pick N
Default to **3 variants**. More than 5 stops being radically different and starts being noise — cap there.
Write down the plan in one line, in the prototype's location or a top-of-file comment:
> "Three variants of the settings page, switchable via `?variant=`, on the existing `/settings` route."
This works whether the user is here to push back or not.
Done when the prototype states one concrete UI question, its host route, and a variant count from three through five.
### 2. Generate radically different variants
Draft each variant. Hold each one to:
- The page's purpose and the data it has access to.
- The project's component library / styling system (TailwindCSS, shadcn, MUI, plain CSS, whatever).
- A clear exported component name, e.g. `VariantA`, `VariantB`, `VariantC`.
Variants must be **structurally different** — different layout, different information hierarchy, different primary affordance, not just different colours. Three slightly-tweaked card grids isn't a UI prototype, it's wallpaper. If two drafts come out too similar, redo one with explicit "do not use a card grid" guidance.
Done when every variant materially differs in layout, information hierarchy, and primary affordance while using the project's existing design system.
### 3. Wire them together
Create a single switcher component on the route:
```tsx
// pseudo-code — adapt to the project's framework
const variant = searchParams.get('variant') ?? 'A';
return (
<>
{variant === 'A' && <VariantA {...data} />}
{variant === 'B' && <VariantB {...data} />}
{variant === 'C' && <VariantC {...data} />}
<PrototypeSwitcher variants={['A','B','C']} current={variant} />
</>
);
```
For sub-shape A (existing page): keep all the existing data fetching above the switcher; only the rendered subtree changes per variant.
For sub-shape B (new page): the throwaway route under `/prototype/<name>` mounts the same switcher.
Done when one route renders every variant from the URL parameter without duplicating data loading.
### 4. Build the floating switcher
A small fixed-position bar at the bottom-centre of the screen with three pieces:
- **Left arrow** — cycles to the previous variant (wraps around).
- **Variant label** — shows the current variant key and, if the variant exports a name, that name too. e.g. `B — Sidebar layout`.
- **Right arrow** — cycles forward (wraps around).
Behaviour:
- Clicking an arrow updates the URL search param (use the framework's router — `router.replace` on Next, `navigate` on React Router, etc) so the variant is shareable and reload-stable.
- Keyboard: `←` and `→` arrow keys also cycle. Don't intercept arrow keys when an `<input>`, `<textarea>`, or `[contenteditable]` is focused.
- Visually distinct from the page (e.g. high-contrast pill, subtle shadow) so it's obviously not part of the design being evaluated.
- Hidden in production builds — gate on `process.env.NODE_ENV !== 'production'` or an equivalent check, so a stray prototype merge can't ship the bar to users.
Put the switcher in a single shared component so both sub-shapes can reuse it. Locate it wherever shared UI lives in the project.
Done when mouse and keyboard controls cycle through every shareable variant without intercepting text-editing keys, and the switcher cannot render in production.
### 5. Evaluate it
In HITL mode, surface the URL and the `?variant=` keys.
The user flips through the variants and may combine elements rather than choosing one unchanged.
In AFK mode, open the URL yourself, compare every variant against the stated objective criteria, and record the observations that support the verdict.
Use HITL instead when the decision depends on taste, product judgment, or UX feel.
Done when the evaluator can compare every variant in its host context and the prototype exposes enough contrast to reach a verdict.
## Production mapping
When the shared [SKILL](SKILL.md) permits production work, keep the full variant set on the throwaway branch and apply the verdict as follows:
- **Sub-shape A** — fold the winner into the existing page and drop the losing variants and switcher from main.
- **Sub-shape B** — promote the winner to a real route and drop the throwaway route and switcher from main.
## Anti-patterns
- **Variants that differ only in colour or copy.** That's a tweak, not a prototype. Real variants disagree about structure.
- **Sharing too much code between variants.** A shared `<Header>` is fine; a shared `<Layout>` defeats the point. Each variant should be free to throw out the layout.
- **Wiring variants to real mutations.** Read-only prototypes are fine. If a variant needs to mutate, point it at a stub — the question is "what should this look like", not "does the backend work".
- **Promoting the prototype directly to production.** The variant code was written under prototype constraints (no tests, minimal error handling). Rewrite it properly when you fold it in.

52
skills/research/SKILL.md Normal file
View File

@@ -0,0 +1,52 @@
---
name: research
description: Investigate a question against high-trust primary sources and capture the cited findings as a Research artifact. Use when a topic needs documentation, API, specification, source-code, or other reading legwork.
---
# Research
Investigate one question and preserve the findings in one cited Research artifact.
Run in the current process.
Isolation and concurrency belong to the caller.
## 1. Resolve the artifact
Resolve the artifact destination before naming the file.
When the caller provides a destination directory, use it exactly.
Otherwise, default to `$(xdg-user-dir DOCUMENTS)/ai-artifacts/projects/<project>` when the AI artifacts vault is available, where `<project>` is the lowercase basename of the current working directory.
If that vault is unavailable, fall back to `./docs` in the current project.
Create the destination directory only when needed.
Before choosing a filename, read the destination's `AGENTS.md`.
Use the naming convention declared there, or infer it from nearby artifacts when the file delegates naming to local context.
Do not assume a counter, numeric prefix, slug shape, or artifact-type suffix unless the destination convention requires it.
If no convention can be determined, choose the least surprising lowercase descriptive Markdown filename and state that the destination did not define a naming convention.
When the caller provides an existing ticket artifact path, use it exactly, do not allocate a filename, and complete that artifact in place.
Otherwise, allocate any identifier required by the destination convention and include `parent` only when an earlier artifact directly caused the research.
Done when one authoritative Research artifact path and its metadata are settled according to the destination convention.
## 2. Investigate the question
Use primary sources such as official documentation, specifications, source code, and first-party APIs rather than relying on secondary accounts.
Follow every substantive claim back to the primary source that owns it.
Use secondary material only to discover primary sources.
When no primary source establishes a needed claim, record that limitation instead of presenting the claim as settled.
Done when the question is answered as far as primary evidence permits and every substantive claim has an owning source or an explicit evidence gap.
## 3. Write the Research artifact
Write the findings to the resolved Markdown file and follow the destination's artifact conventions.
When the resolved file is an existing ticket artifact, preserve its workflow metadata and complete the result in that file.
Keep the question, findings, limitations, and citations sufficient for a future reader to evaluate the result without reconstructing the research session.
Do not create a source dump or research log.
Done when exactly one Research artifact exists at the resolved path and every substantive claim in it cites its source.
## 4. Return the result
Report the artifact path and a concise statement of what the research established or could not establish.
Done when the caller can locate the artifact and understand whether the question was resolved.

121
skills/review/SKILL.md Normal file
View File

@@ -0,0 +1,121 @@
---
name: review
description: Produce a structured, report-only review of a context-selected change set across risk, standards, intent, evidence, and documentation.
---
# review
Review a change set selected from context and produce a structured review report.
This skill is standalone and context-driven.
It does not know about implementation lifecycle, Wayfinder claiming, task resolution, PR creation, or ticket closeout.
It must not edit code.
Reviewer prompts live under [`reviewers/`](reviewers/).
Load the relevant reviewer files before running independent dimension reviews.
## 1. Select and capture the change set
Select the relevant change set from current context.
The change set may be uncommitted changes, branch changes, a PR, explicit files, or another clear source.
State what change set you selected before reviewing.
Ask or stop only when scope ambiguity would make the review untrustworthy.
Capture enough material for reviewers to inspect the selected change set.
This may include a diff, changed-file list, commit list, PR details, relevant intent source, available verification evidence, and repository instructions or standards sources.
Do not overprescribe mode selection or intent-source taxonomy.
Use agent judgment when the selected change set and intent source are clear enough to review.
Done when the chosen change set is explicit and reviewers have enough input to judge it.
## 2. Run dimension reviews
Run independent dimension reviewers through whatever real context boundary the current harness provides.
A separate agent, worker, subprocess, or documented headless session is sufficient when each reviewer can inspect its dimension without seeing the parent review's intermediate conclusions.
Use one reviewer per dimension when practical.
If no real context boundary is available, stop before reviewing and report that independent review cannot be completed in the current runtime.
Do not replace independent reviewers with in-process role switches.
Use these reviewer files:
- [`reviewers/risk.md`](reviewers/risk.md)
- [`reviewers/standards.md`](reviewers/standards.md)
- [`reviewers/intent.md`](reviewers/intent.md)
- [`reviewers/evidence.md`](reviewers/evidence.md)
- [`reviewers/documentation.md`](reviewers/documentation.md)
Each reviewer should follow its own purpose, boundary, rubric, and output expectations.
The shared base finding shape is flexible:
```markdown
- **Severity**: error | warning | info
**Blocking**: yes | no
**Location**: <file, line, command, artifact, or none>
**Finding**: <specific issue>
**Why it matters**: <impact>
**Suggested fix**: <concrete next action, or none>
```
Reviewers may add dimension-specific fields where useful.
Reviewers classify findings as blocking or non-blocking.
The caller or user decides final disposition.
Done when each applicable dimension has returned findings or a clear statement that the dimension does not apply.
## 3. Run targeted checks when useful
Run targeted checks when they are needed to assess a dimension.
Prefer focused commands that clarify evidence, risk, or standards conformance.
Do not turn review into a full external validation pipeline.
If a full suite or expensive check is clearly needed, explain why before relying on it.
Record commands and artifacts that materially influenced the review.
Do not present unrun checks as evidence.
Done when the review has enough evidence for its findings and limitations are stated honestly.
## 4. Aggregate the structured review report
Produce a structured review report, not a gate artifact and not a pass/fail verdict.
Keep sections separate so one dimension does not mask another.
Deduplicate exact duplicates only.
Mention when separate dimensions independently flag the same issue.
Use this report shape unless the selected change set calls for a small adaptation:
```markdown
## Reviewed change set
<what was reviewed and how it was captured>
## Risk
<risk reviewer section>
## Standards
<standards reviewer section>
## Intent
<intent reviewer section>
## Evidence
<evidence reviewer section>
## Documentation
<documentation reviewer section>
## Summary
- Blocking findings: <count>
- Non-blocking findings: <count>
- Targeted checks run: <commands or none>
- Evidence limitations: <limitations or none>
```
Do not invent a pass/fail verdict.
Use blocking findings present or absent as the review's actionable summary.
Done when the report states the reviewed change set, preserves every applicable dimension, and summarizes blocking status and evidence limitations.

View File

@@ -0,0 +1,30 @@
# Documentation reviewer
Review only documentation consequences of the selected change set.
Do not make Risk, Standards, Intent, or Evidence findings unless the issue directly affects documentation correctness.
Look for user-facing, operator-facing, contributor-facing, and agent-facing documentation that should change because behavior, interfaces, commands, options, workflows, or constraints changed.
Also look for stale documentation introduced or left behind by the change.
Do not demand new documentation surfaces for every change.
Prefer updating the existing owner of the fact when one is discoverable.
Documentation findings may be non-blocking when the change is internal and no durable reader would be misled.
They may be blocking when users, operators, future contributors, or agents would reasonably make a wrong decision from stale or missing documentation.
## Output format
```markdown
## Documentation
### Findings
Use `No findings.` when this reviewer has no findings.
- **Severity**: <error|warning|info>
**Blocking**: <yes|no>
**Documentation owner**: <file, artifact, or none known>
**Location**: <file, line, command, artifact, or none>
**Finding**: <specific issue>
**Why it matters**: <reader impact>
**Suggested fix**: <concrete next action, or none>
```

View File

@@ -0,0 +1,38 @@
# Evidence reviewer
Review only whether the selected change set has convincing evidence.
Do not make Risk, Standards, Intent, or Documentation findings unless the issue directly affects evidentiary value.
Check tests, commands, artifacts, manual checks, and any evidence the change provides.
You may recommend or run targeted checks when the orchestrating review context allows it.
Do not present unrun checks as evidence.
Reject source-grep pseudo-tests as proof of behavior.
A test whose only evidence is matching implementation source text, tokens, lines, syntax, prompt phrases, regexes, AST shapes, or incidental snapshots does not prove behavior.
Prefer evidence that demonstrates observable behavior, state, output, side effects, failure modes, or semantic meaning through a public or executable interface.
For declarative artifacts, prefer invoking the real consumer or parsing into a typed or normalized semantic model.
Reading file contents is legitimate when the file itself is the owned output or serialized contract under test.
## Output format
```markdown
## Evidence
Evidence checked:
- <command, test, artifact, or none>
### Findings
Use `No findings.` when this reviewer has no findings.
- **Severity**: <error|warning|info>
**Blocking**: <yes|no>
**Evidence checked**: <command, test, artifact, or none>
**Evidence gap**: <what remains unproven, or none>
**Location**: <file, line, command, artifact, or none>
**Finding**: <specific issue>
**Why it matters**: <trust or verification impact>
**Suggested fix**: <concrete next action, or none>
```

View File

@@ -0,0 +1,33 @@
# Intent reviewer
Review only whether the selected change set satisfies the apparent intent.
Do not make Risk, Standards, Evidence, or Documentation findings unless the issue directly changes intent fidelity.
Use the clearest available intent source from context.
Possible sources include a Wayfinder task, issue, spec, direct request, PR description, branch context, commit messages, or conversation context.
Do not invent requirements.
When intent is inferred rather than explicit, say so and treat it with appropriate caution.
Look for missing requested behavior, partial implementation, behavior outside the selected scope, and implementations that appear to satisfy wording while violating the underlying request.
Do not require remote branch, PR, or CI outcomes when another workflow step owns those outcomes.
## Output format
```markdown
## Intent
Intent source: <source or none>
Intent confidence: <explicit|inferred|unavailable>
### Findings
Use `No findings.` when this reviewer has no findings.
- **Severity**: <error|warning|info>
**Blocking**: <yes|no>
**Intent source**: <task, spec, issue, request, branch context, or none>
**Location**: <file, line, command, artifact, or none>
**Finding**: <specific issue>
**Why it matters**: <intent or correctness impact>
**Suggested fix**: <concrete next action, or none>
```

View File

@@ -0,0 +1,46 @@
# Risk reviewer
Review only risk.
Do not make Standards, Intent, Evidence, or Documentation findings unless the issue directly changes the risk assessment.
Assess how much attention the selected change set warrants before handoff.
Use the worst-factor-wins rubric from the old local review workflow.
Rate each factor Low, Medium, or High with a concise reason:
- Blast radius.
- Reversibility.
- Test coverage.
- Sensitive domain.
- Size and complexity.
- Runtime criticality.
Overall risk is the highest factor.
Risk alone need not block.
Missing evidence for a risky change may be blocking when a future reader should not trust the handoff without more proof.
## Output format
```markdown
## Risk
**Overall: <LOW|MEDIUM|HIGH>**
- Blast radius: <rating> — <reason>
- Reversibility: <rating> — <reason>
- Test coverage: <rating> — <reason>
- Sensitive domain: <rating> — <reason>
- Size and complexity: <rating> — <reason>
- Runtime criticality: <rating> — <reason>
### Findings
Use `No findings.` when this reviewer has no findings.
- **Severity**: <error|warning|info>
**Blocking**: <yes|no>
**Risk factor**: <factor or none>
**Location**: <file, line, command, artifact, or none>
**Finding**: <specific issue>
**Why it matters**: <risk impact>
**Suggested fix**: <concrete next action, or none>
```

View File

@@ -0,0 +1,43 @@
# Standards reviewer
Review only repository standards and maintainability.
Do not make Risk, Intent, Evidence, or Documentation findings unless the issue directly affects standards conformance.
Use repository instructions and discovered standards sources such as `AGENTS.md`, `CONTRIBUTING.md`, coding standards, or nearby conventions.
When documented standards conflict with generic advice, the repository standard wins.
Also apply this smell baseline when tooling or project standards do not already cover the issue:
- Mysterious Name.
- Duplicated Code.
- Feature Envy.
- Data Clumps.
- Primitive Obsession.
- Repeated Switches.
- Shotgun Surgery.
- Divergent Change.
- Speculative Generality.
- Message Chains.
- Middle Man.
- Refused Bequest.
Treat smell findings as judgment calls, not automatic hard violations.
Skip anything deterministic tooling already enforces unless the current review evidence shows the tool is not being run.
## Output format
```markdown
## Standards
### Findings
Use `No findings.` when this reviewer has no findings.
- **Severity**: <error|warning|info>
**Blocking**: <yes|no>
**Standard source**: <file and rule, smell baseline, or convention>
**Location**: <file, line, command, artifact, or none>
**Finding**: <specific issue>
**Why it matters**: <maintainability or standards impact>
**Suggested fix**: <concrete next action, or none>
```

178
skills/slice/SKILL.md Normal file
View File

@@ -0,0 +1,178 @@
---
name: slice
description: Turn a settled plan, spec, conversation, or artifact into Wayfinder implementation tickets for tracer-bullet slices. Run deliberately as /skill:slice when the user asks to slice work.
disable-model-invocation: true
---
# slice
Turn settled source material into implementation-ready Wayfinder tickets.
Use this when the user asks to slice a plan, spec, conversation, or artifact into agent-grabbable implementation work.
Do not use this to resolve planning fog.
If the source contains unresolved decisions, report them instead of inventing implementation slices.
`slice` writes Wayfinder `ticket/implementation` artifacts for implementation slices and `ticket/task/human` artifacts only when the slice cannot proceed without human action.
It does not write legacy `.claude/tasks/` files.
## 1. Gather source material
Work from the current conversation first.
If the caller passes paths, artifact links, issue references, or URLs, read the referenced body and relevant comments or nearby artifact context before slicing.
When source material names a Wayfinder artifact, use that artifact as the default provenance parent.
When no source artifact exists, use the active Wayfinder map if one is clear from context.
When neither exists, create or select the minimal Wayfinder parent required by the destination's artifact rules before writing tickets.
Explore the codebase when the current implementation state is not already understood.
Use the project's established vocabulary and respect relevant ADRs, context artifacts, and repository instructions.
Look for prefactoring that makes the change easier before slicing the behavior.
Done when the source, project context, and default parent artifact are known.
## 2. Resolve the artifact destination
Resolve the destination before naming files.
Use an explicitly supplied destination directory exactly.
Otherwise, default to `$(xdg-user-dir DOCUMENTS)/ai-artifacts/projects/<project>` when the AI-artifacts vault is available, where `<project>` is the lowercase basename of the current working directory.
If that vault is unavailable, fall back to `./docs` in the current project.
Create the destination directory only when a new artifact requires it.
Read the destination's `AGENTS.md` before any artifact write.
Follow the destination's filename, identifier, slug, frontmatter, and wikilink conventions.
If Wayfinder artifact reference is available, read it before writing.
The minimum Wayfinder ticket contract is:
```markdown
---
status: open
parent: "[[<source-artifact-or-map>]]"
blocked-by: []
tags:
- ticket/implementation
---
# <task name>
## Question
<one precise implementation action sized to one agent session>
```
Use `ticket/task/human` instead of `ticket/implementation` when execution or validation requires human input before implementation can proceed.
Do not create claims.
Tickets start open and unclaimed.
Done when the destination and artifact conventions are known.
## 3. Draft slices
Break the work into tracer-bullet slices.
Each normal slice must deliver a narrow complete path through every layer the change requires.
Do not create horizontal layer-only slices.
Reject any proposed slice whose title or delivered behavior is only parsing, only schema, only API, only UI, only tests, or only documentation when those layers are part of one user-visible change.
For a small feature, one slice is usually correct because parsing, behavior, tests, and documentation belong in the same coherent commit.
A completed slice must be demoable or verifiable on its own.
A slice should make sense as one coherent commit.
Put prefactoring slices before behavior slices that depend on them.
Record blocking edges while drafting.
Use blockers only for work that genuinely gates the slice.
A slice with no blockers can start immediately.
Use `parent` for provenance and `blocked-by` for prerequisites.
Use the wide-refactor exception when a single mechanical change has a blast radius that prevents any vertical slice from landing green.
Sequence that work as expand, migration batches, and contract.
The expand slice adds the new form beside the old so nothing breaks.
Each migration batch is sized by blast radius and is blocked by the expand slice.
The contract slice removes the old form and is blocked by every migration batch.
When migration batches cannot stay green alone, keep the sequence but state the integration-branch constraint and add a final integrate-and-verify slice.
Done when every currently implementable slice has a title, type, blockers, delivered behavior, acceptance criteria, and any needed handoff notes.
## 4. Classify ticket type
Use `ticket/implementation` when an agent can complete and verify the implementation slice without live human input.
Use `ticket/task/human` when execution or validation depends on the user before implementation can proceed.
Examples include subjective UI judgment, physical device checks, unavailable credentials, external account approval, or behavior only the human can confirm.
Do not mark a ticket human merely because it is important or risky.
Make the required human involvement explicit in the ticket body.
If slicing reveals a precise unresolved planning question, do not disguise it as an implementation ticket.
Report it and stop, unless the caller asks to create the appropriate Wayfinder research, prototype, or grill ticket.
Done when each proposed slice has the correct ticket type.
## 5. Review before writing
Present the proposed breakdown and wait for approval unless the caller explicitly asks for noninteractive output.
For each slice show exactly this review shape:
```markdown
1. **Title**: <proposed task title>
**Type**: implementation or human task
**Blocked by**: <proposed blockers, or None>
**What it delivers**: <end-to-end behavior or implementation result>
**Acceptance criteria**:
- <observable completion check>
```
Ask the user whether the granularity feels right, whether the blocking edges are correct, and whether any slices should be merged or split.
Iterate until the breakdown is approved.
In explicitly noninteractive mode, write the best breakdown and record any assumptions in `## Implementation Notes`.
Done when the proposed slices are approved or noninteractive assumptions are explicit.
## 6. Write ticket artifacts
Allocate filenames and identifiers according to the destination convention.
Create one Wayfinder ticket artifact per approved slice.
Use the destination's wikilink style when referring to artifacts.
Never refer to a ticket by a bare identifier, filename, or slug in user-facing text.
Each ticket body should use this shape:
```markdown
# <task name>
## Question
<one precise implementation action sized to one agent session>
## What to build
<concise end-to-end behavior or implementation result, not a layer-by-layer checklist>
## Acceptance criteria
- [ ] <observable criterion>
## Implementation Notes
<concise handoff guidance when useful>
```
Keep `## Implementation Notes` empty when no guidance earns its place.
Later implementing agents may append execution notes, verification results, and reasons for deliberately dropped criteria.
Avoid brittle file paths and code snippets unless a prototype or prior decision snippet encodes the decision more precisely than prose can.
Done when every approved slice has exactly one open, unclaimed Wayfinder ticket artifact.
## 7. Advance the Frontier
Re-read the map or parent artifact before editing shared state.
If there is a Wayfinder map, recompute its Frontier from ticket metadata.
A ticket belongs on the Frontier when it is open, every blocker is satisfied, and it has no claim.
The map's Frontier is the only map section that links to ticket artifacts.
Do not copy ticket details into the map.
If the destination uses identifier allocation and concurrent writes caused duplicate identifiers or filenames, preserve pre-existing artifacts, rename current outputs, update wikilinks, and advance the counter as needed.
Done when the ticket artifacts and map Frontier agree with current ticket metadata.
## 8. Report
Report the created ticket artifacts by title using the destination's link style.
Summarize the current Frontier.
Mention any assumptions, unresolved planning questions, or human-dependent validation tickets.
Done when the user can choose the next implementation task from the Frontier.

58
skills/subagents/SKILL.md Normal file
View File

@@ -0,0 +1,58 @@
---
name: subagents
description: Delegate isolated work through Pi subagent tools. Use when a workflow needs an independent worker, fresh context, parallel reviewers, or hidden role separation.
disable-model-invocation: true
---
# subagents
## 1. Confirm support
Prefer tool calls when the current runtime exposes them.
The supported tool names are `subagent_spawn`, `subagent_batch`, `subagent_list`, `subagent_status`, `subagent_result`, and `subagent_cancel`.
If these tools are unavailable, do not pretend that an in-process role switch is a subagent.
State that Pi subagent tools are unavailable in the current runtime.
Do not perform the delegated subagent work in-process.
Done when the run has a supported subagent tool path.
## 2. Prepare a bounded prompt
Give each subagent a self-contained prompt.
Include the role, task, repository path, relevant files or artifacts, constraints, and expected output shape.
Do not expose private planning state that the workflow is trying to isolate.
For test-driven development, send one behavior at a time rather than a backlog.
For review, send one review dimension at a time unless batching independent dimensions.
Use `context: "independent"` for a fresh worker by default.
Use `context: "fork"` only when the child intentionally needs the parent transcript as starting context.
Choose named agents only when the project has configured them.
Do not invent named agents as part of the delegation.
Done when the prompt is narrow enough that the subagent can complete without sharing hidden state or requiring follow-up orchestration.
## 3. Spawn and track workers
Use `subagent_spawn` for one worker.
Use `subagent_batch` when several independent workers can run in parallel.
Both spawn tools return before the child work is complete.
Record accepted child ids and per-entry failures.
Use `subagent_list` or `subagent_status` to track lifecycle when needed.
Use `subagent_result` to retrieve completion output.
Use `subagent_cancel` for stale or no-longer-needed workers.
Do not claim the delegated work is complete until every required child result is available or a failed child has an explicit disposition.
Treat `queued`, `starting`, `running`, and `settling` as incomplete states.
Treat `failed`, `cancelled`, `timed_out`, and `orphaned` as failures unless the caller explicitly accepts the missing result.
Done when every required worker result has been collected or every missing result has a stated disposition.
## 4. Integrate results
Use subagent output as evidence, not as an unquestioned command.
Preserve which child produced each material finding.
When subagents disagree, report the disagreement rather than averaging it away.
When a subagent result changes implementation direction, verify the relevant facts in the parent context before editing code.
Done when the parent has integrated child results into the caller workflow with provenance and limitations visible.

View File

@@ -0,0 +1,77 @@
---
name: test-driven-development
description: Drive red-green-refactor implementation with tests written by someone other than the implementation agent, using public behavior and independent expected values.
---
# test-driven-development
Use test-driven development as process discipline for behavior-bearing changes.
The result is working behavior plus useful tests, not a separate report artifact.
## 1. Establish roles and behavior backlog
Name the two functional roles before writing tests.
The implementation agent changes production code and drives the loop.
The test writer writes and edits tests.
Tests are always written by someone other than the agent doing implementation.
Launch an independent test writer through whatever real context boundary the current harness provides.
A separate agent, worker, subprocess, or documented headless session is sufficient when it cannot see the implementation agent's private backlog.
If no real context boundary is available, stop before writing tests and report that test-driven development cannot be completed in the current runtime.
Do not replace the independent test writer with an in-process role switch.
Do not silently skip the independent-test-writer requirement.
The implementation agent may keep a private behavior backlog.
Keep that backlog isolated from the test writer.
The test writer receives one behavior at a time, not the whole backlog.
Done when roles are explicit and the implementation agent has the next behavior ready without exposing the whole backlog to the test writer.
## 2. Choose the public seam and source of truth
Test public behavior through the interface the code exposes or the task requires.
When the seam is unclear, identify the smallest public seam that can prove the behavior.
Do not test internals just to make RED easy.
Each test needs an independent source of truth for expected values.
Acceptable sources include a spec, task, intent excerpt, worked example, known-good literal, existing behavior being preserved, user clarification, or external standard.
The implementation agent's derived computation is not enough.
Done when the next behavior has a public seam, enough context for the test writer, and an independent expected-value source.
## 3. Run one red-green cycle
Give the test writer exactly one behavior, the public seam and context, the independent expected-value source, and any relevant project test conventions.
The test writer writes or edits one test for that behavior and confirms it fails for the intended reason.
A failing test should fail because the behavior is missing or wrong, not because of import, syntax, fixture, or collection errors.
The implementation agent writes the minimal production code needed to pass that test.
The implementation agent must not edit test-writer-authored tests.
If the test has a mechanical defect, send the error back to the test writer.
If the implementation agent believes the test asserts the wrong semantics, pause for user or intent clarification.
Done when one behavior has a meaningful failing test and then passes through production-code changes made by the implementation agent.
## 4. Repeat behavior by behavior
Repeat the red-green cycle one behavior at a time.
Do not bulk-write tests before implementation.
Do not let the test writer see the behavior backlog.
Do not add speculative behavior while making the current test pass.
Run focused verification as each behavior lands.
Use broader verification when the repository or change warrants it.
Done when every selected behavior has passed through the one-behavior red-green loop or has been deliberately deferred by the implementation agent's judgment.
## 5. Refactor only after green
Never refactor while RED.
After tests pass, the implementation agent may refactor production code.
Refactoring means changing code structure without changing externally observable behavior.
The test writer updates tests only for deliberate public seam changes or test defects.
A test failure during refactoring normally means production behavior broke.
Fix production code unless the public seam changed deliberately.
Done when refactoring, if any, is complete and the relevant tests remain green.

View File

@@ -0,0 +1,154 @@
# Wayfinder artifacts
## Resolve the project
Resolve the artifact destination before naming files.
When the caller provides a destination directory, use it exactly.
Otherwise, default to `$(xdg-user-dir DOCUMENTS)/ai-artifacts/projects/<project>` when the AI artifacts vault is available, where `<project>` is the lowercase basename of the current working directory.
If that vault is unavailable, fall back to `./docs` in the current project.
Create the destination directory only when a new artifact requires it.
Read the destination's `AGENTS.md` before any artifact write.
Allocate identifiers only when the destination convention requires them.
## Names
Use an effort name that identifies one durable effort and is not reused for another map in the project.
Choose each filename from the destination's `AGENTS.md` and nearby artifact convention.
Do not assume a counter, numeric prefix, slug shape, or artifact-type suffix unless the destination convention requires it.
If no convention can be determined, choose the least surprising lowercase descriptive Markdown filename and state that the destination did not define a naming convention.
Refer to artifacts through the link style used by the destination.
Never use a bare identifier as a human-facing reference.
Ticket filenames use the substantive artifact type as their artifact-type suffix: `research`, `prototype`, `grill`, `task`, or `implementation`.
Do not use `ticket` as a filename artifact-type suffix.
## Map
The map is the effort's root artifact and has no `parent`.
It is a route summary rather than the store for ticket resolutions.
Only the Frontier links to tickets or other artifacts.
```markdown
---
status: open
tags:
- wayfinder/map
---
# <effort name>
## Destination
<one or two lines describing what reaching the end of this map looks like>
## Notes
<standing domain, skill, and execution guidance>
## Frontier
- [[<open-unblocked-unclaimed-ticket>]]
## Decisions so far
<one plain-language decision or implementation outcome per resolved ticket, without artifact links>
## Not yet specified
<in-scope fog that is not precise enough to ticket>
## Out of scope
<work consciously ruled beyond the destination>
```
Map status is `open` while any live ticket or fog remains and `complete` when neither remains.
The Frontier is a derived navigation index and the map's only artifact-link section.
Ticket metadata is authoritative.
Repair the Frontier whenever it is missing, stale, or inconsistent with ticket state.
Order Frontier links by the destination's declared ordering unless the user chooses another ticket.
Fall back to filename order when no ordering is declared.
Under **Decisions so far**, record one concise, self-contained decision or implementation outcome for each resolved ticket.
Do not link or identify the resolved ticket, or copy supporting detail from the canonical resolution into the map.
## Tickets
A new ticket starts as:
```markdown
---
status: open
parent: "[[<map-or-surfacing-ticket>]]"
blocked-by: []
tags:
- ticket/<type>
---
# <ticket name>
## Question
<one precise question, prerequisite action, or implementation slice sized to one agent session>
```
Use one of these tags:
- `ticket/research`
- `ticket/prototype/afk`
- `ticket/prototype/hitl`
- `ticket/grill`
- `ticket/task/afk`
- `ticket/task/human`
- `ticket/implementation`
`parent` records provenance.
An initial ticket points to the map.
A ticket surfaced by another ticket points to the surfacing ticket.
An artifact may have many children, which agents find by searching for backlinks to its wikilink.
`blocked-by` records zero or more upstream artifacts that must resolve before the ticket becomes actionable.
It is independent of `parent`.
A ticket blocker is satisfied when its status is `resolved`.
A non-ticket blocker is satisfied when its artifact exists.
A ticket is on the Frontier when its status is `open`, every blocker is satisfied, and it has no claim.
A `ticket/task/afk` ticket is prerequisite work that unblocks the route.
A `ticket/implementation` ticket is a code, configuration, documentation, or test slice that delivers part of an execution map's destination.
## Claims and status
Ticket status is one of:
- `open`
- `claimed`
- `resolved`
- `out-of-scope`
Claim a ticket by setting `status: claimed`, `claimed-by` to the current execution-session identifier, and `claimed-at` to the current timestamp before doing any work.
Use `PI_SESSION_ID` when available and an equivalent harness session identifier otherwise.
Claims do not expire automatically.
The acting agent uses the available context to recover an abandoned claim.
Only resolved tickets contribute decisions or implementation outcomes under **Decisions so far**.
An out-of-scope ticket is closed, while **Out of scope** states the excluded work and reason in plain language without linking or identifying the ticket.
## Results
A ticket is a self-resolving artifact.
Its canonical result lives in that same artifact rather than in a child result artifact.
When invoking `research`, `prototype`, or `implement`, provide the resolved ticket artifact path.
The called skill completes the ticket artifact in place and does not edit the map.
The coordinating Wayfinder agent validates the updated ticket artifact, marks the ticket resolved when the called skill has not already done so, and updates the map.
If a called skill cannot honor this artifact contract, leave the ticket unresolved and record the incompatibility instead of silently storing the result elsewhere.
Navigate the artifact journey forward by finding every note whose `parent` links to the current artifact.
Do not duplicate those relationships through per-artifact Next sections.
## Concurrent writes
Re-read every shared artifact immediately before editing it.
After concurrent workers return, detect duplicate identifiers or filenames, preserve pre-existing artifacts, rename current outputs when required by the destination convention, update their links, and advance any counter required by that convention.
Recompute the Frontier only after returned artifacts and ticket states have been reconciled.

176
skills/wayfinder/SKILL.md Normal file
View File

@@ -0,0 +1,176 @@
---
name: wayfinder
description: Plan or coordinate a huge chunk of work that exceeds one agent session as a durable map of tickets, then resolve them until the way to the destination is clear.
disable-model-invocation: true
---
# Wayfinder
A loose idea has arrived that is too large for one agent session and wrapped in fog.
Wayfinding charts the way to a **destination** rather than charging at it.
It creates a durable map of questions whose resolutions are decisions, findings, prototypes, completed prerequisites, or explicit implementation slices.
Read [`ARTIFACTS.md`](ARTIFACTS.md) before charting or working a map.
It is the single source of truth for how maps, tickets, claims, blocking, resolutions, and the Frontier live in the resolved artifact destination.
## Plan, don't do
Wayfinder plans by default.
The map is complete when nothing remains to decide before someone performs the destination work.
The urge to implement the destination usually marks the edge of a planning map and the time to hand off.
An effort may explicitly permit execution in its Notes, but otherwise preserve resolutions rather than deliver the destination.
A map may explicitly be an execution map when the destination is a tracked implementation effort rather than a route to a later handoff.
Execution maps use the same ticket, claim, and Frontier mechanics, but implementation tickets may deliver slices of the destination.
The destination varies by effort and shapes every ticket.
It may be a spec to hand off, a decision to lock before planning, a change whose route must be understood before implementation, or an implementation effort whose slices need coordination.
## Refer by name
Refer to every map and ticket by its human-readable title using the destination's link style, never by a bare identifier, filename, or slug.
When the destination convention includes an artifact identifier, keep it inside the link without letting it stand in for the name.
## Ticket types
Every ticket is either **HITL**, worked through a live exchange with the human, or **AFK**, driven by the agent.
A HITL ticket only resolves through that exchange.
The agent never speaks for the human's side.
- **Research** (AFK): Investigate documentation, third-party APIs, or resources outside the current working directory through `research`.
The called skill completes the Research ticket artifact in place.
- **Prototype** (AFK or HITL): Raise the fidelity of a logic, state-model, or UI decision through `prototype`.
Use AFK when the question has objective observable criteria the agent can test end to end.
Use HITL when the verdict depends on human judgment, taste, or UX feel.
The called skill completes the Prototype ticket artifact in place after the verdict is reached.
- **Grill** (HITL): Resolve a decision through `grill`.
This is the default ticket type.
- **Task** (AFK or HITL): Perform prerequisite work that fits the map's destination.
In a planning map, a Task earns its place by unblocking a decision rather than delivering part of the destination.
The agent performs it where possible and otherwise gives the human a precise checklist.
- **Implementation** (AFK): Deliver a code, configuration, documentation, or test slice of an execution map through `implement`.
Implementation tickets expect checkout verification, tests where useful, review, commit or handoff, and Wayfinder closeout.
## Fog of war
The map is deliberately incomplete.
Beyond its tickets lies the **fog of war**, where in-scope questions are visible but cannot yet be stated precisely because they depend on unresolved questions.
Resolving a ticket clears the fog ahead of it and graduates newly precise questions into tickets.
Use this test:
- Create a ticket when the question is precise now, even if it is blocked.
- Keep an entry under **Not yet specified** when the question cannot yet be phrased precisely.
Do not pre-slice fog into speculative tickets.
One fog entry may become several tickets or disappear as the frontier advances.
The destination fixes scope.
Work beyond it belongs under **Out of scope**, never under **Not yet specified**.
When an existing ticket proves to be beyond the destination, mark it out of scope and summarize the excluded work and reason in that section without an artifact link.
Do not record a scope boundary as a decision on the route.
## Select the mode
- A loose idea without a map uses **Chart the map**.
- An existing map uses **Work through the map**.
- A session working a map with unblocked AFK Frontier tickets uses **Coordinate workers** inside **Work through the map** when the current harness provides a real isolation or concurrency mechanism.
Worker coordination is the default for AFK Frontier work when an isolation or concurrency mechanism is available.
The current session acts as coordinator.
A worker session resolves exactly one claimed ticket and stops.
A coordinating session may dispatch multiple open Frontier tickets through whatever real isolation or concurrency mechanism the current harness provides.
A coordinating session does not claim tickets it intends to delegate.
Each worker claims its own ticket so accountability remains attached to the session doing the work.
An interactive Wayfinder session may resolve multiple tickets sequentially only when worker coordination is unavailable, unnecessary, or explicitly not selected.
It must complete the full reconcile, claim, resolve, record, and frontier-advance loop before selecting another ticket.
Do not auto-consume HITL tickets without user participation.
## Chart the map
1. **Name the destination.**
Invoke `grill` to settle what this map is finding its way toward.
Done when the destination names the spec, decision, or change at the end of the effort and fixes its scope.
2. **Map the frontier breadth-first.**
Invoke `grill` again to fan out across the whole space without resolving any one branch in depth.
Surface every currently precise question, its blocking relationships, and the remaining fog.
If no fog remains and the whole route fits one session, stop and ask how the user wants to proceed instead of creating a map.
Done when every visible in-scope uncertainty has exactly one home as a precise ticket question or an honest fog entry.
3. **Create the map and tickets.**
Create the map first, then every currently precise ticket, then wire blocking relationships in a second pass according to [`ARTIFACTS.md`](ARTIFACTS.md).
Done when the map is the effort root, every precise question has one ticket, every known blocking edge is represented, and the Frontier is current.
4. **Dispatch Research.**
Invoke `research` for each Research ticket using whatever isolation or concurrency the caller provides.
Reconcile each completed Research ticket according to [`ARTIFACTS.md`](ARTIFACTS.md).
Leave a ticket open with the reason visible when its Research run cannot complete.
Done when every dispatched Research ticket is resolved or records why it remains open.
5. **Stop.**
Stop without resolving a HITL ticket.
Done when charting has created and dispatched the visible route without consuming its human decision work.
## Work through the map
1. **Orient.**
Read the map at low resolution rather than loading every ticket.
Reconcile its derived Frontier against ticket metadata.
Done when the destination, Notes, prior decisions, fog, scope boundary, and current Frontier agree with the artifacts.
2. **Claim one ticket or coordinate workers.**
If unblocked AFK Frontier tickets can be delegated through a real isolation or concurrency mechanism, use **Coordinate workers** instead of claiming a ticket here.
Use the user-named ticket when it is actionable and not delegated.
Otherwise take the first Frontier ticket in the destination's declared ordering, falling back to filename order when no ordering is declared.
Persist the claim before doing any direct work.
Done when exactly one unblocked ticket records this session's claim with `status: claimed`, or the session has switched to worker coordination without claiming delegated tickets.
3. **Resolve by type.**
Invoke `research`, `prototype`, `grill`, or `implement` for the corresponding ticket type.
Perform a Task through the capability or human checklist it requires.
Load related artifacts only when needed.
Done when the question has a resolution, the prerequisite Task is complete, or the implementation slice has a concrete handoff state.
4. **Record the resolution.**
Persist the canonical result in the ticket artifact, resolve the ticket, and append its concise decision or implementation outcome under the map's **Decisions so far** according to [`ARTIFACTS.md`](ARTIFACTS.md).
Do not link or identify the resolved ticket from the map.
Done when the resolution lives in the ticket artifact and the map states only the resulting decision or outcome.
5. **Advance the frontier.**
Create tickets surfaced by the resolution and wire their blockers.
Graduate newly precise fog, remove invalidated tickets, move beyond-destination work out of scope, and recompute the Frontier.
Re-read shared artifacts before each write because other sessions may edit the effort concurrently.
Done when every newly visible question has exactly one home and the map agrees with all current ticket metadata.
6. **Complete, continue, or stop.**
When no unresolved tickets or fog remain, mark the map complete and stop for an explicit handoff instruction.
In a worker session, stop after one ticket is resolved and the frontier is advanced.
In an interactive or coordinating session, continue to another Frontier ticket only after reconciling the map and shared artifacts again.
Otherwise stop.
Done when the map records its current lifecycle state and no destination work has begun without permission.
## Coordinate workers
1. **Select dispatchable tickets.**
Re-read the map and current Frontier before dispatch.
Select only open, unclaimed, unblocked AFK tickets.
Do not dispatch HITL tickets without live user participation.
Include every eligible AFK Frontier ticket unless serial execution or likely conflict requires selecting a smaller batch.
Done when every selected ticket is eligible and no selected ticket has been claimed by the coordinator.
2. **Dispatch workers.**
Send each selected worker exactly one ticket, the artifact path, the map context it needs, and the instruction to claim the ticket itself before work.
Route Research tickets through `research`, AFK Prototype tickets through `prototype`, Implementation tickets through `implement`, and AFK Task tickets through the focused task capability or checklist they require.
Use whatever real isolation or concurrency mechanism the current harness provides.
If no such mechanism exists, stop and report that worker coordination is unavailable in this runtime.
Use parallel workers when selected tickets are independent.
Use serial workers when tickets likely edit the same files, checkout, branch, or shared artifact surfaces.
Done when every selected ticket has either a launched worker or a visible dispatch failure.
3. **Join workers.**
Wait for every launched worker to finish, fail, time out, or be cancelled before treating coordination as complete.
Do not implement in the coordinator while workers are running.
Do not report final handoff while workers are still running.
Done when every launched worker has a terminal result or an explicit recovery status.
4. **Reconcile worker results.**
Re-read every shared artifact touched by returned workers.
Validate each ticket's claim, status, and canonical result.
Detect duplicate surfaced tickets or filenames, preserve pre-existing artifacts, and repair links according to [`ARTIFACTS.md`](ARTIFACTS.md).
Record whether a failed worker's claim remains, was reopened, or needs human recovery.
When Pi subagents supplied the worker results, call `subagent_clear` for each terminal child only after its findings have been reconciled into the artifacts or recovery record.
Done when all returned work is reconciled, every failed or missing worker result has an honest artifact state, and no reconciled terminal subagent remains in the visible work set.
5. **Advance the frontier.**
Recompute the Frontier after reconciliation, not before.
Mark the map `complete` only when no live ticket or fog remains.
Never use `resolved` as a map status.
Done when the map status, Frontier, and ticket metadata agree.

View File

@@ -1,54 +0,0 @@
---
name: wiki
description: Answer from the user's personal Obsidian notes — their "vault", "wiki", or "notes" — read-only. Fires when a question plausibly concerns the user's own recorded knowledge (projects, setup, config, decisions, preferences, or how-to notes they wrote) rather than general world facts, or on "check my notes/vault/wiki", "what do my notes say about…", "do I have anything on…". Returns a synthesized answer with the source note paths, or states plainly that the vault has nothing relevant.
---
# wiki
Answer a question from the user's personal Obsidian vault, read-only.
The vault is a knowledge base the user maintains, and this skill only ever reads it.
It searches the vault and nothing else — not the web, not general knowledge — and the caller decides whether to combine the result with other sources.
The caller may be the user directly, a subagent, or another skill such as `/consume`, and the process below is the same for all three.
## 1. Resolve the vault and its read boundary
The vault lives at `$(xdg-user-dir DOCUMENTS)/notes`.
If that path does not resolve or does not exist, report that the vault is unreachable and stop.
Read `<vault>/.claude/settings.json` and collect every `permissions.deny` entry of the form `Read(<glob>)`.
The glob inside each is a path relative to the vault root, so normalize it — drop the leading `./` and trailing `/**` — to get the off-limits path, turning `Read(./05 journal/**)` into `05 journal/`.
Those paths are off-limits: never read or search under one, and never let its contents reach an answer.
The harness does not apply these rules when you run from outside the vault, so enforcing them is this skill's job.
Done when you have the vault path and the set of off-limits paths (currently `05 journal/`).
## 2. Gather candidate notes
Expand the question into several search terms — synonyms, related concepts, and named entities — so a note worded differently than the question is still found.
Search with ripgrep over the Markdown files directly, not obsidian-cli, so retrieval works whether or not Obsidian is running: `rg -l -i` each term over `*.md` under the vault, excluding each off-limits path with ripgrep's own `-g '!<path>'` syntax.
Then widen for recall through the vault's topic hubs.
Each note in `02 tags/` names a topic, and every note on that topic links to it with a `[[topic]]` wikilink, so for any hub matching the question, add its members — the notes containing `[[<hub-name>]]` — to the candidates.
Done when you have a candidate set of note paths.
## 3. Read to saturation, following links
Read every candidate note.
Within each, follow every `[[wikilink]]` that bears on the question to its note and read that one too, then repeat on those notes' links.
Continue until a full pass surfaces no note you have not already read — saturation.
There is no cap on how many notes you read, and a broad question legitimately pulls in many.
Never follow a link into a denied glob, and read only Markdown — attachments and other binaries are not sources.
Done when every note bearing on the question has been read and a further pass finds nothing new.
## 4. Answer with provenance, or report nothing
If the notes answer the question, synthesize the answer, then list the notes whose content you drew on by path, each with its key excerpt, and cite that path for every claim taken from the vault.
A note that only led you to others, such as an empty topic hub, is a discovery aid rather than a source, so leave it out of the list.
Mark anything sourced from `00 INBOX/` as unvetted, since that folder is raw capture the user has not yet curated.
If nothing relevant was found, say plainly that the vault has nothing on the question.
Do not pad the answer with vault-flavored prose that no note supports, and do not create a note to fill the gap.
Done when the answer cites its source notes, or explicitly reports that the vault has nothing relevant.