Introduce the thinnest complete path that benchmarks a skill's efficacy
against a no-skill baseline and renders an HTML report.
- tests/ convention: a top-level tree mirroring skills/ by full path, with
case directories holding a case.md (CASE-FORMAT.md) and an optional fixture.
- benchmark-skill: a distributed, non-model-invocable runner (/benchmark-skill
<name>) that orchestrates force-invoked new-skill and skill-absent baseline
arms plus a blind per-trial judge as in-session subagents.
- Deterministic core (Python): a pure transform over the collected run data
that sums per-arm usage, collapses each trial to WIN/TIE/LOSS, applies the
efficacy pass rule (wins >= 3, losses <= 1), and renders a self-contained
report. Parameterized over arms and comparisons; two-arm here.
- Fixture-driven no-LLM check (checks/benchmark-core.nix) wired into flake.nix.
- Real tests tree for axi-review; a live run produced its efficacy report.
Review a CLI against the 10 canonical AXI principles black-box: run a
staged battery of read and write probes against a user-provided target,
judge each principle strictly against axi.md, and emit a report-card
table.
Ships as SKILL.md plus AXI-PRINCIPLES.md (per-principle rubric with
probes and PASS/PARTIAL/FAIL criteria) and REPORT-FORMAT.md (the
single-table card format and verdict vocabulary).
Claude-Session: https://claude.ai/code/session_015uc7R6zNYZpCxFWJMXrxwh
Expose lib.mkSkillsShellHook, which takes a list of selected skill derivations
and returns a dev-shell shellHook string. On shell entry the hook symlinks each
selected skill as a direct child of the project's .claude/skills/, pointing into
the store. It is stateless: each entry first removes only store-pointing symlinks
(deselected skills included), then relinks the current selection, leaving
hand-authored real directories untouched and regenerating a self-ignoring
.gitignore of the managed names. Add a nix flake check that sources the produced
hook against a fixture project and asserts placement, stale-removal, the
collision guard, non-store-symlink survival, and the .gitignore contents.
Expose homeModules.default, a home-manager module whose programs.agents.skills
option places operator-selected skill derivations into Claude Code's skills
directory, one recursive home.file per skill, gated on programs.claude-code.enable
and sourcing configDir from the claude-code module. Add a nix flake check that
builds the home-files tree under several operator configurations and asserts
placement, recursion, the enable gate, configDir sourcing, and coexistence with
an operator's own skills.
Add the content tier: a standalone flake that auto-discovers each skill
(a directory containing a SKILL.md, at any depth) and exposes it as an
individually addressable derivation built by lib.mkSkill. Directories
without a SKILL.md are descended through as cosmetic containers; once a
SKILL.md is found, that directory's subfolders are its assets, not
further skills. Skill names must be globally unique across the tree — a
collision is a hard eval-time error, not a warning.
A fixture-driven skill-build check under `nix flake check` exercises the
recursive walk, the builder, the SKILL.md-at-$out-root contract, the
eval-time name passthru, and the collision error. The repo ships no real
skill content yet, so packages.<system> is empty today.
