Turn a benchmark run into a three-arm experiment: alongside the new-skill
and no-skill arms, add a previous-version arm materialized from the
default branch's HEAD, drawn automatically whenever the skill's directory
differs from HEAD and degrading to the two-arm efficacy-only shape
otherwise. Two blind head-to-heads now fall out per trial — efficacy
(new-vs-no-skill) and regression (new-vs-old).
The deterministic core dispatches a pass rule per comparison (efficacy at
wins>=3 and losses<=1, regression at losses<=1 with no wins floor),
gates only Efficacy on the hard assertion, and yields a second
skill-level Regression verdict, green when no case regressed and
not-applicable on a two-arm run. The report gains a second badge, a
headline reading both verdicts together, a previous-version cost row with
a two-ratio footnote, and per-case side-by-side comparisons that
auto-expand on any failure or flagged loss to show the losing-trial
output pair.
The fixture test covers both shapes in one test: a three-arm run
asserting per-arm metrics, both per-case and skill-level verdicts, the
net margins, the three-row cost table and losing-trial evidence, and the
two-arm run retained as the degenerate no-previous-version case.
Introduce the thinnest complete path that benchmarks a skill's efficacy
against a no-skill baseline and renders an HTML report.
- tests/ convention: a top-level tree mirroring skills/ by full path, with
case directories holding a case.md (CASE-FORMAT.md) and an optional fixture.
- benchmark-skill: a distributed, non-model-invocable runner (/benchmark-skill
<name>) that orchestrates force-invoked new-skill and skill-absent baseline
arms plus a blind per-trial judge as in-session subagents.
- Deterministic core (Python): a pure transform over the collected run data
that sums per-arm usage, collapses each trial to WIN/TIE/LOSS, applies the
efficacy pass rule (wins >= 3, losses <= 1), and renders a self-contained
report. Parameterized over arms and comparisons; two-arm here.
- Fixture-driven no-LLM check (checks/benchmark-core.nix) wired into flake.nix.
- Real tests tree for axi-review; a live run produced its efficacy report.
Expose lib.mkSkillsShellHook, which takes a list of selected skill derivations
and returns a dev-shell shellHook string. On shell entry the hook symlinks each
selected skill as a direct child of the project's .claude/skills/, pointing into
the store. It is stateless: each entry first removes only store-pointing symlinks
(deselected skills included), then relinks the current selection, leaving
hand-authored real directories untouched and regenerating a self-ignoring
.gitignore of the managed names. Add a nix flake check that sources the produced
hook against a fixture project and asserts placement, stale-removal, the
collision guard, non-store-symlink survival, and the .gitignore contents.
Expose homeModules.default, a home-manager module whose programs.agents.skills
option places operator-selected skill derivations into Claude Code's skills
directory, one recursive home.file per skill, gated on programs.claude-code.enable
and sourcing configDir from the claude-code module. Add a nix flake check that
builds the home-files tree under several operator configurations and asserts
placement, recursion, the enable gate, configDir sourcing, and coexistence with
an operator's own skills.
Add the content tier: a standalone flake that auto-discovers each skill
(a directory containing a SKILL.md, at any depth) and exposes it as an
individually addressable derivation built by lib.mkSkill. Directories
without a SKILL.md are descended through as cosmetic containers; once a
SKILL.md is found, that directory's subfolders are its assets, not
further skills. Skill names must be globally unique across the tree — a
collision is a hard eval-time error, not a warning.
A fixture-driven skill-build check under `nix flake check` exercises the
recursive walk, the builder, the SKILL.md-at-$out-root contract, the
eval-time name passthru, and the collision error. The repo ships no real
skill content yet, so packages.<system> is empty today.
