Turn a benchmark run into a three-arm experiment: alongside the new-skill
and no-skill arms, add a previous-version arm materialized from the
default branch's HEAD, drawn automatically whenever the skill's directory
differs from HEAD and degrading to the two-arm efficacy-only shape
otherwise. Two blind head-to-heads now fall out per trial — efficacy
(new-vs-no-skill) and regression (new-vs-old).
The deterministic core dispatches a pass rule per comparison (efficacy at
wins>=3 and losses<=1, regression at losses<=1 with no wins floor),
gates only Efficacy on the hard assertion, and yields a second
skill-level Regression verdict, green when no case regressed and
not-applicable on a two-arm run. The report gains a second badge, a
headline reading both verdicts together, a previous-version cost row with
a two-ratio footnote, and per-case side-by-side comparisons that
auto-expand on any failure or flagged loss to show the losing-trial
output pair.
The fixture test covers both shapes in one test: a three-arm run
asserting per-arm metrics, both per-case and skill-level verdicts, the
net margins, the three-row cost table and losing-trial evidence, and the
two-arm run retained as the degenerate no-previous-version case.
Collapse duplication so each rule has one home: the pass-rule threshold
and cost footnote defer to the core, and the case-contract detail defers
to CASE-FORMAT.md. Drop editor-facing sediment from the arms step.
Fixes surfaced by a verify walk-through: define the hard-assertion gate
as passing only when every predicate exits zero (an earlier failure was
maskable by a later success), state that in-session arms run at the
session default temperature, name the fixture directory, and note the
steps run from the repo root.
Bring the feature's design record into git (it was authored across the
benchmark tasks but never committed). Cleaned to the repo's prose
standards on the way in: recast prose semicolons as separate sentences
and split multi-sentence lines to one sentence per line.
Run each benchmark arm in a hermetic fixture-only world materialized
outside the repo, so the no-skill baseline can no longer discover the
skill's assets on disk. The new-skill arm's skill is materialized to an
isolated temp path it is pointed at; the baseline gets no skill and is
told to stay within its world.
A live 5-trial axi-review run confirms the fix: all baseline transcripts
are clean of AXI-PRINCIPLES.md, the baselines invent their own generic
rubric (the honest counterfactual), and the Efficacy verdict is green
5/5 where the contaminated 0004 run had scored red 2/5.
Runner prose only; the deterministic core is untouched.
nix develop places benchmark-skill into ./.claude/skills/ (so /benchmark-skill
is available in this repo) and puts python3 on PATH to run the deterministic
core without a one-off nix shell.
Introduce the thinnest complete path that benchmarks a skill's efficacy
against a no-skill baseline and renders an HTML report.
- tests/ convention: a top-level tree mirroring skills/ by full path, with
case directories holding a case.md (CASE-FORMAT.md) and an optional fixture.
- benchmark-skill: a distributed, non-model-invocable runner (/benchmark-skill
<name>) that orchestrates force-invoked new-skill and skill-absent baseline
arms plus a blind per-trial judge as in-session subagents.
- Deterministic core (Python): a pure transform over the collected run data
that sums per-arm usage, collapses each trial to WIN/TIE/LOSS, applies the
efficacy pass rule (wins >= 3, losses <= 1), and renders a self-contained
report. Parameterized over arms and comparisons; two-arm here.
- Fixture-driven no-LLM check (checks/benchmark-core.nix) wired into flake.nix.
- Real tests tree for axi-review; a live run produced its efficacy report.
Review a CLI against the 10 canonical AXI principles black-box: run a
staged battery of read and write probes against a user-provided target,
judge each principle strictly against axi.md, and emit a report-card
table.
Ships as SKILL.md plus AXI-PRINCIPLES.md (per-principle rubric with
probes and PASS/PARTIAL/FAIL criteria) and REPORT-FORMAT.md (the
single-table card format and verdict vocabulary).
Claude-Session: https://claude.ai/code/session_015uc7R6zNYZpCxFWJMXrxwh
Expose lib.mkSkillsShellHook, which takes a list of selected skill derivations
and returns a dev-shell shellHook string. On shell entry the hook symlinks each
selected skill as a direct child of the project's .claude/skills/, pointing into
the store. It is stateless: each entry first removes only store-pointing symlinks
(deselected skills included), then relinks the current selection, leaving
hand-authored real directories untouched and regenerating a self-ignoring
.gitignore of the managed names. Add a nix flake check that sources the produced
hook against a fixture project and asserts placement, stale-removal, the
collision guard, non-store-symlink survival, and the .gitignore contents.
Expose homeModules.default, a home-manager module whose programs.agents.skills
option places operator-selected skill derivations into Claude Code's skills
directory, one recursive home.file per skill, gated on programs.claude-code.enable
and sourcing configDir from the claude-code module. Add a nix flake check that
builds the home-files tree under several operator configurations and asserts
placement, recursion, the enable gate, configDir sourcing, and coexistence with
an operator's own skills.
Add the content tier: a standalone flake that auto-discovers each skill
(a directory containing a SKILL.md, at any depth) and exposes it as an
individually addressable derivation built by lib.mkSkill. Directories
without a SKILL.md are descended through as cosmetic containers; once a
SKILL.md is found, that directory's subfolders are its assets, not
further skills. Skill names must be globally unique across the tree — a
collision is a hard eval-time error, not a warning.
A fixture-driven skill-build check under `nix flake check` exercises the
recursive walk, the builder, the SKILL.md-at-$out-root contract, the
eval-time name passthru, and the collision error. The repo ships no real
skill content yet, so packages.<system> is empty today.
