Both spec docs and tasks 0001-0008 were mined into the personal wiki by
/consume; delete the scaffolding it consumed, leaving the now-empty spec/
and tasks/ directories in place.
Reworks the consume skill around two acceptance axes and two destinations,
implementing the six issues filed from the first real consume run as one design.
- Add trigger-ability as a second acceptance axis alongside generalizes, with
its two premises; a pull-channel note now requires both (#9).
- Give consume a push channel — the target's own CLAUDE.md — for knowledge that
fails an axis but is still useful: specific residue or proactive rules (#10).
- Name the general-lesson/specific-residue split and route each half (#12).
- Make the step-5 plan report a complete ledger with a per-note trigger line,
push write-set, ref-fixes, flagged-not-authored, and drops (#13).
- Extend project cleanup to reconcile CLAUDE.md (capture-gated removal/trim) and
fix references to deleted files wherever they land (#11).
- Encode a branch-owned output location: project notes flat under
01 sources/claude/projects/, honoring any pre-existing grouping (#14).
Records the resolved vocabulary in .claude/CONTEXT.md and the two-channel write
decision in .claude/adr/0001.
Closes#9Closes#10Closes#11Closes#12Closes#13Closes#14
Add a second pure transform to the benchmark core: a `--leaderboard` mode
that ranks per-skill results models into an index leaderboard, one row per
skill carrying both verdicts, links out to each per-skill report, and sorts
any red first (regressions above efficacy failures), then fragile-but-passing,
then clean green.
The fragile tier reuses the 0006 per-badge chips: the efficacy chip now
carries its win count so the leaderboard reads it against the pass floor
without recomputing margins, and fragility is scoped to the passing tier so a
red row never carries a chip. The runner's SKILL.md gains the no-argument
batch flow and the index invocation. The fixture test covers the tiered sort,
the not-applicable Regression cell, and the per-skill links.
Add the longitudinal layer to the benchmark core: a per-skill JSON-lines
history that appends one summary line per run and trims oldest-first at
fifty, two stacked net-margin trend ribbons (Efficacy and Regression, the
latter leaving a gap for two-arm runs) with the current run ringed and a
net/delta/green-count readout, and per-badge fragility chips that flag the
narrowest passing case on each green axis so a barely-green skill cannot
look robust.
History persistence is opt-in via --history and is the core's only side
effect. Without it the core stays a pure transform. Two committed fixtures
(clean and fragile) and new fixture-test sections cover the append-and-trim,
both ribbons including the two-arm gap, and the chips.
Turn a benchmark run into a three-arm experiment: alongside the new-skill
and no-skill arms, add a previous-version arm materialized from the
default branch's HEAD, drawn automatically whenever the skill's directory
differs from HEAD and degrading to the two-arm efficacy-only shape
otherwise. Two blind head-to-heads now fall out per trial — efficacy
(new-vs-no-skill) and regression (new-vs-old).
The deterministic core dispatches a pass rule per comparison (efficacy at
wins>=3 and losses<=1, regression at losses<=1 with no wins floor),
gates only Efficacy on the hard assertion, and yields a second
skill-level Regression verdict, green when no case regressed and
not-applicable on a two-arm run. The report gains a second badge, a
headline reading both verdicts together, a previous-version cost row with
a two-ratio footnote, and per-case side-by-side comparisons that
auto-expand on any failure or flagged loss to show the losing-trial
output pair.
The fixture test covers both shapes in one test: a three-arm run
asserting per-arm metrics, both per-case and skill-level verdicts, the
net margins, the three-row cost table and losing-trial evidence, and the
two-arm run retained as the degenerate no-previous-version case.
Bring the feature's design record into git (it was authored across the
benchmark tasks but never committed). Cleaned to the repo's prose
standards on the way in: recast prose semicolons as separate sentences
and split multi-sentence lines to one sentence per line.
Run each benchmark arm in a hermetic fixture-only world materialized
outside the repo, so the no-skill baseline can no longer discover the
skill's assets on disk. The new-skill arm's skill is materialized to an
isolated temp path it is pointed at; the baseline gets no skill and is
told to stay within its world.
A live 5-trial axi-review run confirms the fix: all baseline transcripts
are clean of AXI-PRINCIPLES.md, the baselines invent their own generic
rubric (the honest counterfactual), and the Efficacy verdict is green
5/5 where the contaminated 0004 run had scored red 2/5.
Runner prose only; the deterministic core is untouched.
Introduce the thinnest complete path that benchmarks a skill's efficacy
against a no-skill baseline and renders an HTML report.
- tests/ convention: a top-level tree mirroring skills/ by full path, with
case directories holding a case.md (CASE-FORMAT.md) and an optional fixture.
- benchmark-skill: a distributed, non-model-invocable runner (/benchmark-skill
<name>) that orchestrates force-invoked new-skill and skill-absent baseline
arms plus a blind per-trial judge as in-session subagents.
- Deterministic core (Python): a pure transform over the collected run data
that sums per-arm usage, collapses each trial to WIN/TIE/LOSS, applies the
efficacy pass rule (wins >= 3, losses <= 1), and renders a self-contained
report. Parameterized over arms and comparisons; two-arm here.
- Fixture-driven no-LLM check (checks/benchmark-core.nix) wired into flake.nix.
- Real tests tree for axi-review; a live run produced its efficacy report.
Expose lib.mkSkillsShellHook, which takes a list of selected skill derivations
and returns a dev-shell shellHook string. On shell entry the hook symlinks each
selected skill as a direct child of the project's .claude/skills/, pointing into
the store. It is stateless: each entry first removes only store-pointing symlinks
(deselected skills included), then relinks the current selection, leaving
hand-authored real directories untouched and regenerating a self-ignoring
.gitignore of the managed names. Add a nix flake check that sources the produced
hook against a fixture project and asserts placement, stale-removal, the
collision guard, non-store-symlink survival, and the .gitignore contents.
Expose homeModules.default, a home-manager module whose programs.agents.skills
option places operator-selected skill derivations into Claude Code's skills
directory, one recursive home.file per skill, gated on programs.claude-code.enable
and sourcing configDir from the claude-code module. Add a nix flake check that
builds the home-files tree under several operator configurations and asserts
placement, recursion, the enable gate, configDir sourcing, and coexistence with
an operator's own skills.
Add the content tier: a standalone flake that auto-discovers each skill
(a directory containing a SKILL.md, at any depth) and exposes it as an
individually addressable derivation built by lib.mkSkill. Directories
without a SKILL.md are descended through as cosmetic containers; once a
SKILL.md is found, that directory's subfolders are its assets, not
further skills. Skill names must be globally unique across the tree — a
collision is a hard eval-time error, not a warning.
A fixture-driven skill-build check under `nix flake check` exercises the
recursive walk, the builder, the SKILL.md-at-$out-root contract, the
eval-time name passthru, and the collision error. The repo ships no real
skill content yet, so packages.<system> is empty today.
