chore: clear consumed spec and task scaffolding
Both spec docs and tasks 0001-0008 were mined into the personal wiki by /consume; delete the scaffolding it consumed, leaving the now-empty spec/ and tasks/ directories in place.
This commit is contained in:
@@ -1,115 +0,0 @@
|
||||
## Problem Statement
|
||||
|
||||
I manage a collection of Claude Code agent skills and need a way to select which ones apply globally (on every project) versus per-project.
|
||||
Today that per-project selection is driven by three agent-run skills — `setup-skills`, `update-skills`, `remove-skills` — backed by a `.claude/skills-lock.yaml` lockfile and a content-hashing script.
|
||||
Setup copies a skill out of a shared library into a project's tree; update re-hashes to detect drift and applies safe upstream changes; remove deletes it.
|
||||
This is a lot of hand-maintained machinery: an LLM chore to run, a lockfile to keep clean, and a hash-based drift classifier to reason about.
|
||||
I want skills packaged and versioned through Nix instead, so that selection is declarative, updates are a pinned-input bump, and the whole copy-and-hash apparatus disappears.
|
||||
|
||||
## Solution
|
||||
|
||||
A standalone Nix flake (`alexion/skills`) that packages each skill as an individually addressable derivation and exposes optional integration outputs for placing selected skills where Claude Code discovers them.
|
||||
|
||||
- Adding a skill is creating a directory containing a `SKILL.md`; the flake auto-discovers it.
|
||||
- A home-manager config selects skills it wants **globally** (into `~/.claude/skills/`) via a home-manager module.
|
||||
- A project's own flake selects skills it wants **for that project** (into `<project>/.claude/skills/`) via a dev-shell helper, pinning this flake as an input.
|
||||
- "Update" becomes `nix flake update`; "remove" becomes deleting a line from the selecting config; "setup" becomes adding one.
|
||||
- The three management skills, the lockfile, and the hashing script are retired — Nix subsumes all of their responsibilities.
|
||||
|
||||
Nix is accepted as a hard requirement: every machine and every project is Nix/dev-shell based, so there is no need to keep skills as portable, committed-into-the-tree files for non-Nix consumers.
|
||||
|
||||
## User Stories
|
||||
|
||||
1. As a skill author, I want to add a new skill by creating a directory with a `SKILL.md` and nothing else, so that adding a skill requires no edit to the flake.
|
||||
2. As a skill author, I want to organize skills into arbitrarily nested subfolders (e.g. group related workflow skills together), so that the source tree stays browsable as the collection grows.
|
||||
3. As a skill author, I want that grouping to be purely cosmetic, so that where a skill's folder sits never changes how it is selected, placed, or named.
|
||||
4. As an operator, I want to select a set of skills to be active in every project via my home-manager configuration, so that my always-on skills are declared in one place and installed reproducibly.
|
||||
5. As a project maintainer, I want to select a set of skills for a single project via that project's flake, so that only the skills relevant to that project are active there.
|
||||
6. As a project maintainer, I want the selected skills to appear under the project's `.claude/skills/` automatically when I enter its dev shell, so that I never run a manual install step.
|
||||
7. As a project maintainer, I want the committed record of which skills a project uses to be the project's `flake.nix` selection plus its pinned input, so that the selection diffs cleanly in git and reproduces exactly on any machine.
|
||||
8. As a project maintainer, I want the generated skill symlinks kept out of git, so that machine-specific store paths never get committed.
|
||||
9. As a project maintainer, I want to keep my own hand-authored, project-private skills in `.claude/skills/` alongside the Nix-delivered ones, so that a project can have bespoke skills without adding them to the shared repo.
|
||||
10. As a project maintainer, I want the dev shell to only ever touch the skills it manages, so that my hand-authored skills are never removed or clobbered.
|
||||
11. As an operator, I want to update all pinned skills with `nix flake update`, so that picking up upstream skill changes is a single, reviewable, atomic operation.
|
||||
12. As an operator, I want a skill I previously selected to disappear when I remove it from my selection and re-enter the shell (or rebuild), so that removal needs no cleanup command.
|
||||
13. As an operator, I want to promote a skill from per-project to global (or vice versa) by moving one line between a project flake and my home-manager config, so that changing a skill's reach never moves files.
|
||||
14. As a skill author, I want two skills that would resolve to the same name to fail the build with a clear error, so that a silent clobber can never happen.
|
||||
15. As a skill author, I want the packaging to know nothing about `.claude/skills` or Claude Code, so that when a skill later needs to feed a different agent/tool the content is not welded to one placement convention.
|
||||
16. As an operator, I want my global skill placement to compose with other modules that write into the same skills directory (e.g. a tool that self-installs its own skill), so that independent sources coexist without collisions.
|
||||
17. As a maintainer, I want `nix flake check` to fail if a skill is malformed, if the home-manager placement regresses, or if the dev-shell helper stops producing a valid hook, so that breakage is caught before it ships.
|
||||
|
||||
## Implementation Decisions
|
||||
|
||||
### Overall structure
|
||||
|
||||
- The flake has two clearly separated tiers: a **content tier** (tool-agnostic skill packaging) and an **integration tier** (optional, Claude-Code-specific placement). A skill derivation in the content tier must never reference `.claude/skills` or anything Claude-Code-specific.
|
||||
- Flake scaffolding uses `flake-utils.lib.eachSystem` over `x86_64-linux`, `aarch64-linux`, `aarch64-darwin`, with `nixpkgs` tracking unstable.
|
||||
An explicit system list is passed rather than `eachDefaultSystem`, because that default set includes `x86_64-darwin`, which recent nixpkgs dropped and whose `legacyPackages` now throws.
|
||||
|
||||
### Content tier
|
||||
|
||||
- Skills are **auto-discovered** by recursively walking the `skills/` tree; any directory containing a `SKILL.md` is a skill, at any depth. Once a `SKILL.md` is found, that directory is a skill and its own subfolders are its assets, not further skills.
|
||||
- Directories that do not contain a `SKILL.md` are organizational containers only; the walk descends through them. This nesting is **cosmetic**: it never reaches the placement location and is not a selectable unit (no "select a whole group" — that is a purely additive future change if ever wanted).
|
||||
- Each skill becomes an **individually addressable derivation**, exposed at `packages.<system>.<skill-name>`. There is no `packages.<system>.default` — a catalog has no single default skill.
|
||||
- A skill is addressed by its **name** (leaf directory / frontmatter `name`), independent of its source path. Placement flattens the source path away — a skill nested at `skills/workflow/to-spec/` is placed as a direct child `.../skills/to-spec/`, because Claude Code only discovers direct children of the skills root.
|
||||
- **Skill-derivation contract** (the interchange primitive, shared with self-packaging tool repos like `gitea-axi`):
|
||||
- `$out` contains `SKILL.md` at its root, plus any assets.
|
||||
- The derivation carries its **name as an eval-time attribute** (e.g. `pname` / a passthru), so placement can form `.../skills/<name>` without reading `$out` (no import-from-derivation).
|
||||
- **Names must be globally unique** across the whole tree. A collision detected during the discovery walk is a **hard build error** with a clear message, not a warning or a lint.
|
||||
- A `lib.mkSkill` builder is exposed (turns a skill source directory into a conforming derivation). Exposing it costs nothing and lets a tool repo reuse it, even though tool repos are expected to self-package today.
|
||||
|
||||
### Cross-skill references
|
||||
|
||||
- There is **no dependency/closure machinery.** Skills interact by **name-based invocation** (the Skill tool), not by reading each other's files. A skill that references another skill relies on that skill being loaded (globally or in the same project), not on any relative path resolving on disk.
|
||||
- Design convention, enforced by convention only (no lint): **skills are self-contained** — a skill never reaches into another skill's files; anything it needs at a path, it carries itself. The retired management skills (which read siblings' `LOCKFILE.md` and executed a sibling's script) are exactly the anti-pattern this convention forbids.
|
||||
|
||||
### Integration tier — selection
|
||||
|
||||
- Selection is **by derivation**, not by name string. Consumers pass skill derivations pulled from `packages.<system>` (e.g. bind `packages.<system>` to a short local name and list the skills off it).
|
||||
- Rationale: skills are derivations in this ecosystem, so a derivation-based API composes with any skill satisfying the contract, from any repo, and keeps this flake ignorant of skills it does not own (respecting the boundary that tool-specific skills live in and are packaged by their own repos, not registered here).
|
||||
|
||||
### Integration tier — global placement (home-manager)
|
||||
|
||||
- Exposed as `homeModules.default`.
|
||||
- The operator-facing option is **`programs.agents.skills`**, a `listOf package` with default `[]`. `agents` is deliberately an umbrella namespace (room for future `programs.agents.<other>`), and it has **no shared `programs.agents.enable`** — each sub-feature self-gates, so the namespace stays a clean, mergeable surface a different repo could also extend without an ownership conflict. An empty list is a no-op.
|
||||
- For each selected skill the module writes an individual home file at `${claude-code.configDir}/skills/<name>` with `source = <skill derivation>` and **`recursive = true`**, gated on `programs.claude-code.enable`, reading `configDir` from the claude-code module.
|
||||
- `recursive = true` is a **hard requirement, not a style choice**: it forces home-manager to materialize `.../skills/<name>/` as a real directory of per-file symlinks rather than claiming the directory as one opaque symlink. That is what allows this module, the operator's own skills declarations, and self-placing tool modules to coexist under one `skills/` tree.
|
||||
- The module deliberately does **not** feed `programs.claude-code.skills`. That option is single-valued and would collide with an operator already setting it; per-skill `home.file` composes where the option does not.
|
||||
|
||||
### Integration tier — per-project placement (dev shell)
|
||||
|
||||
- Exposed as `lib.mkSkillsShellHook`, which takes a list of selected skill derivations and returns a shellHook string a project drops into its dev shell.
|
||||
- On shell entry the hook symlinks each selected skill as a **direct child** of `<project>/.claude/skills/`, pointing into the Nix store.
|
||||
- **Stateless lifecycle:** each entry first removes only the symlinks under `.claude/skills/` that point into the store (unambiguously "ours"), then recreates the current selection. This yields free stale-removal (deselect a skill → its symlink is gone next entry) and never touches real directories. No manifest or state file.
|
||||
- **Coexistence with hand-authored skills:** real (non-symlink) skill directories under `.claude/skills/` are left untouched. The hook maintains a generated, self-ignoring `.claude/skills/.gitignore` that lists the names it manages (and ignores itself), so Nix-delivered symlinks stay out of git while hand-authored skills remain tracked.
|
||||
- The committed record of a project's selection is its **`flake.nix`** (the selection list) plus its pinned `flake.lock`; the generated symlinks are gitignored.
|
||||
- The hook is a plain shell string relying on ambient POSIX tools; it does not need a home-manager context (projects are not home-manager-managed).
|
||||
|
||||
### Retired
|
||||
|
||||
- `setup-skills`, `update-skills`, `remove-skills`, the `skills-lock.yaml` lockfile concept, and the directory-hashing script are all replaced by Nix and are **not** carried into this repo. Their responsibilities map to: selection in a flake (setup / remove) and `nix flake update` (update).
|
||||
|
||||
## Testing Decisions
|
||||
|
||||
- A good test here exercises the flake's **public outputs** as a consumer would observe them — a built skill, an evaluated-and-built module, an instantiated helper — not the internal shape of the discovery walk or builder. The outputs are the surface; there is no separate production code path needing its own seam.
|
||||
- The single seam is **`nix flake check`**, with three focused checks beneath it (prior art: the `gitea-axi` flake's package-build and `checks/home-manager-module.nix` checks):
|
||||
1. **Skill-build check** — builds every auto-discovered skill derivation. This one check transitively exercises the recursive discovery walk, the `mkSkill` builder, the `SKILL.md`-at-`$out`-root contract, and the name-uniqueness hard error (which fires at eval and so also surfaces here).
|
||||
2. **Home-manager-module composition check** — instantiates `homeModules.default` under a sample home-manager configuration that selects a couple of skills with `programs.claude-code` enabled, and builds the resulting home-files derivation. This is the load-bearing check: it proves per-skill `home.file`, `recursive = true`, the `claude-code.enable` gate, and `configDir` sourcing compose as intended. Direct analogue of gitea-axi's home-manager-module check.
|
||||
3. **shellHook assertion** — instantiates `lib.mkSkillsShellHook` with sample skills and asserts the produced hook is non-empty and references the expected store paths / skill names.
|
||||
- Prefer the highest seam: none of these introduce a bespoke test hook into production logic; they evaluate/realize the flake outputs directly.
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- **Dotfiles and any migration of existing skills.** This repo owns only the packaging and integration outputs. Moving current skills out of the dotfiles tree, rewiring the dotfiles home-manager configuration to consume this flake, deleting the retired management skills from wherever they currently live, and reviewing/rewriting individual skills are all handled separately and by hand. The repo starts with flake machinery and no skill content; skills are added incrementally, "as needed."
|
||||
- **Integration with tool-specific skills.** Skills that ship inside a tool's own repo (e.g. `gitea-axi`) self-package and self-place; they are not listed in or delivered by this flake. This flake stays ignorant of them. The shared skill-derivation contract is the only thing in common.
|
||||
- **Selectable groups.** Nested folders are cosmetic; selecting a whole group as a unit is not built (additive later if wanted).
|
||||
- **A formal cross-skill dependency system.** Not built; name-based invocation plus the self-containment convention is the whole mechanism.
|
||||
- **Non-Nix portability.** Skills are not required to work on a machine without Nix; there is no committed-into-the-tree copy for non-Nix consumers.
|
||||
- **Multiple harnesses.** Placement targets Claude Code only. The `programs.agents` namespace is chosen to leave room for other harnesses later, but no second harness is implemented now.
|
||||
|
||||
## Further Notes
|
||||
|
||||
- The global-vs-per-project distinction is no longer a directory (`library/` is gone) or a file-copy state; it is purely **which config selects a skill**. The same per-skill derivation is selected by home-manager for the global set and by a project's dev shell for that project's set.
|
||||
- The two integration outputs place skills through different mechanisms because they run in different contexts: home-manager (`home.file`, `recursive = true`) for the global set, and a dev-shell shellHook (store symlinks) for the per-project set. Both operate on the same by-derivation selection and the same skill-derivation contract.
|
||||
- `programs.agents` is a generic namespace; if this repo is ever made public it is a mild land-grab worth revisiting, but it is appropriate for a personal ecosystem.
|
||||
- The design intentionally mirrors `gitea-axi` (flake scaffolding, per-skill `home.file` with `recursive = true`, module shape, check style) so the two repos stay consistent and the placement lessons already encoded in gitea-axi's ADRs carry over.
|
||||
@@ -1,288 +0,0 @@
|
||||
## Problem Statement
|
||||
|
||||
I write agent skills, but I have no way to know whether a skill is actually pulling its weight.
|
||||
A skill can read well, be faithfully followed, and still make the agent's output no better than it would have been with no skill at all.
|
||||
That skill is worthless, yet nothing in my current setup would catch it.
|
||||
I want a repeatable way to prove that a given skill genuinely improves the agent's work before I trust it.
|
||||
|
||||
I also edit skills over time, and an edit can quietly make a skill worse.
|
||||
A skill that still beats no-skill but has degraded from its released version is a regression I currently cannot see, because measuring only against no-skill sets too low a bar to catch it.
|
||||
I want the same harness to tell me, the moment I edit a released skill, whether my change held quality steady or regressed against the version that is already trusted.
|
||||
|
||||
## Solution
|
||||
|
||||
A skill-benchmarking convention plus an agent-run harness that measures a skill's **efficacy against a no-skill baseline** and, whenever a released version exists, its **regression against that previous version**.
|
||||
|
||||
For each skill I author a set of test cases under a top-level `tests/` tree.
|
||||
Running the harness on a skill executes every case as a controlled experiment: the same realistic prompt is given to several arms, several times, and a blind judge decides which arm's output better satisfies the case's author-written expectations.
|
||||
Every run has a **new-skill arm** (my working tree) and a **no-skill baseline arm**.
|
||||
Whenever the skill already exists on the main branch, the run adds a third **previous-version arm** materialized from main's `HEAD`.
|
||||
Two blind head-to-head comparisons fall out of this: **efficacy** (new-vs-no-skill) tells me whether the skill earns its keep, and **regression** (new-vs-old) tells me whether my edit made it worse.
|
||||
|
||||
A skill "passes" efficacy only when the new arm reliably beats the no-skill baseline — a skill the agent follows faithfully but that does not beat the baseline is reported as adding nothing.
|
||||
A skill "passes" regression when the new arm reliably does *not* lose to the previous version — a clean edit that holds quality steady passes, and only a real degradation fails.
|
||||
The two are reported as **separate verdicts**, because "this skill never worked" and "I just broke a skill that did work" are different problems.
|
||||
|
||||
The harness runs in-session as an "AI script": it orchestrates the arms and the judge as subagents, and it hands the boring mechanical work — counting results, summing token usage, applying the pass rules, rendering the report — to a small committed program so the numbers are exact and reproducible.
|
||||
It produces a self-contained HTML report with the two verdicts, a per-arm cost table, trend lines across recent runs, and drill-down evidence for anything that failed or regressed.
|
||||
|
||||
The whole thing is a distributed skill named `benchmark-skill`, invoked as `/benchmark-skill <name>`, so any repo that follows the `tests/` convention can benchmark its own skills.
|
||||
|
||||
## User Stories
|
||||
|
||||
1. As a skill author, I want to prove a skill improves the agent's output versus doing the task with no skill, so that I do not keep a skill that reads well but adds nothing.
|
||||
2. As a skill author, I want each test case to pin down explicit expectations I write myself, so that the judge measures against my intent rather than inventing its own bar.
|
||||
3. As a skill author, I want the same realistic prompt given to every arm, so that the only difference measured is which version of the skill (or no skill) produced the output.
|
||||
4. As a skill author, I want the with-skill arms to actually load and follow the skill, so that I am measuring the skill's effect and not whether it happened to trigger.
|
||||
5. As a skill author, I want each case to carry a hermetic, committed fixture, so that a run is reproducible and needs no live external state.
|
||||
6. As a skill author who writes conversation-driven skills, I want a case to seed a prior conversation, so that I can benchmark skills whose input is a discussion rather than a file tree.
|
||||
7. As a skill author, I want a mechanical, deterministic gate (hard assertions) separate from the judged comparison, so that a malformed output fails immediately without spending judgment on it.
|
||||
8. As a skill author, I want a run repeated several times at a realistic temperature, so that the result reflects whether the skill reliably helps rather than helping once by luck.
|
||||
9. As a developer iterating on a skill, I want trend lines across recent runs, so that I can tell a real change from run-to-run noise after I edit the skill.
|
||||
10. As a developer, I want a per-arm table of turns, tokens, and imputed cost, so that I can see what the skill's quality gain costs and whether my edit made it more expensive.
|
||||
11. As a developer, I want the report to open on the verdicts and expand only what failed or regressed, so that a clean run is calm and a problem run puts the evidence in front of me.
|
||||
12. As a developer, I want a green run to still flag its most fragile case on each axis, so that a skill that barely passed cannot masquerade as robust.
|
||||
13. As a developer, I want to benchmark a single skill quickly, so that my edit-and-recheck loop is cheap.
|
||||
14. As a developer, I want to benchmark all skills at once and see a leaderboard, so that I know at a glance which of my skills are green and which regressed.
|
||||
15. As an operator, I want the whole battery to run on my normal interactive subscription, so that a benchmark does not draw down a separate metered automation credit pool.
|
||||
16. As a maintainer, I want the mechanical scoring and rendering to be committed, tested code, so that the report's numbers are exact and reproducible rather than re-derived by the agent each run.
|
||||
17. As a maintainer, I want the test tree kept out of the packaged skills, so that installing a skill never drags its fixtures along.
|
||||
18. As a maintainer, I want reports and history kept out of git, so that machine-specific run artifacts never get committed.
|
||||
19. As a developer updating a skill, I want the new version compared against the released version on main and not only against no-skill, so that a skill that silently got worse but still beats no-skill is caught rather than passing green.
|
||||
20. As a developer, I want efficacy and regression reported as two independent verdicts, so that I can tell "this skill never worked" apart from "I just broke a skill that did work."
|
||||
21. As a developer, I want a clean refactor that holds output quality steady to pass the regression check, so that a deliberate no-op change is not failed merely for tying the previous version.
|
||||
22. As a developer, I want the regression comparison to appear automatically the moment I edit a released skill, so that I never have to remember to opt into it.
|
||||
|
||||
## Implementation Decisions
|
||||
|
||||
### What is measured
|
||||
|
||||
- The benchmark measures **efficacy against a no-skill baseline** and, when a previous version exists, **regression against that previous version**.
|
||||
Trigger-correctness (does a model-invokable skill fire on a realistic prompt) and format-conformance are explicitly not measured here.
|
||||
They are separable evals that can be added later without disturbing this one.
|
||||
- Both dimensions are judged on **output quality**, with cost reported alongside but **never gating**.
|
||||
A skill that improves quality is worth keeping even when it costs more tokens, and the cost columns are there to inform, not to fail.
|
||||
|
||||
### The arms
|
||||
|
||||
- Every case is a controlled experiment with a **new-skill arm** (my working tree) and a **no-skill baseline arm**, and — whenever the skill already exists on the main branch — a third **previous-version arm** materialized from main's `HEAD`.
|
||||
- Two blind head-to-head comparisons are drawn from the arms per trial: **efficacy** pairs the new arm against the no-skill arm, and **regression** pairs the new arm against the previous-version arm.
|
||||
- Every arm is given the **identical realistic prompt**, authored once and arm-agnostically, so the arms differ only in which skill (or no skill) is present.
|
||||
- The new arm and the previous-version arm are both **force-invoked**: each subagent is pointed at its own isolated skill materialization and told to use it, reading that skill's own file and any assets it references, so the real skill machinery is exercised rather than a reconstruction.
|
||||
The previous-version arm is force-invoked identically to the new arm, pointed at the materialization of the old skill instead of the new arm's copy of the working-tree directory, so the skill version is the only difference between them.
|
||||
- The baseline arm receives the bare prompt with the skill **absent from its context**, which is the honest counterfactual of the skill not existing.
|
||||
- Force-invoking uniformly is a deliberate consequence of running in-session.
|
||||
An earlier design varied arm construction by whether a skill was model-invokable, but in-session subagents cannot faithfully reproduce skill auto-discovery, so the harness force-invokes every with-skill arm and leaves triggering to a future, separate eval.
|
||||
|
||||
### The regression comparison
|
||||
|
||||
- The previous version is **always main's `HEAD`** — the repo's default branch, resolved rather than hardcoded to the literal name "main" — and the new version is **the working tree**.
|
||||
Anchoring to the released state on main, rather than to whatever was last committed on the working branch, means the regression question is always "did my in-progress edit degrade from the trusted version," which is the moment of real risk.
|
||||
- The third arm runs **only when the skill directory differs from main's `HEAD`**.
|
||||
A brand-new skill that does not yet exist on main, or a skill whose directory is unchanged from main, has no meaningful previous version, so the run degrades to the two-arm efficacy-only shape.
|
||||
- **Cases are held fixed to the working tree.**
|
||||
All arms run against today's prompt, today's fixture, today's expectations, so the skill version is the only variable differing between the new and previous-version arms.
|
||||
Letting a case drift with the version would change two things at once and rob the new-vs-old verdict of meaning.
|
||||
The conservative consequence is accepted: if a case's expectations were rewritten alongside the skill, the old version is judged against a bar it was never written for, which surfaces a possible regression for a human to eyeball rather than silently excusing it.
|
||||
- The **hard-assertion gate applies to the new arm only**, exactly as before.
|
||||
The gate exists to catch the current skill emitting malformed output.
|
||||
Running it against the previous version would manufacture spurious failures for any assertion introduced in the very edit under test, and the no-skill arm was already never gated.
|
||||
- The **judge and the soft criteria are reused unchanged** for the regression comparison.
|
||||
A case's author-written soft criteria describe what a good answer looks like, which does not depend on the opponent, so the same blind judge grounds both head-to-heads and simply receives a different pair of outputs.
|
||||
|
||||
### Expectations per case
|
||||
|
||||
- A case's expectations come in two tiers.
|
||||
- **Hard assertions** are executable shell predicates that form a deterministic, LLM-free gate.
|
||||
They run against two provided values: `$OUTPUT`, the path to the arm's captured final message, and `$WORLD`, the path to that arm's fresh fixture copy.
|
||||
A non-zero exit fails the assertion.
|
||||
A case may have zero hard assertions.
|
||||
- **Soft criteria** are natural-language statements the judge grounds the head-to-heads on.
|
||||
A case has at least one.
|
||||
- This snippet, from the case-format decision, encodes the authoring contract more precisely than prose:
|
||||
|
||||
```
|
||||
---
|
||||
description: <one-line scenario>
|
||||
---
|
||||
## Prompt
|
||||
<the realistic user request, given identically to every arm>
|
||||
|
||||
## Seed (optional; conversation-based skills only)
|
||||
**User:** ...
|
||||
**Assistant:** ...
|
||||
|
||||
## Hard assertions (executable; $OUTPUT = arm's final message, $WORLD = its fixture copy)
|
||||
```sh
|
||||
test -f "$WORLD/review.md"
|
||||
grep -qE '<pattern>' "$OUTPUT"
|
||||
```
|
||||
|
||||
## Soft criteria
|
||||
- <a statement the judge grounds the comparisons on>
|
||||
```
|
||||
|
||||
### The judge
|
||||
|
||||
- The judge is an **LLM subagent**, one **blind judge per trial per comparison**, shown both arms' outputs as unlabelled A and B with the **order randomized** per trial.
|
||||
- The judge is grounded on the case's soft criteria rather than free-forming its own standard, and its prompt explicitly instructs it to discount mere length and formatting differences, since a skill can otherwise "win" by being more verbose.
|
||||
- The same judge machinery serves efficacy and regression.
|
||||
Only the pair of outputs it is handed differs.
|
||||
- Variance is handled by repeating trials rather than by a per-trial panel.
|
||||
A panel is escalated to only for a comparison whose trials come back consistently split.
|
||||
|
||||
### Trials, temperature, and the pass rules
|
||||
|
||||
- Each case runs **5 paired trials** at a **realistic temperature**, configurable per skill, with **no reuse of arms or judgments across arms or runs**.
|
||||
Trial *i*'s new-skill output is judged against trial *i*'s no-skill output for efficacy and against trial *i*'s previous-version output for regression.
|
||||
A realistic temperature is chosen because most skills guide open-ended reasoning, and a big part of a skill's value is making a good outcome reliable across that variance, which temperature zero would hide.
|
||||
A per-skill temperature-zero override remains available for a skill that genuinely wraps a mechanical task.
|
||||
- The **efficacy pass rule**:
|
||||
- Per trial, the graded head-to-head collapses to **WIN, TIE, or LOSS** for the new arm, and a tie counts as a non-win.
|
||||
- Per case, efficacy **passes** when **wins ≥ 3 and losses ≤ 1** across the 5 trials, and **any** loss is flagged in the report for human review.
|
||||
A failed hard assertion fails the case outright regardless of the head-to-head.
|
||||
- The **regression pass rule** inverts, because holding quality steady is the goal:
|
||||
- Per trial, the new-vs-old head-to-head collapses to **WIN, TIE, or LOSS** for the new arm, where a TIE means "as good as the previous version" (a success), a LOSS is the regression being hunted, and a WIN is a bonus improvement.
|
||||
- Per case, regression **passes** when **losses ≤ 1** across the 5 trials, with **no wins floor** — ties and wins both count as non-regressions — and **any** loss is flagged for human review.
|
||||
- The wins floor is dropped deliberately, so a deliberate no-op edit that ties the previous version is a pass rather than a failure.
|
||||
- Tolerating one loss on each axis, rather than demanding zero, keeps a single spurious judge miscall at temperature above zero from reddening a genuinely good result, while still surfacing every regression and failing on a second loss.
|
||||
|
||||
### The two verdicts
|
||||
|
||||
- A skill carries **two independent verdicts**: an **Efficacy** verdict, green when every case passes efficacy, and a **Regression** verdict, green when no case regressed.
|
||||
- The two are reported side by side rather than collapsed into one, because their four crossings carry genuinely different meanings:
|
||||
- efficacy-green + regression-green → the edit is safe and the skill earns its keep.
|
||||
- efficacy-green + **regression-red** → the skill still beats no-skill but got worse than the released version — the exact regression this feature exists to catch.
|
||||
- **efficacy-red** + regression-green → the skill does not beat no-skill, but the edit did not make it worse (it was already dead weight).
|
||||
- efficacy-red + regression-red → the skill does not beat baseline and the edit made it worse.
|
||||
- The report headline reads the two verdicts together into one "so what" (for example, "Still valuable, but this edit regressed two cases").
|
||||
- On a two-arm run with no previous version, only the Efficacy verdict is meaningful and the Regression verdict reads not-applicable.
|
||||
|
||||
### Fixtures and isolation
|
||||
|
||||
- Every case ships its own **hermetic, committed fixture**, and each arm-and-trial combination runs against a **fresh copy** of it, so writes from one run never leak into another.
|
||||
- File-and-tree skills get a fixture directory.
|
||||
Conversation-driven skills get a **seed transcript** in role-tagged form, injected as the subagent's prior context before the prompt.
|
||||
- Each arm-and-trial runs in a **fixture-only world**: the arm subagent's working directory is its fresh fixture copy rather than the repo root, and nothing under `skills/` or the grading `tests/` tree sits on any path it explores from there.
|
||||
This is what makes the no-skill baseline an honest counterfactual, because it cannot discover and read the skill's assets off disk, and it also stops any arm from reading its own case's soft criteria or hard assertions and tuning its answer to the bar it will be judged against.
|
||||
- A with-skill arm is handed its skill as an **isolated temp materialization** placed outside the fixture, and is pointed there to force-invoke it.
|
||||
The new-skill arm's materialization is a copy of the working-tree skill directory, so it reflects uncommitted edits.
|
||||
The previous-version arm's is the same skill directory checked out at main's `HEAD`, since each skill is packaged as its own self-contained derivation and does not need the rest of the repo.
|
||||
The no-skill arm is handed nothing.
|
||||
Each temp materialization is cleaned up after the run, like the fresh fixture copies.
|
||||
- Isolation here is deliberately **soft**: an in-session subagent shares the machine and could in principle reach the repo by absolute path.
|
||||
Relocating each arm's working directory to its fixture, and instructing the baseline to stay within it, moves contamination from near-certain to requiring an arm to deliberately wander outside its world.
|
||||
A hard filesystem guarantee would require an OS sandbox, which is out of scope.
|
||||
- A live-target fixture that seeds and tears down an external resource per run is out of scope for now.
|
||||
|
||||
### Location and the test tree
|
||||
|
||||
- Tests live in a top-level **`tests/` tree that mirrors `skills/` by full path**, so a skill's cosmetic nesting is mirrored and its tests sit at the same relative path.
|
||||
- Placing tests outside `skills/` is deliberate: each skill is packaged as its own derivation and placed into a project's skills directory, so bundling fixtures inside a skill would bloat every installation.
|
||||
The test tree is neither content nor integration output — it is repo infrastructure that is never packaged or placed.
|
||||
- A skill's tests sit as case directories at the mirror point, each holding a `case.md` and an optional fixture.
|
||||
|
||||
### The harness
|
||||
|
||||
- The harness runs **in-session as an AI script**, orchestrating the arms and judges as subagents via the workflow mechanism.
|
||||
- Running in-session is a billing decision: it stays on my **interactive subscription quota**, avoiding the separate metered automation credit pool that headless `claude -p` and the Agent SDK draw from since mid-2026.
|
||||
- The full per-arm metric set is recovered by **reading each arm-subagent's transcript file**, which records per-message token usage.
|
||||
Summing that usage yields raw tokens across the four components, the pricing-weighted cost-equivalent-token figure that the gitea-axi bench also reports, and an imputed dollar cost, while the turn count comes from the transcript's tool-call rounds.
|
||||
- The **mechanical work is done by a small committed program, not by agent reasoning** (see the deterministic core below).
|
||||
|
||||
### The deterministic core
|
||||
|
||||
- A committed program owns every mechanical, non-judgment step so the results are exact and reproducible: summing token usage across every arm and computing the cost figures, collapsing each trial of each comparison to WIN/TIE/LOSS, applying the two pass rules and the two verdicts, parsing `case.md`, rendering the HTML report, and appending to and trimming the run history.
|
||||
- It is **parameterized over the number of arms and the two comparisons**, so it handles both the three-arm and the degenerate two-arm run through one path.
|
||||
- It is a pure transform: given the collected run data (per-arm transcript usage, hard-assertion results, and the per-trial judge verdicts for both comparisons) it returns the results model, the rendered HTML, and the updated history.
|
||||
- It ships with the `benchmark-skill` skill and is invoked by that skill's prose, which keeps the agent confined to the judgment work — running arms and judging — that it is actually good at.
|
||||
|
||||
### The runner skill
|
||||
|
||||
- The runner is a **distributed skill** named `benchmark-skill`, auto-discovered and packaged like any other skill in this repo, and marked non-model-invocable so it is only ever run deliberately.
|
||||
- It is invoked as `/benchmark-skill <name>` to benchmark one skill, and with no argument to benchmark every skill.
|
||||
- Choosing the two-arm or three-arm shape is **fully automatic and carries no new invocation surface**: the harness diffs the skill directory against main's `HEAD` and adds the previous-version arm exactly when a released version exists to compare against.
|
||||
- Because it is distributed and relies only on the `tests/` convention, any repo following that convention can benchmark its own skills.
|
||||
- Validating the shape of the `tests/` tree (that a `case.md` parses, that a fixture referenced exists, that a test directory maps to a real skill, and reporting skills that have no tests) is the runner's own responsibility at run time, not a separate build-time check.
|
||||
|
||||
### The report
|
||||
|
||||
- The report uses a **layered shape**: a verdict scorecard on top with progressive disclosure beneath.
|
||||
A failed or regressed case auto-expands with its losing-trial evidence, while passing cases stay collapsed but can be opened.
|
||||
- Its sections run top to bottom.
|
||||
First a header with the **two verdict badges** (Efficacy and Regression), run metadata, and a jump-to-failure link shown only when something failed or regressed.
|
||||
Then the **trend ribbons**.
|
||||
Then the **verdict headline** reading both badges together.
|
||||
Then a **per-arm cost table**.
|
||||
And last the **cases** in stable authored order.
|
||||
- There are **two stacked trend ribbons** — an **Efficacy** ribbon and a **Regression** ribbon — each a net-margin (wins minus losses) sparkline over roughly the last seven runs, with verdict-colored per-run dots, the current run ringed, and a readout of the current net, the change versus the previous run, and how many recent runs were green.
|
||||
The Regression ribbon's dot is **absent for any run that was two-arm**.
|
||||
The Regression series is comparable across runs only while main is unchanged.
|
||||
Because the previous-version arm is always main's `HEAD`, merging the branch moves main and resets the meaningful regression history, which is accepted rather than normalized.
|
||||
The ribbons are what let an edit's real change be told apart from run-to-run noise.
|
||||
- The **cost table** has **three rows — new-skill, previous-version, no-skill** — each with turns, raw tokens, cost-equivalent tokens, and imputed cost, with a footnote that absolute cost is inflated by shared-context cache overhead and that the trustworthy signals are two ratios: **new-vs-no-skill** (what the skill costs over nothing) and **new-vs-old** (what this edit added or saved).
|
||||
The previous-version row drops out on a two-arm run.
|
||||
- Each case panel shows **both comparisons side by side**, each with its per-trial WIN/TIE/LOSS strip and its pass result.
|
||||
A case **auto-expands when either comparison fails or carries a flagged loss**, so a regression opens the panel even when efficacy is green, and the auto-expanded evidence shows the losing-trial output pair for whichever comparison failed.
|
||||
- **Cases render in stable authored order and are never reshuffled by verdict**, so their positions are learnable and diff cleanly across runs, with failures reached via auto-expansion and the header's jump link rather than by sorting them to the top.
|
||||
- A **clean run rests fully collapsed**, except that fragility is flagged **per badge**: a green Efficacy badge carries a "narrowest margin" chip on the case fewest trial-flips from failing efficacy, and a green Regression badge independently carries one on the case sitting at exactly one loss.
|
||||
A case that is fragile on both axes shows both chips, so a barely-green skill cannot look as safe as a clean sweep on either axis.
|
||||
- The report carries a **thin, constrained narrative layer** — a headline "so what" and per-case one-liners — under the rule that every sentence it writes must be backed by a value or judge quote visible on the same screen.
|
||||
The per-trial judge rationale and the mechanical "why it failed" are inherent evidence, not part of that narrative layer.
|
||||
|
||||
### All-skills output and files
|
||||
|
||||
- Benchmarking all skills produces an **index leaderboard** where each skill row carries **both badges (Efficacy and Regression)**.
|
||||
- The sort promotes **any red first**, with **regressions ordered above efficacy failures** — a regression means "you just broke something that was working," the more urgent signal while iterating — then fragile-but-passing skills, then clean green.
|
||||
A skill with no previous version reads not-applicable in its Regression cell.
|
||||
- The leaderboard links out to separate per-skill report files.
|
||||
- All report artifacts live under a git-ignored `tests/.reports/` directory, flat and keyed by unique skill name, since skill names are globally unique in this repo.
|
||||
- Per-skill HTML is **latest-only and overwritten each run**, because the longitudinal data lives in a per-skill history file to which the harness appends one summary line per run, capped at roughly the last fifty and trimmed oldest-first.
|
||||
- Each history line records **both the efficacy and the regression net-margin and pass/fail**, plus a flag for whether the run was two-arm or three-arm, so both ribbons can plot their own sparkline and correctly show a gap for any two-arm run.
|
||||
- That history is **local and ephemeral** — it lives inside the ignored reports directory and resets if that directory is cleaned — consistent with reports being transient artifacts.
|
||||
Promoting it to durable, committed history is a later, additive change.
|
||||
|
||||
## Testing Decisions
|
||||
|
||||
- The judgment half of this feature — the arms and the blind judges — is inherently non-deterministic and is **not unit-tested**.
|
||||
Its correctness is established by running the harness on a real skill and reading the report.
|
||||
- The one deterministic, testable seam is the **deterministic core**.
|
||||
A good test here exercises that core's external behavior as its caller observes it: fed **committed fixtures** — sample arm transcripts for all three arms, hard-assertion results, and the per-trial judge verdicts for both comparisons — it must produce the expected per-arm metrics, the expected per-case and per-skill verdicts on both axes, and a correct report including the two ribbons and the three-row cost table.
|
||||
The **two-arm run is retained as the degenerate "no previous version" case** in the same test, so both shapes are covered.
|
||||
It is tested at that single seam, not through the internals of the token-summing or the rendering.
|
||||
- This isolates exactly the error-prone arithmetic, counting, pass-rules, and templating that must not be re-derived by the agent, and it needs no LLM to run.
|
||||
- Prior art is `checks/shell-hook.nix`, which runs a produced artifact against a fixture project and asserts on the observed result.
|
||||
The `home-manager-module` check, which builds an output and inspects it, is the same style.
|
||||
- Whether this fixture test is also wired into `nix flake check` is left open and deferred, in keeping with the earlier decision not to add a build-time check for this feature.
|
||||
The test can exist and run without being a flake gate.
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- **Trigger-correctness** — whether a model-invokable skill fires on a realistic prompt.
|
||||
It is a distinct dimension the in-session harness cannot cleanly measure, and it is deferred to a separate eval.
|
||||
- **Format-conformance** as its own measure, which the efficacy comparison largely subsumes.
|
||||
- **Live-target fixtures** that seed and tear down an external resource per run.
|
||||
Only hermetic committed fixtures are supported for now.
|
||||
- **Headless or API execution** of the battery, which would draw the metered automation credit pool.
|
||||
A headless run of a single skill remains only a fallback for the unlikely case that transcript files lack usage data.
|
||||
- **Cost as a gate.** Cost is always reported and never fails a skill on its own.
|
||||
- **A pinned or blessed baseline, and comparison against arbitrary refs.** The previous version is always main's `HEAD`.
|
||||
Comparing against a marked version-to-beat or two chosen historical refs is a later, additive refinement.
|
||||
- **A `--no-regression` fast-path** for skipping the previous-version arm mid-edit.
|
||||
The shape is fully automatic for now.
|
||||
An opt-out flag can be added later.
|
||||
- **Durable or normalized trend history.** History is local and ephemeral, and the regression series is not normalized across a main move, until a later opt-in.
|
||||
- **Selectable, per-skill-kind arm construction.** The harness force-invokes every with-skill arm uniformly.
|
||||
- **Hard filesystem isolation of the arms.** Each arm runs in a fixture-only world with its working directory relocated off the repo root, which is soft isolation that an arm could defeat only by deliberately reaching for an absolute repo path.
|
||||
An OS sandbox that makes the skill genuinely unreachable is a later hardening.
|
||||
|
||||
## Further Notes
|
||||
|
||||
- The gitea-axi benchmark is the **methodology source**, not a template: its honest-baseline discipline, its per-arm comparison table, and its pricing-weighted cost-equivalent-token unit carry over, while its temperature-zero, headless, standalone-script shape does not, because a reasoning skill needs a realistic temperature and the billing reality pushes the run in-session.
|
||||
- The shared-context cache overhead means a per-arm absolute cost reads higher than a standalone run would.
|
||||
The inter-arm ratios are what stay trustworthy, and the report says so.
|
||||
- A prototype of the report was built during design and is the visual reference for the layered shape, the trend ribbons, the fragility chips, and the clean-versus-failing resting states.
|
||||
- "Fragile," for a green-run chip, means the passing case that is the fewest trial-flips from failing on that badge's axis — closest to dropping under three wins for efficacy, or over one loss for regression.
|
||||
- Because the previous version is always main's `HEAD`, a change committed directly to main stops being regression-testable once the working tree matches `HEAD`.
|
||||
Regression is designed for the branch-in-progress workflow (working tree versus released main).
|
||||
The deferred pinned-baseline feature would cover the commit-to-main case.
|
||||
Reference in New Issue
Block a user