chore: clear consumed spec and task scaffolding
Both spec docs and tasks 0001-0008 were mined into the personal wiki by /consume; delete the scaffolding it consumed, leaving the now-empty spec/ and tasks/ directories in place.
This commit is contained in:
@@ -1,53 +0,0 @@
|
||||
---
|
||||
spec: nix-skill-packaging
|
||||
---
|
||||
|
||||
## What to build
|
||||
|
||||
The foundational tracer bullet: a standalone Nix flake that packages each skill as an individually addressable derivation, plus the check that proves it.
|
||||
|
||||
The flake scaffolding uses `flake-utils.lib.eachSystem` over `x86_64-linux`, `aarch64-linux`, `aarch64-darwin`, with `nixpkgs` tracking unstable.
|
||||
An explicit system list is passed rather than `eachDefaultSystem`, whose default set includes the now-dropped `x86_64-darwin`.
|
||||
|
||||
The content tier recursively walks the `skills/` tree: any directory containing a `SKILL.md` is a skill (at any depth), and once found, its own subfolders are its assets rather than further skills. Directories without a `SKILL.md` are cosmetic organizational containers the walk descends through — the nesting never reaches the placement name and is not a selectable unit.
|
||||
|
||||
Each discovered skill becomes an individually addressable derivation exposed at `packages.<system>.<skill-name>`, built by an exposed `lib.mkSkill` builder. There is no `packages.<system>.default`. A skill is addressed by its name (leaf directory / frontmatter `name`), independent of its source path. The derivation satisfies the skill-derivation contract: `$out` contains `SKILL.md` at its root plus any assets, and the derivation carries its name as an eval-time attribute (e.g. `pname` / a passthru) so placement can form `.../skills/<name>` without import-from-derivation. Nothing in this tier references `.claude/skills` or anything Claude-Code-specific.
|
||||
|
||||
Names must be globally unique across the whole tree; a collision detected during the discovery walk is a hard eval-time build error with a clear message, not a warning.
|
||||
|
||||
Verified by the skill-build check under `nix flake check`, which builds every auto-discovered skill derivation and thereby transitively exercises the recursive walk, the `mkSkill` builder, the `SKILL.md`-at-`$out`-root contract, and the name-uniqueness hard error. Because the repo ships with no real skill content, this check drives fixture skills (à la gitea-axi's `runCommandLocal` fixtures), including a collision fixture that must fail the build.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [x] `flake.nix` iterates the three systems via `flake-utils.lib.eachSystem` (explicit list, not `eachDefaultSystem`) with `nixpkgs` unstable.
|
||||
- [x] The `skills/` tree is auto-discovered recursively: any directory containing a `SKILL.md` is a skill at any depth; its subfolders become its assets, not further skills.
|
||||
- [x] Directories without a `SKILL.md` are traversed as cosmetic containers only and never affect a skill's placement name.
|
||||
- [x] Each discovered skill is exposed at `packages.<system>.<skill-name>`; there is no `packages.<system>.default`.
|
||||
- [x] `lib.mkSkill` is exposed and turns a skill source directory into a conforming derivation.
|
||||
- [x] A built skill's `$out` contains `SKILL.md` at its root, and the derivation carries its name as an eval-time attribute (no import-from-derivation needed to read it).
|
||||
- [x] No content-tier derivation references `.claude/skills` or anything Claude-Code-specific.
|
||||
- [x] Two skills that resolve to the same name fail the build at eval with a clear collision message.
|
||||
- [x] `nix flake check` includes a skill-build check that builds every auto-discovered skill; a name-collision fixture makes it fail.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
Files: `flake.nix` (scaffolding + wiring), `lib/mk-skill.nix` (builder), `lib/discover.nix` (recursive walk + uniqueness), `checks/skill-build.nix` with fixtures under `checks/fixtures/`, and an empty `skills/.gitkeep` root.
|
||||
|
||||
- **Name source — leaf directory name, not frontmatter.**
|
||||
The spec's "What to build" phrases the name as "leaf directory / frontmatter `name`", but the acceptance criteria only ever require the leaf directory name (criteria 2 and 6).
|
||||
Discovery and `mkSkill` derive the name purely from the leaf directory (`mkSkill`'s `name` defaults to `builtins.baseNameOf src`).
|
||||
The two are equal by Claude Code convention; parsing `SKILL.md` YAML frontmatter at eval would add real complexity for no behavioural gain when they agree, so the frontmatter branch is deliberately not implemented.
|
||||
If divergence between directory name and frontmatter name ever needs enforcing, that is an additive validation for later.
|
||||
|
||||
- **Eval-time name attribute.**
|
||||
Carried as `passthru.skillName` (in addition to the derivation's own `name`), so the integration tier can form `.../skills/<name>` at eval time without import-from-derivation.
|
||||
|
||||
- **Empty `skills/` root.**
|
||||
The repo ships no real skill content, so `skills/` holds only a `.gitkeep` and `packages.<system>` is an empty set today.
|
||||
The `skill-build` check therefore drives fixture skills (a top-level skill, one under a cosmetic container with an asset subfolder, and one nested several containers deep) plus a separate two-skills-one-name collision fixture that `builtins.tryEval` confirms fails discovery at eval.
|
||||
|
||||
- **Real skills also folded into `checks` (beyond the literal criteria).**
|
||||
Each real discovered skill is added to `checks.<system>` alongside `skill-build`, so once skills land `nix flake check` builds each one and fails on a malformed skill (spec user story 17).
|
||||
This is empty today and is a natural extension of criterion 9's "builds every auto-discovered skill", not new scope.
|
||||
|
||||
- **Integration tier (home-manager module, dev-shell shellHook) is intentionally absent** — it belongs to tasks 0002 and 0003. No `home-manager` flake input is added yet for that reason.
|
||||
@@ -1,48 +0,0 @@
|
||||
---
|
||||
spec: nix-skill-packaging
|
||||
blocked-by: 0001-content-tier-skill-packaging
|
||||
---
|
||||
|
||||
## What to build
|
||||
|
||||
The global (every-project) placement output: a home-manager module that installs an operator's selected skills into `~/.claude/skills/`, plus the check that proves its composition.
|
||||
|
||||
Exposed as `homeModules.default`. The operator-facing option is `programs.agents.skills`, a `listOf package` with default `[]` (an empty list is a no-op). `agents` is deliberately an umbrella namespace with no shared `programs.agents.enable` — each sub-feature self-gates so the namespace stays a clean, mergeable surface another repo could extend. Selection is by derivation: consumers pass skill derivations pulled from `packages.<system>`, not name strings.
|
||||
|
||||
For each selected skill the module writes an individual `home.file` at `${claude-code.configDir}/skills/<name>` with `source = <skill derivation>` and `recursive = true`, gated on `programs.claude-code.enable`, reading `configDir` from the claude-code module. `recursive = true` is a hard requirement: it materializes `.../skills/<name>/` as a real directory of per-file symlinks so this module, the operator's own skills declarations, and self-placing tool modules coexist under one `skills/` tree. The module deliberately does not feed the single-valued `programs.claude-code.skills` option, which would collide with an operator already setting it.
|
||||
|
||||
Verified by the home-manager-module composition check under `nix flake check`: it instantiates `homeModules.default` under a sample home-manager configuration selecting a couple of skills with `programs.claude-code` enabled, builds the resulting home-files derivation, and asserts the per-skill `home.file`, `recursive = true`, the `claude-code.enable` gate, and `configDir` sourcing all compose as intended (direct analogue of gitea-axi's home-manager-module check).
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [x] `homeModules.default` is exposed.
|
||||
- [x] It defines `programs.agents.skills` as `listOf package` with default `[]`, and an empty list installs nothing.
|
||||
- [x] There is no shared `programs.agents.enable`; the skills feature self-gates.
|
||||
- [x] Selection is by derivation (skills pulled from `packages.<system>`), not by name string.
|
||||
- [x] Each selected skill is written as an individual `home.file` at `${claude-code.configDir}/skills/<name>` with `recursive = true`, sourcing `configDir` from the claude-code module.
|
||||
- [x] Placement is gated on `programs.claude-code.enable`; with it off, no skill files are written.
|
||||
- [x] The module does not set `programs.claude-code.skills`.
|
||||
- [x] The module composes with an operator's own skills declarations and self-placing tool modules under one `skills/` tree without collision.
|
||||
- [x] `nix flake check` includes a home-manager-module composition check that builds the home-files derivation for a sample selection and asserts the above.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
Files: `home-manager-module.nix` (the module, at the repo root, mirroring gitea-axi's placement), `checks/home-manager-module.nix` (the composition check), and `flake.nix` wiring (a `homeModules` output plus the check registration and the new `home-manager` input).
|
||||
|
||||
- **Placement name comes from `skillName`.**
|
||||
The module reads each derivation's `passthru.skillName` — the eval-time name attribute task 0001 established for exactly this purpose — to form `.../skills/<name>` without import-from-derivation.
|
||||
|
||||
- **`homeModules` exposes an `agents-skills` alias beside `default`.**
|
||||
The criteria only require `homeModules.default`; the named alias is an additive convenience mirroring gitea-axi's `homeModules` shape, and `default` points at it.
|
||||
|
||||
- **`home-manager` flake input follows this flake's nixpkgs.**
|
||||
It exists solely so `nix flake check` can evaluate the module against real home-manager; a consumer importing the module supplies their own home-manager and pkgs, so the input has no bearing on what they get.
|
||||
|
||||
- **Recursive placement is proven by the entry's type, not by file existence.**
|
||||
A `test -f skills/<name>/SKILL.md` follows symlinks and cannot distinguish a recursive per-file tree from one opaque symlink over the whole skill, so the check asserts the entry is a real directory (`test -d` and `! -L`).
|
||||
A mutation to `recursive = false` fails the check.
|
||||
|
||||
- **A non-default `configDir` scenario was added during review.**
|
||||
The other scenarios all run at the default `configDir`, so a module hardcoding `.claude/skills/` would have passed them identically.
|
||||
One configuration now sets a custom `configDir` and asserts placement follows it, closing the "configDir sourcing" leg of the check.
|
||||
A mutation hardcoding `.claude` fails the check.
|
||||
@@ -1,55 +0,0 @@
|
||||
---
|
||||
spec: nix-skill-packaging
|
||||
blocked-by: 0001-content-tier-skill-packaging
|
||||
---
|
||||
|
||||
## What to build
|
||||
|
||||
The per-project placement output: a dev-shell helper that drops a project's selected skills into its own `.claude/skills/`, plus the check that proves it.
|
||||
|
||||
Exposed as `lib.mkSkillsShellHook`, which takes a list of selected skill derivations and returns a shellHook string a project drops into its dev shell. On shell entry the hook symlinks each selected skill as a direct child of `<project>/.claude/skills/`, pointing into the Nix store. Selection is by derivation (skills pulled from `packages.<system>`), and the hook is a plain shell string relying on ambient POSIX tools — it needs no home-manager context.
|
||||
|
||||
Lifecycle is stateless with no manifest or state file: each entry first removes only the symlinks under `.claude/skills/` that point into the store (unambiguously "ours"), then recreates the current selection. This gives free stale-removal (deselect a skill → its symlink is gone next entry) and never touches real directories. Real (non-symlink) skill directories under `.claude/skills/` — hand-authored, project-private skills — are left untouched. The hook maintains a generated, self-ignoring `.claude/skills/.gitignore` listing the names it manages (and ignoring itself), so Nix-delivered symlinks stay out of git while hand-authored skills remain tracked. The committed record of a project's selection is its `flake.nix` selection list plus its pinned `flake.lock`.
|
||||
|
||||
Verified by the shellHook assertion check under `nix flake check`: it instantiates `lib.mkSkillsShellHook` with sample skills and asserts the produced hook is non-empty and references the expected store paths / skill names.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [x] `lib.mkSkillsShellHook` is exposed, takes a list of selected skill derivations, and returns a shellHook string.
|
||||
- [x] The hook is a plain shell string using ambient POSIX tools and needs no home-manager context.
|
||||
- [x] On entry, each selected skill is symlinked as a direct child of `<project>/.claude/skills/` pointing into the store.
|
||||
- [x] Each entry first removes only store-pointing symlinks under `.claude/skills/`, then recreates the current selection — deselecting a skill removes its symlink next entry.
|
||||
- [x] Real (non-symlink) skill directories under `.claude/skills/` are never touched.
|
||||
- [x] The hook maintains a generated, self-ignoring `.claude/skills/.gitignore` listing the names it manages, keeping store symlinks out of git while hand-authored skills stay tracked.
|
||||
- [x] No manifest or state file is used.
|
||||
- [x] `nix flake check` includes a shellHook assertion that instantiates the helper with sample skills and asserts the hook is non-empty and references the expected store paths / skill names.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
Files: `lib/mk-skills-shell-hook.nix` (the helper), `checks/shell-hook.nix` (the check), and the `flake.nix` wiring (a `mkSkillsShellHook` binding in the top-level `lib` output, plus the check registration).
|
||||
|
||||
- **Placement name comes from `skillName`.**
|
||||
The hook reads each derivation's `passthru.skillName` (from task 0001) to name its symlink, so the selection stays by-derivation and the name needs no `$out` read.
|
||||
|
||||
- **The whole hook runs in a subshell.**
|
||||
It defines helper variables and a function, so wrapping it in `( … )` keeps those from leaking into the operator's interactive shell.
|
||||
Verified by a real `bash` run: after sourcing, `_mkskills_managed` and the link function are absent from the parent shell.
|
||||
|
||||
- **Name collision with a hand-authored skill: the real directory wins.**
|
||||
If a selected skill's name already exists as a real (non-symlink) directory, the hook skips it with a stderr warning and leaves the directory in place.
|
||||
A skipped skill is deliberately kept out of the managed set, so it is never added to the `.gitignore` and the hand-authored directory stays tracked.
|
||||
Only skills the hook actually links are listed in the `.gitignore`.
|
||||
|
||||
- **Cleanup is precise to store-pointing symlinks.**
|
||||
The stale-removal sweep removes a direct-child symlink only when `readlink` shows it targets the Nix store, so a symlink a project points elsewhere is left alone.
|
||||
The check's fixture includes such a non-store symlink and asserts it survives.
|
||||
|
||||
- **The check executes the hook, beyond the literal testing decision.**
|
||||
The task's testing decision only requires asserting the hook string is non-empty and references the expected store paths and names.
|
||||
The check does that, then additionally sources the hook against a fixture project across two successive selections and asserts placement, stateless stale-removal, the collision guard, the non-store-symlink survival, the `.gitignore` contents, and that no state file appears.
|
||||
This is a strict superset that proves the spec-mandated lifecycle (criteria 3–7) rather than trusting a string match, and each added assertion maps to a stated requirement.
|
||||
|
||||
- **POSIX-tool notes.**
|
||||
The hook relies on `readlink` to classify symlinks; it is not in POSIX but is present on every Nix-based machine this targets, and there is no clean substitute.
|
||||
The link step avoids the non-POSIX `ln -n` overwrite flag by removing any residual symlink first, then using a plain `ln -s`.
|
||||
A single quote in a skill name would break the generated shell string; skill names are kebab-case by convention, so hardening against that is left out as out of scope.
|
||||
@@ -1,117 +0,0 @@
|
||||
---
|
||||
spec: skill-benchmarking
|
||||
blocked-by: 0001-content-tier-skill-packaging
|
||||
---
|
||||
|
||||
## What to build
|
||||
|
||||
The foundational tracer bullet: the thinnest complete path that benchmarks one skill's **efficacy against a no-skill baseline** and produces a report.
|
||||
Running `/benchmark-skill <name>` on a real skill executes its cases and returns an HTML report carrying a single **Efficacy verdict**.
|
||||
|
||||
This slice establishes three things end to end.
|
||||
|
||||
**The `tests/` convention.**
|
||||
Tests live in a top-level `tests/` tree that mirrors `skills/` by full path, so a skill's cosmetic nesting is mirrored and its tests sit at the same relative path.
|
||||
Tests are deliberately outside `skills/` so they are never packaged or placed when a skill is installed.
|
||||
A skill's tests are case directories at the mirror point, each holding a `case.md` and an optional fixture.
|
||||
A `case.md` carries a one-line `description`, a `## Prompt` given identically to every arm, an optional `## Seed` transcript in role-tagged form for conversation-driven skills, an optional `## Hard assertions` block of executable shell predicates, and a `## Soft criteria` list with at least one entry.
|
||||
Every case ships a hermetic, committed fixture, and each arm-and-trial combination runs against a fresh copy of it so writes never leak between runs.
|
||||
|
||||
The `case.md` shape (from the case-format decision) is the authoring contract:
|
||||
|
||||
```
|
||||
---
|
||||
description: <one-line scenario>
|
||||
---
|
||||
## Prompt
|
||||
<the realistic user request, given identically to every arm>
|
||||
|
||||
## Seed (optional; conversation-based skills only)
|
||||
**User:** ...
|
||||
**Assistant:** ...
|
||||
|
||||
## Hard assertions (executable; $OUTPUT = arm's final message, $WORLD = its fixture copy)
|
||||
```sh
|
||||
test -f "$WORLD/review.md"
|
||||
grep -qE '<pattern>' "$OUTPUT"
|
||||
```
|
||||
|
||||
## Soft criteria
|
||||
- <a statement the judge grounds the comparisons on>
|
||||
```
|
||||
|
||||
**The `benchmark-skill` runner skill.**
|
||||
A distributed skill, auto-discovered and packaged like any other skill in this repo, marked non-model-invocable so it only ever runs deliberately, invoked `/benchmark-skill <name>`.
|
||||
It runs in-session as an AI script, orchestrating the arms and the judge as subagents via the workflow mechanism, so the whole battery stays on the interactive subscription quota rather than the metered automation credit pool.
|
||||
It validates the shape of the target skill's `tests/` tree at run time — that each `case.md` parses, that a referenced fixture exists, that the test directory maps to a real skill — and reports a skill that has no tests.
|
||||
For each case it runs a **new-skill arm** (the working tree) and a **no-skill baseline arm**.
|
||||
The new arm is force-invoked: its subagent is pointed at the skill's directory and told to use it, reading that skill's own file and any assets it references, so the real skill machinery is exercised.
|
||||
The baseline arm receives the bare prompt with the skill absent from its context, the honest counterfactual of the skill not existing.
|
||||
Every arm is given the identical realistic prompt authored once and arm-agnostically.
|
||||
Each case runs 5 paired trials at a realistic temperature, configurable per skill, with a per-skill temperature-zero override for a skill that wraps a mechanical task, and with no reuse of arms or judgments across arms or runs.
|
||||
Per trial, one blind judge subagent decides between the two arms' outputs shown as unlabelled A and B with the order randomized, grounded on the case's soft criteria rather than free-forming its own standard, and instructed to discount mere length and formatting differences.
|
||||
The hard-assertion gate runs against the new arm only, with `$OUTPUT` the path to the arm's captured final message and `$WORLD` the path to its fresh fixture copy; a non-zero exit fails the assertion, and a failed hard assertion fails the case outright regardless of the head-to-head.
|
||||
|
||||
**The deterministic core.**
|
||||
A small committed program, shipped with the `benchmark-skill` skill and invoked by its prose, owns every mechanical, non-judgment step so the numbers are exact and reproducible rather than re-derived by the agent each run.
|
||||
It is a pure transform: given the collected run data (per-arm transcript usage, hard-assertion results, and the per-trial judge verdicts) it returns the results model and the rendered HTML.
|
||||
The run-history file and the trend ribbons that consume it are deferred to a later slice (0006); this slice's report is latest-only.
|
||||
It is structured parameterized over the number of arms and the comparisons from the start, but this slice exercises only the two-arm efficacy shape.
|
||||
It parses `case.md`, sums each arm's transcript token usage into the metric set — raw tokens across the four components, the pricing-weighted cost-equivalent-token figure, an imputed dollar cost, and the turn count from the transcript's tool-call rounds — collapses each efficacy trial to WIN, TIE, or LOSS for the new arm (a tie is a non-win), and applies the efficacy pass rule.
|
||||
Efficacy passes for a case when wins ≥ 3 and losses ≤ 1 across the 5 trials, and any loss is flagged for human review.
|
||||
The **Efficacy verdict** for the skill is green when every case passes efficacy.
|
||||
It renders a self-contained HTML report: an Efficacy verdict badge and run metadata at the top, a per-arm cost table (new-skill and no-skill rows, each with turns, raw tokens, cost-equivalent tokens, and imputed cost, footnoted that absolute cost is inflated by shared-context cache overhead so the trustworthy signal is the new-vs-no-skill ratio), and the cases in stable authored order.
|
||||
Cost is reported alongside quality but never gates a verdict.
|
||||
|
||||
The deterministic core is unit-tested at its single external seam, in the style of `checks/shell-hook.nix` and `checks/home-manager-module.nix`: fed committed fixtures (sample new-skill and no-skill arm transcripts, hard-assertion results, and per-trial judge verdicts) it must produce the expected per-arm metrics, the expected per-case and skill-level Efficacy verdict, and a correct report.
|
||||
The test needs no LLM to run.
|
||||
|
||||
All report artifacts live under a git-ignored `tests/.reports/` directory.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [x] A `tests/` tree at the repo top level mirrors `skills/` by full path; a skill's tests sit as case directories at the mirror point, each with a `case.md` and optional committed fixture.
|
||||
- [x] `case.md` parses its `description`, `## Prompt`, optional `## Seed`, optional `## Hard assertions`, and `## Soft criteria` (at least one) per the authoring contract.
|
||||
- [x] `benchmark-skill` is an auto-discovered, packaged, non-model-invocable distributed skill invoked as `/benchmark-skill <name>`.
|
||||
- [x] The runner validates the target's `tests/` tree at run time (each `case.md` parses, referenced fixtures exist, the directory maps to a real skill) and reports a skill with no tests.
|
||||
- [x] For each case the runner orchestrates a force-invoked new-skill arm and a skill-absent no-skill baseline arm as subagents via the workflow mechanism, entirely in-session.
|
||||
- [x] Every arm receives the identical prompt; each arm-and-trial runs against a fresh copy of the case fixture; a conversation-driven case injects its `## Seed` transcript as the subagent's prior context.
|
||||
- [x] Each case runs 5 paired trials at a per-skill-configurable realistic temperature, with a per-skill temperature-zero override, and no reuse of arms or judgments across arms or runs.
|
||||
- [x] Per trial, one blind judge subagent compares the two arms as randomized unlabelled A/B, grounded on the case's soft criteria and instructed to discount length and formatting.
|
||||
- [x] The hard-assertion gate runs against the new arm only with `$OUTPUT` and `$WORLD` provided; a non-zero exit fails the assertion and a failed assertion fails the case outright.
|
||||
- [x] The deterministic core is a committed, pure-transform program shipped with the skill, structured parameterized over arms and comparisons though exercised here two-arm.
|
||||
- [x] The core sums per-arm transcript usage into raw tokens, cost-equivalent tokens, imputed dollar cost, and turn count, and collapses each efficacy trial to WIN/TIE/LOSS (tie = non-win).
|
||||
- [x] The core applies the efficacy pass rule (wins ≥ 3 and losses ≤ 1), flags any loss, and yields a skill-level Efficacy verdict that is green when every case passes.
|
||||
- [x] The core renders a self-contained HTML report with the Efficacy badge, run metadata, a two-row per-arm cost table with the cache-overhead footnote, and cases in stable authored order; cost never gates.
|
||||
- [x] A fixture-driven unit test (no LLM), in the style of the existing `checks/`, feeds sample transcripts, hard-assertion results, and judge verdicts to the core and asserts the metrics, the Efficacy verdict, and the report.
|
||||
- [x] Report artifacts are written under a git-ignored `tests/.reports/` directory.
|
||||
- [x] At least one existing real skill carries an authored `tests/` tree (a `case.md` with a fixture), and a live `/benchmark-skill <name>` run exercises it end to end and produces its efficacy report — proving the tracer bullet actually fires.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
Files: `skills/benchmark-skill/SKILL.md` (the in-session runner prose), `skills/benchmark-skill/CASE-FORMAT.md` (the `case.md` authoring contract), `skills/benchmark-skill/core/benchmark_core.py` (the deterministic core), `checks/benchmark-core.nix` + `checks/fixtures/benchmark/run-bundle.json` (the fixture-driven no-LLM check, wired into `flake.nix`), `tests/skills/axi-review/basic-cli-review/` (the real authored test tree with a committed `fixture/greet`), and a `tests/.reports/` `.gitignore` rule.
|
||||
|
||||
- **Core language — Python 3.**
|
||||
The deterministic core is Python stdlib only, chosen because it renders HTML and parses JSON far more cleanly than the repo's shell/jq idiom, keeping the error-prone arithmetic and templating in committed, tested code rather than re-derived by the agent.
|
||||
This machine has no `python3` on its global PATH, so the check pulls `pkgs.python3` as a build input and the live run invokes the core via `nix shell nixpkgs#python3`.
|
||||
A distributed consumer repo is expected to provide `python3` at run time.
|
||||
|
||||
- **Live run — baseline contamination is a real harness limitation.**
|
||||
The live `/benchmark-skill axi-review` ran all 5 paired trials end to end and produced `tests/.reports/axi-review.html` with real per-arm token metrics recovered from the arm transcripts, proving the tracer bullet fires.
|
||||
The Efficacy verdict came out **red** (new arm won 2/5, baseline won 3/5).
|
||||
That red is substantially an artifact: the no-skill baseline subagent, running in this repo on disk, discovered and used the skill's `AXI-PRINCIPLES.md` rubric (its own output states it "judged each principle against the canonical rubric in `AXI-PRINCIPLES.md`"), so it was not a clean no-skill counterfactual.
|
||||
This is a limitation of in-session benchmarking — a subagent can read repo-resident skill assets — not a defect in the core, the pass rule, or the skill's design.
|
||||
Hardening the baseline arm's filesystem isolation is a follow-up.
|
||||
|
||||
- **Trial temperature.**
|
||||
Subagent temperature is not directly settable through the workflow's `agent()` surface, so the arms ran at the session's default (realistic) temperature.
|
||||
The per-skill temperature and the temperature-zero override remain documented knobs in `SKILL.md` for a future headless path.
|
||||
|
||||
- **Parameterization kept per spec.**
|
||||
The core carries the arms list and comparisons list (each with a `rule`) from the start, per "structured parameterized over the number of arms and the comparisons from the start", though this slice exercises only the two-arm efficacy shape and the single efficacy pass rule.
|
||||
The regression arm and its inverted rule arrive in task 0005.
|
||||
|
||||
- **Review follow-through.**
|
||||
A per-run pricing override was removed as speculative config.
|
||||
Documented-standard breaches (semicolons in authored prose/comments, a comment sentence-per-line) were fixed.
|
||||
`CASE-FORMAT.md`'s all-caps name is retained to match the repo's existing skill-companion-doc convention (`AXI-PRINCIPLES.md`, `REPORT-FORMAT.md`).
|
||||
@@ -1,77 +0,0 @@
|
||||
---
|
||||
spec: skill-benchmarking
|
||||
blocked-by: 0008-arm-isolation-hardening
|
||||
---
|
||||
|
||||
## What to build
|
||||
|
||||
The second head-to-head: a **previous-version arm** that turns a run into a three-arm experiment and adds a **Regression verdict** beside the Efficacy one, so an edit that silently made a trusted skill worse is caught rather than passing green.
|
||||
|
||||
The previous version is always main's `HEAD` — the repo's default branch, resolved rather than hardcoded to the literal name "main" — and the new version is the working tree.
|
||||
Anchoring to the released state on main makes the regression question always "did my in-progress edit degrade from the trusted version," which is the moment of real risk.
|
||||
The third arm is added **fully automatically and carries no new invocation surface**: the runner diffs the skill's directory against main's `HEAD` and adds the previous-version arm exactly when the two differ.
|
||||
A brand-new skill absent from main, or a skill unchanged from `HEAD`, has no meaningful previous version, so the run degrades to the two-arm efficacy-only shape.
|
||||
|
||||
The previous-version arm's skill is materialized by checking out just the skill's directory at main's `HEAD` into a temp path, since each skill is a self-contained derivation that does not need the rest of the repo.
|
||||
That temp materialization is cleaned up after the run, like the fresh fixture copies.
|
||||
The previous-version arm is force-invoked identically to the new arm, pointed at this temp checkout instead of the new arm's copy of the working-tree directory, so the skill version is the only difference between them.
|
||||
Cases are held fixed to the working tree — all arms run against today's prompt, fixture, and expectations — so the skill version is the only variable differing between the new and previous-version arms.
|
||||
The hard-assertion gate stays on the new arm only, and the blind judge and the case's soft criteria are reused unchanged for the regression comparison; only the pair of outputs handed to the judge differs.
|
||||
|
||||
Two blind head-to-heads are now drawn per trial: **efficacy** pairs the new arm against the no-skill arm, and **regression** pairs the new arm against the previous-version arm.
|
||||
Trial *i*'s new-skill output is judged against trial *i*'s no-skill output for efficacy and against trial *i*'s previous-version output for regression.
|
||||
|
||||
The regression pass rule inverts, because holding quality steady is the goal.
|
||||
Per trial, the new-vs-old head-to-head collapses to WIN, TIE, or LOSS for the new arm, where a TIE means "as good as the previous version" (a success), a LOSS is the regression being hunted, and a WIN is a bonus improvement.
|
||||
Per case, regression passes when losses ≤ 1 across the 5 trials, with no wins floor — ties and wins both count as non-regressions — and any loss is flagged for human review.
|
||||
Dropping the wins floor means a deliberate no-op edit that ties the previous version passes rather than failing merely for tying.
|
||||
|
||||
The deterministic core is exercised across its full three-arm, two-comparison path (the same parameterized program from 0004, now run with both comparisons rather than re-architected).
|
||||
It gains the regression pass rule and a skill-level **Regression verdict**, green when no case regressed.
|
||||
The per-run result now carries the regression net-margin and a two-arm/three-arm flag, but persisting those into the durable per-run history line is deferred to the trend-history slice (0006), which owns history persistence; this slice's report remains latest-only with no ribbons.
|
||||
The two verdicts are reported side by side as independent badges, because their four crossings carry genuinely different meanings, and the report headline reads both badges together into one "so what."
|
||||
On a two-arm run with no previous version the Regression verdict reads not-applicable.
|
||||
The cost table gains a **previous-version row** (turns, raw tokens, cost-equivalent tokens, imputed cost), which drops out on a two-arm run, and the footnote names the two trustworthy ratios: new-vs-no-skill and new-vs-old.
|
||||
Each case panel shows both comparisons side by side, each with its per-trial WIN/TIE/LOSS strip and its pass result, and auto-expands when either comparison fails or carries a flagged loss — so a regression opens the panel even when efficacy is green — surfacing the losing-trial output pair for whichever comparison failed.
|
||||
|
||||
The core's fixture unit test is extended so the same test covers both shapes: the three-arm run (sample transcripts for all three arms and per-trial judge verdicts for both comparisons) asserting the per-arm metrics, both per-case and skill-level verdicts, and a report with the three-row cost table, and the two-arm run retained as the degenerate "no previous version" case.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [x] The previous version resolves to the default branch's `HEAD` dynamically, never a hardcoded "main" literal, and the new version is the working tree.
|
||||
- [x] The third arm is added automatically, with no new flag or argument, exactly when the skill directory differs from `HEAD`; otherwise the run stays two-arm efficacy-only.
|
||||
- [x] The previous-version arm materializes by checking out just the skill's directory at `HEAD` into a temp path, force-invoked against it identically to the new arm, and the temp path is cleaned up after the run.
|
||||
- [x] All arms run against the working-tree case (prompt, fixture, expectations), so the skill version is the only variable between the new and previous-version arms.
|
||||
- [x] The hard-assertion gate remains new-arm-only; the same blind judge and soft criteria are reused for the regression comparison, differing only in the output pair.
|
||||
- [x] Per trial, efficacy is judged new-vs-no-skill and regression new-vs-previous, using that trial's own outputs with no cross-arm reuse.
|
||||
- [x] The regression pass rule passes a case when losses ≤ 1 across 5 trials with no wins floor (ties and wins are non-regressions), flagging any loss.
|
||||
- [x] The core, unchanged in structure, runs the full three-arm two-comparison path and yields a skill-level Regression verdict green when no case regressed.
|
||||
- [x] The report shows Efficacy and Regression as two independent badges with a headline reading both together; Regression reads not-applicable on a two-arm run.
|
||||
- [x] The cost table gains a previous-version row that drops out on a two-arm run, with a footnote naming the new-vs-no-skill and new-vs-old ratios.
|
||||
- [x] Each case panel shows both comparisons side by side with per-trial WIN/TIE/LOSS strips, auto-expanding on any fail or flagged loss and showing the losing-trial output pair for the failed comparison.
|
||||
- [x] The core's fixture unit test covers both the three-arm and the degenerate two-arm shapes in the same test, asserting metrics, both verdicts, and the three-row cost table.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The deterministic half — the pass rules, the two verdicts, the net margins, and the report — lives in `core/benchmark_core.py` and is fully covered by the extended `checks/benchmark-core.nix` fixture test.
|
||||
The orchestration half — resolving `HEAD`, diff-gating the third arm, materializing the previous-version checkout, and drawing the two head-to-heads (criteria 1–6) — is specified in the runner's `SKILL.md` prose, since that half runs in-session as an AI script and, per the spec's Testing Decisions, is not unit-tested but established by running the harness.
|
||||
|
||||
- **Branching deviated from the standard `/implement` flow, at the user's direction.**
|
||||
Task 0005 is blocked by 0008, which stacks on 0004, and none of that had been merged to `main` — so branching 0005 off a fresh `main` would have lost the entire benchmark foundation.
|
||||
`main` was fast-forwarded to the `task-0004` stack tip (a clean linear superset) and pushed, then `task-0005` was branched off it.
|
||||
- **The hard-assertion gate feeds Efficacy only, not Regression.**
|
||||
The spec defines the regression pass rule purely as losses ≤ 1 and never mentions the gate, and the gate exists to catch the *current* skill emitting malformed output — an absolute property of the new arm, whereas regression is relative to the previous version.
|
||||
So a hard failure fails the case's Efficacy axis outright while its Regression axis is judged purely on the head-to-head.
|
||||
The three-arm fixture's `hard-gate-fail` case pins this: Efficacy fails on the gate, Regression passes.
|
||||
- **The run bundle gained a per-trial `outputs` map (arm id → final message).**
|
||||
0004 carried no output text; the losing-trial evidence pair this slice requires needs it, so the core reads `outputs` to surface the new arm's output beside the one it lost to, only for flagged-loss trials of a failed or flagged comparison.
|
||||
- **Arm order is canonical: new-skill, previous-version, no-skill.**
|
||||
The cost table renders arms in bundle order, and a Spec-axis review caught that the initial new/no-skill/previous order put the previous-version row last rather than in the spec's stated middle position.
|
||||
The fixtures and the `SKILL.md` example now emit arms in the spec order.
|
||||
- **Comparisons render side by side via a CSS grid.**
|
||||
A Spec-axis review caught that the two comparisons were stacked vertically rather than laid out side by side as the report shape requires; a `.cmps` grid container now places them beside each other, wrapping to one column on a narrow viewport.
|
||||
- **Arm shape is derived from the presence of a regression comparison, not the arm count**, so a run is two-arm exactly when no regression comparison was drawn.
|
||||
- **The two committed fixtures replace 0004's single `run-bundle.json`.**
|
||||
`three-arm-bundle.json` is the primary shape and `two-arm-bundle.json` is retained as the degenerate "no previous version" case, both exercised by the one test.
|
||||
They are machine-generated for accuracy; the generator is not committed.
|
||||
- **A live three-arm `/benchmark-skill` run is the natural follow-up validation** (as 0008 did for isolation) but is not gated by this slice's acceptance criteria, so it was not run here.
|
||||
@@ -1,59 +0,0 @@
|
||||
---
|
||||
spec: skill-benchmarking
|
||||
blocked-by: 0005-regression-arm-and-two-verdicts
|
||||
---
|
||||
|
||||
## What to build
|
||||
|
||||
The longitudinal layer that lets a real change be told apart from run-to-run noise, plus the fragility flags that stop a barely-green skill from masquerading as robust.
|
||||
|
||||
**History.**
|
||||
The deterministic core appends one summary line per run to a per-skill history file and trims it, oldest-first, at roughly the last fifty.
|
||||
Each line records both the efficacy and the regression net-margin (wins minus losses) and pass/fail, plus a flag for whether the run was two-arm or three-arm, so both ribbons can plot their own sparkline and correctly show a gap for any two-arm run.
|
||||
The history is local and ephemeral: it lives inside the git-ignored reports directory and resets if that directory is cleaned, consistent with reports being transient artifacts.
|
||||
|
||||
**Trend ribbons.**
|
||||
The report grows two stacked trend ribbons — an Efficacy ribbon and a Regression ribbon — each a net-margin sparkline over roughly the last seven runs, with verdict-colored per-run dots, the current run ringed, and a readout of the current net, the change versus the previous run, and how many recent runs were green.
|
||||
The Regression ribbon's dot is absent for any run that was two-arm.
|
||||
The Regression series is comparable across runs only while main is unchanged; because the previous-version arm is always main's `HEAD`, merging the branch moves main and resets the meaningful regression history, which is accepted rather than normalized.
|
||||
|
||||
**Fragility chips and the clean resting state.**
|
||||
A clean run rests fully collapsed, and fragility is flagged per badge.
|
||||
A green Efficacy badge carries a "narrowest margin" chip on the case fewest trial-flips from failing efficacy — closest to dropping under three wins.
|
||||
A green Regression badge independently carries a chip on the case sitting at exactly one loss.
|
||||
A case fragile on both axes shows both chips, so a barely-green skill cannot look as safe as a clean sweep on either axis.
|
||||
|
||||
The core's fixture unit test is extended to cover the history append-and-trim, the two ribbons' rendering including the two-arm gap in the regression series, and the per-badge fragility chips.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [x] The core appends one summary line per run to a per-skill history file and trims oldest-first at roughly fifty lines.
|
||||
- [x] Each history line records the efficacy and regression net-margins and pass/fail plus a two-arm/three-arm flag.
|
||||
- [x] The history file lives inside the git-ignored reports directory and is treated as ephemeral (resets when that directory is cleaned).
|
||||
- [x] The report renders two stacked ribbons (Efficacy and Regression), each a net-margin sparkline over ~7 runs with verdict-colored dots, the current run ringed, and a net/delta/green-count readout.
|
||||
- [x] The Regression ribbon shows no dot for any run that was two-arm.
|
||||
- [x] A clean run rests fully collapsed.
|
||||
- [x] A green Efficacy badge carries a narrowest-margin chip on the case fewest trial-flips from failing efficacy; a green Regression badge independently carries a chip on the case at exactly one loss; a both-axes-fragile case shows both.
|
||||
- [x] The core's fixture unit test covers the history append-and-trim, both ribbons (including the two-arm gap), and the per-badge fragility chips.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The whole slice lives in the deterministic core, `core/benchmark_core.py`, and is fully covered by the extended `checks/benchmark-core.nix` fixture test.
|
||||
The runner's `SKILL.md` step 6 gains the `--history tests/.reports/<name>.history.jsonl` argument and a description of the append-trim and the ribbons, since the orchestration half runs in-session and is established by running the harness rather than unit-tested.
|
||||
|
||||
- **The standard `/implement` branching flow was followed this time.**
|
||||
Task 0005's PR had merged to `main`, so the blocker was reachable: `main` was fast-forwarded to `origin/main` and `task-0006` branched off it, without the manual stack-tip fast-forward 0005 needed.
|
||||
- **History persistence is opt-in via `--history` and is the core's only side effect.**
|
||||
Without the flag the core stays a pure transform and renders no ribbons, so the 0005-era report shape is unchanged.
|
||||
The history is JSON-lines — one run summary object per line — read, appended, trimmed to fifty, and rewritten.
|
||||
- **The ribbon delta compares against the previous run that carries a value on that axis, not the literal previous line.**
|
||||
On the regression axis this skips a two-arm gap rather than blanking the change readout, consistent with the spec accepting regression discontinuity across a main move.
|
||||
- **The green-count readout shows `green/applicable`, not a bare green count.**
|
||||
The denominator is the number of runs in the window that have a value on that axis, so a regression readout excludes two-arm runs it cannot score, which reads more honestly than counting them against the total.
|
||||
- **The sparkline draws a faint dashed zero baseline** when the plotted window straddles zero, as a readability aid for telling a positive net-margin run from a negative one.
|
||||
- **Two committed fixtures were added, machine-generated for accuracy with the generator not committed** (the 0005 precedent).
|
||||
The existing red fixtures carry no green axis and so exercise no chips, so `clean-bundle.json` (fully clean, green/green) pins the fully-collapsed resting state and the lone narrowest-margin efficacy chip, and `fragile-bundle.json` (green but fragile) pins the independent per-badge chips and the both-axes case carrying both.
|
||||
- **The efficacy chip is a single case, the regression chip is every case at exactly one loss.**
|
||||
This asymmetry follows the wording: "narrowest margin" is comparative and picks one case (ties broken by authored order), while "the case sitting at exactly one loss" is an absolute condition any number of cases can meet.
|
||||
- **A case at exactly one regression loss both auto-expands and carries its chip.**
|
||||
"A clean run rests fully collapsed" holds because a clean run has zero losses, so nothing trips the 0005 flagged-loss auto-expand — the chip is what still flags fragility on a case that has no loss to open it (a green efficacy case sitting at exactly three wins).
|
||||
@@ -1,46 +0,0 @@
|
||||
---
|
||||
spec: skill-benchmarking
|
||||
blocked-by: 0006-trend-history-and-ribbons
|
||||
---
|
||||
|
||||
## What to build
|
||||
|
||||
The batch view: `/benchmark-skill` with no argument benchmarks every skill and produces an index leaderboard, so an author knows at a glance which skills are green and which regressed.
|
||||
|
||||
Invoked with no argument, the runner benchmarks every skill that follows the `tests/` convention and renders an index leaderboard.
|
||||
Each skill row carries both badges, Efficacy and Regression, and a skill with no previous version reads not-applicable in its Regression cell.
|
||||
The sort promotes any red first, with regressions ordered above efficacy failures — a regression means "you just broke something that was working," the more urgent signal while iterating — then fragile-but-passing skills, then clean green.
|
||||
The fragile-but-passing tier reuses the per-badge fragility signal introduced with the trend layer (0006) rather than recomputing it here.
|
||||
The leaderboard links out to the separate per-skill report files produced by a single-skill run.
|
||||
|
||||
The per-skill report files and the index live flat under the `tests/.reports/` directory (its git-ignore established in 0004), keyed by unique skill name, since skill names are globally unique in this repo.
|
||||
Per-skill HTML is latest-only and overwritten each run, because the longitudinal data already lives in the per-skill history file.
|
||||
|
||||
The core's fixture unit test is extended to cover the leaderboard rendering: the sort order across a mix of regressed, efficacy-failed, fragile, and clean skills, the not-applicable Regression cell for a skill with no previous version, and the links to per-skill files.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [x] `/benchmark-skill` with no argument benchmarks every skill following the `tests/` convention and renders an index leaderboard.
|
||||
- [x] Each leaderboard row carries both the Efficacy and Regression badges; a skill with no previous version reads not-applicable in its Regression cell.
|
||||
- [x] The sort promotes any red first with regressions above efficacy failures, then fragile-but-passing skills, then clean green.
|
||||
- [x] The leaderboard links out to the separate per-skill report files.
|
||||
- [x] The per-skill report files and the leaderboard index live flat under `tests/.reports/` (its git-ignore established in 0004), keyed by unique skill name.
|
||||
- [x] Per-skill HTML is latest-only and overwritten each run.
|
||||
- [x] The core's fixture unit test covers the leaderboard sort order, the not-applicable Regression cell, and the per-skill links.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The leaderboard is a second pure transform in the deterministic core, `core/benchmark_core.py`, fully covered by the extended `checks/benchmark-core.nix` fixture test.
|
||||
The in-session orchestration half — enumerate every skill with tests, run steps 1–6 for each, then render the index — is documented in `SKILL.md` step 7 rather than unit-tested, following the 0006 split where the workflow half is established by running the harness, not the fixture test.
|
||||
Per the invocation, the benchmark harness itself was not run.
|
||||
|
||||
- **The leaderboard consumes per-skill results JSONs, not bundles, so the core stays a pure transform.**
|
||||
The single-skill flow already writes `tests/.reports/<name>.results.json`; the batch flow feeds every one of those to `--leaderboard`, which sorts and renders the index to `tests/.reports/index.html`.
|
||||
- **The CLI gains a `--leaderboard` mode that reuses `--json`/`--html`.**
|
||||
The positional argument was widened from a single `bundle` to `inputs` (`nargs="+"`) with an explicit count guard, so the single-skill contract `core.py <bundle> --json … --html …` is unchanged and every existing call site still works.
|
||||
- **The fragile-but-passing tier reuses the 0006 per-badge chips rather than recomputing margins.**
|
||||
The regression chip already means "one loss from regressing." A green run always chips its narrowest efficacy case, so chip presence alone cannot tell a barely-green skill from a roomy one; the efficacy chip therefore now carries its win count, and the leaderboard reads that count against the existing `EFFICACY_WINS_FLOOR` to decide efficacy fragility — the spec's own definition, "closest to dropping under three wins."
|
||||
- **Fragility is scoped to the fragile-but-passing tier (review finding).**
|
||||
A red skill carries no fragility chip even when a still-passing axis sits at its edge, so the chip stays the marker of tier 2 rather than leaking onto a red row.
|
||||
- **Two leaderboard fixtures are authored inline in the check, alongside the four scored models.**
|
||||
No committed bundle reaches the clean-green tier (the clean bundle's narrowest efficacy case sits on the floor, so it is itself fragile) or the efficacy-red-with-a-still-fragile-regression-axis crossing, so `robust-skill` and `leaky-skill` are synthesized as small results JSONs the way 0006 synthesized its 55-line history seed.
|
||||
@@ -1,66 +0,0 @@
|
||||
---
|
||||
spec: skill-benchmarking
|
||||
blocked-by: 0004-efficacy-benchmark-tracer-bullet
|
||||
---
|
||||
|
||||
## What to build
|
||||
|
||||
The isolation fix the tracer bullet flagged as a follow-up: give every arm a hermetic **fixture-only world** so the no-skill baseline stops discovering the skill on disk, turning that run's contaminated red into an honest counterfactual.
|
||||
|
||||
In 0004 the arms ran with the repo as their working directory.
|
||||
The no-skill baseline, solving its task inside the real repo, grepped up `skills/axi-review/AXI-PRINCIPLES.md` and graded against it — its own output admitted as much — so the measured "no-skill" arm was not the counterfactual of the skill not existing.
|
||||
The Efficacy verdict came out red as a direct artifact of that contamination rather than a real result.
|
||||
|
||||
Each arm-and-trial now runs in a fixture-only world.
|
||||
The arm subagent's working directory is its fresh fixture copy rather than the repo root, and nothing under `skills/` or the grading `tests/` tree sits on any path it explores from there.
|
||||
This is what makes the no-skill baseline honest, because it can no longer find and read the skill's assets, and it also stops any arm from reading its own case's soft criteria or hard assertions and tuning its answer to the bar it will be judged against.
|
||||
|
||||
A with-skill arm is handed its skill as an **isolated temp materialization** placed outside the fixture, and the subagent is pointed there to force-invoke it.
|
||||
For this slice's two-arm shape only the new-skill arm has one: a copy of the working-tree skill directory, so it reflects uncommitted edits and carries none of its repo surroundings.
|
||||
The no-skill arm is handed nothing.
|
||||
Each temp materialization is cleaned up after the run, like the fresh fixture copies.
|
||||
|
||||
Isolation here is deliberately soft.
|
||||
An in-session subagent shares the machine and could in principle reach the repo by absolute path, so the baseline is additionally instructed to stay within its working directory.
|
||||
Relocating each arm's world and giving that instruction moves contamination from near-certain to requiring an arm to deliberately wander outside its world.
|
||||
A hard filesystem guarantee would require an OS sandbox and is out of scope.
|
||||
|
||||
`$WORLD` binds to the arm's fixture copy — now also its working directory — and `$OUTPUT` to its captured final message, with the hard-assertion gate unchanged from 0004.
|
||||
The deterministic core and its fixture unit test are untouched, because isolation is a runner orchestration concern rather than a scoring one.
|
||||
|
||||
Correctness is established by re-running the harness on the real skill and reading the result.
|
||||
A live `/benchmark-skill axi-review` run confirms the baseline arm's transcript no longer references `AXI-PRINCIPLES.md` or any `skills/` or `tests/` path, and reports the now-uncontaminated Efficacy verdict that replaces 0004's documented-artifact red.
|
||||
|
||||
This slice establishes the "materialize a with-skill arm's skill into an isolated temp path" pattern that the previous-version arm reuses for the old skill version.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [x] Each arm-and-trial subagent runs with its working directory set to its fresh fixture copy, never the repo root.
|
||||
- [x] Nothing under `skills/` or the `tests/` tree is present on any relative path an arm explores from its world.
|
||||
- [x] The new-skill arm's skill is materialized into an isolated temp path outside the fixture — a copy of the working-tree directory that reflects uncommitted edits — and the subagent is pointed there to force-invoke it, with the temp materialization cleaned up after the run.
|
||||
- [x] The no-skill baseline is handed no skill materialization and is instructed to stay within its working directory.
|
||||
- [x] `$WORLD` binds to the arm's fixture copy and `$OUTPUT` to its captured final message, with the hard-assertion gate semantics unchanged from 0004.
|
||||
- [x] The deterministic core and its fixture unit test are unchanged, since isolation is an orchestration concern and not a scoring one.
|
||||
- [x] A live `/benchmark-skill axi-review` run confirms the baseline arm's transcript references no `AXI-PRINCIPLES.md`, `skills/`, or `tests/` path, and its Efficacy verdict reflects the honest counterfactual rather than the earlier contamination artifact.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
Changed `skills/benchmark-skill/SKILL.md` only: the world/materialization steps in §3, the `$WORLD` binding in §4, and the scratch cleanup in §6.
|
||||
`core/benchmark_core.py` and its check were left untouched, and the unchanged core was confirmed to still run clean on its committed fixture.
|
||||
|
||||
- **Worlds are materialized outside the repo, not merely relocated within it.**
|
||||
The spec calls for a "fixture-only world", and the sharp form that actually kills contamination is a world under a system temp path (`mktemp -d`), because a world still under `tests/.reports/` leaves `skills/` and the grading `tests/` tree reachable by upward navigation.
|
||||
The prose now says so explicitly.
|
||||
- **Live verification passed and reversed 0004's contaminated result.**
|
||||
A real `/benchmark-skill axi-review`-shaped run of 5 paired trials, orchestrated through the workflow mechanism with each arm's world under a temp path, came back with the new arm winning 5/5 and the Efficacy verdict **green** — where 0004 had scored red at 2/5 purely because the baseline had read the skill's `AXI-PRINCIPLES.md`.
|
||||
All five no-skill-baseline transcripts were clean of any `AXI-PRINCIPLES.md`, `skills/`, or `tests/` reference, and the baselines instead invented their own generic CLI rubric — the honest counterfactual of the skill not existing.
|
||||
- **Soft isolation is realized by instruction, since a subagent's working directory is not hard-settable through the workflow surface.**
|
||||
Each arm is told to `cd` into its world first and work only there, and the baseline is additionally told to stay within it.
|
||||
This is the achievable in-session bar, and hard OS-sandbox isolation stays deferred per the spec's "Out of Scope".
|
||||
- **Review follow-through.**
|
||||
A spec-axis review caught that "copy the case fixture into the world" could be read as nesting the fixture one level down, which would break `$WORLD/<file>` predicates.
|
||||
The wording now states that the world *is* the fresh fixture copy, so a `$WORLD/<file>` predicate resolves against the fixture root.
|
||||
A semicolon in one acceptance-criterion line was recast.
|
||||
- **The feature spec is not part of this commit.**
|
||||
`.claude/spec/skill-benchmarking.md` carries the isolation decisions but remains untracked, matching how 0004 was committed (code and task file, without the spec), and committing it here would pull in prose-standard breaches from sections written earlier.
|
||||
- The materialization pattern this establishes for the new arm is reused by 0005's previous-version arm, which points at a temp checkout of the skill at main's `HEAD` instead of a copy of the working-tree directory.
|
||||
Reference in New Issue
Block a user