alexion/skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: efficacy-benchmark tracer bullet (task 0004) #4

Merged
alexion merged 5 commits from task-0004-efficacy-benchmark-tracer-bullet into main 2026-07-24 15:03:26 -04:00
Conversation 0 Commits 5 Files Changed 12 +1591
alexion commented 2026-07-24 08:56:53 -04:00
Owner
Copy Link

Task: .claude/tasks/0004-efficacy-benchmark-tracer-bullet.md

Summary

The efficacy-benchmark tracer bullet — the thinnest complete path that benchmarks a skill against a no-skill baseline and renders an HTML report.

  • tests/ convention — a top-level tree mirroring skills/ by full path; case directories hold a case.md (contract in CASE-FORMAT.md) and an optional committed fixture, kept outside skills/ so they never get packaged.
  • benchmark-skill — an auto-discovered, packaged, non-model-invocable runner (disable-model-invocation: true), invoked /benchmark-skill <name>. It validates the target's tests/ tree, then orchestrates a force-invoked new-skill arm and a skill-absent baseline arm plus a blind per-trial judge as in-session subagents via the workflow mechanism.
  • Deterministic core (core/benchmark_core.py) — a pure transform over the collected run data: sums per-arm transcript usage into raw/cost-equivalent/imputed-cost/turns, collapses each trial to WIN/TIE/LOSS (tie = non-win), applies the efficacy pass rule (wins ≥ 3, losses ≤ 1), flags losses, and renders a self-contained HTML report with the Efficacy badge, a two-row cost table with the cache-overhead footnote, and cases in stable authored order. Parameterized over arms and comparisons; two-arm here.
  • Check — checks/benchmark-core.nix feeds committed fixtures to the core and asserts the metrics, verdicts, and report with no LLM; wired into flake.nix.
  • Live proof — a real tests/ tree for axi-review plus a live 5-trial /benchmark-skill axi-review run that produced its efficacy report end to end.

Deviations

  • The core is Python 3 (run via nix shell nixpkgs#python3 on this host, which lacks a global python3); it cleanly owns the HTML/JSON work the shell idiom would fight.
  • The live run's Efficacy verdict came out red (new 2/5, baseline 3/5), but that is a baseline-contamination artifact: the in-session no-skill subagent read the skill's AXI-PRINCIPLES.md rubric from the repo on disk, so it was not a clean counterfactual. Filesystem isolation of the baseline arm is a follow-up; the core, pass rule, and skill design are unaffected.
  • A per-run pricing override was removed as speculative config.
  • Trial temperature ran at the session default (subagent temperature isn't settable via the workflow agent() surface); the per-skill temperature and zero-override remain documented knobs.

Review

Risk

Overall: Low

  • Blast radius: Low — all new files; only additive edits to .gitignore and one flake.nix entry, no existing callers touched.
  • Reversibility: Low — pure additions, no migrations or published-API changes.
  • Test coverage: Low — the core is exercised end-to-end by checks/benchmark-core.nix against a committed fixture.
  • Sensitive domain: Low — no auth/payments/permissions; hard-assertion sh runs only against local fixtures in dev tooling.
  • Size & complexity: Medium — ~1,100 lines, mostly repetitive fixture JSON and docs; the scoring logic is small and linear.
  • Runtime criticality: Low — developer-only tooling, invoked deliberately.

Unaddressed findings

Standards

  • CASE-FORMAT.md all-caps name (soft naming call) — kept: matches the repo's existing skill-companion-doc convention (AXI-PRINCIPLES.md, REPORT-FORMAT.md), which overrides the baseline.
  • Speculative Generality: the per-comparison rule field and arms/comparisons parameterization are unused by this two-arm slice — kept: the spec mandates "structured parameterized over the number of arms and the comparisons from the start"; the regression rule lands in task 0005.
  • Duplicated usage blocks across fixture trials — kept: inert committed test data, not logic.

Spec

  • The core does not literally "parse case.md" (spec line 60) — no change: parsing lives in the runner prose, and the pure-transform split matches spec lines 57 and 82 and is the better design.
Task: `.claude/tasks/0004-efficacy-benchmark-tracer-bullet.md` ## Summary The efficacy-benchmark tracer bullet — the thinnest complete path that benchmarks a skill against a no-skill baseline and renders an HTML report. - **`tests/` convention** — a top-level tree mirroring `skills/` by full path; case directories hold a `case.md` (contract in `CASE-FORMAT.md`) and an optional committed fixture, kept outside `skills/` so they never get packaged. - **`benchmark-skill`** — an auto-discovered, packaged, non-model-invocable runner (`disable-model-invocation: true`), invoked `/benchmark-skill <name>`. It validates the target's `tests/` tree, then orchestrates a force-invoked new-skill arm and a skill-absent baseline arm plus a blind per-trial judge as in-session subagents via the workflow mechanism. - **Deterministic core** (`core/benchmark_core.py`) — a pure transform over the collected run data: sums per-arm transcript usage into raw/cost-equivalent/imputed-cost/turns, collapses each trial to WIN/TIE/LOSS (tie = non-win), applies the efficacy pass rule (wins ≥ 3, losses ≤ 1), flags losses, and renders a self-contained HTML report with the Efficacy badge, a two-row cost table with the cache-overhead footnote, and cases in stable authored order. Parameterized over arms and comparisons; two-arm here. - **Check** — `checks/benchmark-core.nix` feeds committed fixtures to the core and asserts the metrics, verdicts, and report with no LLM; wired into `flake.nix`. - **Live proof** — a real `tests/` tree for `axi-review` plus a live 5-trial `/benchmark-skill axi-review` run that produced its efficacy report end to end. ### Deviations - The core is **Python 3** (run via `nix shell nixpkgs#python3` on this host, which lacks a global `python3`); it cleanly owns the HTML/JSON work the shell idiom would fight. - The live run's Efficacy verdict came out **red** (new 2/5, baseline 3/5), but that is a **baseline-contamination artifact**: the in-session no-skill subagent read the skill's `AXI-PRINCIPLES.md` rubric from the repo on disk, so it was not a clean counterfactual. Filesystem isolation of the baseline arm is a follow-up; the core, pass rule, and skill design are unaffected. - A per-run pricing override was removed as speculative config. - Trial temperature ran at the session default (subagent temperature isn't settable via the workflow `agent()` surface); the per-skill temperature and zero-override remain documented knobs. ## Review ### Risk **Overall: Low** - Blast radius: Low — all new files; only additive edits to `.gitignore` and one `flake.nix` entry, no existing callers touched. - Reversibility: Low — pure additions, no migrations or published-API changes. - Test coverage: Low — the core is exercised end-to-end by `checks/benchmark-core.nix` against a committed fixture. - Sensitive domain: Low — no auth/payments/permissions; hard-assertion `sh` runs only against local fixtures in dev tooling. - Size & complexity: Medium — ~1,100 lines, mostly repetitive fixture JSON and docs; the scoring logic is small and linear. - Runtime criticality: Low — developer-only tooling, invoked deliberately. ### Unaddressed findings **Standards** - `CASE-FORMAT.md` all-caps name (soft naming call) — kept: matches the repo's existing skill-companion-doc convention (`AXI-PRINCIPLES.md`, `REPORT-FORMAT.md`), which overrides the baseline. - Speculative Generality: the per-comparison `rule` field and arms/comparisons parameterization are unused by this two-arm slice — kept: the spec mandates "structured parameterized over the number of arms and the comparisons from the start"; the regression rule lands in task 0005. - Duplicated `usage` blocks across fixture trials — kept: inert committed test data, not logic. **Spec** - The core does not literally "parse `case.md`" (spec line 60) — no change: parsing lives in the runner prose, and the pure-transform split matches spec lines 57 and 82 and is the better design.
alexion added 1 commit 2026-07-24 08:56:53 -04:00
feat: add efficacy-benchmark tracer bullet (task 0004) e35ed9d05b 
Introduce the thinnest complete path that benchmarks a skill's efficacy
against a no-skill baseline and renders an HTML report.

- tests/ convention: a top-level tree mirroring skills/ by full path, with
  case directories holding a case.md (CASE-FORMAT.md) and an optional fixture.
- benchmark-skill: a distributed, non-model-invocable runner (/benchmark-skill
  <name>) that orchestrates force-invoked new-skill and skill-absent baseline
  arms plus a blind per-trial judge as in-session subagents.
- Deterministic core (Python): a pure transform over the collected run data
  that sums per-arm usage, collapses each trial to WIN/TIE/LOSS, applies the
  efficacy pass rule (wins >= 3, losses <= 1), and renders a self-contained
  report. Parameterized over arms and comparisons; two-arm here.
- Fixture-driven no-LLM check (checks/benchmark-core.nix) wired into flake.nix.
- Real tests tree for axi-review; a live run produced its efficacy report.
alexion added 1 commit 2026-07-24 09:01:31 -04:00
feat: add dev shell with python3 and the benchmark skill (task 0004) ab2c05acb2 
nix develop places benchmark-skill into ./.claude/skills/ (so /benchmark-skill
is available in this repo) and puts python3 on PATH to run the deterministic
core without a one-off nix shell.
alexion added 1 commit 2026-07-24 14:28:08 -04:00
feat: isolate benchmark arms in fixture-only worlds (task 0008) 613e423617 
Run each benchmark arm in a hermetic fixture-only world materialized
outside the repo, so the no-skill baseline can no longer discover the
skill's assets on disk. The new-skill arm's skill is materialized to an
isolated temp path it is pointed at; the baseline gets no skill and is
told to stay within its world.

A live 5-trial axi-review run confirms the fix: all baseline transcripts
are clean of AXI-PRINCIPLES.md, the baselines invent their own generic
rubric (the honest counterfactual), and the Efficacy verdict is green
5/5 where the contaminated 0004 run had scored red 2/5.

Runner prose only; the deterministic core is untouched.
alexion added 1 commit 2026-07-24 14:42:22 -04:00
docs: add skill-benchmarking spec 3e96db664c 
Bring the feature's design record into git (it was authored across the
benchmark tasks but never committed). Cleaned to the repo's prose
standards on the way in: recast prose semicolons as separate sentences
and split multi-sentence lines to one sentence per line.
alexion added 1 commit 2026-07-24 15:01:58 -04:00
refactor: tighten benchmark-skill instructions per craft-skill audit 5db98dcab9 
Collapse duplication so each rule has one home: the pass-rule threshold
and cost footnote defer to the core, and the case-contract detail defers
to CASE-FORMAT.md. Drop editor-facing sediment from the arms step.

Fixes surfaced by a verify walk-through: define the hard-assertion gate
as passing only when every predicate exits zero (an earlier failure was
maskable by a later success), state that in-session arms run at the
session default temperature, name the fixture directory, and note the
steps run from the repo root.
alexion merged commit 5db98dcab9 into main 2026-07-24 15:03:26 -04:00
alexion deleted branch task-0004-efficacy-benchmark-tracer-bullet 2026-07-24 15:03:26 -04:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
alexion
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: alexion/skills#4
Delete Branch "task-0004-efficacy-benchmark-tracer-bullet"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?