skills/.claude/tasks/0004-efficacy-benchmark-tracer-bullet.md

---
spec: skill-benchmarking
blocked-by: 0001-content-tier-skill-packaging
---

## What to build

The foundational tracer bullet: the thinnest complete path that benchmarks one skill's **efficacy against a no-skill baseline** and produces a report.
Running `/benchmark-skill <name>` on a real skill executes its cases and returns an HTML report carrying a single **Efficacy verdict**.

This slice establishes three things end to end.

**The `tests/` convention.**
Tests live in a top-level `tests/` tree that mirrors `skills/` by full path, so a skill's cosmetic nesting is mirrored and its tests sit at the same relative path.
Tests are deliberately outside `skills/` so they are never packaged or placed when a skill is installed.
A skill's tests are case directories at the mirror point, each holding a `case.md` and an optional fixture.
A `case.md` carries a one-line `description`, a `## Prompt` given identically to every arm, an optional `## Seed` transcript in role-tagged form for conversation-driven skills, an optional `## Hard assertions` block of executable shell predicates, and a `## Soft criteria` list with at least one entry.
Every case ships a hermetic, committed fixture, and each arm-and-trial combination runs against a fresh copy of it so writes never leak between runs.

The `case.md` shape (from the case-format decision) is the authoring contract:

```
---
description: <one-line scenario>
---
## Prompt
<the realistic user request, given identically to every arm>

## Seed            (optional; conversation-based skills only)
**User:** ...
**Assistant:** ...

## Hard assertions   (executable; $OUTPUT = arm's final message, $WORLD = its fixture copy)
​```sh
test -f "$WORLD/review.md"
grep -qE '<pattern>' "$OUTPUT"
​```

## Soft criteria
- <a statement the judge grounds the comparisons on>
```

**The `benchmark-skill` runner skill.**
A distributed skill, auto-discovered and packaged like any other skill in this repo, marked non-model-invocable so it only ever runs deliberately, invoked `/benchmark-skill <name>`.
It runs in-session as an AI script, orchestrating the arms and the judge as subagents via the workflow mechanism, so the whole battery stays on the interactive subscription quota rather than the metered automation credit pool.
It validates the shape of the target skill's `tests/` tree at run time — that each `case.md` parses, that a referenced fixture exists, that the test directory maps to a real skill — and reports a skill that has no tests.
For each case it runs a **new-skill arm** (the working tree) and a **no-skill baseline arm**.
The new arm is force-invoked: its subagent is pointed at the skill's directory and told to use it, reading that skill's own file and any assets it references, so the real skill machinery is exercised.
The baseline arm receives the bare prompt with the skill absent from its context, the honest counterfactual of the skill not existing.
Every arm is given the identical realistic prompt authored once and arm-agnostically.
Each case runs 5 paired trials at a realistic temperature, configurable per skill, with a per-skill temperature-zero override for a skill that wraps a mechanical task, and with no reuse of arms or judgments across arms or runs.
Per trial, one blind judge subagent decides between the two arms' outputs shown as unlabelled A and B with the order randomized, grounded on the case's soft criteria rather than free-forming its own standard, and instructed to discount mere length and formatting differences.
The hard-assertion gate runs against the new arm only, with `$OUTPUT` the path to the arm's captured final message and `$WORLD` the path to its fresh fixture copy; a non-zero exit fails the assertion, and a failed hard assertion fails the case outright regardless of the head-to-head.

**The deterministic core.**
A small committed program, shipped with the `benchmark-skill` skill and invoked by its prose, owns every mechanical, non-judgment step so the numbers are exact and reproducible rather than re-derived by the agent each run.
It is a pure transform: given the collected run data (per-arm transcript usage, hard-assertion results, and the per-trial judge verdicts) it returns the results model and the rendered HTML.
The run-history file and the trend ribbons that consume it are deferred to a later slice (0006); this slice's report is latest-only.
It is structured parameterized over the number of arms and the comparisons from the start, but this slice exercises only the two-arm efficacy shape.
It parses `case.md`, sums each arm's transcript token usage into the metric set — raw tokens across the four components, the pricing-weighted cost-equivalent-token figure, an imputed dollar cost, and the turn count from the transcript's tool-call rounds — collapses each efficacy trial to WIN, TIE, or LOSS for the new arm (a tie is a non-win), and applies the efficacy pass rule.
Efficacy passes for a case when wins ≥ 3 and losses ≤ 1 across the 5 trials, and any loss is flagged for human review.
The **Efficacy verdict** for the skill is green when every case passes efficacy.
It renders a self-contained HTML report: an Efficacy verdict badge and run metadata at the top, a per-arm cost table (new-skill and no-skill rows, each with turns, raw tokens, cost-equivalent tokens, and imputed cost, footnoted that absolute cost is inflated by shared-context cache overhead so the trustworthy signal is the new-vs-no-skill ratio), and the cases in stable authored order.
Cost is reported alongside quality but never gates a verdict.

The deterministic core is unit-tested at its single external seam, in the style of `checks/shell-hook.nix` and `checks/home-manager-module.nix`: fed committed fixtures (sample new-skill and no-skill arm transcripts, hard-assertion results, and per-trial judge verdicts) it must produce the expected per-arm metrics, the expected per-case and skill-level Efficacy verdict, and a correct report.
The test needs no LLM to run.

All report artifacts live under a git-ignored `tests/.reports/` directory.

## Acceptance criteria

- [x] A `tests/` tree at the repo top level mirrors `skills/` by full path; a skill's tests sit as case directories at the mirror point, each with a `case.md` and optional committed fixture.
- [x] `case.md` parses its `description`, `## Prompt`, optional `## Seed`, optional `## Hard assertions`, and `## Soft criteria` (at least one) per the authoring contract.
- [x] `benchmark-skill` is an auto-discovered, packaged, non-model-invocable distributed skill invoked as `/benchmark-skill <name>`.
- [x] The runner validates the target's `tests/` tree at run time (each `case.md` parses, referenced fixtures exist, the directory maps to a real skill) and reports a skill with no tests.
- [x] For each case the runner orchestrates a force-invoked new-skill arm and a skill-absent no-skill baseline arm as subagents via the workflow mechanism, entirely in-session.
- [x] Every arm receives the identical prompt; each arm-and-trial runs against a fresh copy of the case fixture; a conversation-driven case injects its `## Seed` transcript as the subagent's prior context.
- [x] Each case runs 5 paired trials at a per-skill-configurable realistic temperature, with a per-skill temperature-zero override, and no reuse of arms or judgments across arms or runs.
- [x] Per trial, one blind judge subagent compares the two arms as randomized unlabelled A/B, grounded on the case's soft criteria and instructed to discount length and formatting.
- [x] The hard-assertion gate runs against the new arm only with `$OUTPUT` and `$WORLD` provided; a non-zero exit fails the assertion and a failed assertion fails the case outright.
- [x] The deterministic core is a committed, pure-transform program shipped with the skill, structured parameterized over arms and comparisons though exercised here two-arm.
- [x] The core sums per-arm transcript usage into raw tokens, cost-equivalent tokens, imputed dollar cost, and turn count, and collapses each efficacy trial to WIN/TIE/LOSS (tie = non-win).
- [x] The core applies the efficacy pass rule (wins ≥ 3 and losses ≤ 1), flags any loss, and yields a skill-level Efficacy verdict that is green when every case passes.
- [x] The core renders a self-contained HTML report with the Efficacy badge, run metadata, a two-row per-arm cost table with the cache-overhead footnote, and cases in stable authored order; cost never gates.
- [x] A fixture-driven unit test (no LLM), in the style of the existing `checks/`, feeds sample transcripts, hard-assertion results, and judge verdicts to the core and asserts the metrics, the Efficacy verdict, and the report.
- [x] Report artifacts are written under a git-ignored `tests/.reports/` directory.
- [x] At least one existing real skill carries an authored `tests/` tree (a `case.md` with a fixture), and a live `/benchmark-skill <name>` run exercises it end to end and produces its efficacy report — proving the tracer bullet actually fires.

## Implementation Notes

Files: `skills/benchmark-skill/SKILL.md` (the in-session runner prose), `skills/benchmark-skill/CASE-FORMAT.md` (the `case.md` authoring contract), `skills/benchmark-skill/core/benchmark_core.py` (the deterministic core), `checks/benchmark-core.nix` + `checks/fixtures/benchmark/run-bundle.json` (the fixture-driven no-LLM check, wired into `flake.nix`), `tests/skills/axi-review/basic-cli-review/` (the real authored test tree with a committed `fixture/greet`), and a `tests/.reports/` `.gitignore` rule.

- **Core language — Python 3.**
  The deterministic core is Python stdlib only, chosen because it renders HTML and parses JSON far more cleanly than the repo's shell/jq idiom, keeping the error-prone arithmetic and templating in committed, tested code rather than re-derived by the agent.
  This machine has no `python3` on its global PATH, so the check pulls `pkgs.python3` as a build input and the live run invokes the core via `nix shell nixpkgs#python3`.
  A distributed consumer repo is expected to provide `python3` at run time.

- **Live run — baseline contamination is a real harness limitation.**
  The live `/benchmark-skill axi-review` ran all 5 paired trials end to end and produced `tests/.reports/axi-review.html` with real per-arm token metrics recovered from the arm transcripts, proving the tracer bullet fires.
  The Efficacy verdict came out **red** (new arm won 2/5, baseline won 3/5).
  That red is substantially an artifact: the no-skill baseline subagent, running in this repo on disk, discovered and used the skill's `AXI-PRINCIPLES.md` rubric (its own output states it "judged each principle against the canonical rubric in `AXI-PRINCIPLES.md`"), so it was not a clean no-skill counterfactual.
  This is a limitation of in-session benchmarking — a subagent can read repo-resident skill assets — not a defect in the core, the pass rule, or the skill's design.
  Hardening the baseline arm's filesystem isolation is a follow-up.

- **Trial temperature.**
  Subagent temperature is not directly settable through the workflow's `agent()` surface, so the arms ran at the session's default (realistic) temperature.
  The per-skill temperature and the temperature-zero override remain documented knobs in `SKILL.md` for a future headless path.

- **Parameterization kept per spec.**
  The core carries the arms list and comparisons list (each with a `rule`) from the start, per "structured parameterized over the number of arms and the comparisons from the start", though this slice exercises only the two-arm efficacy shape and the single efficacy pass rule.
  The regression arm and its inverted rule arrive in task 0005.

- **Review follow-through.**
  A per-run pricing override was removed as speculative config.
  Documented-standard breaches (semicolons in authored prose/comments, a comment sentence-per-line) were fixed.
  `CASE-FORMAT.md`'s all-caps name is retained to match the repo's existing skill-companion-doc convention (`AXI-PRINCIPLES.md`, `REPORT-FORMAT.md`).