skills/skills/benchmark-skill/SKILL.md

name, description, disable-model-invocation

name	description	disable-model-invocation
benchmark-skill	Benchmark a skill's efficacy against a no-skill baseline and, when a released version exists, its regression against that version, rendering an HTML report. Run deliberately as /benchmark-skill <name> for one skill, or /benchmark-skill with no argument for the all-skills leaderboard, never automatically.	true

benchmark-skill

Prove that a skill genuinely improves the agent's work rather than reading well and adding nothing.

Running /benchmark-skill <name> executes that skill's authored test cases as a controlled experiment and produces a self-contained HTML report carrying two independent verdicts: an Efficacy verdict (does the skill beat no-skill) and a Regression verdict (did my in-progress edit degrade it from the released version). Running /benchmark-skill with no argument benchmarks every skill that follows the tests/ convention and renders an index leaderboard over their reports (step 7). Each case runs a new-skill arm (the working tree) and a no-skill baseline arm, and — whenever the skill already exists on main and its directory differs from that released state — a third previous-version arm materialized from main's HEAD. Each arm runs several times, and a blind judge decides which arm's output better satisfies the case's author-written expectations. Two head-to-heads fall out of the arms per trial: efficacy pairs the new arm against no-skill, and regression pairs it against the previous version. When the skill is brand-new or unchanged from main, there is no previous version to compare against and the run degrades to the two-arm efficacy-only shape, with the Regression verdict reading not-applicable.

This skill runs in-session as an AI script: you orchestrate the arms and judges as subagents via the workflow mechanism, so the whole battery stays on the interactive subscription quota. The mechanical, non-judgment work — summing token usage, collapsing verdicts, applying the pass rules, rendering the report — is done by a committed program, core/benchmark_core.py, so the numbers are exact and reproducible rather than re-derived each run. You do the judgment work — running arms and judging — that the agent is actually good at.

The authoring contract for a test case is CASE-FORMAT.md. Read it before you read the target skill's tests.

Report artifacts live under a git-ignored tests/.reports/ directory at the repo root, flat and keyed by skill name. Run from the repo root: every path in the steps below is relative to it.

1. Resolve the target and validate its tests tree

The user passes the skill name as <name>. With no <name> you are in the all-skills mode: skip to step 7, which enumerates the skills and drives steps 1–6 for each. Find the skill's directory by its leaf name under skills/ (any depth), and find its tests at the mirror point under tests/skills/…/<name>/. If no skill directory maps to <name>, stop and say so. If the skill exists but has no tests directory or no case directories, report that the skill has no tests and stop — there is nothing to benchmark.

Validate the shape of the tests tree at run time:

• Each case directory has a case.md that parses per CASE-FORMAT.md.
• Any fixture a case references exists.
• The test directory maps to a real skill.

Report any malformed case and stop. A benchmark on a broken tree would produce meaningless numbers.

Done when every case parses, its fixture exists, and you have the case list in stable authored order.

2. Fix the arms and per-run settings

Always run a new-skill arm and a no-skill baseline arm. Add a third previous-version arm exactly when a released version exists to compare against, decided automatically with no flag or argument:

• Resolve the repo's default branch dynamically — git symbolic-ref refs/remotes/origin/HEAD (or git remote show origin), never a hardcoded "main" literal — and call its tip HEAD.
• Diff the skill's directory against HEAD (git diff --quiet HEAD -- <skill-dir>). Add the previous-version arm only when the directory both exists at HEAD and differs from it.
• A brand-new skill absent from HEAD, or a skill unchanged from it, has no meaningful previous version, so the run stays two-arm efficacy-only.

Arms run at the session's realistic default temperature (around 1.0), so the result reflects whether the skill reliably helps across variance rather than helping once by luck. Per-arm temperature is not settable through the in-session workflow surface, so a skill's temperature-zero override for a genuinely mechanical task is a documented knob for the future headless path, not something to set here.

Each case runs 5 paired trials. For each trial, trial i's new-skill output is judged against trial i's no-skill output for efficacy and — on a three-arm run — against trial i's previous-version output for regression. There is no reuse of arms or judgments across arms or runs.

3. Run the arms and judges as subagents

Use the workflow mechanism (the Workflow tool) to fan out the arms and judges. For each case, for each of the 5 trials, run every arm, then judge each comparison's pair.

Every arm receives the identical ## Prompt, authored once and arm-agnostically. For a conversation-driven case, inject the ## Seed transcript as the subagent's prior context before the prompt.

Give each arm-and-trial a hermetic fixture-only world. Make a fresh copy of the case fixture outside the repo — under a system temp path such as one from mktemp -d, never under tests/.reports/. That copy is the world: it is the arm subagent's working directory and the $WORLD the hard-assertion gate reads, so a $WORLD/<file> predicate resolves against the fixture root. Because the world lives outside the repo, nothing under skills/ or the grading tests/ tree sits on any path the arm reaches from there, so the baseline cannot discover the skill's assets and no arm can read its own case's soft criteria or hard assertions. Each arm-and-trial gets its own fresh world, so writes never leak between them.

• New-skill arm — force-invoke the skill. Materialize the skill into an isolated temp path outside the world — a copy of its working-tree directory, so it reflects uncommitted edits and carries none of its repo surroundings. Point the subagent at that copy, tell it to use that skill, and have it read the skill's SKILL.md and any assets it references, so the real skill machinery is exercised. Its working directory is the fresh world, and it also receives the prompt.
• Previous-version arm (three-arm runs only) — force-invoked identically to the new-skill arm, differing only in which skill it points at. Materialize the skill's directory at the default branch's HEAD into its own isolated temp path (git --work-tree=<temp> checkout HEAD -- <skill-dir>, or git archive HEAD <skill-dir> piped into the temp path), since each skill is self-contained and needs none of the rest of the repo. Point the subagent at that checkout and otherwise treat it exactly as the new-skill arm — same world, same prompt.
• No-skill baseline arm — the honest counterfactual of the skill not existing. Materialize no skill for it: give it the bare prompt with the skill absent from its context, and do not mention the skill or hint that one exists. Instruct it to stay within its working directory, since the isolation is soft and the subagent shares the machine.

The cases are held fixed to the working tree: all arms run against today's prompt, fixture, and expectations, so the skill version is the only variable between the new and previous-version arms.

Capture each arm's final message to a file — this is the $OUTPUT the hard-assertion gate reads and the text the judge compares.

The judge — one blind judge subagent per trial per comparison, so a three-arm trial draws two judgments (efficacy and regression) and a two-arm trial draws one. For each comparison, show the judge the two arms' final messages as unlabelled A and B, with the order randomized per trial (record which of A/B is the new arm so you can map the verdict back). The same judge machinery serves both comparisons; only the pair of outputs handed over differs — efficacy pairs new against no-skill, regression pairs new against the previous version. Ground it on the case's ## Soft criteria rather than letting it free-form its own standard, and instruct it explicitly to discount mere length and formatting differences — a skill must not win by being more verbose. Have it return a single winner: A, B, or tie.

4. Run the hard-assertion gate

Run the case's ## Hard assertions against the new arm only, once per trial, with $OUTPUT and $WORLD bound as CASE-FORMAT.md defines — that trial's new-arm final message and its world. Record each trial's pass or fail into the bundle for the core to score. A case with no ## Hard assertions block simply has no gate.

5. Collect the run data

The deterministic core is a pure transform: it takes the collected data and returns the results model and HTML. Assemble one run bundle JSON with this shape and write it under tests/.reports/.work/<name>-bundle.json:

The shape below is the three-arm run. For a two-arm run, drop the previous arm, drop the regression comparison, and drop the previous key from each trial's outputs and usage — the core reads the arm shape off the presence of a regression comparison.

{
  "skill": "<name>",
  "generatedAt": "<ISO-8601 timestamp>",
  "temperature": 1.0,
  "trialsPerCase": 5,
  "arms": [
    {"id": "new", "label": "New skill"},
    {"id": "previous", "label": "Previous version"},
    {"id": "baseline", "label": "No skill"}
  ],
  "comparisons": [
    {"id": "efficacy",   "label": "Efficacy",   "new": "new", "against": "baseline", "rule": "efficacy"},
    {"id": "regression", "label": "Regression", "new": "new", "against": "previous", "rule": "regression"}
  ],
  "cases": [
    {
      "name": "<case directory name>",
      "description": "<from case.md frontmatter>",
      "softCriteria": ["<each ## Soft criteria entry>"],
      "trials": [
        {
          "hard": {"ran": true, "pass": true},
          "comparisons": {
            "efficacy":   {"winner": "new", "rationale": "<judge's one line>"},
            "regression": {"winner": "tie", "rationale": "<judge's one line>"}
          },
          "outputs": {
            "new":      "<new arm's final message>",
            "previous": "<previous-version arm's final message>",
            "baseline": "<no-skill arm's final message>"
          },
          "usage": {
            "new":      {"input": 0, "output": 0, "cacheCreation": 0, "cacheRead": 0, "turns": 0},
            "previous": {"input": 0, "output": 0, "cacheCreation": 0, "cacheRead": 0, "turns": 0},
            "baseline": {"input": 0, "output": 0, "cacheCreation": 0, "cacheRead": 0, "turns": 0}
          }
        }
      ]
    }
  ]
}

• comparisons carries one entry per head-to-head, keyed by the comparison id. regression is present only on a three-arm run. Each winner is an arm id ("new", "baseline", or "previous") or "tie", mapped back from that comparison's blind A/B answer.
• hard carries the gate result for that trial's new arm. Omit it or set ran: false when the case has no hard assertions. The gate feeds the Efficacy axis only. The Regression axis is judged purely on the head-to-head.
• outputs is each arm's captured final message, keyed by arm id. The core surfaces the losing-trial pair as report evidence, so a failed comparison shows the new arm's output beside the one it lost to.
• usage is the per-arm-per-trial transcript usage. Recover it by reading each arm-subagent's transcript file (the per-agent JSONL the workflow writes) and summing each message's token usage into the four components — input, output, cacheCreation (cache-creation input tokens), cacheRead (cache-read input tokens). turns is the count of tool-call rounds in that transcript. If a transcript genuinely lacks usage data, record zeros rather than guessing — cost never gates a verdict.

6. Score and render

Run the core over the bundle, writing the report and a machine-readable model:

python3 skills/benchmark-skill/core/benchmark_core.py \
  tests/.reports/.work/<name>-bundle.json \
  --json tests/.reports/<name>.results.json \
  --html tests/.reports/<name>.html \
  --history tests/.reports/<name>.history.jsonl

The core collapses each trial of each comparison to a WIN, TIE, or LOSS for the new arm, applies each comparison's pass rule — efficacy passes at wins ≥ 3 and losses ≤ 1, regression passes at losses ≤ 1 with no wins floor — flags any loss for human review, and yields the two skill-level verdicts: Efficacy (green when every case beats no-skill) and Regression (green when no case degraded, not-applicable on a two-arm run). With --history it appends this run's summary line — each axis's net margin and pass/fail plus the two-arm/three-arm flag — to the per-skill history file, trimming it oldest-first at roughly the last fifty runs. That history file lives in the git-ignored reports directory and is ephemeral: cleaning the directory resets it, consistent with reports being transient artifacts. It renders the self-contained HTML report — the two verdict badges, two stacked trend ribbons plotting each axis's net-margin over the recent runs (the regression ribbon leaving a gap for any two-arm run), a headline reading the badges together, run metadata, the per-arm cost table (a previous-version row on a three-arm run), and the cases in stable authored order, each auto-expanding on any failure or flagged loss to show the losing-trial output pair — alongside the machine-readable model. A clean run rests fully collapsed, and each green badge still flags its most fragile case with a chip so a barely-green skill cannot look robust. Cost is reported but never gates a verdict.

Clean up the scratch when done: remove the tests/.reports/.work/ directory and every per-arm world and skill materialization you created under the system temp path — including the previous-version checkout.

Done when tests/.reports/<name>.html exists. Report both verdicts, the path to the report, and any flagged losses or regressions to the user.

7. Benchmark every skill and render the leaderboard

This step runs only in the all-skills mode — /benchmark-skill with no <name>.

Enumerate every skill the same way step 1 resolves one: each skill directory under skills/ (any depth) whose mirror point under tests/skills/…/ holds case directories. A skill with no tests directory is skipped rather than failing the batch; note which skills were skipped. Run steps 1–6 for each skill in turn, so each writes its own per-skill report and results model under tests/.reports/, flat and keyed by skill name — the per-skill HTML is latest-only and overwritten each run, since the longitudinal data lives in the per-skill history file.

Then render the index over every per-skill results model:

python3 skills/benchmark-skill/core/benchmark_core.py --leaderboard \
  tests/.reports/*.results.json \
  --html tests/.reports/index.html \
  --generated-at "<ISO-8601 timestamp>"

The core sorts the rows — any red first with regressions ordered above efficacy failures, then fragile-but-passing skills, then clean green — and links each row out to its <name>.html. A skill with no previous version reads not-applicable in its Regression cell. The fragile-but-passing tier reuses the per-badge fragility signal from the per-skill reports: a case one loss from regressing, or a narrowest efficacy margin sitting on the pass floor.

Done when tests/.reports/index.html exists. Report the leaderboard path, and per skill both verdicts and any flagged losses or regressions, to the user.