feat: add efficacy-benchmark tracer bullet (task 0004)
Introduce the thinnest complete path that benchmarks a skill's efficacy against a no-skill baseline and renders an HTML report. - tests/ convention: a top-level tree mirroring skills/ by full path, with case directories holding a case.md (CASE-FORMAT.md) and an optional fixture. - benchmark-skill: a distributed, non-model-invocable runner (/benchmark-skill <name>) that orchestrates force-invoked new-skill and skill-absent baseline arms plus a blind per-trial judge as in-session subagents. - Deterministic core (Python): a pure transform over the collected run data that sums per-arm usage, collapses each trial to WIN/TIE/LOSS, applies the efficacy pass rule (wins >= 3, losses <= 1), and renders a self-contained report. Parameterized over arms and comparisons; two-arm here. - Fixture-driven no-LLM check (checks/benchmark-core.nix) wired into flake.nix. - Real tests tree for axi-review; a live run produced its efficacy report.
This commit is contained in:
150
skills/benchmark-skill/SKILL.md
Normal file
150
skills/benchmark-skill/SKILL.md
Normal file
@@ -0,0 +1,150 @@
|
||||
---
|
||||
name: benchmark-skill
|
||||
description: Benchmark a skill's efficacy against a no-skill baseline and render an HTML report. Run deliberately as /benchmark-skill <name>, never automatically.
|
||||
disable-model-invocation: true
|
||||
---
|
||||
|
||||
# benchmark-skill
|
||||
|
||||
Prove that a skill genuinely improves the agent's work rather than reading well and adding nothing.
|
||||
|
||||
Running `/benchmark-skill <name>` executes that skill's authored test cases as a controlled experiment and produces a self-contained HTML report carrying a single **Efficacy verdict**.
|
||||
Each case runs a **new-skill arm** (the working tree) and a **no-skill baseline arm**, several times, and a blind judge decides which arm's output better satisfies the case's author-written expectations.
|
||||
|
||||
This skill runs **in-session as an AI script**: you orchestrate the arms and judges as subagents via the workflow mechanism, so the whole battery stays on the interactive subscription quota.
|
||||
The mechanical, non-judgment work — summing token usage, collapsing verdicts, applying the pass rule, rendering the report — is done by a committed program, [`core/benchmark_core.py`](core/benchmark_core.py), so the numbers are exact and reproducible rather than re-derived each run.
|
||||
You do the judgment work — running arms and judging — that the agent is actually good at.
|
||||
|
||||
The authoring contract for a test case is [`CASE-FORMAT.md`](CASE-FORMAT.md).
|
||||
Read it before you read the target skill's tests.
|
||||
|
||||
Report artifacts live under a git-ignored `tests/.reports/` directory at the repo root, flat and keyed by skill name.
|
||||
|
||||
## 1. Resolve the target and validate its tests tree
|
||||
|
||||
The user passes the skill name as `<name>`.
|
||||
Find the skill's directory by its leaf name under `skills/` (any depth), and find its tests at the mirror point under `tests/skills/…/<name>/`.
|
||||
If no skill directory maps to `<name>`, stop and say so.
|
||||
If the skill exists but has no tests directory or no case directories, report that the skill has no tests and stop — there is nothing to benchmark.
|
||||
|
||||
Validate the shape of the tests tree at run time:
|
||||
|
||||
- Each case directory has a `case.md` that parses per `CASE-FORMAT.md` — a `description`, a `## Prompt`, at least one `## Soft criteria` entry, and optional `## Seed` and `## Hard assertions` blocks.
|
||||
- Any fixture a case references exists.
|
||||
- The test directory maps to a real skill.
|
||||
|
||||
Report any malformed case and stop.
|
||||
A benchmark on a broken tree would produce meaningless numbers.
|
||||
|
||||
Done when every case parses, its fixture exists, and you have the case list in stable authored order.
|
||||
|
||||
## 2. Fix the arms and per-run settings
|
||||
|
||||
This slice is **two-arm efficacy only**: a new-skill arm and a no-skill baseline arm.
|
||||
(The previous-version regression arm is a later slice.
|
||||
The core already handles extra arms, so do not remove the two-arm shape.)
|
||||
|
||||
Choose the trial temperature: a realistic temperature (around 1.0) by default, so the result reflects whether the skill *reliably* helps across variance.
|
||||
A skill may pin a temperature-zero override when it wraps a genuinely mechanical task — honor that override if the skill declares one.
|
||||
|
||||
Each case runs **5 paired trials**.
|
||||
Trial *i*'s new-skill output is judged against trial *i*'s no-skill output.
|
||||
There is no reuse of arms or judgments across arms or runs.
|
||||
|
||||
## 3. Run the arms and judges as subagents
|
||||
|
||||
Use the workflow mechanism (the `Workflow` tool) to fan out the arms and judges.
|
||||
For each case, for each of the 5 trials, run both arms, then judge the pair.
|
||||
|
||||
**Both arms receive the identical `## Prompt`, authored once and arm-agnostically.**
|
||||
Each arm-and-trial combination runs against a **fresh copy** of the case fixture, made under `tests/.reports/.work/`, so writes never leak between runs.
|
||||
For a conversation-driven case, inject the `## Seed` transcript as the subagent's prior context before the prompt.
|
||||
|
||||
- **New-skill arm** — force-invoke the skill.
|
||||
Point the subagent at the skill's own directory, tell it to use that skill, and have it read the skill's `SKILL.md` and any assets it references, so the real skill machinery is exercised.
|
||||
Give it the fresh fixture copy as its working target and the prompt.
|
||||
- **No-skill baseline arm** — the honest counterfactual of the skill not existing.
|
||||
Give it the bare prompt with the skill absent from its context.
|
||||
Do not mention the skill or hint that one exists.
|
||||
|
||||
Capture each arm's **final message** to a file — this is the `$OUTPUT` the hard-assertion gate reads and the text the judge compares.
|
||||
|
||||
**The judge** — one blind judge subagent per trial.
|
||||
Show it the two arms' final messages as unlabelled **A** and **B**, with the order randomized per trial (record which of A/B is the new arm so you can map the verdict back).
|
||||
Ground it on the case's `## Soft criteria` rather than letting it free-form its own standard, and instruct it explicitly to **discount mere length and formatting differences** — a skill must not win by being more verbose.
|
||||
Have it return a single winner: A, B, or tie.
|
||||
|
||||
## 4. Run the hard-assertion gate
|
||||
|
||||
Run the case's `## Hard assertions` against the **new arm only**, once per trial.
|
||||
Bind `$OUTPUT` to the path of that trial's new-arm final message and `$WORLD` to the path of that trial's fresh new-arm fixture copy.
|
||||
Run each predicate.
|
||||
A non-zero exit fails the assertion.
|
||||
Record pass/fail per trial — a failed hard assertion fails the case outright regardless of the head-to-head.
|
||||
A case with no `## Hard assertions` block simply has no gate.
|
||||
|
||||
## 5. Collect the run data
|
||||
|
||||
The deterministic core is a pure transform: it takes the collected data and returns the results model and HTML.
|
||||
Assemble one **run bundle** JSON with this shape and write it under `tests/.reports/.work/<name>-bundle.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"skill": "<name>",
|
||||
"generatedAt": "<ISO-8601 timestamp>",
|
||||
"temperature": 1.0,
|
||||
"trialsPerCase": 5,
|
||||
"arms": [
|
||||
{"id": "new", "label": "New skill"},
|
||||
{"id": "baseline", "label": "No skill"}
|
||||
],
|
||||
"comparisons": [
|
||||
{"id": "efficacy", "label": "Efficacy", "new": "new", "against": "baseline", "rule": "efficacy"}
|
||||
],
|
||||
"cases": [
|
||||
{
|
||||
"name": "<case directory name>",
|
||||
"description": "<from case.md frontmatter>",
|
||||
"softCriteria": ["<each ## Soft criteria entry>"],
|
||||
"trials": [
|
||||
{
|
||||
"hard": {"ran": true, "pass": true},
|
||||
"comparisons": {"efficacy": {"winner": "new", "rationale": "<judge's one line>"}},
|
||||
"usage": {
|
||||
"new": {"input": 0, "output": 0, "cacheCreation": 0, "cacheRead": 0, "turns": 0},
|
||||
"baseline": {"input": 0, "output": 0, "cacheCreation": 0, "cacheRead": 0, "turns": 0}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- **`winner`** is the arm id (`"new"` or `"baseline"`) or `"tie"`, mapped back from the judge's blind A/B answer.
|
||||
- **`hard`** carries the gate result for that trial's new arm.
|
||||
Omit it or set `ran: false` when the case has no hard assertions.
|
||||
- **`usage`** is the per-arm-per-trial transcript usage.
|
||||
Recover it by reading each arm-subagent's transcript file (the per-agent JSONL the workflow writes) and summing each message's token usage into the four components — `input`, `output`, `cacheCreation` (cache-creation input tokens), `cacheRead` (cache-read input tokens).
|
||||
`turns` is the count of tool-call rounds in that transcript.
|
||||
If a transcript genuinely lacks usage data, record zeros rather than guessing — cost never gates a verdict.
|
||||
|
||||
## 6. Score and render
|
||||
|
||||
Run the core over the bundle, writing the report and a machine-readable model:
|
||||
|
||||
```sh
|
||||
python3 skills/benchmark-skill/core/benchmark_core.py \
|
||||
tests/.reports/.work/<name>-bundle.json \
|
||||
--json tests/.reports/<name>.results.json \
|
||||
--html tests/.reports/<name>.html
|
||||
```
|
||||
|
||||
The core collapses each efficacy trial to WIN, TIE, or LOSS for the new arm (a tie is a non-win), applies the pass rule — **efficacy passes for a case when wins ≥ 3 and losses ≤ 1** — flags any loss for human review, and yields a skill-level **Efficacy verdict** that is green only when every case passes.
|
||||
It renders a self-contained HTML report: the Efficacy badge and run metadata, a two-row per-arm cost table (footnoted that absolute cost is inflated by shared-context cache overhead, so the trustworthy signal is the new-vs-no-skill ratio), and the cases in stable authored order.
|
||||
Cost is reported alongside quality but never gates the verdict.
|
||||
|
||||
Clean up the `tests/.reports/.work/` scratch directory when done.
|
||||
|
||||
Done when `tests/.reports/<name>.html` exists.
|
||||
Report the Efficacy verdict, the path to the report, and any flagged losses to the user.
|
||||
Reference in New Issue
Block a user