alexion
e35ed9d05b
Introduce the thinnest complete path that benchmarks a skill's efficacy against a no-skill baseline and renders an HTML report. - tests/ convention: a top-level tree mirroring skills/ by full path, with case directories holding a case.md (CASE-FORMAT.md) and an optional fixture. - benchmark-skill: a distributed, non-model-invocable runner (/benchmark-skill <name>) that orchestrates force-invoked new-skill and skill-absent baseline arms plus a blind per-trial judge as in-session subagents. - Deterministic core (Python): a pure transform over the collected run data that sums per-arm usage, collapses each trial to WIN/TIE/LOSS, applies the efficacy pass rule (wins >= 3, losses <= 1), and renders a self-contained report. Parameterized over arms and comparisons; two-arm here. - Fixture-driven no-LLM check (checks/benchmark-core.nix) wired into flake.nix. - Real tests tree for axi-review; a live run produced its efficacy report.
884 B
884 B
description
| description |
|---|
| Review a tiny CLI against the 10 AXI principles and produce a report card. |
Prompt
There is a small command-line tool named greet in the current directory (./greet).
Review it against the 10 AXI (Agent eXperience Interface) principles by running it black-box, and give me a report card that scores each principle with a concrete verdict and a fix for anything that isn't a clean pass.
Hard assertions
grep -qiE 'AXI|principle' "$OUTPUT"
grep -qiE 'PASS|PARTIAL|FAIL' "$OUTPUT"
Soft criteria
- The review scores each of the 10 AXI principles with an explicit verdict grounded in the tool's actual observed output, not in guesses about its source.
- Every principle that is not a clean pass carries a concrete, actionable fix.
- The review is driven by running the
greettool and citing what it emitted, rather than by generic CLI advice.