alexion
e35ed9d05b
Introduce the thinnest complete path that benchmarks a skill's efficacy against a no-skill baseline and renders an HTML report. - tests/ convention: a top-level tree mirroring skills/ by full path, with case directories holding a case.md (CASE-FORMAT.md) and an optional fixture. - benchmark-skill: a distributed, non-model-invocable runner (/benchmark-skill <name>) that orchestrates force-invoked new-skill and skill-absent baseline arms plus a blind per-trial judge as in-session subagents. - Deterministic core (Python): a pure transform over the collected run data that sums per-arm usage, collapses each trial to WIN/TIE/LOSS, applies the efficacy pass rule (wins >= 3, losses <= 1), and renders a self-contained report. Parameterized over arms and comparisons; two-arm here. - Fixture-driven no-LLM check (checks/benchmark-core.nix) wired into flake.nix. - Real tests tree for axi-review; a live run produced its efficacy report.
18 lines
884 B
Markdown
18 lines
884 B
Markdown
---
|
|
description: Review a tiny CLI against the 10 AXI principles and produce a report card.
|
|
---
|
|
## Prompt
|
|
There is a small command-line tool named `greet` in the current directory (`./greet`).
|
|
Review it against the 10 AXI (Agent eXperience Interface) principles by running it black-box, and give me a report card that scores each principle with a concrete verdict and a fix for anything that isn't a clean pass.
|
|
|
|
## Hard assertions
|
|
```sh
|
|
grep -qiE 'AXI|principle' "$OUTPUT"
|
|
grep -qiE 'PASS|PARTIAL|FAIL' "$OUTPUT"
|
|
```
|
|
|
|
## Soft criteria
|
|
- The review scores each of the 10 AXI principles with an explicit verdict grounded in the tool's actual observed output, not in guesses about its source.
|
|
- Every principle that is not a clean pass carries a concrete, actionable fix.
|
|
- The review is driven by running the `greet` tool and citing what it emitted, rather than by generic CLI advice.
|