feat: add efficacy-benchmark tracer bullet (task 0004)
Introduce the thinnest complete path that benchmarks a skill's efficacy against a no-skill baseline and renders an HTML report. - tests/ convention: a top-level tree mirroring skills/ by full path, with case directories holding a case.md (CASE-FORMAT.md) and an optional fixture. - benchmark-skill: a distributed, non-model-invocable runner (/benchmark-skill <name>) that orchestrates force-invoked new-skill and skill-absent baseline arms plus a blind per-trial judge as in-session subagents. - Deterministic core (Python): a pure transform over the collected run data that sums per-arm usage, collapses each trial to WIN/TIE/LOSS, applies the efficacy pass rule (wins >= 3, losses <= 1), and renders a self-contained report. Parameterized over arms and comparisons; two-arm here. - Fixture-driven no-LLM check (checks/benchmark-core.nix) wired into flake.nix. - Real tests tree for axi-review; a live run produced its efficacy report.
This commit is contained in:
17
tests/skills/axi-review/basic-cli-review/case.md
Normal file
17
tests/skills/axi-review/basic-cli-review/case.md
Normal file
@@ -0,0 +1,17 @@
|
||||
---
|
||||
description: Review a tiny CLI against the 10 AXI principles and produce a report card.
|
||||
---
|
||||
## Prompt
|
||||
There is a small command-line tool named `greet` in the current directory (`./greet`).
|
||||
Review it against the 10 AXI (Agent eXperience Interface) principles by running it black-box, and give me a report card that scores each principle with a concrete verdict and a fix for anything that isn't a clean pass.
|
||||
|
||||
## Hard assertions
|
||||
```sh
|
||||
grep -qiE 'AXI|principle' "$OUTPUT"
|
||||
grep -qiE 'PASS|PARTIAL|FAIL' "$OUTPUT"
|
||||
```
|
||||
|
||||
## Soft criteria
|
||||
- The review scores each of the 10 AXI principles with an explicit verdict grounded in the tool's actual observed output, not in guesses about its source.
|
||||
- Every principle that is not a clean pass carries a concrete, actionable fix.
|
||||
- The review is driven by running the `greet` tool and citing what it emitted, rather than by generic CLI advice.
|
||||
41
tests/skills/axi-review/basic-cli-review/fixture/greet
Executable file
41
tests/skills/axi-review/basic-cli-review/fixture/greet
Executable file
@@ -0,0 +1,41 @@
|
||||
#!/usr/bin/env sh
|
||||
# greet: a tiny CLI, deliberately imperfect, for exercising an AXI review.
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
greet — say hello
|
||||
|
||||
USAGE:
|
||||
greet hello <name> print a greeting
|
||||
greet json <name> print the greeting as JSON
|
||||
greet --help show this help
|
||||
EOF
|
||||
}
|
||||
|
||||
case "$1" in
|
||||
--help|-h|help)
|
||||
usage
|
||||
;;
|
||||
hello)
|
||||
if [ -z "$2" ]; then
|
||||
echo "error: missing name" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "Hello, $2!"
|
||||
;;
|
||||
json)
|
||||
if [ -z "$2" ]; then
|
||||
echo "error: missing name" >&2
|
||||
exit 1
|
||||
fi
|
||||
printf '{"greeting":"Hello","name":"%s"}\n' "$2"
|
||||
;;
|
||||
"")
|
||||
usage >&2
|
||||
exit 2
|
||||
;;
|
||||
*)
|
||||
echo "error: unknown command '$1'" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
Reference in New Issue
Block a user