alexion
e35ed9d05b
Introduce the thinnest complete path that benchmarks a skill's efficacy against a no-skill baseline and renders an HTML report. - tests/ convention: a top-level tree mirroring skills/ by full path, with case directories holding a case.md (CASE-FORMAT.md) and an optional fixture. - benchmark-skill: a distributed, non-model-invocable runner (/benchmark-skill <name>) that orchestrates force-invoked new-skill and skill-absent baseline arms plus a blind per-trial judge as in-session subagents. - Deterministic core (Python): a pure transform over the collected run data that sums per-arm usage, collapses each trial to WIN/TIE/LOSS, applies the efficacy pass rule (wins >= 3, losses <= 1), and renders a self-contained report. Parameterized over arms and comparisons; two-arm here. - Fixture-driven no-LLM check (checks/benchmark-core.nix) wired into flake.nix. - Real tests tree for axi-review; a live run produced its efficacy report.
42 lines
714 B
Bash
Executable File
42 lines
714 B
Bash
Executable File
#!/usr/bin/env sh
|
|
# greet: a tiny CLI, deliberately imperfect, for exercising an AXI review.
|
|
|
|
usage() {
|
|
cat <<'EOF'
|
|
greet — say hello
|
|
|
|
USAGE:
|
|
greet hello <name> print a greeting
|
|
greet json <name> print the greeting as JSON
|
|
greet --help show this help
|
|
EOF
|
|
}
|
|
|
|
case "$1" in
|
|
--help|-h|help)
|
|
usage
|
|
;;
|
|
hello)
|
|
if [ -z "$2" ]; then
|
|
echo "error: missing name" >&2
|
|
exit 1
|
|
fi
|
|
echo "Hello, $2!"
|
|
;;
|
|
json)
|
|
if [ -z "$2" ]; then
|
|
echo "error: missing name" >&2
|
|
exit 1
|
|
fi
|
|
printf '{"greeting":"Hello","name":"%s"}\n' "$2"
|
|
;;
|
|
"")
|
|
usage >&2
|
|
exit 2
|
|
;;
|
|
*)
|
|
echo "error: unknown command '$1'" >&2
|
|
exit 2
|
|
;;
|
|
esac
|