feat: add efficacy-benchmark tracer bullet (task 0004)
Introduce the thinnest complete path that benchmarks a skill's efficacy against a no-skill baseline and renders an HTML report. - tests/ convention: a top-level tree mirroring skills/ by full path, with case directories holding a case.md (CASE-FORMAT.md) and an optional fixture. - benchmark-skill: a distributed, non-model-invocable runner (/benchmark-skill <name>) that orchestrates force-invoked new-skill and skill-absent baseline arms plus a blind per-trial judge as in-session subagents. - Deterministic core (Python): a pure transform over the collected run data that sums per-arm usage, collapses each trial to WIN/TIE/LOSS, applies the efficacy pass rule (wins >= 3, losses <= 1), and renders a self-contained report. Parameterized over arms and comparisons; two-arm here. - Fixture-driven no-LLM check (checks/benchmark-core.nix) wired into flake.nix. - Real tests tree for axi-review; a live run produced its efficacy report.
This commit is contained in:
41
tests/skills/axi-review/basic-cli-review/fixture/greet
Executable file
41
tests/skills/axi-review/basic-cli-review/fixture/greet
Executable file
@@ -0,0 +1,41 @@
|
||||
#!/usr/bin/env sh
|
||||
# greet: a tiny CLI, deliberately imperfect, for exercising an AXI review.
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
greet — say hello
|
||||
|
||||
USAGE:
|
||||
greet hello <name> print a greeting
|
||||
greet json <name> print the greeting as JSON
|
||||
greet --help show this help
|
||||
EOF
|
||||
}
|
||||
|
||||
case "$1" in
|
||||
--help|-h|help)
|
||||
usage
|
||||
;;
|
||||
hello)
|
||||
if [ -z "$2" ]; then
|
||||
echo "error: missing name" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "Hello, $2!"
|
||||
;;
|
||||
json)
|
||||
if [ -z "$2" ]; then
|
||||
echo "error: missing name" >&2
|
||||
exit 1
|
||||
fi
|
||||
printf '{"greeting":"Hello","name":"%s"}\n' "$2"
|
||||
;;
|
||||
"")
|
||||
usage >&2
|
||||
exit 2
|
||||
;;
|
||||
*)
|
||||
echo "error: unknown command '$1'" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
Reference in New Issue
Block a user