feat: add efficacy-benchmark tracer bullet (task 0004)

Introduce the thinnest complete path that benchmarks a skill's efficacy
against a no-skill baseline and renders an HTML report.

- tests/ convention: a top-level tree mirroring skills/ by full path, with
  case directories holding a case.md (CASE-FORMAT.md) and an optional fixture.
- benchmark-skill: a distributed, non-model-invocable runner (/benchmark-skill
  <name>) that orchestrates force-invoked new-skill and skill-absent baseline
  arms plus a blind per-trial judge as in-session subagents.
- Deterministic core (Python): a pure transform over the collected run data
  that sums per-arm usage, collapses each trial to WIN/TIE/LOSS, applies the
  efficacy pass rule (wins >= 3, losses <= 1), and renders a self-contained
  report. Parameterized over arms and comparisons; two-arm here.
- Fixture-driven no-LLM check (checks/benchmark-core.nix) wired into flake.nix.
- Real tests tree for axi-review; a live run produced its efficacy report.
This commit is contained in:
2026-07-24 08:55:55 -04:00
parent 1caf18d72d
commit e35ed9d05b
10 changed files with 1226 additions and 0 deletions

View File

@@ -0,0 +1,41 @@
#!/usr/bin/env sh
# greet: a tiny CLI, deliberately imperfect, for exercising an AXI review.
usage() {
cat <<'EOF'
greet — say hello
USAGE:
greet hello <name> print a greeting
greet json <name> print the greeting as JSON
greet --help show this help
EOF
}
case "$1" in
--help|-h|help)
usage
;;
hello)
if [ -z "$2" ]; then
echo "error: missing name" >&2
exit 1
fi
echo "Hello, $2!"
;;
json)
if [ -z "$2" ]; then
echo "error: missing name" >&2
exit 1
fi
printf '{"greeting":"Hello","name":"%s"}\n' "$2"
;;
"")
usage >&2
exit 2
;;
*)
echo "error: unknown command '$1'" >&2
exit 2
;;
esac