feat: add efficacy-benchmark tracer bullet (task 0004)

Introduce the thinnest complete path that benchmarks a skill's efficacy
against a no-skill baseline and renders an HTML report.

- tests/ convention: a top-level tree mirroring skills/ by full path, with
  case directories holding a case.md (CASE-FORMAT.md) and an optional fixture.
- benchmark-skill: a distributed, non-model-invocable runner (/benchmark-skill
  <name>) that orchestrates force-invoked new-skill and skill-absent baseline
  arms plus a blind per-trial judge as in-session subagents.
- Deterministic core (Python): a pure transform over the collected run data
  that sums per-arm usage, collapses each trial to WIN/TIE/LOSS, applies the
  efficacy pass rule (wins >= 3, losses <= 1), and renders a self-contained
  report. Parameterized over arms and comparisons; two-arm here.
- Fixture-driven no-LLM check (checks/benchmark-core.nix) wired into flake.nix.
- Real tests tree for axi-review; a live run produced its efficacy report.
This commit is contained in:
2026-07-24 08:55:55 -04:00
parent 1caf18d72d
commit e35ed9d05b
10 changed files with 1226 additions and 0 deletions

3
.gitignore vendored
View File

@@ -6,3 +6,6 @@ result-*
# Ignore automatically generated direnv output
.direnv
# Benchmark run artifacts: reports and local history, keyed by skill name.
tests/.reports/