alexion
e35ed9d05b
Introduce the thinnest complete path that benchmarks a skill's efficacy against a no-skill baseline and renders an HTML report. - tests/ convention: a top-level tree mirroring skills/ by full path, with case directories holding a case.md (CASE-FORMAT.md) and an optional fixture. - benchmark-skill: a distributed, non-model-invocable runner (/benchmark-skill <name>) that orchestrates force-invoked new-skill and skill-absent baseline arms plus a blind per-trial judge as in-session subagents. - Deterministic core (Python): a pure transform over the collected run data that sums per-arm usage, collapses each trial to WIN/TIE/LOSS, applies the efficacy pass rule (wins >= 3, losses <= 1), and renders a self-contained report. Parameterized over arms and comparisons; two-arm here. - Fixture-driven no-LLM check (checks/benchmark-core.nix) wired into flake.nix. - Real tests tree for axi-review; a live run produced its efficacy report.
12 lines
250 B
Plaintext
12 lines
250 B
Plaintext
# ---> Nix
|
|
# Ignore build outputs from performing a nix-build or `nix build` command
|
|
result
|
|
result-*
|
|
|
|
# Ignore automatically generated direnv output
|
|
.direnv
|
|
|
|
# Benchmark run artifacts: reports and local history, keyed by skill name.
|
|
tests/.reports/
|
|
|