alexion
d445b2bd2b
Add the benchmark-harness spec, three supporting ADRs (cost-equivalent token metric, single-user seed, guard-based tool isolation), and the 0022-0030 task breakdown that slices the harness into foundational seams, an integrating single-cell runner, and the reporting layer.
20 lines
1.7 KiB
Markdown
20 lines
1.7 KiB
Markdown
---
|
|
spec: benchmark-harness
|
|
---
|
|
|
|
## What to build
|
|
|
|
The pure scoring seam that turns a completed run into a deterministic pass/fail, plus the scoring-spec contract each task pairs with.
|
|
|
|
For a mutation task, the checker diffs the entire post-run repository state against the expected end state, so both the intended change and any collateral damage are caught. Comparison runs after normalization: volatile identifiers and timestamps are dropped, comments are matched by author and body, and labels are compared as sets. For a read task, the checker matches required facts in the agent's final report against the seeded ground truth, with no LLM judge.
|
|
|
|
This slice also defines the scoring-spec contract — a task's expected end state (for mutations) or its required answer facts (for reads) — that the checker consumes and that the runner and task suite will produce. The checker is fed synthetic state snapshots and expected states; capturing live state from a real repository is the runner's job.
|
|
|
|
## Acceptance criteria
|
|
|
|
- [ ] Given synthetic actual and expected state snapshots, the full-state diff passes when they match after normalization and fails when the actual state is missing the intended change.
|
|
- [ ] The diff fails when the actual state carries collateral change beyond the intended mutation.
|
|
- [ ] Normalization drops volatile identifiers and timestamps, matches comments by author and body, and compares label sets order-independently.
|
|
- [ ] The read-task answer-match passes when the required facts are present in the final report and fails when a required fact is missing.
|
|
- [ ] The scoring-spec contract expresses both a mutation's expected end state and a read's required answer facts.
|