alexion/gitea-axi
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add benchmark harness spec, ADRs, and task breakdown
All checks were successful
CI / test (pull_request) Successful in 50s
Details
CI / test (push) Successful in 49s
Details

Browse Source 
Add the benchmark-harness spec, three supporting ADRs (cost-equivalent
token metric, single-user seed, guard-based tool isolation), and the
0022-0030 task breakdown that slices the harness into foundational
seams, an integrating single-cell runner, and the reporting layer.
This commit was merged in pull request #22.
This commit is contained in:
alexion
2026-07-15 10:04:28 -04:00
parent b5955cd7b8
commit d445b2bd2b
13 changed files with 377 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
.claude/tasks/0024-bench-checker-and-scoring-spec.md Normal file
View File

@@ -0,0 +1,19 @@
---
spec: benchmark-harness
---
## What to build
The pure scoring seam that turns a completed run into a deterministic pass/fail, plus the scoring-spec contract each task pairs with.
For a mutation task, the checker diffs the entire post-run repository state against the expected end state, so both the intended change and any collateral damage are caught. Comparison runs after normalization: volatile identifiers and timestamps are dropped, comments are matched by author and body, and labels are compared as sets. For a read task, the checker matches required facts in the agent's final report against the seeded ground truth, with no LLM judge.
This slice also defines the scoring-spec contract — a task's expected end state (for mutations) or its required answer facts (for reads) — that the checker consumes and that the runner and task suite will produce. The checker is fed synthetic state snapshots and expected states; capturing live state from a real repository is the runner's job.
## Acceptance criteria
- [ ] Given synthetic actual and expected state snapshots, the full-state diff passes when they match after normalization and fails when the actual state is missing the intended change.
- [ ] The diff fails when the actual state carries collateral change beyond the intended mutation.
- [ ] Normalization drops volatile identifiers and timestamps, matches comments by author and body, and compares label sets order-independently.
- [ ] The read-task answer-match passes when the required facts are present in the final report and fails when a required fact is missing.
- [ ] The scoring-spec contract expresses both a mutation's expected end state and a read's required answer facts.