alexion/gitea-axi
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add benchmark checker and scoring spec (task 0024) #25

Merged
alexion merged 1 commits from task-0024-bench-checker-and-scoring-spec into main 2026-07-16 07:39:47 -04:00
Conversation 0 Commits 1 Files Changed 6 +873 -5
alexion commented 2026-07-16 07:34:03 -04:00
Owner
Copy Link

Task: .claude/tasks/0024-bench-checker-and-scoring-spec.md

Summary

Adds the benchmark harness's pure scoring seam and the scoring-spec contract it consumes, in bench/ (dev-only, excluded from the npm package):

  • bench/scoring-spec.ts — the contract: RepoState (labels, issues, pull requests, with comments/reviews/assignees and volatile id/timestamp fields), RequiredFact, and the ScoringSpec discriminated union (mutation expected end state | read required answer facts).
  • bench/checker.ts — the logic: checkMutation (full-state diff), checkReadAnswer (deterministic, no LLM judge), and score which dispatches on task kind. Mirrors the existing result.ts (shape) / store.ts (logic) split.

Mutation scoring diffs the entire post-run state against the expected end state, so both the intended change and collateral damage are caught. Normalization drops volatile ids/timestamps, matches comments by author and body, and compares label/assignee/review sets order-independently. Read scoring matches each required fact's anyOf renderings against the report (case/whitespace-normalized). Failures carry human-readable differences naming the diverging entity/label/comment/fact.

Built test-first (44 bench tests). No acceptance criteria dropped; all five satisfied.

Deviations

  • The diff covers the whole PR/issue surface (reviews, assignees, label definitions) beyond the comment/label-set clauses the criteria name — this is what "diffing the entire post-run repository state" requires and what the contract must express for the scored suite's review/merge/assignee tasks.
  • A code-review finding was fixed: reviews are now matched order-independently (with a test), consistent with comments and labels.

Review

Overall: Low

  • Blast radius: Low — two new self-contained bench/ files plus tests; nothing imports them yet (later slices consume the seam).
  • Reversibility: Low — all-new files, no migrations or published surface.
  • Test coverage: Low (well-covered) — focused unit tests for checkMutation, checkReadAnswer, score dispatch, error paths, and the contract shape.
  • Sensitive domain: Low — pure scoring/diff logic; no auth, permissions, concurrency, or migrations.
  • Size & complexity: Low — straightforward map/diff helpers, flat control flow.
  • Runtime criticality: Low — bench/ is dev-only tooling, excluded from the published npm package.

Standards (unaddressed)

  • Duplicated score mismatch guards (judgement call) — left as-is: the two throw arms read clearly and explicitly; extracting a helper for two three-line guards adds indirection for little gain. (The other two Duplicated-Code findings — issue/PR field diffing and the repeated order-independent-collection matching — were addressed by extracting diffConversation and a shared matchByKey primitive.)

Spec (unaddressed)

  • Read-answer substring matching is naive (e.g. "#42" matches inside "#420") — deliberate: kept as deterministic case/whitespace-normalized substring matching with per-fact anyOf alternatives, no LLM judge. Disambiguation is the task suite's job (choosing robust anyOf renderings), not the checker's. Documented in the task's Implementation Notes.
  • Read facts vs. "seeded ground truth" phrasing (partial) — the required facts carried in the ScoringSpec are the read task's ground truth; this is the intended contract shape.
Task: `.claude/tasks/0024-bench-checker-and-scoring-spec.md` ## Summary Adds the benchmark harness's pure scoring seam and the scoring-spec contract it consumes, in `bench/` (dev-only, excluded from the npm package): - **`bench/scoring-spec.ts`** — the contract: `RepoState` (labels, issues, pull requests, with comments/reviews/assignees and volatile id/timestamp fields), `RequiredFact`, and the `ScoringSpec` discriminated union (`mutation` expected end state | `read` required answer facts). - **`bench/checker.ts`** — the logic: `checkMutation` (full-state diff), `checkReadAnswer` (deterministic, no LLM judge), and `score` which dispatches on task kind. Mirrors the existing `result.ts` (shape) / `store.ts` (logic) split. Mutation scoring diffs the entire post-run state against the expected end state, so both the intended change and collateral damage are caught. Normalization drops volatile ids/timestamps, matches comments by author and body, and compares label/assignee/review sets order-independently. Read scoring matches each required fact's `anyOf` renderings against the report (case/whitespace-normalized). Failures carry human-readable `differences` naming the diverging entity/label/comment/fact. Built test-first (44 bench tests). No acceptance criteria dropped; all five satisfied. ### Deviations - The diff covers the whole PR/issue surface (reviews, assignees, label definitions) beyond the comment/label-set clauses the criteria name — this is what "diffing the entire post-run repository state" requires and what the contract must express for the scored suite's review/merge/assignee tasks. - A code-review finding was fixed: reviews are now matched order-independently (with a test), consistent with comments and labels. ## Review **Overall: Low** - Blast radius: Low — two new self-contained `bench/` files plus tests; nothing imports them yet (later slices consume the seam). - Reversibility: Low — all-new files, no migrations or published surface. - Test coverage: Low (well-covered) — focused unit tests for `checkMutation`, `checkReadAnswer`, `score` dispatch, error paths, and the contract shape. - Sensitive domain: Low — pure scoring/diff logic; no auth, permissions, concurrency, or migrations. - Size & complexity: Low — straightforward map/diff helpers, flat control flow. - Runtime criticality: Low — `bench/` is dev-only tooling, excluded from the published npm package. ### Standards (unaddressed) - **Duplicated `score` mismatch guards** (judgement call) — left as-is: the two throw arms read clearly and explicitly; extracting a helper for two three-line guards adds indirection for little gain. (The other two Duplicated-Code findings — issue/PR field diffing and the repeated order-independent-collection matching — were addressed by extracting `diffConversation` and a shared `matchByKey` primitive.) ### Spec (unaddressed) - **Read-answer substring matching is naive** (e.g. `"#42"` matches inside `"#420"`) — deliberate: kept as deterministic case/whitespace-normalized substring matching with per-fact `anyOf` alternatives, no LLM judge. Disambiguation is the task suite's job (choosing robust `anyOf` renderings), not the checker's. Documented in the task's Implementation Notes. - **Read facts vs. "seeded ground truth" phrasing** (partial) — the required facts carried in the `ScoringSpec` *are* the read task's ground truth; this is the intended contract shape.
alexion added 1 commit 2026-07-16 07:34:03 -04:00
feat: add benchmark checker and scoring spec (task 0024)
All checks were successful
CI / test (pull_request) Successful in 51s
Details
CI / test (push) Successful in 48s
Details
66f576a2b7
alexion merged commit 66f576a2b7 into main 2026-07-16 07:39:47 -04:00
alexion deleted branch task-0024-bench-checker-and-scoring-spec 2026-07-16 07:39:47 -04:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
alexion
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: alexion/gitea-axi#25
Delete Branch "task-0024-bench-checker-and-scoring-spec"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?