feat: persist agent report on read result records (task 0032)
All checks were successful
CI / test (pull_request) Successful in 52s
All checks were successful
CI / test (pull_request) Successful in 52s
Retain the agent's final report on the benchmark result record for read tasks, so a failed read is diagnosable directly from the stored record instead of only carrying an opaque `incorrect` tag. The runner resolves the scoring spec once and records `run.finalReport` when the spec is a read; mutation records omit the field entirely. The sample store needs no change — it serializes whatever record it is handed. This is the prerequisite for confirming the read-open-issue-count failure from real report text before the state-aware count-line change (task 0033).
This commit is contained in:
@@ -72,6 +72,15 @@ export interface ResultRecord {
|
||||
|
||||
/** Pass/fail outcome with a failure tag. */
|
||||
outcome: Outcome;
|
||||
|
||||
/**
|
||||
* The agent's final report, retained for read tasks so a failed read is
|
||||
* diagnosable directly from the record — the exact text the agent submitted,
|
||||
* distinguishing a wrong answer from a right one phrased in words the checker's
|
||||
* accepted-phrasing list did not match. Absent for mutation tasks, which are
|
||||
* scored by diffing repository state and have no agent report to record.
|
||||
*/
|
||||
report?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
Reference in New Issue
Block a user