feat: persist the tool transcript on every result record

Records stored only a run's token/turn totals, so an arm's turn cost —
the dominant driver of cache-read tokens — could not be diagnosed from
the store. Retain the ordered transcript of tool invocations (the exact
shell commands, MCP calls, and built-in tools the run made) on every
scored record, absent only for a hung run that produced no transcript.

The canonical TranscriptEntry shape lives on the record (result.ts); the
isolation audit's ToolUse now aliases it so the persisted and audited
shapes cannot drift.
This commit is contained in:
2026-07-17 14:43:43 -04:00
parent 05b72f6987
commit 14d494afa2
4 changed files with 40 additions and 8 deletions

View File

@@ -45,6 +45,19 @@ export type FailureTag = "incorrect" | "confused" | "hung";
/** The pass/fail outcome of a run, tagged with the failure mode when it fails. */
export type Outcome = { pass: true } | { pass: false; failure: FailureTag };
/**
* One tool invocation as it is recorded in a run's transcript, in the order it
* executed. This is the canonical shape the harness both audits for isolation
* (see audit.ts, whose `ToolUse` aliases this) and persists on the record for
* diagnosis. A `shell` entry keeps the exact command line the agent ran; an `mcp`
* entry names the server and tool it called; `other` names a built-in tool that
* reaches no Gitea channel.
*/
export type TranscriptEntry =
| { kind: "shell"; command: string }
| { kind: "mcp"; server: string; tool: string }
| { kind: "other"; name: string };
/**
* One completed `(arm, task, trial)` run. Carries the metrics the headline and
* supporting views are computed from, plus the tags those views group by.
@@ -81,6 +94,15 @@ export interface ResultRecord {
* scored by diffing repository state and have no agent report to record.
*/
report?: string;
/**
* The ordered transcript of tool invocations the run made, retained on every
* scored run so its turn cost is diagnosable directly from the record — the
* exact command sequence, which is how an arm's turn count (the dominant driver
* of cache-read tokens) is explained. Absent only for a hung run, which
* produced no completed transcript to record.
*/
transcript?: TranscriptEntry[];
}
/**