chore(vendor): import pi 1.3.0
Imported as a linear vendor commit so forge rebases do not replay the upstream root tree. git-subtree-dir: third_party/pi git-subtree-split: 71023aec1546d6d677a65669c5665c0927ce0dd0
This commit is contained in:
230
third_party/pi/packages/evals/README.md
vendored
230
third_party/pi/packages/evals/README.md
vendored
@@ -1,153 +1,129 @@
|
||||
# Pi evals
|
||||
|
||||
Pi evals are behavioral, model-backed checks for Pi workflows. They adapt a real `AgentSession` to `vitest-evals`, run
|
||||
it in isolated temporary project and agent directories, and attach native Pi session artifacts.
|
||||
Use them to measure end-to-end behavior and compare prompts, tools, skills, models, or other harness configurations.
|
||||
Behavioral evals for Pi's coding agent, built with `vitest-evals`.
|
||||
|
||||
## Running evals
|
||||
## File conventions
|
||||
|
||||
Run from the repository root with a default provider and model:
|
||||
Eval definitions are flat under `evals/`:
|
||||
|
||||
- `*.docs.eval.ts` is a documentation-lift eval. `eval:docs` runs each case in isolated `without_docs` and `with_docs` containers and reports lift.
|
||||
- Other `*.eval.ts` files are host evals. `eval:host` runs them with Vitest on this machine. They are ordinary vitest-evals suites, not paired comparisons.
|
||||
|
||||
Runner code lives in `src/`:
|
||||
|
||||
- `cli.ts` orchestrates a comparison
|
||||
- `docker.ts` builds the two images, discovers cases, and runs one isolated arm
|
||||
- `plan.ts` expands cases into `(case, variant, repetition)` tasks
|
||||
- `report.ts` reads Vitest JSON, pairs arms, and computes lift
|
||||
- `harness.ts` is the vitest-evals adapter
|
||||
|
||||
Eval suites and their fixtures live under `evals/`. Image build files live in `docker/`.
|
||||
|
||||
## Run evals
|
||||
|
||||
Host evals (smoke, documentation audit) and documentation-lift evals need `PI_PROVIDER` and `PI_MODEL`.
|
||||
|
||||
```bash
|
||||
npm run eval -- --provider openai --model gpt-5.6-sol
|
||||
PI_PROVIDER=openai-codex PI_MODEL=gpt-5.6-sol npm run eval -w packages/evals
|
||||
```
|
||||
|
||||
The equivalent environment variables are:
|
||||
That runs host evals, then the documentation comparison. Extra CLI flags after `--` go to `eval:docs` only.
|
||||
|
||||
Host only:
|
||||
|
||||
```bash
|
||||
PI_PROVIDER=openai PI_MODEL=gpt-5.6-sol npm run eval
|
||||
PI_PROVIDER=openai-codex PI_MODEL=gpt-5.6-sol npm run eval:host -w packages/evals
|
||||
```
|
||||
|
||||
CLI values take precedence and become defaults for harnesses that do not select a model explicitly. Provider and model must be supplied together. The runner also allows no default when every executed harness configures its own model.
|
||||
Authentication comes from Pi's normal `ModelRuntime`, including Pi subscription credentials and provider API-key
|
||||
environment variables.
|
||||
|
||||
Additional arguments are forwarded to Vitest:
|
||||
One host suite:
|
||||
|
||||
```bash
|
||||
npm run eval -- src/extensions.eval.ts
|
||||
npm run eval -- -t "creates, reloads, and uses"
|
||||
PI_PROVIDER=openai-codex PI_MODEL=gpt-5.6-sol \
|
||||
npm run eval:host -w packages/evals -- evals/documentation-audit.eval.ts
|
||||
```
|
||||
|
||||
Each invocation prints an ignored `.eval/` artifact directory. `runs.jsonl` indexes completed harness runs and their
|
||||
native Pi session JSONL attachments under `sessions/`. These files may contain prompts, responses, source code, and tool
|
||||
output.
|
||||
## Run documentation comparisons
|
||||
|
||||
## Writing evals
|
||||
From the repository root:
|
||||
|
||||
Follow [`vitest-evals`](https://github.com/getsentry/vitest-evals) for general suite, judge, assertion, and normalized
|
||||
trace guidance. Pi-specific evals use `createPiCodingAgentHarness(...)` from `src/pi-harness.ts`, with one harness bound
|
||||
to each `describeEval(...)` suite:
|
||||
```bash
|
||||
npm run eval:docs -w packages/evals -- \
|
||||
--provider openai-codex \
|
||||
--model gpt-5.6-sol
|
||||
```
|
||||
|
||||
`PI_PROVIDER` and `PI_MODEL` provide the same defaults. Both values are required.
|
||||
|
||||
The default is one run per variant. Increase repetitions explicitly when measuring stability:
|
||||
|
||||
```bash
|
||||
npm run eval:docs -w packages/evals -- \
|
||||
evals/extensions.docs.eval.ts \
|
||||
--runs-per-variant 5
|
||||
```
|
||||
|
||||
`PI_EVAL_RUNS_PER_VARIANT=5` is equivalent. Vitest filters are applied during discovery:
|
||||
|
||||
```bash
|
||||
npm run eval:docs -w packages/evals -- -t "adds the model"
|
||||
```
|
||||
|
||||
The runner:
|
||||
|
||||
1. Mounts the repository ephemerally for a Docker build, packs the current workspace packages using the repository's consumer-install machinery, then creates separate `without_docs` and `with_docs` images from the staged runtime.
|
||||
2. Discovers the selected cases in both images and requires identical cohorts.
|
||||
3. Plans every `(case, variant, model, runNumber)` arm before execution.
|
||||
4. Runs each arm in a fresh container. A failed or missing arm is recorded and the planned cohort continues.
|
||||
5. Reads native Vitest JSON through `@vitest-evals/core/node` when a report exists.
|
||||
6. Pairs exact arms and writes the comparison report. Blocked pairs withhold headline lift; the process exits nonzero.
|
||||
|
||||
Repetition order alternates by run number to reduce order bias.
|
||||
|
||||
## Documentation variants
|
||||
|
||||
`without_docs` omits the coding-agent `README.md`, `CHANGELOG.md`, `docs/`, and `examples/`, then removes the Pi documentation-routing section from the default system prompt.
|
||||
|
||||
`with_docs` includes those files and uses the unchanged default prompt.
|
||||
|
||||
Both variants install the same local workspace tarballs. Existing npm overrides ensure coding-agent's internal Pi dependencies also come from the current repository rather than the registry. Documentation and source files from internal dependency packages are removed symmetrically so they cannot act as alternate instructions. Startup validates the image allowlist and verifies that the installed coding-agent package resolves from `dist/`. Eval definitions, evaluator helpers, fixtures, and Vitest configuration are root-owned and unreadable after the harness permanently drops to an unprivileged UID. Each run receives a new home, agent directory, workspace, session directory, and container filesystem.
|
||||
|
||||
Documentation evals allow only `read`, `write`, `edit`, `grep`, `find`, and `ls` by default. They do not expose shell or web-search tools. Provider traffic still requires container network access, so Docker alone cannot prove that arbitrary code written by an agent never uses the network.
|
||||
|
||||
## Results
|
||||
|
||||
Each invocation creates an ignored `.eval/<timestamp>_<id>/` directory containing:
|
||||
|
||||
- `protocol.json`: model, image IDs, cases, tasks, and protocol digest.
|
||||
- `expected-runs.json`: the complete planned cohort.
|
||||
- `observations.jsonl`: normalized outcomes and telemetry.
|
||||
- `tasks/*/vitest.json`: native JSON for each isolated arm.
|
||||
- `<variant>/sessions/*/session.jsonl`: native Pi sessions.
|
||||
- `report.json` and `report.txt`: paired comparisons.
|
||||
|
||||
A pair contributes to pass-rate lift only when both arms produce exactly one score. Missing, duplicate, skipped, pending, unscored, or errored arms block the pair. If any pair in an eval set is blocked, headline pass rates are withheld. Missing telemetry remains unavailable rather than being treated as zero.
|
||||
|
||||
The report flags no lift, negative deltas, saturated controls or treatments, and observed flakiness. One repetition cannot establish stability.
|
||||
|
||||
Artifacts may contain prompts, responses, generated code, and tool output.
|
||||
|
||||
## Write an eval
|
||||
|
||||
Use one ordinary `describeEval(...)` suite and one explicit `run(...)` call per case:
|
||||
|
||||
```ts
|
||||
import { expect } from "vitest";
|
||||
import { describeEval } from "vitest-evals";
|
||||
import { createPiCodingAgentHarness } from "./pi-harness.ts";
|
||||
import { describeEval, StructuredOutputJudge } from "vitest-evals";
|
||||
import { createPiDocumentationEvalHarness } from "../src/harness.ts";
|
||||
|
||||
const harness = createPiCodingAgentHarness({ noTools: "all" });
|
||||
const harness = createPiDocumentationEvalHarness();
|
||||
const judge = StructuredOutputJudge({ expected: { ok: true }, match: "strict", allowExtras: false });
|
||||
|
||||
describeEval("Pi smoke", { harness }, (it) => {
|
||||
it("answers a factual question", async ({ run }) => {
|
||||
const result = await run("What is the capital of France? Reply with only the city name.");
|
||||
expect(result.output).toBe("Paris");
|
||||
});
|
||||
describeEval("Target workflow", { harness, judges: [judge], judgeThreshold: null }, (it) => {
|
||||
it("completes the task", async ({ run }) => {
|
||||
await run("Complete the target task.");
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### Configuring the Pi harness
|
||||
The outer runner owns variants, repetitions, isolation, identity, persistence, and reporting. Eval files should contain only scenario setup, the model task, and deterministic grading.
|
||||
|
||||
`createPiCodingAgentHarness(...)` accepts:
|
||||
|
||||
- `name`: stable harness identity used by reports and comparisons.
|
||||
- `model`: optional `{ provider, id }` selection. It overrides the runner's default model.
|
||||
- `noTools`: Pi's tool-disable configuration.
|
||||
- `transformSystemPrompt`: transforms the complete default prompt before the eval starts.
|
||||
- `output`: transforms the final response and `AgentSession` into a JSON-safe domain result.
|
||||
|
||||
An explicitly selected model makes model-comparison harnesses independent of the runner default:
|
||||
|
||||
```ts
|
||||
const harness = createPiCodingAgentHarness({
|
||||
name: "claude-opus-4-6",
|
||||
model: { provider: "anthropic", id: "claude-opus-4-6" },
|
||||
});
|
||||
```
|
||||
|
||||
A run accepts either one prompt or a sequence of prompt and reload steps. Reload steps are useful when the preceding
|
||||
prompt creates or changes Pi resources:
|
||||
|
||||
```ts
|
||||
const result = await run([
|
||||
{ type: "prompt", content: "Create a Pi extension." },
|
||||
{ type: "reload" },
|
||||
{ type: "prompt", content: "Use the extension." },
|
||||
]);
|
||||
```
|
||||
|
||||
### Transforming harness output
|
||||
|
||||
Use `output` to expose scenario-specific, JSON-safe behavior without adding that behavior to the generic Pi adapter:
|
||||
|
||||
```ts
|
||||
const harness = createPiCodingAgentHarness({
|
||||
output: ({ response, session }) => ({
|
||||
response,
|
||||
activeTools: session.getActiveToolNames(),
|
||||
extensionErrors: session.resourceLoader.getExtensions().errors,
|
||||
}),
|
||||
});
|
||||
```
|
||||
|
||||
Assert application behavior on `result.output`. Assert model and tool traces on `result.session`, using
|
||||
`vitest-evals` helpers such as `toolCalls(...)`.
|
||||
|
||||
### Writing comparative eval sets
|
||||
|
||||
Use `evalHarnessTable(...)` with Vitest's native `describe.for(...)` to run the same inputs against multiple harnesses.
|
||||
Harnesses may differ by prompt, tools, skills, model, or any other Pi configuration:
|
||||
|
||||
```ts
|
||||
import { describe } from "vitest";
|
||||
import { createJudge, describeEval } from "vitest-evals";
|
||||
import { evalHarnessTable } from "./vitest-evals/harness-table.ts";
|
||||
|
||||
const TargetTaskJudge = createJudge<string, string>("TargetTaskJudge", ({ output }) => ({
|
||||
score: output === "expected result" ? 1 : 0,
|
||||
}));
|
||||
|
||||
const harnessTable = evalHarnessTable(
|
||||
"target skill effectiveness",
|
||||
{
|
||||
baseline: withoutTargetSkillHarness,
|
||||
candidate: withTargetSkillHarness,
|
||||
repetitions: 6,
|
||||
},
|
||||
);
|
||||
|
||||
describe.for(harnessTable)("$name repetition $repetition", ({ harness }) => {
|
||||
describeEval("target skill effectiveness", { harness, judges: [TargetTaskJudge], judgeThreshold: null }, (it) => {
|
||||
it("completes the target task", async ({ run }) => {
|
||||
await run("Complete the target task.");
|
||||
});
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
Comparative suites should record correctness with deterministic or model-backed judges and set `judgeThreshold: null`.
|
||||
This keeps a low score as an observation instead of making the Vitest invocation fail. Use hard assertions only for
|
||||
suite invariants and infrastructure contracts. `expect.soft(...)` still fails the test and is not a scoring mechanism.
|
||||
|
||||
The Pi harness snapshots native session JSONL before deleting its temporary workspace. An eval-only `afterEach` hook
|
||||
registers that snapshot against the explicit Vitest test task before reporters run.
|
||||
|
||||
Harness names must be stable and unique within an eval set. The grouping key combines repetition with a non-empty string
|
||||
`input.id` when available, otherwise with a SHA-256 hash of strict canonical JSON input. Use `candidate` for one treatment
|
||||
or `candidates` for multiple treatments. Each candidate is compared only with the declared baseline. For each matched
|
||||
input and repetition, the reporter computes pass-rate lift from each run's recorded average judge score, treating a score
|
||||
of at least `1` as passing. Lift is the candidate pass rate minus the baseline pass rate, in percentage points. Missing
|
||||
judge scores are reported as incomplete observations. Tokens, latency, and estimated cost remain separate
|
||||
candidate-minus-baseline paired deltas; missing telemetry remains unavailable. If execution-order randomization becomes
|
||||
necessary, use Vitest's built-in sequence shuffling.
|
||||
|
||||
See the [`skill-eval-harness`](https://github.com/adewale/skill-eval-harness/) guidance for comparative-eval methodology,
|
||||
repetition strategy, trustworthy judges, and telemetry interpretation.
|
||||
Use `judgeThreshold: null` for comparative scoring. A low score is data, not an infrastructure failure. Reserve Vitest assertions for broken suite invariants.
|
||||
|
||||
44
third_party/pi/packages/evals/docker/Dockerfile
vendored
Normal file
44
third_party/pi/packages/evals/docker/Dockerfile
vendored
Normal file
@@ -0,0 +1,44 @@
|
||||
FROM node:24-slim AS builder
|
||||
WORKDIR /work
|
||||
# Build against an ephemeral repository mount. Only outputs staged under
|
||||
# /opt/evaluator persist; node_modules and dist created in /repo are discarded.
|
||||
RUN --mount=type=bind,source=.,target=/repo,rw \
|
||||
--mount=type=cache,target=/root/.npm \
|
||||
cd /repo \
|
||||
&& npm ci --ignore-scripts \
|
||||
&& npm run build:offline \
|
||||
&& node packages/evals/docker/install-runtime.mjs /opt/evaluator
|
||||
|
||||
# Both variants use the same installed runtime. The control removes only the
|
||||
# coding-agent documentation whose effect the experiment measures.
|
||||
FROM builder AS without-docs-install
|
||||
RUN rm -rf \
|
||||
/opt/evaluator/install/node_modules/@earendil-works/pi-coding-agent/README.md \
|
||||
/opt/evaluator/install/node_modules/@earendil-works/pi-coding-agent/CHANGELOG.md \
|
||||
/opt/evaluator/install/node_modules/@earendil-works/pi-coding-agent/docs \
|
||||
/opt/evaluator/install/node_modules/@earendil-works/pi-coding-agent/examples
|
||||
|
||||
FROM node:24-slim AS evaluator
|
||||
COPY --from=builder /opt/evaluator/root /repo
|
||||
WORKDIR /repo/packages/evals
|
||||
# Vitest starts as root so it can load evaluator code. Before prompting the
|
||||
# model, the harness permanently drops to UID 65532. Keep evaluator sources
|
||||
# root-only so model tools and generated code cannot inspect judges or fixtures.
|
||||
RUN chmod -R go-rwx \
|
||||
/repo/packages/evals/src \
|
||||
/repo/packages/evals/evals \
|
||||
/repo/packages/evals/docker \
|
||||
&& chmod go-rwx \
|
||||
/repo/vitest.base.ts \
|
||||
/repo/packages/evals/vitest.evals.config.ts \
|
||||
&& mkdir -p /run/pi-eval-secrets \
|
||||
&& chmod 0700 /run/pi-eval-secrets
|
||||
ENTRYPOINT ["node", "--experimental-strip-types", "docker/entrypoint.ts"]
|
||||
|
||||
FROM evaluator AS without_docs
|
||||
COPY --from=without-docs-install /opt/evaluator/install/node_modules /repo/node_modules
|
||||
ENV PI_EVAL_VARIANT=without_docs
|
||||
|
||||
FROM evaluator AS with_docs
|
||||
COPY --from=builder /opt/evaluator/install/node_modules /repo/node_modules
|
||||
ENV PI_EVAL_VARIANT=with_docs
|
||||
7
third_party/pi/packages/evals/docker/Dockerfile.dockerignore
vendored
Normal file
7
third_party/pi/packages/evals/docker/Dockerfile.dockerignore
vendored
Normal file
@@ -0,0 +1,7 @@
|
||||
.git
|
||||
**/.eval
|
||||
**/node_modules
|
||||
**/dist
|
||||
**/coverage
|
||||
**/.env
|
||||
**/.env.*
|
||||
152
third_party/pi/packages/evals/docker/entrypoint.ts
vendored
Normal file
152
third_party/pi/packages/evals/docker/entrypoint.ts
vendored
Normal file
@@ -0,0 +1,152 @@
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { chownSync, copyFileSync, existsSync, lstatSync, mkdirSync, readFileSync, readdirSync, statSync } from "node:fs";
|
||||
import { createRequire } from "node:module";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const variant = process.env.PI_EVAL_VARIANT;
|
||||
|
||||
function parseId(name: string): number {
|
||||
const value = Number(process.env[name]);
|
||||
if (!Number.isSafeInteger(value) || value < 1) throw new Error(`${name} must be a positive integer.`);
|
||||
return value;
|
||||
}
|
||||
|
||||
const sandboxUid = parseId("PI_EVAL_SANDBOX_UID");
|
||||
const sandboxGid = parseId("PI_EVAL_SANDBOX_GID");
|
||||
|
||||
function assertDirectoryEntries(path: string, expectedEntries: readonly string[]): void {
|
||||
const actual = readdirSync(path).sort();
|
||||
const expected = [...expectedEntries].sort();
|
||||
if (JSON.stringify(actual) !== JSON.stringify(expected)) {
|
||||
throw new Error(`Unexpected eval image contents in ${path}: ${actual.join(", ")}`);
|
||||
}
|
||||
}
|
||||
|
||||
function assertWorkspace(): void {
|
||||
assertDirectoryEntries("/repo", ["node_modules", "package.json", "packages", "vitest.base.ts"]);
|
||||
assertDirectoryEntries("/repo/packages", ["evals"]);
|
||||
assertDirectoryEntries("/repo/packages/evals", ["docker", "evals", "package.json", "src", "vitest.evals.config.ts"]);
|
||||
const codingAgentDir = "/repo/node_modules/@earendil-works/pi-coding-agent";
|
||||
for (const name of ["package.json", "dist/index.js"]) {
|
||||
if (!existsSync(join(codingAgentDir, name))) throw new Error(`Installed coding-agent is missing ${name}.`);
|
||||
}
|
||||
const internalScope = "/repo/node_modules/@earendil-works";
|
||||
for (const packageName of readdirSync(internalScope)) {
|
||||
if (packageName === "pi-coding-agent") continue;
|
||||
const packageDirectory = join(internalScope, packageName);
|
||||
for (const entry of readdirSync(packageDirectory, { withFileTypes: true })) {
|
||||
if (
|
||||
(entry.isDirectory() && ["docs", "examples", "src", "test", "tests"].includes(entry.name)) ||
|
||||
(entry.isFile() && /^(?:readme|changelog)(?:\..+)?$/i.test(entry.name))
|
||||
) {
|
||||
throw new Error(`Internal dependency documentation is model-visible: ${packageName}/${entry.name}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (variant === "without_docs") {
|
||||
for (const name of ["README.md", "CHANGELOG.md", "docs", "examples"]) {
|
||||
if (existsSync(join(codingAgentDir, name))) throw new Error(`without_docs contains coding-agent ${name}.`);
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (variant !== "with_docs") throw new Error("Eval image has no valid PI_EVAL_VARIANT.");
|
||||
for (const path of [
|
||||
join(codingAgentDir, "README.md"),
|
||||
join(codingAgentDir, "CHANGELOG.md"),
|
||||
join(codingAgentDir, "docs/models.md"),
|
||||
join(codingAgentDir, "examples/README.md"),
|
||||
]) {
|
||||
if (!existsSync(path) || readFileSync(path).length === 0) throw new Error(`Missing documentation: ${path}`);
|
||||
}
|
||||
}
|
||||
|
||||
function chownTree(path: string, uid: number, gid: number): void {
|
||||
const stats = lstatSync(path);
|
||||
if (stats.isDirectory() && !stats.isSymbolicLink()) {
|
||||
for (const entry of readdirSync(path)) chownTree(join(path, entry), uid, gid);
|
||||
}
|
||||
chownSync(path, uid, gid);
|
||||
}
|
||||
|
||||
function assertRootOnly(path: string): void {
|
||||
const stats = statSync(path);
|
||||
if (stats.uid !== 0 || (stats.mode & 0o077) !== 0) {
|
||||
throw new Error(`Evaluator source must be readable only by root: ${path}`);
|
||||
}
|
||||
}
|
||||
|
||||
function assertSandboxCannotRead(path: string): void {
|
||||
const method = statSync(path).isDirectory() ? "readdirSync" : "readFileSync";
|
||||
const probe = spawnSync(process.execPath, ["--eval", `require("node:fs").${method}(${JSON.stringify(path)})`], {
|
||||
uid: sandboxUid,
|
||||
gid: sandboxGid,
|
||||
stdio: "ignore",
|
||||
});
|
||||
if (probe.status === 0) throw new Error(`The model-facing user can read evaluator source: ${path}`);
|
||||
}
|
||||
|
||||
assertWorkspace();
|
||||
const evalRoot = "/repo/packages/evals/evals";
|
||||
for (const path of [
|
||||
evalRoot,
|
||||
"/repo/packages/evals/src",
|
||||
"/repo/packages/evals/docker",
|
||||
"/repo/packages/evals/vitest.evals.config.ts",
|
||||
"/repo/vitest.base.ts",
|
||||
]) {
|
||||
assertRootOnly(path);
|
||||
assertSandboxCannotRead(path);
|
||||
}
|
||||
const codingAgentEntry = fileURLToPath(import.meta.resolve("@earendil-works/pi-coding-agent"));
|
||||
if (codingAgentEntry !== "/repo/node_modules/@earendil-works/pi-coding-agent/dist/index.js") {
|
||||
throw new Error(`Eval does not resolve pi-coding-agent from dist: ${codingAgentEntry}`);
|
||||
}
|
||||
|
||||
const agentDir = "/tmp/pi-eval-host-agent";
|
||||
mkdirSync(agentDir, { recursive: true });
|
||||
const authSource = "/run/pi-eval-secrets/auth.json";
|
||||
if (existsSync(authSource)) copyFileSync(authSource, join(agentDir, "auth.json"));
|
||||
chownTree(agentDir, sandboxUid, sandboxGid);
|
||||
chownTree("/artifacts", sandboxUid, sandboxGid);
|
||||
process.env.HOME = "/tmp/pi-eval-bootstrap";
|
||||
process.env.USERPROFILE = process.env.HOME;
|
||||
process.env.PI_CODING_AGENT_DIR = agentDir;
|
||||
process.env.PI_EVAL_CONTAINER = "1";
|
||||
process.umask(0o022);
|
||||
|
||||
const require = createRequire(import.meta.url);
|
||||
const vitestCli = resolve(dirname(require.resolve("vitest/package.json")), "vitest.mjs");
|
||||
const inputArgs = process.argv.slice(2);
|
||||
const discover = inputArgs.includes("--discover");
|
||||
const vitestArgs = inputArgs.filter((argument) => argument !== "--discover");
|
||||
const evalFiles = vitestArgs.filter((argument) => argument.endsWith(".eval.ts"));
|
||||
if (evalFiles.length === 0) throw new Error("The container runner requires an explicit eval file.");
|
||||
|
||||
for (const evalFile of evalFiles) {
|
||||
assertRootOnly(evalFile);
|
||||
assertSandboxCannotRead(evalFile);
|
||||
}
|
||||
|
||||
const commandArgs = discover
|
||||
? [vitestCli, "list", "--config", "vitest.evals.config.ts", ...vitestArgs, "--json=/artifacts/discovered-tests.json"]
|
||||
: [
|
||||
vitestCli,
|
||||
"run",
|
||||
"--config",
|
||||
"vitest.evals.config.ts",
|
||||
...vitestArgs,
|
||||
"--reporter=vitest-evals/reporter",
|
||||
"--reporter=json",
|
||||
"--outputFile=/artifacts/vitest.json",
|
||||
];
|
||||
const result = spawnSync(process.execPath, commandArgs, { cwd: packageRoot, stdio: "inherit", env: process.env });
|
||||
if (result.error) throw result.error;
|
||||
|
||||
const artifactUid = process.env.PI_EVAL_ARTIFACT_UID;
|
||||
const artifactGid = process.env.PI_EVAL_ARTIFACT_GID;
|
||||
if (artifactUid !== undefined && artifactGid !== undefined) {
|
||||
chownTree("/artifacts", parseId("PI_EVAL_ARTIFACT_UID"), parseId("PI_EVAL_ARTIFACT_GID"));
|
||||
}
|
||||
process.exit(result.status ?? 1);
|
||||
70
third_party/pi/packages/evals/docker/install-runtime.mjs
vendored
Normal file
70
third_party/pi/packages/evals/docker/install-runtime.mjs
vendored
Normal file
@@ -0,0 +1,70 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { cpSync, existsSync, mkdirSync, readFileSync, readdirSync, rmSync } from "node:fs";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { installConsumer } from "../../../scripts/local-package-install.mjs";
|
||||
import { produceArtifactSet } from "../../../scripts/package-artifacts.mjs";
|
||||
import { codingAgentName } from "../../../scripts/coding-agent-smoke.mjs";
|
||||
|
||||
const repositoryRoot = resolve(dirname(fileURLToPath(import.meta.url)), "../../..");
|
||||
const outputDirectory = process.argv[2];
|
||||
if (!outputDirectory || process.argv.length !== 3) {
|
||||
throw new Error("Usage: node packages/evals/docker/install-runtime.mjs <output-directory>");
|
||||
}
|
||||
|
||||
const evalPackage = JSON.parse(readFileSync(join(repositoryRoot, "packages/evals/package.json"), "utf8"));
|
||||
const artifactSet = produceArtifactSet({
|
||||
build: false,
|
||||
outDir: join(outputDirectory, "artifacts"),
|
||||
repoRoot: repositoryRoot,
|
||||
// Docker build contexts may omit Git metadata; this artifact set is consumed only within this image build.
|
||||
source: null,
|
||||
});
|
||||
const artifactNames = new Set(artifactSet.packages.map((pkg) => pkg.name));
|
||||
const evaluatorDependencies = Object.entries(evalPackage.devDependencies).filter(([name]) => !artifactNames.has(name));
|
||||
const installDirectory = join(outputDirectory, "install");
|
||||
installConsumer({ artifactSet, directory: installDirectory, packageNames: [codingAgentName] });
|
||||
execFileSync(
|
||||
"npm",
|
||||
[
|
||||
"install",
|
||||
"--ignore-scripts",
|
||||
"--omit=dev",
|
||||
"--no-audit",
|
||||
"--no-fund",
|
||||
"--no-save",
|
||||
...evaluatorDependencies.map(([name, version]) => `${name}@${version}`),
|
||||
],
|
||||
{ cwd: installDirectory, stdio: "inherit" },
|
||||
);
|
||||
|
||||
for (const packageName of artifactNames) {
|
||||
if (packageName === codingAgentName) continue;
|
||||
const packageDirectory = join(installDirectory, "node_modules", ...packageName.split("/"));
|
||||
if (!existsSync(packageDirectory)) continue;
|
||||
for (const entry of readdirSync(packageDirectory, { withFileTypes: true })) {
|
||||
if (
|
||||
(entry.isDirectory() && ["docs", "examples", "src", "test", "tests"].includes(entry.name)) ||
|
||||
(entry.isFile() && /^(?:readme|changelog)(?:\..+)?$/i.test(entry.name))
|
||||
) {
|
||||
rmSync(join(packageDirectory, entry.name), { force: true, recursive: true });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const runtimeRoot = join(outputDirectory, "root");
|
||||
const runtimeEvalRoot = join(runtimeRoot, "packages/evals");
|
||||
mkdirSync(join(runtimeEvalRoot, "docker"), { recursive: true });
|
||||
for (const file of ["package.json", "vitest.base.ts"]) {
|
||||
cpSync(join(repositoryRoot, file), join(runtimeRoot, file));
|
||||
}
|
||||
for (const file of ["package.json", "vitest.evals.config.ts"]) {
|
||||
cpSync(join(repositoryRoot, "packages/evals", file), join(runtimeEvalRoot, file));
|
||||
}
|
||||
for (const directory of ["src", "evals"]) {
|
||||
cpSync(join(repositoryRoot, "packages/evals", directory), join(runtimeEvalRoot, directory), { recursive: true });
|
||||
}
|
||||
cpSync(
|
||||
join(repositoryRoot, "packages/evals/docker/entrypoint.ts"),
|
||||
join(runtimeEvalRoot, "docker/entrypoint.ts"),
|
||||
);
|
||||
154
third_party/pi/packages/evals/evals/acme-server.ts
vendored
Normal file
154
third_party/pi/packages/evals/evals/acme-server.ts
vendored
Normal file
@@ -0,0 +1,154 @@
|
||||
import { createServer, type IncomingMessage, type Server, type ServerResponse } from "node:http";
|
||||
|
||||
export const OPENAI_PROVIDER_ID = "acme";
|
||||
export const OPENAI_MODEL_ID = "acme-chat";
|
||||
export const OPENAI_PROBE_PROMPT = "Reply with ACME_OK.";
|
||||
export const OPENAI_PROBE_RESPONSE = "ACME_OK";
|
||||
export const STREAM_PROVIDER_ID = "acme-stream";
|
||||
export const STREAM_MODEL_ID = "acme-stream-chat";
|
||||
export const STREAM_PROBE_PROMPT = "Reply with ACME_STREAM_OK.";
|
||||
export const STREAM_PROBE_RESPONSE = "ACME_STREAM_OK";
|
||||
|
||||
export const STREAM_API_DOCUMENTATION = `${JSON.stringify(
|
||||
{
|
||||
name: "Acme Streaming API",
|
||||
request: {
|
||||
method: "POST",
|
||||
path: "/generate",
|
||||
headers: { "content-type": "application/json", "x-acme-key": "resolved credential" },
|
||||
body: { model: STREAM_MODEL_ID, messages: [{ role: "user", content: "Hello" }], stream: true },
|
||||
},
|
||||
response: {
|
||||
contentType: "application/x-ndjson",
|
||||
events: [
|
||||
{ type: "text_delta", text: "Hello" },
|
||||
{ type: "usage", input_tokens: 3, output_tokens: 2 },
|
||||
{ type: "done", reason: "stop" },
|
||||
],
|
||||
},
|
||||
},
|
||||
null,
|
||||
2,
|
||||
)}\n`;
|
||||
|
||||
type ServerMode = "openai" | "stream";
|
||||
|
||||
function isRecord(value: unknown): value is Record<string, unknown> {
|
||||
return typeof value === "object" && value !== null && !Array.isArray(value);
|
||||
}
|
||||
|
||||
function rejectRequest(response: ServerResponse, status: number, message: string): void {
|
||||
response.writeHead(status, { "content-type": "text/plain" });
|
||||
response.end(message);
|
||||
}
|
||||
|
||||
async function readJson(request: IncomingMessage): Promise<unknown> {
|
||||
let body = "";
|
||||
for await (const chunk of request) body += chunk.toString();
|
||||
return JSON.parse(body);
|
||||
}
|
||||
|
||||
export type AcmeServer = {
|
||||
start(): Promise<void>;
|
||||
stop(): Promise<void>;
|
||||
reset(): void;
|
||||
origin(): string;
|
||||
baseUrl(): string;
|
||||
validRequestReceived(): boolean;
|
||||
};
|
||||
|
||||
export function createAcmeServer(mode: ServerMode): AcmeServer {
|
||||
let server: Server | undefined;
|
||||
let serverOrigin = "";
|
||||
let receivedValidRequest = false;
|
||||
|
||||
async function handle(request: IncomingMessage, response: ServerResponse): Promise<void> {
|
||||
const expectedPath = mode === "openai" ? "/v1/chat/completions" : "/generate";
|
||||
if (request.url !== expectedPath) return rejectRequest(response, 404, "Unknown endpoint");
|
||||
if (request.method !== "POST") return rejectRequest(response, 405, "Expected POST");
|
||||
if (!request.headers["content-type"]?.startsWith("application/json")) {
|
||||
return rejectRequest(response, 415, "Expected application/json");
|
||||
}
|
||||
let payload: unknown;
|
||||
try {
|
||||
payload = await readJson(request);
|
||||
} catch {
|
||||
return rejectRequest(response, 400, "Invalid JSON");
|
||||
}
|
||||
if (!isRecord(payload)) return rejectRequest(response, 422, "Expected a JSON object");
|
||||
const messages = Array.isArray(payload.messages) ? payload.messages : [];
|
||||
const userMessage = messages.find(
|
||||
(message): message is Record<string, unknown> => isRecord(message) && message.role === "user",
|
||||
);
|
||||
const prompt = typeof userMessage?.content === "string" ? userMessage.content : undefined;
|
||||
|
||||
if (mode === "stream") {
|
||||
if (request.headers["x-acme-key"] !== "resolved-stream-key") {
|
||||
return rejectRequest(response, 401, "Invalid Acme Stream credential");
|
||||
}
|
||||
if (payload.model !== STREAM_MODEL_ID || prompt === undefined || payload.stream !== true) {
|
||||
return rejectRequest(response, 422, "Invalid Acme Stream request");
|
||||
}
|
||||
receivedValidRequest = prompt === STREAM_PROBE_PROMPT;
|
||||
response.writeHead(200, { "content-type": "application/x-ndjson" });
|
||||
response.write(`${JSON.stringify({ type: "text_delta", text: "ACME_" })}\n`);
|
||||
response.write(`${JSON.stringify({ type: "text_delta", text: "STREAM_OK" })}\n`);
|
||||
response.write(`${JSON.stringify({ type: "usage", input_tokens: 4, output_tokens: 3 })}\n`);
|
||||
response.end(`${JSON.stringify({ type: "done", reason: "stop" })}\n`);
|
||||
return;
|
||||
}
|
||||
|
||||
if (request.headers.authorization !== "Bearer resolved-acme-key") {
|
||||
return rejectRequest(response, 401, "Invalid Acme credential");
|
||||
}
|
||||
if (payload.model !== OPENAI_MODEL_ID || prompt === undefined || payload.stream !== true) {
|
||||
return rejectRequest(response, 422, "Invalid OpenAI-compatible request");
|
||||
}
|
||||
receivedValidRequest = prompt === OPENAI_PROBE_PROMPT;
|
||||
response.writeHead(200, { "content-type": "text/event-stream", "cache-control": "no-cache" });
|
||||
response.write(
|
||||
`data: ${JSON.stringify({
|
||||
id: "chatcmpl-acme",
|
||||
object: "chat.completion.chunk",
|
||||
created: 0,
|
||||
model: OPENAI_MODEL_ID,
|
||||
choices: [{ index: 0, delta: { role: "assistant", content: OPENAI_PROBE_RESPONSE }, finish_reason: null }],
|
||||
})}\n\n`,
|
||||
);
|
||||
response.write(
|
||||
`data: ${JSON.stringify({
|
||||
id: "chatcmpl-acme",
|
||||
object: "chat.completion.chunk",
|
||||
created: 0,
|
||||
model: OPENAI_MODEL_ID,
|
||||
choices: [{ index: 0, delta: {}, finish_reason: "stop" }],
|
||||
usage: { prompt_tokens: 3, completion_tokens: 2 },
|
||||
})}\n\n`,
|
||||
);
|
||||
response.end("data: [DONE]\n\n");
|
||||
}
|
||||
|
||||
return {
|
||||
async start() {
|
||||
server = createServer((request, response) => void handle(request, response));
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
server?.once("error", reject);
|
||||
server?.listen(0, "127.0.0.1", resolve);
|
||||
});
|
||||
const address = server.address();
|
||||
if (!address || typeof address === "string") throw new Error("Acme fixture server did not bind a TCP port.");
|
||||
serverOrigin = `http://127.0.0.1:${address.port}`;
|
||||
},
|
||||
async stop() {
|
||||
if (!server) return;
|
||||
await new Promise<void>((resolve, reject) => server?.close((error) => (error ? reject(error) : resolve())));
|
||||
server = undefined;
|
||||
},
|
||||
reset() {
|
||||
receivedValidRequest = false;
|
||||
},
|
||||
origin: () => serverOrigin,
|
||||
baseUrl: () => `${serverOrigin}/v1`,
|
||||
validRequestReceived: () => receivedValidRequest,
|
||||
};
|
||||
}
|
||||
120
third_party/pi/packages/evals/evals/configured-runtime.ts
vendored
Normal file
120
third_party/pi/packages/evals/evals/configured-runtime.ts
vendored
Normal file
@@ -0,0 +1,120 @@
|
||||
import { join } from "node:path";
|
||||
import { type Api, type Context, contentText, type Model, type ModelsSimpleStreamOptions } from "@earendil-works/pi-ai";
|
||||
import { ModelRuntime } from "@earendil-works/pi-coding-agent";
|
||||
|
||||
type ModelFields = {
|
||||
id: string;
|
||||
name: string;
|
||||
provider: string;
|
||||
reasoning: boolean;
|
||||
input: Array<"text" | "image">;
|
||||
cost: { input: number; output: number; cacheRead: number; cacheWrite: number };
|
||||
contextWindow: number;
|
||||
maxTokens: number;
|
||||
};
|
||||
|
||||
export type ProviderRuntimeOutput = {
|
||||
result:
|
||||
| {
|
||||
validRequestReceived: boolean;
|
||||
model: ModelFields;
|
||||
response: { text: string; stopReason: string; inputTokens: number; outputTokens: number };
|
||||
}
|
||||
| { error: string };
|
||||
};
|
||||
|
||||
export type AddedModelOutput = {
|
||||
result: { model: ModelFields; existingModelsPreserved: boolean } | { error: string };
|
||||
};
|
||||
|
||||
export type ProviderScenario = {
|
||||
providerId: string;
|
||||
modelId: string;
|
||||
createContext(): Context;
|
||||
options?: ModelsSimpleStreamOptions;
|
||||
validRequestReceived(): boolean;
|
||||
};
|
||||
|
||||
function modelFields(model: Model<Api>): ModelFields {
|
||||
return {
|
||||
id: model.id,
|
||||
name: model.name,
|
||||
provider: model.provider,
|
||||
reasoning: model.reasoning,
|
||||
input: [...model.input],
|
||||
cost: {
|
||||
input: model.cost.input,
|
||||
output: model.cost.output,
|
||||
cacheRead: model.cost.cacheRead,
|
||||
cacheWrite: model.cost.cacheWrite,
|
||||
},
|
||||
contextWindow: model.contextWindow,
|
||||
maxTokens: model.maxTokens,
|
||||
};
|
||||
}
|
||||
|
||||
export function loadConfiguredModelRuntime(agentDir: string): Promise<ModelRuntime> {
|
||||
return ModelRuntime.create({
|
||||
modelsPath: join(agentDir, "models.json"),
|
||||
authPath: join(agentDir, "auth.json"),
|
||||
modelsStorePath: join(agentDir, "models-store.json"),
|
||||
allowModelNetwork: false,
|
||||
});
|
||||
}
|
||||
|
||||
export async function inspectProvider(
|
||||
runtime: ModelRuntime,
|
||||
scenario: ProviderScenario,
|
||||
): Promise<ProviderRuntimeOutput> {
|
||||
await runtime.refresh({ allowNetwork: false });
|
||||
const configurationError = runtime.getError();
|
||||
if (configurationError) return { result: { error: configurationError } };
|
||||
const model = runtime.getModel(scenario.providerId, scenario.modelId);
|
||||
if (!model) {
|
||||
return { result: { error: `Model ${scenario.providerId}/${scenario.modelId} is unavailable after reload.` } };
|
||||
}
|
||||
try {
|
||||
const response = await runtime.completeSimple(model, scenario.createContext(), scenario.options);
|
||||
return {
|
||||
result: {
|
||||
validRequestReceived: scenario.validRequestReceived(),
|
||||
model: modelFields(model),
|
||||
response: {
|
||||
text: contentText(response.content),
|
||||
stopReason: response.stopReason,
|
||||
inputTokens: response.usage.input,
|
||||
outputTokens: response.usage.output,
|
||||
},
|
||||
},
|
||||
};
|
||||
} catch (error) {
|
||||
return { result: { error: error instanceof Error ? error.message : String(error) } };
|
||||
}
|
||||
}
|
||||
|
||||
export async function inspectAddedModel(
|
||||
runtime: ModelRuntime,
|
||||
providerId: string,
|
||||
modelId: string,
|
||||
): Promise<AddedModelOutput> {
|
||||
const pristineRuntime = await ModelRuntime.create({ modelsPath: null, allowModelNetwork: false });
|
||||
const pristineProvider = pristineRuntime.getProvider(providerId);
|
||||
if (!pristineProvider) return { result: { error: `Built-in provider ${providerId} is unavailable.` } };
|
||||
const existingModelIds = pristineProvider.getModels().map(({ id }) => id);
|
||||
if (existingModelIds.length === 0) {
|
||||
return { result: { error: `Built-in provider ${providerId} has no models.` } };
|
||||
}
|
||||
await runtime.refresh({ allowNetwork: false });
|
||||
const configurationError = runtime.getError();
|
||||
if (configurationError) return { result: { error: configurationError } };
|
||||
const model = runtime.getModel(providerId, modelId);
|
||||
if (!model) {
|
||||
return { result: { error: `Model ${providerId}/${modelId} is unavailable after reload.` } };
|
||||
}
|
||||
return {
|
||||
result: {
|
||||
model: modelFields(model),
|
||||
existingModelsPreserved: existingModelIds.every((id) => runtime.getModel(providerId, id) !== undefined),
|
||||
},
|
||||
};
|
||||
}
|
||||
72
third_party/pi/packages/evals/evals/custom-provider.docs.eval.ts
vendored
Normal file
72
third_party/pi/packages/evals/evals/custom-provider.docs.eval.ts
vendored
Normal file
@@ -0,0 +1,72 @@
|
||||
import { afterAll, beforeAll, beforeEach } from "vitest";
|
||||
import { describeEval, StructuredOutputJudge } from "vitest-evals";
|
||||
import {
|
||||
createAcmeServer,
|
||||
STREAM_API_DOCUMENTATION,
|
||||
STREAM_MODEL_ID,
|
||||
STREAM_PROBE_PROMPT,
|
||||
STREAM_PROBE_RESPONSE,
|
||||
STREAM_PROVIDER_ID,
|
||||
} from "./acme-server.ts";
|
||||
import { inspectProvider, type ProviderRuntimeOutput } from "./configured-runtime.ts";
|
||||
import { createPiDocumentationEvalHarness } from "../src/harness.ts";
|
||||
|
||||
const DOCUMENTATION_PATH = "fixtures/acme-stream-api.json";
|
||||
const STREAM_API_KEY = "resolved-stream-key";
|
||||
const server = createAcmeServer("stream");
|
||||
|
||||
beforeAll(() => {
|
||||
process.env.ACME_STREAM_API_KEY = STREAM_API_KEY;
|
||||
return server.start();
|
||||
});
|
||||
beforeEach(() => server.reset());
|
||||
afterAll(() => {
|
||||
delete process.env.ACME_STREAM_API_KEY;
|
||||
return server.stop();
|
||||
});
|
||||
|
||||
const expected: ProviderRuntimeOutput = {
|
||||
result: {
|
||||
validRequestReceived: true,
|
||||
model: {
|
||||
id: STREAM_MODEL_ID,
|
||||
name: "Acme Stream Chat",
|
||||
provider: STREAM_PROVIDER_ID,
|
||||
reasoning: false,
|
||||
input: ["text"],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 16384,
|
||||
maxTokens: 2048,
|
||||
},
|
||||
response: { text: STREAM_PROBE_RESPONSE, stopReason: "stop", inputTokens: 4, outputTokens: 3 },
|
||||
},
|
||||
};
|
||||
|
||||
const harness = createPiDocumentationEvalHarness({
|
||||
workspaceFiles: { [DOCUMENTATION_PATH]: STREAM_API_DOCUMENTATION },
|
||||
output: ({ session }) =>
|
||||
inspectProvider(session.modelRuntime, {
|
||||
providerId: STREAM_PROVIDER_ID,
|
||||
modelId: STREAM_MODEL_ID,
|
||||
createContext: () => ({
|
||||
messages: [{ role: "user", content: STREAM_PROBE_PROMPT, timestamp: 0 }],
|
||||
}),
|
||||
options: { env: { ACME_STREAM_API_KEY: STREAM_API_KEY }, maxTokens: 32 },
|
||||
validRequestReceived: server.validRequestReceived,
|
||||
}),
|
||||
});
|
||||
const judge = StructuredOutputJudge({ expected, match: "strict", allowExtras: false });
|
||||
|
||||
describeEval("Add custom streaming provider", { harness, judges: [judge], judgeThreshold: null }, (it) => {
|
||||
it("implements a provider from an API fixture", async ({ run }) => {
|
||||
await run([
|
||||
{
|
||||
type: "prompt",
|
||||
content: `Configure this running Pi installation with Acme Stream as a provider. Do not create a project package or modify project source. Its provider ID is ${STREAM_PROVIDER_ID}, its API is at ${server.origin()}, and its API documentation is in ./${DOCUMENTATION_PATH}. Read its credential from the ACME_STREAM_API_KEY environment variable.
|
||||
|
||||
It offers one model, ${STREAM_MODEL_ID}, shown as “Acme Stream Chat”. The model accepts text, does not support reasoning, has a 16,384-token context window and a 2,048-token maximum output, and has no usage cost.`,
|
||||
},
|
||||
{ type: "reload" },
|
||||
]);
|
||||
});
|
||||
});
|
||||
66
third_party/pi/packages/evals/evals/documentation-audit.eval.ts
vendored
Normal file
66
third_party/pi/packages/evals/evals/documentation-audit.eval.ts
vendored
Normal file
@@ -0,0 +1,66 @@
|
||||
import { globSync } from "node:fs";
|
||||
import { resolve } from "node:path";
|
||||
import { Type } from "@earendil-works/pi-ai";
|
||||
import { defineTool } from "@earendil-works/pi-coding-agent";
|
||||
import { expect } from "vitest";
|
||||
import { describeEval, toolCalls } from "vitest-evals";
|
||||
import { createPiCodingAgentHarness } from "../src/harness.ts";
|
||||
|
||||
const TOOL_NAME = "submit_documentation_audit";
|
||||
const submitAudit = defineTool({
|
||||
name: TOOL_NAME,
|
||||
label: "Submit documentation audit",
|
||||
description: "Submit the final verdict after completing the documentation investigation.",
|
||||
promptSnippet: "Submit the final documentation audit as validated structured data",
|
||||
parameters: Type.Object(
|
||||
{
|
||||
verdict: Type.Union([Type.Literal("match"), Type.Literal("mismatch"), Type.Literal("inconclusive")]),
|
||||
explanation: Type.String({ minLength: 1, maxLength: 2000 }),
|
||||
},
|
||||
{ additionalProperties: false },
|
||||
),
|
||||
constrainedSampling: { type: "json_schema", strict: "prefer" },
|
||||
async execute(_toolCallId, params) {
|
||||
return {
|
||||
content: [{ type: "text", text: "Documentation audit submitted." }],
|
||||
details: params,
|
||||
terminate: true,
|
||||
};
|
||||
},
|
||||
});
|
||||
|
||||
const repositoryRoot = resolve(import.meta.dirname, "../../..");
|
||||
const docsRoot = resolve(repositoryRoot, "packages/coding-agent/docs");
|
||||
const pages = globSync("**/*.md", { cwd: docsRoot })
|
||||
.map((path) => ({ path: path.replaceAll("\\", "/") }))
|
||||
.sort((left, right) => left.path.localeCompare(right.path));
|
||||
const harness = createPiCodingAgentHarness({
|
||||
name: "documentation-page-audit",
|
||||
tools: ["read", "grep", "find", "ls", TOOL_NAME],
|
||||
customTools: [submitAudit],
|
||||
});
|
||||
|
||||
describeEval("Audit documentation against implementation", { harness }, (it) => {
|
||||
it.for(pages)("$path matches the implementation", async ({ path }, { run }) => {
|
||||
const documentationPath = resolve(docsRoot, path);
|
||||
const result = await run(`Audit this Pi documentation page against the repository implementation.
|
||||
|
||||
Documentation page: ${documentationPath}
|
||||
Repository root: ${repositoryRoot}
|
||||
|
||||
Read the complete page and the relevant implementation. Report a mismatch only for a clear, user-visible contradiction between an explicit documentation claim and actual runtime behavior. Follow the runtime path; names, comments, types, isolated helpers, and tests are not sufficient evidence by themselves.
|
||||
|
||||
Missing internal detail, ambiguous wording, hypothetical misuse, and undocumented edge cases are not mismatches. If the evidence is not decisive, report inconclusive. Otherwise report match.
|
||||
|
||||
For a mismatch, quote the claim, cite the implementation path and symbol, and state the concrete behavior a user would observe.
|
||||
|
||||
Call ${TOOL_NAME} exactly once as your final action. Do not return prose.`);
|
||||
|
||||
const auditCalls = toolCalls(result).filter((call) => call.name === TOOL_NAME);
|
||||
expect(auditCalls).toHaveLength(1);
|
||||
const auditCall = auditCalls[0];
|
||||
expect(auditCall?.status).toBe("ok");
|
||||
const explanation = auditCall?.arguments?.explanation;
|
||||
expect(auditCall?.arguments?.verdict, typeof explanation === "string" ? explanation : undefined).toBe("match");
|
||||
});
|
||||
});
|
||||
60
third_party/pi/packages/evals/evals/extensions.docs.eval.ts
vendored
Normal file
60
third_party/pi/packages/evals/evals/extensions.docs.eval.ts
vendored
Normal file
@@ -0,0 +1,60 @@
|
||||
import { contentText } from "@earendil-works/pi-ai";
|
||||
import { describeEval, StructuredOutputJudge, ToolCallJudge } from "vitest-evals";
|
||||
import { createPiDocumentationEvalHarness, DOCUMENTATION_EVAL_TOOLS } from "../src/harness.ts";
|
||||
|
||||
const TOOL_NAME = "hello";
|
||||
const TOOL_ARGUMENTS = { name: "Bob" };
|
||||
const TOOL_RESULT = "Hello, Bob!";
|
||||
|
||||
const harness = createPiDocumentationEvalHarness({
|
||||
tools: [...DOCUMENTATION_EVAL_TOOLS, TOOL_NAME],
|
||||
output: ({ response, session }) => {
|
||||
const extensions = session.resourceLoader.getExtensions();
|
||||
const result = [...session.messages]
|
||||
.reverse()
|
||||
.find((message) => message.role === "toolResult" && message.toolName === TOOL_NAME && !message.isError);
|
||||
return {
|
||||
response,
|
||||
extensionErrors: extensions.errors,
|
||||
toolLoaded: extensions.extensions.some(({ tools }) => tools.has(TOOL_NAME)),
|
||||
toolResult: result?.role === "toolResult" ? contentText(result.content) : null,
|
||||
};
|
||||
},
|
||||
});
|
||||
|
||||
describeEval(
|
||||
"Create and use a tool extension",
|
||||
{
|
||||
harness,
|
||||
judges: [
|
||||
StructuredOutputJudge({
|
||||
expected: {
|
||||
response: TOOL_RESULT,
|
||||
extensionErrors: [],
|
||||
toolLoaded: true,
|
||||
toolResult: TOOL_RESULT,
|
||||
},
|
||||
match: "strict",
|
||||
allowExtras: false,
|
||||
}),
|
||||
ToolCallJudge({ expectedTools: [{ name: TOOL_NAME, arguments: TOOL_ARGUMENTS }] }),
|
||||
],
|
||||
judgeThreshold: null,
|
||||
},
|
||||
(it) => {
|
||||
it("creates, reloads, and invokes the extension", async ({ run }) => {
|
||||
await run([
|
||||
{
|
||||
type: "prompt",
|
||||
content:
|
||||
"Configure this running Pi installation with an extension containing a hello tool that takes a name and returns a greeting. Do not create project source. For example, passing Bob should return `Hello, Bob!`.",
|
||||
},
|
||||
{ type: "reload" },
|
||||
{
|
||||
type: "prompt",
|
||||
content: "Use the hello tool to greet Bob. Respond with exactly the tool's greeting and nothing else.",
|
||||
},
|
||||
]);
|
||||
});
|
||||
},
|
||||
);
|
||||
40
third_party/pi/packages/evals/evals/models.docs.eval.ts
vendored
Normal file
40
third_party/pi/packages/evals/evals/models.docs.eval.ts
vendored
Normal file
@@ -0,0 +1,40 @@
|
||||
import { describeEval, StructuredOutputJudge } from "vitest-evals";
|
||||
import { inspectAddedModel, type AddedModelOutput } from "./configured-runtime.ts";
|
||||
import { createPiDocumentationEvalHarness } from "../src/harness.ts";
|
||||
|
||||
const PROVIDER_ID = "openai";
|
||||
const MODEL_ID = "fixture-chat";
|
||||
const MODEL_NAME = "Fixture Chat";
|
||||
|
||||
const expected: AddedModelOutput = {
|
||||
result: {
|
||||
model: {
|
||||
id: MODEL_ID,
|
||||
name: MODEL_NAME,
|
||||
provider: PROVIDER_ID,
|
||||
reasoning: true,
|
||||
input: ["text"],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 32768,
|
||||
maxTokens: 4096,
|
||||
},
|
||||
existingModelsPreserved: true,
|
||||
},
|
||||
};
|
||||
|
||||
const harness = createPiDocumentationEvalHarness({
|
||||
output: ({ session }) => inspectAddedModel(session.modelRuntime, PROVIDER_ID, MODEL_ID),
|
||||
});
|
||||
const judge = StructuredOutputJudge({ expected, match: "strict", allowExtras: false });
|
||||
|
||||
describeEval("Add model to existing provider", { harness, judges: [judge], judgeThreshold: null }, (it) => {
|
||||
it("adds the model without replacing existing models", async ({ run }) => {
|
||||
await run([
|
||||
{
|
||||
type: "prompt",
|
||||
content: `Configure this running Pi installation with a new \`${PROVIDER_ID}/${MODEL_ID}\` model. Do not create project-local configuration. Show it as “${MODEL_NAME}”. It accepts text, supports reasoning, has a 32,768-token context window and a 4,096-token maximum output, and has no usage cost.`,
|
||||
},
|
||||
{ type: "reload" },
|
||||
]);
|
||||
});
|
||||
});
|
||||
62
third_party/pi/packages/evals/evals/openai-provider.docs.eval.ts
vendored
Normal file
62
third_party/pi/packages/evals/evals/openai-provider.docs.eval.ts
vendored
Normal file
@@ -0,0 +1,62 @@
|
||||
import { afterAll, beforeAll, beforeEach } from "vitest";
|
||||
import { describeEval, StructuredOutputJudge } from "vitest-evals";
|
||||
import {
|
||||
createAcmeServer,
|
||||
OPENAI_MODEL_ID,
|
||||
OPENAI_PROBE_PROMPT,
|
||||
OPENAI_PROBE_RESPONSE,
|
||||
OPENAI_PROVIDER_ID,
|
||||
} from "./acme-server.ts";
|
||||
import { inspectProvider, type ProviderRuntimeOutput } from "./configured-runtime.ts";
|
||||
import { createPiDocumentationEvalHarness } from "../src/harness.ts";
|
||||
|
||||
const server = createAcmeServer("openai");
|
||||
|
||||
beforeAll(() => server.start());
|
||||
beforeEach(() => server.reset());
|
||||
afterAll(() => server.stop());
|
||||
|
||||
const expected: ProviderRuntimeOutput = {
|
||||
result: {
|
||||
validRequestReceived: true,
|
||||
model: {
|
||||
id: OPENAI_MODEL_ID,
|
||||
name: "Acme Chat",
|
||||
provider: OPENAI_PROVIDER_ID,
|
||||
reasoning: false,
|
||||
input: ["text"],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 32768,
|
||||
maxTokens: 4096,
|
||||
},
|
||||
response: { text: OPENAI_PROBE_RESPONSE, stopReason: "stop", inputTokens: 3, outputTokens: 2 },
|
||||
},
|
||||
};
|
||||
|
||||
const harness = createPiDocumentationEvalHarness({
|
||||
output: ({ session }) =>
|
||||
inspectProvider(session.modelRuntime, {
|
||||
providerId: OPENAI_PROVIDER_ID,
|
||||
modelId: OPENAI_MODEL_ID,
|
||||
createContext: () => ({
|
||||
messages: [{ role: "user", content: OPENAI_PROBE_PROMPT, timestamp: 0 }],
|
||||
}),
|
||||
options: { env: { ACME_API_KEY: "resolved-acme-key" }, maxTokens: 32 },
|
||||
validRequestReceived: server.validRequestReceived,
|
||||
}),
|
||||
});
|
||||
const judge = StructuredOutputJudge({ expected, match: "strict", allowExtras: false });
|
||||
|
||||
describeEval("Add OpenAI-compatible provider", { harness, judges: [judge], judgeThreshold: null }, (it) => {
|
||||
it("configures a provider that works through Pi", async ({ run }) => {
|
||||
await run([
|
||||
{
|
||||
type: "prompt",
|
||||
content: `Configure this running Pi installation with Acme as a provider. Do not create project-local configuration. Its provider ID is ${OPENAI_PROVIDER_ID}, its API is at ${server.baseUrl()}, and it uses OpenAI Chat Completions. Read its API key from the ACME_API_KEY environment variable.
|
||||
|
||||
The provider offers one model, ${OPENAI_MODEL_ID}, shown as “Acme Chat”. It accepts text, does not support reasoning, has a 32,768-token context window and a 4,096-token maximum output, and has no usage cost.`,
|
||||
},
|
||||
{ type: "reload" },
|
||||
]);
|
||||
});
|
||||
});
|
||||
18
third_party/pi/packages/evals/evals/smoke.eval.ts
vendored
Normal file
18
third_party/pi/packages/evals/evals/smoke.eval.ts
vendored
Normal file
@@ -0,0 +1,18 @@
|
||||
import { expect } from "vitest";
|
||||
import { describeEval } from "vitest-evals";
|
||||
import { createPiCodingAgentHarness } from "../src/harness.ts";
|
||||
|
||||
const harness = createPiCodingAgentHarness({ noTools: "all" });
|
||||
|
||||
describeEval("Answer a basic prompt", { harness }, (it) => {
|
||||
it("returns the expected answer", async ({ run }) => {
|
||||
const result = await run("What's the capital of France? Respond with only the city name.");
|
||||
expect(result.output.trim()).toBe("Paris");
|
||||
expect(result.errors).toEqual([]);
|
||||
expect(result.usage).toMatchObject({
|
||||
provider: process.env.PI_PROVIDER,
|
||||
model: process.env.PI_MODEL,
|
||||
});
|
||||
expect(result.usage.totalTokens).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
264
third_party/pi/packages/evals/evals/tui.docs.eval.ts
vendored
Normal file
264
third_party/pi/packages/evals/evals/tui.docs.eval.ts
vendored
Normal file
@@ -0,0 +1,264 @@
|
||||
import { setTimeout } from "node:timers/promises";
|
||||
import { stripVTControlCharacters } from "node:util";
|
||||
import type { AssistantMessage } from "@earendil-works/pi-ai";
|
||||
import {
|
||||
type AgentSession,
|
||||
AgentSessionRuntime,
|
||||
type AgentSessionServices,
|
||||
InteractiveMode,
|
||||
SessionManager,
|
||||
createAgentSessionFromServices,
|
||||
} from "@earendil-works/pi-coding-agent";
|
||||
import { Levenshtein } from "autoevals";
|
||||
import { createJudge, describeEval } from "vitest-evals";
|
||||
import { createPiDocumentationEvalHarness, type PiCodingAgentInput } from "../src/harness.ts";
|
||||
|
||||
const CONTEXT_WINDOW = 272_000;
|
||||
const BUILT_IN_CONTEXT_PATTERN = /\d+(?:\.\d+)?%\/272k(?: \(auto\))?/;
|
||||
const CONTEXT_FIXTURES = [
|
||||
{ percent: 42.2, expectedBar: "████░░░░░░ 42%" },
|
||||
{ percent: 65, expectedBar: "███████░░░ 65%" },
|
||||
{ percent: 120, expectedBar: "██████████ 100%" },
|
||||
] as const;
|
||||
|
||||
class RecordingTerminal {
|
||||
readonly columns = 100;
|
||||
readonly rows = 30;
|
||||
readonly kittyProtocolActive = true;
|
||||
private writes: string[] = [];
|
||||
private resizeHandler?: () => void;
|
||||
|
||||
start(_onInput: (data: string) => void, onResize: () => void): void {
|
||||
this.resizeHandler = onResize;
|
||||
}
|
||||
|
||||
stop(): void {
|
||||
this.resizeHandler = undefined;
|
||||
}
|
||||
|
||||
async drainInput(_maxMs?: number, _idleMs?: number): Promise<void> {}
|
||||
|
||||
write(data: string): void {
|
||||
this.writes.push(data);
|
||||
}
|
||||
|
||||
moveBy(_lines: number): void {}
|
||||
hideCursor(): void {}
|
||||
showCursor(): void {}
|
||||
clearLine(): void {}
|
||||
clearFromCursor(): void {}
|
||||
clearScreen(): void {}
|
||||
setTitle(_title: string): void {}
|
||||
setProgress(_active: boolean): void {}
|
||||
|
||||
requestResize(): void {
|
||||
this.resizeHandler?.();
|
||||
}
|
||||
|
||||
async takeRenders(): Promise<string[]> {
|
||||
await setTimeout(30);
|
||||
const output = this.writes.join("");
|
||||
this.writes = [];
|
||||
const startMarker = "\x1b[?2026h";
|
||||
const endMarker = "\x1b[?2026l";
|
||||
const renders: string[] = [];
|
||||
let offset = 0;
|
||||
while (true) {
|
||||
const start = output.indexOf(startMarker, offset);
|
||||
if (start === -1) return renders;
|
||||
const contentStart = start + startMarker.length;
|
||||
const end = output.indexOf(endMarker, contentStart);
|
||||
if (end === -1) throw new Error("TUI emitted an incomplete synchronized render.");
|
||||
renders.push(stripVTControlCharacters(output.slice(contentStart, end)));
|
||||
offset = end + endMarker.length;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type TuiFooterOutput = {
|
||||
builtInStatusLine: string | null;
|
||||
observations: Array<{
|
||||
percent: number;
|
||||
renderedOutput: string;
|
||||
contextUsage: { tokens: number | null; contextWindow: number; percent: number | null } | null;
|
||||
}>;
|
||||
extensionErrors: string[];
|
||||
};
|
||||
|
||||
function lines(output: string): string[] {
|
||||
return output.split(/[\r\n]+/).filter((line) => line.trim().length > 0);
|
||||
}
|
||||
|
||||
function createContextMessage(model: NonNullable<AgentSession["model"]>, percent: number): AssistantMessage {
|
||||
const tokens = Math.round((model.contextWindow * percent) / 100);
|
||||
return {
|
||||
role: "assistant",
|
||||
content: [{ type: "text", text: "Context usage fixture" }],
|
||||
api: model.api,
|
||||
provider: model.provider,
|
||||
model: model.id,
|
||||
usage: {
|
||||
input: tokens,
|
||||
output: 0,
|
||||
cacheRead: 0,
|
||||
cacheWrite: 0,
|
||||
totalTokens: tokens,
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 },
|
||||
},
|
||||
stopReason: "stop",
|
||||
timestamp: Date.now(),
|
||||
};
|
||||
}
|
||||
|
||||
async function inspectContextFooter(session: AgentSession, agentDir: string): Promise<TuiFooterOutput> {
|
||||
const activeModel = session.model;
|
||||
if (!activeModel) throw new Error("TUI footer oracle requires an active model.");
|
||||
const model = { ...activeModel, contextWindow: CONTEXT_WINDOW };
|
||||
const services: AgentSessionServices = {
|
||||
cwd: session.sessionManager.getCwd(),
|
||||
agentDir,
|
||||
modelRuntime: session.modelRuntime,
|
||||
settingsManager: session.settingsManager,
|
||||
resourceLoader: session.resourceLoader,
|
||||
diagnostics: [],
|
||||
};
|
||||
const oracleSession = (
|
||||
await createAgentSessionFromServices({
|
||||
services,
|
||||
sessionManager: SessionManager.inMemory(services.cwd),
|
||||
model,
|
||||
thinkingLevel: "high",
|
||||
noTools: "all",
|
||||
})
|
||||
).session;
|
||||
const runtime = new AgentSessionRuntime(oracleSession, services, async () => {
|
||||
throw new Error("The TUI footer oracle does not replace sessions.");
|
||||
});
|
||||
const terminal = new RecordingTerminal();
|
||||
const mode = new InteractiveMode(runtime, { tuiMode: "regular", terminal });
|
||||
const previousOffline = process.env.PI_OFFLINE;
|
||||
process.env.PI_OFFLINE = "1";
|
||||
try {
|
||||
oracleSession.sessionManager.appendMessage({
|
||||
...createContextMessage(model, 0),
|
||||
usage: {
|
||||
input: 106_000,
|
||||
output: 5_800,
|
||||
cacheRead: 606_000,
|
||||
cacheWrite: 0,
|
||||
totalTokens: 717_800,
|
||||
cost: { input: 1.008, output: 0, cacheRead: 0, cacheWrite: 0, total: 1.008 },
|
||||
},
|
||||
});
|
||||
let builtInStatusLine: string | null = null;
|
||||
const observations: TuiFooterOutput["observations"] = [];
|
||||
for (const [index, { percent }] of CONTEXT_FIXTURES.entries()) {
|
||||
oracleSession.agent.state.messages = [createContextMessage(model, percent)];
|
||||
if (index === 0) await mode.init();
|
||||
else terminal.requestResize();
|
||||
const renders = await terminal.takeRenders();
|
||||
if (index === 0) {
|
||||
builtInStatusLine = renders.flatMap(lines).find((line) => BUILT_IN_CONTEXT_PATTERN.test(line)) ?? null;
|
||||
}
|
||||
observations.push({
|
||||
percent,
|
||||
renderedOutput: renders.at(-1) ?? "",
|
||||
contextUsage: oracleSession.getContextUsage() ?? null,
|
||||
});
|
||||
}
|
||||
return {
|
||||
builtInStatusLine,
|
||||
observations,
|
||||
extensionErrors: session.resourceLoader.getExtensions().errors.map(({ error }) => error),
|
||||
};
|
||||
} finally {
|
||||
try {
|
||||
mode.stop("resume-hint");
|
||||
} finally {
|
||||
try {
|
||||
await runtime.dispose();
|
||||
} finally {
|
||||
if (previousOffline === undefined) delete process.env.PI_OFFLINE;
|
||||
else process.env.PI_OFFLINE = previousOffline;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const harness = createPiDocumentationEvalHarness({
|
||||
output: ({ session, agentDir }) => inspectContextFooter(session, agentDir),
|
||||
});
|
||||
|
||||
const contextFooterJudge = createJudge<PiCodingAgentInput, TuiFooterOutput>(
|
||||
"ContextFooterJudge",
|
||||
async ({ output }) => {
|
||||
let valid = output.extensionErrors.length === 0;
|
||||
const similarities: number[] = [];
|
||||
const contextMarker = "{context}";
|
||||
const builtInStatusLine = output.builtInStatusLine?.trim().replace(/\s+/g, " ");
|
||||
const reference = builtInStatusLine?.replace(BUILT_IN_CONTEXT_PATTERN, contextMarker);
|
||||
if (!reference?.includes(contextMarker)) valid = false;
|
||||
for (const [index, { percent, expectedBar }] of CONTEXT_FIXTURES.entries()) {
|
||||
const observation = output.observations[index];
|
||||
if (!observation || observation.percent !== percent) {
|
||||
valid = false;
|
||||
continue;
|
||||
}
|
||||
if (
|
||||
observation.contextUsage?.contextWindow !== CONTEXT_WINDOW ||
|
||||
Math.round((observation.contextUsage?.percent ?? -1) * 10) / 10 !== percent
|
||||
) {
|
||||
valid = false;
|
||||
}
|
||||
const actualStatusLine = lines(observation.renderedOutput).find(
|
||||
(line) => line.includes(expectedBar) || BUILT_IN_CONTEXT_PATTERN.test(line),
|
||||
);
|
||||
if (!actualStatusLine) {
|
||||
valid = false;
|
||||
continue;
|
||||
}
|
||||
const normalizedStatusLine = actualStatusLine.trim().replace(/\s+/g, " ");
|
||||
if (!normalizedStatusLine.includes(expectedBar) || BUILT_IN_CONTEXT_PATTERN.test(normalizedStatusLine)) {
|
||||
valid = false;
|
||||
continue;
|
||||
}
|
||||
for (const { expectedBar: otherBar } of CONTEXT_FIXTURES) {
|
||||
if (otherBar !== expectedBar && normalizedStatusLine.includes(otherBar)) valid = false;
|
||||
}
|
||||
if (reference) {
|
||||
const result = await Levenshtein({
|
||||
expected: reference,
|
||||
output: normalizedStatusLine.replace(expectedBar, contextMarker),
|
||||
});
|
||||
if (result.score !== null) similarities.push(result.score);
|
||||
}
|
||||
}
|
||||
const retained = similarities.length === CONTEXT_FIXTURES.length ? Math.min(...similarities) : 0;
|
||||
return {
|
||||
score: valid ? 1 : 0,
|
||||
metadata: {
|
||||
rationale: valid
|
||||
? `Footer retained ${Math.round(retained * 100)}% of the built-in status text.`
|
||||
: "Context progress behavior was incorrect.",
|
||||
retainedPercent: Math.round(retained * 100),
|
||||
},
|
||||
};
|
||||
},
|
||||
);
|
||||
|
||||
describeEval(
|
||||
"Customize the interactive context footer",
|
||||
{ harness, judges: [contextFooterJudge], judgeThreshold: null },
|
||||
(it) => {
|
||||
it("replaces numeric context usage with a progress bar", async ({ run }) => {
|
||||
await run([
|
||||
{
|
||||
type: "prompt",
|
||||
content:
|
||||
"Change the Pi footer replacing the built-in context info, e.g. `42.2%/272k (auto)`, with a ten-cell progress bar to make it show ████░░░░░░ 42% instead. Clamp percentages to the 0-100 range. Just replace the context. Otherwise leave everything else the same.",
|
||||
},
|
||||
{ type: "reload" },
|
||||
]);
|
||||
});
|
||||
},
|
||||
);
|
||||
19
third_party/pi/packages/evals/package.json
vendored
19
third_party/pi/packages/evals/package.json
vendored
@@ -1,20 +1,23 @@
|
||||
{
|
||||
"name": "@earendil-works/pi-evals",
|
||||
"version": "0.85.1",
|
||||
"version": "1.3.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"clean": "shx rm -rf .eval",
|
||||
"eval": "node scripts/run-evals.mjs",
|
||||
"eval": "npm run eval:host && npm run eval:docs --",
|
||||
"eval:host": "vitest run --config vitest.evals.config.ts --project host",
|
||||
"eval:docs": "node --experimental-strip-types src/cli.ts",
|
||||
"test": "vitest run --config vitest.test.config.ts"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@earendil-works/pi-ai": "^0.85.1",
|
||||
"@earendil-works/pi-coding-agent": "^0.85.1",
|
||||
"@earendil-works/pi-ai": "1.3.0",
|
||||
"@earendil-works/pi-coding-agent": "1.3.0",
|
||||
"@types/node": "22.19.19",
|
||||
"shx": "0.4.0",
|
||||
"typescript": "5.9.3",
|
||||
"vitest-evals": "0.15.0",
|
||||
"vitest": "4.1.9"
|
||||
"@vitest-evals/core": "0.15.0",
|
||||
"autoevals": "0.3.0",
|
||||
"shx": "0.3.4",
|
||||
"vitest": "4.1.11",
|
||||
"vitest-evals": "0.15.0"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,97 +0,0 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { mkdirSync } from "node:fs";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { createRequire } from "node:module";
|
||||
import { dirname, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const artifactDirectory = process.env.PI_EVAL_ARTIFACT_DIR
|
||||
? resolve(packageRoot, process.env.PI_EVAL_ARTIFACT_DIR)
|
||||
: resolve(
|
||||
packageRoot,
|
||||
".eval",
|
||||
`${new Date().toISOString().replaceAll(":", "-")}_${randomUUID()}`,
|
||||
);
|
||||
const args = process.argv.slice(2);
|
||||
let provider;
|
||||
let model;
|
||||
let hasCliModelSelection = false;
|
||||
const vitestArgs = [];
|
||||
|
||||
for (let index = 0; index < args.length; index += 1) {
|
||||
const arg = args[index];
|
||||
if (arg === "--provider" || arg === "--model") {
|
||||
const value = args[index + 1];
|
||||
if (!value || value.startsWith("-")) {
|
||||
console.error(`Missing value for ${arg}`);
|
||||
process.exit(1);
|
||||
}
|
||||
if (arg === "--provider") provider = value;
|
||||
else model = value;
|
||||
hasCliModelSelection = true;
|
||||
index += 1;
|
||||
continue;
|
||||
}
|
||||
if (arg.startsWith("--provider=")) {
|
||||
provider = arg.slice("--provider=".length);
|
||||
hasCliModelSelection = true;
|
||||
continue;
|
||||
}
|
||||
if (arg.startsWith("--model=")) {
|
||||
model = arg.slice("--model=".length);
|
||||
hasCliModelSelection = true;
|
||||
continue;
|
||||
}
|
||||
vitestArgs.push(arg);
|
||||
}
|
||||
|
||||
provider = provider?.trim() || undefined;
|
||||
model = model?.trim() || undefined;
|
||||
if (hasCliModelSelection) {
|
||||
if (!provider || !model) {
|
||||
console.error("CLI model selection requires both --provider and --model.");
|
||||
process.exit(1);
|
||||
}
|
||||
} else {
|
||||
provider = process.env.PI_PROVIDER?.trim() || undefined;
|
||||
model = process.env.PI_MODEL?.trim() || undefined;
|
||||
if (Boolean(provider) !== Boolean(model)) {
|
||||
console.error("Default model selection requires both PI_PROVIDER and PI_MODEL.");
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
const require = createRequire(import.meta.url);
|
||||
const vitestPackagePath = require.resolve("vitest/package.json");
|
||||
const vitestCliPath = resolve(dirname(vitestPackagePath), "vitest.mjs");
|
||||
|
||||
mkdirSync(artifactDirectory, { recursive: true, mode: 0o700 });
|
||||
console.error(`[eval] default-model=${provider && model ? `${provider}/${model}` : "none"}`);
|
||||
console.error(`[eval] artifacts=${artifactDirectory}`);
|
||||
const childEnvironment = {
|
||||
...process.env,
|
||||
PI_EVAL_ARTIFACT_DIR: artifactDirectory,
|
||||
};
|
||||
if (provider && model) {
|
||||
childEnvironment.PI_PROVIDER = provider;
|
||||
childEnvironment.PI_MODEL = model;
|
||||
} else {
|
||||
delete childEnvironment.PI_PROVIDER;
|
||||
delete childEnvironment.PI_MODEL;
|
||||
}
|
||||
const result = spawnSync(
|
||||
process.execPath,
|
||||
[vitestCliPath, "run", "--config", "vitest.config.ts", ...vitestArgs],
|
||||
{
|
||||
cwd: packageRoot,
|
||||
stdio: "inherit",
|
||||
env: childEnvironment,
|
||||
},
|
||||
);
|
||||
|
||||
if (result.error) {
|
||||
throw result.error;
|
||||
}
|
||||
|
||||
process.exit(result.status ?? 1);
|
||||
192
third_party/pi/packages/evals/src/cli.ts
vendored
Normal file
192
third_party/pi/packages/evals/src/cli.ts
vendored
Normal file
@@ -0,0 +1,192 @@
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { globSync } from "node:fs";
|
||||
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
||||
import { dirname, relative, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { buildImages, createDockerContext, discoverCases, requireEvalAuthFile, runTask } from "./docker.ts";
|
||||
import { createTaskPlan, type DiscoveredEvalCase, DOCUMENTATION_VARIANTS, parseDiscoveredCases } from "./plan.ts";
|
||||
import {
|
||||
type EvalObservation,
|
||||
erroredObservation,
|
||||
formatEvalComparisonReport,
|
||||
readTaskObservation,
|
||||
summarizeEvalObservations,
|
||||
} from "./report.ts";
|
||||
|
||||
type EvalCliOptions = {
|
||||
provider?: string;
|
||||
model?: string;
|
||||
runsPerVariant: number;
|
||||
requestedFiles: string[];
|
||||
discoveryArgs: string[];
|
||||
};
|
||||
|
||||
const RUNNER_OPTIONS = new Set(["--provider", "--model", "--runs-per-variant"]);
|
||||
const FILTER_OPTIONS = new Set(["-t", "--testNamePattern"]);
|
||||
|
||||
function parseEvalCli(args: readonly string[], environment: NodeJS.ProcessEnv = process.env): EvalCliOptions {
|
||||
let provider: string | undefined;
|
||||
let model: string | undefined;
|
||||
let runsText: string | undefined;
|
||||
let cliSelectedModel = false;
|
||||
const requestedFiles: string[] = [];
|
||||
const discoveryArgs: string[] = [];
|
||||
|
||||
for (let index = 0; index < args.length; index += 1) {
|
||||
const argument = args[index];
|
||||
if (argument.endsWith(".docs.eval.ts")) {
|
||||
requestedFiles.push(argument);
|
||||
continue;
|
||||
}
|
||||
if (FILTER_OPTIONS.has(argument)) {
|
||||
const value = args[index + 1];
|
||||
if (!value) throw new Error(`Missing value for ${argument}.`);
|
||||
discoveryArgs.push(argument, value);
|
||||
index += 1;
|
||||
continue;
|
||||
}
|
||||
if (argument.startsWith("--testNamePattern=")) {
|
||||
if (argument.length === "--testNamePattern=".length) throw new Error("Missing value for --testNamePattern.");
|
||||
discoveryArgs.push(argument);
|
||||
continue;
|
||||
}
|
||||
|
||||
const equals = argument.indexOf("=");
|
||||
const name = equals === -1 ? argument : argument.slice(0, equals);
|
||||
if (RUNNER_OPTIONS.has(name)) {
|
||||
const value = equals === -1 ? args[index + 1] : argument.slice(equals + 1);
|
||||
if (!value || (equals === -1 && value.startsWith("-"))) throw new Error(`Missing value for ${name}.`);
|
||||
if (name === "--provider") provider = value;
|
||||
else if (name === "--model") model = value;
|
||||
else runsText = value;
|
||||
if (name !== "--runs-per-variant") cliSelectedModel = true;
|
||||
if (equals === -1) index += 1;
|
||||
continue;
|
||||
}
|
||||
throw new Error(`Unsupported eval argument: ${argument}`);
|
||||
}
|
||||
|
||||
provider = provider?.trim() || undefined;
|
||||
model = model?.trim() || undefined;
|
||||
if (cliSelectedModel) {
|
||||
if (!provider || !model) throw new Error("CLI model selection requires both --provider and --model.");
|
||||
} else {
|
||||
provider = environment.PI_PROVIDER?.trim() || undefined;
|
||||
model = environment.PI_MODEL?.trim() || undefined;
|
||||
if (Boolean(provider) !== Boolean(model)) throw new Error("Set both PI_PROVIDER and PI_MODEL, or neither.");
|
||||
}
|
||||
|
||||
const configuredRuns = (runsText ?? environment.PI_EVAL_RUNS_PER_VARIANT)?.trim();
|
||||
const runsPerVariant = configuredRuns ? Number(configuredRuns) : 1;
|
||||
if (!Number.isSafeInteger(runsPerVariant) || runsPerVariant < 1) {
|
||||
throw new Error("Runs per variant must be a positive integer.");
|
||||
}
|
||||
|
||||
return { provider, model, runsPerVariant, requestedFiles, discoveryArgs };
|
||||
}
|
||||
|
||||
const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
function artifactRunId(): string {
|
||||
return `${new Date().toISOString().replaceAll(":", "-")}_${randomUUID()}`;
|
||||
}
|
||||
|
||||
function containerPath(path: string): string {
|
||||
const absolute = resolve(packageRoot, path);
|
||||
const packageRelative = relative(packageRoot, absolute);
|
||||
if (packageRelative.startsWith("..")) throw new Error(`Eval file must be inside ${packageRoot}: ${path}`);
|
||||
return packageRelative.replaceAll("\\", "/");
|
||||
}
|
||||
|
||||
function normalizeDiscoveredFile(path: string): string {
|
||||
const prefix = "/repo/packages/evals/";
|
||||
if (!path.startsWith(prefix)) throw new Error(`Discovered eval path is outside the container package: ${path}`);
|
||||
return path.slice(prefix.length);
|
||||
}
|
||||
|
||||
function compareDiscovery(left: readonly DiscoveredEvalCase[], right: readonly DiscoveredEvalCase[]): void {
|
||||
const identities = (cases: readonly DiscoveredEvalCase[]) =>
|
||||
cases.map(({ fullName, file }) => JSON.stringify([fullName, normalizeDiscoveredFile(file)])).sort();
|
||||
if (JSON.stringify(identities(left)) !== JSON.stringify(identities(right))) {
|
||||
throw new Error("Documentation variants discovered different eval cases.");
|
||||
}
|
||||
}
|
||||
|
||||
const cli = parseEvalCli(process.argv.slice(2));
|
||||
if (!cli.provider || !cli.model) throw new Error("Set PI_PROVIDER and PI_MODEL, or pass --provider and --model.");
|
||||
const selectedModel = { provider: cli.provider, model: cli.model };
|
||||
const modelIdentity = `${selectedModel.provider}/${selectedModel.model}`;
|
||||
const files = (
|
||||
cli.requestedFiles.length > 0 ? cli.requestedFiles : globSync("evals/**/*.docs.eval.ts", { cwd: packageRoot })
|
||||
)
|
||||
.map(containerPath)
|
||||
.sort();
|
||||
if (files.length === 0) throw new Error("No documentation eval files were selected.");
|
||||
for (const file of files) {
|
||||
if (!file.endsWith(".docs.eval.ts")) throw new Error(`Documentation runner cannot execute non-doc eval: ${file}`);
|
||||
}
|
||||
|
||||
const artifactDirectory = resolve(packageRoot, ".eval", artifactRunId());
|
||||
const authPath = requireEvalAuthFile(selectedModel.provider);
|
||||
await mkdir(artifactDirectory, { recursive: true, mode: 0o700 });
|
||||
const images = buildImages();
|
||||
if (images.with_docs.id === images.without_docs.id)
|
||||
throw new Error("Documentation variants resolved to the same image.");
|
||||
const docker = createDockerContext(
|
||||
images,
|
||||
artifactDirectory,
|
||||
authPath,
|
||||
selectedModel.provider,
|
||||
selectedModel.model,
|
||||
cli.runsPerVariant,
|
||||
);
|
||||
|
||||
const discoveries = [];
|
||||
for (const variant of DOCUMENTATION_VARIANTS) {
|
||||
const path = discoverCases(docker, variant, files, cli.discoveryArgs);
|
||||
discoveries.push(parseDiscoveredCases(JSON.parse(await readFile(path, "utf8"))));
|
||||
}
|
||||
compareDiscovery(discoveries[0], discoveries[1]);
|
||||
const cases = discoveries[0].map((evalCase) => ({ ...evalCase, file: normalizeDiscoveredFile(evalCase.file) }));
|
||||
if (cases.length === 0) throw new Error("No documentation eval cases matched the selection.");
|
||||
const tasks = createTaskPlan(cases, modelIdentity, cli.runsPerVariant);
|
||||
|
||||
const protocol = {
|
||||
schemaVersion: 1,
|
||||
model: modelIdentity,
|
||||
runsPerVariant: cli.runsPerVariant,
|
||||
images,
|
||||
files,
|
||||
cases: cases.map(({ evalSet, caseId, file }) => ({ evalSet, caseId, file })),
|
||||
tasks,
|
||||
};
|
||||
const protocolText = JSON.stringify(protocol);
|
||||
const protocolDigest = createHash("sha256").update(protocolText).digest("hex");
|
||||
await writeFile(
|
||||
resolve(artifactDirectory, "protocol.json"),
|
||||
`${JSON.stringify({ ...protocol, protocolDigest }, null, 2)}\n`,
|
||||
);
|
||||
await writeFile(resolve(artifactDirectory, "expected-runs.json"), `${JSON.stringify(tasks, null, 2)}\n`);
|
||||
|
||||
const observations: EvalObservation[] = [];
|
||||
for (const [index, task] of tasks.entries()) {
|
||||
console.log(
|
||||
`\n[${index + 1}/${tasks.length}] ${task.evalSet} > ${task.caseId} | ${task.variant} | ${task.model} | run ${task.runNumber}`,
|
||||
);
|
||||
const reportPath = runTask(docker, task);
|
||||
const observation: EvalObservation = reportPath
|
||||
? await readTaskObservation(task, reportPath, artifactDirectory)
|
||||
: erroredObservation(task);
|
||||
observations.push(observation);
|
||||
await writeFile(
|
||||
resolve(artifactDirectory, "observations.jsonl"),
|
||||
`${observations.map((item) => JSON.stringify(item)).join("\n")}\n`,
|
||||
);
|
||||
}
|
||||
|
||||
const report = summarizeEvalObservations(protocolDigest, tasks, observations);
|
||||
const reportText = formatEvalComparisonReport(report);
|
||||
await writeFile(resolve(artifactDirectory, "report.json"), `${JSON.stringify(report, null, 2)}\n`);
|
||||
await writeFile(resolve(artifactDirectory, "report.txt"), `${reportText}\n`);
|
||||
console.log(`\n${reportText}`);
|
||||
console.log(`\nArtifacts: ${artifactDirectory}`);
|
||||
if (report.blockedPairs.length > 0) process.exitCode = 1;
|
||||
175
third_party/pi/packages/evals/src/docker.ts
vendored
Normal file
175
third_party/pi/packages/evals/src/docker.ts
vendored
Normal file
@@ -0,0 +1,175 @@
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import { existsSync, mkdirSync, readFileSync, statSync } from "node:fs";
|
||||
import { homedir } from "node:os";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import type { DocumentationVariant, EvalTask } from "./plan.ts";
|
||||
|
||||
const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const repositoryRoot = resolve(packageRoot, "../..");
|
||||
|
||||
export type BuiltImages = Record<DocumentationVariant, { name: string; id: string }>;
|
||||
|
||||
type DockerContext = {
|
||||
images: BuiltImages;
|
||||
artifactDirectory: string;
|
||||
authPath: string;
|
||||
provider: string;
|
||||
model: string;
|
||||
runsPerVariant: number;
|
||||
};
|
||||
|
||||
function execute(command: string, args: readonly string[], capture = false): { status: number; stdout: string } {
|
||||
const result = spawnSync(command, args, {
|
||||
cwd: packageRoot,
|
||||
stdio: capture ? ["ignore", "pipe", "inherit"] : "inherit",
|
||||
encoding: "utf8",
|
||||
});
|
||||
if (result.error) throw result.error;
|
||||
return { status: result.status ?? 1, stdout: capture ? result.stdout : "" };
|
||||
}
|
||||
|
||||
function requireSuccess(command: string, args: readonly string[]): void {
|
||||
const { status } = execute(command, args);
|
||||
if (status !== 0) throw new Error(`${command} exited with status ${status}.`);
|
||||
}
|
||||
|
||||
export function buildImages(): BuiltImages {
|
||||
const prefix = `pi-evals-${createHash("sha256").update(repositoryRoot).digest("hex").slice(0, 12)}`;
|
||||
const images = {
|
||||
without_docs: { name: `${prefix}-without-docs:local`, id: "" },
|
||||
with_docs: { name: `${prefix}-with-docs:local`, id: "" },
|
||||
} satisfies BuiltImages;
|
||||
for (const variant of ["without_docs", "with_docs"] as const) {
|
||||
requireSuccess("docker", [
|
||||
"build",
|
||||
"--target",
|
||||
variant,
|
||||
"--tag",
|
||||
images[variant].name,
|
||||
"--file",
|
||||
join(packageRoot, "docker", "Dockerfile"),
|
||||
repositoryRoot,
|
||||
]);
|
||||
const inspected = execute("docker", ["image", "inspect", "--format", "{{.Id}}", images[variant].name], true);
|
||||
if (inspected.status !== 0 || !inspected.stdout.trim())
|
||||
throw new Error(`Cannot inspect ${images[variant].name}.`);
|
||||
images[variant].id = inspected.stdout.trim();
|
||||
}
|
||||
return images;
|
||||
}
|
||||
|
||||
function environment(name: string, value: string): string[] {
|
||||
return ["--env", `${name}=${value}`];
|
||||
}
|
||||
|
||||
export function requireEvalAuthFile(provider: string): string {
|
||||
const path = join(
|
||||
process.env.PI_CODING_AGENT_DIR ? resolve(process.env.PI_CODING_AGENT_DIR) : join(homedir(), ".pi", "agent"),
|
||||
"auth.json",
|
||||
);
|
||||
if (!existsSync(path) || !statSync(path).isFile())
|
||||
throw new Error(`Eval authentication file does not exist: ${path}`);
|
||||
let credentials: unknown;
|
||||
try {
|
||||
credentials = JSON.parse(readFileSync(path, "utf8"));
|
||||
} catch (error) {
|
||||
throw new Error(`Eval authentication file is invalid: ${path}`, { cause: error });
|
||||
}
|
||||
if (
|
||||
typeof credentials !== "object" ||
|
||||
credentials === null ||
|
||||
Array.isArray(credentials) ||
|
||||
!Object.hasOwn(credentials, provider)
|
||||
) {
|
||||
throw new Error(`Eval authentication file has no credential for provider ${provider}.`);
|
||||
}
|
||||
return path;
|
||||
}
|
||||
|
||||
function dockerArgs(
|
||||
context: DockerContext,
|
||||
variant: DocumentationVariant,
|
||||
outputDirectory: string,
|
||||
entrypointArgs: readonly string[],
|
||||
): string[] {
|
||||
mkdirSync(outputDirectory, { recursive: true, mode: 0o700 });
|
||||
const args = [
|
||||
"run",
|
||||
"--rm",
|
||||
"--read-only",
|
||||
"--tmpfs",
|
||||
"/tmp:rw,exec,mode=1777",
|
||||
"--tmpfs",
|
||||
"/repo/node_modules/.vite-temp:rw,exec,mode=1777",
|
||||
"--mount",
|
||||
`type=bind,source=${outputDirectory},target=/artifacts`,
|
||||
...environment("PI_EVAL_ARTIFACT_DIR", "/artifacts"),
|
||||
...environment("PI_EVAL_RUNS_PER_VARIANT", String(context.runsPerVariant)),
|
||||
...environment("PI_EVAL_SANDBOX_UID", "65532"),
|
||||
...environment("PI_EVAL_SANDBOX_GID", "65532"),
|
||||
...environment("PI_PROVIDER", context.provider),
|
||||
...environment("PI_MODEL", context.model),
|
||||
];
|
||||
if (typeof process.getuid === "function" && typeof process.getgid === "function") {
|
||||
args.push(
|
||||
...environment("PI_EVAL_ARTIFACT_UID", String(process.getuid())),
|
||||
...environment("PI_EVAL_ARTIFACT_GID", String(process.getgid())),
|
||||
);
|
||||
}
|
||||
args.push("--mount", `type=bind,source=${context.authPath},target=/run/pi-eval-secrets/auth.json,readonly`);
|
||||
args.push(context.images[variant].name, ...entrypointArgs);
|
||||
return args;
|
||||
}
|
||||
|
||||
export function createDockerContext(
|
||||
images: BuiltImages,
|
||||
artifactDirectory: string,
|
||||
authPath: string,
|
||||
provider: string,
|
||||
model: string,
|
||||
runsPerVariant: number,
|
||||
): DockerContext {
|
||||
return { images, artifactDirectory, authPath, provider, model, runsPerVariant };
|
||||
}
|
||||
|
||||
export function discoverCases(
|
||||
context: DockerContext,
|
||||
variant: DocumentationVariant,
|
||||
files: readonly string[],
|
||||
vitestArgs: readonly string[],
|
||||
): string {
|
||||
const outputDirectory = join(context.artifactDirectory, "discovery", variant);
|
||||
const args = dockerArgs(context, variant, outputDirectory, [
|
||||
"--discover",
|
||||
"--project",
|
||||
"docs",
|
||||
...files,
|
||||
...vitestArgs,
|
||||
]);
|
||||
const { status } = execute("docker", args);
|
||||
if (status !== 0) throw new Error(`${variant} eval discovery failed.`);
|
||||
return join(outputDirectory, "discovered-tests.json");
|
||||
}
|
||||
|
||||
function taskDirectoryName(task: EvalTask): string {
|
||||
const identity = JSON.stringify([task.evalSet, task.caseId, task.variant, task.model, task.runNumber]);
|
||||
return createHash("sha256").update(identity).digest("hex");
|
||||
}
|
||||
|
||||
export function runTask(context: DockerContext, task: EvalTask): string | undefined {
|
||||
const outputDirectory = join(context.artifactDirectory, "tasks", taskDirectoryName(task));
|
||||
const testName = `${task.evalSet} ${task.caseId}`;
|
||||
const exactName = `^${testName.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}$`;
|
||||
const args = dockerArgs(context, task.variant, outputDirectory, [
|
||||
"--project",
|
||||
"docs",
|
||||
task.file,
|
||||
"--testNamePattern",
|
||||
exactName,
|
||||
]);
|
||||
execute("docker", args);
|
||||
const reportPath = join(outputDirectory, "vitest.json");
|
||||
return existsSync(reportPath) ? reportPath : undefined;
|
||||
}
|
||||
140
third_party/pi/packages/evals/src/extensions.eval.ts
vendored
140
third_party/pi/packages/evals/src/extensions.eval.ts
vendored
@@ -1,140 +0,0 @@
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { describe, expect } from "vitest";
|
||||
import { createJudge, describeEval } from "vitest-evals";
|
||||
import { createPiCodingAgentHarness, type PiCodingAgentInput } from "./pi-harness.ts";
|
||||
import { recordEvalSourceArtifact } from "./vitest-evals/artifacts.ts";
|
||||
import { evalHarnessTable } from "./vitest-evals/harness-table.ts";
|
||||
|
||||
type ExtensionAuthoringOutput = {
|
||||
response: string;
|
||||
systemPromptHasGuidelines: boolean;
|
||||
systemPromptHasPiDocs: boolean;
|
||||
extensionErrors: Array<{ path: string; error: string }>;
|
||||
loadedExtensions: Array<{ path: string; tools: string[] }>;
|
||||
extensionSource: string | null;
|
||||
};
|
||||
|
||||
function createExtensionAuthoringHarness(name: string, transformSystemPrompt?: (defaultPrompt: string) => string) {
|
||||
return createPiCodingAgentHarness({
|
||||
name,
|
||||
...(transformSystemPrompt ? { transformSystemPrompt } : {}),
|
||||
output: ({ response, session }) => {
|
||||
const extensions = session.resourceLoader.getExtensions();
|
||||
const extensionPath = join(session.sessionManager.getCwd(), ".pi", "extensions", "hello.ts");
|
||||
const extensionSource = existsSync(extensionPath) ? readFileSync(extensionPath, "utf8") : null;
|
||||
return {
|
||||
response,
|
||||
systemPromptHasGuidelines: session.systemPrompt.includes("\nGuidelines:\n"),
|
||||
systemPromptHasPiDocs: session.systemPrompt.includes("\nPi documentation (read only"),
|
||||
extensionErrors: extensions.errors,
|
||||
loadedExtensions: extensions.extensions.map(({ path, tools }) => ({
|
||||
path,
|
||||
tools: [...tools.keys()],
|
||||
})),
|
||||
extensionSource,
|
||||
};
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function excludeGuidelinesAndDocumentation(defaultPrompt: string): string {
|
||||
const guidelinesStart = defaultPrompt.indexOf("\nGuidelines:\n");
|
||||
if (guidelinesStart === -1) throw new Error("Default Pi system prompt has no Guidelines section.");
|
||||
return defaultPrompt.slice(0, guidelinesStart);
|
||||
}
|
||||
|
||||
function prepareDefaultPromptOverride(defaultPrompt: string): string {
|
||||
const cwdStart = defaultPrompt.lastIndexOf("\nCurrent working directory: ");
|
||||
if (cwdStart === -1) throw new Error("Default Pi system prompt has no working-directory section.");
|
||||
return defaultPrompt.slice(0, cwdStart);
|
||||
}
|
||||
|
||||
const ExtensionAuthoringJudge = createJudge<PiCodingAgentInput, ExtensionAuthoringOutput>(
|
||||
"ExtensionAuthoringJudge",
|
||||
({ output, toolCalls }) => {
|
||||
const failures: string[] = [];
|
||||
if (output.extensionSource === null) {
|
||||
failures.push("generated extension source is unavailable");
|
||||
} else {
|
||||
const imports = Array.from(
|
||||
output.extensionSource.matchAll(/\b(?:from|import)\s+["']([^"']+)["']/g),
|
||||
(match) => match[1],
|
||||
);
|
||||
if (!imports.includes("@earendil-works/pi-coding-agent")) {
|
||||
failures.push("extension does not import the canonical @earendil-works/pi-coding-agent package");
|
||||
}
|
||||
if (imports.some((specifier) => specifier.startsWith("@mariozechner/"))) {
|
||||
failures.push("extension imports a legacy @mariozechner package");
|
||||
}
|
||||
if (imports.some((specifier) => specifier.startsWith("@sinclair/typebox"))) {
|
||||
failures.push('extension imports legacy "@sinclair/typebox" instead of "typebox"');
|
||||
}
|
||||
}
|
||||
if (output.extensionErrors.length > 0) failures.push("extension loader reported errors");
|
||||
if (!output.loadedExtensions.some(({ tools }) => tools.includes("hello"))) {
|
||||
failures.push('no loaded extension registered the "hello" tool');
|
||||
}
|
||||
if (
|
||||
!toolCalls.some(
|
||||
(call) =>
|
||||
call.name === "hello" &&
|
||||
call.status === "ok" &&
|
||||
call.arguments?.name === "Bob" &&
|
||||
call.result === "Hello, Bob!",
|
||||
)
|
||||
) {
|
||||
failures.push('no successful hello({ name: "Bob" }) call returned "Hello, Bob!"');
|
||||
}
|
||||
if (output.response !== "Hello, Bob!") failures.push('final response was not exactly "Hello, Bob!"');
|
||||
|
||||
return {
|
||||
score: failures.length === 0 ? 1 : 0,
|
||||
metadata: {
|
||||
rationale: failures.length === 0 ? "Extension authoring workflow completed." : failures.join("; "),
|
||||
},
|
||||
};
|
||||
},
|
||||
);
|
||||
|
||||
const extensionHarnessTable = evalHarnessTable("Pi extension authoring system prompt", {
|
||||
baseline: createExtensionAuthoringHarness("system-prompt-without-docs", excludeGuidelinesAndDocumentation),
|
||||
candidate: createExtensionAuthoringHarness("default-system-prompt", prepareDefaultPromptOverride),
|
||||
});
|
||||
|
||||
describe.for(extensionHarnessTable)("$name", ({ harness }) => {
|
||||
describeEval(
|
||||
"Pi extension authoring system prompt",
|
||||
{ harness, judges: [ExtensionAuthoringJudge], judgeThreshold: null },
|
||||
(it) => {
|
||||
it("creates, reloads, and uses a hello extension", async ({ run, task }) => {
|
||||
const result = await run([
|
||||
{
|
||||
type: "prompt",
|
||||
content:
|
||||
"Create a Pi extension with a hello tool that takes a name and returns a greeting. For example, passing Bob should return `Hello, Bob!`.",
|
||||
},
|
||||
{ type: "reload" },
|
||||
{
|
||||
type: "prompt",
|
||||
content:
|
||||
"Use the hello tool to greet Bob. Respond with exactly the tool's greeting and nothing else.",
|
||||
},
|
||||
]);
|
||||
if (result.output.extensionSource !== null) {
|
||||
const runId = result.artifacts?.runId;
|
||||
if (typeof runId !== "string") throw new Error("Pi eval run did not record a run ID.");
|
||||
await recordEvalSourceArtifact(task, runId, {
|
||||
name: "hello.ts",
|
||||
contentType: "text/typescript",
|
||||
body: result.output.extensionSource,
|
||||
bodyEncoding: "utf-8",
|
||||
});
|
||||
}
|
||||
const expectsFullPrompt = harness.name === "default-system-prompt";
|
||||
expect(result.output.systemPromptHasGuidelines).toBe(expectsFullPrompt);
|
||||
expect(result.output.systemPromptHasPiDocs).toBe(expectsFullPrompt);
|
||||
});
|
||||
},
|
||||
);
|
||||
});
|
||||
537
third_party/pi/packages/evals/src/harness.ts
vendored
Normal file
537
third_party/pi/packages/evals/src/harness.ts
vendored
Normal file
@@ -0,0 +1,537 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import type { Dirent } from "node:fs";
|
||||
import { existsSync } from "node:fs";
|
||||
import { chmod, chown, lstat, mkdir, mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path";
|
||||
import { performance } from "node:perf_hooks";
|
||||
import { contentText, InMemoryCredentialStore } from "@earendil-works/pi-ai";
|
||||
import { getCurrentSystemPrompt } from "@earendil-works/pi-ai/utils/transcript";
|
||||
import {
|
||||
type AgentSession,
|
||||
type CreateAgentSessionOptions,
|
||||
createAgentSessionFromServices,
|
||||
createAgentSessionServices,
|
||||
getAgentDir,
|
||||
type InlineExtension,
|
||||
ModelRuntime,
|
||||
readStoredCredential,
|
||||
SessionManager,
|
||||
} from "@earendil-works/pi-coding-agent";
|
||||
import {
|
||||
attachHarnessRunToError,
|
||||
createHarness,
|
||||
type Harness,
|
||||
type HarnessContext,
|
||||
type JsonValue,
|
||||
normalizeHarnessRun,
|
||||
normalizeRecord,
|
||||
type SimpleHarnessResult,
|
||||
type TranscriptEvent,
|
||||
toJsonValue,
|
||||
type UsageSummary,
|
||||
} from "vitest-evals/harness";
|
||||
import type { DocumentationVariant } from "./plan.ts";
|
||||
import { PI_SESSION_SNAPSHOT_ARTIFACT } from "./report.ts";
|
||||
|
||||
type PiRunDiagnostics = {
|
||||
events: TranscriptEvent[];
|
||||
metadata: Record<string, unknown>;
|
||||
usage: UsageSummary;
|
||||
};
|
||||
|
||||
export type PiCodingAgentInput = string | Array<{ type: "prompt"; content: string } | { type: "reload" }>;
|
||||
|
||||
export type PiCodingAgentModelSelection = {
|
||||
provider: string;
|
||||
id: string;
|
||||
};
|
||||
|
||||
export type PiCodingAgentHarnessOptions = {
|
||||
name?: string;
|
||||
model?: PiCodingAgentModelSelection;
|
||||
noTools?: CreateAgentSessionOptions["noTools"];
|
||||
tools?: CreateAgentSessionOptions["tools"];
|
||||
customTools?: CreateAgentSessionOptions["customTools"];
|
||||
workspaceFiles?: Readonly<Record<string, string>>;
|
||||
transformSystemPrompt?: (defaultPrompt: string) => string;
|
||||
expectedPiDocumentation?: boolean;
|
||||
};
|
||||
|
||||
export type PiCodingAgentHarnessWithOutput<TOutput extends JsonValue> = PiCodingAgentHarnessOptions & {
|
||||
output: (args: {
|
||||
response: string;
|
||||
session: AgentSession;
|
||||
systemPrompt: string;
|
||||
agentDir: string;
|
||||
}) => TOutput | Promise<TOutput>;
|
||||
};
|
||||
|
||||
export function resolveModelSelection(
|
||||
explicitModel: PiCodingAgentModelSelection | undefined,
|
||||
environment: { PI_PROVIDER?: string; PI_MODEL?: string } = process.env,
|
||||
): PiCodingAgentModelSelection {
|
||||
const provider = (explicitModel?.provider ?? environment.PI_PROVIDER)?.trim();
|
||||
const id = (explicitModel?.id ?? environment.PI_MODEL)?.trim();
|
||||
if (!provider || !id) {
|
||||
throw new Error("Select a harness model explicitly or set both PI_PROVIDER and PI_MODEL as defaults.");
|
||||
}
|
||||
return { provider, id };
|
||||
}
|
||||
|
||||
export function applyIsolatedEnvironment(home: string, agentDir: string): () => void {
|
||||
const overrides = { HOME: home, USERPROFILE: home, PI_CODING_AGENT_DIR: agentDir };
|
||||
const previous = new Map<string, string | undefined>();
|
||||
for (const name of Object.keys(process.env)) {
|
||||
if (!name.startsWith("PI_EVAL_")) continue;
|
||||
previous.set(name, process.env[name]);
|
||||
delete process.env[name];
|
||||
}
|
||||
for (const [name, value] of Object.entries(overrides)) {
|
||||
if (!previous.has(name)) previous.set(name, process.env[name]);
|
||||
process.env[name] = value;
|
||||
}
|
||||
return () => {
|
||||
for (const [name, value] of previous) {
|
||||
if (value === undefined) delete process.env[name];
|
||||
else process.env[name] = value;
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
type SandboxIdentity = { uid: number; gid: number };
|
||||
|
||||
function parseSandboxId(name: "PI_EVAL_SANDBOX_UID" | "PI_EVAL_SANDBOX_GID"): number | undefined {
|
||||
const value = process.env[name];
|
||||
if (value === undefined) return undefined;
|
||||
const id = Number(value);
|
||||
if (!Number.isSafeInteger(id) || id < 1) throw new Error(`${name} must be a positive integer.`);
|
||||
return id;
|
||||
}
|
||||
|
||||
function resolveSandboxIdentity(): SandboxIdentity | undefined {
|
||||
const uid = parseSandboxId("PI_EVAL_SANDBOX_UID");
|
||||
const gid = parseSandboxId("PI_EVAL_SANDBOX_GID");
|
||||
if (uid === undefined && gid === undefined) return undefined;
|
||||
if (uid === undefined || gid === undefined) {
|
||||
throw new Error("Set both PI_EVAL_SANDBOX_UID and PI_EVAL_SANDBOX_GID, or neither.");
|
||||
}
|
||||
return { uid, gid };
|
||||
}
|
||||
|
||||
async function chownTree(path: string, identity: SandboxIdentity): Promise<void> {
|
||||
const stats = await lstat(path);
|
||||
if (stats.isDirectory() && !stats.isSymbolicLink()) {
|
||||
await Promise.all((await readdir(path)).map((entry) => chownTree(join(path, entry), identity)));
|
||||
}
|
||||
await chown(path, identity.uid, identity.gid);
|
||||
}
|
||||
|
||||
async function protectTransformedModules(): Promise<string[]> {
|
||||
const protectedPaths: string[] = [];
|
||||
for (const directory of await readdir(tmpdir(), { withFileTypes: true })) {
|
||||
if (!directory.isDirectory()) continue;
|
||||
const cacheRoot = join(tmpdir(), directory.name);
|
||||
const ssrRoot = join(cacheRoot, "ssr");
|
||||
let files: Dirent[];
|
||||
try {
|
||||
files = await readdir(ssrRoot, { withFileTypes: true });
|
||||
} catch (error) {
|
||||
if (error instanceof Error && "code" in error && error.code === "ENOENT") continue;
|
||||
throw error;
|
||||
}
|
||||
await Promise.all([chmod(cacheRoot, 0o755), chmod(ssrRoot, 0o755)]);
|
||||
for (const file of files) {
|
||||
if (!file.isFile()) continue;
|
||||
const path = join(ssrRoot, file.name);
|
||||
await chmod(path, 0o600);
|
||||
protectedPaths.push(path);
|
||||
}
|
||||
}
|
||||
return protectedPaths;
|
||||
}
|
||||
|
||||
async function enterToolSandbox(root: string, identity: SandboxIdentity | undefined): Promise<void> {
|
||||
if (!identity) return;
|
||||
if (
|
||||
typeof process.getuid !== "function" ||
|
||||
typeof process.geteuid !== "function" ||
|
||||
typeof process.setuid !== "function" ||
|
||||
typeof process.setgid !== "function" ||
|
||||
typeof process.setgroups !== "function"
|
||||
) {
|
||||
throw new Error("The eval filesystem sandbox requires POSIX user APIs.");
|
||||
}
|
||||
if (process.getuid() !== 0 || process.geteuid() !== 0) {
|
||||
throw new Error("The eval runner must start as root before entering the unprivileged tool sandbox.");
|
||||
}
|
||||
|
||||
const protectedPaths = await protectTransformedModules();
|
||||
await chownTree(root, identity);
|
||||
process.setgroups([]);
|
||||
process.setgid(identity.gid);
|
||||
process.setuid(identity.uid);
|
||||
if (process.getuid() !== identity.uid || process.geteuid() !== identity.uid) {
|
||||
throw new Error("Failed to enter the unprivileged eval tool sandbox.");
|
||||
}
|
||||
for (const path of protectedPaths) {
|
||||
try {
|
||||
await readFile(path);
|
||||
} catch (error) {
|
||||
if (error instanceof Error && "code" in error && (error.code === "EACCES" || error.code === "ENOENT")) {
|
||||
continue;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
throw new Error(`The unprivileged eval process can read a transformed eval module: ${path}`);
|
||||
}
|
||||
}
|
||||
|
||||
function toTranscriptEvents(messages: AgentSession["messages"]): TranscriptEvent[] {
|
||||
const events: TranscriptEvent[] = [];
|
||||
for (const message of messages) {
|
||||
if (message.role === "user") {
|
||||
events.push({ type: "message", role: "user", content: contentText(message.content) });
|
||||
continue;
|
||||
}
|
||||
if (message.role === "assistant") {
|
||||
const text = contentText(message.content);
|
||||
if (text) events.push({ type: "message", role: "assistant", content: text });
|
||||
for (const part of message.content) {
|
||||
if (part.type !== "toolCall") continue;
|
||||
events.push({
|
||||
type: "tool_call",
|
||||
id: part.id,
|
||||
name: part.name,
|
||||
arguments: normalizeRecord(part.arguments),
|
||||
});
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (message.role === "toolResult") {
|
||||
const text = contentText(message.content);
|
||||
events.push({
|
||||
type: "tool_result",
|
||||
toolCallId: message.toolCallId,
|
||||
name: message.toolName,
|
||||
content: message.content.every((part) => part.type === "text") ? text : toJsonValue(message.content),
|
||||
...(message.isError ? { error: { message: text || "Tool failed" } } : {}),
|
||||
});
|
||||
}
|
||||
}
|
||||
return events;
|
||||
}
|
||||
|
||||
async function seedWorkspace(workspace: string, files: Readonly<Record<string, string>> | undefined): Promise<void> {
|
||||
for (const [name, content] of Object.entries(files ?? {})) {
|
||||
if (!name || isAbsolute(name)) throw new TypeError(`Invalid workspace fixture path: ${name}`);
|
||||
const path = resolve(workspace, name);
|
||||
const pathFromWorkspace = relative(workspace, path);
|
||||
if (pathFromWorkspace === ".." || pathFromWorkspace.startsWith(`..${sep}`) || isAbsolute(pathFromWorkspace)) {
|
||||
throw new TypeError(`Workspace fixture escapes the workspace: ${name}`);
|
||||
}
|
||||
await mkdir(dirname(path), { recursive: true });
|
||||
await writeFile(path, content, { mode: 0o600 });
|
||||
}
|
||||
}
|
||||
|
||||
async function promptAgent(session: AgentSession, input: string, signal: AbortSignal | undefined): Promise<string> {
|
||||
signal?.throwIfAborted();
|
||||
const previousMessageCount = session.messages.length;
|
||||
await session.prompt(input);
|
||||
const assistant = session.messages
|
||||
.slice(previousMessageCount)
|
||||
.reverse()
|
||||
.find((message) => message.role === "assistant");
|
||||
if (!assistant) throw new Error("Agent run completed without an assistant message.");
|
||||
if (assistant.stopReason !== "stop" && assistant.stopReason !== "toolUse") {
|
||||
throw new Error(
|
||||
assistant.errorMessage ?? `Agent run ended with unexpected stop reason: ${assistant.stopReason}.`,
|
||||
);
|
||||
}
|
||||
const output = session.getLastAssistantText();
|
||||
if (!output && assistant.stopReason === "stop") throw new Error("Agent run produced no assistant text.");
|
||||
return output ?? "";
|
||||
}
|
||||
|
||||
export function verifySystemPrompt(
|
||||
systemPrompt: string,
|
||||
options: Pick<PiCodingAgentHarnessOptions, "name" | "expectedPiDocumentation">,
|
||||
): string {
|
||||
if (options.expectedPiDocumentation === undefined) return systemPrompt;
|
||||
if (!systemPrompt.includes("\n<rules>\n")) {
|
||||
throw new Error(`Pi system prompt lost its rules in the ${options.name} eval variant.`);
|
||||
}
|
||||
const hasDocumentation = systemPrompt.includes("\n<docs>\nPi documentation (read only");
|
||||
if (hasDocumentation !== options.expectedPiDocumentation) {
|
||||
throw new Error(`Pi system prompt does not match the ${options.name} eval variant.`);
|
||||
}
|
||||
return systemPrompt;
|
||||
}
|
||||
|
||||
async function runPiCodingAgent<TOutput extends JsonValue>(
|
||||
input: PiCodingAgentInput,
|
||||
signal: AbortSignal | undefined,
|
||||
setArtifact: HarnessContext["setArtifact"],
|
||||
options: PiCodingAgentHarnessOptions | PiCodingAgentHarnessWithOutput<TOutput>,
|
||||
): Promise<SimpleHarnessResult<string | TOutput>> {
|
||||
const startedAt = performance.now();
|
||||
signal?.throwIfAborted();
|
||||
const selection = resolveModelSelection(options.model);
|
||||
const hostAgentDir = getAgentDir();
|
||||
const sandboxIdentity = resolveSandboxIdentity();
|
||||
const root = await mkdtemp(join(tmpdir(), "pi-eval-"));
|
||||
const workspace = join(root, "workspace");
|
||||
const isolatedHome = join(root, "home");
|
||||
const agentDir = join(isolatedHome, ".pi", "agent");
|
||||
const extensionFactories: InlineExtension[] = [];
|
||||
let forcedSystemPrompt: string | undefined;
|
||||
if (options.transformSystemPrompt) {
|
||||
const transform = options.transformSystemPrompt;
|
||||
extensionFactories.push({
|
||||
name: "eval-system-prompt-transform",
|
||||
hidden: true,
|
||||
factory: (pi) => {
|
||||
pi.on("before_agent_start", ({ systemPrompt }) => {
|
||||
forcedSystemPrompt = transform(systemPrompt);
|
||||
return { systemPrompt: forcedSystemPrompt };
|
||||
});
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
let sessionManager: SessionManager | undefined;
|
||||
let session: AgentSession | undefined;
|
||||
let result: SimpleHarnessResult<string | TOutput> | undefined;
|
||||
let runDiagnostics: PiRunDiagnostics | undefined;
|
||||
let runError: unknown;
|
||||
const cleanupErrors: unknown[] = [];
|
||||
let hiddenCredentialEnvironment: { name: string; value: string } | undefined;
|
||||
const restoreEnvironment = applyIsolatedEnvironment(isolatedHome, agentDir);
|
||||
try {
|
||||
const authPath = join(hostAgentDir, "auth.json");
|
||||
const credentials = new InMemoryCredentialStore();
|
||||
const storedCredential = readStoredCredential(selection.provider, authPath);
|
||||
if (storedCredential) await credentials.modify(selection.provider, async () => storedCredential);
|
||||
const modelRuntime = await ModelRuntime.create({ credentials });
|
||||
await Promise.all([mkdir(workspace), mkdir(agentDir, { recursive: true })]);
|
||||
await seedWorkspace(workspace, options.workspaceFiles);
|
||||
const model = modelRuntime.getModel(selection.provider, selection.id);
|
||||
if (!model) throw new Error(`Eval model not found: ${selection.provider}/${selection.id}`);
|
||||
const auth = await modelRuntime.getAuth(model);
|
||||
if (!auth) {
|
||||
throw new Error(`Eval model has no configured authentication: ${selection.provider}/${selection.id}`);
|
||||
}
|
||||
if (!storedCredential && auth.auth.apiKey) {
|
||||
await modelRuntime.setRuntimeApiKey(selection.provider, auth.auth.apiKey);
|
||||
}
|
||||
if (sandboxIdentity) {
|
||||
await rm(authPath, { force: true });
|
||||
const credentialEnvironmentValue = auth.source ? process.env[auth.source] : undefined;
|
||||
if (auth.source && /^[A-Z][A-Z0-9_]*$/.test(auth.source) && credentialEnvironmentValue) {
|
||||
hiddenCredentialEnvironment = { name: auth.source, value: credentialEnvironmentValue };
|
||||
delete process.env[auth.source];
|
||||
}
|
||||
}
|
||||
|
||||
const services = await createAgentSessionServices({
|
||||
cwd: workspace,
|
||||
modelRuntime,
|
||||
resourceLoaderOptions: { extensionFactories },
|
||||
});
|
||||
signal?.throwIfAborted();
|
||||
sessionManager = SessionManager.create(workspace, join(root, "sessions"));
|
||||
setArtifact("runId", sessionManager.getSessionId());
|
||||
session = (
|
||||
await createAgentSessionFromServices({
|
||||
services,
|
||||
sessionManager,
|
||||
model,
|
||||
thinkingLevel: "off",
|
||||
tools: options.tools,
|
||||
noTools: options.noTools,
|
||||
customTools: options.customTools,
|
||||
})
|
||||
).session;
|
||||
|
||||
const expectedInlinePaths = new Set(extensionFactories.map(({ name }) => `<inline:${name}>`));
|
||||
const unexpectedExtensions = session.extensionRunner
|
||||
.getExtensionPaths()
|
||||
.filter((path) => !expectedInlinePaths.has(path));
|
||||
if (unexpectedExtensions.length > 0) {
|
||||
throw new Error(`Isolated eval loaded unexpected extensions: ${unexpectedExtensions.join(", ")}`);
|
||||
}
|
||||
|
||||
await enterToolSandbox(root, sandboxIdentity);
|
||||
let response: string | undefined;
|
||||
const steps = typeof input === "string" ? [{ type: "prompt" as const, content: input }] : input;
|
||||
let abortPromise: Promise<void> | undefined;
|
||||
const abort = () => {
|
||||
abortPromise ??= session!.abort();
|
||||
};
|
||||
signal?.addEventListener("abort", abort, { once: true });
|
||||
try {
|
||||
for (const step of steps) {
|
||||
if (step.type === "reload") {
|
||||
await session.reload();
|
||||
continue;
|
||||
}
|
||||
response = await promptAgent(session, step.content, signal);
|
||||
}
|
||||
} finally {
|
||||
signal?.removeEventListener("abort", abort);
|
||||
if (abortPromise) await abortPromise;
|
||||
}
|
||||
if (response === undefined) {
|
||||
throw new Error("Pi eval input must include at least one prompt step.");
|
||||
}
|
||||
// A forced prompt is not recorded in the transcript, so use the one the transform
|
||||
// extension sent; otherwise the replayed transcript prompt is what the provider received.
|
||||
const systemPrompt = forcedSystemPrompt ?? getCurrentSystemPrompt(session.messages);
|
||||
const stats = session.getSessionStats();
|
||||
const hasPricing = [model.cost, ...(model.cost.tiers ?? [])].some(
|
||||
({ input: inputCost, output: outputCost, cacheRead, cacheWrite }) =>
|
||||
inputCost > 0 || outputCost > 0 || cacheRead > 0 || cacheWrite > 0,
|
||||
);
|
||||
runDiagnostics = {
|
||||
events: toTranscriptEvents(session.messages),
|
||||
metadata: { systemPromptSha256: createHash("sha256").update(systemPrompt).digest("hex") },
|
||||
usage: {
|
||||
provider: model.provider,
|
||||
model: model.id,
|
||||
inputTokens: stats.tokens.input,
|
||||
outputTokens: stats.tokens.output,
|
||||
totalTokens: stats.tokens.total,
|
||||
toolCalls: stats.toolCalls,
|
||||
metadata: {
|
||||
cacheReadTokens: stats.tokens.cacheRead,
|
||||
cacheWriteTokens: stats.tokens.cacheWrite,
|
||||
...(hasPricing ? { estimatedCostUsd: stats.cost } : {}),
|
||||
},
|
||||
},
|
||||
};
|
||||
verifySystemPrompt(systemPrompt, options);
|
||||
const output =
|
||||
"output" in options ? await options.output({ response, session, systemPrompt, agentDir }) : response;
|
||||
result = { output, ...runDiagnostics };
|
||||
} catch (error) {
|
||||
runError = error;
|
||||
} finally {
|
||||
if (sessionManager) {
|
||||
const sessionPath = sessionManager.getSessionFile();
|
||||
if (!sessionPath || !existsSync(sessionPath)) {
|
||||
cleanupErrors.push(new Error("Pi eval produced no session file."));
|
||||
} else {
|
||||
try {
|
||||
setArtifact(PI_SESSION_SNAPSHOT_ARTIFACT, await readFile(sessionPath, "utf8"));
|
||||
} catch (error) {
|
||||
cleanupErrors.push(error);
|
||||
}
|
||||
}
|
||||
}
|
||||
try {
|
||||
session?.dispose();
|
||||
} catch (error) {
|
||||
cleanupErrors.push(error);
|
||||
}
|
||||
try {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
} catch (error) {
|
||||
cleanupErrors.push(error);
|
||||
}
|
||||
restoreEnvironment();
|
||||
if (hiddenCredentialEnvironment) {
|
||||
process.env[hiddenCredentialEnvironment.name] = hiddenCredentialEnvironment.value;
|
||||
}
|
||||
}
|
||||
|
||||
let failure = runError;
|
||||
if (runError !== undefined && cleanupErrors.length > 0) {
|
||||
failure = new AggregateError([runError, ...cleanupErrors], "Agent run failed and cleanup also failed.");
|
||||
} else if (cleanupErrors.length === 1) {
|
||||
failure = cleanupErrors[0];
|
||||
} else if (cleanupErrors.length > 1) {
|
||||
failure = new AggregateError(cleanupErrors, "Agent cleanup failed.");
|
||||
}
|
||||
if (failure !== undefined) {
|
||||
if (runDiagnostics) {
|
||||
const partialRun = normalizeHarnessRun(input, {
|
||||
...runDiagnostics,
|
||||
errors: [failure],
|
||||
timings: { totalMs: performance.now() - startedAt },
|
||||
});
|
||||
throw attachHarnessRunToError(failure, partialRun);
|
||||
}
|
||||
throw failure;
|
||||
}
|
||||
if (!result) throw new Error("Pi eval completed without a result.");
|
||||
return { ...result, timings: { totalMs: performance.now() - startedAt } };
|
||||
}
|
||||
|
||||
export function createPiCodingAgentHarness<TOutput extends JsonValue>(
|
||||
options: PiCodingAgentHarnessWithOutput<TOutput>,
|
||||
): Harness<PiCodingAgentInput, TOutput>;
|
||||
export function createPiCodingAgentHarness(options?: PiCodingAgentHarnessOptions): Harness<PiCodingAgentInput, string>;
|
||||
export function createPiCodingAgentHarness<TOutput extends JsonValue>(
|
||||
options: PiCodingAgentHarnessOptions | PiCodingAgentHarnessWithOutput<TOutput> = {},
|
||||
): Harness<PiCodingAgentInput, string | TOutput> {
|
||||
return createHarness<PiCodingAgentInput, string | TOutput>({
|
||||
name: options.name ?? "pi-coding-agent",
|
||||
run: ({ input, signal, setArtifact }) => runPiCodingAgent(input, signal, setArtifact, options),
|
||||
});
|
||||
}
|
||||
|
||||
/** Documentation evals intentionally exclude shell and unrestricted network tools. */
|
||||
export const DOCUMENTATION_EVAL_TOOLS = ["read", "write", "edit", "grep", "find", "ls"] as const;
|
||||
|
||||
export function resolveDocumentationVariant(
|
||||
value: string | undefined = process.env.PI_EVAL_VARIANT,
|
||||
): DocumentationVariant {
|
||||
if (value === "without_docs" || value === "with_docs") return value;
|
||||
throw new TypeError('PI_EVAL_VARIANT must be "without_docs" or "with_docs".');
|
||||
}
|
||||
|
||||
export function excludePiDocumentation(defaultPrompt: string): string {
|
||||
const documentationStartMarker = "\n<docs>\n";
|
||||
const documentationEndMarker = "\n</docs>";
|
||||
const documentationStart = defaultPrompt.indexOf(documentationStartMarker);
|
||||
if (documentationStart === -1) throw new Error("Default Pi system prompt has no Pi documentation section.");
|
||||
const documentationEnd = defaultPrompt.indexOf(documentationEndMarker, documentationStart);
|
||||
if (documentationEnd === -1) throw new Error("Default Pi system prompt has no complete Pi documentation section.");
|
||||
const cwdStart = defaultPrompt.lastIndexOf("\n<cwd>\n");
|
||||
if (cwdStart < documentationEnd) throw new Error("Default Pi system prompt has no working-directory section.");
|
||||
return (
|
||||
defaultPrompt.slice(0, documentationStart) + defaultPrompt.slice(documentationEnd + documentationEndMarker.length)
|
||||
);
|
||||
}
|
||||
|
||||
type DocumentationHarnessOptions = Omit<
|
||||
PiCodingAgentHarnessOptions,
|
||||
"name" | "transformSystemPrompt" | "expectedPiDocumentation"
|
||||
>;
|
||||
type DocumentationHarnessWithOutput<TOutput extends JsonValue> = Omit<
|
||||
PiCodingAgentHarnessWithOutput<TOutput>,
|
||||
"name" | "transformSystemPrompt" | "expectedPiDocumentation"
|
||||
>;
|
||||
|
||||
export function createPiDocumentationEvalHarness<TOutput extends JsonValue>(
|
||||
options: DocumentationHarnessWithOutput<TOutput>,
|
||||
): Harness<PiCodingAgentInput, TOutput>;
|
||||
export function createPiDocumentationEvalHarness(
|
||||
options?: DocumentationHarnessOptions,
|
||||
): Harness<PiCodingAgentInput, string>;
|
||||
export function createPiDocumentationEvalHarness<TOutput extends JsonValue>(
|
||||
options: DocumentationHarnessOptions | DocumentationHarnessWithOutput<TOutput> = {},
|
||||
) {
|
||||
if (process.env.PI_EVAL_CONTAINER !== "1" || !resolveSandboxIdentity()) {
|
||||
throw new Error("Documentation evals must run in the isolated container sandbox.");
|
||||
}
|
||||
const variant = resolveDocumentationVariant();
|
||||
return createPiCodingAgentHarness({
|
||||
...options,
|
||||
name: variant,
|
||||
tools: options.tools ?? [...DOCUMENTATION_EVAL_TOOLS],
|
||||
...(variant === "without_docs" ? { transformSystemPrompt: excludePiDocumentation } : {}),
|
||||
expectedPiDocumentation: variant === "with_docs",
|
||||
});
|
||||
}
|
||||
257
third_party/pi/packages/evals/src/pi-harness.ts
vendored
257
third_party/pi/packages/evals/src/pi-harness.ts
vendored
@@ -1,257 +0,0 @@
|
||||
import { existsSync } from "node:fs";
|
||||
import { mkdir, mkdtemp, readFile, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { performance } from "node:perf_hooks";
|
||||
import { contentText } from "@earendil-works/pi-ai";
|
||||
import {
|
||||
type AgentSession,
|
||||
type CreateAgentSessionOptions,
|
||||
createAgentSessionFromServices,
|
||||
createAgentSessionServices,
|
||||
ModelRuntime,
|
||||
SessionManager,
|
||||
SettingsManager,
|
||||
} from "@earendil-works/pi-coding-agent";
|
||||
import {
|
||||
createHarness,
|
||||
type Harness,
|
||||
type HarnessContext,
|
||||
type JsonValue,
|
||||
normalizeRecord,
|
||||
type SimpleHarnessResult,
|
||||
type TranscriptEvent,
|
||||
toJsonValue,
|
||||
} from "vitest-evals/harness";
|
||||
import { PI_SESSION_SNAPSHOT_ARTIFACT } from "./vitest-evals/artifacts.ts";
|
||||
|
||||
export type PiCodingAgentInput = string | Array<{ type: "prompt"; content: string } | { type: "reload" }>;
|
||||
|
||||
type PiCodingAgentModelSelection = {
|
||||
provider: string;
|
||||
id: string;
|
||||
};
|
||||
|
||||
type PiCodingAgentHarnessOptions = {
|
||||
name?: string;
|
||||
model?: PiCodingAgentModelSelection;
|
||||
noTools?: CreateAgentSessionOptions["noTools"];
|
||||
transformSystemPrompt?: (defaultPrompt: string) => string;
|
||||
};
|
||||
|
||||
type PiCodingAgentHarnessWithOutput<TOutput extends JsonValue> = PiCodingAgentHarnessOptions & {
|
||||
output: (args: { response: string; session: AgentSession }) => TOutput | Promise<TOutput>;
|
||||
};
|
||||
|
||||
export function resolveModelSelection(
|
||||
explicitModel: PiCodingAgentModelSelection | undefined,
|
||||
environment: { PI_PROVIDER?: string; PI_MODEL?: string } = process.env,
|
||||
): PiCodingAgentModelSelection {
|
||||
const provider = (explicitModel?.provider ?? environment.PI_PROVIDER)?.trim();
|
||||
const id = (explicitModel?.id ?? environment.PI_MODEL)?.trim();
|
||||
if (!provider || !id) {
|
||||
throw new Error("Select a harness model explicitly or set both PI_PROVIDER and PI_MODEL as defaults.");
|
||||
}
|
||||
return { provider, id };
|
||||
}
|
||||
|
||||
function toTranscriptEvents(messages: AgentSession["messages"]): TranscriptEvent[] {
|
||||
const events: TranscriptEvent[] = [];
|
||||
for (const message of messages) {
|
||||
if (message.role === "user") {
|
||||
events.push({ type: "message", role: "user", content: contentText(message.content) });
|
||||
} else if (message.role === "assistant") {
|
||||
const text = contentText(message.content);
|
||||
if (text) events.push({ type: "message", role: "assistant", content: text });
|
||||
for (const part of message.content) {
|
||||
if (part.type === "toolCall") {
|
||||
events.push({
|
||||
type: "tool_call",
|
||||
id: part.id,
|
||||
name: part.name,
|
||||
arguments: normalizeRecord(part.arguments),
|
||||
});
|
||||
}
|
||||
}
|
||||
} else if (message.role === "toolResult") {
|
||||
const text = contentText(message.content);
|
||||
events.push({
|
||||
type: "tool_result",
|
||||
toolCallId: message.toolCallId,
|
||||
name: message.toolName,
|
||||
content: message.content.every((part) => part.type === "text") ? text : toJsonValue(message.content),
|
||||
...(message.isError ? { error: { message: text || "Tool failed" } } : {}),
|
||||
});
|
||||
}
|
||||
}
|
||||
return events;
|
||||
}
|
||||
|
||||
async function promptAgent(session: AgentSession, input: string, signal: AbortSignal | undefined): Promise<string> {
|
||||
signal?.throwIfAborted();
|
||||
const previousMessageCount = session.messages.length;
|
||||
await session.prompt(input);
|
||||
const assistant = session.messages
|
||||
.slice(previousMessageCount)
|
||||
.reverse()
|
||||
.find((message) => message.role === "assistant");
|
||||
if (!assistant) throw new Error("Agent run completed without an assistant message.");
|
||||
if (assistant.stopReason !== "stop") {
|
||||
throw new Error(
|
||||
assistant.errorMessage ?? `Agent run ended with unexpected stop reason: ${assistant.stopReason}.`,
|
||||
);
|
||||
}
|
||||
const output = session.getLastAssistantText();
|
||||
if (!output) throw new Error("Agent run produced no assistant text.");
|
||||
return output;
|
||||
}
|
||||
|
||||
async function runPiCodingAgent<TOutput extends JsonValue>(
|
||||
input: PiCodingAgentInput,
|
||||
signal: AbortSignal | undefined,
|
||||
setArtifact: HarnessContext["setArtifact"],
|
||||
options: PiCodingAgentHarnessOptions | PiCodingAgentHarnessWithOutput<TOutput>,
|
||||
): Promise<SimpleHarnessResult<string | TOutput>> {
|
||||
const startedAt = performance.now();
|
||||
signal?.throwIfAborted();
|
||||
const selection = resolveModelSelection(options.model);
|
||||
const modelRuntime = await ModelRuntime.create();
|
||||
const model = modelRuntime.getModel(selection.provider, selection.id);
|
||||
if (!model) throw new Error(`Eval model not found: ${selection.provider}/${selection.id}`);
|
||||
|
||||
const root = await mkdtemp(join(tmpdir(), "pi-eval-"));
|
||||
const cwd = join(root, "workspace");
|
||||
const agentDir = join(root, "agent");
|
||||
let transformedSystemPrompt: string | undefined;
|
||||
let sessionManager: SessionManager | undefined;
|
||||
let session: AgentSession | undefined;
|
||||
let outcome: { success: true; result: SimpleHarnessResult<string | TOutput> } | { success: false; error: unknown };
|
||||
try {
|
||||
await Promise.all([mkdir(cwd), mkdir(agentDir)]);
|
||||
const services = await createAgentSessionServices({
|
||||
cwd,
|
||||
agentDir,
|
||||
modelRuntime,
|
||||
settingsManager: SettingsManager.inMemory(),
|
||||
...(options.transformSystemPrompt
|
||||
? { resourceLoaderOptions: { systemPromptOverride: () => transformedSystemPrompt } }
|
||||
: {}),
|
||||
});
|
||||
signal?.throwIfAborted();
|
||||
sessionManager = SessionManager.create(cwd, join(root, "sessions"));
|
||||
setArtifact("runId", sessionManager.getSessionId());
|
||||
session = (
|
||||
await createAgentSessionFromServices({
|
||||
services,
|
||||
sessionManager,
|
||||
model,
|
||||
thinkingLevel: "off",
|
||||
noTools: options.noTools,
|
||||
})
|
||||
).session;
|
||||
|
||||
const evalSession = session;
|
||||
if (options.transformSystemPrompt) {
|
||||
transformedSystemPrompt = options.transformSystemPrompt(evalSession.systemPrompt);
|
||||
if (!transformedSystemPrompt.trim()) throw new Error("Transformed eval system prompt must not be empty.");
|
||||
await evalSession.reload();
|
||||
}
|
||||
let abortPromise: Promise<void> | undefined;
|
||||
const abort = () => {
|
||||
abortPromise ??= evalSession.abort();
|
||||
};
|
||||
signal?.addEventListener("abort", abort, { once: true });
|
||||
try {
|
||||
signal?.throwIfAborted();
|
||||
if (evalSession.extensionRunner.getExtensionPaths().length !== 0) {
|
||||
throw new Error("Expected an isolated eval session to start without extensions.");
|
||||
}
|
||||
const steps = typeof input === "string" ? [{ type: "prompt" as const, content: input }] : input;
|
||||
let response: string | undefined;
|
||||
for (const step of steps) {
|
||||
if (step.type === "prompt") {
|
||||
response = await promptAgent(evalSession, step.content, signal);
|
||||
} else {
|
||||
await evalSession.reload();
|
||||
}
|
||||
}
|
||||
if (response === undefined) throw new Error("Pi eval input must include at least one prompt step.");
|
||||
const output = "output" in options ? await options.output({ response, session: evalSession }) : response;
|
||||
const stats = evalSession.getSessionStats();
|
||||
const hasPricing = [model.cost, ...(model.cost.tiers ?? [])].some(
|
||||
({ input, output, cacheRead, cacheWrite }) => input > 0 || output > 0 || cacheRead > 0 || cacheWrite > 0,
|
||||
);
|
||||
outcome = {
|
||||
success: true,
|
||||
result: {
|
||||
output,
|
||||
events: toTranscriptEvents(evalSession.messages),
|
||||
usage: {
|
||||
provider: model.provider,
|
||||
model: model.id,
|
||||
inputTokens: stats.tokens.input,
|
||||
outputTokens: stats.tokens.output,
|
||||
totalTokens: stats.tokens.total,
|
||||
toolCalls: stats.toolCalls,
|
||||
metadata: {
|
||||
cacheReadTokens: stats.tokens.cacheRead,
|
||||
cacheWriteTokens: stats.tokens.cacheWrite,
|
||||
...(hasPricing ? { estimatedCostUsd: stats.cost } : {}),
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
} finally {
|
||||
signal?.removeEventListener("abort", abort);
|
||||
if (abortPromise) await abortPromise;
|
||||
}
|
||||
} catch (error) {
|
||||
outcome = { success: false, error };
|
||||
}
|
||||
|
||||
const cleanupErrors: unknown[] = [];
|
||||
if (sessionManager) {
|
||||
try {
|
||||
const sessionPath = sessionManager.getSessionFile();
|
||||
if (sessionPath && existsSync(sessionPath)) {
|
||||
setArtifact(PI_SESSION_SNAPSHOT_ARTIFACT, await readFile(sessionPath, "utf8"));
|
||||
}
|
||||
} catch (error) {
|
||||
cleanupErrors.push(error);
|
||||
}
|
||||
}
|
||||
try {
|
||||
session?.dispose();
|
||||
} catch (error) {
|
||||
cleanupErrors.push(error);
|
||||
}
|
||||
try {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
} catch (error) {
|
||||
cleanupErrors.push(error);
|
||||
}
|
||||
|
||||
if (!outcome.success) {
|
||||
if (cleanupErrors.length === 0) throw outcome.error;
|
||||
throw new AggregateError([outcome.error, ...cleanupErrors], "Agent run failed and cleanup also failed.");
|
||||
}
|
||||
if (cleanupErrors.length === 1) throw cleanupErrors[0];
|
||||
if (cleanupErrors.length > 1) throw new AggregateError(cleanupErrors, "Agent cleanup failed.");
|
||||
return {
|
||||
...outcome.result,
|
||||
timings: { totalMs: performance.now() - startedAt },
|
||||
};
|
||||
}
|
||||
|
||||
export function createPiCodingAgentHarness<TOutput extends JsonValue>(
|
||||
options: PiCodingAgentHarnessWithOutput<TOutput>,
|
||||
): Harness<PiCodingAgentInput, TOutput>;
|
||||
export function createPiCodingAgentHarness(options?: PiCodingAgentHarnessOptions): Harness<PiCodingAgentInput, string>;
|
||||
export function createPiCodingAgentHarness<TOutput extends JsonValue>(
|
||||
options: PiCodingAgentHarnessOptions | PiCodingAgentHarnessWithOutput<TOutput> = {},
|
||||
) {
|
||||
return createHarness<PiCodingAgentInput, string | TOutput>({
|
||||
name: options.name ?? "pi-coding-agent",
|
||||
run: ({ input, signal, setArtifact }) => runPiCodingAgent(input, signal, setArtifact, options),
|
||||
});
|
||||
}
|
||||
59
third_party/pi/packages/evals/src/plan.ts
vendored
Normal file
59
third_party/pi/packages/evals/src/plan.ts
vendored
Normal file
@@ -0,0 +1,59 @@
|
||||
export const DOCUMENTATION_VARIANTS = ["without_docs", "with_docs"] as const;
|
||||
export type DocumentationVariant = (typeof DOCUMENTATION_VARIANTS)[number];
|
||||
|
||||
export type DiscoveredEvalCase = {
|
||||
file: string;
|
||||
fullName: string;
|
||||
evalSet: string;
|
||||
caseId: string;
|
||||
};
|
||||
|
||||
export type EvalTask = DiscoveredEvalCase & {
|
||||
variant: DocumentationVariant;
|
||||
model: string;
|
||||
runNumber: number;
|
||||
};
|
||||
|
||||
function isRecord(value: unknown): value is Record<string, unknown> {
|
||||
return typeof value === "object" && value !== null && !Array.isArray(value);
|
||||
}
|
||||
|
||||
export function parseDiscoveredCases(value: unknown): DiscoveredEvalCase[] {
|
||||
if (!Array.isArray(value)) throw new TypeError("Discovered eval cases must be an array.");
|
||||
const identities = new Set<string>();
|
||||
return value.map((item) => {
|
||||
if (!isRecord(item) || typeof item.name !== "string" || typeof item.file !== "string") {
|
||||
throw new TypeError("Discovered eval case is invalid.");
|
||||
}
|
||||
const [evalSet, caseId, ...extra] = item.name.split(" > ");
|
||||
if (!evalSet?.trim() || !caseId?.trim() || extra.length > 0) {
|
||||
throw new TypeError(`Documentation eval must use "<eval set> > <case>": ${item.name}`);
|
||||
}
|
||||
const identity = JSON.stringify([evalSet, caseId]);
|
||||
if (identities.has(identity)) throw new TypeError(`Duplicate eval case identity: ${item.name}`);
|
||||
identities.add(identity);
|
||||
return { file: item.file, fullName: item.name, evalSet, caseId };
|
||||
});
|
||||
}
|
||||
|
||||
export function createTaskPlan(
|
||||
cases: readonly DiscoveredEvalCase[],
|
||||
model: string,
|
||||
runsPerVariant: number,
|
||||
): EvalTask[] {
|
||||
if (!model.includes("/") || model.startsWith("/") || model.endsWith("/")) {
|
||||
throw new TypeError("Model identity must contain a provider and model.");
|
||||
}
|
||||
if (!Number.isSafeInteger(runsPerVariant) || runsPerVariant < 1) {
|
||||
throw new TypeError("Runs per variant must be a positive integer.");
|
||||
}
|
||||
const tasks: EvalTask[] = [];
|
||||
for (const evalCase of cases) {
|
||||
for (let runNumber = 1; runNumber <= runsPerVariant; runNumber += 1) {
|
||||
const variants: DocumentationVariant[] =
|
||||
runNumber % 2 === 1 ? ["without_docs", "with_docs"] : ["with_docs", "without_docs"];
|
||||
for (const variant of variants) tasks.push({ ...evalCase, variant, model, runNumber });
|
||||
}
|
||||
}
|
||||
return tasks;
|
||||
}
|
||||
483
third_party/pi/packages/evals/src/report.ts
vendored
Normal file
483
third_party/pi/packages/evals/src/report.ts
vendored
Normal file
@@ -0,0 +1,483 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { styleText } from "node:util";
|
||||
import type { ReportCase } from "@vitest-evals/core";
|
||||
import { readReportWorkspace, readVitestJsonReportFile } from "@vitest-evals/core/node";
|
||||
import type { DocumentationVariant, EvalTask } from "./plan.ts";
|
||||
|
||||
export const PI_SESSION_SNAPSHOT_ARTIFACT = "piSessionJsonl";
|
||||
|
||||
export type EvalRunIdentity = {
|
||||
evalSet: string;
|
||||
caseId: string;
|
||||
variant: DocumentationVariant;
|
||||
model: string;
|
||||
runNumber: number;
|
||||
};
|
||||
|
||||
export type ExpectedEvalRun = EvalRunIdentity;
|
||||
|
||||
type EvalMetrics = {
|
||||
inputTokens?: number;
|
||||
outputTokens?: number;
|
||||
cacheReadTokens?: number;
|
||||
cacheWriteTokens?: number;
|
||||
totalTokens?: number;
|
||||
toolCalls?: number;
|
||||
totalMs?: number;
|
||||
estimatedCostUsd?: number;
|
||||
};
|
||||
|
||||
export type EvalObservation = EvalRunIdentity &
|
||||
EvalMetrics &
|
||||
({ outcome: "scored"; score: number } | { outcome: "unscored" | "skipped" | "pending" | "errored" });
|
||||
|
||||
export type PairedMetricSummary = {
|
||||
eligiblePairs: number;
|
||||
controlMean: number | null;
|
||||
treatmentMean: number | null;
|
||||
meanDelta: number | null;
|
||||
};
|
||||
|
||||
export type EvalSetComparison = {
|
||||
evalSet: string;
|
||||
totalPairs: number;
|
||||
eligiblePairs: number;
|
||||
blockedPairs: number;
|
||||
controlPassRate: number | null;
|
||||
treatmentPassRate: number | null;
|
||||
lift: number | null;
|
||||
flags: Array<"no-lift" | "negative-delta" | "control-saturated" | "treatment-saturated" | "flaky">;
|
||||
totalTokens: PairedMetricSummary;
|
||||
toolCalls: PairedMetricSummary;
|
||||
totalMs: PairedMetricSummary;
|
||||
estimatedCostUsd: PairedMetricSummary;
|
||||
};
|
||||
|
||||
export type BlockedPair = Omit<EvalRunIdentity, "variant"> & { reasons: string[] };
|
||||
|
||||
export type OperationalMetricTotal = { availableRuns: number; total: number | null };
|
||||
|
||||
export type VariantTotals = {
|
||||
variant: DocumentationVariant;
|
||||
runs: number;
|
||||
inputTokens: OperationalMetricTotal;
|
||||
outputTokens: OperationalMetricTotal;
|
||||
cacheReadTokens: OperationalMetricTotal;
|
||||
cacheWriteTokens: OperationalMetricTotal;
|
||||
totalTokens: OperationalMetricTotal;
|
||||
toolCalls: OperationalMetricTotal;
|
||||
totalMs: OperationalMetricTotal;
|
||||
estimatedCostUsd: OperationalMetricTotal;
|
||||
};
|
||||
|
||||
export type EvalComparisonReport = {
|
||||
schemaVersion: 3;
|
||||
protocolDigest: string;
|
||||
control: "without_docs";
|
||||
treatment: "with_docs";
|
||||
comparisons: EvalSetComparison[];
|
||||
blockedPairs: BlockedPair[];
|
||||
operationalTotals: VariantTotals[];
|
||||
};
|
||||
|
||||
function optionalMetric(value: unknown, name: string): number | undefined {
|
||||
if (value === undefined) return undefined;
|
||||
if (typeof value !== "number" || !Number.isFinite(value) || value < 0) {
|
||||
throw new TypeError(`${name} must be a finite non-negative number.`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function validateScore(value: unknown): number | undefined {
|
||||
if (value === null || value === undefined) return undefined;
|
||||
if (typeof value !== "number" || !Number.isFinite(value) || value < 0 || value > 1) {
|
||||
throw new TypeError("Eval score must be between 0 and 1.");
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
export function classifyCaseStatus(status: ReportCase["status"]): "errored" | "skipped" | "pending" | undefined {
|
||||
if (status === "failed") return "errored";
|
||||
if (status === "skipped" || status === "todo" || status === "disabled") return "skipped";
|
||||
if (status === "pending") return "pending";
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function taskIdentity(task: EvalTask): EvalRunIdentity {
|
||||
return {
|
||||
evalSet: task.evalSet,
|
||||
caseId: task.caseId,
|
||||
variant: task.variant,
|
||||
model: task.model,
|
||||
runNumber: task.runNumber,
|
||||
};
|
||||
}
|
||||
|
||||
export function erroredObservation(task: EvalTask): EvalObservation {
|
||||
return { ...taskIdentity(task), outcome: "errored" };
|
||||
}
|
||||
|
||||
async function persistSession(caseResult: ReportCase, task: EvalTask, artifactDirectory: string): Promise<void> {
|
||||
const session = caseResult.harness?.run?.artifacts?.[PI_SESSION_SNAPSHOT_ARTIFACT];
|
||||
if (typeof session !== "string") return;
|
||||
const identity = JSON.stringify([task.evalSet, task.caseId, task.variant, task.model, task.runNumber]);
|
||||
const directory = join(
|
||||
artifactDirectory,
|
||||
task.variant,
|
||||
"sessions",
|
||||
createHash("sha256").update(identity).digest("hex"),
|
||||
);
|
||||
await mkdir(directory, { recursive: true, mode: 0o700 });
|
||||
await writeFile(join(directory, "session.jsonl"), session, { mode: 0o600 });
|
||||
}
|
||||
|
||||
export async function readTaskObservation(
|
||||
task: EvalTask,
|
||||
reportPath: string,
|
||||
artifactDirectory: string,
|
||||
): Promise<EvalObservation> {
|
||||
const identity = taskIdentity(task);
|
||||
const loaded = await Promise.all([readReportWorkspace([reportPath]), readVitestJsonReportFile(reportPath)]).catch(
|
||||
() => undefined,
|
||||
);
|
||||
if (!loaded) return { ...identity, outcome: "errored" };
|
||||
const [{ workspace }, rawReport] = loaded;
|
||||
const reportedFullName = `${task.evalSet} ${task.caseId}`;
|
||||
const assertions = rawReport.testResults.flatMap(({ assertionResults }) => assertionResults);
|
||||
if (assertions.length !== 1) return { ...identity, outcome: "errored" };
|
||||
const assertion = assertions[0];
|
||||
if (assertion.fullName !== reportedFullName) return { ...identity, outcome: "errored" };
|
||||
const statusOutcome = classifyCaseStatus(assertion.status);
|
||||
if (statusOutcome === "skipped" || statusOutcome === "pending") return { ...identity, outcome: statusOutcome };
|
||||
if (workspace.cases.length !== 1) return { ...identity, outcome: "errored" };
|
||||
const caseResult = workspace.cases[0];
|
||||
if (caseResult.fullName !== reportedFullName) return { ...identity, outcome: "errored" };
|
||||
if (caseResult.status !== assertion.status) return { ...identity, outcome: "errored" };
|
||||
const run = caseResult.harness?.run;
|
||||
if (!run) return { ...identity, outcome: "errored" };
|
||||
await persistSession(caseResult, task, artifactDirectory);
|
||||
const actualModel = run.usage.provider && run.usage.model ? `${run.usage.provider}/${run.usage.model}` : undefined;
|
||||
if (actualModel !== task.model) return { ...identity, outcome: "errored" };
|
||||
let metrics: EvalMetrics = {};
|
||||
try {
|
||||
metrics = {
|
||||
inputTokens: optionalMetric(run.usage.inputTokens, "inputTokens"),
|
||||
outputTokens: optionalMetric(run.usage.outputTokens, "outputTokens"),
|
||||
cacheReadTokens: optionalMetric(run.usage.metadata?.cacheReadTokens, "cacheReadTokens"),
|
||||
cacheWriteTokens: optionalMetric(run.usage.metadata?.cacheWriteTokens, "cacheWriteTokens"),
|
||||
totalTokens: optionalMetric(run.usage.totalTokens, "totalTokens"),
|
||||
toolCalls: optionalMetric(run.usage.toolCalls, "toolCalls"),
|
||||
totalMs: optionalMetric(run.timings?.totalMs, "totalMs"),
|
||||
estimatedCostUsd: optionalMetric(run.usage.metadata?.estimatedCostUsd, "estimatedCostUsd"),
|
||||
};
|
||||
} catch {
|
||||
return { ...identity, outcome: "errored" };
|
||||
}
|
||||
if (statusOutcome === "errored" || run.errors.length > 0) {
|
||||
return { ...identity, ...metrics, outcome: "errored" };
|
||||
}
|
||||
let score: number | undefined;
|
||||
try {
|
||||
score = validateScore(caseResult.eval?.avgScore);
|
||||
} catch {
|
||||
return { ...identity, ...metrics, outcome: "errored" };
|
||||
}
|
||||
if (score === undefined) return { ...identity, ...metrics, outcome: "unscored" };
|
||||
return { ...identity, ...metrics, outcome: "scored", score };
|
||||
}
|
||||
|
||||
const CONTROL = "without_docs";
|
||||
const TREATMENT = "with_docs";
|
||||
const VARIANTS = [CONTROL, TREATMENT] as const;
|
||||
|
||||
type Pair = { control: EvalObservation & { outcome: "scored" }; treatment: EvalObservation & { outcome: "scored" } };
|
||||
type PairGroup = Omit<EvalRunIdentity, "variant"> & {
|
||||
expected: Map<DocumentationVariant, number>;
|
||||
observations: Map<DocumentationVariant, EvalObservation[]>;
|
||||
};
|
||||
|
||||
function mean(values: readonly number[]): number | null {
|
||||
return values.length === 0 ? null : values.reduce((sum, value) => sum + value, 0) / values.length;
|
||||
}
|
||||
|
||||
function difference(treatment: number, control: number): number {
|
||||
return Number((treatment - control).toPrecision(15));
|
||||
}
|
||||
|
||||
function pairKey(identity: Omit<EvalRunIdentity, "variant">): string {
|
||||
return JSON.stringify([identity.evalSet, identity.caseId, identity.model, identity.runNumber]);
|
||||
}
|
||||
|
||||
function groupPairs(expectedRuns: readonly ExpectedEvalRun[], observations: readonly EvalObservation[]): PairGroup[] {
|
||||
const groups = new Map<string, PairGroup>();
|
||||
const getGroup = (identity: EvalRunIdentity): PairGroup => {
|
||||
const key = pairKey(identity);
|
||||
const existing = groups.get(key);
|
||||
if (existing) return existing;
|
||||
const group = {
|
||||
evalSet: identity.evalSet,
|
||||
caseId: identity.caseId,
|
||||
model: identity.model,
|
||||
runNumber: identity.runNumber,
|
||||
expected: new Map<DocumentationVariant, number>(),
|
||||
observations: new Map<DocumentationVariant, EvalObservation[]>(),
|
||||
};
|
||||
groups.set(key, group);
|
||||
return group;
|
||||
};
|
||||
for (const expected of expectedRuns) {
|
||||
const group = getGroup(expected);
|
||||
group.expected.set(expected.variant, (group.expected.get(expected.variant) ?? 0) + 1);
|
||||
}
|
||||
for (const observation of observations) {
|
||||
const group = getGroup(observation);
|
||||
const runs = group.observations.get(observation.variant) ?? [];
|
||||
runs.push(observation);
|
||||
group.observations.set(observation.variant, runs);
|
||||
}
|
||||
return [...groups.values()].sort(
|
||||
(left, right) =>
|
||||
left.evalSet.localeCompare(right.evalSet) ||
|
||||
left.caseId.localeCompare(right.caseId) ||
|
||||
left.model.localeCompare(right.model) ||
|
||||
left.runNumber - right.runNumber,
|
||||
);
|
||||
}
|
||||
|
||||
function resolvePair(group: PairGroup): { pair?: Pair; blocked?: BlockedPair } {
|
||||
const reasons: string[] = [];
|
||||
for (const variant of VARIANTS) {
|
||||
const expected = group.expected.get(variant) ?? 0;
|
||||
const observed = group.observations.get(variant) ?? [];
|
||||
if (expected !== 1) reasons.push(`${variant}: design expected 1 run, found ${expected}`);
|
||||
if (observed.length !== expected) {
|
||||
reasons.push(
|
||||
`${variant}: expected ${expected} observation${expected === 1 ? "" : "s"}, found ${observed.length}`,
|
||||
);
|
||||
}
|
||||
if (expected === 1 && observed.length === 1 && observed[0].outcome !== "scored") {
|
||||
reasons.push(`${variant}: ${observed[0].outcome}`);
|
||||
}
|
||||
}
|
||||
if (reasons.length > 0) {
|
||||
return {
|
||||
blocked: {
|
||||
evalSet: group.evalSet,
|
||||
caseId: group.caseId,
|
||||
model: group.model,
|
||||
runNumber: group.runNumber,
|
||||
reasons,
|
||||
},
|
||||
};
|
||||
}
|
||||
return {
|
||||
pair: {
|
||||
control: group.observations.get(CONTROL)![0] as EvalObservation & { outcome: "scored" },
|
||||
treatment: group.observations.get(TREATMENT)![0] as EvalObservation & { outcome: "scored" },
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function summarizeMetric(
|
||||
pairs: readonly Pair[],
|
||||
select: (observation: EvalObservation) => number | undefined,
|
||||
): PairedMetricSummary {
|
||||
const control: number[] = [];
|
||||
const treatment: number[] = [];
|
||||
for (const pair of pairs) {
|
||||
const controlValue = select(pair.control);
|
||||
const treatmentValue = select(pair.treatment);
|
||||
if (controlValue === undefined || treatmentValue === undefined) continue;
|
||||
control.push(controlValue);
|
||||
treatment.push(treatmentValue);
|
||||
}
|
||||
const controlMean = mean(control);
|
||||
const treatmentMean = mean(treatment);
|
||||
return {
|
||||
eligiblePairs: control.length,
|
||||
controlMean,
|
||||
treatmentMean,
|
||||
meanDelta: controlMean === null || treatmentMean === null ? null : difference(treatmentMean, controlMean),
|
||||
};
|
||||
}
|
||||
|
||||
function operationalTotal(
|
||||
runs: readonly EvalObservation[],
|
||||
select: (observation: EvalObservation) => number | undefined,
|
||||
): OperationalMetricTotal {
|
||||
const values = runs.flatMap((run) => {
|
||||
const value = select(run);
|
||||
return value === undefined ? [] : [value];
|
||||
});
|
||||
return {
|
||||
availableRuns: values.length,
|
||||
total: values.length === 0 ? null : values.reduce((sum, value) => sum + value, 0),
|
||||
};
|
||||
}
|
||||
|
||||
function variantTotals(observations: readonly EvalObservation[], variant: DocumentationVariant): VariantTotals {
|
||||
const runs = observations.filter((observation) => observation.variant === variant);
|
||||
return {
|
||||
variant,
|
||||
runs: runs.length,
|
||||
inputTokens: operationalTotal(runs, ({ inputTokens }) => inputTokens),
|
||||
outputTokens: operationalTotal(runs, ({ outputTokens }) => outputTokens),
|
||||
cacheReadTokens: operationalTotal(runs, ({ cacheReadTokens }) => cacheReadTokens),
|
||||
cacheWriteTokens: operationalTotal(runs, ({ cacheWriteTokens }) => cacheWriteTokens),
|
||||
totalTokens: operationalTotal(runs, ({ totalTokens }) => totalTokens),
|
||||
toolCalls: operationalTotal(runs, ({ toolCalls }) => toolCalls),
|
||||
totalMs: operationalTotal(runs, ({ totalMs }) => totalMs),
|
||||
estimatedCostUsd: operationalTotal(runs, ({ estimatedCostUsd }) => estimatedCostUsd),
|
||||
};
|
||||
}
|
||||
|
||||
function comparisonFlags(pairs: readonly Pair[], controlPassRate: number | null, treatmentPassRate: number | null) {
|
||||
const flags: Array<"no-lift" | "negative-delta" | "control-saturated" | "treatment-saturated" | "flaky"> = [];
|
||||
if (controlPassRate !== null && treatmentPassRate !== null) {
|
||||
if (controlPassRate === treatmentPassRate) flags.push("no-lift");
|
||||
if (treatmentPassRate < controlPassRate) flags.push("negative-delta");
|
||||
if (controlPassRate === 1) flags.push("control-saturated");
|
||||
if (treatmentPassRate === 1) flags.push("treatment-saturated");
|
||||
}
|
||||
const outcomes = new Map<string, Set<string>>();
|
||||
for (const pair of pairs) {
|
||||
const key = JSON.stringify([pair.control.caseId, pair.control.variant]);
|
||||
const control = outcomes.get(key) ?? new Set<string>();
|
||||
control.add(String(pair.control.score >= 1));
|
||||
outcomes.set(key, control);
|
||||
const treatmentKey = JSON.stringify([pair.treatment.caseId, pair.treatment.variant]);
|
||||
const treatment = outcomes.get(treatmentKey) ?? new Set<string>();
|
||||
treatment.add(String(pair.treatment.score >= 1));
|
||||
outcomes.set(treatmentKey, treatment);
|
||||
}
|
||||
if ([...outcomes.values()].some((values) => values.size > 1)) flags.push("flaky");
|
||||
return flags;
|
||||
}
|
||||
|
||||
export function summarizeEvalObservations(
|
||||
protocolDigest: string,
|
||||
expectedRuns: readonly ExpectedEvalRun[],
|
||||
observations: readonly EvalObservation[],
|
||||
): EvalComparisonReport {
|
||||
const groups = groupPairs(expectedRuns, observations);
|
||||
const blockedPairs: BlockedPair[] = [];
|
||||
const pairsByEvalSet = new Map<string, Pair[]>();
|
||||
const totalsByEvalSet = new Map<string, number>();
|
||||
for (const group of groups) {
|
||||
totalsByEvalSet.set(group.evalSet, (totalsByEvalSet.get(group.evalSet) ?? 0) + 1);
|
||||
const { pair, blocked } = resolvePair(group);
|
||||
if (blocked) blockedPairs.push(blocked);
|
||||
if (pair) pairsByEvalSet.set(group.evalSet, [...(pairsByEvalSet.get(group.evalSet) ?? []), pair]);
|
||||
}
|
||||
const comparisons = [...totalsByEvalSet]
|
||||
.sort(([left], [right]) => left.localeCompare(right))
|
||||
.map(([evalSet, totalPairs]) => {
|
||||
const pairs = pairsByEvalSet.get(evalSet) ?? [];
|
||||
const blockedPairCount = totalPairs - pairs.length;
|
||||
const publishHeadline = blockedPairCount === 0 && pairs.length > 0;
|
||||
const controlPassRate = publishHeadline
|
||||
? pairs.filter(({ control }) => control.score >= 1).length / pairs.length
|
||||
: null;
|
||||
const treatmentPassRate = publishHeadline
|
||||
? pairs.filter(({ treatment }) => treatment.score >= 1).length / pairs.length
|
||||
: null;
|
||||
return {
|
||||
evalSet,
|
||||
totalPairs,
|
||||
eligiblePairs: pairs.length,
|
||||
blockedPairs: blockedPairCount,
|
||||
controlPassRate,
|
||||
treatmentPassRate,
|
||||
lift:
|
||||
controlPassRate === null || treatmentPassRate === null
|
||||
? null
|
||||
: difference(treatmentPassRate, controlPassRate),
|
||||
flags: comparisonFlags(pairs, controlPassRate, treatmentPassRate),
|
||||
totalTokens: summarizeMetric(pairs, ({ totalTokens }) => totalTokens),
|
||||
toolCalls: summarizeMetric(pairs, ({ toolCalls }) => toolCalls),
|
||||
totalMs: summarizeMetric(pairs, ({ totalMs }) => totalMs),
|
||||
estimatedCostUsd: summarizeMetric(pairs, ({ estimatedCostUsd }) => estimatedCostUsd),
|
||||
};
|
||||
});
|
||||
return {
|
||||
schemaVersion: 3,
|
||||
protocolDigest,
|
||||
control: CONTROL,
|
||||
treatment: TREATMENT,
|
||||
comparisons,
|
||||
blockedPairs,
|
||||
operationalTotals: [variantTotals(observations, CONTROL), variantTotals(observations, TREATMENT)],
|
||||
};
|
||||
}
|
||||
|
||||
function percentage(value: number | null): string {
|
||||
return value === null ? "unavailable" : `${(value * 100).toFixed(1)}%`;
|
||||
}
|
||||
|
||||
function signed(value: number, digits: number): string {
|
||||
return `${value >= 0 ? "+" : ""}${value.toFixed(digits)}`;
|
||||
}
|
||||
|
||||
function pairedMetric(label: string, metric: PairedMetricSummary, unit = ""): string {
|
||||
if (metric.meanDelta === null || metric.controlMean === null || metric.treatmentMean === null) {
|
||||
return ` ${label.padStart(10)} unavailable`;
|
||||
}
|
||||
return ` ${label.padStart(10)} ${signed(metric.meanDelta, 1)}${unit} (with ${metric.treatmentMean.toFixed(1)}${unit}, without ${metric.controlMean.toFixed(1)}${unit}, ${metric.eligiblePairs} pairs)`;
|
||||
}
|
||||
|
||||
function operationalMetric(metric: OperationalMetricTotal, runs: number, format: (total: number) => string): string {
|
||||
if (metric.total === null) return `unavailable (0/${runs} measured)`;
|
||||
const coverage = metric.availableRuns === runs ? "" : ` (${metric.availableRuns}/${runs} measured)`;
|
||||
return `${format(metric.total)}${coverage}`;
|
||||
}
|
||||
|
||||
export function formatEvalComparisonReport(report: EvalComparisonReport): string {
|
||||
if (report.comparisons.length === 0) return "";
|
||||
const lines = [styleText("bold", "Documentation Eval Comparisons")];
|
||||
for (const comparison of report.comparisons) {
|
||||
lines.push(` ${comparison.evalSet}`);
|
||||
lines.push(` Pairs ${comparison.eligiblePairs}/${comparison.totalPairs} eligible`);
|
||||
if (comparison.lift === null) {
|
||||
lines.push(
|
||||
comparison.blockedPairs > 0
|
||||
? " Pass rate withheld because pairs are blocked"
|
||||
: " Pass rate unavailable",
|
||||
);
|
||||
} else {
|
||||
lines.push(
|
||||
` Pass rate ${signed(comparison.lift * 100, 1)} pp (with ${percentage(comparison.treatmentPassRate)}, without ${percentage(comparison.controlPassRate)})`,
|
||||
);
|
||||
}
|
||||
if (comparison.flags.length > 0) lines.push(` Flags ${comparison.flags.join(", ")}`);
|
||||
lines.push(pairedMetric("Tokens", comparison.totalTokens));
|
||||
lines.push(pairedMetric("Tools", comparison.toolCalls));
|
||||
lines.push(pairedMetric("Latency", comparison.totalMs, "ms"));
|
||||
const cost = comparison.estimatedCostUsd;
|
||||
if (cost.meanDelta === null || cost.controlMean === null || cost.treatmentMean === null) {
|
||||
lines.push(" Est. cost unavailable");
|
||||
} else {
|
||||
lines.push(
|
||||
` Est. cost ${cost.meanDelta >= 0 ? "+" : "-"}$${Math.abs(cost.meanDelta).toFixed(4)} (with $${cost.treatmentMean.toFixed(4)}, without $${cost.controlMean.toFixed(4)}, ${cost.eligiblePairs} pairs)`,
|
||||
);
|
||||
}
|
||||
}
|
||||
lines.push(" Operational totals");
|
||||
for (const totals of report.operationalTotals) {
|
||||
const tokens = operationalMetric(totals.totalTokens, totals.runs, (total) => `${total} tokens`);
|
||||
const tools = operationalMetric(totals.toolCalls, totals.runs, (total) => `${total} tools`);
|
||||
const latency = operationalMetric(totals.totalMs, totals.runs, (total) => `${(total / 1000).toFixed(2)}s`);
|
||||
const cost = operationalMetric(totals.estimatedCostUsd, totals.runs, (total) => `$${total.toFixed(4)} cost`);
|
||||
lines.push(` ${totals.variant}: ${totals.runs} runs, ${tokens}, ${tools}, ${latency}, ${cost}`);
|
||||
}
|
||||
if (report.blockedPairs.length > 0) {
|
||||
lines.push(" Blocked pairs");
|
||||
for (const blocked of report.blockedPairs) {
|
||||
lines.push(
|
||||
` ${blocked.evalSet}/${blocked.caseId}/${blocked.model}/run-${blocked.runNumber}: ${blocked.reasons.join("; ")}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
return lines.join("\n");
|
||||
}
|
||||
17
third_party/pi/packages/evals/src/smoke.eval.ts
vendored
17
third_party/pi/packages/evals/src/smoke.eval.ts
vendored
@@ -1,17 +0,0 @@
|
||||
import { expect } from "vitest";
|
||||
import { describeEval } from "vitest-evals";
|
||||
import { createPiCodingAgentHarness } from "./pi-harness.ts";
|
||||
|
||||
const piCodingAgentHarness = createPiCodingAgentHarness({ noTools: "all" });
|
||||
|
||||
describeEval("Pi Coding Agent smoke", { harness: piCodingAgentHarness }, (it) => {
|
||||
it("runs a basic prompt end to end", async ({ run }) => {
|
||||
const result = await run("What's the capital of France? Respond with only the city name.");
|
||||
|
||||
expect(result.output.trim()).toBe("Paris");
|
||||
expect(result.errors).toEqual([]);
|
||||
expect(result.usage.provider).toBe(process.env.PI_PROVIDER);
|
||||
expect(result.usage.model).toBe(process.env.PI_MODEL);
|
||||
expect(result.usage.totalTokens).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
@@ -1,113 +0,0 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import { basename, join, relative } from "node:path";
|
||||
import {
|
||||
type RunnerTestCase,
|
||||
recordArtifact,
|
||||
type TestArtifact,
|
||||
type TestArtifactBase,
|
||||
type TestAttachment,
|
||||
} from "vitest";
|
||||
import type { HarnessRun } from "vitest-evals/harness";
|
||||
|
||||
export const PI_SESSION_SNAPSHOT_ARTIFACT = "piSessionJsonl";
|
||||
|
||||
const evalSessionArtifactKey = Symbol("pi-evals-session-artifact");
|
||||
const evalSourceArtifactKey = Symbol("pi-evals-source-artifact");
|
||||
|
||||
interface PiSessionAttachment extends TestAttachment {
|
||||
name: "session.jsonl";
|
||||
contentType: "application/jsonl";
|
||||
body: string;
|
||||
bodyEncoding: "utf-8";
|
||||
}
|
||||
|
||||
export interface SourceAttachment extends TestAttachment {
|
||||
name: string;
|
||||
contentType: string;
|
||||
body: string;
|
||||
bodyEncoding: "utf-8";
|
||||
}
|
||||
|
||||
interface PiSessionArtifact extends TestArtifactBase {
|
||||
type: "@earendil-works/pi-evals:session";
|
||||
runId: string;
|
||||
attachments: [PiSessionAttachment] | [];
|
||||
}
|
||||
|
||||
interface SourceArtifact extends TestArtifactBase {
|
||||
type: "@earendil-works/pi-evals:source";
|
||||
runId: string;
|
||||
attachments: [SourceAttachment] | [];
|
||||
}
|
||||
|
||||
declare module "vitest" {
|
||||
interface TestArtifactRegistry {
|
||||
[evalSessionArtifactKey]: PiSessionArtifact;
|
||||
[evalSourceArtifactKey]: SourceArtifact;
|
||||
}
|
||||
}
|
||||
|
||||
export async function recordEvalSessionArtifact(
|
||||
task: Readonly<RunnerTestCase>,
|
||||
run: Pick<HarnessRun, "artifacts">,
|
||||
): Promise<void> {
|
||||
const runId = run.artifacts?.runId;
|
||||
const session = run.artifacts?.[PI_SESSION_SNAPSHOT_ARTIFACT];
|
||||
if (session === undefined) return;
|
||||
if (typeof runId !== "string" || typeof session !== "string") {
|
||||
throw new TypeError("Pi eval session artifact metadata is invalid.");
|
||||
}
|
||||
await recordArtifact(task, {
|
||||
type: "@earendil-works/pi-evals:session",
|
||||
runId,
|
||||
attachments: [
|
||||
{
|
||||
name: "session.jsonl",
|
||||
contentType: "application/jsonl",
|
||||
body: session,
|
||||
bodyEncoding: "utf-8",
|
||||
},
|
||||
],
|
||||
});
|
||||
}
|
||||
|
||||
export async function recordEvalSourceArtifact(
|
||||
task: Readonly<RunnerTestCase>,
|
||||
runId: string,
|
||||
attachment: SourceAttachment,
|
||||
): Promise<void> {
|
||||
await recordArtifact(task, {
|
||||
type: "@earendil-works/pi-evals:source",
|
||||
runId,
|
||||
attachments: [attachment],
|
||||
});
|
||||
}
|
||||
|
||||
export async function persistEvalArtifactReferences(
|
||||
artifacts: ReadonlyArray<TestArtifact>,
|
||||
runId: string,
|
||||
artifactDirectory: string,
|
||||
): Promise<Array<{ name: string; path: string }>> {
|
||||
const references: Array<{ name: string; path: string }> = [];
|
||||
for (const artifact of artifacts) {
|
||||
if (
|
||||
(artifact.type !== "@earendil-works/pi-evals:session" &&
|
||||
artifact.type !== "@earendil-works/pi-evals:source") ||
|
||||
artifact.runId !== runId
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
const category = artifact.type === "@earendil-works/pi-evals:session" ? "sessions" : "sources";
|
||||
for (const attachment of artifact.attachments) {
|
||||
const name = basename(attachment.name);
|
||||
if (name !== attachment.name) throw new TypeError(`Invalid eval artifact name: ${attachment.name}`);
|
||||
const directory = join(artifactDirectory, category, createHash("sha256").update(runId).digest("hex"));
|
||||
await mkdir(directory, { recursive: true, mode: 0o700 });
|
||||
const path = join(directory, name);
|
||||
await writeFile(path, attachment.body, { encoding: "utf8", mode: 0o600 });
|
||||
references.push({ name, path: relative(artifactDirectory, path) });
|
||||
}
|
||||
}
|
||||
return references;
|
||||
}
|
||||
@@ -1,193 +0,0 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import {
|
||||
attachHarnessRunToError,
|
||||
getHarnessRunFromError,
|
||||
type Harness,
|
||||
type HarnessRun,
|
||||
type JsonValue,
|
||||
} from "vitest-evals/harness";
|
||||
|
||||
export const EVAL_HARNESS_ITERATION_ARTIFACT = "vitestEvalsHarnessIteration";
|
||||
|
||||
export type EvalHarnessIterationArtifact = {
|
||||
schemaVersion: 1;
|
||||
evalSet: string;
|
||||
groupKey: string;
|
||||
harness: string;
|
||||
baseline: string;
|
||||
candidates: string[];
|
||||
repetition: number;
|
||||
};
|
||||
|
||||
export type EvalHarnessTableRow<TInput, TOutput extends JsonValue | undefined> = {
|
||||
harness: Harness<TInput, TOutput>;
|
||||
name: string;
|
||||
repetition: number;
|
||||
};
|
||||
|
||||
export type EvalHarnessTablePairOptions<TInput, TOutput extends JsonValue | undefined> = {
|
||||
baseline: Harness<TInput, TOutput>;
|
||||
candidate: Harness<TInput, TOutput>;
|
||||
repetitions?: number;
|
||||
};
|
||||
|
||||
export type EvalHarnessTableCandidatesOptions<TInput, TOutput extends JsonValue | undefined> = {
|
||||
baseline: Harness<TInput, TOutput>;
|
||||
candidates: readonly Harness<TInput, TOutput>[];
|
||||
repetitions?: number;
|
||||
};
|
||||
|
||||
export type EvalHarnessTableOptions<TInput, TOutput extends JsonValue | undefined> =
|
||||
| EvalHarnessTablePairOptions<TInput, TOutput>
|
||||
| EvalHarnessTableCandidatesOptions<TInput, TOutput>;
|
||||
|
||||
type EvalHarnessIterationPlan = Omit<EvalHarnessIterationArtifact, "groupKey">;
|
||||
|
||||
export function parseEvalHarnessIterationArtifact(
|
||||
value: JsonValue | undefined,
|
||||
): EvalHarnessIterationArtifact | undefined {
|
||||
if (value === null || value === undefined || typeof value !== "object" || Array.isArray(value)) return undefined;
|
||||
const { schemaVersion, evalSet, groupKey, harness, baseline, candidates, repetition } = value;
|
||||
if (
|
||||
schemaVersion !== 1 ||
|
||||
typeof evalSet !== "string" ||
|
||||
typeof groupKey !== "string" ||
|
||||
typeof harness !== "string" ||
|
||||
typeof baseline !== "string" ||
|
||||
!Array.isArray(candidates) ||
|
||||
!candidates.every((name): name is string => typeof name === "string") ||
|
||||
typeof repetition !== "number"
|
||||
) {
|
||||
return undefined;
|
||||
}
|
||||
return { schemaVersion, evalSet, groupKey, harness, baseline, candidates, repetition };
|
||||
}
|
||||
|
||||
function canonicalizeJson(value: unknown, ancestors: WeakSet<object>): JsonValue {
|
||||
if (value === null || typeof value === "string" || typeof value === "boolean") return value;
|
||||
if (typeof value === "number") {
|
||||
if (!Number.isFinite(value)) throw new TypeError("Eval input must contain only finite numbers.");
|
||||
return value;
|
||||
}
|
||||
if (typeof value !== "object") throw new TypeError("Eval input must be JSON-serializable.");
|
||||
if (ancestors.has(value)) throw new TypeError("Eval input must not contain circular references.");
|
||||
|
||||
ancestors.add(value);
|
||||
try {
|
||||
if (Array.isArray(value)) {
|
||||
const result: JsonValue[] = [];
|
||||
for (let index = 0; index < value.length; index += 1) {
|
||||
if (!Object.hasOwn(value, index)) throw new TypeError("Eval input arrays must not be sparse.");
|
||||
result.push(canonicalizeJson(value[index], ancestors));
|
||||
}
|
||||
return result;
|
||||
}
|
||||
const prototype = Object.getPrototypeOf(value);
|
||||
if (prototype !== Object.prototype && prototype !== null) {
|
||||
throw new TypeError("Eval input must contain only plain objects and arrays.");
|
||||
}
|
||||
const entries: Array<[string, unknown]> = Object.entries(value);
|
||||
return Object.fromEntries(
|
||||
entries
|
||||
.sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0))
|
||||
.map(([key, item]): [string, JsonValue] => [key, canonicalizeJson(item, ancestors)]),
|
||||
);
|
||||
} finally {
|
||||
ancestors.delete(value);
|
||||
}
|
||||
}
|
||||
|
||||
function deriveInputKey(input: unknown): string {
|
||||
if (typeof input === "object" && input !== null && !Array.isArray(input) && "id" in input) {
|
||||
const id = input.id;
|
||||
if (typeof id === "string" && id.trim()) return id.trim();
|
||||
}
|
||||
const canonicalInput = JSON.stringify(canonicalizeJson(input, new WeakSet()));
|
||||
if (canonicalInput === undefined) throw new TypeError("Eval input must be JSON-serializable.");
|
||||
return createHash("sha256").update(canonicalInput).digest("hex");
|
||||
}
|
||||
|
||||
export function deriveEvalGroupKey(input: unknown, repetition: number): string {
|
||||
return JSON.stringify([deriveInputKey(input), repetition]);
|
||||
}
|
||||
|
||||
function validateOptions<TInput, TOutput extends JsonValue | undefined>(
|
||||
evalSet: string,
|
||||
baseline: Harness<TInput, TOutput>,
|
||||
candidates: readonly Harness<TInput, TOutput>[],
|
||||
repetitions: number,
|
||||
): void {
|
||||
if (!evalSet.trim()) throw new TypeError("evalSet must not be empty.");
|
||||
if (candidates.length === 0) throw new TypeError("At least one candidate harness is required.");
|
||||
const harnesses = [baseline, ...candidates];
|
||||
const names = new Set(harnesses.map((harness) => harness.name));
|
||||
if (names.size !== harnesses.length) throw new TypeError("Harness names must be unique within an eval set.");
|
||||
if (!Number.isSafeInteger(repetitions) || repetitions < 1) {
|
||||
throw new TypeError("repetitions must be a positive integer.");
|
||||
}
|
||||
}
|
||||
|
||||
function withIterationArtifact<TInput, TOutput extends JsonValue | undefined>(
|
||||
harness: Harness<TInput, TOutput>,
|
||||
plan: EvalHarnessIterationPlan,
|
||||
): Harness<TInput, TOutput> {
|
||||
return {
|
||||
name: harness.name,
|
||||
async run(input, context) {
|
||||
const groupKey = deriveEvalGroupKey(input, plan.repetition);
|
||||
const artifact: EvalHarnessIterationArtifact = { ...plan, groupKey };
|
||||
context.setArtifact(EVAL_HARNESS_ITERATION_ARTIFACT, artifact);
|
||||
const attachIterationArtifact = <TRun extends HarnessRun>(run: TRun): TRun => {
|
||||
run.artifacts = { ...context.artifacts, ...run.artifacts, [EVAL_HARNESS_ITERATION_ARTIFACT]: artifact };
|
||||
return run;
|
||||
};
|
||||
try {
|
||||
return attachIterationArtifact(await harness.run(input, context));
|
||||
} catch (error) {
|
||||
const partialRun = getHarnessRunFromError(error);
|
||||
if (partialRun) {
|
||||
throw attachHarnessRunToError(error, attachIterationArtifact(partialRun));
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export function evalHarnessTable<TInput, TOutput extends JsonValue | undefined>(
|
||||
evalSet: string,
|
||||
options: EvalHarnessTablePairOptions<TInput, TOutput>,
|
||||
): EvalHarnessTableRow<TInput, TOutput>[];
|
||||
export function evalHarnessTable<TInput, TOutput extends JsonValue | undefined>(
|
||||
evalSet: string,
|
||||
options: EvalHarnessTableCandidatesOptions<TInput, TOutput>,
|
||||
): EvalHarnessTableRow<TInput, TOutput>[];
|
||||
export function evalHarnessTable<TInput, TOutput extends JsonValue | undefined>(
|
||||
evalSet: string,
|
||||
options: EvalHarnessTableOptions<TInput, TOutput>,
|
||||
): EvalHarnessTableRow<TInput, TOutput>[] {
|
||||
const repetitions = options.repetitions ?? 1;
|
||||
const candidates = "candidate" in options ? [options.candidate] : options.candidates;
|
||||
validateOptions(evalSet, options.baseline, candidates, repetitions);
|
||||
|
||||
const rows: EvalHarnessTableRow<TInput, TOutput>[] = [];
|
||||
const harnesses = [options.baseline, ...candidates];
|
||||
for (let repetition = 1; repetition <= repetitions; repetition += 1) {
|
||||
for (const harness of harnesses) {
|
||||
const plan: EvalHarnessIterationPlan = {
|
||||
schemaVersion: 1,
|
||||
evalSet,
|
||||
harness: harness.name,
|
||||
baseline: options.baseline.name,
|
||||
candidates: candidates.map(({ name }) => name),
|
||||
repetition,
|
||||
};
|
||||
rows.push({
|
||||
harness: withIterationArtifact(harness, plan),
|
||||
name: harness.name,
|
||||
repetition,
|
||||
});
|
||||
}
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
@@ -1,111 +0,0 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { appendFile, mkdir } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import type { Reporter, SerializedError, TestCase, TestModule, TestRunEndReason, Vitest } from "vitest/node";
|
||||
import { isHarnessRun } from "vitest-evals/harness";
|
||||
import { PI_SESSION_SNAPSHOT_ARTIFACT, persistEvalArtifactReferences } from "./artifacts.ts";
|
||||
import { EVAL_HARNESS_ITERATION_ARTIFACT, parseEvalHarnessIterationArtifact } from "./harness-table.ts";
|
||||
import { formatHarnessComparisonReport, type HarnessObservation, summarizeHarnessComparisons } from "./summary.ts";
|
||||
|
||||
function readFiniteNumber(value: unknown): number | undefined {
|
||||
return typeof value === "number" && Number.isFinite(value) ? value : undefined;
|
||||
}
|
||||
|
||||
async function appendHarnessRunReport(test: TestCase): Promise<void> {
|
||||
const artifactDirectory = process.env.PI_EVAL_ARTIFACT_DIR?.trim();
|
||||
if (!artifactDirectory) return;
|
||||
const harness = test.meta().harness;
|
||||
if (!harness || !isHarnessRun(harness.run)) return;
|
||||
|
||||
const run = harness.run;
|
||||
const artifactRunId = run.artifacts?.runId;
|
||||
const runId = typeof artifactRunId === "string" ? artifactRunId : randomUUID();
|
||||
const metadata = Object.fromEntries(
|
||||
Object.entries(run.artifacts ?? {}).filter(([name]) => name !== "runId" && name !== PI_SESSION_SNAPSHOT_ARTIFACT),
|
||||
);
|
||||
const record = {
|
||||
schemaVersion: 1,
|
||||
runId,
|
||||
test: {
|
||||
id: test.id,
|
||||
file: test.module.relativeModuleId,
|
||||
name: test.name,
|
||||
fullName: test.fullName,
|
||||
status: test.result().state,
|
||||
},
|
||||
harness: harness.name,
|
||||
usage: run.usage,
|
||||
...(run.timings ? { timings: run.timings } : {}),
|
||||
...(run.errors.length > 0 ? { errors: run.errors } : {}),
|
||||
artifacts: await persistEvalArtifactReferences(test.artifacts(), runId, artifactDirectory),
|
||||
...(Object.keys(metadata).length > 0 ? { metadata } : {}),
|
||||
};
|
||||
await mkdir(artifactDirectory, { recursive: true, mode: 0o700 });
|
||||
await appendFile(join(artifactDirectory, "runs.jsonl"), `${JSON.stringify(record)}\n`, {
|
||||
encoding: "utf8",
|
||||
flag: "a",
|
||||
mode: 0o600,
|
||||
});
|
||||
}
|
||||
|
||||
function collectHarnessObservations(modules: ReadonlyArray<TestModule>): HarnessObservation[] {
|
||||
const observations: HarnessObservation[] = [];
|
||||
for (const module of modules) {
|
||||
for (const test of module.children.allTests()) {
|
||||
const harness = test.meta().harness;
|
||||
if (!harness || !isHarnessRun(harness.run)) continue;
|
||||
const run = harness.run;
|
||||
const iteration = parseEvalHarnessIterationArtifact(run.artifacts?.[EVAL_HARNESS_ITERATION_ARTIFACT]);
|
||||
if (!iteration) continue;
|
||||
const score = readFiniteNumber(test.meta().eval?.avgScore);
|
||||
const estimatedCostUsd = readFiniteNumber(run.usage.metadata?.estimatedCostUsd);
|
||||
const observation = {
|
||||
evalSet: iteration.evalSet,
|
||||
groupKey: iteration.groupKey,
|
||||
testName: test.name,
|
||||
file: module.relativeModuleId,
|
||||
harness: iteration.harness,
|
||||
baseline: iteration.baseline,
|
||||
candidates: iteration.candidates,
|
||||
repetition: iteration.repetition,
|
||||
...(run.usage.totalTokens === undefined ? {} : { totalTokens: run.usage.totalTokens }),
|
||||
...(run.timings?.totalMs === undefined ? {} : { totalMs: run.timings.totalMs }),
|
||||
...(estimatedCostUsd === undefined ? {} : { estimatedCostUsd }),
|
||||
};
|
||||
if (run.errors.length > 0) observations.push({ ...observation, outcome: "errored" });
|
||||
else if (score !== undefined) observations.push({ ...observation, outcome: "scored", score });
|
||||
else {
|
||||
const state = test.result().state;
|
||||
const outcome = state === "passed" ? "unscored" : state === "failed" ? "errored" : state;
|
||||
observations.push({ ...observation, outcome });
|
||||
}
|
||||
}
|
||||
}
|
||||
return observations;
|
||||
}
|
||||
|
||||
export default class EvalHarnessReporter implements Reporter {
|
||||
private vitest: Vitest | undefined;
|
||||
|
||||
onInit(vitest: Vitest): void {
|
||||
this.vitest = vitest;
|
||||
}
|
||||
|
||||
async onTestCaseResult(test: TestCase): Promise<void> {
|
||||
await appendHarnessRunReport(test);
|
||||
}
|
||||
|
||||
onTestRunEnd(
|
||||
modules: ReadonlyArray<TestModule>,
|
||||
_errors: ReadonlyArray<SerializedError>,
|
||||
reason: TestRunEndReason,
|
||||
): void {
|
||||
if (reason === "interrupted") {
|
||||
this.vitest?.logger.log("\nEval comparisons unavailable: test run interrupted.");
|
||||
return;
|
||||
}
|
||||
const report = summarizeHarnessComparisons(collectHarnessObservations(modules));
|
||||
const formatted = formatHarnessComparisonReport(report);
|
||||
if (formatted) this.vitest?.logger.log(`\n${formatted}`);
|
||||
}
|
||||
}
|
||||
@@ -1,8 +0,0 @@
|
||||
import { afterEach } from "vitest";
|
||||
import type {} from "vitest-evals";
|
||||
import { recordEvalSessionArtifact } from "./artifacts.ts";
|
||||
|
||||
afterEach(async ({ task }) => {
|
||||
const run = task.meta.harness?.run;
|
||||
if (run) await recordEvalSessionArtifact(task, run);
|
||||
});
|
||||
@@ -1,438 +0,0 @@
|
||||
import { styleText } from "node:util";
|
||||
|
||||
type HarnessObservationOutcome = "scored" | "unscored" | "skipped" | "pending" | "errored";
|
||||
|
||||
type HarnessObservationBase = {
|
||||
evalSet: string;
|
||||
groupKey: string;
|
||||
testName: string;
|
||||
file: string;
|
||||
harness: string;
|
||||
baseline: string;
|
||||
candidates: string[];
|
||||
repetition: number;
|
||||
totalTokens?: number;
|
||||
totalMs?: number;
|
||||
estimatedCostUsd?: number;
|
||||
};
|
||||
|
||||
export type HarnessObservation = HarnessObservationBase &
|
||||
({ outcome: "scored"; score: number } | { outcome: Exclude<HarnessObservationOutcome, "scored">; score?: never });
|
||||
|
||||
export type PairedMetricSummary = {
|
||||
totalPairs: number;
|
||||
eligiblePairs: number;
|
||||
baselineMean: number | null;
|
||||
candidateMean: number | null;
|
||||
meanDelta: number | null;
|
||||
};
|
||||
|
||||
export type CorrectnessLiftSummary = {
|
||||
totalPairs: number;
|
||||
eligiblePairs: number;
|
||||
baselinePassRate: number | null;
|
||||
candidatePassRate: number | null;
|
||||
lift: number | null;
|
||||
baselineWins: number;
|
||||
candidateWins: number;
|
||||
ties: number;
|
||||
};
|
||||
|
||||
export type HarnessPairComparison = {
|
||||
baseline: string;
|
||||
candidate: string;
|
||||
correctness: CorrectnessLiftSummary;
|
||||
totalTokens: PairedMetricSummary;
|
||||
totalMs: PairedMetricSummary;
|
||||
estimatedCostUsd: PairedMetricSummary;
|
||||
};
|
||||
|
||||
export type HarnessComparisonDiagnostic = {
|
||||
evalSet: string;
|
||||
groupKey: string;
|
||||
testName: string;
|
||||
file: string;
|
||||
repetition: number;
|
||||
harness: string;
|
||||
reason: "missing-observation" | "duplicate-observation" | "harness-error" | "missing-score" | "unscorable-outcome";
|
||||
};
|
||||
|
||||
export type HarnessEvalSetReport = {
|
||||
evalSet: string;
|
||||
comparisons: HarnessPairComparison[];
|
||||
};
|
||||
|
||||
export type HarnessComparisonReport = {
|
||||
schemaVersion: 1;
|
||||
evalSets: HarnessEvalSetReport[];
|
||||
diagnostics: HarnessComparisonDiagnostic[];
|
||||
};
|
||||
|
||||
type HarnessDescriptor = {
|
||||
name: string;
|
||||
index: number;
|
||||
};
|
||||
|
||||
type ObservationGroup = {
|
||||
evalSet: string;
|
||||
groupKey: string;
|
||||
testName: string;
|
||||
file: string;
|
||||
repetition: number;
|
||||
observationsByHarness: Map<string, HarnessObservation[]>;
|
||||
};
|
||||
|
||||
type EvalSetData = {
|
||||
baseline: HarnessDescriptor;
|
||||
candidatesByName: Map<string, HarnessDescriptor>;
|
||||
groupsByKey: Map<string, ObservationGroup>;
|
||||
};
|
||||
|
||||
type ObservationPair = {
|
||||
baseline: HarnessObservation;
|
||||
candidate: HarnessObservation;
|
||||
};
|
||||
|
||||
function getOrCreate<K, V extends object>(map: Map<K, V>, key: K, create: () => V): V {
|
||||
const existing = map.get(key);
|
||||
if (existing !== undefined) return existing;
|
||||
const value = create();
|
||||
map.set(key, value);
|
||||
return value;
|
||||
}
|
||||
|
||||
function mean(values: readonly number[]): number | null {
|
||||
return values.length === 0 ? null : values.reduce((sum, value) => sum + value, 0) / values.length;
|
||||
}
|
||||
|
||||
function preciseDifference(left: number, right: number): number {
|
||||
return Number((left - right).toPrecision(15));
|
||||
}
|
||||
|
||||
function groupObservations(observations: readonly HarnessObservation[]): Map<string, EvalSetData> {
|
||||
const evalSets = new Map<string, EvalSetData>();
|
||||
for (const observation of observations) {
|
||||
const evalSet = getOrCreate(evalSets, observation.evalSet, () => ({
|
||||
baseline: { name: observation.baseline, index: 0 },
|
||||
candidatesByName: new Map(),
|
||||
groupsByKey: new Map(),
|
||||
}));
|
||||
|
||||
for (const [index, name] of observation.candidates.entries()) {
|
||||
const existing = evalSet.candidatesByName.get(name);
|
||||
if (!existing || index < existing.index) evalSet.candidatesByName.set(name, { name, index });
|
||||
}
|
||||
|
||||
const group = getOrCreate(
|
||||
evalSet.groupsByKey,
|
||||
JSON.stringify([observation.file, observation.testName, observation.groupKey]),
|
||||
() => ({
|
||||
evalSet: observation.evalSet,
|
||||
groupKey: observation.groupKey,
|
||||
testName: observation.testName,
|
||||
file: observation.file,
|
||||
repetition: observation.repetition,
|
||||
observationsByHarness: new Map(),
|
||||
}),
|
||||
);
|
||||
getOrCreate(group.observationsByHarness, observation.harness, (): HarnessObservation[] => []).push(observation);
|
||||
}
|
||||
return evalSets;
|
||||
}
|
||||
|
||||
function orderedHarnesses(evalSet: EvalSetData): HarnessDescriptor[] {
|
||||
return [
|
||||
evalSet.baseline,
|
||||
...[...evalSet.candidatesByName.values()].sort(
|
||||
(left, right) => left.index - right.index || left.name.localeCompare(right.name),
|
||||
),
|
||||
];
|
||||
}
|
||||
|
||||
function orderedCandidates(evalSet: EvalSetData): HarnessDescriptor[] {
|
||||
return [...evalSet.candidatesByName.values()].sort(
|
||||
(left, right) => left.index - right.index || left.name.localeCompare(right.name),
|
||||
);
|
||||
}
|
||||
|
||||
function orderedGroups(evalSet: EvalSetData): ObservationGroup[] {
|
||||
return [...evalSet.groupsByKey.values()].sort(
|
||||
(left, right) => left.groupKey.localeCompare(right.groupKey) || left.repetition - right.repetition,
|
||||
);
|
||||
}
|
||||
|
||||
function collectDiagnostics(
|
||||
harnesses: readonly HarnessDescriptor[],
|
||||
groups: readonly ObservationGroup[],
|
||||
): HarnessComparisonDiagnostic[] {
|
||||
const diagnostics: HarnessComparisonDiagnostic[] = [];
|
||||
for (const group of groups) {
|
||||
for (const { name: harness } of harnesses) {
|
||||
const observations = group.observationsByHarness.get(harness) ?? [];
|
||||
let reason: HarnessComparisonDiagnostic["reason"] | undefined;
|
||||
if (observations.length === 0) reason = "missing-observation";
|
||||
else if (observations.length > 1) reason = "duplicate-observation";
|
||||
else if (observations[0].outcome === "errored") reason = "harness-error";
|
||||
else if (observations[0].outcome === "unscored") {
|
||||
reason = "missing-score";
|
||||
} else if (observations[0].outcome !== "scored") {
|
||||
reason = "unscorable-outcome";
|
||||
}
|
||||
if (!reason) continue;
|
||||
diagnostics.push({
|
||||
evalSet: group.evalSet,
|
||||
groupKey: group.groupKey,
|
||||
testName: group.testName,
|
||||
file: group.file,
|
||||
repetition: group.repetition,
|
||||
harness,
|
||||
reason,
|
||||
});
|
||||
}
|
||||
}
|
||||
return diagnostics;
|
||||
}
|
||||
|
||||
function pairObservations(
|
||||
groups: readonly ObservationGroup[],
|
||||
baselineHarness: string,
|
||||
candidateHarness: string,
|
||||
): ObservationPair[] {
|
||||
const pairs: ObservationPair[] = [];
|
||||
for (const group of groups) {
|
||||
const baseline = group.observationsByHarness.get(baselineHarness) ?? [];
|
||||
const candidate = group.observationsByHarness.get(candidateHarness) ?? [];
|
||||
if (baseline.length === 1 && candidate.length === 1) {
|
||||
pairs.push({ baseline: baseline[0], candidate: candidate[0] });
|
||||
}
|
||||
}
|
||||
return pairs;
|
||||
}
|
||||
|
||||
function summarizeMetric(
|
||||
pairs: readonly ObservationPair[],
|
||||
select: (observation: HarnessObservation) => number | undefined,
|
||||
totalPairs: number,
|
||||
): PairedMetricSummary {
|
||||
const baselineValues: number[] = [];
|
||||
const candidateValues: number[] = [];
|
||||
for (const { baseline, candidate } of pairs) {
|
||||
if (baseline.outcome !== "scored" || candidate.outcome !== "scored") continue;
|
||||
const baselineValue = select(baseline);
|
||||
const candidateValue = select(candidate);
|
||||
if (
|
||||
baselineValue === undefined ||
|
||||
candidateValue === undefined ||
|
||||
!Number.isFinite(baselineValue) ||
|
||||
!Number.isFinite(candidateValue)
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
baselineValues.push(baselineValue);
|
||||
candidateValues.push(candidateValue);
|
||||
}
|
||||
|
||||
const baselineMean = mean(baselineValues);
|
||||
const candidateMean = mean(candidateValues);
|
||||
return {
|
||||
totalPairs,
|
||||
eligiblePairs: baselineValues.length,
|
||||
baselineMean,
|
||||
candidateMean,
|
||||
meanDelta:
|
||||
baselineMean === null || candidateMean === null ? null : preciseDifference(candidateMean, baselineMean),
|
||||
};
|
||||
}
|
||||
|
||||
function summarizeCorrectness(pairs: readonly ObservationPair[], totalPairs: number): CorrectnessLiftSummary {
|
||||
let eligiblePairs = 0;
|
||||
let baselinePasses = 0;
|
||||
let candidatePasses = 0;
|
||||
let baselineWins = 0;
|
||||
let candidateWins = 0;
|
||||
let ties = 0;
|
||||
|
||||
for (const { baseline, candidate } of pairs) {
|
||||
if (baseline.outcome !== "scored" || candidate.outcome !== "scored") continue;
|
||||
eligiblePairs += 1;
|
||||
const baselinePassed = baseline.score >= 1;
|
||||
const candidatePassed = candidate.score >= 1;
|
||||
if (baselinePassed) baselinePasses += 1;
|
||||
if (candidatePassed) candidatePasses += 1;
|
||||
if (baselinePassed === candidatePassed) ties += 1;
|
||||
else if (baselinePassed) baselineWins += 1;
|
||||
else candidateWins += 1;
|
||||
}
|
||||
|
||||
const baselinePassRate = eligiblePairs === 0 ? null : baselinePasses / eligiblePairs;
|
||||
const candidatePassRate = eligiblePairs === 0 ? null : candidatePasses / eligiblePairs;
|
||||
return {
|
||||
totalPairs,
|
||||
eligiblePairs,
|
||||
baselinePassRate,
|
||||
candidatePassRate,
|
||||
lift:
|
||||
baselinePassRate === null || candidatePassRate === null
|
||||
? null
|
||||
: preciseDifference(candidatePassRate, baselinePassRate),
|
||||
baselineWins,
|
||||
candidateWins,
|
||||
ties,
|
||||
};
|
||||
}
|
||||
|
||||
function compareHarnesses(
|
||||
baseline: HarnessDescriptor,
|
||||
candidate: HarnessDescriptor,
|
||||
groups: readonly ObservationGroup[],
|
||||
): HarnessPairComparison {
|
||||
const pairs = pairObservations(groups, baseline.name, candidate.name);
|
||||
return {
|
||||
baseline: baseline.name,
|
||||
candidate: candidate.name,
|
||||
correctness: summarizeCorrectness(pairs, groups.length),
|
||||
totalTokens: summarizeMetric(pairs, ({ totalTokens }) => totalTokens, groups.length),
|
||||
totalMs: summarizeMetric(pairs, ({ totalMs }) => totalMs, groups.length),
|
||||
estimatedCostUsd: summarizeMetric(pairs, ({ estimatedCostUsd }) => estimatedCostUsd, groups.length),
|
||||
};
|
||||
}
|
||||
|
||||
export function summarizeHarnessComparisons(observations: readonly HarnessObservation[]): HarnessComparisonReport {
|
||||
const evalSets: HarnessEvalSetReport[] = [];
|
||||
const diagnostics: HarnessComparisonDiagnostic[] = [];
|
||||
for (const [evalSet, data] of [...groupObservations(observations)].sort(([left], [right]) =>
|
||||
left.localeCompare(right),
|
||||
)) {
|
||||
const harnesses = orderedHarnesses(data);
|
||||
const candidates = orderedCandidates(data);
|
||||
const groups = orderedGroups(data);
|
||||
evalSets.push({
|
||||
evalSet,
|
||||
comparisons: candidates.map((candidate) => compareHarnesses(data.baseline, candidate, groups)),
|
||||
});
|
||||
diagnostics.push(...collectDiagnostics(harnesses, groups));
|
||||
}
|
||||
|
||||
return {
|
||||
schemaVersion: 1,
|
||||
evalSets,
|
||||
diagnostics: diagnostics.sort(
|
||||
(left, right) =>
|
||||
left.evalSet.localeCompare(right.evalSet) ||
|
||||
left.file.localeCompare(right.file) ||
|
||||
left.groupKey.localeCompare(right.groupKey) ||
|
||||
left.repetition - right.repetition ||
|
||||
left.harness.localeCompare(right.harness),
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
function formatPercentage(value: number | null): string {
|
||||
return value === null ? "unavailable" : `${(value * 100).toFixed(1)}%`;
|
||||
}
|
||||
|
||||
function formatSigned(value: number, fractionDigits: number): string {
|
||||
return `${value >= 0 ? "+" : ""}${value.toFixed(fractionDigits)}`;
|
||||
}
|
||||
|
||||
function formatCoverage(eligiblePairs: number, totalPairs: number): string {
|
||||
return styleText("gray", `(${eligiblePairs}/${totalPairs} pairs)`);
|
||||
}
|
||||
|
||||
function formatReportLine(label: string, value: string): string {
|
||||
return ` ${styleText("gray", label.padStart(9))} ${value}`;
|
||||
}
|
||||
|
||||
function colorDelta(value: number, formatted: string, positiveIsBetter: boolean): string {
|
||||
if (value === 0) return styleText("gray", formatted);
|
||||
const improved = positiveIsBetter ? value > 0 : value < 0;
|
||||
return styleText(improved ? "green" : "red", formatted);
|
||||
}
|
||||
|
||||
function formatMetric(
|
||||
label: string,
|
||||
metric: PairedMetricSummary,
|
||||
formatValue: (value: number) => string,
|
||||
formatDelta: (value: number) => string,
|
||||
comparisonPairs: number,
|
||||
): string {
|
||||
const coverage =
|
||||
metric.eligiblePairs === 0 || metric.eligiblePairs === comparisonPairs
|
||||
? ""
|
||||
: ` ${formatCoverage(metric.eligiblePairs, metric.totalPairs)}`;
|
||||
if (metric.baselineMean === null || metric.candidateMean === null || metric.meanDelta === null) {
|
||||
return formatReportLine(label, `${styleText("yellow", "unavailable")}${coverage}`);
|
||||
}
|
||||
const delta = colorDelta(metric.meanDelta, formatDelta(metric.meanDelta), false);
|
||||
const values = styleText(
|
||||
"gray",
|
||||
`(candidate ${formatValue(metric.candidateMean)}, baseline ${formatValue(metric.baselineMean)})`,
|
||||
);
|
||||
return formatReportLine(label, `${delta} ${values}${coverage}`);
|
||||
}
|
||||
|
||||
export function formatHarnessComparisonReport(report: HarnessComparisonReport): string {
|
||||
if (report.evalSets.every(({ comparisons }) => comparisons.length === 0)) return "";
|
||||
const lines = [styleText("bold", "Eval Comparisons")];
|
||||
for (const evalSet of report.evalSets) {
|
||||
lines.push(` ${evalSet.evalSet}`);
|
||||
for (const [index, comparison] of evalSet.comparisons.entries()) {
|
||||
if (index > 0) lines.push("");
|
||||
const { correctness } = comparison;
|
||||
lines.push(formatReportLine("Baseline", comparison.baseline));
|
||||
lines.push(
|
||||
formatReportLine(
|
||||
"Candidate",
|
||||
`${comparison.candidate} ${formatCoverage(correctness.eligiblePairs, correctness.totalPairs)}`,
|
||||
),
|
||||
);
|
||||
if (correctness.lift === null) {
|
||||
lines.push(formatReportLine("Pass rate", styleText("yellow", "unavailable")));
|
||||
} else {
|
||||
const lift = correctness.lift * 100;
|
||||
const delta = colorDelta(lift, `${formatSigned(lift, 1)} pp`, true);
|
||||
const values = styleText(
|
||||
"gray",
|
||||
`(candidate ${formatPercentage(correctness.candidatePassRate)}, baseline ${formatPercentage(correctness.baselinePassRate)})`,
|
||||
);
|
||||
lines.push(formatReportLine("Pass rate", `${delta} ${values}`));
|
||||
}
|
||||
lines.push(
|
||||
formatMetric(
|
||||
"Tokens",
|
||||
comparison.totalTokens,
|
||||
(value) => value.toFixed(1),
|
||||
(value) => formatSigned(value, 1),
|
||||
correctness.eligiblePairs,
|
||||
),
|
||||
);
|
||||
lines.push(
|
||||
formatMetric(
|
||||
"Latency",
|
||||
comparison.totalMs,
|
||||
(value) => `${value.toFixed(1)}ms`,
|
||||
(value) => `${formatSigned(value, 1)}ms`,
|
||||
correctness.eligiblePairs,
|
||||
),
|
||||
);
|
||||
lines.push(
|
||||
formatMetric(
|
||||
"Est. cost",
|
||||
comparison.estimatedCostUsd,
|
||||
(value) => `$${value.toFixed(4)}`,
|
||||
(value) => `${value >= 0 ? "+" : "-"}$${Math.abs(value).toFixed(4)}`,
|
||||
correctness.eligiblePairs,
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
if (report.diagnostics.length > 0) {
|
||||
lines.push(` ${styleText("yellow", "Incomplete observations")}`);
|
||||
for (const diagnostic of report.diagnostics) {
|
||||
lines.push(
|
||||
` ${diagnostic.reason}: ${diagnostic.file}/${diagnostic.testName} repetition ${diagnostic.repetition}, harness ${diagnostic.harness}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
return lines.join("\n");
|
||||
}
|
||||
128
third_party/pi/packages/evals/test/acme-server.test.ts
vendored
Normal file
128
third_party/pi/packages/evals/test/acme-server.test.ts
vendored
Normal file
@@ -0,0 +1,128 @@
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it } from "vitest";
|
||||
import {
|
||||
type AcmeServer,
|
||||
createAcmeServer,
|
||||
OPENAI_MODEL_ID,
|
||||
OPENAI_PROBE_PROMPT,
|
||||
OPENAI_PROBE_RESPONSE,
|
||||
STREAM_MODEL_ID,
|
||||
STREAM_PROBE_PROMPT,
|
||||
STREAM_PROBE_RESPONSE,
|
||||
} from "../evals/acme-server.ts";
|
||||
|
||||
type Fixture = {
|
||||
mode: "openai" | "stream";
|
||||
url: (server: AcmeServer) => string;
|
||||
headers: Record<string, string>;
|
||||
unauthorizedHeaders: Record<string, string>;
|
||||
model: string;
|
||||
prompt: string;
|
||||
response: string;
|
||||
contentType: string;
|
||||
unauthorized: string;
|
||||
invalid: string;
|
||||
};
|
||||
|
||||
const fixtures: Fixture[] = [
|
||||
{
|
||||
mode: "openai",
|
||||
url: (server) => `${server.baseUrl()}/chat/completions`,
|
||||
headers: { authorization: "Bearer resolved-acme-key" },
|
||||
unauthorizedHeaders: { authorization: "Bearer wrong-key" },
|
||||
model: OPENAI_MODEL_ID,
|
||||
prompt: OPENAI_PROBE_PROMPT,
|
||||
response: OPENAI_PROBE_RESPONSE,
|
||||
contentType: "text/event-stream",
|
||||
unauthorized: "Invalid Acme credential",
|
||||
invalid: "Invalid OpenAI-compatible request",
|
||||
},
|
||||
{
|
||||
mode: "stream",
|
||||
url: (server) => `${server.origin()}/generate`,
|
||||
headers: { "x-acme-key": "resolved-stream-key" },
|
||||
unauthorizedHeaders: { "x-acme-key": "wrong-key" },
|
||||
model: STREAM_MODEL_ID,
|
||||
prompt: STREAM_PROBE_PROMPT,
|
||||
response: STREAM_PROBE_RESPONSE,
|
||||
contentType: "application/x-ndjson",
|
||||
unauthorized: "Invalid Acme Stream credential",
|
||||
invalid: "Invalid Acme Stream request",
|
||||
},
|
||||
];
|
||||
|
||||
function body(model: string, prompt: string, stream = true) {
|
||||
return { model, messages: [{ role: "user", content: prompt }], stream };
|
||||
}
|
||||
|
||||
async function post(url: string, headers: Record<string, string>, payload: unknown) {
|
||||
return fetch(url, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json", ...headers },
|
||||
body: JSON.stringify(payload),
|
||||
});
|
||||
}
|
||||
|
||||
function streamedText(mode: Fixture["mode"], raw: string): string {
|
||||
if (mode === "stream") {
|
||||
return raw
|
||||
.split("\n")
|
||||
.filter(Boolean)
|
||||
.map((line) => JSON.parse(line) as { type?: string; text?: string })
|
||||
.filter((event) => event.type === "text_delta")
|
||||
.map((event) => event.text ?? "")
|
||||
.join("");
|
||||
}
|
||||
return raw
|
||||
.split("\n")
|
||||
.filter((line) => line.startsWith("data: ") && line !== "data: [DONE]")
|
||||
.map((line) => JSON.parse(line.slice("data: ".length)) as { choices?: Array<{ delta?: { content?: string } }> })
|
||||
.map((chunk) => chunk.choices?.[0]?.delta?.content ?? "")
|
||||
.join("");
|
||||
}
|
||||
|
||||
describe.each(fixtures)("$mode fixture", (fixture) => {
|
||||
const server = createAcmeServer(fixture.mode);
|
||||
|
||||
beforeAll(() => server.start());
|
||||
beforeEach(() => server.reset());
|
||||
afterAll(() => server.stop());
|
||||
|
||||
it("accepts the probe request and records it", async () => {
|
||||
const response = await post(fixture.url(server), fixture.headers, body(fixture.model, fixture.prompt));
|
||||
expect(response.status).toBe(200);
|
||||
expect(response.headers.get("content-type")).toContain(fixture.contentType);
|
||||
expect(streamedText(fixture.mode, await response.text())).toBe(fixture.response);
|
||||
expect(server.validRequestReceived()).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects a bad credential with 401 and does not record a probe", async () => {
|
||||
const response = await post(
|
||||
fixture.url(server),
|
||||
fixture.unauthorizedHeaders,
|
||||
body(fixture.model, fixture.prompt),
|
||||
);
|
||||
expect(response.status).toBe(401);
|
||||
expect(await response.text()).toBe(fixture.unauthorized);
|
||||
expect(server.validRequestReceived()).toBe(false);
|
||||
});
|
||||
|
||||
it("rejects a malformed request with 422 and does not record a probe", async () => {
|
||||
const response = await post(fixture.url(server), fixture.headers, body(fixture.model, fixture.prompt, false));
|
||||
expect(response.status).toBe(422);
|
||||
expect(await response.text()).toBe(fixture.invalid);
|
||||
expect(server.validRequestReceived()).toBe(false);
|
||||
});
|
||||
|
||||
it("does not treat a non-probe success as the probe", async () => {
|
||||
const response = await post(fixture.url(server), fixture.headers, body(fixture.model, "hello"));
|
||||
expect(response.status).toBe(200);
|
||||
expect(server.validRequestReceived()).toBe(false);
|
||||
});
|
||||
|
||||
it("clears the probe flag on reset", async () => {
|
||||
await post(fixture.url(server), fixture.headers, body(fixture.model, fixture.prompt));
|
||||
expect(server.validRequestReceived()).toBe(true);
|
||||
server.reset();
|
||||
expect(server.validRequestReceived()).toBe(false);
|
||||
});
|
||||
});
|
||||
125
third_party/pi/packages/evals/test/comparison.test.ts
vendored
Normal file
125
third_party/pi/packages/evals/test/comparison.test.ts
vendored
Normal file
@@ -0,0 +1,125 @@
|
||||
import { stripVTControlCharacters } from "node:util";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
type EvalObservation,
|
||||
type ExpectedEvalRun,
|
||||
formatEvalComparisonReport,
|
||||
summarizeEvalObservations,
|
||||
} from "../src/report.ts";
|
||||
|
||||
function scored(
|
||||
variant: EvalObservation["variant"],
|
||||
runNumber: number,
|
||||
score: number,
|
||||
metrics: Partial<{
|
||||
totalTokens: number;
|
||||
toolCalls: number;
|
||||
totalMs: number;
|
||||
estimatedCostUsd: number;
|
||||
}> = {},
|
||||
): EvalObservation {
|
||||
return {
|
||||
evalSet: "tool access",
|
||||
caseId: "create",
|
||||
variant,
|
||||
model: "fixture/model",
|
||||
runNumber,
|
||||
outcome: "scored",
|
||||
score,
|
||||
totalTokens: 100,
|
||||
toolCalls: 2,
|
||||
totalMs: 1000,
|
||||
estimatedCostUsd: 0.01,
|
||||
...metrics,
|
||||
};
|
||||
}
|
||||
|
||||
function errored(variant: EvalObservation["variant"], runNumber: number, totalTokens?: number): EvalObservation {
|
||||
return {
|
||||
evalSet: "tool access",
|
||||
caseId: "create",
|
||||
variant,
|
||||
model: "fixture/model",
|
||||
runNumber,
|
||||
outcome: "errored",
|
||||
totalTokens,
|
||||
};
|
||||
}
|
||||
|
||||
function expectedFor(runNumbers: readonly number[]): ExpectedEvalRun[] {
|
||||
return runNumbers.flatMap((runNumber) =>
|
||||
(["without_docs", "with_docs"] as const).map((variant) => ({
|
||||
evalSet: "tool access",
|
||||
caseId: "create",
|
||||
variant,
|
||||
model: "fixture/model",
|
||||
runNumber,
|
||||
})),
|
||||
);
|
||||
}
|
||||
|
||||
describe("summarizeEvalObservations", () => {
|
||||
it("computes paired lift and efficiency deltas", () => {
|
||||
const observations = [
|
||||
scored("without_docs", 1, 0, { totalTokens: 100, toolCalls: 3, totalMs: 1000 }),
|
||||
scored("with_docs", 1, 1, { totalTokens: 120, toolCalls: 2, totalMs: 800 }),
|
||||
scored("without_docs", 2, 1, { totalTokens: 200 }),
|
||||
scored("with_docs", 2, 1, { totalTokens: 180 }),
|
||||
];
|
||||
const report = summarizeEvalObservations("digest", expectedFor([1, 2]), observations);
|
||||
expect(report.comparisons).toEqual([
|
||||
expect.objectContaining({
|
||||
evalSet: "tool access",
|
||||
totalPairs: 2,
|
||||
eligiblePairs: 2,
|
||||
blockedPairs: 0,
|
||||
controlPassRate: 0.5,
|
||||
treatmentPassRate: 1,
|
||||
lift: 0.5,
|
||||
totalTokens: { eligiblePairs: 2, controlMean: 150, treatmentMean: 150, meanDelta: 0 },
|
||||
toolCalls: { eligiblePairs: 2, controlMean: 2.5, treatmentMean: 2, meanDelta: -0.5 },
|
||||
}),
|
||||
]);
|
||||
});
|
||||
|
||||
it("fails closed for incomplete and errored pairs while retaining totals", () => {
|
||||
const observations: EvalObservation[] = [
|
||||
scored("without_docs", 1, 0),
|
||||
errored("with_docs", 1, 120),
|
||||
scored("without_docs", 2, 1, { totalTokens: 200 }),
|
||||
];
|
||||
const report = summarizeEvalObservations("digest", expectedFor([1, 2]), observations);
|
||||
expect(report.comparisons[0]).toEqual(
|
||||
expect.objectContaining({ totalPairs: 2, eligiblePairs: 0, blockedPairs: 2, lift: null }),
|
||||
);
|
||||
expect(report.blockedPairs).toEqual([
|
||||
expect.objectContaining({ runNumber: 1, reasons: ["with_docs: errored"] }),
|
||||
expect.objectContaining({ runNumber: 2, reasons: ["with_docs: expected 1 observation, found 0"] }),
|
||||
]);
|
||||
expect(report.operationalTotals[0].totalTokens).toEqual({ availableRuns: 2, total: 300 });
|
||||
});
|
||||
|
||||
it("blocks duplicate observations and keeps missing metrics distinct from zero", () => {
|
||||
const withoutTokens = scored("without_docs", 1, 1);
|
||||
delete withoutTokens.totalTokens;
|
||||
const report = summarizeEvalObservations("digest", expectedFor([1]), [
|
||||
withoutTokens,
|
||||
withoutTokens,
|
||||
scored("with_docs", 1, 1, { totalTokens: 0 }),
|
||||
]);
|
||||
expect(report.blockedPairs[0].reasons).toEqual(["without_docs: expected 1 observation, found 2"]);
|
||||
expect(report.operationalTotals[0].totalTokens).toEqual({ availableRuns: 0, total: null });
|
||||
expect(report.operationalTotals[1].totalTokens).toEqual({ availableRuns: 1, total: 0 });
|
||||
});
|
||||
|
||||
it("formats blocked comparisons and operational totals", () => {
|
||||
const report = summarizeEvalObservations("digest", expectedFor([1, 2]), [
|
||||
scored("without_docs", 1, 1),
|
||||
scored("with_docs", 1, 1),
|
||||
]);
|
||||
const formatted = stripVTControlCharacters(formatEvalComparisonReport(report));
|
||||
expect(formatted).toContain("Documentation Eval Comparisons");
|
||||
expect(formatted).toContain("Pass rate withheld because pairs are blocked");
|
||||
expect(formatted).toContain("without_docs: 1 runs");
|
||||
});
|
||||
});
|
||||
196
third_party/pi/packages/evals/test/configured-runtime.test.ts
vendored
Normal file
196
third_party/pi/packages/evals/test/configured-runtime.test.ts
vendored
Normal file
@@ -0,0 +1,196 @@
|
||||
import { mkdtemp, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from "vitest";
|
||||
import {
|
||||
createAcmeServer,
|
||||
OPENAI_MODEL_ID,
|
||||
OPENAI_PROBE_PROMPT,
|
||||
OPENAI_PROBE_RESPONSE,
|
||||
OPENAI_PROVIDER_ID,
|
||||
} from "../evals/acme-server.ts";
|
||||
import { inspectAddedModel, inspectProvider, loadConfiguredModelRuntime } from "../evals/configured-runtime.ts";
|
||||
|
||||
const PROVIDER_ID = OPENAI_PROVIDER_ID;
|
||||
const MODEL = {
|
||||
id: OPENAI_MODEL_ID,
|
||||
name: "Acme Chat",
|
||||
provider: PROVIDER_ID,
|
||||
reasoning: false,
|
||||
input: ["text"],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 32768,
|
||||
maxTokens: 4096,
|
||||
};
|
||||
const server = createAcmeServer("openai");
|
||||
const temporaryDirectories: string[] = [];
|
||||
|
||||
afterEach(async () => {
|
||||
await Promise.all(temporaryDirectories.splice(0).map((path) => rm(path, { recursive: true, force: true })));
|
||||
});
|
||||
beforeAll(() => server.start());
|
||||
beforeEach(() => server.reset());
|
||||
afterAll(() => server.stop());
|
||||
|
||||
async function agentDirWith(modelsJson: unknown) {
|
||||
const directory = await mkdtemp(join(tmpdir(), "pi-eval-provider-probe-"));
|
||||
temporaryDirectories.push(directory);
|
||||
await writeFile(join(directory, "models.json"), `${JSON.stringify(modelsJson)}\n`);
|
||||
return directory;
|
||||
}
|
||||
|
||||
async function runtimeWith(modelsJson: unknown) {
|
||||
return loadConfiguredModelRuntime(await agentDirWith(modelsJson));
|
||||
}
|
||||
|
||||
function acmeModelsJson() {
|
||||
return {
|
||||
providers: {
|
||||
[PROVIDER_ID]: {
|
||||
baseUrl: server.baseUrl(),
|
||||
api: "openai-completions",
|
||||
apiKey: "$ACME_API_KEY",
|
||||
models: [
|
||||
{
|
||||
id: MODEL.id,
|
||||
name: MODEL.name,
|
||||
reasoning: MODEL.reasoning,
|
||||
input: MODEL.input,
|
||||
cost: MODEL.cost,
|
||||
contextWindow: MODEL.contextWindow,
|
||||
maxTokens: MODEL.maxTokens,
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function probe(content: string, apiKey: string) {
|
||||
return {
|
||||
providerId: PROVIDER_ID,
|
||||
modelId: OPENAI_MODEL_ID,
|
||||
createContext: () => ({
|
||||
messages: [{ role: "user" as const, content, timestamp: 0 }],
|
||||
}),
|
||||
options: { env: { ACME_API_KEY: apiKey }, maxTokens: 32 },
|
||||
validRequestReceived: server.validRequestReceived,
|
||||
};
|
||||
}
|
||||
|
||||
describe("inspectProvider", () => {
|
||||
it("probes a models.json provider through ModelRuntime", async () => {
|
||||
const runtime = await runtimeWith(acmeModelsJson());
|
||||
await expect(inspectProvider(runtime, probe(OPENAI_PROBE_PROMPT, "resolved-acme-key"))).resolves.toEqual({
|
||||
result: {
|
||||
validRequestReceived: true,
|
||||
model: MODEL,
|
||||
response: { text: OPENAI_PROBE_RESPONSE, stopReason: "stop", inputTokens: 3, outputTokens: 2 },
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps a non-probe completion distinct from a valid probe", async () => {
|
||||
const runtime = await runtimeWith(acmeModelsJson());
|
||||
await expect(inspectProvider(runtime, probe("hello", "resolved-acme-key"))).resolves.toEqual({
|
||||
result: {
|
||||
validRequestReceived: false,
|
||||
model: MODEL,
|
||||
response: { text: OPENAI_PROBE_RESPONSE, stopReason: "stop", inputTokens: 3, outputTokens: 2 },
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("returns a structured error when the configured model is missing", async () => {
|
||||
const runtime = await runtimeWith({ providers: {} });
|
||||
await expect(inspectProvider(runtime, probe(OPENAI_PROBE_PROMPT, "resolved-acme-key"))).resolves.toEqual({
|
||||
result: { error: `Model ${PROVIDER_ID}/${OPENAI_MODEL_ID} is unavailable after reload.` },
|
||||
});
|
||||
expect(server.validRequestReceived()).toBe(false);
|
||||
});
|
||||
|
||||
it("returns a structured error when models.json cannot be parsed", async () => {
|
||||
const directory = await mkdtemp(join(tmpdir(), "pi-eval-provider-probe-"));
|
||||
temporaryDirectories.push(directory);
|
||||
await writeFile(join(directory, "models.json"), "{");
|
||||
const output = await inspectProvider(
|
||||
await loadConfiguredModelRuntime(directory),
|
||||
probe(OPENAI_PROBE_PROMPT, "resolved-acme-key"),
|
||||
);
|
||||
expect(output).toEqual({
|
||||
result: { error: expect.stringContaining("Failed to parse models.json") },
|
||||
});
|
||||
expect(server.validRequestReceived()).toBe(false);
|
||||
});
|
||||
|
||||
it("does not treat an unauthorized completion as a valid probe", async () => {
|
||||
const runtime = await runtimeWith(acmeModelsJson());
|
||||
await expect(inspectProvider(runtime, probe(OPENAI_PROBE_PROMPT, "wrong-key"))).resolves.toEqual({
|
||||
result: {
|
||||
validRequestReceived: false,
|
||||
model: MODEL,
|
||||
response: { text: "", stopReason: "error", inputTokens: 0, outputTokens: 0 },
|
||||
},
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("inspectAddedModel", () => {
|
||||
const PROVIDER = "openai";
|
||||
const MODEL_ID = "fixture-chat";
|
||||
const ADDED_MODEL = {
|
||||
id: MODEL_ID,
|
||||
name: "Fixture Chat",
|
||||
provider: PROVIDER,
|
||||
reasoning: true,
|
||||
input: ["text"],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 32768,
|
||||
maxTokens: 4096,
|
||||
};
|
||||
|
||||
it("loads an added model without dropping built-in models", async () => {
|
||||
const directory = await agentDirWith({
|
||||
providers: {
|
||||
[PROVIDER]: {
|
||||
models: [
|
||||
{
|
||||
id: ADDED_MODEL.id,
|
||||
name: ADDED_MODEL.name,
|
||||
reasoning: ADDED_MODEL.reasoning,
|
||||
input: ADDED_MODEL.input,
|
||||
cost: ADDED_MODEL.cost,
|
||||
contextWindow: ADDED_MODEL.contextWindow,
|
||||
maxTokens: ADDED_MODEL.maxTokens,
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
});
|
||||
await expect(inspectAddedModel(await loadConfiguredModelRuntime(directory), PROVIDER, MODEL_ID)).resolves.toEqual(
|
||||
{
|
||||
result: { model: ADDED_MODEL, existingModelsPreserved: true },
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
it("returns a structured error when the added model is missing", async () => {
|
||||
const directory = await agentDirWith({ providers: {} });
|
||||
await expect(inspectAddedModel(await loadConfiguredModelRuntime(directory), PROVIDER, MODEL_ID)).resolves.toEqual(
|
||||
{
|
||||
result: { error: `Model ${PROVIDER}/${MODEL_ID} is unavailable after reload.` },
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
it("returns a structured error when models.json cannot be parsed", async () => {
|
||||
const directory = await mkdtemp(join(tmpdir(), "pi-eval-provider-probe-"));
|
||||
temporaryDirectories.push(directory);
|
||||
await writeFile(join(directory, "models.json"), "{");
|
||||
await expect(inspectAddedModel(await loadConfiguredModelRuntime(directory), PROVIDER, MODEL_ID)).resolves.toEqual(
|
||||
{
|
||||
result: { error: expect.stringContaining("Failed to parse models.json") },
|
||||
},
|
||||
);
|
||||
});
|
||||
});
|
||||
116
third_party/pi/packages/evals/test/harness.test.ts
vendored
Normal file
116
third_party/pi/packages/evals/test/harness.test.ts
vendored
Normal file
@@ -0,0 +1,116 @@
|
||||
import { homedir } from "node:os";
|
||||
import { getDocsPath, getExamplesPath, getReadmePath } from "@earendil-works/pi-coding-agent";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { buildSystemPrompt } from "../../coding-agent/src/core/system-prompt.ts";
|
||||
import {
|
||||
applyIsolatedEnvironment,
|
||||
createPiDocumentationEvalHarness,
|
||||
DOCUMENTATION_EVAL_TOOLS,
|
||||
excludePiDocumentation,
|
||||
resolveDocumentationVariant,
|
||||
resolveModelSelection,
|
||||
verifySystemPrompt,
|
||||
} from "../src/harness.ts";
|
||||
|
||||
describe("resolveModelSelection", () => {
|
||||
it("prefers an explicit harness model", () => {
|
||||
expect(
|
||||
resolveModelSelection(
|
||||
{ provider: "anthropic", id: "claude-opus-4-6" },
|
||||
{ PI_PROVIDER: "openai-codex", PI_MODEL: "gpt-5.6-sol" },
|
||||
),
|
||||
).toEqual({ provider: "anthropic", id: "claude-opus-4-6" });
|
||||
});
|
||||
|
||||
it("uses trimmed environment defaults", () => {
|
||||
expect(resolveModelSelection(undefined, { PI_PROVIDER: " openai-codex ", PI_MODEL: " gpt-5.6-sol " })).toEqual({
|
||||
provider: "openai-codex",
|
||||
id: "gpt-5.6-sol",
|
||||
});
|
||||
});
|
||||
|
||||
it.each([{}, { PI_PROVIDER: "openai-codex" }, { PI_MODEL: "gpt-5.6-sol" }])(
|
||||
"rejects incomplete model selection",
|
||||
(environment) => {
|
||||
expect(() => resolveModelSelection(undefined, environment)).toThrow("Select a harness model explicitly");
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
describe("isolateProcessEnvironment", () => {
|
||||
it("removes runner metadata and restores the process environment", () => {
|
||||
vi.stubEnv("PI_EVAL_VARIANT", "with_docs");
|
||||
vi.stubEnv("PI_EVAL_ARTIFACT_DIR", "/tmp/artifacts");
|
||||
const oldHome = process.env.HOME;
|
||||
try {
|
||||
const restore = applyIsolatedEnvironment("/tmp/eval-home", "/tmp/eval-agent");
|
||||
try {
|
||||
expect(homedir()).toBe("/tmp/eval-home");
|
||||
expect(process.env.PI_CODING_AGENT_DIR).toBe("/tmp/eval-agent");
|
||||
expect(process.env.PI_EVAL_VARIANT).toBeUndefined();
|
||||
expect(process.env.PI_EVAL_ARTIFACT_DIR).toBeUndefined();
|
||||
} finally {
|
||||
restore();
|
||||
}
|
||||
expect(process.env.HOME).toBe(oldHome);
|
||||
expect(process.env.PI_EVAL_VARIANT).toBe("with_docs");
|
||||
} finally {
|
||||
vi.unstubAllEnvs();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("documentation variant", () => {
|
||||
it.each(["without_docs", "with_docs"] as const)("accepts %s", (variant) => {
|
||||
expect(resolveDocumentationVariant(variant)).toBe(variant);
|
||||
});
|
||||
|
||||
it.each([undefined, "", "other"])("rejects invalid variant %s", (variant) => {
|
||||
expect(() => resolveDocumentationVariant(variant)).toThrow("PI_EVAL_VARIANT");
|
||||
});
|
||||
|
||||
it("strips only the documentation routing section from the default Pi prompt", () => {
|
||||
const prompt = buildSystemPrompt({
|
||||
cwd: "/workspace",
|
||||
selectedTools: [...DOCUMENTATION_EVAL_TOOLS],
|
||||
});
|
||||
expect(prompt).toContain("\n<docs>\nPi documentation (read only");
|
||||
expect(prompt).toContain("\n<rules>\n");
|
||||
expect(prompt).toContain("\n<cwd>\n/workspace\n</cwd>");
|
||||
expect(prompt).toContain("docs/models.md");
|
||||
|
||||
const stripped = excludePiDocumentation(prompt);
|
||||
expect(stripped).toContain("\n<rules>\n");
|
||||
expect(stripped).toContain("\n<cwd>\n/workspace\n</cwd>");
|
||||
expect(stripped).not.toContain("<docs>");
|
||||
expect(stripped).not.toContain("Pi documentation");
|
||||
expect(stripped).not.toContain("docs/models.md");
|
||||
expect(stripped).not.toContain(getReadmePath());
|
||||
expect(stripped).not.toContain(getDocsPath());
|
||||
expect(stripped).not.toContain(getExamplesPath());
|
||||
});
|
||||
|
||||
it("verifies the prompt that was sent", () => {
|
||||
const prompt = buildSystemPrompt({
|
||||
cwd: "/workspace",
|
||||
selectedTools: [...DOCUMENTATION_EVAL_TOOLS],
|
||||
});
|
||||
const stripped = excludePiDocumentation(prompt);
|
||||
|
||||
expect(verifySystemPrompt(stripped, { name: "without_docs", expectedPiDocumentation: false })).toBe(stripped);
|
||||
expect(() => verifySystemPrompt(prompt, { name: "without_docs", expectedPiDocumentation: false })).toThrow(
|
||||
"does not match",
|
||||
);
|
||||
});
|
||||
|
||||
it("fails closed when prompt markers are missing", () => {
|
||||
expect(() => excludePiDocumentation("Instructions")).toThrow("no Pi documentation section");
|
||||
expect(() => excludePiDocumentation("\n<docs>\nPi documentation\n</docs>")).toThrow(
|
||||
"no working-directory section",
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects documentation harnesses outside the container sandbox", () => {
|
||||
expect(() => createPiDocumentationEvalHarness()).toThrow("isolated container sandbox");
|
||||
});
|
||||
});
|
||||
@@ -1,34 +0,0 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { resolveModelSelection } from "../src/pi-harness.ts";
|
||||
|
||||
describe("resolveModelSelection", () => {
|
||||
it("prefers an explicit harness model over environment defaults", () => {
|
||||
expect(
|
||||
resolveModelSelection(
|
||||
{ provider: "anthropic", id: "claude-opus-4-6" },
|
||||
{ PI_PROVIDER: "openai-codex", PI_MODEL: "gpt-5.6-sol" },
|
||||
),
|
||||
).toEqual({ provider: "anthropic", id: "claude-opus-4-6" });
|
||||
});
|
||||
|
||||
it("uses trimmed environment defaults when the harness has no explicit model", () => {
|
||||
expect(resolveModelSelection(undefined, { PI_PROVIDER: " openai-codex ", PI_MODEL: " gpt-5.6-sol " })).toEqual({
|
||||
provider: "openai-codex",
|
||||
id: "gpt-5.6-sol",
|
||||
});
|
||||
});
|
||||
|
||||
it.each([
|
||||
[undefined, {}],
|
||||
[undefined, { PI_PROVIDER: "openai-codex" }],
|
||||
[undefined, { PI_MODEL: "gpt-5.6-sol" }],
|
||||
[
|
||||
{ provider: "", id: "gpt-5.6-sol" },
|
||||
{ PI_PROVIDER: "openai-codex", PI_MODEL: "gpt-5.6-sol" },
|
||||
],
|
||||
] as const)("rejects an incomplete model selection", (explicitModel, environment) => {
|
||||
expect(() => resolveModelSelection(explicitModel, environment)).toThrow(
|
||||
"Select a harness model explicitly or set both PI_PROVIDER and PI_MODEL as defaults.",
|
||||
);
|
||||
});
|
||||
});
|
||||
42
third_party/pi/packages/evals/test/plan.test.ts
vendored
Normal file
42
third_party/pi/packages/evals/test/plan.test.ts
vendored
Normal file
@@ -0,0 +1,42 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { createTaskPlan, parseDiscoveredCases } from "../src/plan.ts";
|
||||
|
||||
const discovered = [{ name: "Add model > adds the model", file: "evals/models.docs.eval.ts" }];
|
||||
|
||||
describe("parseDiscoveredCases", () => {
|
||||
it("derives stable case identity from ordinary Vitest names", () => {
|
||||
expect(parseDiscoveredCases(discovered)).toEqual([
|
||||
{
|
||||
fullName: "Add model > adds the model",
|
||||
evalSet: "Add model",
|
||||
caseId: "adds the model",
|
||||
file: "evals/models.docs.eval.ts",
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("rejects ambiguous and duplicate identities", () => {
|
||||
expect(() => parseDiscoveredCases([{ name: "adds the model", file: "model.ts" }])).toThrow("<eval set> > <case>");
|
||||
expect(() => parseDiscoveredCases([...discovered, ...discovered])).toThrow("Duplicate eval case identity");
|
||||
});
|
||||
});
|
||||
|
||||
describe("createTaskPlan", () => {
|
||||
it("creates one isolated task per case, variant, model, and repetition", () => {
|
||||
const cases = parseDiscoveredCases(discovered);
|
||||
const tasks = createTaskPlan(cases, "fixture/model", 2);
|
||||
expect(tasks).toHaveLength(4);
|
||||
expect(tasks.map(({ variant, runNumber }) => [variant, runNumber])).toEqual([
|
||||
["without_docs", 1],
|
||||
["with_docs", 1],
|
||||
["with_docs", 2],
|
||||
["without_docs", 2],
|
||||
]);
|
||||
});
|
||||
|
||||
it("rejects invalid model identities and repetitions", () => {
|
||||
const cases = parseDiscoveredCases(discovered);
|
||||
expect(() => createTaskPlan(cases, "model", 1)).toThrow("provider and model");
|
||||
expect(() => createTaskPlan(cases, "fixture/model", 0)).toThrow("positive integer");
|
||||
});
|
||||
});
|
||||
208
third_party/pi/packages/evals/test/report.test.ts
vendored
Normal file
208
third_party/pi/packages/evals/test/report.test.ts
vendored
Normal file
@@ -0,0 +1,208 @@
|
||||
import { mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import type { EvalTask } from "../src/plan.ts";
|
||||
import { classifyCaseStatus, PI_SESSION_SNAPSHOT_ARTIFACT, readTaskObservation } from "../src/report.ts";
|
||||
|
||||
const temporaryDirectories: string[] = [];
|
||||
afterEach(async () => {
|
||||
await Promise.all(temporaryDirectories.splice(0).map((path) => rm(path, { recursive: true, force: true })));
|
||||
});
|
||||
|
||||
const task: EvalTask = {
|
||||
file: "evals/example.docs.eval.ts",
|
||||
fullName: "Example workflow > handles the case",
|
||||
evalSet: "Example workflow",
|
||||
caseId: "handles the case",
|
||||
variant: "without_docs",
|
||||
model: "fixture/model",
|
||||
runNumber: 1,
|
||||
};
|
||||
const SESSION = '{"type":"session"}\n';
|
||||
|
||||
type ReportAssertion = {
|
||||
status?: "passed" | "skipped" | "pending" | "failed";
|
||||
meta?: Record<string, unknown>;
|
||||
};
|
||||
|
||||
async function writeTaskReport(assertion: ReportAssertion = {}) {
|
||||
const directory = await mkdtemp(join(tmpdir(), "pi-eval-report-test-"));
|
||||
temporaryDirectories.push(directory);
|
||||
const reportPath = join(directory, "vitest.json");
|
||||
const status = assertion.status ?? "passed";
|
||||
await writeFile(
|
||||
reportPath,
|
||||
JSON.stringify({
|
||||
numFailedTests: 0,
|
||||
numPassedTests: status === "passed" ? 1 : 0,
|
||||
numPendingTests: status === "pending" || status === "skipped" ? 1 : 0,
|
||||
numTodoTests: 0,
|
||||
numTotalTests: 1,
|
||||
startTime: 0,
|
||||
success: true,
|
||||
testResults: [
|
||||
{
|
||||
message: "",
|
||||
name: "/repo/packages/evals/evals/example.docs.eval.ts",
|
||||
status: "passed",
|
||||
assertionResults: [
|
||||
{
|
||||
ancestorTitles: [task.evalSet],
|
||||
fullName: `${task.evalSet} ${task.caseId}`,
|
||||
status,
|
||||
title: task.caseId,
|
||||
failureMessages: [],
|
||||
meta: assertion.meta ?? {},
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
return { directory, reportPath };
|
||||
}
|
||||
|
||||
function scoredMeta(overrides?: {
|
||||
avgScore?: number | null;
|
||||
model?: string;
|
||||
errors?: Array<Record<string, string>>;
|
||||
artifacts?: Record<string, string>;
|
||||
}): Record<string, unknown> {
|
||||
return {
|
||||
eval: {
|
||||
avgScore: overrides && "avgScore" in overrides ? overrides.avgScore : 0.5,
|
||||
scores: [{ name: "StructuredOutputJudge", score: 0.5 }],
|
||||
thresholdFailed: false,
|
||||
},
|
||||
harness: {
|
||||
name: "without_docs",
|
||||
run: {
|
||||
output: { ok: true },
|
||||
session: { events: [{ type: "message", role: "user", content: "prompt" }] },
|
||||
usage: {
|
||||
provider: "fixture",
|
||||
model: overrides?.model ?? "model",
|
||||
inputTokens: 10,
|
||||
outputTokens: 5,
|
||||
totalTokens: 15,
|
||||
toolCalls: 1,
|
||||
metadata: { cacheReadTokens: 2, cacheWriteTokens: 3, estimatedCostUsd: 0.01 },
|
||||
},
|
||||
timings: { totalMs: 1234 },
|
||||
artifacts: overrides?.artifacts ?? { runId: "run-1", [PI_SESSION_SNAPSHOT_ARTIFACT]: SESSION },
|
||||
errors: overrides?.errors ?? [],
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function readObservation(assertion?: ReportAssertion) {
|
||||
const { directory, reportPath } = await writeTaskReport(assertion);
|
||||
return { directory, observation: await readTaskObservation(task, reportPath, directory) };
|
||||
}
|
||||
|
||||
describe("classifyCaseStatus", () => {
|
||||
it.each(["skipped", "todo", "disabled"] as const)("maps %s to skipped", (status) => {
|
||||
expect(classifyCaseStatus(status)).toBe("skipped");
|
||||
});
|
||||
|
||||
it("maps failed infrastructure to errored", () => {
|
||||
expect(classifyCaseStatus("failed")).toBe("errored");
|
||||
});
|
||||
});
|
||||
|
||||
describe("readTaskObservation", () => {
|
||||
it("preserves a skipped outcome when no harness run exists", async () => {
|
||||
const { observation } = await readObservation({ status: "skipped" });
|
||||
expect(observation).toMatchObject({ outcome: "skipped" });
|
||||
});
|
||||
|
||||
it("preserves a pending outcome when no harness run exists", async () => {
|
||||
const { observation } = await readObservation({ status: "pending" });
|
||||
expect(observation).toMatchObject({ outcome: "pending" });
|
||||
});
|
||||
|
||||
it("preserves metrics from a failed eval with a partial harness run", async () => {
|
||||
const { directory, observation } = await readObservation({
|
||||
status: "failed",
|
||||
meta: scoredMeta({ errors: [{ message: "Prompt verification failed" }] }),
|
||||
});
|
||||
expect(observation).toEqual({
|
||||
evalSet: task.evalSet,
|
||||
caseId: task.caseId,
|
||||
variant: task.variant,
|
||||
model: task.model,
|
||||
runNumber: task.runNumber,
|
||||
outcome: "errored",
|
||||
inputTokens: 10,
|
||||
outputTokens: 5,
|
||||
cacheReadTokens: 2,
|
||||
cacheWriteTokens: 3,
|
||||
totalTokens: 15,
|
||||
toolCalls: 1,
|
||||
totalMs: 1234,
|
||||
estimatedCostUsd: 0.01,
|
||||
});
|
||||
const hashes = await readdir(join(directory, task.variant, "sessions"));
|
||||
await expect(
|
||||
readFile(join(directory, task.variant, "sessions", hashes[0], "session.jsonl"), "utf8"),
|
||||
).resolves.toBe(SESSION);
|
||||
});
|
||||
|
||||
it("records an errored outcome when a passed eval has no harness run", async () => {
|
||||
const { observation } = await readObservation({ status: "passed" });
|
||||
expect(observation).toMatchObject({ outcome: "errored" });
|
||||
});
|
||||
|
||||
it("reads a scored harness run and persists the session artifact", async () => {
|
||||
const { directory, observation } = await readObservation({ meta: scoredMeta() });
|
||||
expect(observation).toEqual({
|
||||
evalSet: task.evalSet,
|
||||
caseId: task.caseId,
|
||||
variant: task.variant,
|
||||
model: task.model,
|
||||
runNumber: task.runNumber,
|
||||
outcome: "scored",
|
||||
score: 0.5,
|
||||
inputTokens: 10,
|
||||
outputTokens: 5,
|
||||
cacheReadTokens: 2,
|
||||
cacheWriteTokens: 3,
|
||||
totalTokens: 15,
|
||||
toolCalls: 1,
|
||||
totalMs: 1234,
|
||||
estimatedCostUsd: 0.01,
|
||||
});
|
||||
const hashes = await readdir(join(directory, task.variant, "sessions"));
|
||||
expect(hashes).toHaveLength(1);
|
||||
await expect(
|
||||
readFile(join(directory, task.variant, "sessions", hashes[0], "session.jsonl"), "utf8"),
|
||||
).resolves.toBe(SESSION);
|
||||
});
|
||||
|
||||
it("treats a zero score as scored data", async () => {
|
||||
const { observation } = await readObservation({ meta: scoredMeta({ avgScore: 0 }) });
|
||||
expect(observation).toMatchObject({ outcome: "scored", score: 0 });
|
||||
});
|
||||
|
||||
it("records an unscored outcome when a completed eval has no score", async () => {
|
||||
const { observation } = await readObservation({ meta: scoredMeta({ avgScore: null }) });
|
||||
expect(observation).toMatchObject({ outcome: "unscored" });
|
||||
});
|
||||
|
||||
it("records an errored outcome when the reported model does not match the task", async () => {
|
||||
const { observation } = await readObservation({ meta: scoredMeta({ model: "other" }) });
|
||||
expect(observation).toMatchObject({ outcome: "errored" });
|
||||
});
|
||||
|
||||
it("records an errored outcome when a completed harness run contains errors", async () => {
|
||||
const { observation } = await readObservation({ meta: scoredMeta({ errors: [{ message: "boom" }] }) });
|
||||
expect(observation).toMatchObject({ outcome: "errored" });
|
||||
});
|
||||
|
||||
it("still scores a completed eval when the session artifact is missing", async () => {
|
||||
const { observation } = await readObservation({ meta: scoredMeta({ artifacts: { runId: "run-1" } }) });
|
||||
expect(observation).toMatchObject({ outcome: "scored", score: 0.5 });
|
||||
});
|
||||
});
|
||||
@@ -1,103 +0,0 @@
|
||||
import { mkdtemp, readFile, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { expect, it } from "vitest";
|
||||
import {
|
||||
persistEvalArtifactReferences,
|
||||
recordEvalSessionArtifact,
|
||||
recordEvalSourceArtifact,
|
||||
} from "../../src/vitest-evals/artifacts.ts";
|
||||
|
||||
it("records session and source artifacts against the explicit test task", async ({ task }) => {
|
||||
const runId = "run-1";
|
||||
await recordEvalSessionArtifact(task, {
|
||||
artifacts: { runId, piSessionJsonl: '{"type":"session"}\n' },
|
||||
});
|
||||
await recordEvalSourceArtifact(task, runId, {
|
||||
name: "hello.ts",
|
||||
contentType: "text/typescript",
|
||||
body: "export default function () {}\n",
|
||||
bodyEncoding: "utf-8",
|
||||
});
|
||||
|
||||
expect(task.artifacts).toContainEqual(
|
||||
expect.objectContaining({
|
||||
type: "@earendil-works/pi-evals:session",
|
||||
runId,
|
||||
attachments: [
|
||||
expect.objectContaining({
|
||||
name: "session.jsonl",
|
||||
body: '{"type":"session"}\n',
|
||||
bodyEncoding: "utf-8",
|
||||
contentType: "application/jsonl",
|
||||
}),
|
||||
],
|
||||
}),
|
||||
);
|
||||
expect(task.artifacts).toContainEqual(
|
||||
expect.objectContaining({
|
||||
type: "@earendil-works/pi-evals:source",
|
||||
runId,
|
||||
attachments: [
|
||||
expect.objectContaining({
|
||||
name: "hello.ts",
|
||||
body: "export default function () {}\n",
|
||||
bodyEncoding: "utf-8",
|
||||
contentType: "text/typescript",
|
||||
}),
|
||||
],
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it("persists and selects attachments belonging to the reported run", async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), "pi-eval-artifact-report-test-"));
|
||||
try {
|
||||
const references = await persistEvalArtifactReferences(
|
||||
[
|
||||
{
|
||||
type: "@earendil-works/pi-evals:session",
|
||||
runId: "run-1",
|
||||
attachments: [
|
||||
{
|
||||
name: "session.jsonl",
|
||||
body: '{"type":"session"}\n',
|
||||
bodyEncoding: "utf-8",
|
||||
contentType: "application/jsonl",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
type: "@earendil-works/pi-evals:session",
|
||||
runId: "run-2",
|
||||
attachments: [],
|
||||
},
|
||||
{
|
||||
type: "@earendil-works/pi-evals:source",
|
||||
runId: "run-1",
|
||||
attachments: [
|
||||
{
|
||||
name: "hello.ts",
|
||||
body: "export default function () {}\n",
|
||||
bodyEncoding: "utf-8",
|
||||
contentType: "text/typescript",
|
||||
},
|
||||
],
|
||||
},
|
||||
{ type: "internal:annotation", annotation: { message: "other", type: "info" } },
|
||||
],
|
||||
"run-1",
|
||||
root,
|
||||
);
|
||||
expect(references).toEqual([
|
||||
{ name: "session.jsonl", path: expect.stringMatching(/^sessions\/[a-f0-9]{64}\/session\.jsonl$/) },
|
||||
{ name: "hello.ts", path: expect.stringMatching(/^sources\/[a-f0-9]{64}\/hello\.ts$/) },
|
||||
]);
|
||||
for (const { name, path } of references) {
|
||||
const expected = name === "session.jsonl" ? '{"type":"session"}\n' : "export default function () {}\n";
|
||||
expect(await readFile(join(root, path), "utf8")).toBe(expected);
|
||||
}
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
@@ -1,94 +0,0 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { createHarness, type HarnessContext } from "vitest-evals/harness";
|
||||
import {
|
||||
deriveEvalGroupKey,
|
||||
EVAL_HARNESS_ITERATION_ARTIFACT,
|
||||
evalHarnessTable,
|
||||
parseEvalHarnessIterationArtifact,
|
||||
} from "../../src/vitest-evals/harness-table.ts";
|
||||
|
||||
describe("deriveEvalGroupKey", () => {
|
||||
it("combines a trimmed string input ID with repetition", () => {
|
||||
expect(deriveEvalGroupKey({ id: " input-1 ", prompt: "hello" }, 2)).toBe(JSON.stringify(["input-1", 2]));
|
||||
});
|
||||
|
||||
it("hashes canonical JSON independently of object key order", () => {
|
||||
expect(deriveEvalGroupKey({ first: 1, second: [true, "value"] }, 1)).toBe(
|
||||
deriveEvalGroupKey({ second: [true, "value"], first: 1 }, 1),
|
||||
);
|
||||
expect(deriveEvalGroupKey({ first: 1 }, 1)).not.toBe(deriveEvalGroupKey({ first: 2 }, 1));
|
||||
expect(deriveEvalGroupKey({ first: 1 }, 1)).not.toBe(deriveEvalGroupKey({ first: 1 }, 2));
|
||||
expect(deriveEvalGroupKey(["first", "second"], 1)).not.toBe(deriveEvalGroupKey(["second", "first"], 1));
|
||||
});
|
||||
|
||||
it("rejects non-JSON and circular input", () => {
|
||||
const circular: { self?: unknown } = {};
|
||||
circular.self = circular;
|
||||
expect(() => deriveEvalGroupKey(new Date(0), 1)).toThrow("only plain objects and arrays");
|
||||
expect(() => deriveEvalGroupKey(Array(1), 1)).toThrow("must not be sparse");
|
||||
expect(() => deriveEvalGroupKey(circular, 1)).toThrow("must not contain circular references");
|
||||
});
|
||||
});
|
||||
|
||||
function createFakeHarness(name: string) {
|
||||
return createHarness<{ id: string }, { harness: string; inputId: string }>({
|
||||
name,
|
||||
run: ({ input }) => ({
|
||||
output: { harness: name, inputId: input.id },
|
||||
events: [
|
||||
{ type: "message", role: "user", content: input.id },
|
||||
{ type: "message", role: "assistant", content: name },
|
||||
],
|
||||
}),
|
||||
});
|
||||
}
|
||||
|
||||
const harnessTable = evalHarnessTable("local multi-harness eval", {
|
||||
baseline: createFakeHarness("withoutSkill"),
|
||||
candidates: [createFakeHarness("withSkill")],
|
||||
repetitions: 2,
|
||||
});
|
||||
|
||||
describe("evalHarnessTable", () => {
|
||||
it("plans repetitions in declaration order", () => {
|
||||
expect(harnessTable.map(({ name, repetition }) => ({ name, repetition }))).toEqual([
|
||||
{ name: "withoutSkill", repetition: 1 },
|
||||
{ name: "withSkill", repetition: 1 },
|
||||
{ name: "withoutSkill", repetition: 2 },
|
||||
{ name: "withSkill", repetition: 2 },
|
||||
]);
|
||||
});
|
||||
|
||||
it("accepts a singular candidate", () => {
|
||||
const rows = evalHarnessTable("singular candidate", {
|
||||
baseline: createFakeHarness("baseline"),
|
||||
candidate: createFakeHarness("candidate"),
|
||||
});
|
||||
|
||||
expect(rows.map(({ name }) => name)).toEqual(["baseline", "candidate"]);
|
||||
});
|
||||
|
||||
it("attaches iteration metadata to every wrapped harness run", async () => {
|
||||
for (const row of harnessTable) {
|
||||
const artifacts: HarnessContext["artifacts"] = {};
|
||||
const context: HarnessContext = {
|
||||
artifacts,
|
||||
setArtifact(name, value) {
|
||||
artifacts[name] = value;
|
||||
},
|
||||
};
|
||||
const result = await row.harness.run({ id: "first" }, context);
|
||||
|
||||
expect(result.output).toEqual({ harness: row.name, inputId: "first" });
|
||||
expect(parseEvalHarnessIterationArtifact(result.artifacts?.[EVAL_HARNESS_ITERATION_ARTIFACT])).toEqual({
|
||||
schemaVersion: 1,
|
||||
evalSet: "local multi-harness eval",
|
||||
groupKey: deriveEvalGroupKey({ id: "first" }, row.repetition),
|
||||
harness: row.name,
|
||||
baseline: "withoutSkill",
|
||||
candidates: ["withSkill"],
|
||||
repetition: row.repetition,
|
||||
});
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -1,232 +0,0 @@
|
||||
import { stripVTControlCharacters } from "node:util";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
formatHarnessComparisonReport,
|
||||
type HarnessObservation,
|
||||
summarizeHarnessComparisons,
|
||||
} from "../../src/vitest-evals/summary.ts";
|
||||
|
||||
type ObservationResult = "passed" | "failed" | Exclude<HarnessObservation["outcome"], "scored">;
|
||||
|
||||
function observation(
|
||||
harness: string,
|
||||
testName: string,
|
||||
result: ObservationResult,
|
||||
metrics: Pick<HarnessObservation, "totalTokens" | "totalMs" | "estimatedCostUsd"> = {},
|
||||
baseline = "without-tools",
|
||||
candidates: string[] = ["with-tools"],
|
||||
): HarnessObservation {
|
||||
const base = {
|
||||
evalSet: "tool access",
|
||||
groupKey: JSON.stringify([testName, 1]),
|
||||
testName,
|
||||
file: "src/tool-access.eval.ts",
|
||||
harness,
|
||||
baseline,
|
||||
candidates,
|
||||
repetition: 1,
|
||||
...metrics,
|
||||
};
|
||||
if (result === "passed" || result === "failed") {
|
||||
return { ...base, outcome: "scored", score: result === "passed" ? 1 : 0 };
|
||||
}
|
||||
return { ...base, outcome: result };
|
||||
}
|
||||
|
||||
describe("summarizeHarnessComparisons", () => {
|
||||
it("computes paired correctness lift separately from efficiency deltas", () => {
|
||||
const report = summarizeHarnessComparisons([
|
||||
observation("without-tools", "create", "failed", {
|
||||
totalTokens: 100,
|
||||
totalMs: 1000,
|
||||
estimatedCostUsd: 0.01,
|
||||
}),
|
||||
observation("with-tools", "create", "passed", {
|
||||
totalTokens: 120,
|
||||
totalMs: 800,
|
||||
estimatedCostUsd: 0.02,
|
||||
}),
|
||||
observation("without-tools", "inspect", "passed", { totalTokens: 200 }),
|
||||
observation("with-tools", "inspect", "passed", { totalTokens: 180 }),
|
||||
]);
|
||||
|
||||
expect(report.evalSets).toHaveLength(1);
|
||||
expect(report.evalSets[0]?.comparisons).toEqual([
|
||||
expect.objectContaining({
|
||||
baseline: "without-tools",
|
||||
candidate: "with-tools",
|
||||
correctness: {
|
||||
totalPairs: 2,
|
||||
eligiblePairs: 2,
|
||||
baselinePassRate: 0.5,
|
||||
candidatePassRate: 1,
|
||||
lift: 0.5,
|
||||
baselineWins: 0,
|
||||
candidateWins: 1,
|
||||
ties: 1,
|
||||
},
|
||||
totalTokens: {
|
||||
totalPairs: 2,
|
||||
eligiblePairs: 2,
|
||||
baselineMean: 150,
|
||||
candidateMean: 150,
|
||||
meanDelta: 0,
|
||||
},
|
||||
totalMs: {
|
||||
totalPairs: 2,
|
||||
eligiblePairs: 1,
|
||||
baselineMean: 1000,
|
||||
candidateMean: 800,
|
||||
meanDelta: -200,
|
||||
},
|
||||
estimatedCostUsd: {
|
||||
totalPairs: 2,
|
||||
eligiblePairs: 1,
|
||||
baselineMean: 0.01,
|
||||
candidateMean: 0.02,
|
||||
meanDelta: 0.01,
|
||||
},
|
||||
}),
|
||||
]);
|
||||
expect(report.diagnostics).toEqual([]);
|
||||
});
|
||||
|
||||
it("reports missing observations without coercing them to failures or zero telemetry", () => {
|
||||
const report = summarizeHarnessComparisons([
|
||||
observation("without-tools", "create", "failed"),
|
||||
observation("with-tools", "create", "passed"),
|
||||
observation("without-tools", "inspect", "passed"),
|
||||
]);
|
||||
const comparison = report.evalSets[0]?.comparisons[0];
|
||||
|
||||
expect(comparison?.correctness).toEqual({
|
||||
totalPairs: 2,
|
||||
eligiblePairs: 1,
|
||||
baselinePassRate: 0,
|
||||
candidatePassRate: 1,
|
||||
lift: 1,
|
||||
baselineWins: 0,
|
||||
candidateWins: 1,
|
||||
ties: 0,
|
||||
});
|
||||
expect(comparison?.totalTokens).toEqual({
|
||||
totalPairs: 2,
|
||||
eligiblePairs: 0,
|
||||
baselineMean: null,
|
||||
candidateMean: null,
|
||||
meanDelta: null,
|
||||
});
|
||||
expect(report.diagnostics).toContainEqual(
|
||||
expect.objectContaining({
|
||||
testName: "inspect",
|
||||
harness: "with-tools",
|
||||
reason: "missing-observation",
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it("keeps identical inputs in different test files separate", () => {
|
||||
const report = summarizeHarnessComparisons([
|
||||
observation("without-tools", "shared", "failed"),
|
||||
observation("with-tools", "shared", "passed"),
|
||||
{ ...observation("without-tools", "shared", "passed"), file: "src/other.eval.ts" },
|
||||
{ ...observation("with-tools", "shared", "passed"), file: "src/other.eval.ts" },
|
||||
]);
|
||||
|
||||
expect(report.evalSets[0]?.comparisons[0]?.correctness).toEqual(
|
||||
expect.objectContaining({ totalPairs: 2, eligiblePairs: 2 }),
|
||||
);
|
||||
expect(report.diagnostics).toEqual([]);
|
||||
});
|
||||
|
||||
it("does not score harness errors as correctness failures", () => {
|
||||
const report = summarizeHarnessComparisons([
|
||||
observation("without-tools", "create", "errored", { totalTokens: 100 }),
|
||||
observation("with-tools", "create", "passed", { totalTokens: 100 }),
|
||||
]);
|
||||
|
||||
expect(report.evalSets[0]?.comparisons[0]?.correctness).toEqual(
|
||||
expect.objectContaining({ totalPairs: 1, eligiblePairs: 0 }),
|
||||
);
|
||||
expect(report.evalSets[0]?.comparisons[0]?.totalTokens.eligiblePairs).toBe(0);
|
||||
expect(report.diagnostics).toContainEqual(
|
||||
expect.objectContaining({ harness: "without-tools", reason: "harness-error" }),
|
||||
);
|
||||
});
|
||||
|
||||
it("does not derive correctness from completed Vitest tests without judge scores", () => {
|
||||
const withoutScore = observation("without-tools", "create", "unscored");
|
||||
const withScore = observation("with-tools", "create", "unscored");
|
||||
|
||||
const report = summarizeHarnessComparisons([withoutScore, withScore]);
|
||||
|
||||
expect(report.evalSets[0]?.comparisons[0]?.correctness.eligiblePairs).toBe(0);
|
||||
expect(report.diagnostics).toEqual([
|
||||
expect.objectContaining({ harness: "with-tools", reason: "missing-score" }),
|
||||
expect.objectContaining({ harness: "without-tools", reason: "missing-score" }),
|
||||
]);
|
||||
});
|
||||
|
||||
it("compares each candidate with the declared baseline", () => {
|
||||
const candidates = ["second", "third"];
|
||||
const report = summarizeHarnessComparisons([
|
||||
observation("first", "input", "passed", {}, "first", candidates),
|
||||
observation("second", "input", "passed", {}, "first", candidates),
|
||||
observation("third", "input", "passed", {}, "first", candidates),
|
||||
]);
|
||||
|
||||
expect(report.evalSets[0]?.comparisons.map(({ baseline, candidate }) => [baseline, candidate])).toEqual([
|
||||
["first", "second"],
|
||||
["first", "third"],
|
||||
]);
|
||||
});
|
||||
|
||||
it("retains a declared harness with no completed observations", () => {
|
||||
const report = summarizeHarnessComparisons([observation("without-tools", "create", "failed")]);
|
||||
|
||||
expect(report.evalSets[0]?.comparisons).toHaveLength(1);
|
||||
expect(report.evalSets[0]?.comparisons[0]?.correctness.eligiblePairs).toBe(0);
|
||||
expect(report.diagnostics).toContainEqual(
|
||||
expect.objectContaining({
|
||||
testName: "create",
|
||||
harness: "with-tools",
|
||||
reason: "missing-observation",
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it("reports duplicate and unscorable observations once across multiple harness pairs", () => {
|
||||
const candidates = ["second", "third"];
|
||||
const report = summarizeHarnessComparisons([
|
||||
observation("first", "duplicate", "passed", {}, "first", candidates),
|
||||
observation("first", "duplicate", "failed", {}, "first", candidates),
|
||||
observation("second", "duplicate", "passed", {}, "first", candidates),
|
||||
observation("third", "duplicate", "passed", {}, "first", candidates),
|
||||
observation("first", "skipped", "skipped", {}, "first", candidates),
|
||||
observation("second", "skipped", "passed", {}, "first", candidates),
|
||||
observation("third", "skipped", "passed", {}, "first", candidates),
|
||||
]);
|
||||
|
||||
expect(report.diagnostics.filter(({ reason }) => reason === "duplicate-observation")).toEqual([
|
||||
expect.objectContaining({ testName: "duplicate", harness: "first" }),
|
||||
]);
|
||||
expect(report.diagnostics.filter(({ reason }) => reason === "unscorable-outcome")).toEqual([
|
||||
expect.objectContaining({ testName: "skipped", harness: "first" }),
|
||||
]);
|
||||
});
|
||||
|
||||
it("formats lift and telemetry availability for the terminal report", () => {
|
||||
const report = summarizeHarnessComparisons([
|
||||
observation("without-tools", "create", "failed", { totalMs: 34853.7 }),
|
||||
observation("with-tools", "create", "passed", { totalMs: 30694.2 }),
|
||||
]);
|
||||
|
||||
const formatted = stripVTControlCharacters(formatHarnessComparisonReport(report));
|
||||
expect(formatted).toContain("Eval Comparisons");
|
||||
expect(formatted).toContain(" Baseline without-tools");
|
||||
expect(formatted).toContain("Candidate with-tools (1/1 pairs)");
|
||||
expect(formatted).toContain("Pass rate +100.0 pp (candidate 100.0%, baseline 0.0%)");
|
||||
expect(formatted).toContain(" Tokens unavailable");
|
||||
expect(formatted).toContain(" Latency -4159.5ms (candidate 30694.2ms, baseline 34853.7ms)");
|
||||
});
|
||||
});
|
||||
2
third_party/pi/packages/evals/tsconfig.json
vendored
2
third_party/pi/packages/evals/tsconfig.json
vendored
@@ -6,5 +6,5 @@
|
||||
"moduleResolution": "NodeNext",
|
||||
"types": ["node", "vitest"]
|
||||
},
|
||||
"include": ["src/**/*.ts"]
|
||||
"include": ["src/**/*.ts", "evals/**/*.ts", "docker/**/*.ts", "test/**/*.ts", "*.config.ts"]
|
||||
}
|
||||
|
||||
20
third_party/pi/packages/evals/vitest.config.ts
vendored
20
third_party/pi/packages/evals/vitest.config.ts
vendored
@@ -1,20 +0,0 @@
|
||||
import { defineConfig, mergeConfig } from "vitest/config";
|
||||
import baseConfig, { workspaceSourcePaths } from "../../vitest.base.ts";
|
||||
|
||||
export default mergeConfig(
|
||||
baseConfig,
|
||||
defineConfig({
|
||||
test: {
|
||||
environment: "node",
|
||||
fileParallelism: false,
|
||||
include: ["src/**/*.eval.ts"],
|
||||
testTimeout: 120000,
|
||||
hookTimeout: 30000,
|
||||
setupFiles: ["./src/vitest-evals/setup.ts"],
|
||||
reporters: ["vitest-evals/reporter", "./src/vitest-evals/reporter.ts"],
|
||||
},
|
||||
resolve: {
|
||||
alias: [{ find: /^@earendil-works\/pi-coding-agent$/, replacement: workspaceSourcePaths.codingAgentIndex }],
|
||||
},
|
||||
}),
|
||||
);
|
||||
46
third_party/pi/packages/evals/vitest.evals.config.ts
vendored
Normal file
46
third_party/pi/packages/evals/vitest.evals.config.ts
vendored
Normal file
@@ -0,0 +1,46 @@
|
||||
import { defineConfig, mergeConfig } from "vitest/config";
|
||||
import baseConfig, { workspaceSourcePaths } from "../../vitest.base.ts";
|
||||
|
||||
const evalConfig = defineConfig({
|
||||
test: {
|
||||
watch: false,
|
||||
passWithNoTests: false,
|
||||
pool: "forks",
|
||||
maxWorkers: 1,
|
||||
fileParallelism: false,
|
||||
projects: [
|
||||
{
|
||||
extends: true,
|
||||
test: {
|
||||
name: "docs",
|
||||
include: ["evals/**/*.docs.eval.ts"],
|
||||
sequence: { concurrent: false },
|
||||
testTimeout: 300_000,
|
||||
hookTimeout: 300_000,
|
||||
},
|
||||
},
|
||||
{
|
||||
extends: true,
|
||||
test: {
|
||||
name: "host",
|
||||
include: ["evals/**/*.eval.ts"],
|
||||
exclude: ["evals/**/*.docs.eval.ts"],
|
||||
sequence: { concurrent: false },
|
||||
testTimeout: 300_000,
|
||||
hookTimeout: 300_000,
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
});
|
||||
|
||||
const localConfig = mergeConfig(
|
||||
baseConfig,
|
||||
defineConfig({
|
||||
resolve: {
|
||||
alias: [{ find: /^@earendil-works\/pi-coding-agent$/, replacement: workspaceSourcePaths.codingAgentIndex }],
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
export default process.env.PI_EVAL_CONTAINER === "1" ? evalConfig : mergeConfig(localConfig, evalConfig);
|
||||
Reference in New Issue
Block a user