alexion/dev
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Squashed 'third_party/pi/' content from commit d981de122

Browse Source 
git-subtree-dir: third_party/pi
git-subtree-split: d981de1229ef899957bbe968bc8dcda02a21f477
This commit is contained in:
alexion
2026-09-14 10:40:08 -04:00
commit 67b361f0e9
1669 changed files with 397627 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

214
packages/agent/docs/mobile-handoff/01-harness/01-delta/FINDINGS.md Normal file
View File

@@ -0,0 +1,214 @@
# 01-delta: known defects and measured findings
Everything here was found **after** `delta.md` and the prototype implementation beside it were written. None of it is applied to the code in this directory. Production now lives in `packages/chord/src/delta/index.ts`: flush-time dirty tracking resolves D1, and production remeasurement closes D2 without a new API. See [`append-decision.md`](append-decision.md). Each item below remains the original problem, evidence, and candidate-fix record.
Every number was measured with **`node --experimental-strip-types`**, not `tsx`.
That matters: tsx transpiles through esbuild and inflated the same benchmark by
**2.6x** (183 µs vs 63 µs per write). Do not benchmark this module through a
transpiler.
---
## D1. Interleaved paths defeat coalescing (correctness-adjacent, unbounded memory)
**Severity: high.** This is the one to fix first.
### Problem
`flush()` coalesces only **adjacent** ops. A producer that alternates between two
paths therefore never coalesces anything, and the ops array grows with the number
of writes rather than with the size of the value.
This is not hypothetical. It is exactly what bash does: it updates the output text
and a byte counter on every chunk.
```ts
const next = t.state.content[0].text + chunk;
t.state.content[0].text = next.length > CAP ? next.slice(next.length - CAP) : next;
t.state.truncation.totalBytes += chunk.length; // <- the other path
```
### Evidence
50 KB window, 200-byte chunks, no flush in between:
| held-back writes | ops | bytes |
| --- | --- | --- |
| 1 | 3 | 0.3 KB |
| 100 | 201 | 26 KB |
| **1000** | **2001** | **264 KB** |
Without interleaving the same workload gives 1 op and 51 KB. The window is
bounded at 50 KB; the ops describing it are not.
### Why it matters beyond memory
Any design where the sink **holds back** — rate limiting, backpressure, a slow
consumer — depends on held-back writes costing nothing. They currently cost
everything. See `../../04-tool-output/rate-limiting.md`.
### Candidate fix
Fold at **record** time, keyed by path rather than by adjacency: one slot per
path holding at most two ops (`[s]`, `[d]`, `[t]`, `[a]`, or `[t, a]`), emitted in
first-touch order.
A prototype reached **2 ops / 51 KB** on the same workload. It is not included
here because it was not verified to the standard the rest of this unit meets.
**Four folding rules, each a specific claim about the vocabulary:**
1. A later `s` supersedes an earlier `s` on the same path — but **not** an earlier
`d`, which must survive or the key changes position on reinsertion (§5.1).
2. Consecutive `a` merge. Once the merged append is **strictly longer** than the
value, the window has been sliding and the append is the more expensive
spelling, so collapse to `s`. Strictly: while a string is still *growing*
toward its cap the append equals the value, and collapsing there would resend
the whole string every batch.
3. `t` and `a` on one path **commute** — the truncate drops from the front, the
append adds to the end — so a rolling window's `t,a,t,a,...` folds to one of
each. Without this, pairwise merging never fires at all.
4. `t` or `a` following a pending `s` changes nothing: the `s` already means "this
path becomes its current value".
**The trap that prototype hit.** Slots emit in first-touch order, which loses the
fact that a child write happened *before* a later parent write:
```
script : a={x:1}; a.b=99; a={c:2}
producer: {"a":{"c":2}}
replica : {"a":{"c":2,"b":99}} <- WRONG
```
A set or delete must invalidate every pending slot **below** its path, at record
time. The existing `verify-dead` style test passed 1190 sequences while this was
broken, because its generator only nested one level. Any test for this must
generate genuinely nested paths with parent overwrites interleaved between child
writes.
---
## D2. `overlap()` was 94.5% of prototype tracker time — closed
**Historical severity: high. Production decision:** do not add an explicit append/truncate API. The prototype profile was dominated by a slow `startsWith` path; production uses a flattened-slice comparison and measured 17.8–18.7 µs per 200 KB assistant append flush and 2.43–2.46 µs per 50 KB rolling-window flush locally. See [`append-decision.md`](append-decision.md). The original evidence follows.
### Problem
The tracker infers "you appended and evicted" by searching two 50 KB strings.
Each new value produced by `next.slice(-CAP)` is a V8 `SlicedString` — a lazy
substring — and `indexOf`/`endsWith` force it to **flatten**, copying 50 KB per
call.
### Evidence
CPU profile of a 20 000-write rolling-window loop, node, flush at the end:
```
94.5% overlap
2.7% (anon)
1.2% (garbage collector)
0.5% set
0.3% emit
0.0% fold
```
And the flattening is directly observable:
| `overlap()` on a 50 KB window | µs |
| --- | --- |
| the same two strings reused | 37 |
| **a fresh sliced string each call** | **149** |
An earlier benchmark reported 29 µs because it reused two already-flattened
strings 2000 times — a case that never occurs in the real loop. **Do not benchmark
string operations against a reused fixture.**
### Candidate fix
Do not infer what the producer already knows. `ToolOutput` performs the windowing,
so it can emit the ops directly rather than assigning a new string and having the
tracker search for the difference:
```ts
out.append(chunk); // -> ["t", path, dropped], ["a", path, chunk]
```
Two integers — chars appended, chars evicted — describe everything that happened
between any two observations of a bounded window. A prototype of this shape held
**exactly the window** in memory under every load, with no ops array, no overlap
detection and no folding rules at all.
This does **not** remove `overlap`: it stays for the general case where a producer
assigns a whole new string and the delta genuinely has to be discovered. It
removes it from the hot path.
### Do not
Do not "optimise" `overlap` itself before doing the above. Its algorithm is
already correct and bounded (`maxOverlapScan`); the cost is string flattening,
which no probe strategy avoids.
---
## D3. Nested-path proxy overhead is next, and currently invisible
**Severity: unknown — measure before acting.**
`overlap` dominates so completely that nothing else shows in a profile. But two
benchmarks disagree in a way that points at path depth:
| | µs per write |
| --- | --- |
| top-level path (`{ text }`) | 63 |
| nested path (`{ content: [{ text }] }`) | 244 |
4x for depth alone. Each nested property access wraps a child proxy and builds a
cache key by joining the path, so a read of `content[0].text` allocates on every
access.
**Re-profile only if production delta cost becomes material.** This may be the next bottleneck or it may be
nothing; the current numbers cannot distinguish them because `overlap` swamps both.
---
## D4. Things measured and found NOT worth doing
Recorded so nobody repeats the work. All were under **1.5% combined** in the
profile above.
**A path trie replacing `JSON.stringify(path)` keys.** Genuinely faster in
isolation — 6x on the `LaneSnapshot` shape, 93x at 500 distinct paths — because
stringify cost scales with path depth while a trie walk is one `Map.get` per
segment. But the whole slot layer is 0.5% of runtime. Build it only if a current production profile says so.
**Trie as storage vs trie as index.** If a trie is ever built, it should map path
to a **position in the ops array**, not hold the ops. Holding them loses first-touch
ordering and forces a sort at flush to rebuild it. Measured difference: none
(1.4x / 0.9x / 1.0x). The argument is structural, not performance.
**A `maxDepth` heuristic to skip descendant scanning.** Rejected: the flat map was
losing on the common case too, so the heuristic would have hidden a real cost
rather than removed it.
**Eager string concatenation in the `a`+`a` merge.** Real (it rebuilds a growing
string per write) but worth ~5%. Compare lengths and materialise only when
emitting.
---
## D5. Measurement errors made while producing these numbers
Listed because each one produced a confident wrong conclusion, and the same traps
are still there.
- **Benchmarking through `tsx`** — inflated everything 2.6x.
- **Reusing a flattened string** in a string benchmark — hid the real cost 4x.
- **Repetitive fixtures** (`"x".repeat(n)`) — overlap detection hits its candidate
bound and gives up, so the benchmark measures the fallback path, not the real one.
Use varied text.
- **Measuring a cache hit as if it were work** — `Markdown.render()` returns
`cachedLines` when text and width are unchanged, so a repeat-the-same-call loop
measured 0.065 ms for something that costs 0.86 ms.
- **Reasoning about complexity while the constant dominated** — twice.
- **Measuring heap instead of retained size** — a fixture generating 800 x 1 MB
strings reported 40 MB of "growth" that was its own garbage.

39
packages/agent/docs/mobile-handoff/01-harness/01-delta/append-decision.md Normal file
View File

@@ -0,0 +1,39 @@
# Decision: no explicit text append/truncate API
**Status: closed.** Do not add `appendText`, explicit truncate, a text-specific dirty-node kind, or proxy-to-tracker lookup machinery.
## Why it was considered
The tracker marks dirty paths and compares them with its accepted baseline at `flush()`. For a growing or rolling string, confirming the relationship can scan the retained string even though the producer already knows it appended or evicted text.
The original benchmark made this look expensive because its append fast path used `after.startsWith(before)`. V8 walked the producer's cons string character by character. Production Chord now uses:
```ts
if (after.length > before.length && after.slice(0, before.length) === before) {
// emit append
}
```
The slice flattens once and comparison uses the native string path. Flush-time dirty tracking also removed the prototype's retained-op amplification.
## Local confirmation
Measured 2026-09-01 against `origin/dev` at `1a7bc80e7`, using `packages/chord/src/delta/index.ts` directly under Node 26.0.0 on an Apple M5 Max. Each workload used 3,000 warmups followed by 11 samples of 10,000 mutation-plus-flush iterations; a second process reproduced the result.
| Workload | Median µs/flush, run 1 | Median µs/flush, run 2 |
| --- | ---: | ---: |
| 200 KB assistant string, append 8 characters | 18.68 | 17.81 |
| 50 KB rolling window, slide 32 varied characters | 2.46 | 2.43 |
| Transcript push, one small entry | 0.74 | 0.78 |
The assistant tracker started with 200,000 varied characters, appended `" abcdef"`, and flushed each append. The rolling tracker started with 50,000 varied characters, then assigned `text.slice(32) + chunk` using a distinct 32-character `chunk:<base36 index>:durable-stream` value on every flush; every flush was asserted to emit `t` + `a`. The transcript tracker pushed `{ id: "e<index>", text: "message <index>" }` and asserted one `p` per flush. Setup and garbage collection were outside each timed loop.
At 100 assistant updates per second, the growing-string case consumes about 1.8 ms/s, approximately 0.18% of one core. The measured rolling-window cost is itself too small to justify the abandoned explicit API.
## Why the API is rejected
The prototype required a text-specific dirty-node state, proxy-to-tracker lookup, repeated-append/drop folding, and interaction rules for later whole-value replacement. It produced subtle silent failures: one implementation fell through to the ordinary differ while tests still passed, and another drifted when drops reached into earlier appends.
That complexity does not justify saving microseconds below the surrounding replication, isolation, and rendering costs. Keep ordinary string mutation and the generic Chord op vocabulary.
Reopen only when a production profile shows delta flush time is a meaningful fraction of a real workload. Re-measure the generic fast path first; a regression there is cheaper and safer to fix than adding a producer-specific API.

826
packages/agent/docs/mobile-handoff/01-harness/01-delta/delta-impl.ts Normal file
View File

@@ -0,0 +1,826 @@
// ─────────────────────────────────────────────────────────────────────────────
// harness/delta — intent-recording change tracking over plain JSON.
//
// Depends on nothing else in the harness. Session storage, the runtime and the
// facet host consume it; keep the arrows pointing that way.
// ─────────────────────────────────────────────────────────────────────────────
export type JsonValue = null | boolean | number | string | JsonValue[] | { [k: string]: JsonValue };
export type Seg = string | number;
export type Path = readonly Seg[];
export type NonEmptyPath = readonly [Seg, ...Seg[]];
/** A path inline, or an id assigned by the encoder on second use. */
export type PathRef<P extends Path = Path> = P | number;
/**
* Tuples are the form — in memory, on the wire, on disk.
*
* `r` is the ONLY op that replaces a whole value. `s`/`d`/`a`/`t` cannot target
* the root: the type forbids it. `p` may, and only because a tracked value can
* itself be an array — but a `p` that replaces its entire target is normalised to
* `r`/`s` at record time, so a root `p` is always a partial modification.
*
* `Op` knows nothing about the path dictionary. Interning, id references and
* omitted paths live in `WireOp` and exist only between `encode` and `decode`.
*/
export type Op =
| readonly ["r", JsonValue]
| readonly ["s", NonEmptyPath, JsonValue]
| readonly ["d", NonEmptyPath]
| readonly ["a", NonEmptyPath, string]
| readonly ["t", NonEmptyPath, number]
| readonly ["p", Path, number, number, JsonValue[]];
/**
* What crosses a boundary. Adds two compressions and nothing else:
*
* ["#", id, path] defines an id, emitted on a path's SECOND use
* a numeric PathRef references a previously defined id
* a shortened tuple reuses the previous op's path; arity disambiguates
*
* ["r", value] carries no path, so it encodes to itself — which is why isBase
* works unchanged on either vocabulary.
*/
export type WireOp =
| readonly ["r", JsonValue]
| readonly ["s", PathRef<NonEmptyPath>, JsonValue] | readonly ["s", JsonValue]
| readonly ["d", PathRef<NonEmptyPath>] | readonly ["d"]
| readonly ["a", PathRef<NonEmptyPath>, string] | readonly ["a", string]
| readonly ["t", PathRef<NonEmptyPath>, number] | readonly ["t", number]
| readonly ["p", PathRef, number, number, JsonValue[]]
| readonly ["p", number, number, JsonValue[]]
| readonly ["#", number, Path];
// ─── Classification ──────────────────────────────────────────────────────────
export const isReplace = (op: Op | WireOp): boolean => op[0] === "r";
/**
* A batch begins with a replacement. Flush guarantees `r` is at index 0 or absent,
* so this is exact rather than a heuristic.
*/
export const isBase = (ops: readonly (Op | WireOp)[]): boolean =>
ops.length > 0 && ops[0]![0] === "r";
// ─── Overlap ─────────────────────────────────────────────────────────────────
/**
* Longest suffix of `a` that is a prefix of `b`. Probes with indexOf and verifies
* with endsWith, so the hot loops are native. A hand-written KMP is
* asymptotically equivalent and ~47x slower in practice.
*
* Always correct: the returned n satisfies a.endsWith(b.slice(0, n)).
*/
export function overlap(a: string, b: string, scan: number, probe = 64, maxCandidates = 8): number {
if (a.length === 0 || b.length === 0 || scan === 0) return 0;
const tail = a.length > scan ? a.slice(a.length - scan) : a;
// A probe of length h can only find overlaps of at least h — the head must
// actually occur in `a`. So try a long head first (few candidates, and it
// catches the large overlaps a rolling window produces), then fall back to one
// character, which finds any overlap at the cost of more candidates.
//
// Candidates are bounded because repetitive output — a build log, or any run of
// one character — makes a long head match at thousands of positions, each
// costing a full endsWith. Giving up returns 0, which emits a set: larger, never
// wrong. Unbounded, a 50 KB window of one repeated character took 4.7 s per
// 2000 slides against 90 ms bounded.
for (const h of [Math.min(probe, b.length), 1]) {
const head = b.slice(0, h);
let tried = 0;
for (let k = tail.indexOf(head); k !== -1; k = tail.indexOf(head, k + 1)) {
if (++tried > maxCandidates) break;
const n = tail.length - k;
if (n <= b.length && tail.endsWith(b.slice(0, n))) return n;
}
if (h === 1) break;
}
return 0;
}
// ─── Tracker ─────────────────────────────────────────────────────────────────
export interface TrackerOptions {
maxOverlapScan?: number;
coalesce?: boolean;
}
export interface Tracker<T extends object> {
/**
* Mutate this. Plain TS: assignment, push, splice, delete, all of it.
*
* Assigning to it replaces the whole value: emits `["r", next]` and discards
* ops recorded before it, which describe a value that no longer exists.
*
* `tracker.state = tracker.target` is therefore a checkpoint — it forces a base
* batch without changing anything. Discarding the pending ops is correct
* because the proxy mutates the target directly, so the value already carries
* them.
*/
state: T;
/** The untracked backing object. Reads only — writing here emits nothing. */
readonly target: T;
flush(): Op[];
/**
* Clear pending ops and make the next flush a base batch, without changing
* the value.
*
* Recovery replays from the last base batch, so a producer that wants "at most
* N batches to replay" calls this every N. The pending ops are already
* reflected in the value — the proxy mutates the target directly — so
* discarding them loses nothing.
*/
rebase(): void;
/** Discard without emitting. */
discard(): void;
readonly dirty: boolean;
}
const isObj = (v: unknown): v is object => v !== null && typeof v === "object";
const clone = <T>(v: T): T => (isObj(v) ? (structuredClone(v) as T) : v);
const INDEX = /^(?:0|[1-9]\d*)$/;
const norm = (t: object, k: string | symbol): Seg | symbol =>
typeof k === "symbol" ? k : Array.isArray(t) && INDEX.test(k) ? Number(k) : k;
const MUTATORS = new Set(["push", "pop", "shift", "unshift", "splice", "sort", "reverse", "fill", "copyWithin"]);
export function track<T extends object>(root: T, options: TrackerOptions = {}): Tracker<T> {
const scan = options.maxOverlapScan ?? 65_536;
const doCoalesce = options.coalesce ?? true;
let ops: Op[] = [];
const emit = (op: Op) => {
ops.push(op);
};
/**
* Reject reserved keys at record time, not just on apply.
*
* `JSON.parse('{"__proto__":{}}')` yields an OWN property, so a producer can
* hold state containing one without doing anything unusual. The tracker would
* then emit a path the applier refuses — state that cannot be replicated. Fail
* where the mistake is, at the write.
*/
const guard = (seg: Seg | symbol): Seg => {
if (typeof seg === "symbol") throw new UnsafePathError(String(seg));
if (typeof seg === "string" && FORBIDDEN.has(seg)) throw new UnsafePathError(seg);
return seg;
};
/**
* A splice covering the whole target is a replacement written the long way.
* Only the interception knows the pre-splice length, so this is a record-time
* rewrite, not something flush can recover.
*/
const spliceOrReplace = (path: Path, before: number, index: number, remove: number, items: JsonValue[]) => {
if (index === 0 && remove === before) {
if (path.length === 0) emit(["r", items]);
else emit(["s", path as NonEmptyPath, items]);
return;
}
emit(["p", [...path], index, remove, items]);
};
const cache = new WeakMap<object, Map<string, object>>();
const wrap = <V extends object>(obj: V, path: Path): V => {
let byPath = cache.get(obj);
if (byPath === undefined) cache.set(obj, (byPath = new Map()));
const cacheKey = path.join("\u0000");
const hit = byPath.get(cacheKey);
if (hit !== undefined) return hit as V;
const proxy = new Proxy(obj, {
get(t, k, r) {
if (Array.isArray(t) && typeof k === "string" && MUTATORS.has(k)) {
return (...args: unknown[]) => {
const before = t.length;
const result = (Array.prototype[k as "push"] as Function).apply(t, args);
const items = (xs: unknown[]) => xs.map(clone) as JsonValue[];
switch (k) {
case "push": spliceOrReplace(path, before, before, 0, items(args)); break;
case "unshift": spliceOrReplace(path, before, 0, 0, items(args)); break;
case "pop": if (before > 0) spliceOrReplace(path, before, before - 1, 1, []); break;
case "shift": if (before > 0) spliceOrReplace(path, before, 0, 1, []); break;
case "splice": {
const raw = Number(args[0] ?? 0);
const index = raw < 0 ? Math.max(0, before + raw) : Math.min(raw, before);
const remove = args.length < 2 ? before - index : Math.max(0, Math.min(Number(args[1]), before - index));
spliceOrReplace(path, before, index, remove, items(args.slice(2)));
break;
}
// sort/reverse/fill/copyWithin permute wholesale; intent is not a splice.
default:
if (path.length === 0) emit(["r", clone(t) as JsonValue]);
else emit(["s", path as NonEmptyPath, clone(t) as JsonValue]);
}
return result;
};
}
const v = Reflect.get(t, k, r);
return isObj(v) ? wrap(v, [...path, guard(norm(t, k))]) : v;
},
set(t, k, v, r) {
// `length` is a real mutation but not a document location.
if (Array.isArray(t) && k === "length") {
const before = t.length;
const next = Number(v);
if (next < before) spliceOrReplace(path, before, next, before - next, []);
else if (next > before) emit(["p", [...path], before, 0, Array(next - before).fill(null)]);
return Reflect.set(t, k, v, r);
}
const at = [...path, guard(norm(t, k))] as unknown as NonEmptyPath;
if (v === undefined) { // JSON has no undefined
emit(["d", at]);
return Reflect.deleteProperty(t, k);
}
const prev = (t as Record<string | symbol, unknown>)[k];
if (typeof prev === "string" && typeof v === "string") {
if (prev === v) return true;
if (v.length > prev.length && v.startsWith(prev)) {
emit(["a", at, v.slice(prev.length)]); // fast path, no probe
} else {
const ov = overlap(prev, v, scan);
if (ov === 0) emit(["s", at, v]);
else {
emit(["t", at, prev.length - ov]);
if (v.length > ov) emit(["a", at, v.slice(ov)]);
}
}
} else {
assertJsonValue(v);
emit(["s", at, clone(v) as JsonValue]);
}
return Reflect.set(t, k, v, r);
},
deleteProperty(t, k) {
emit(["d", [...path, guard(norm(t, k))] as unknown as NonEmptyPath]);
return Reflect.deleteProperty(t, k);
},
});
byPath.set(cacheKey, proxy);
return proxy as V;
};
let state = wrap(root, []);
// The first flush is always a base batch. A consumer starts with nothing, so
// a stream that opens with deltas has nothing to apply them to — and that
// fails at runtime, in the consumer, rather than where the mistake is.
let forceBase = true;
return {
get state() { return state; },
// A setter, so the obvious thing works. Without it `tracker.state = next`
// silently swaps the proxy for a plain object and stops tracking.
set state(next: T) {
ops = [];
root = next;
state = wrap(root, []);
forceBase = true;
},
get target() { return root; },
// Same effect as assigning the current value back, minus the re-wrap: the
// value has not changed, so the proxy cache is still valid.
rebase() { ops = []; forceBase = true; },
get dirty() { return ops.length > 0; },
discard() { ops = []; },
flush() {
if (forceBase) {
forceBase = false;
ops = [];
return [["r", clone(root) as JsonValue]];
}
if (ops.length === 0) return [];
let out = ops;
ops = [];
// Everything before a replacement is dead.
const lastReplace = out.findLastIndex(isReplace);
if (lastReplace > 0) out = out.slice(lastReplace);
if (doCoalesce) out = coalesce(dropDead(out));
return out;
},
};
}
const samePath = (a: Op, b: Op): boolean => {
const pa = pathOf(a);
const pb = pathOf(b);
if (pa === undefined || pb === undefined) return false;
return JSON.stringify(pa) === JSON.stringify(pb);
};
function pathOf(op: Op): Path | undefined {
return op[0] === "r" ? undefined : op[1];
}
/**
* Drop ops that a later op makes unreachable.
*
* Ops apply in order, so an op is dead if a later one overwrites the whole
* subtree it lives in. Only `s`, `d` and `r` do that — `a`, `t` and `p` modify
* what is already there, so they never dominate.
*
* O(n * d): walk backwards keeping a map of overwritten paths, and for each op
* test its own prefixes against that map. `d` is the path depth, which is small
* and bounded. The obvious formulation — compare each op against every
* dominator — is O(n^2 * d) and degrades on exactly the wide-flush input this
* pass exists to clean up. Measured: 12-14 microseconds per op, flat from 100 to
* 5000 ops, and 10001 redundant writes collapse to 2 emitted ops.
*
* OBJECT KEY ORDER IS NOT PRESERVED. A producer that does
* `set c; set a; delete c; set c` leaves `c` last, because deleting and
* reinserting moves it. The replica receives only the surviving `set c`, applies
* it to a base where `c` never existed, and `c` lands first. Values are
* identical — verified over 1823 randomised sequences with zero value
* mismatches — but the serialised forms can differ.
*
* This is fine because nothing compares a replica against its producer: the
* consumer holds the state and nobody diffs it back. The one visible case is a
* consumer that iterates keys for display, where two views could order rows
* differently. It requires a delete-and-recreate of the same key within one
* flush. Do not hash or content-address a replicated value.
*/
function dropDead(ops: Op[]): Op[] {
const overwritten = new Map<string, "s" | "d">();
const kept: Op[] = [];
for (let i = ops.length - 1; i >= 0; i--) {
const op = ops[i]!;
if (op[0] === "r") {
kept.push(op);
overwritten.clear();
overwritten.set("", "s"); // the root is now overwritten
continue;
}
const path = opPath(op);
if (path === undefined) { kept.push(op); continue; } // arity form, or an interned id
let dead = overwritten.has("");
let key = "";
for (let k = 0; k < path.length && !dead; k++) {
key += `\u0000${String(path[k])}`;
const kind = overwritten.get(key);
if (kind === undefined) continue;
// A `d` at exactly a later `s`'s path is NOT dead. Dropping it leaves the
// key in its original slot, while the producer's delete-then-set moved it
// to the end: same value, different serialisation.
if (k === path.length - 1 && op[0] === "d" && kind === "s") break;
dead = true;
}
if (dead) continue;
kept.push(op);
if (op[0] === "s" || op[0] === "d") overwritten.set(key, op[0]);
}
return kept.reverse();
}
/** Total, because `Op` paths are always inline. */
function opPath(op: Op): Path | undefined {
return op[0] === "r" ? undefined : op[1];
}
/** Merge adjacent ops describing the same path. Lossless. */
function coalesce(ops: Op[]): Op[] {
const out: Op[] = [];
for (const op of ops) {
const last = out[out.length - 1];
if (last !== undefined && samePath(last, op)) {
if (last[0] === "a" && op[0] === "a") {
out[out.length - 1] = ["a", last[1], `${last[2]}${op[2]}`];
continue;
}
if (last[0] === "s" && op[0] === "s") { // later write wins
out[out.length - 1] = op;
continue;
}
if (last[0] === "s" && op[0] === "a" && typeof last[2] === "string") {
out[out.length - 1] = ["s", last[1], `${last[2]}${op[2]}`];
continue;
}
}
out.push(op);
}
return out;
}
// ─── Path safety ─────────────────────────────────────────────────────────────
/**
* Segments that reach the prototype chain.
*
* `JSON.parse` is safe on its own — it makes `__proto__` an own property. What is
* not safe is `parent[key] = value`, which is exactly what an applier does, and
* paths are data: `["s", ["__proto__", "isAdmin"], true]` pollutes
* `Object.prototype` for the whole process.
*
* Ops arrive from a facet, a plugin compartment, or a tool whose details may echo
* model output, so none of it is trusted input.
*/
const FORBIDDEN = new Set(["__proto__", "constructor", "prototype"]);
export class UnsafePathError extends Error {
// Not a parameter property: Node's --experimental-strip-types rejects those,
// and these files are meant to run under it directly.
readonly segment: Seg;
constructor(segment: Seg) {
super(`unsafe path segment: ${String(segment)}`);
this.segment = segment;
this.name = "UnsafePathError";
}
}
/**
* Rejects anything JSON.stringify would not round-trip.
*
* structuredClone is NOT a JSON check: it clones a Map happily, and the op then
* carries `{}` while the producer keeps a real Map — producer and replica diverge
* silently, forever.
*/
export function assertJsonValue(value: unknown, seen = new Set<unknown>()): void {
if (value === null) return;
const t = typeof value;
if (t === "string" || t === "boolean") return;
if (t === "number") {
if (!Number.isFinite(value as number)) throw new TypeError("non-finite number is not JSON");
return;
}
if (t !== "object") throw new TypeError(`${t} is not a JsonValue`);
if (seen.has(value)) throw new TypeError("cyclic value is not JSON");
seen.add(value);
if (Array.isArray(value)) {
for (const v of value) assertJsonValue(v, seen);
} else {
const proto = Object.getPrototypeOf(value);
if (proto !== Object.prototype && proto !== null) {
throw new TypeError(`${(value as object).constructor?.name ?? "object"} is not a JsonValue`);
}
if (Object.getOwnPropertySymbols(value).length > 0) {
throw new TypeError("symbol keys are not JSON");
}
for (const v of Object.values(value as object)) assertJsonValue(v, seen);
}
seen.delete(value);
}
/**
* Verb, arity and payload shape for a **decoded** op: paths inline, no `#`, no
* short forms. `apply` uses this.
*
* Validating `Op` against the wire grammar would be laxer than the type: a
* two-element `["s", value]` would pass, and `apply` would then read the value as
* a path. Each vocabulary gets the validator that matches it.
*/
export function assertValidOp(op: unknown): void {
if (!Array.isArray(op) || op.length === 0) throw new TypeError("op is not a tuple");
switch (op[0]) {
case "r": if (op.length !== 2) throw new TypeError("r arity"); assertJsonValue(op[1]); return;
case "s": if (op.length !== 3) throw new TypeError("s arity"); assertPathArg(op[1]); assertJsonValue(op[2]); return;
case "d": if (op.length !== 2) throw new TypeError("d arity"); assertPathArg(op[1]); return;
case "a": if (op.length !== 3 || typeof op[2] !== "string") throw new TypeError("a shape"); assertPathArg(op[1]); return;
case "t": if (op.length !== 3 || !Number.isInteger(op[2])) throw new TypeError("t shape"); assertPathArg(op[1]); return;
case "p": {
if (op.length !== 5) throw new TypeError("p arity");
assertPathArg(op[1]);
if (!Number.isInteger(op[2]) || op[2] < 0) throw new TypeError("p index");
if (!Number.isInteger(op[3]) || op[3] < 0) throw new TypeError("p remove");
if (!Array.isArray(op[4])) throw new TypeError("p items");
for (const v of op[4]) assertJsonValue(v);
return;
}
// Silently skipping an unknown verb is how a newer producer's op vanishes.
default: throw new TypeError(`unknown op verb: ${String(op[0])}`);
}
}
function assertPathArg(p: unknown): void {
if (!Array.isArray(p)) throw new TypeError("path is not an array");
assertSafePath(p as Path);
}
/** The same, for the wire grammar: ids and short forms are legal here. */
export function assertValidWireOp(op: unknown): void {
if (!Array.isArray(op) || op.length === 0) throw new TypeError("op is not a tuple");
const [verb] = op as unknown[];
const okRef = (r: unknown): void => {
if (typeof r === "number") {
if (!Number.isInteger(r) || r < 0) throw new TypeError("bad path id");
return;
}
// A string is not a path. Unchecked, `"a".slice(0, -1)` is `""`, so it
// resolves to the ROOT and writes there — a path that is not a path, accepted.
if (!Array.isArray(r)) throw new TypeError("path is not an array");
assertSafePath(r as Path);
};
switch (verb) {
case "r": if (op.length !== 2) throw new TypeError("r arity"); assertJsonValue(op[1]); return;
case "s": if (op.length === 3) { okRef(op[1]); assertJsonValue(op[2]); }
else if (op.length === 2) assertJsonValue(op[1]);
else throw new TypeError("s arity"); return;
case "d": if (op.length === 2) okRef(op[1]); else if (op.length !== 1) throw new TypeError("d arity"); return;
case "a": if (op.length === 3) { okRef(op[1]); if (typeof op[2] !== "string") throw new TypeError("a value"); }
else if (op.length === 2) { if (typeof op[1] !== "string") throw new TypeError("a value"); }
else throw new TypeError("a arity"); return;
case "t": if (op.length === 3) { okRef(op[1]); if (!Number.isInteger(op[2])) throw new TypeError("t count"); }
else if (op.length === 2) { if (!Number.isInteger(op[1])) throw new TypeError("t count"); }
else throw new TypeError("t arity"); return;
case "p": {
const [i, r, items] = op.length === 5 ? [op[2], op[3], op[4]] : op.length === 4 ? [op[1], op[2], op[3]] : [];
if (items === undefined) throw new TypeError("p arity");
if (op.length === 5) okRef(op[1]);
if (!Number.isInteger(i) || (i as number) < 0) throw new TypeError("p index");
if (!Number.isInteger(r) || (r as number) < 0) throw new TypeError("p remove");
if (!Array.isArray(items)) throw new TypeError("p items");
for (const v of items) assertJsonValue(v);
return;
}
case "#": {
if (op.length !== 3 || !Number.isInteger(op[1]) || !Array.isArray(op[2])) throw new TypeError("# shape");
assertSafePath(op[2] as Path);
return;
}
// Silently skipping an unknown verb is how a newer producer's op vanishes.
default: throw new TypeError(`unknown op verb: ${String(verb)}`);
}
}
export function assertSafePath(path: Path): void {
for (const seg of path) {
if (typeof seg === "string") {
if (FORBIDDEN.has(seg)) throw new UnsafePathError(seg);
} else if (!Number.isInteger(seg) || seg < 0) {
throw new UnsafePathError(seg);
}
}
}
/**
* An index may address an existing element or append exactly one past the end.
*
* This is not an arbitrary cap — it is what keeps the value a `JsonValue`. A
* sparse array does not survive a JSON round trip: holes serialise to `null` and
* return as real properties, so `arr[7] = x` on a length-3 array already produces
* state a replica cannot match. Rejecting the write is more honest than silently
* diverging.
*
* It also removes the denial of service it would otherwise permit:
* `["s", ["xs", 4294967290], 1]` allocates a 4.29-billion-entry array from one op.
* Growth stays possible and stays proportional — the tracker already emits
* `arr.length = n` as a splice of explicit nulls, whose op size grows with the
* gap, so a large growth costs a large op rather than a small one.
*/
function assertIndexInRange(parent: readonly unknown[], index: number): void {
if (index > parent.length) throw new UnsafePathError(index);
}
// ─── Applier ─────────────────────────────────────────────────────────────────
export class PathError extends Error {
readonly path: Path | number;
constructor(path: Path | number) {
super(`unresolvable path: ${JSON.stringify(path)}`);
this.path = path;
this.name = "PathError";
}
}
/**
* Apply ops to a plain mutable value. Returns the value, because `r` replaces it
* outright and cannot be done in place.
*
* Takes decoded ops. Path ids and omitted paths are a wire concern — run
* `decode` first if the ops came from a boundary.
*/
export function apply<T>(target: T, ops: readonly Op[]): T {
let root = target as unknown as JsonValue;
for (const op of ops) {
assertValidOp(op);
if (op[0] === "r") {
// Adopted, not copied. The consumer owns the batch it was handed.
//
// Fanning one batch out to several consumers in-process therefore makes
// their replicas alias each other. That is an ownership rule, not a
// defect: copy the batch at the fan-out point, or let each consumer
// decode its own. A batch that crosses a real boundary is already
// distinct, because serialisation produces fresh objects.
root = op[1];
continue;
}
const path = op[1];
assertSafePath(path);
if (op[0] === "p") {
const target_ = path.length === 0 ? root : resolve(root, path);
if (!Array.isArray(target_)) throw new PathError(path);
target_.splice(op[2], op[3], ...op[4]);
continue;
}
// s/d/a/t can never target the root — the type forbids it.
const parent = resolve(root, path.slice(0, -1)) as Record<Seg, JsonValue>;
const key = path[path.length - 1]!;
if (Array.isArray(parent) && typeof key === "number") assertIndexInRange(parent, key);
// defineProperty rather than assignment: a setter inherited from the prototype
// chain would otherwise run on write.
const write = (value: JsonValue) => {
Object.defineProperty(parent, key, { value, writable: true, enumerable: true, configurable: true });
};
const read = (): unknown => (Object.hasOwn(parent, key) ? parent[key] : undefined);
switch (op[0]) {
case "s": write(op[2]); break;
case "d": Array.isArray(parent) ? (parent as unknown as JsonValue[]).splice(Number(key), 1) : delete parent[key]; break;
case "a": write(`${String(read() ?? "")}${op[2]}`); break;
case "t": write(String(read() ?? "").slice(op[2])); break;
}
}
return root as unknown as T;
}
function resolve(root: JsonValue, path: Path): JsonValue {
let node: JsonValue = root;
for (const seg of path) {
if (!isObj(node)) throw new PathError(path);
// Own properties only: an inherited getter must not run, and a walk must not
// escape the value into the prototype chain.
if (!Object.hasOwn(node, seg as PropertyKey)) throw new PathError(path);
node = (node as Record<Seg, JsonValue>)[seg]!;
}
if (!isObj(node)) throw new PathError(path);
return node;
}
// ─── Codec ───────────────────────────────────────────────────────────────────
//
// Path interning and arity omission, per `delta.md` §4. These live between the
// tracker and a boundary; `Op` and `apply` know nothing about them.
//
// ONE PAIR PER STREAM. The table spans an entire subscription or file: a path
// interned in batch 3 is referenced in batch 40. A second consumer that
// subscribes at batch 40 has never seen the definition, so it needs its own
// encoder — sharing one across consumers hands the late subscriber ids it cannot
// resolve. A base batch does not rescue it, because `["r", value]` carries no
// path refs and leaves the table empty.
const pathKey = (path: Path): string => path.join("\u0000");
export interface Encoder {
encode(ops: readonly Op[]): WireOp[];
}
/**
* Intern on SECOND use. A definition costs more than the path it replaces, so
* interning on first use loses on the many paths written exactly once —
* measured worse than no interning at all (`delta.md` §4.1).
*/
export function encoder(): Encoder {
const seen = new Set<string>();
const ids = new Map<string, number>();
let nextId = 0;
let previous: string | undefined; // last path in THIS batch
return {
encode(ops) {
// Arity omission is scoped to a batch. Letting it span batches would make
// a batch's first op depend on the previous batch's last one, so a reader
// that skips or reorders a batch decodes into the wrong path. Ids are the
// only cross-batch state, and the dictionary makes those explicit.
previous = undefined;
const out: WireOp[] = [];
for (const op of ops) {
if (op[0] === "r") {
out.push(op);
// A base batch is a RECOVERY POINT: a reader replays from the last one
// with a fresh decoder. So everything after it must be self-contained.
// Keeping ids across a replacement emits references to definitions the
// reader never saw — recovery fails with an unresolvable path id.
seen.clear();
ids.clear();
nextId = 0;
previous = undefined;
continue;
}
const path = op[1];
const key = pathKey(path);
// Same path as the previous op: drop the ref entirely.
if (key === previous) {
switch (op[0]) {
case "s": out.push(["s", op[2]]); break;
case "d": out.push(["d"]); break;
case "a": out.push(["a", op[2]]); break;
case "t": out.push(["t", op[2]]); break;
case "p": out.push(["p", op[2], op[3], op[4]]); break;
}
continue;
}
let ref: PathRef = path;
const existing = ids.get(key);
if (existing !== undefined) {
ref = existing;
} else if (seen.has(key)) {
const id = nextId++;
ids.set(key, id);
out.push(["#", id, path]); // second use: define, then reference
ref = id;
} else {
seen.add(key); // first use: inline
}
switch (op[0]) {
case "s": out.push(["s", ref as PathRef<NonEmptyPath>, op[2]]); break;
case "d": out.push(["d", ref as PathRef<NonEmptyPath>]); break;
case "a": out.push(["a", ref as PathRef<NonEmptyPath>, op[2]]); break;
case "t": out.push(["t", ref as PathRef<NonEmptyPath>, op[2]]); break;
case "p": out.push(["p", ref, op[2], op[3], op[4]]); break;
}
previous = key;
}
return out;
},
};
}
export interface Decoder {
decode(wire: readonly WireOp[]): Op[];
}
export function decoder(): Decoder {
const paths = new Map<number, Path>();
return {
decode(wire) {
let previous: Path | undefined; // scoped to the batch, as in encode
const out: Op[] = [];
for (const op of wire) {
assertValidWireOp(op);
if (op[0] === "#") {
assertSafePath(op[2]);
paths.set(op[1], op[2]);
continue;
}
if (op[0] === "r") {
out.push(op);
previous = undefined;
continue;
}
// Arity tells us whether a ref is present: the short forms omit it.
const short =
(op[0] === "d" && op.length === 1) ||
(op[0] !== "d" && op[0] !== "p" && op.length === 2) ||
(op[0] === "p" && op.length === 4);
let path: Path;
if (short) {
if (previous === undefined) throw new PathError([]);
path = previous;
} else {
const ref = op[1] as PathRef;
if (typeof ref === "number") {
const resolved = paths.get(ref);
if (resolved === undefined) throw new PathError(ref);
path = resolved;
} else {
path = ref;
}
previous = path;
}
switch (op[0]) {
case "s": out.push(["s", path as NonEmptyPath, (short ? op[1] : op[2]) as JsonValue]); break;
case "d": out.push(["d", path as NonEmptyPath]); break;
case "a": out.push(["a", path as NonEmptyPath, (short ? op[1] : op[2]) as string]); break;
case "t": out.push(["t", path as NonEmptyPath, (short ? op[1] : op[2]) as number]); break;
case "p": {
const [i, r, items] = short
? [op[1] as number, op[2] as number, op[3] as JsonValue[]]
: [op[2] as number, op[3] as number, op[4] as JsonValue[]];
out.push(["p", path, i, r, items]);
break;
}
}
}
return out;
},
};
}

113
packages/agent/docs/mobile-handoff/01-harness/01-delta/delta.bench.ts Normal file
View File

@@ -0,0 +1,113 @@
/**
* Not a test — a guard against the cliffs found during implementation. Reports
* units you can reason about: cost per WRITE, and bytes on the wire relative to
* bytes consumed.
*
* npx vitest bench delta.bench.ts
*
* The headline finding: cost is per write, not per byte. A 100-byte chunk costs
* about the same as a 1 KB chunk, because the fixed cost is the window slice and
* the overlap probe. So chunk size — set by the exec env's coalescing interval,
* not by this package — decides both throughput and wire size.
*/
import { bench, describe } from "vitest";
import { decoder, encoder, track } from "./delta-impl.ts";
const WORDS = "the quick brown fox jumps over lazy dog while considering strategies".split(" ");
const text = (n: number): string => {
let s = "";
while (s.length < n) s += `${WORDS[Math.floor(Math.random() * WORDS.length)]} `;
return s.slice(0, n);
};
const CAP = 50_000;
const WRITES = 500;
const FLUSH_EVERY = 20;
/** One write = one chunk handed to the sink. Returns nothing; bench times it. */
function drive(initial: string, chunkSize: number, cap = CAP): void {
const t = track({ out: initial });
for (let i = 0; i < WRITES; i++) {
const next = t.state.out + text(chunkSize);
t.state.out = next.length > cap ? next.slice(next.length - cap) : next;
if (i % FLUSH_EVERY === FLUSH_EVERY - 1) t.flush();
}
t.flush();
}
/**
* Measured on the reference machine, 500 writes each. Reproduced here so a
* regression is legible rather than just "slower".
*
* append only, 100 B 127 µs/write 8k writes/s 101% of input on wire
* rolling, 100 B 176 µs/write 6k writes/s 135% of input on wire
* rolling, 1 KB 68 µs/write 15k writes/s 104% of input on wire
* rolling, 8 KB 366 µs/write 3k writes/s 31% of input on wire
* pathological, 100 B 307 µs/write 3k writes/s 727% of input on wire
*
* Two things to notice.
*
* 135% at 100-byte chunks: past the cap every write emits `truncate` + `append`,
* and two op envelopes exceed a 100-byte payload — we put MORE on the wire than we
* consumed. At 8 KB it is 31%. Flush granularity is therefore a size lever, not
* just an update-rate lever, and the exec env's coalescing is what keeps chunks
* off the bad end of that curve.
*
* The pathological row is bounded only because the overlap candidate scan is
* capped. Unbounded it was 4.7 s for 2000 writes against 93 ms; the cap makes it
* give up and emit a set — larger, never wrong.
*/
describe("rolling window, 50 KB cap", () => {
bench("100 B chunks", () => { drive(text(CAP), 100); });
bench("1 KB chunks", () => { drive(text(CAP), 1024); });
bench("8 KB chunks", () => { drive(text(CAP), 8192); });
bench("pathological: one repeated character", () => { drive("x".repeat(CAP), 100); });
});
describe("below the cap", () => {
// The dominant case for most tools: startsWith fast path, no probe at all,
// and one append per flush rather than a truncate/append pair.
bench("append only, never fills", () => { drive("", 100, Number.MAX_SAFE_INTEGER); });
});
// ── Codec: what path interning and arity omission actually save ────────────
//
// Reported as wire bytes relative to inline paths, on three shapes:
// - one hot path (a rolling tool-output window): omission does the work
// - a few alternating paths (lane state): interning does the work
// - many distinct paths: neither helps, and definitions cost a little
describe("codec", () => {
const streams = {
"one hot path": () => {
const t = track({ content: [{ type: "text", text: "" }] });
return Array.from({ length: 200 }, (_, i) => {
(t.state.content[0] as { text: string }).text += `line ${i} of build output\n`;
return t.flush();
});
},
"four alternating paths": () => {
const t = track({ a: "", b: "", c: "", d: "" });
return Array.from({ length: 200 }, (_, i) => {
for (const k of ["a", "b", "c", "d"] as const) t.state[k] += `${i}`;
return t.flush();
});
},
"200 distinct paths": () => {
const root: Record<string, number> = {};
for (let i = 0; i < 200; i++) root[`f${i}`] = 0;
const t = track(root);
return Array.from({ length: 5 }, (_, r) => {
for (let i = 0; i < 200; i++) t.state[`f${i}`] = r + 1;
return t.flush();
});
},
};
for (const [label, build] of Object.entries(streams)) {
bench(label, () => {
const enc = encoder();
const dec = decoder();
for (const ops of build()) dec.decode(enc.encode(ops));
});
}
});

133
packages/agent/docs/mobile-handoff/01-harness/01-delta/delta.examples.ts Normal file
View File

@@ -0,0 +1,133 @@
/**
* Runnable illustrations of the two things that are easy to get wrong.
*
* node --experimental-strip-types delta.examples.ts
*
* Nothing here is a test — `delta.test.ts` covers the same ground with
* assertions. This exists to be read.
*/
import { apply, decoder, encoder, isBase, track, type Op } from "./delta-impl.ts";
const show = (label: string, ops: Op[]) => {
console.log(` ${label.padEnd(22)} ${ops.length} op(s), base=${isBase(ops)}`);
for (const op of ops) console.log(` ${trunc(JSON.stringify(op), 66)}`);
};
const trunc = (s: string, n: number) => (s.length <= n ? s : `${s.slice(0, n)}…`);
// ─────────────────────────────────────────────────────────────────────────
// 1. Replacing the whole value
// ─────────────────────────────────────────────────────────────────────────
console.log("\n1. REPLACING THE WHOLE VALUE\n");
const before = { user: { id: "u1", name: "ada" }, items: ["a", "b"], note: "n".repeat(120) };
const after = { user: { id: "u2", name: "bob" }, items: ["x"], note: "m".repeat(120) };
{
// The wrong way. `state` is a property of the TRACKER, not of the tracked
// object. Without a setter this swaps the proxy for a plain object and every
// later mutation is silently untracked — no error, no ops.
//
// The implementation defines `state` as a setter precisely so this works.
const t = track<Record<string, unknown>>(structuredClone(before));
t.flush(); // drain the opening base batch
t.state.items = ["changed first"]; // recorded…
t.state = structuredClone(after); // …then discarded: it described a dead value
show("assign to .state", t.flush());
t.state.user = { id: "u3" }; // still tracked afterwards
show(" and still tracked", t.flush());
}
{
// Dead-op elimination: a field written three times emits one op, and a parent
// replaced after its child emits one. Object key order is not preserved by
// this pass — see delta.md §5.1.
const t = track<Record<string, any>>(structuredClone(before));
t.flush(); // drain the opening base batch
t.state.items = ["first"];
t.state.items = ["second"];
t.state.user.name = "dropped";
t.state.user = { id: "u9" };
show("4 writes, 2 survive", t.flush());
}
// ─────────────────────────────────────────────────────────────────────────
// 2. Why checkpointing exists
// ─────────────────────────────────────────────────────────────────────────
console.log("\n2. RECOVERY LENGTH\n");
/**
* A bash-shaped workload: output arrives continuously and the sink writes
* durably on an interval. Each flush is one batch appended to a list.
* On resume the reader folds back to the last BASE batch, so that distance is
* the cost of recovery.
*/
function simulate(writes: number, checkpointEvery: number | null) {
const CAP = 50_000;
const t = track({ out: "" });
const isBaseBatch: boolean[] = [];
for (let i = 0; i < writes; i++) {
const next = `${t.state.out}[${i}] cc -c src/file${i}.c -o build/file${i}.o\n`;
t.state.out = next.length > CAP ? next.slice(next.length - CAP) : next;
if (checkpointEvery !== null && i % checkpointEvery === checkpointEvery - 1) t.rebase();
const ops = t.flush();
if (ops.length > 0) isBaseBatch.push(isBase(ops));
}
const lastBase = isBaseBatch.lastIndexOf(true);
return { written: isBaseBatch.length, replay: isBaseBatch.length - lastBase - 1 };
}
for (const [label, every] of [["never", null], ["rebase() every 50", 50]] as const) {
const r = simulate(500, every);
console.log(` ${label.padEnd(18)} ${r.written} batches written → ${r.replay} to replay on recovery`);
}
console.log(`
Nothing produces a base batch on its own: flush() emits ops, and a replacement
happens only when the producer asks for one. So a stream of appends stays a
stream of deltas indefinitely, and recovery grows without bound.
Two callers need rebase():
- pendingToolOutput, during a long-running command. Without it a ten-minute
build leaves hundreds of batches to fold on resume.
- a facet host on resubscribe. facets.md §9.2 requires the first batch of a
subscription to be a base batch, and the host must be able to produce one
on demand.
`);
// ─────────────────────────────────────────────────────────────────────────
// 3. A replica always agrees
// ─────────────────────────────────────────────────────────────────────────
console.log("3. PRODUCER AND REPLICA AGREE\n");
{
// The whole loop: tracker -> encode -> [boundary] -> decode -> apply.
const t = track<Record<string, unknown>>(structuredClone(before));
const enc = encoder();
const dec = decoder();
let replica: unknown;
const step = (label: string, mutate: () => void) => {
mutate();
const ops = dec.decode(enc.encode(t.flush()));
if (ops.length === 0) return;
// No base-batch branch: apply handles `r` by replacing, and tolerates an
// undefined target because a root op never reads it.
replica = apply(replica, ops);
const same = JSON.stringify(replica) === JSON.stringify(t.target);
console.log(` ${label.padEnd(22)} ${ops.length} op(s) ${isBase(ops) ? "BASE " : "delta"} matches: ${same}`);
};
step("opening base batch", () => {});
step("append to a string", () => { (t.state.note as string); t.state.note = `${t.state.note}!`; });
step("push", () => { (t.state.items as string[]).push("c"); });
step("nested set", () => { (t.state.user as Record<string, unknown>).name = "cy"; });
step("delete", () => { delete t.state.note; });
step("whole replacement", () => { t.state = structuredClone(after); });
step("rebase", () => { t.rebase(); });
}
console.log();

617
packages/agent/docs/mobile-handoff/01-harness/01-delta/delta.md Normal file
View File

@@ -0,0 +1,617 @@
# Delta Tracking and the Op Vocabulary
> **Production status:** landed in `packages/chord/src/delta/index.ts`, with tests in `packages/chord/test/delta.test.ts`. Chord owns the dependency-free `Op`/`WireOp`, tracker, applier, codec, and validation boundary; Session storage, the Harness, and facets consume it. The implementation and tests beside this document are historical prototype and benchmark evidence, not production source.
>
> The landed tracker computes deltas at `flush()` from a dirty tree and baseline rather than retaining one op per mutation. That resolves FINDINGS D1. Production remeasurement also closes FINDINGS D2: the generic string path is below surrounding replication/rendering costs, so an explicit append/truncate API is rejected ([decision](append-decision.md)).
One mechanism covers assistant partials, tool output, tool details, lane state, and arbitrary facet state — on the wire and in durable storage.
```bash
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run test/delta.test.ts
# run from packages/chord
```
## 1. Why not an existing library
Immer, Valtio, Mutative and Colyseus all **record effects, not intent**. A string
is a leaf, so `text += chunk` is one write of the whole new string; there is no
representation in which the library could know you appended. `unshift` is the
same story — the engine shifts every index, so each is a write.
Measured, all four:
| operation | what they emit |
| --- | --- |
| `text += "x"` on 40 KB | `replace` carrying 40 KB |
| `arr.unshift(x)` on 100 items | ~100 index writes |
| `arr.pop()` | a write to a `length` path, which is not a document location |
Immer's own pitfalls page states the guarantee: patches are correct and
**explicitly not minimal**. Colyseus documents the same array weakness — removing
the first of 20 items costs 38 extra bytes.
Diffing base against result ourselves does not fix it either. A differ sees two
values, not intent. Prefix comparison catches pure growth, but a rolling window
that drops from the front *and* appends is neither a prefix nor a suffix of the
old value; measured on 5000 elements the fallback was
`{index: 0, remove: 5000, items: 5000}` — a full replace, in exactly the case the
optimisation existed for.
So the win is not a blind whole-value differ. The tracker records which paths and array operations became dirty, then compares only those subtrees against its accepted baseline at `flush()`.
## 2. Ops
**Tuples are the form. Everywhere. There is no keyed variant.**
```ts
export type Seg = string | number;
export type Path = readonly Seg[];
/** Non-empty: `s`/`d`/`a`/`t` cannot target the root. Enforced by the type. */
export type NonEmptyPath = readonly [Seg, ...Seg[]];
export type Op =
| readonly ["r", JsonValue] // replace the value
| readonly ["s", NonEmptyPath, JsonValue] // set
| readonly ["d", NonEmptyPath] // delete
| readonly ["a", NonEmptyPath, string] // append
| readonly ["t", NonEmptyPath, number] // truncate, in chars
| readonly ["p", Path, number, number, JsonValue[]] // splice: index, remove, items
```
Interning, id references and omitted paths are **not** here — they live in
`WireOp` and exist only between `encode` and `decode` (§4).
**`r` is the only op that replaces a whole value.** It is an op rather than a
frame kind for the same reason `a` and `t` exist: they are specialisations of `s`
that are smaller and state intent.
Do not encode a replacement as a set at the root path. Base-batch detection is a
**correctness boundary** — recovery stops there (§9) — so it must be a token
comparison, `ops[0]?.[0] === "r"`, and not a path inspection. A path-inspecting
predicate has to handle the two-element arity form, where `op[1]` is a value
rather than a path, and misclassifies `["s", []]`.
**Only `p` may target the root**, and only because a tracked value can itself be
an array — `entries.push(x)` on a root array is `["p", [], 3, 0, [x]]`. A `p`
covering its entire target is normalised at flush time to `r` (root) or `s`
(nested), so a root `p` is always a partial modification. The other four verbs
take a `NonEmptyPath`: `["s", [], v]` and `["d", []]` do not typecheck.
**Do not add a keyed in-memory shape with a codec at the serialization boundary.**
Two representations of one thing means two appliers, two size estimates, and a
conversion nobody needs. Ops are tuples in memory, on the wire, and on disk;
readability is what a debug formatter is for.
Six verbs. No `move`, `copy` or `test`: the first two are size optimisations for
a case we do not have, and `test` belongs to a conflict model we do not have,
since there is exactly one authoritative writer.
`chars` counts UTF-16 code units, matching `String.prototype.slice`. **Not bytes.**
Byte caps are a producer concern and never cross a boundary.
The format is JSON-Patch-*shaped*, not RFC 6902 conformant. That costs nothing:
RFC 6902 has had no successor since 2013, still has the same six ops, and nothing
anywhere adds a string splice. Immer was never conformant either.
## 3. The tracker
State is **plain TypeScript**. No containers, no handles, no schema, no
decorators. You mutate normally.
```ts
const t = track(laneView);
t.state.operation.streamingMessage.content[0].text += delta; // -> append
t.state.transcript.push(entry); // -> splice
t.state.tools[0].details.failures.push({ name, msg }); // -> splice
delete t.state.config.model; // -> delete
const ops = t.flush();
```
Three mechanisms, one per case:
**Objects** — ordinary `set` / `deleteProperty` traps.
**Arrays — mutator methods intercepted in the `get` trap.** `arr.push(x)` passes
through `get(arr, "push")` first, so the tracker returns its own function that
records `splice(len, 0, [x])` and then delegates. Intent is captured *before* the
engine performs its index writes, which is why `unshift` is one op rather than
O(n). This is the thing no other library does.
**Strings — dirty marking on the `set` trap, overlap detection at `flush()`.** Strings are primitives, so there is nothing to intercept. Given the accepted `prev` and final `next`, find the longest suffix of `prev` that is a prefix of `next`:
```
overlap === prev.length -> append(next.slice(overlap))
overlap > 0 -> truncate(prev.length - overlap) + append(rest)
overlap === 0 -> set(next)
```
Always correct, because the overlap is **verified**, not guessed. The rolling
window works:
```ts
s.out = s.out.slice(4) + "!!!";
// ["truncate", ["out"], 4]
// ["append", ["out"], "!!!"]
```
### 3.1 Overlap must not use a hand-written KMP
A KMP failure function is asymptotically right and **47x slower in practice**,
because it runs character by character in JS. Measured on 2000 window slides over
a 50 KB string: 2870 ms for KMP, 61 ms for a native probe.
The probe: `indexOf` a short head of `next` within `prev`, verify each candidate
with `endsWith`. Both are native. Verified identical to KMP across 200 window
slides.
A `startsWith` fast path runs first, so a pure append — the dominant case — skips
the probe entirely.
### 3.2 Rules the implementation must hold
**Inserted values are adopted; emitted payloads are cloned at flush.** Callers may retain read-only references but must not mutate adopted objects outside the tracker. `flush()` clones payloads and advances a cloned accepted baseline, so emitted ops never alias producer state.
**`x = undefined` normalises to `delete`.** JSON has no `undefined`, and a `set`
with a missing value is indistinguishable from a lost value after a
`JSON.stringify` round trip.
**`arr.length = n` is translated, never emitted as a path.** Shrink becomes a
splice, grow becomes a splice of nulls. `length` is a real mutation but not a
document location — this is the bug Immer ships (issue 208). Dropping it silently,
as an early prototype did, means `arr.length = 0` never reaches the replica.
**Numeric array keys are normalised to numbers.** Proxy traps deliver `"0"`;
paths carry `0`. Valtio ships the string form and it is worse for both size and
comparison.
**Proxies are cached per object and path**, so `s.a === s.a` and proxies are not
rebuilt on every access.
**Scope is `JsonValue` only, and it must be CHECKED, not assumed.**
`structuredClone` is not a JSON check — it clones `Map`, `Set`, `Date` and
`RegExp` happily, and the result then differs from what `JSON.stringify` produces,
so producer and replica diverge silently. Assert the value at record time and
reject symbol keys. See §7.4 for what leaks without it.
**Exactly two ops reach the root**, and the applier's root branch must handle
both: `r` always, and `p` when the tracked value is itself an array.
`s`/`d`/`a`/`t` cannot, by type, which makes the branch's fallthrough provably
unreachable. Handling only `r` there passes every hand-written test and fails on
the first randomised sequence that produces a root splice.
**Two writes to one address inside a transaction chain.** When several writes to
the same address are emitted in one commit, each must be recorded against the
previous one's result, not against pre-transaction state. The applier replays them
in order; anything else applies the second against a stale base. This did not fire
in the workload that surfaced it — it would have corrupted silently when it did.
### 3.2.1 Replacing the whole value
Assigning to `state` replaces it:
```ts
tracker.state = next; // emits ["r", next]; discards ops recorded before it
```
`state` **must be a setter on the tracker**, not a plain property. Without one,
`tracker.state = next` swaps the proxy for a plain object and every later
mutation is silently untracked — no error, no ops, `target` unchanged. It is also
the first thing anyone tries.
Prior ops are discarded because they describe a value that no longer exists.
Only a *full* replacement collapses to `r`. Rewrite part of the value and the ops
survive, because they are genuinely cheaper:
```
partial rewrite 2 op(s), base=false
["s",["user"],{"id":"u2","name":"bob"}]
["s",["items"],["x"]]
```
### 3.2.2 The first flush is a base batch
`track()` opens a stream, and a consumer starts with nothing, so the first flush
always emits `["r", value]` — carrying any mutations made before it:
```ts
const t = track({ x: 0, l: [] });
t.state.x = 100;
t.state.l.push("xyz");
t.flush(); // [["r", { x: 100, l: ["xyz"] }]]
t.state.x = 101;
t.flush(); // [["s", ["x"], 101]]
```
Requiring the producer to remember a `rebase()` first would fail at runtime, in
the consumer, far from the mistake.
### 3.2.3 Forcing a base batch later
```ts
tracker.rebase(); // next flush is ["r", value]; value unchanged
```
Discarding the pending ops is correct: the proxy mutates the target directly, so
the value already carries them. `tracker.state = tracker.state` has the same effect in the landed implementation, but `rebase()` states the intent directly.
**Nothing produces a base batch on its own.** `flush()` emits ops; a replacement
happens only when the producer asks for one. So a stream of appends stays a
stream of deltas indefinitely.
Recovery replays from the last base batch (§9). From `delta.examples.ts`, a
bash-shaped workload of 500 durable writes into a 50 KB rolling window:
| | batches written | to replay on recovery |
| --- | --- | --- |
| never | 500 | **499** |
| `rebase()` every 50 | 500 | 0 |
Two callers need this:
- **A durable sink.** `pendingToolOutput` accumulates one batch per checkpoint for
as long as a command runs. A `make -j8` running for ten minutes otherwise leaves
hundreds of batches to fold on resume. Checkpointing every N makes "at most N
batches to replay" a policy rather than an accident.
- **A facet host on resubscribe.** `facets.md` §9.2 requires the first batch of
every subscription to be a base batch, and a consumer that sees a gap
resubscribes to get one. The host has to be able to produce one on demand rather
than wait for one to happen by chance.
### 3.3 Known gaps
- `sort` / `reverse` / `fill` / `copyWithin` mark the array dirty and emit the resulting structural/index changes rather than preserving the producer's method intent. Add a dedicated op only if a measured workload needs one.
- A manual index-shift loop (`for (…) a[i] = a[i+1]`) can still cost O(n) sets. Correct, not minimal, and unavoidable — the producer genuinely wrote every element.
- Rolling-window string assignment still runs overlap discovery at flush, deliberately: production measurements do not justify text-specific tracker state or API surface ([decision](append-decision.md)).
### 3.4 Constraints the string algorithm must hold
Each of these is cheap to get wrong in a way that passes hand-written tests.
**A fixed-length probe cannot find an overlap shorter than the probe.** The head
must actually occur in `a`: `"abcdefgh"` -> `"defghxyz"` overlaps by 5, and a
64-character head cannot occur in an 8-character string. Try a long head first,
then fall back to a one-character head. A test using only 50 KB strings, where
overlaps are huge, will pass against a broken implementation.
**Bound the candidate scan.** Repetitive output — a build log, or a run of one
character — makes a long head match at thousands of positions, each costing a full
`endsWith`. Unbounded, 2000 slides over a 50 KB window take 4.7 s against 93 ms
bounded. Exceeding the bound returns 0, which emits a set: larger, never wrong.
**Property tests before anything depends on this.** 3000 random mutation
round-trips find the root-splice hole in §3.2 on the first run; the hand-written
cases do not. `delta.test.ts` is the suite to port.
## 4. The codec
Two vocabularies, not one.
`Op` is what the tracker produces and `apply` consumes. **Paths are always
inline.** It knows nothing about a dictionary.
`WireOp` is what crosses a boundary. It adds exactly two compressions:
```
["#", id, path] defines an id, emitted on a path's SECOND use
a numeric PathRef references a previously defined id
a shortened tuple reuses the previous op's path; arity disambiguates
```
`encoder().encode(ops): WireOp[]` and `decoder().decode(wire): Op[]` are the only places either exists. Keeping them out of `Op` means `apply` has no id resolution, no `#`
case, and no previous-path state — three branches removed from the hot path —
and dirty-tree generation always works with inline paths before encoding.
`["r", value]` carries no path, so it encodes to itself. That is why `isBase`
works unchanged on either vocabulary.
### 4.1 Two rules that are easy to get wrong
**One encoder/decoder pair per stream.** The id table spans a whole subscription
or file: a path interned in batch 3 is referenced in batch 40. A second consumer
that subscribes at batch 40 has never seen the definition, so it needs its own
encoder. Sharing one across consumers hands the late subscriber ids it cannot
resolve, and a base batch does not rescue it — `["r", value]` carries no refs and
leaves the table empty.
**Reset the table on a base batch.** A reader replays from the *last* base batch
with a fresh decoder, so everything after one must be self-contained. Carrying
ids across a replacement emits references to definitions the reader never saw:
```
4 : [["a",0,"x4"],["a",1,"y4"]]
5 BASE: [["r",{…}]]
6 : [["a",0,"x6"]] <- id 0 was defined in batch 1
-> RECOVERY FAILED: PathError - unresolvable path: 0
```
**Arity omission is scoped to a batch.** Letting it span batches makes a batch's
first op depend on the previous batch's last one, so a reader that skips or
reorders a batch decodes into the wrong path. Ids are the only cross-batch state,
and the dictionary makes those explicit.
### 4.2 Intern on second use, not first
A definition costs more than the path it replaces, so interning on first use
loses on every path written exactly once — and most are. Measured on a bulk
workload: **255.5 KB first-use versus 179.6 KB with no interning at all.**
### 4.3 Measured
Wire bytes against inline paths, round-trip verified:
| stream | inline | wire | saved |
| --- | --- | --- | --- |
| one hot path (a rolling tool-output window) | 11,290 B | 7,538 B | **33.2%** |
| four alternating paths (lane state) | 14,160 B | 11,032 B | **22.1%** |
| 200 distinct paths | 16,455 B | 16,035 B | 2.6% |
Omission does the work in the first case, interning in the second. In the third
neither helps and the definitions cost a little — which is the case second-use
interning exists to bound.
## 5. Flush emits ops, and drops the dead ones
`flush()` computes ops for the dirty paths against the last accepted baseline. It does not retain one op per mutation, so repeated and interleaved writes stay bounded by changed state rather than write count. There is no size comparison or replacement heuristic: a replacement is something the producer asks for by assigning `state` or calling `rebase()`.
An earlier design compared op bytes against the value's serialised size and
replaced when ops were larger. It was removed. Measured against emitting ops
unconditionally, on six workloads, it changed the output in two — both requiring
hundreds of distinct paths to change in a single flush. Neither the harness nor
the replication path does that: `LaneSnapshot` is folded one event at a time, and
the widest case (`run_end`) touches four fields. The rule cost a per-flush
comparison, a running size estimate, and an invalidation rule for the ops that
could not maintain it.
### 5.1 Dirty-tree collapse
The landed tracker records only which subtrees are dirty. At flush it compares each dirty subtree's accepted baseline with its final value and emits the surviving structural change. Repeated writes to one field collapse naturally; alternating writes to two fields retain two dirty paths rather than one op per mutation; a parent replacement subsumes dirty descendants where the final comparison permits it. `packages/chord/test/delta.test.ts` pins the interleaved rolling-window case at at most three ops after 1,000 alternating writes.
This replaces the prototype's backwards dead-op pass and adjacent-only coalescer. Do not port those algorithms into Chord: flush-time generation is the D1 fix.
> **Object key order is not a replicated invariant.** Values round-trip, but delete-and-reinsert activity within one flush can produce a different insertion order on a replica. Do not hash or content-address a replicated value, and sort explicitly where display order matters.
## 6. There is no frame type
The logical batch is `Op[]`; transport and durable storage carry the statefully encoded `WireOp[]`. Do not wrap either batch.
**`seq` does not belong in the payload.** It would defend against a lossy
transport we do not have: a durable list element already carries `seq` from
storage, an in-process callback cannot skip, and SSE either delivers in order or
breaks — and a break means resubscribe, which means a base batch. A second copy of
storage's `seq` could only disagree with the first. The SSE binding stamps `id:`,
which is where transport metadata belongs.
**Nothing else needs wrapping either.** A `kind` discriminator is redundant once a
replacement is an op (§2), and the address already says which value a batch
belongs to. What would remain is a struct around an array.
A producer batch is just `Op[]`, and its encoded boundary form is `WireOp[]`. A **base batch** is one whose first op is `r`;
`isBase(ops)` is `ops[0]?.[0] === "r"`, exact rather than heuristic because flush
guarantees `r` appears at index 0 or not at all (§5).
Everything a frame used to do is now done by something that already existed:
| was on the frame | now |
| --- | --- |
| `seq` | the list element's `seq`, or the SSE `id:` |
| `kind: "replace"` | the `r` op |
| which value it belongs to | the address |
| "this is a snapshot" | the `"base"` storage tag |
**Resubscription is a base batch plus buffered batches** — the same thing the lane
adapter already does for the harness: snapshot first, then whatever accumulated
while the client caught up. There is no `Last-Event-ID` and no cross-subscription
resume; that would need a retained op log, which §5 deliberately does not keep.
A consumer that sees a gap, fails to resolve a path, or connects cold takes one
path: ask for a base batch. That is why there is no `Rebase` type.
The cost, stated plainly: **a consumer cannot detect staleness from the payload
alone.** It relies on the transport reporting a break. For SSE and in-process that
is sound. If a lossy or multiplexed transport ever appears, sequencing goes on that
transport's envelope — not back into the ops.
## 7. Safety
Ops arrive from a facet, a plugin compartment, or a tool whose details may echo
model output. **None of it is trusted input**, and the boundary where untrusted
data meets trusted machinery is where this design has to hold.
The reference point is CVE-2025-55182 — RCE in React Server Components, CVSS 10.0,
exploited in the wild. Their Flight protocol is a compact tagged wire format that
reconstructs structure, so the resemblance is real. The failure was *"fails to
validate the structure correctly... treats the fake object as genuine"*: a forged
Chunk resolved as a Promise and exposed internal state containing gadgets to reach
`Function`.
**We are structurally safer, and not because of diligence:**
| | Flight | ops |
| --- | --- | --- |
| can describe runtime objects | yes — Chunks resolve as Promises | no |
| can reference code or modules | yes — client components | no |
| values | arbitrary object graphs | `JsonValue` |
| worst case from a forged payload | RCE | corrupted replica state |
Flight *must* reference code; that is its job. An op can only put a `JsonValue` at
a path, so there is no first link in a gadget chain. This is why an attacker cannot
plant a non-resolving `then` on `Object.prototype`: a data-only `then` is not
callable, and `await` resolves normally when `then` is not callable.
> **Rule: no op may name something the host resolves.** Not a mutation name, not a
> component id, not a module reference. Mutation names were removed for an
> unrelated reason (§8); this is the second and better reason never to reintroduce
> them, because it is the ingredient that made Flight exploitable.
### 7.1 Paths are the dangerous part
`JSON.parse` is safe on its own — `{"__proto__":{}}` becomes an *own* property.
The hazard is `parent[key]`, which is exactly what applying a path does.
A single key is not enough. `x["__proto__"] = v` swaps *x's own* parent, which is
local. It takes a **walk** — `x["__proto__"]["polluted"] = v` — to reach the shared
prototype, and a path is precisely a walk.
`constructor` is worse than `__proto__`, because `({}).constructor.constructor` is
`Function`. That ladder is closed here only because op values cannot be functions;
it is closed properly by rejecting the segment.
**Reserved path segments: `__proto__`, `constructor`, `prototype`.** Rejected at
record time *and* at apply time, including through an interned path id.
They are reserved as *segments*, not as values. An object with a literal
`"__proto__"` key replicates fine as a whole value; only a path walking *through*
it is refused. This is a genuine restriction on what is mutable — document it, do
not pretend it away.
Two further measures on the same hazard:
- **Write with `Object.defineProperty`**, never assignment, so an inherited setter
cannot run.
- **Resolve own properties only** (`Object.hasOwn`), so a walk cannot escape into
the prototype chain and an inherited getter cannot fire. This blocks
`constructor` a second time, since it is inherited.
What prototype pollution actually buys an attacker here is narrower than it first
appears: **it flips defaults, it does not overwrite explicit values.** An own
property shadows. So `{name:"bob", isAdmin:false}` is unaffected; an option bag read
as `opts.skipSandbox` is not. Our own `ShellExecOptions`, `ShellOutputCaptureOptions`,
`ListReadOptions` and `TrackerOptions` are exactly that shape. Note also that
`"x" in {}` and `const {x = false} = {}` both lie under pollution — which is why
the applier uses `Object.hasOwn`.
### 7.2 Array indices
An index may address an existing element or append exactly one past the end.
This is not an arbitrary cap; it is what keeps the value a `JsonValue`. A sparse
array does not survive a JSON round trip — holes serialise to `null` and return as
real properties — so `arr[7] = x` on a length-3 array already produces state a
replica cannot match. Rejecting the write is more honest than diverging.
It removes a denial of service as a side effect rather than as its purpose:
`["s",["xs",4294967290],1]` would otherwise allocate a 4.29-billion-entry array
from one op. Growth stays available and stays proportional, because
`arr.length = n` is emitted as a splice of explicit nulls whose op size grows with
the gap.
### 7.3 Validate op structure on decode
This is the RSC lesson applied to us. A decoder must not trust tuple shape.
Measured against an unvalidated applier:
| malformed op | result |
| --- | --- |
| `["p",["xs"],0,0,"not-an-array"]` | string spread into the array: `["n","o","t",…]` |
| `["s","a",9]` — path is a string | accepted; `"a".slice(0,-1)` is `""`, so it wrote at the root |
| `["ZZZ",["a"],9]` | silently ignored, replica diverges with no error |
Validate: verb is known, arity matches the verb, a path is an array of strings and
non-negative integers, `p` carries integer index and count plus an array of items,
`#` defines an array path. An unknown verb is an **error**, not a no-op — silently
skipping it is how a newer producer's op disappears and a replica drifts.
**One validator per vocabulary.** `assertValidOp` guards `apply` and rejects ids,
short forms and `#`; `assertValidWireOp` guards `decode` and permits them.
Validating a decoded op against the wire grammar is laxer than its own type: a
two-element `["s", value]` would pass, and `apply` would then read the value as a
path.
### 7.4 The tracker's cage leaks on types, not shape
A facet cannot forge an op: it mutates plain objects and the tracker builds the
tuples, so shape is well-formed by construction. Values and keys are another
matter, and `structuredClone` is **not** a JSON check.
| what a facet writes | what happens |
| --- | --- |
| a function | throws (`DataCloneError`) |
| a BigInt, a cycle | throws |
| `new Map([[1,2]])` | op carries `{}`, producer keeps a real Map — **silent divergence** |
| `new Date(0)` | op carries an ISO string, producer keeps a Date |
| `state[Symbol("s")] = 1` | emits `["s",[null],1]` — **a malformed op from our own tracker** |
So the tracker must check what it is given, at record time, where the mistake is:
- **Reject symbol keys.** They are not path segments.
- **Assert `JsonValue` on every recorded value.** Reject `Map`, `Set`, `Date`,
`RegExp`, typed arrays and class instances explicitly. The scope restriction in
§3.2 was a comment; it has to be a check.
A getter on state is safe — the trap records the computed result — and a facet
value that *looks* like an op is nested inside `["s", path, value]` and can never
be read as a top-level op, because nothing flattens.
### 7.5 Applier
```ts
export function apply<T>(target: T | undefined, ops: readonly Op[]): T;
```
Six verbs, no domain knowledge, no library, no tool code, no registry lookup, and
**no path table** — ids and omitted paths are resolved by `decode` before `apply`
sees anything (§4).
It returns the value rather than mutating in place, because `r` replaces it
outright.
Runs against a **plain mutable object the consumer owns**. It must not be pointed
at a value produced by Immer, which deep-freezes and would throw.
A page of code in any language, which is the property that keeps non-JS consumers
viable. Colyseus ships decoders for C#, Lua and Haxe on the same basis.
## 8. What this removes from the codebase
Concrete deletions, not simplifications in principle:
- **Immer**, from the harness and from the facet layer. See §1 for why a
patch-producing library cannot serve this purpose.
- **Per-type reducers.** There is one applier, `apply(target, ops)`, with no
domain knowledge and no registry. Nobody writes a fold for `ToolOutputState` or
for a plugin's shape.
- **`detailMutations` and `initialDetails`**, and with them the argument that
`details: unknown` forces special handling. A structural tracker never needs
the type.
- **`Rebase` as a reducer return value.** A fold that cannot apply leaves state
alone and the host sends a base batch.
- **Mutation names on the wire**, and therefore mutation-name version skew. Names
never cross a boundary, so adding or renaming one is not a breaking change.
And three things not to build, each of which looks reasonable until §1:
- **A blind whole-value differ.** The landed tracker compares only dirty subtrees against its accepted baseline; it does not scan unrelated state.
- **A keyed op shape plus a codec.** Tuples are the form everywhere (§2).
- **A frame wrapper.** What travels is `Op[]` (§6).
## 9. Durable form
A tracked value is stored as a **list of encoded `WireOp[]` batches**, one appended per flush. Base batches carry the storage tag `"base"`; one stateful decoder per value decodes them before `apply`.
Recovery reads backwards to the last base batch and applies forward:
```ts
readList(address, { order: "desc", stopAtTag: "base", limit: 100 })
```
`stopAtTag` is a stop condition *within a page*: if no base batch is in the page,
the consumer pages again with the cursor. The tag lives on the storage record
beside `seq`, never inside the value, so storage never parses ops. See
[scopes.md](../02-scopes/scopes.md) §11.
This is what makes "a replacement truncates recovery" (§5) real rather than
aspirational — a reader stops at the last base batch instead of replaying from
the beginning.
## 10. Open questions
- Prefix interning (a trie over path heads) if a workload emerges with many
distinct paths sharing long prefixes. Second-use interning (§4) removes the
pathological case; this would go further.
- Whether array reordering needs a dedicated op; the landed tracker currently marks the array dirty and emits the resulting structural/index changes.
- Cross-language replicas: the applier is a page of code in any language, but the
interning tables and the wire framing are not specified for a non-JS consumer.
Address interning in the JSONL log is the same trick as path interning, applied
one layer up over `namespace` + `key`. It is **not** an open question — it is
specified in [scopes.md](../02-scopes/scopes.md) §12 and is a separate dictionary from this one.

563
packages/agent/docs/mobile-handoff/01-harness/01-delta/delta.test.ts Normal file
View File

@@ -0,0 +1,563 @@
import { describe, expect, it } from "vitest";
import { apply, assertJsonValue, assertValidOp, assertValidWireOp, decoder, encoder, isBase, overlap, track, type Op, type WireOp } from "./delta-impl.ts";
const PAD = "p".repeat(400); // see "adaptive emission": a delta must be able to win
describe("tracker: intent", () => {
it("records an append, not a replacement", () => {
const t = track({ s: "", pad: PAD });
t.flush(); // drain the base batch
t.state.s += "ab";
t.state.s += "cd";
const ops = t.flush();
expect(ops).toEqual([["a", ["s"], "abcd"]]);
expect(apply({ s: "", pad: PAD }, ops)).toEqual({ s: "abcd", pad: PAD });
});
it("recovers truncate+append from a rolling window", () => {
// The case every effect-recording library degrades to a whole-value set.
const t = track({ s: "abcdefgh", pad: PAD });
t.flush(); // drain the base batch
t.state.s = t.state.s.slice(3) + "xyz";
const ops = t.flush();
expect(ops).toEqual([["t", ["s"], 3], ["a", ["s"], "xyz"]]);
expect(apply({ s: "abcdefgh", pad: PAD }, ops)).toEqual({ s: "defghxyz", pad: PAD });
});
it("records array intent as one splice", () => {
const t = track({ xs: [1, 2], pad: PAD });
t.flush(); // drain the base batch
t.state.xs.push(3);
expect(t.flush()).toEqual([["p", ["xs"], 2, 0, [3]]]);
});
it("normalises undefined to a delete", () => {
const t = track({ a: 1, pad: PAD });
t.flush(); // drain the base batch
(t.state as Record<string, unknown>).a = undefined;
expect(t.flush()).toEqual([["d", ["a"]]]);
});
});
describe("tracker: root ops", () => {
it("splices a value that is itself an array", () => {
const t = track([1, 2, 3, PAD]);
t.flush(); // drain the base batch
t.state.push(4);
const ops = t.flush();
expect(ops).toEqual([["p", [], 4, 0, [4]]]);
expect(apply([1, 2, 3, PAD], ops)).toEqual([1, 2, 3, PAD, 4]);
});
it("normalises a splice covering the whole root to a replacement", () => {
const t = track([1, 2, 3]);
t.flush(); // drain the base batch
t.state.splice(0, 3, 9);
const ops = t.flush();
expect(ops).toEqual([["r", [9]]]);
expect(isBase(ops)).toBe(true);
});
it("normalises a nested splice-all to a set, not a replacement", () => {
const t = track({ xs: [1, 2, 3], pad: PAD });
t.flush(); // drain the base batch
t.state.xs.splice(0, 3, 9);
expect(t.flush()).toEqual([["s", ["xs"], [9]]]);
});
it("normalises length = 0 on the root", () => {
const t = track([1, 2, 3]);
t.flush(); // drain the base batch
t.state.length = 0;
const ops = t.flush();
expect(ops).toEqual([["r", []]]);
expect(apply([1, 2, 3], ops)).toEqual([]);
});
});
describe("replacing the whole value", () => {
// The proxy cannot observe `tracker.state = next`: `state` belongs to the
// tracker, not to the tracked object, so assigning to it swaps the proxy for a
// plain object and silently stops tracking. Replacement must be a method.
it("assigning to state emits a base batch and keeps tracking", () => {
const t = track<Record<string, unknown>>({ p: 1, q: PAD });
t.flush(); // drain the base batch
t.state.p = 2;
t.state = { r: 9, s: "new" };
const ops = t.flush();
expect(ops).toEqual([["r", { r: 9, s: "new" }]]);
expect(isBase(ops)).toBe(true);
expect(apply({}, ops)).toEqual(t.target);
t.state.r = 10; // the new value is tracked
expect(t.flush()).toEqual([["s", ["r"], 10]]);
});
it("assigning to state discards ops recorded before it", () => {
// They describe a value that no longer exists.
const t = track<Record<string, unknown>>({ p: 1, q: PAD });
t.flush(); // drain the base batch
t.state.p = 2;
t.state = { z: 1 };
expect(t.flush()).toEqual([["r", { z: 1 }]]);
});
it("a partial rewrite keeps its ops", () => {
const t = track<Record<string, unknown>>({ a: "x".repeat(200), b: "y".repeat(200) });
t.flush(); // drain the base batch
t.state.a = "p";
const ops = t.flush();
expect(isBase(ops)).toBe(false);
expect(ops).toEqual([["s", ["a"], "p"]]);
});
it("rebase() forces a base batch without changing the value", () => {
// This is the checkpoint. Recovery replays from the last base batch, so a
// producer must be able to bound that rather than depend on the adaptive
// size rule, which compares each flush independently.
const t = track<Record<string, unknown>>({ p: 1, pad: PAD });
t.flush(); // drain the base batch
t.rebase();
const ops = t.flush();
expect(isBase(ops)).toBe(true);
expect(apply({}, ops)).toEqual(t.target);
});
it("rebase() applies once, not to every later flush", () => {
const t = track<Record<string, unknown>>({ p: 1, pad: PAD });
t.flush(); // drain the base batch
t.rebase();
t.flush();
t.state.p = 2;
expect(t.flush()).toEqual([["s", ["p"], 2]]);
});
});
describe("the first flush", () => {
it("is always a base batch", () => {
// A consumer starts with nothing, so a stream opening with deltas has
// nothing to apply them to. Requiring the producer to remember would fail
// at runtime, in the consumer, rather than where the mistake is.
const t = track<Record<string, unknown>>({ x: 0 });
t.state.x = 100;
const ops = t.flush();
expect(isBase(ops)).toBe(true);
expect(ops).toEqual([["r", { x: 100 }]]); // carries mutations made before it
});
it("is followed by deltas", () => {
const t = track<Record<string, unknown>>({ x: 0, pad: PAD });
t.flush();
t.state.x = 1;
expect(t.flush()).toEqual([["s", ["x"], 1]]);
});
it("emits nothing when untouched", () => {
// An unmutated tracker has nothing to say; the base batch is owed to a
// consumer, and there is no consumer until someone flushes.
const t = track<Record<string, unknown>>({ x: 0 });
expect(t.flush()).toEqual([["r", { x: 0 }]]);
expect(t.flush()).toEqual([]);
});
});
describe("apply and fan-out", () => {
it("adopts an `r` payload rather than copying it", () => {
// The consumer owns the batch it was handed. Copying every base batch
// defensively doubles the memory of the one op carrying the whole value.
//
// The rule that follows: do not hand one batch to two in-process
// consumers. This test pins the aliasing so nobody "fixes" it by adding a
// clone back.
const t = track<Record<string, unknown>>({ n: 0 });
const batch = t.flush();
const a = apply(undefined, batch) as Record<string, unknown>;
const b = apply(undefined, batch) as Record<string, unknown>;
a.n = 1;
expect(b.n).toBe(1);
});
it("does not alias the producer", () => {
const t = track<Record<string, unknown>>({ x: 0 });
const replica = apply(undefined, t.flush()) as Record<string, unknown>;
t.state.x = 999;
t.flush();
expect(replica.x).toBe(0);
});
it("folds a whole stream without a base-batch branch", () => {
// `apply` handles `r` by replacing, and tolerates an undefined target,
// so a consumer needs no isBase check and no clone of its own.
const t = track<Record<string, any>>({ x: 0, l: [] as string[] });
const enc = encoder();
const dec = decoder();
let replica: unknown;
const send = () => { replica = apply(replica, dec.decode(enc.encode(t.flush()))); };
t.state.x = 100;
t.state.l.push("xyz");
send();
t.state.x = 101;
send();
expect(replica).toEqual(t.target);
});
});
describe("dead-op elimination", () => {
// Ops apply in order, so an op is dead if a later one overwrites the whole
// subtree it lives in. Only s/d/r dominate; a/t/p modify what is there.
const run = (mutate: (s: any) => void, init: any = { a: { b: 1 }, x: 1, y: 2 }) => {
const t = track<any>(structuredClone(init));
t.flush(); // drain the base batch
mutate(t.state);
return t.flush();
};
it("collapses repeated writes to one field", () => {
expect(run((s) => { s.x = 1; s.x = 2; s.x = 3; })).toEqual([["s", ["x"], 3]]);
});
it("drops a write superseded by a later, non-adjacent one", () => {
expect(run((s) => { s.x = 10; s.y = 20; s.x = 30; }))
.toEqual([["s", ["y"], 20], ["s", ["x"], 30]]);
});
it("drops a child write when the parent is replaced after it", () => {
expect(run((s) => { s.a.b = 99; s.a = { c: 5 }; })).toEqual([["s", ["a"], { c: 5 }]]);
});
it("keeps a child write that follows the parent replacement", () => {
expect(run((s) => { s.a = { b: 1 }; s.a.b = 7; }))
.toEqual([["s", ["a"], { b: 1 }], ["s", ["a", "b"], 7]]);
});
it("collapses set-then-delete to the delete", () => {
expect(run((s) => { s.x = 5; delete s.x; })).toEqual([["d", ["x"]]]);
});
it("keeps delete-then-set, to preserve key position", () => {
// Dropping the delete would leave the key in its original slot while the
// producer moved it to the end. Same value, different serialisation.
expect(run((s) => { delete s.x; s.x = 5; })).toEqual([["d", ["x"]], ["s", ["x"], 5]]);
});
it("is linear in the number of ops", () => {
// The naive formulation compares every op against every dominator, which
// is quadratic and degrades on exactly the wide flush this pass cleans up.
const wide = (n: number) => {
const root: Record<string, number> = {};
for (let i = 0; i < n; i++) root[`f${i}`] = i;
const t = track(root);
t.flush(); // drain the base batch
const started = performance.now();
for (let i = 0; i < n; i++) t.state[`f${i}`] = i + 1;
t.flush();
return performance.now() - started;
};
wide(200); // warm
const small = Math.max(wide(250), 0.1);
const large = wide(2500);
expect(large / small).toBeLessThan(40); // linear would be ~10x
});
it("collapses a pathological redundant producer", () => {
const t = track<any>({ a: { b: 1 }, x: 1 });
t.flush(); // drain the base batch
for (let i = 0; i < 2000; i++) { t.state.x = i; t.state.a.b = i; }
t.state.a = { done: true };
expect(t.flush()).toHaveLength(2);
});
});
describe("flush", () => {
it("drops everything before a replacement", () => {
const t = track([1, 2]);
t.flush(); // drain the base batch
t.state.push(3);
t.state.splice(0, 3, 7);
expect(t.flush()).toEqual([["r", [7]]]);
});
it("keeps the prefix when the replacement is nested", () => {
// `s` on a subtree is not a root replacement, so earlier ops stay live.
const t = track({ a: 1, xs: [1, 2, 3], pad: PAD });
t.flush(); // drain the base batch
t.state.a = 2;
t.state.xs.splice(0, 3, 9);
expect(t.flush()).toEqual([["s", ["a"], 2], ["s", ["xs"], [9]]]);
});
it("rejects a constructor walk", () => {
// ({}).constructor.constructor is Function — the classic escape ladder.
expect(() => apply({}, JSON.parse('[["s",["constructor","prototype","gadget"],true]]'))).toThrow();
expect(({} as Record<string, unknown>).gadget).toBeUndefined();
});
it("rejects a forbidden path reached through an interned id", () => {
const dict = new Map<number, readonly (string | number)[]>([[0, ["__proto__", "w"]]]);
expect(() => apply({}, [["s", 0, true]], (id) => dict.get(id)!)).toThrow();
});
it("does not run an inherited setter", () => {
Object.defineProperty(Object.prototype, "trap", {
get() { throw new Error("inherited getter ran"); },
configurable: true,
});
try {
expect(() => apply({}, [["s", ["trap"], 1]])).not.toThrow();
} finally {
delete (Object.prototype as Record<string, unknown>).trap;
}
});
it("allows a reserved name as a VALUE key", () => {
// Reserved as segments, not as values.
const out = apply({}, [["s", ["a"], JSON.parse('{"__proto__":{"z":1}}')]]);
expect(({} as Record<string, unknown>).z).toBeUndefined();
expect(out).toBeDefined();
});
});
describe("safety: array indices", () => {
// An index may address an existing element or append exactly one past the end.
// Not an arbitrary cap: a sparse array does not survive a JSON round trip, so
// arr[7] = x on a length-3 array already produces unreplicable state.
it("writes an existing index", () => {
expect(apply({ xs: [1, 2, 3] }, [["s", ["xs", 1], 9]])).toEqual({ xs: [1, 9, 3] });
});
it("appends one past the end", () => {
expect(apply({ xs: [1, 2, 3] }, [["s", ["xs", 3], 9]])).toEqual({ xs: [1, 2, 3, 9] });
});
it("rejects a gap", () => {
expect(() => apply({ xs: [1, 2, 3] }, [["s", ["xs", 5], 9]])).toThrow();
});
it("rejects a huge index", () => {
// Would otherwise allocate 4.29 billion entries from one op.
expect(() => apply({ xs: [] }, [["s", ["xs", 4_294_967_290], 1]])).toThrow();
});
it("allows explicit growth with nulls", () => {
expect(apply({ xs: [1] }, [["p", ["xs"], 1, 0, [null, null, 9]]])).toEqual({ xs: [1, null, null, 9] });
});
});
describe("safety: op structure", () => {
// The CVE-2025-55182 lesson: a decoder must not trust tuple shape.
it("rejects an unknown verb rather than skipping it", () => {
// Silently skipping is how a newer producer's op vanishes and a replica drifts.
expect(() => apply({ a: 1 }, [["ZZZ", ["a"], 9] as never])).toThrow();
});
it("rejects non-array splice items", () => {
// Unvalidated, this spreads a string: ["n","o","t","-","a",…]
expect(() => apply({ xs: [1] }, [["p", ["xs"], 0, 0, "not-an-array"] as never])).toThrow();
});
it("rejects a string path", () => {
// Unvalidated, "a".slice(0,-1) is "", so it resolved to the ROOT and wrote.
expect(() => apply({ a: 1 }, [["s", "a", 9] as never])).toThrow();
});
it("rejects a non-tuple op", () => {
expect(() => apply({ a: 1 }, [{ op: "s" } as never])).toThrow();
expect(() => apply({ a: 1 }, [null as never])).toThrow();
});
it("clamps a splice remove past the end, as Array.prototype.splice does", () => {
// Deterministic and identical on both sides — not a hole.
expect(apply({ xs: [1, 2] }, [["p", ["xs"], 0, 1e9, []]])).toEqual({ xs: [] });
});
});
describe("safety: what a facet can put on the wire", () => {
// A facet cannot forge an op — it mutates plain objects and the tracker builds
// the tuples. Values and keys are the leak, and structuredClone is NOT a JSON
// check: it clones a Map happily, and the op then carries {} while the producer
// keeps a real Map.
const rejects = (name: string, mutate: (s: Record<string, unknown>) => void) =>
it(`rejects ${name}`, () => {
const t = track<Record<string, unknown>>({ pad: PAD });
t.flush(); // drain the base batch
expect(() => mutate(t.state)).toThrow();
});
rejects("a function", (s) => { s.f = () => {}; });
rejects("a Map", (s) => { s.m = new Map([[1, 2]]); });
rejects("a Date", (s) => { s.d = new Date(0); });
rejects("a BigInt", (s) => { s.b = 1n; });
rejects("a cycle", (s) => { const a: Record<string, unknown> = {}; a.self = a; s.c = a; });
rejects("a symbol key", (s) => { s[Symbol("k") as unknown as string] = 1; });
it("allows a getter on state, recording the computed result", () => {
const t = track<Record<string, unknown>>({ pad: PAD });
t.flush(); // drain the base batch
Object.defineProperty(t.target, "g", { get: () => 42, enumerable: true });
t.state.x = 1;
expect(t.flush()).toEqual([["s", ["x"], 1]]);
});
it("accepts a value that merely looks like an op", () => {
// Nested inside ["s", path, value]; nothing flattens, so it cannot be read
// as a top-level op.
const t = track<Record<string, unknown>>({ pad: PAD });
t.flush(); // drain the base batch
t.state.x = ["r", { evil: true }];
expect(t.flush()).toEqual([["s", ["x"], ["r", { evil: true }]]]);
});
});
describe("assertions", () => {
it("assertJsonValue rejects non-JSON", () => {
for (const bad of [new Map(), new Set(), new Date(), /re/, () => {}, Number.NaN, Infinity]) {
expect(() => assertJsonValue(bad)).toThrow();
}
for (const good of [null, true, 1, "s", [1, "a"], { a: { b: [null] } }]) {
expect(() => assertJsonValue(good)).not.toThrow();
}
});
it("assertValidOp accepts decoded ops and rejects wire forms", () => {
// Validating Op against the wire grammar would be laxer than the type: a
// two-element ["s", value] would pass, and apply would read the value as a
// path. Each vocabulary gets the validator that matches it.
for (const op of [
["r", { a: 1 }], ["s", ["a"], 1], ["d", ["a"]],
["a", ["a"], "x"], ["t", ["a"], 2], ["p", ["a"], 0, 0, []],
] as never[]) {
expect(() => assertValidOp(op)).not.toThrow();
}
for (const wireOnly of [["s", 1], ["d"], ["a", "x"], ["t", 2], ["p", 0, 0, []], ["#", 0, ["a"]], ["s", 0, 1]] as never[]) {
expect(() => assertValidOp(wireOnly)).toThrow();
expect(() => assertValidWireOp(wireOnly)).not.toThrow();
}
});
});
describe("codec: path interning and arity omission", () => {
// One pair per stream. The table spans a whole subscription or file, so a
// second consumer joining later needs its own encoder.
const roundTrip = (batches: Op[][]) => {
const enc = encoder();
const dec = decoder();
return batches.map((ops) => dec.decode(enc.encode(ops)));
};
it("round-trips a stream exactly", () => {
const t = track<Record<string, any>>({ a: { deep: "" }, b: { deep: "" } });
t.flush(); // drain the base batch
const batches: Op[][] = [];
for (let i = 0; i < 6; i++) {
t.state.a.deep += `x${i}`;
t.state.b.deep += `y${i}`;
batches.push(t.flush());
}
expect(roundTrip(batches)).toEqual(batches);
});
it("interns on second use, not first", () => {
const enc = encoder();
const p = ["a", "deep"] as const;
const first = enc.encode([["a", p, "1"]]);
const second = enc.encode([["a", p, "2"]]);
expect(first).toEqual([["a", p, "1"]]); // inline
expect(second).toEqual([["#", 0, p], ["a", 0, "2"]]); // define, then use
});
it("omits the path when it repeats", () => {
const enc = encoder();
const p = ["a"] as const;
expect(enc.encode([["s", p, 1], ["s", p, 2]])).toEqual([["s", p, 1], ["s", 2]]);
});
it("resets the table on a base batch, so recovery is self-contained", () => {
// A reader replays from the LAST base batch with a fresh decoder. Ids
// defined before it were never seen, so keeping them breaks recovery with
// an unresolvable path id.
const enc = encoder();
const p = ["a", "deep"] as const;
enc.encode([["a", p, "1"]]);
enc.encode([["a", p, "2"]]); // id 0 assigned
const base = enc.encode([["r", { a: { deep: "x" } }]]);
const after = enc.encode([["a", p, "3"]]);
expect(base).toEqual([["r", { a: { deep: "x" } }]]);
expect(after).toEqual([["a", p, "3"]]); // inline again
const dec = decoder();
expect(() => dec.decode(base)).not.toThrow();
expect(dec.decode(after)).toEqual([["a", p, "3"]]);
});
it("survives recovery from the last base batch", () => {
const enc = encoder();
const t = track<Record<string, any>>({ a: { deep: "" }, b: { deep: "" } });
t.flush(); // drain the base batch
const wire: WireOp[][] = [];
for (let i = 0; i < 8; i++) {
t.state.a.deep += `x${i}`;
t.state.b.deep += `y${i}`;
if (i === 5) t.rebase();
wire.push(enc.encode(t.flush()));
}
const lastBase = wire.map(isBase).lastIndexOf(true);
const dec = decoder();
let replica: unknown;
for (const w of wire.slice(lastBase)) {
const ops = dec.decode(w);
replica = replica === undefined
? structuredClone((ops[0] as ["r", unknown])[1])
: apply(replica, ops);
}
expect(replica).toEqual(t.target);
});
it("round-trips random streams", () => {
for (let round = 0; round < 300; round++) {
const t = track<Record<string, any>>({ a: { p: "", q: "" }, b: [] as unknown[], c: 0 });
t.flush(); // drain the base batch
const batches: Op[][] = [];
for (let i = 0; i < 8; i++) {
const r = Math.random();
if (r < 0.3) t.state.a.p += "x";
else if (r < 0.5) t.state.a.q += "y";
else if (r < 0.65) t.state.b.push(i);
else if (r < 0.8) t.state.c = i;
else if (r < 0.9) delete t.state.c;
else t.rebase();
const ops = t.flush();
if (ops.length > 0) batches.push(ops);
}
expect(roundTrip(batches)).toEqual(batches);
}
});
});
describe("property: random round-trip", () => {
// 3000 random sequences found the root-splice hole on the first run, before any
// hand-written case did.
const rnd = (d = 0): unknown => {
const r = Math.random();
if (d > 2 || r < 0.3) return Math.floor(Math.random() * 5);
if (r < 0.45) return ["x", "y", null, true][Math.floor(Math.random() * 4)];
if (r < 0.7) return Array.from({ length: Math.floor(Math.random() * 4) }, () => rnd(d + 1));
const o: Record<string, unknown> = {};
for (const k of ["a", "b", "c"]) if (Math.random() < 0.6) o[k] = rnd(d + 1);
return o;
};
it("a replica always matches the producer", () => {
let checked = 0;
for (let i = 0; i < 3000; i++) {
const base = rnd();
if (typeof base !== "object" || base === null) continue;
const t = track(structuredClone(base) as object);
t.flush(); // drain the base batch
const next = rnd();
if (Array.isArray(t.state) && Array.isArray(next)) {
(t.state as unknown[]).splice(0, (t.state as unknown[]).length, ...next);
} else if (!Array.isArray(t.state) && typeof next === "object" && next !== null && !Array.isArray(next)) {
const s = t.state as Record<string, unknown>;
for (const k of Object.keys(s)) if (!(k in next)) delete s[k];
for (const [k, v] of Object.entries(next)) s[k] = v;
} else continue;
expect(apply(structuredClone(base), t.flush())).toEqual(t.target);
checked++;
}
// Most random pairs are shape-incompatible and skipped; this only guards
// against the loop silently checking nothing.
expect(checked).toBeGreaterThan(200);
});
});

285
packages/agent/docs/mobile-handoff/01-harness/01-delta/delta.ts Normal file
View File

@@ -0,0 +1,285 @@
// ─────────────────────────────────────────────────────────────────────────────
// harness/delta — intent-recording change tracking over plain JSON.
//
// Depends on nothing else in the harness. Session storage, the runtime and the
// facet host consume it; keep the arrows pointing that way.
//
// State is plain TS. You mutate it normally. Ops come out carrying what you
// *did*, not what changed — `text += chunk` yields an append, not a 50 KB
// replacement; `arr.unshift(x)` yields one splice, not O(n) index writes.
// ─────────────────────────────────────────────────────────────────────────────
export type JsonValue = null | boolean | number | string | JsonValue[] | { [k: string]: JsonValue };
export type Seg = string | number;
export type Path = readonly Seg[];
/** Non-empty: `s`/`d`/`a`/`t` cannot target the root. Enforced by the type. */
export type NonEmptyPath = readonly [Seg, ...Seg[]];
/** A path inline on first use, an interned id thereafter. */
export type PathRef<P extends Path = Path> = P | number;
/**
* Tuples are the form — in memory, on the wire, on disk. There is no keyed
* variant and no codec between them.
*
* `r` is the ONLY op that replaces a whole value. It is an op rather than a
* frame kind for the same reason `a` and `t` exist: they are specialisations of
* `s` that are smaller and state intent. It also makes base detection a token
* comparison rather than a path inspection, which matters because recovery stops
* at the last base batch — a correctness boundary.
*
* Only `p` may target the root, and only because a tracked value can itself be
* an array. A `p` covering its entire target is normalised at RECORD time to `r`
* or `s`, so a root `p` is always a partial modification.
*
* The two-element forms reuse the previous op's path; arity disambiguates.
*/
export type Op =
| readonly ["r", JsonValue]
| readonly ["s", NonEmptyPath, JsonValue]
| readonly ["d", NonEmptyPath]
| readonly ["a", NonEmptyPath, string]
| readonly ["t", NonEmptyPath, number]
| readonly ["p", Path, number, number, JsonValue[]];
/**
* What crosses a boundary. Adds path interning and arity omission, and nothing
* else. `["r", value]` carries no path, so it encodes to itself — which is why
* `isBase` works on either vocabulary.
*/
export type WireOp =
| readonly ["r", JsonValue]
| readonly ["s", PathRef<NonEmptyPath>, JsonValue] | readonly ["s", JsonValue]
| readonly ["d", PathRef<NonEmptyPath>] | readonly ["d"]
| readonly ["a", PathRef<NonEmptyPath>, string] | readonly ["a", string]
| readonly ["t", PathRef<NonEmptyPath>, number] | readonly ["t", number]
| readonly ["p", PathRef, number, number, JsonValue[]]
| readonly ["p", number, number, JsonValue[]]
| readonly ["#", number, Path];
/**
* There is no frame type. What travels is `Op[]`.
*
* A **base batch** is one whose first op is `r`; the first batch of a stream is
* always one, and storage tags it "base" so recovery can stop there. Sequencing
* lives on the transport — a list element's `seq`, or an SSE `id:` — never in the
* payload. See `delta.md` §6.
*/
// ─── Safety ──────────────────────────────────────────────────────────────────
/**
* Path segments that reach the prototype chain. Rejected at record time AND at
* apply time, including through an interned id.
*
* Reserved as SEGMENTS, not as values: an object with a literal `__proto__` key
* replicates fine as a whole value; only a path walking through it is refused.
* That is a real restriction on what is mutable — see `delta.md` §7.1.
*/
export declare const RESERVED_SEGMENTS: ReadonlySet<string>;
export declare class UnsafePathError extends Error {
readonly segment: Seg;
}
/** Rejects reserved segments, symbol keys, and negative or non-integer indices. */
export declare function assertSafePath(path: Path): void;
/**
* Rejects anything `JSON.stringify` would not round-trip: Map, Set, Date, RegExp,
* typed arrays, class instances, functions, BigInt, cycles.
*
* `structuredClone` is NOT a JSON check — it clones a Map happily, and the op then
* carries `{}` while the producer keeps a real Map. Silent divergence.
*/
export declare function assertJsonValue(value: unknown): asserts value is JsonValue;
/**
* Validate an op's structure before applying it. A decoder must not trust tuple
* shape: an unvalidated applier spreads a string into an array given
* `["p",["xs"],0,0,"not-an-array"]`, and writes at the root given a string path.
*
* An unknown verb is an error, not a no-op — skipping it is how a newer producer's
* op vanishes and a replica drifts. This is the CVE-2025-55182 lesson
* (`delta.md` §7.3).
*/
export declare function assertValidOp(op: unknown): asserts op is Op;
/**
* The same, for the wire grammar, where ids and short forms are legal.
* `decode` uses this; `apply` uses `assertValidOp`. Validating a decoded op
* against the wire grammar is laxer than its type — a two-element `["s", value]`
* would pass and then be read as a path.
*/
export declare function assertValidWireOp(op: unknown): asserts op is WireOp;
// ─── Classification ─────────────────────────────────────────────────────────
//
// Ops are tuples, so classification is easy to hand-roll and easy to get wrong.
// `op[0] === "s"` is not enough — a set on any path is "s"; what makes a base is
// that it targets the ROOT. And because a tuple may omit the PathRef (arity
// disambiguates), `op[1]` is a PathRef or a value depending on length. These
// exist so nobody writes that comparison twice.
/** True for the `r` op. */
export declare function isReplace(op: Op): boolean;
/**
* True when this batch begins with `r` — a **base batch**. Exact rather than a
* heuristic, because flush guarantees `r` appears at index 0 or not at all.
* Storage tags such elements `"base"` so recovery can stop at the last one.
*/
export declare function isBase(ops: readonly (Op | WireOp)[]): boolean;
// ─── Codec ───────────────────────────────────────────────────────────────────
/**
* Path interning and arity omission. ONE PAIR PER STREAM — the id table spans a
* whole subscription or file, so a consumer joining later needs its own.
*
* The table resets on a base batch, because a reader replays from the last one
* with a fresh decoder and everything after it must be self-contained.
*/
export declare function encoder(): { encode(ops: readonly Op[]): WireOp[] };
export declare function decoder(): { decode(wire: readonly WireOp[]): Op[] };
// ─── Tracker ─────────────────────────────────────────────────────────────────
export interface TrackerOptions {
/**
* Cap on characters compared when deriving append/truncate from a string
* write. A window slide's overlap lies near the end, so a cap bounds cost
* without changing the result. Default 65536; 0 disables detection.
*/
maxOverlapScan?: number;
/** Merge compatible adjacent ops on flush. Default true. */
coalesce?: boolean;
}
export interface Tracker<T extends object> {
/**
* Mutate this. Plain TS: assignment, push, splice, delete, all of it.
*
* Assigning to it replaces the whole value: emits `["r", next]` and discards
* ops recorded before it. It must be a SETTER on the tracker — a plain
* property would be overwritten, swapping the proxy for a plain object and
* silently ending tracking.
*
* To force a base batch without changing the value, call `rebase()`.
*/
state: T;
/** The untracked backing object. Reads only — writing here emits nothing. */
readonly target: T;
/**
* Drain. Empty when nothing changed.
*
* The FIRST flush is always a base batch, carrying the value including any
* mutations made before it — a consumer starts with nothing, so a stream must
* open with something to apply deltas to.
*
* Ops that a later op makes unreachable are dropped first — a field written
* three times emits one op, and a parent replaced after its child emits one.
* Object key order is not preserved by that pass; see `delta.md` §5.
*/
flush(): Op[];
/** Discard without emitting. For a producer that just sent a base batch. */
discard(): void;
get dirty(): boolean;
}
export declare function track<T extends object>(root: T, options?: TrackerOptions): Tracker<T>;
// ─── Applier ─────────────────────────────────────────────────────────────────
/**
* Apply ops to a plain mutable object. Five verbs plus the dictionary entry, no
* domain knowledge, no library, no registry lookup.
*
* Returns the value, because a root set replaces it outright and cannot be done
* in place. Throws `PathError` when a path does not resolve, which a consumer
* treats exactly like a sequence gap: request a fresh base frame.
*
* Must not be pointed at a value produced by Immer, which deep-freezes.
*/
/**
* Fold ops into a value and return it — returned rather than mutated in place,
* because `r` replaces the value outright. Tolerates an `undefined` target,
* since a root op never reads it, so a consumer needs no base-batch branch.
*
* An `r` payload is **adopted, not copied**: the consumer owns the batch it was
* handed. Do not hand one batch to two in-process consumers, or their replicas
* will alias.
*/
export declare function apply<T>(target: T, ops: readonly Op[]): T;
export declare class PathError extends Error {
readonly path: Path;
}
// ─── Replica ─────────────────────────────────────────────────────────────────
export interface Replica<T> {
/** Current value. `undefined` until the first base frame. */
readonly value: T | undefined;
/**
* Feed one batch, with the sequence number the transport supplied. Returns
* "gap" when `seq` is not contiguous or a path failed to resolve; the caller
* then requests a fresh base batch.
*/
receive(ops: readonly Op[], seq: number): "ok" | "gap";
}
export declare function replica<T>(): Replica<T>;
// ─── Semantics ───────────────────────────────────────────────────────────────
//
// ROOT OPS
// `["s", [], v]` replaces the value; `["p", [], …]` splices it when the value
// is an array. The applier must special-case `path.length === 0` before
// resolving a parent — a prototype handling only a root set threw on the first
// randomised sequence that produced a root splice.
//
// STRINGS
// A write of `next` over `prev` derives ops from the longest suffix of `prev`
// that is a prefix of `next`:
// overlap === prev.length -> append(next.slice(overlap))
// overlap > 0 -> truncate(prev.length - overlap) + append(rest)
// overlap === 0 -> set(next)
// Always correct: the overlap is verified, not guessed. This is what makes a
// rolling window cheap — the case every effect-recording library degrades on.
//
// Use a native probe, not a hand-written KMP: `indexOf` a short head of `next`
// in `prev`, verify with `endsWith`. Measured 47x faster over 2000 slides on a
// 50 KB string, verified identical across 200.
//
// `t` counts UTF-16 code units, matching String.prototype.slice. Not bytes.
//
// ARRAYS
// Mutator methods are intercepted in the `get` trap, so intent is recorded
// before the engine performs index writes:
// push/unshift/pop/shift/splice -> one `p`
// sort/reverse/fill/copyWithin -> `s` of the whole array (rare)
// Direct index writes are honest `s` ops. `a.length = n` is translated —
// shrink to a splice, grow to a splice of nulls — never emitted as a path,
// because `length` is not a document location.
//
// TRANSACTIONS
// Several writes to one address in one commit must each be recorded against
// the previous one's result, not against pre-transaction state. The applier
// replays in order; anything else applies the second against a stale base.
//
// ALIASING
// Op payloads are structured-cloned at record time. Without this a later
// mutation retroactively changes an already-emitted op and a local replay
// aliases the source — an observed bug, not a hypothetical.
//
// UNDEFINED
// `x = undefined` normalises to `d`. JSON has no undefined, and a set with a
// missing value is indistinguishable from a lost value after a round trip.
//
// SCOPE
// JsonValue only. No Map, Set, Date, RegExp, class instances, getters, symbol
// keys, or cycles. That keeps this to five traps and one string algorithm, and
// makes structuredClone on payloads always safe.

604
packages/agent/docs/mobile-handoff/01-harness/02-scopes/implementation-handoff.md Normal file
View File

@@ -0,0 +1,604 @@
# Scoped storage Step 1 — actionable implementation handoff
**Status: actionable, not implemented.**
**Prepared against:** `c4b0e35ab` (`dev`). Re-audit named source files if the implementation baseline changes materially.
This handoff implements storage scopes only. **Read this file before [`scopes.md`](scopes.md).** It supersedes conflicting details there, especially per-scope sequence spaces, operation-derived orphan detection, one-use scope IDs, raw-string scope arguments, sidecars ending in `.jsonl`, the proposed interned retirement tuple, and bundling JSONL delta/address encoding into the same change. The broader motivation and measurements in `scopes.md` remain design evidence; its signatures and implementation order are not current where this file differs.
Step 2—Chord-encoded JSONL values, address interning, compact tuple records, and their measurements—is a separate package. Do not begin Step 2 until Step 1 has been tested, reviewed, explicitly approved by the user, committed, and pushed.
## 0. Delivery protocol
This package has a mandatory user checkpoint:
1. Implement Step 1 only.
2. Run every required focused test, `npm run check`, and `./test.sh`.
3. Review the complete diff and report results. Delegated reviews, if used, run with provider `anthropic` and model `claude-fable-5`.
4. **Stop and wait for explicit user approval.**
5. After approval, commit only this package's files and push that commit.
6. Confirm the pushed commit before designing or implementing Step 2.
Do not combine the Step 1 and Step 2 commits. Do not commit or push before approval.
## 1. Mandatory reading
Read completely before editing:
1. `packages/agent/docs/harness.md` §§0.2–0.9, 1.1–1.7, 2.7–2.8, 3.7–3.8, 3.13, 4.4–4.8, 5.4, Part 7, Part 9.
2. `packages/agent/docs/values.md`.
3. `packages/agent/docs/mobile-handoff/README.md`.
4. [`scopes.md`](scopes.md), using this handoff where they conflict.
5. `../01-delta/delta.md` for vocabulary ownership only; Step 1 does not store Chord ops.
6. `../04-tool-output/harness-tools.md` §§7.1–7.7 for later-consumer context; do not implement its output redesign here.
7. `../05-assistant-output/message-update.md` §7 for later-consumer context; do not implement it here.
8. `packages/agent/src/harness/session/{types,values,commit,in-memory-storage-state,memory,session,index}.ts`.
9. Every file under `packages/agent/src/harness/session/jsonl/` and the `FileSystem` declaration/implementation in `packages/agent/src/harness/{types,env/nodejs}.ts`.
10. `packages/agent/src/harness/runtime/{progress,lane}.ts`, `runtime/drive/{terminal,response,deferred,tools,reconcile,structural,boundary}.ts`, and every caller of `operationCleanupWrites`. Inspect `tool-placement.ts` through the grep audit, but current source has no scoped delete there.
11. `packages/session-backends/sqlite-node/src/{index,sqlite/index}.ts`, `sqlite/{storage,repo}.ts`, `migrations/001_initial.sql`, and `session/{values,session-sequences}.ts`.
12. Shared storage/repository conformance and every focused test named in §9.
13. `scopes.variance.ts` for the covariant-address/invariant-write phantom mechanism only. Its current no-ID `EphemeralScope` and `retireScope(id: string)` signatures are superseded by §2 and must be updated in Slice A.
Do not use `dist/` as implementation input. Format 4 remains WIP; no R11 migration is part of this package.
## 2. Fixed scope contract
### 2.1 Runtime scope identity
A scope ID is a reusable logical lifetime name, not a globally one-use token:
```ts
export type SessionScope = { readonly kind: "session" };
export interface EphemeralScope {
readonly kind: "ephemeral";
readonly id: string;
}
export type Scope = SessionScope | EphemeralScope;
export function ephemeralScope(id: string): EphemeralScope;
```
`ephemeralScope(id)` requires a non-empty well-formed Unicode string and returns a frozen value. Equal IDs identify the same physical scope; object identity has no meaning. JSONL encodes IDs with `encodeURIComponent`. The encoded component must be at most 180 ASCII characters; reject longer IDs and lone-surrogate input. The physical filename always has a fixed `scope-` prefix, so encoded `.`/`..` values cannot become path segments. Do not directly interpolate unencoded caller input into a path.
No create/open-scope transaction exists. The first scoped value/list write creates physical state. A later `retireScope(scope)` ends everything in that scope through its assigned global sequence. A write under the same ID after retirement begins a new logical lifetime.
Harness operation scopes use `ephemeralScope(operationId)`. Generic application scopes may use stable names. Storage never parses a scope ID as an operation ID and never reads Harness operation state to decide lifetime.
### 2.2 Reuse and retirement boundary
Global sequence order defines reusable generations:
```text
seq 10 scoped write S
seq 20 scoped write S
seq 30 retireScope(S)
seq 40 scoped write S # new lifetime
```
For a given scope ID, only scoped records with `seq` greater than its latest committed retirement sequence are live. Memory and SQLite realize this by deleting current rows/maps at retirement and allowing later writes to recreate them. JSONL uses the retirement sequence as its replay boundary (§6).
Retirement is idempotent in the storage sense: retiring a scope with no current state is legal and advances the boundary. A later write still begins a new lifetime. Storage does not maintain an all-time used-ID registry or reject reuse.
A scoped write reaching the commit line after retirement is therefore a new-lifetime write. Harness invocation fencing and settlement drains must continue to ensure a late assistant/tool progress job cannot recreate operation output after terminal retirement. Add explicit tests for this boundary.
### 2.3 Explicit retirement is the sole authority
```ts
export function retireScope(scope: EphemeralScope): Write<SessionScope>;
```
Storage never infers orphanage from operation absence, namespace, current state, or a missing owner. The only valid lifetime boundary is a committed `retireScope` write. If an owner never retires its scope, that is an owner defect and the scope remains live.
Crash behavior:
- crash before the retirement transaction commits: the scope remains live and reopens;
- crash after retirement commits but before JSONL unlink: replay sees the retirement boundary, ignores pre-boundary sidecar records, and retries physical deletion;
- crash during a later reused lifetime: records after the latest retirement reopen normally.
Repository deletion separately removes every physical sidecar belonging to the deleted Session. It does not need operation semantics.
### 2.4 Address and write typing
Addresses carry a covariant scope tag; writes carry an invariant scope tag. Preserve value-type invariance independently:
```ts
interface Value<T, Sc extends Scope = SessionScope> { /* existing fields + scope */ }
interface ValueList<T, Sc extends Scope = SessionScope> { /* existing fields + scope */ }
declare function value<T>(namespace: string, key?: string): Value<T, SessionScope>;
declare function value<T>(namespace: string, key: string, scope: EphemeralScope): Value<T, EphemeralScope>;
declare function list<T>(namespace: string, key?: string): ValueList<T, SessionScope>;
declare function list<T>(namespace: string, key: string, scope: EphemeralScope): ValueList<T, EphemeralScope>;
```
A session-scoped address has no runtime scope ID. An ephemeral address carries its scope ID. Scope is part of physical identity: equal namespace/key addresses in session scope and ephemeral scope, or in two different ephemeral IDs, are distinct.
`setValue`, `deleteValue`, `appendList`, and `deleteList` preserve the address scope in `Write<Sc>`. Entry and usage writes are session-scoped. `retireScope` is session-scoped even though applying it deletes ephemeral state.
All writes in an ordinary transaction have one static scope. A session transaction may include session values, entries, usage, and one or more `retireScope` writes. It may not include a direct ephemeral set/delete/append. An ephemeral transaction may include value/list writes only. At runtime, every ephemeral write in one transaction must carry the same scope ID; two different IDs have the same TypeScript tag and require this assertion.
Make the minimum generic changes necessary to preserve this through `Write`, `CommittedWrite`, `Storage.commit`, Session mutation capabilities, `CommitDecision`, and lane commands. Follow the variance proof in `scopes.variance.ts`; do not make readers invariant or propagate unnecessary scope type parameters through read-only APIs.
### 2.5 One global sequence space
Memory, JSONL main records, JSONL sidecar records, and SQLite rows share one Session-global sequence space. Every admitted commit remains serialized and receives increasing sequences in admission order. Gaps remain legal.
- Memory continues using one `nextSeq`.
- SQLite continues allocating from `sessions.next_seq` inside the write transaction.
- JSONL main and sidecar appends share one commit queue and one resident `nextSeq`.
JSONL open computes the high-water mark from the main header and every complete record in every matching sidecar **before** deleting or ignoring retired physical files. A retired sidecar's sequences may then become gaps but can never be reused. Torn uncommitted final transactions do not advance the durable high-water mark.
No reservation record, per-scope sequence counter, or range allocator is permitted.
## 3. List tags
Step 1 adds only the mechanism required by later tracked-output recovery:
```ts
export interface ListElement<T> {
seq: number;
value: T;
tag?: string;
}
export interface ListReadOptions {
cursor?: ListCursor;
order?: "asc" | "desc";
limit?: number;
stopAtTag?: string;
}
export function appendList<T, Sc extends Scope>(
address: ValueList<T, Sc>,
element: NoInfer<T>,
tag?: string,
): ListAppendWrite<Sc>;
```
Tags are non-empty strings when present. Storage stores and returns the tag without interpreting the element.
Read semantics:
1. Apply address scope, exclusive cursor, and order.
2. Inspect at most the normalized `limit` elements.
3. Stop at and include the first element carrying `stopAtTag`.
4. Return that possibly shortened page.
`stopAtTag` never searches beyond the page limit. If absent from the page, the caller pages again from the last returned sequence. Do **not** add the draft `tag` filter; no current consumer needs it and combining filtering with the existing cursor shape is ambiguous.
SQLite may fetch the indexed `limit` rows and truncate at the first returned tag in TypeScript; storage still never parses payloads. Preserve the existing primary-key query plan with no temporary sort.
## 4. Step 1 durable payloads remain unchanged
This package changes lifetime and routing, not output representation:
- `pendingToolOutput` remains `Value<AgentToolResult<unknown>, EphemeralScope>`;
- `pendingAssistantFrames` remains `ValueList<AssistantMessageFrame, EphemeralScope>`;
- `operationToolMemo` remains a `JsonValue` scalar;
- JSONL transaction records remain readable keyed objects;
- assistant progress still appends one frame per accepted frame;
- tool checkpoints still replace whole snapshots;
- `message_update` and `tool_update` event shapes do not change;
- safe replay retains its current checkpoint behavior, including the known delete-before-replay bug owned by the later tool-output package.
Do not rename `pendingAssistantFrames`, introduce `pendingAssistantOutput`, store `WireOp[]`, add Chord trackers/codecs, redesign `AgentHarnessTool`, or change output cadence here.
## 5. Runtime ownership and cleanup
### 5.1 Addresses moved to the operation scope
Construct these addresses with `ephemeralScope(operationId)`:
- `operationToolMemo(operationId, invocationId, name)`;
- `pendingToolOutput(operationId, invocationId)`;
- `pendingAssistantFrames(operationId, responseEntryId)`.
All other current built-ins remain session-scoped. In particular, `pi.op.state`, `pi.op.meta`, `pi.op.tool_args`, `pi.op.preparation`, and `pi.pending.entry` remain in the main scope because their writes coordinate atomically with lane/operation state.
### 5.2 No cross-file cleanup transaction
Remove direct ephemeral deletes from transactions that also write session state:
- assistant response settlement;
- deferred-response supersession;
- tool outcome staging in `drive/tools.ts`;
- cancellation/recovery decisions;
- terminal per-address cleanup.
Current `drive/tool-placement.ts` has no ephemeral delete; do not invent a change there.
`operationCleanupWrites` stops scanning tool memos and tool outputs and stops constructing a state-directed assistant-frame delete. It keeps session-scoped operation/meta/args/preparation/pending-entry cleanup and adds exactly one `retireScope(ephemeralScope(operationId))` in the universal terminal suffix.
After removing the current staging/terminal scans, remove `operationToolMemoPrefix` and `pendingToolOutputPrefix`; current source has no other production consumers. Update the normative exported prefix-constructor count from five to three and update exact constructor tests rather than retaining dead inventory APIs.
An ephemeral-only delete remains legal. For example, the current safe-replay checkpoint delete remains because its transaction contains no session write; fixing its behavior belongs to the later tool-output package, not scoped storage or JSONL optimization.
### 5.3 Unreachable scoped residue
After a response settles, a deferred response is superseded, or a tool reaches `outcome_ready`, its former frames/checkpoints/memos may remain physically and logically addressable inside the still-active operation scope. Current scalar operation state no longer references them; no recovery or snapshot path scans the scope to infer authority. The terminal retirement deletes the whole scope.
Call this **unreachable scoped residue**, not orphanage. Update `harness.md` invariants that currently require an `outcome_ready` call to have no physically present memo/checkpoint. The replacement invariant is that settled child state never consumes or exposes residue and terminal retirement removes all scoped state atomically with operation completion.
Close and fault are controlled crashes and never retire the scope. Reopen restores scoped progress only through addresses derived from current authoritative operation state.
## 6. JSONL sidecars
### 6.1 Physical layout
The main session file remains unchanged except for committed scope-retirement records. Every main file owns one known sidecar directory at `${mainPath}.scopes`; this requires no dirname API or `FileSystem` extension. Create it lazily with `createDir(..., { recursive: true })`. Discover scopes with `listDir` on that exact path. A sidecar filename is exactly `scope-${encodeURIComponent(scope.id)}.scope`, subject to §2.1's 180-character encoded-component limit. The suffix does not end in `.jsonl`, and the sidecar directory itself is a directory, so repository listing never mistakes either for a Session.
A sidecar begins with this exact header:
```ts
interface JsonlScopeHeader {
v: 4;
kind: "scope_header";
sessionId: string;
scopeId: string;
storageVersion: 1;
}
```
Validate the exact session/scope identity before replay. Create the first header through a temp file inside `${mainPath}.scopes` plus atomic rename. Temporary filenames end in `.tmp` and are never discovered as sidecars. Use the existing `FileSystem` capability; do not add a dirname method, extend `JsonlStorageOptions`, or add Node-only filesystem access to agent core.
Step 1 sidecar writes retain the current object record spelling and carry/validate their scope ID in committed shapes. Each physical transaction remains one complete line or one array line. Sidecar transactions may contain only value/list writes for that exact scope.
### 6.2 Retirement record
The main log records an explicit committed write:
```ts
interface CommittedScopeRetireWrite {
kind: "scope";
op: "retire";
seq: number;
scopeId: string;
}
```
Its Step 1 JSON representation is the corresponding readable object. Do not use the draft `['!', addrId, seq]`: a scope is not an interned value/list address. Compact tuple spelling belongs to Step 2 and must be designed there.
Applying a committed retirement removes resident values/lists in that scope. After the complete main transaction is durable, JSONL synchronously serializes lifecycle cleanup on the same commit queue: close any owned sidecar resource, attempt unlink, then release the queue. Failure to unlink is non-fatal to the already-committed business transaction; the sidecar remains logically bounded by the retirement sequence and deletion is retried on reopen.
### 6.3 Replay and reusable IDs
Open performs these logical stages before admitting writes:
1. Read/repair the main file and parse complete main transactions.
2. Discover and validate matching sidecars; ignore unrelated files.
3. Parse complete sidecar transactions and repair torn final transactions.
4. Compute the global sequence high-water mark from the header and **all** complete main/sidecar records, including records later excluded by retirement.
5. Determine the latest retirement sequence per scope from main transactions.
6. Keep sidecar writes whose sequence is greater than that scope's latest retirement.
7. Merge retained main and sidecar transactions by global sequence and replay them in global order, preserving transaction boundaries and rejecting duplicate/non-monotonic sequences.
8. Advance to the precomputed high-water mark.
9. Remove sidecars with no records after their latest retirement; cleanup failure does not make retired contents live.
Main transactions have sequence gaps where sidecar transactions occurred. Sidecars have gaps where other scopes/main commits occurred. Both remain valid.
A sidecar containing records after a retirement is an active reused lifetime and must not be deleted. Serialize retirement cleanup against later writes so an old unlink cannot race and remove a newly reused sidecar.
Malformed interior records or identity mismatch in an active sidecar are storage corruption and fail open. A torn final transaction is discarded wholly. Temporary files left by interrupted atomic creation are cleanup artifacts and never become sidecars.
### 6.4 Legacy v3
Legacy v3 has no scopes. An ordinary Harness operation commits session-scoped acceptance before any progress write, so it upgrades first in the normal path. Still define and test the generic storage behavior when the first caller write is ephemeral: complete the existing v3-to-v4 main rewrite/usage adjustment under the global commit queue, then commit the scoped transaction without placing it in the main file. A crash before the sidecar append leaves no committed caller transaction and may reuse its uncommitted sequence.
### 6.5 Repository and fork behavior
- Repository `list()` ignores `${mainPath}.scopes` directories and their files through its existing non-file/`.jsonl` filtering.
- Repository `delete()` removes the main file through its current path, then removes `${mainPath}.scopes` recursively with `force: true`; a scope-directory cleanup failure rejects deletion after the main-file result is known, without inferring Harness state. Missing scope directories are legal.
- `JsonlStorage.close()` drains its admitted main/sidecar commits as today; `JsonlSessionRepo.close()` retains its current lifecycle behavior because repository ownership is a separate package. No progress scope is retired merely because a handle closes.
- Preserve the current fork allowlist exactly. Do not add generic application values or lists to either fork scope. Ensure source snapshots and backend fork readers ignore all ephemeral values/lists regardless of namespace; current destinations remain logically unchanged except that ephemeral state cannot leak.
- Fork destinations contain no sidecars or retirement state.
- WP08 may later replace fork materialization but must preserve this scope policy.
J1 compaction is excluded. When J1 eventually lands, it may omit a historical retirement record only after no physical sidecar can be made live by losing that boundary.
## 7. Memory backend
Extend `InMemoryStorageState` physical value/list identity with scope. Session scope and each ephemeral ID are distinct. Preserve one global `nextSeq` and current stats behavior.
Applying `CommittedScopeRetireWrite` deletes every scalar/list in that scope in the same synchronous application of the surrounding transaction. It does not delete entries or usage, which cannot be ephemeral. A later scoped write recreates state under the same ID.
Snapshots expose scope so fork code can reject ephemeral state, but `createForkSnapshot` keeps its current built-in allowlist and must not start copying generic session-scoped application values or lists. Instrumentation must still report exact committed write order, including retirement.
## 8. SQLite backend
Change WIP schema in place with no migration:
```sql
scalar_values(
session_id TEXT NOT NULL,
scope_id TEXT NOT NULL,
namespace TEXT NOT NULL,
key TEXT NOT NULL,
seq INTEGER NOT NULL,
value TEXT NOT NULL,
PRIMARY KEY(session_id, scope_id, namespace, key)
) WITHOUT ROWID;
list_values(
session_id TEXT NOT NULL,
scope_id TEXT NOT NULL,
namespace TEXT NOT NULL,
key TEXT NOT NULL,
seq INTEGER NOT NULL,
value TEXT NOT NULL,
tag TEXT,
PRIMARY KEY(session_id, scope_id, namespace, key, seq)
) WITHOUT ROWID;
```
Use `scope_id = ''` for session scope and the exact scope ID otherwise. Every point read, prefix scan, set/delete, append/delete, snapshot, and query-plan assertion includes scope identity.
Inside the same `BEGIN IMMEDIATE` transaction, a retirement write executes:
```sql
DELETE FROM scalar_values WHERE session_id = ? AND scope_id = ?;
DELETE FROM list_values WHERE session_id = ? AND scope_id = ?;
```
It still consumes its assigned global sequence through `sessions.next_seq`, even though no retired-scope tombstone row remains. A later write under that ID recreates rows naturally. Retirement rollback must restore both scoped rows and all sibling main writes.
Fork paths copy only `scope_id = ''`. Shared-container deletion remains Session-scoped and removes all values/lists regardless of scope along with the other Session rows.
## 9. Required tests
Port tests before or with each implementation slice. Use the repository Vitest binary from the owning package. Do not run the full Vitest suite directly.
### 9.1 Type and constructor tests
- Session addresses default to `SessionScope`.
- Ephemeral overloads retain `EphemeralScope` while preserving invariant `T`.
- Address scope is covariant for reads; write scope is invariant.
- Same-scope session transactions typecheck.
- Same-scope ephemeral transactions typecheck.
- Session writes plus `retireScope` typecheck.
- Direct session + ephemeral writes in one transaction fail with `@ts-expect-error`.
- Two ephemeral IDs evade static distinction but fail the runtime same-ID assertion.
- Scope object identity is irrelevant; equal IDs access the same state.
- Empty and overlong encoded IDs reject; separators/Unicode encode without path escape.
- Session and ephemeral addresses with equal namespace/key do not alias.
### 9.2 Shared backend conformance
Run identically on Memory, JSONL, and SQLite:
- ephemeral scalar set/get/replace/delete;
- ephemeral list append/page/whole-list-delete;
- independent scopes with equal namespace/key;
- session scope independent from equal ephemeral address;
- mixed session/ephemeral transaction rejection before mutation;
- two ephemeral IDs in one transaction rejection before mutation;
- retirement deletes every value/list in the exact scope and no other scope;
- retirement atomic with session sibling writes;
- retirement of absent state is legal;
- write after retirement recreates a new lifetime;
- repeated retirement advances the boundary without making later writes disappear;
- main, scope A, scope B, retirement, and reused-A commits receive globally increasing sequences in admission order;
- retired/deleted physical elements leave legal gaps and first post-reopen commit allocates above every complete prior record;
- list tags round-trip, page limits remain bounded, and ascending/descending `stopAtTag` includes the marker;
- stats ignore scoped values/lists and retirement;
- close drains admitted scoped commits and rejects later reads/writes.
### 9.3 JSONL focused tests
- first scoped write creates exactly one valid sidecar and no main value/list record;
- several writes in one scoped transaction remain one physical sidecar line;
- main and multiple sidecars replay in global sequence order;
- header `nextSeq` and all complete sidecar records contribute to reopen high-water;
- torn sidecar final transaction is discarded wholly and repaired;
- malformed interior active sidecar and header identity mismatch fail open;
- crash before retirement leaves scoped data live;
- crash after main retirement but before unlink leaves data logically absent and cleanup retries;
- unlink failure does not reject the already-committed terminal transaction;
- reuse after retirement replays only records after the latest retirement sequence;
- multiple retirement/reuse cycles choose the latest boundary;
- retirement cleanup is serialized so a later reuse cannot be deleted by an earlier unlink;
- ignored pre-boundary records still prevent sequence reuse;
- sidecars and temp files never appear in repository listing;
- repository deletion removes all sidecars;
- forks exclude generic application-owned ephemeral values/lists;
- legacy-v3 first session write and first scoped-write paths both produce valid v4 state;
- close/reopen preserves active operation progress and does not retire it.
### 9.4 SQLite focused tests
- schema and query helpers include `scope_id`; list rows preserve nullable tags;
- one transaction atomically deletes scoped scalar/list rows and writes terminal main state;
- forced failure after scope deletion rolls back deletion and sibling writes;
- reuse after retirement recreates rows under the same scope ID;
- `sessions.next_seq` advances for retirement;
- list paging plan still uses the scoped primary key with no temp b-tree;
- both per-file and shared-container layouts isolate Session and scope IDs;
- fork and Session deletion exclude/remove scoped rows correctly.
### 9.5 Harness integration
Update existing exact-write tests rather than weakening them:
- assistant/deferred settlement drops mixed-scope frame deletes;
- tool outcome staging drops mixed-scope checkpoint/memo deletes;
- unreachable residue is never consumed by snapshot/recovery after its owning child settles;
- safe-replay's current ephemeral-only delete remains ordered and fenced;
- every terminal leaf emits one retirement in the documented cleanup position;
- close/fault emits no retirement and reopen recovers current frames/checkpoint/memos;
- cancellation and unknown-outcome recovery retire only at terminal completion;
- late queued assistant/tool progress cannot commit after terminal retirement or reopen the scope;
- terminal cleanup no longer scans memo/output prefixes;
- every focused test and fixture found by the §12 old-delete/prefix greps is updated deliberately; do not rely on a historical numeric test count;
- every terminal path reached through `operationCleanupWrites` callers in `drive/{response,reconcile,structural,boundary}.ts` is covered, including each operation family and cancellation.
Likely focused files include:
```text
packages/agent/test/harness/values.test.ts
packages/agent/test/harness/{memory,jsonl}-storage*.test.ts
packages/agent/test/harness/{memory,jsonl}-session-repo*.test.ts
packages/agent/test/harness/runtime/drive-{terminal,retry-deferred,tools,reconcile,generation}.test.ts
packages/session-backends/sqlite-node/test/{storage,storage-conformance,repo,repo-conformance}.test.ts
```
Use compiler and grep guards to find additional affected tests rather than assuming this list is exhaustive.
## 10. Implementation slices
### Slice A — contract, types, Memory reference, list tags
Primary files:
```text
CREATE packages/agent/src/harness/session/scope.ts for `SessionScope`, `EphemeralScope`, `Scope`, `ephemeralScope`, scope phantoms/helpers, and runtime scope-ID validation
MODIFY packages/agent/src/harness/session/{types,values,commit,in-memory-storage-state,memory,session,index}.ts
MODIFY packages/agent/src/harness/session/testing/{conformance/storage,instrumented-storage,storage-decorator,gating-storage}.ts
MODIFY packages/agent/test/harness/{values,memory-conformance,memory-storage,storage-backed-session}.test.ts
MODIFY packages/agent/docs/mobile-handoff/01-harness/02-scopes/scopes.variance.ts
```
1. Add scope values/address/write variance and reusable-ID semantics.
2. Add retirement committed write and one-scope transaction validation.
3. Add tag/`stopAtTag` behavior.
4. Implement Memory physical identity, retirement, global sequences, snapshots.
5. Pass type tests and Memory conformance before continuing.
### Slice B — SQLite
Primary files:
```text
MODIFY packages/session-backends/sqlite-node/src/sqlite/migrations/001_initial.sql
MODIFY packages/session-backends/sqlite-node/src/sqlite/{storage,repo}.ts
MODIFY packages/session-backends/sqlite-node/src/sqlite/session/{values,session-sequences}.ts
MODIFY packages/session-backends/sqlite-node/test/{storage,storage-conformance,repo,repo-conformance}.test.ts
```
1. Add scoped schema/query identity and tags.
2. Implement atomic `DELETE ... WHERE scope_id` retirement.
3. Preserve global sequence allocation, stats, branch indexes, shared-container behavior.
4. Exclude ephemeral rows from forks.
5. Pass SQLite focused/conformance tests before continuing.
### Slice C — JSONL sidecars
Primary files:
```text
CREATE packages/agent/src/harness/session/jsonl/scope-files.ts for fixed directory/file naming, exact header parsing/serialization, discovery, atomic first creation, torn-tail parsing/repair, and recursive cleanup helpers
MODIFY packages/agent/src/harness/session/jsonl/{types,codec,storage,repo,legacy-v3,index}.ts
MODIFY packages/agent/test/harness/jsonl-{storage,storage-conformance,session-repo,session-repo-conformance,v3-migration}.test.ts
```
1. Add sidecar naming/header/create/discovery.
2. Route scoped commits to sidecars while allocating globally.
3. Persist main retirement records and serialized unlink.
4. Implement multi-file high-water calculation and globally ordered replay with reusable boundaries.
5. Handle torn tails, corruption, v3, listing, delete, close, and forks.
6. Pass all JSONL tests before runtime migration.
### Slice D — Harness migration
Primary files:
```text
MODIFY packages/agent/src/harness/session/values.ts
MODIFY packages/agent/src/harness/runtime/{progress,lane}.ts to propagate scoped write generics through progress channels and lane commands
MODIFY packages/agent/src/harness/runtime/drive/{terminal,response,deferred,tools,reconcile,structural,boundary}.ts
INSPECT packages/agent/src/harness/runtime/drive/tool-placement.ts; current source needs no scoped cleanup change
MODIFY all focused runtime tests found by the §12 greps
```
1. Scope memo/tool/frame addresses by operation ID.
2. Remove mixed-scope deletes and cleanup scans.
3. Add terminal retirement in the universal suffix.
4. Preserve current output/replay/event behavior.
5. Prove late-write fencing and every terminal/close/recovery path.
### Slice E — documentation and final validation
Update current normative/reference docs:
```text
packages/agent/docs/harness.md
packages/agent/docs/values.md
packages/agent/docs/post-wp05-roadmap.md
packages/agent/docs/mobile-handoff/README.md
packages/agent/docs/mobile-handoff/01-harness/02-scopes/scopes.md
packages/session-backends/sqlite-node/README.md
packages/agent/src/harness/telemetry.ts and generated `packages/agent/docs/telemetry-schema.md`: add the `scope` session-write item kind without implementing spans
```
Do not rewrite historical WP00–WP07 handoffs or released changelog sections. On `dev`, do not add changelog entries under the repository's main-only rule.
Run:
```bash
npm run check
./test.sh
```
Also run every modified focused test after each slice. Record full command results for the approval checkpoint.
## 11. Exclusions
Do not include:
- Chord `Op`/`WireOp` storage integration;
- JSONL address/path interning or compact tuple records;
- pending-output address renames;
- `ToolOutput`, tool API changes, replay seeding, memo/checkpoint atomicity, progress cadence, rate limiting, exec-env changes;
- compact `message_update`, assistant incremental reducer changes, or protocol replication changes;
- J1 main-log snapshot compaction;
- WP08 fork redesign beyond excluding ephemeral state from current forks;
- repository lifecycle contract changes;
- telemetry implementation beyond keeping the declared schema accurate for the new write kind;
- migrations or compatibility for older WIP format-4 SQLite schemas;
- application-level automatic scope ownership, operation-derived orphan inference, leases, TTLs, or background scope collection.
If implementation requires an excluded item, stop and revise this handoff before expanding scope.
## 12. Grep guards
Before final review, inspect every remaining match:
```text
operationToolMemoPrefix
pendingToolOutputPrefix
deleteList(pendingAssistantFrames
deleteValue(pendingToolOutput
scopeId
retireScope
stopAtTag
```
Expected outcomes:
- no session-scoped transaction directly deletes an ephemeral address;
- no terminal path omits retirement;
- no backend point read/write omits scope identity;
- no fork copies ephemeral state;
- no JSONL sidecar is treated as a Session file;
- no per-scope sequence counter exists;
- no operation-state/orphan inference exists in storage.
## 13. Exit condition
Step 1 is implementation-complete when:
- scope/address/write typing rejects mixed-scope transactions and runtime validation rejects two ephemeral IDs in one transaction;
- IDs are reusable through latest-retirement global sequence boundaries with no historical ID registry;
- Memory, JSONL, and SQLite share one global sequence space and pass common conformance;
- SQLite retirement deletes exact scoped rows atomically with terminal state;
- JSONL writes ephemeral values/lists only to sidecars, records retirement only in the main log, replays all files in global order, preserves high-water marks, and deletes retired sidecars without operation inference;
- list tags and bounded `stopAtTag` work identically across backends;
- current tool/assistant payload and event formats are unchanged;
- every terminal path retires once, while close/fault preserves active scope recovery;
- forks and repository listing/deletion handle sidecars correctly;
- focused tests, `npm run check`, and `./test.sh` pass;
- final review has no blocker;
- the complete diff and results have been presented to the user and implementation has stopped for approval.
Only after explicit approval may Step 1 be committed and pushed. Step 2 starts only after that push is confirmed.

574
packages/agent/docs/mobile-handoff/01-harness/02-scopes/scopes.md Normal file
View File

@@ -0,0 +1,574 @@
# Session Storage: Scopes
> **Status:** Design evidence. The actionable Step 1 contract is [`implementation-handoff.md`](implementation-handoff.md), which supersedes this document on sequence allocation, reusable scope IDs, retirement authority/records, sidecar layout, implementation order, and the separation of JSONL encoding into Step 2.
>
> **Scope:** `packages/agent/src/harness/session/`, plus the call sites in
> `runtime/` listed in §10.
>
> **Depends on [01-delta](../01-delta/delta.md)** for the landed Chord op vocabulary that pending state is stored in (§2, §11) and for the `WireOp` form the record encoding carries (§12). The two
> are otherwise independent and compound: encoding shrinks every write, scopes
> move one class of write out of the main log entirely.
>
> Every claim here was checked against `runtime/lane.ts`, `runtime/progress.ts`
> and `runtime/drive/*.ts`. The type-level enforcement in §6 was verified with
> `tsc --strict`, including that mixed-scope commits are the *only* thing it
> rejects.
## 1. The problem
Operation state — pending tool output, pending assistant output — matters only
while the operation runs. It is crash-recovery scaffolding, not history.
The JSONL log is append-only, so it persists anyway. Worse, deleting it *adds*
records: `deleteValue` and `deleteList` write a line each, and those lines also
persist. Today only a full snapshot rewrite reclaims the space, and no in-place
compaction exists (`jsonl/storage.ts` has `createFromSnapshot`, used for fork).
Measured on 20 operations, each 400 assistant frames plus 2 tools at 60
checkpoints of a 50 KB window:
| | size |
| --- | --- |
| total JSONL | 93.89 MB |
| settled transcript entries — the part that is actually history | **0.06 MB** |
Three orders of magnitude of the file is scaffolding for finished operations.
**Two independent fixes, and they are not alternatives:**
- **Encoding** ([delta.md](../01-delta/delta.md)): value writes carry Chord ops rather than whole values, and addresses are interned. Takes the same workload to **5.32 MB in a single file**, with no change to atomicity. The op vocabulary and flush-time tracker have landed; storage integration has not.
- **Scopes** (this document): pending state leaves the main log entirely, so it is
never history to begin with.
Do the encoding first. Scopes are worth it because 5.32 MB of dead weight per 20
operations still accumulates, and there is no compaction pass to reclaim it.
## 2. Scope on the address
Scope has two halves that must not be confused: a **type-level tag** carrying no
data, and a **runtime scope id** that names the file.
```ts
export type SessionScope = { readonly kind: "session" };
export type EphemeralScope = { readonly kind: "ephemeral" };
export type Scope = SessionScope | EphemeralScope;
// Passing a scopeId is what makes an address ephemeral.
export function value<T>(namespace: string, key: string): Value<T, SessionScope>;
export function value<T>(namespace: string, key: string, scopeId: string): Value<T, EphemeralScope>;
export function list<T>(namespace: string, key: string): ValueList<T, SessionScope>;
export function list<T>(namespace: string, key: string, scopeId: string): ValueList<T, EphemeralScope>;
/** A main-log record retiring an ephemeral scope. See §5. */
export function retireScope(id: string): Write<SessionScope>;
```
The suffix is not decoration: `Session` already names the session interface in
`harness/session/types.ts`, and reusing it produces `TS2440`/`TS2484` plus an
ambiguous re-export from `session/index.ts`.
```ts
interface Value<T, Sc extends Scope = SessionScope> extends ScopedAddress<Sc> {
readonly namespace: string;
readonly key: string;
/** Present iff Sc is Ephemeral. Routes the write and names the sidecar. */
readonly scopeId?: string;
}
```
The id is a plain runtime string, because it is an operation id — generated at
runtime and unavailable to the type system. That is precisely why §6 can
distinguish *session from ephemeral* statically but cannot distinguish *two
different ephemeral scopes*: the tag is in the type, the id is not.
```ts
// Durable lists carry encoded batches; one encoder/decoder pair belongs to each value stream.
export const pendingToolOutput = (operationId: string, invocationId: string) =>
list<WireOp[]>("pi.pending.tool_output", `${operationId}:${invocationId}`, operationId);
export const pendingAssistantOutput = (operationId: string, entryId: string) =>
list<WireOp[]>("pi.pending.assistant_output", `${operationId}:${entryId}`, operationId);
export const operationToolMemo = (operationId: string, invocationId: string, name: string) =>
value<JsonValue>("pi.op.tool_memo", `${operationId}:${invocationId}:${name}`, operationId);
// unchanged — no scopeId, so Session by inference
export const laneStateValue = (lane: string) => value<DurableLaneState>("pi.lane.state", lane);
```
**Scope is the file.** A session-scoped write goes to the main log; an
ephemeral-scoped write goes to `<session>.<scopeId>.jsonl`.
Other backends need no sidecars at all: in-memory drops a `Map` keyed by scope id,
SQLite issues one `DELETE ... WHERE scope = ?`. Scope is meaningful to them only as
a lifetime hint. It is JSONL that needs the file split, and that is where the
constraints in §4 and §5 come from.
## 3. Which addresses are scoped, and why
The boundary is drawn by **write isolation**, not by lifetime.
Scoping everything under `pi.op.*` on the grounds that it dies with the operation
looks right and is wrong. `lane.ts` shows why: **every** operation commit bundles
`operationState` with `laneState`.
```ts
// lane.ts:405/431, 659/660, 749/750, 1071/1072 — the same shape each time
writes: [
...decision.writes,
setValue(operationStateValue(operationId), decision.operationState),
setValue(laneStateValue(this.name), durableLaneState(...)),
]
```
That pair is coordination state: lane state says the operation is at step N,
operation state holds the data for step N. Splitting them across files makes a
crash produce a lane that believes it is somewhere the data does not support.
The **bulk** values behave differently. `openProgress` commits exactly one write
(`runtime/progress.ts:51`), and `pendingToolOutput` / `pendingAssistantFrames`
appear in multi-write transactions only as *deletes*.
| | addresses |
| --- | --- |
| **ephemeral (sidecar)** | `pi.pending.tool_output`, `pi.pending.assistant_output`, `pi.op.tool_memo` |
| **session (main log)** | `pi.lane.state`, `pi.op.state`, `pi.op.meta`, `pi.result`, `pi.pending.entry`, `pi.branch.tip`, `pi.op.tool_args`, `pi.op.preparation` |
`operationToolMemo` is ephemeral so that the memo-plus-checkpoint bundling in
`harness-tools.md` §7.5 is a single-file transaction. `terminal.ts:34` already
treats `operationToolMemoPrefix` and `pendingToolOutputPrefix` as the same cleanup
class, so this matches how the code already thinks about them.
The size win is unaffected by leaving `operationState` behind: effectively all of
the 93.89 MB in §1 is pending tool and assistant output.
## 4. Two files are not atomic
Serialising on the Session line gives **ordering, not atomicity**. Two `write()`
calls to two descriptors, two fsyncs, no journal spanning them. A crash between
leaves the files disagreeing, and JSONL has nothing to repair it with.
So the design does not try to make cross-file transactions work. It ensures none
exist:
> **All writes in a transaction share one scope.**
That is achievable because it is already true of the code — see §5 for the audit.
Two consequences of *not* needing cross-file commits:
- **No write ordering between files is required.** The commit-record pattern
(sidecar first, fsync, then a main-log record acknowledging it) is unnecessary if
no transaction spans files. There is nothing to acknowledge.
- **No commit marker for sidecar-only writes.** One tempting design is a small
main-log record for `openProgress`, since nothing acknowledges its write. It is
not needed: that transaction is one write in one file and already atomic. Torn
tails are handled the same way as in the main log — every line is a complete op
or a whole batch, so a reader stops at the last intact one.
What remains is a **durability** choice, not a consistency one: if the sidecar is
fsynced less aggressively, recent checkpoints can be lost. That is the same bounded
loss the checkpoint interval already accepts, but it should be a deliberate policy
rather than an accident.
## 5. Audit: does the existing code satisfy the rule?
Every commit in `lane.ts` (12 sites) and every `writes:` producer in
`runtime/drive/*.ts` (32 sites) was read.
**Result: yes, with one adjustment.** No transaction writes to two files. The
spanning cases are all *deletes* of ephemeral state bundled with main-log writes:
- `response.ts:340` — the settle commit: `insertEntry`, `insertUsage`,
`setValue(branchTip)`, plus `deleteList(pendingAssistantFrames(...))`.
- `terminal.ts:50` `operationCleanupWrites` — bundles main-log deletes with
`toolMemos` and `toolOutputs` deletes, returned into the same array as
`operationResultValue` and `laneStateValue`.
- `tools.ts:230, 257` — `deleteValue(pendingToolOutput)` with main-log writes.
- `deferred.ts:157` — `deleteList(pendingAssistantFrames)`.
**The adjustment: retirement becomes a main-log record.**
```ts
retireScope(operationId): Write<SessionScope>
```
The log is the truth; the sidecar file is a cache. Recovery reads the retire record
and ignores — then removes — the sidecar. Unlinking eagerly after commit is a pure
optimisation; losing the unlink costs disk, never correctness.
This makes all four sites above single-scope, because the individual ephemeral
deletes disappear. It also *simplifies* `operationCleanupWrites`: the `scanValues`
calls for `operationToolMemoPrefix` and `pendingToolOutputPrefix` are replaced by
one `retireScope`.
And it makes the open-time sweep principled rather than heuristic. A sidecar is
dead iff the main log retired it or its operation is absent — no inference from
operation status.
### 5.1 Two ephemeral scopes in one transaction cannot arise
A lane holds `operation: { meta, state } | null` — **singular**. Every write set
uses one `operationId`, either `drive.operationId` or the newly created one in
`startOperation`, which asserts no operation is active. Retire and start are always
separate transactions: the settle commits at `lane.ts:427` and `response.ts:339`
set lane state to `null`.
Concurrency is across **lanes**, and each lane has at most one active operation, so
two ephemeral scopes never meet in a commit. The runtime assertion in §6 is defence
against a future change, not a live gap.
## 6. Static enforcement
> **`scopes.variance.ts` ships alongside this doc** and is the executable form of
> this section: `npx tsc --noEmit --strict --lib es2023 scopes.variance.ts`.
> Silence means the rule holds. Its three `@ts-expect-error` lines fail the
> compile if they ever start type-checking, so it catches the enforcement being
> weakened as well as broken.
Two phantom types, differing only in where `Sc` appears. This distinction is
load-bearing and is the single easiest thing to get wrong here.
```ts
declare const storedScopeType: unique symbol;
/** Addresses: COVARIANT — Sc only in return position. */
export interface ScopedAddress<Sc extends Scope> {
readonly [storedScopeType]?: () => Sc;
}
/** Writes: INVARIANT — Sc in both parameter and return position. */
export interface Scoped<Sc extends Scope> {
readonly [storedScopeType]?: (scope: Sc) => Sc;
}
export interface Value<T, Sc extends Scope = SessionScope>
extends StoredAddressBase, ScopedAddress<Sc> { … }
export interface ValueSetWrite<Sc extends Scope> extends Scoped<Sc> { … }
```
**Why they differ.** Reading an address is safe at any scope — `getValue` does not
care which file a value lives in, so `Value<T, SessionScope>` must be usable where
`Value<T, Scope>` is expected. Forming a transaction is *not* safe at any scope,
because two files are not atomic, so a commit must pin down exactly one.
Enforcement therefore belongs where transactions are formed, not where addresses
are read. `setValue<T, Sc>(address: Value<T, Sc>, …): Write<Sc>` is the bridge: it
infers the scope from a covariant address and stamps it onto an invariant write.
> **Getting this wrong is expensive.** Making addresses invariant too breaks every
> reader — `getValue`, `scanValues`, `readList` and everything downstream — and
> produced a 36-error cascade across storage backends that had done nothing wrong.
> The tempting workaround, making each reader generic over `Sc` to mean "any
> scope", propagates a type parameter through the whole storage stack to express
> something variance gives for free.
Verified with `tsc --noEmit --strict`. Reads pass at both scopes:
```ts
getValue(laneState); // Value<T, SessionScope>
getValue(pendingOutput); // Value<T, EphemeralScope>
```
Single-scope commits pass:
```ts
commit([setValue(laneState, a), setValue(operationState, b)]);
commit([setValue(toolMemo, m), setValue(pendingOutput, o)]);
commit([setValue(laneState, a), retireScope("op_1")]);
```
Mixed-scope commits fail, and are the *only* things that fail:
```ts
commit([setValue(laneState, a), setValue(pendingOutput, o)]); // ERROR
commit([setValue(pendingOutput, o), retireScope("op_1")]); // ERROR
```
### 6.1 What the type system cannot catch
Two different ephemeral scopes both type as `Write<EphemeralScope>`, because the id
is a runtime string (§2). A runtime assertion at commit covers it — compare
`scopeId` across all writes — and per §5.1 it should never fire.
## 7. Recovery and sweeping
On open, enumerate sidecars alongside the main log and load them. Their records
participate in replay exactly as main-log records do.
A sidecar is removed when the main log holds a `retireScope` record for it, or when
its operation is absent — a crash between commit and unlink. Torn-tail semantics
are unchanged: each line is self-contained, so a crash mid-write loses exactly the
trailing line.
## 8. Sequence numbers
Scoped writes get their own sequence space per scope. Nothing outside a scope
orders against its contents, and keeping them separate means `nextSeq` in the main
header does not account for numbers consumed by files that will be deleted.
Shared numbering would leave large gaps after several operations settled, and the
high-water mark would need separate persistence.
## 9. Measured effect
Same workload as §1:
| | size | of today |
| --- | --- | --- |
| today, single JSONL | 93.89 MB | — |
| ops + interned addresses, single JSONL | 5.32 MB | 5.7% |
| **plus scopes — main log** | **0.06 MB** | **0.06%** |
| — sidecars, retired on settle | 5.26 MB | peak 0.26 MB per operation |
The 5.7% figure is **rate-dependent and must not be quoted alone**: it holds at
2 KB of tool output per checkpoint and inverts above the cap, where a whole-value
replacement wins.
The scope result is **not** rate-dependent. The main log holds settled entries
regardless of encoding, and that is the number that governs session file growth.
## 10. What changes in the existing code
The compiler finds these for you: once addresses carry a scope and `Write<Sc>` is
invariant, every site that mixes scopes in one transaction fails to typecheck.
This list is what a full pass turned up, so you know when you are done.
**New, in `session/values.ts` and `session/types.ts`**
`SessionScope` / `EphemeralScope` / `Scope`; the two phantoms from §6;
`Value<T, Sc>` and `ValueList<T, Sc>`; `scopeId` on the address; the `value` /
`list` overloads; `retireScope`; scope-preserving `setValue` / `deleteValue` /
`appendList` / `deleteList`; `Write<Sc>` with entries, usage and retirement
session-only by construction.
**Generified**
`CommitDecision<TResult, Sc>`, `lane.command<TResult, Sc>`, `Storage.commit<Sc>`.
Give `Sc` a default of `SessionScope`: conditional types are not inference sites,
so without a default a mixed array falls back to the constraint `Scope` and every
call site fails. `settleOperation` and `continueOperation` stay **non**-generic —
see §5.
**Committed writes**
`CommittedScopeRetireWrite`, and `scopeId` carried through the committed shapes so
`JsonlStorage` can route on it.
**Call sites that must change** — each currently mixes scopes in one transaction:
| site | what it does today | what it becomes |
| --- | --- | --- |
| `drive/terminal.ts` `operationCleanupWrites` | enumerates tool memos and tool outputs via `scanValues`, deletes each alongside session deletes | one `retireScope(operationId)`; two of its four scans disappear |
| `drive/response.ts` (settle) | `deleteList(pendingAssistantOutput)` bundled with operation state | drop it — the sidecar is discarded at retire |
| `drive/deferred.ts` (superseded response) | same | same |
| `drive/tools.ts` (tool settle) | `deleteValue(pendingToolOutput)` plus memo deletes in a session write array | drop them |
| `drive/tool-placement.ts` | `deleteValue(pendingToolOutput)` | drop it |
The pattern is the same everywhere: **intra-operation cleanup of ephemeral state
is impossible**, because those commits also write operation and lane state. The
sidecar is discarded wholesale at retire. The cost is that a multi-turn operation
holds superseded pending output until settle — bounded by turns, and never
reaching the main log.
**Storage**
Sidecar routing in `JsonlStorage.commit` (assert one scope id per transaction,
route the append), unlink on a `scope` record **after** the commit, and an
open-time sweep that loads sidecars and removes any the main log already retired.
In-memory drops a `Map` keyed by scope id; SQLite issues one
`DELETE ... WHERE scope = ?` and treats a retire record as a no-op, because it has
no sidecar and the terminal transaction deletes the scoped values explicitly.
**Expected test churn**
Nine tests assert the old cleanup write set and will fail — `drive-terminal` (3),
`drive-retry-deferred` (2), `drive-tools`, `drive-reconcile`, `drive-generation`,
and one pinning reserved namespaces. They expect six or seven individual deletes
where the new code emits fewer plus one `retireScope`. That is the change landing,
not a regression.
## 11. List tags and stop conditions
Tracked state is stored as a list of op batches (`delta.md` §9). Recovery needs "the
batches since the last base batch" without unpacking every row, which lists cannot
express today: `ListReadOptions` has `cursor`, `order` and `limit` only.
`BranchScan` already solves the same problem for entries, and its vocabulary is
the one to copy — `stopAtType` / `stopAtId` for a terminator, plain field names
for a filter.
```ts
appendList<T, Sc>(address: ValueList<T, Sc>, element: T, tag?: string): ListAppendWrite<Sc>;
export interface ListElement<T> {
seq: number;
value: T;
tag?: string;
}
export interface ListReadOptions {
cursor?: ListCursor;
order?: "asc" | "desc";
limit?: number;
/** Include elements up to and including the first carrying this tag, then stop. */
stopAtTag?: string;
/** Return only elements carrying this tag. */
tag?: string;
}
```
**`stopAtTag` is a stop condition within a page, not a guarantee.** It can only end
a page earlier than `limit` would; it never overrides it. If the tagged element is
not in the page, the consumer sees no tagged element and pages again with the
cursor — the same loop `readAssistantFrames` already runs. There is no ordering
question between the two bounds and no "not found" error.
For that loop to work, `readList` must return the tag on each element. Otherwise a
consumer cannot tell "page ended at the tag" from "page ended at the limit"
without parsing the value, which is the thing the tag exists to avoid.
**The producer sets the tag, not storage.** For a tracked value the caller
already knows, because `isBase(ops)` is a token comparison on the first op
(`delta.md` §2):
```ts
const ops = tracker.flush();
if (ops.length === 0) return;
writes: [appendList(address, enc.encode(ops), isBase(ops) ? "base" : undefined)];
```
Storage must never inspect an element to derive a tag. That is what keeps op
batches opaque to the storage layer, and it is the same property that makes
`EntryScan.type` work: the discriminant is a stored column, not something derived
from the payload.
**The tag lives on the storage record**, beside `seq` and `value`:
```jsonc
["l",7,9,[["r",{…}]],"base"] // per §12.1
```
Storage must never parse an element to evaluate a predicate. That is what keeps
ops opaque to storage and the durable path free of domain knowledge
(`harness-tools.md` §7.7). It is also why `EntryScan.type` works: the discriminant
is a stored column, not something derived from the payload.
`order` stays `"asc" | "desc"` rather than `BranchScan`'s
`"newestFirst" | "oldestFirst"`. Branch order is semantic — a walk from a tip
through a tree. List order is over `seq`. Borrowing the branch words would imply a
traversal that is not happening.
Backends: SQLite gets a `tag` column and a real predicate. JSONL holds the list in
memory already, so it scans backwards and checks a field — no worse than today.
In-memory likewise.
The primitive generalises past frames: any list wanting "the tail since the last
checkpoint" gets it.
## 12. JSONL record encoding
Two dictionaries at two layers. They are the same trick and are independent: the
**address** dictionary is over `namespace` + `key` on the record; the **path**
dictionary is over paths *inside* a value's ops (`delta.md` §4).
### 12.1 Records
```
["@", addrId, namespace, key] address definition
["v", addrId, seq, wireOps] value write — WireOp[], delta.md §4
["l", addrId, seq, element, tag?] list append — element is WireOp[]
for a tracked value
["x", addrId, seq] delete (value or list)
["!", addrId, seq] retire an ephemeral scope
```
**Record verbs and op verbs are separate vocabularies read at different levels**,
but they must not look alike. `!` rather than `r` for retire: `r` is delta's
replace op, and a reader scanning a file should not have to remember which nesting
level they are at to know what a tuple means.
Retire takes an address id like everything else, defined by an `["@", id, …]`
whose namespace is the scope. That keeps one interning rule for the whole file
rather than a special case for one record.
Entries and usage rows keep their current keyed form: they are written once, never
repeat an address, and are read by machinery that has nothing to do with ops.
An address is defined on its **second** use, matching path interning — a
definition is pure overhead for an address written once, and a session has many of
those.
### 12.2 Worked example
Four writes across two addresses, before and after.
```jsonc
// today — 1250 bytes
{"kind":"value","op":"set","seq":7,"namespace":"pi.op.state","key":"01a04cf6-…","value":{"at":"starting","control":{…},"settings":{…},"latestAssistantEntryId":null}}
{"kind":"value","op":"set","seq":8,"namespace":"pi.lane.state","key":"main","value":{"currentOperationId":"01a04cf6-…",…}}
{"kind":"value","op":"set","seq":9,"namespace":"pi.op.state","key":"01a04cf6-…","value":{"at":"checkpoint",…}}
{"kind":"value","op":"set","seq":12,"namespace":"pi.op.state","key":"01a04cf6-…","value":{"at":"assistant.ready",…}}
```
```jsonc
// with both dictionaries — 547 bytes
["@",0,"pi.op.state","01a04cf6-…"]
["v",0,7,[["r",{"at":"starting","control":{…},"settings":{…},"latestAssistantEntryId":null}]]]
["@",1,"pi.lane.state","main"]
["v",1,8,[["r",{"currentOperationId":"01a04cf6-…",…}]]]
["v",0,9,[["#",0,["at"]],["s",0,"checkpoint"]]]
["v",0,12,[["s","assistant.ready"]]]
```
| line | before | after |
| --- | --- | --- |
| address definition | – | 61 |
| op.state -> `starting` (base batch) | 353 | 256 |
| address definition | – | 31 |
| lane.state (base batch) | 181 | 114 |
| op.state -> `checkpoint` | 355 | **48** |
| op.state -> `assistant.ready` | 361 | **37** |
| **total** | **1250** | **547** (44%) |
The shape matters more than the total. Base batches barely shrink — 353 to 256 is
envelope only, since the value ships whole either way. The transitions collapse
~8x because they carry one changed field instead of the whole state including the
never-changing `settings` block.
Both dictionaries appear on line 5: `0` in `["v",0,9,…]` is an **address** id,
while `["#",0,["at"]]` defines a **path** id inside the value's ops. Same trick,
different layers, separate tables. Line 6 then drops the path entirely — arity
omission, because it targets the same path as the previous op in that batch.
**Do not quote 44% as the file-level saving.** Dictionary entries are one-time, so
a real operation's ~11 op.state writes amortise them and the ratio improves; but
transcript entries are untouched, and they were 18 KB of a 67 KB tool-heavy run.
Applied there this hits the ~27 KB of value writes and would take the file down by
roughly a fifth, not by half.
### 12.3 Reading
A reader builds both tables while replaying, so a torn tail costs exactly the
trailing lines and no header needs rewriting. A snapshot rewrite starts fresh
tables and re-emits definitions naturally.
A numeric `addrId` for which no definition has been seen is a corrupt file, not a
recoverable state — unlike a missing path id, which cannot occur because path
definitions are inside the same record as their first use.
## 13. Open questions
- File-handle pressure across concurrent lanes, each with a live sidecar. Probably
fine, unmeasured.
- fsync policy for sidecars (§4) — matched to the main log, or relaxed given the
contents are bounded-loss scaffolding.
- Whether a long-running operation should rotate its sidecar. `rebase()`
(`delta.md` §3.2.3) bounds *recovery* length but not file size, so a command
running for hours still grows its sidecar without bound.
- Whether an in-place compaction pass for the main log is worth building anyway.
`createFromSnapshot` plus atomic replace is the mechanism; scopes reduce the need
but do not remove it, since superseded session-scoped values still accumulate.

65
packages/agent/docs/mobile-handoff/01-harness/02-scopes/scopes.variance.ts Normal file
View File

@@ -0,0 +1,65 @@
/**
* Executable form of scopes.md §6.
*
* npx tsc --noEmit --strict --lib es2023 scopes.variance.ts
*
* Silence means the rule holds: reads accept either scope, single-scope commits
* pass, and mixed-scope commits are the ONLY thing rejected. The three
* `@ts-expect-error` lines fail the compile if they ever start type-checking,
* so this catches the enforcement being weakened as well as broken.
*
* The variance split is the part that is easy to get backwards. Making addresses
* invariant too breaks every reader — getValue, scanValues, readList and
* everything downstream — and the tempting fix, making each reader generic over
* Sc, propagates a type parameter through the whole storage stack to express
* what variance gives for free.
*/
declare const tag: unique symbol;
type SessionScope = { readonly kind: "session" };
type EphemeralScope = { readonly kind: "ephemeral" };
type Scope = SessionScope | EphemeralScope;
/** Addresses: COVARIANT — Sc only in return position. Readers take any scope. */
interface ScopedAddress<Sc extends Scope> { readonly [tag]?: () => Sc }
/** Writes: INVARIANT — Sc in both positions. Mixed commits cannot unify. */
interface Scoped<Sc extends Scope> { readonly [tag]?: (s: Sc) => Sc }
interface Value<T, Sc extends Scope = SessionScope> extends ScopedAddress<Sc> { k: "v" }
interface ValueList<T, Sc extends Scope = SessionScope> extends ScopedAddress<Sc> { k: "l" }
interface VSet<Sc extends Scope = Scope> extends Scoped<Sc> { kind: "value"; op: "set" }
interface VDel<Sc extends Scope = Scope> extends Scoped<Sc> { kind: "value"; op: "delete" }
interface LApp<Sc extends Scope = Scope> extends Scoped<Sc> { kind: "list"; op: "append" }
interface EntryWrite { kind: "entry" }
interface RetireWrite extends Scoped<SessionScope> { kind: "scope" }
type Write<Sc extends Scope = SessionScope> =
| VSet<Sc> | VDel<Sc> | LApp<Sc>
| (Sc extends SessionScope ? EntryWrite | RetireWrite : never);
declare function setValue<T, Sc extends Scope>(a: Value<T, Sc>, v: T): VSet<Sc>;
declare function deleteValue<T, Sc extends Scope>(a: Value<T, Sc>): VDel<Sc>;
declare function appendList<T, Sc extends Scope>(a: ValueList<T, Sc>, v: T): LApp<Sc>;
declare function retireScope(id: string): RetireWrite;
declare function insertEntry(e: unknown): EntryWrite;
declare function commit<Sc extends Scope = SessionScope>(w: readonly Write<Sc>[]): void;
declare function getValue<T>(a: Value<T, Scope>): T; // reader: any scope
const laneState = null as unknown as Value<string>; // session
const pendingOut = null as unknown as ValueList<string, EphemeralScope>; // ephemeral
const toolMemo = null as unknown as Value<string, EphemeralScope>;
// READS accept either scope — that is what covariance buys.
getValue(laneState);
getValue(toolMemo);
// Single-scope commits pass.
commit([setValue(laneState, "a")]);
commit([setValue(laneState, "a"), insertEntry({}), retireScope("op_1")]);
commit([appendList(pendingOut, "x"), setValue(toolMemo, "m")]);
// @ts-expect-error mixed scopes in one transaction
commit([setValue(laneState, "a"), appendList(pendingOut, "x")]);
// @ts-expect-error a retire is session-only, so it cannot join an ephemeral commit
commit([appendList(pendingOut, "x"), retireScope("op_1")]);
// @ts-expect-error entries are session-only
commit<EphemeralScope>([insertEntry({})]);

172
packages/agent/docs/mobile-handoff/01-harness/03-execenv/bash.after.ts Normal file
View File

@@ -0,0 +1,172 @@
import { type Static, Type } from "typebox";
import type { Context } from "../context.ts";
import type { AgentHarnessTool } from "../types.ts";
import { getOrThrow } from "../types.ts";
import { DEFAULT_MAX_BYTES, DEFAULT_MAX_LINES, formatSize, type TruncationResult } from "../utils/truncate.ts";
import type { ExecutionToolContext } from "./tool-context.ts";
const MAX_TIMEOUT_SECONDS = 2_147_483_647 / 1000;
/**
* Coalescing cadence for live updates. The env owns capping and coalescing now
* (`executionenv.md` §3), so bash no longer has an update throttle or a
* checkpoint interval of its own.
*/
const BASH_UPDATE_INTERVAL_MS = 100;
const bashSchema = Type.Object({
command: Type.String({ description: "Bash command to execute" }),
timeout: Type.Optional(Type.Number({ description: "Timeout in seconds (optional, no default timeout)" })),
});
export type BashToolInput = Static<typeof bashSchema>;
export interface BashToolDetails {
truncation?: TruncationResult;
/** Set by the exec env, on the machine the command ran on. */
spillPath?: string;
}
export interface BashExecution {
command: string;
cwd: string;
env: Record<string, string>;
inheritEnv: boolean;
}
export type BashPrepare<TContext extends ExecutionToolContext = ExecutionToolContext> = (
execution: BashExecution,
toolContext: TContext,
context: Context,
) => void | Promise<void>;
export interface BashToolOptions<TContext extends ExecutionToolContext = ExecutionToolContext> {
commandPrefix?: string;
prepare?: BashPrepare<TContext>;
}
function validateTimeout(timeout: number | undefined): void {
if (timeout === undefined) return;
if (!Number.isFinite(timeout) || timeout <= 0) {
throw new Error("Invalid timeout: must be a finite number of seconds");
}
if (timeout > MAX_TIMEOUT_SECONDS) {
throw new Error(`Invalid timeout: maximum is ${MAX_TIMEOUT_SECONDS} seconds`);
}
}
export function createBashTool<TContext extends ExecutionToolContext = ExecutionToolContext>(
options?: BashToolOptions<TContext>,
): AgentHarnessTool<TContext, typeof bashSchema, BashToolDetails | undefined> {
return {
name: "bash",
label: "bash",
description: `Execute a bash command in the current working directory. Returns stdout and stderr. Output is truncated to last ${DEFAULT_MAX_LINES} lines or ${DEFAULT_MAX_BYTES / 1024}KB (whichever is hit first). If truncated, full output is saved to a temp file. Optionally provide a timeout in seconds.`,
parameters: bashSchema,
async execute(_toolCallId, { command, timeout }, onUpdate, toolContext, _invocation, context) {
validateTimeout(timeout);
const { env } = toolContext;
const execution: BashExecution = {
command: options?.commandPrefix ? `${options.commandPrefix}\n${command}` : command,
cwd: env.cwd,
env: {},
inheritEnv: true,
};
await options?.prepare?.(execution, toolContext, context);
// The env caps, coalesces and spills at the source. bash keeps a mirror
// only to build the settled result; it never truncates and never chooses
// a spill path.
let view = "";
let truncation: TruncationResult | undefined;
let spillPath: string | undefined;
const publish = (checkpoint: boolean): void => {
onUpdate(
{
content: [{ type: "text" as const, text: view }],
details: { truncation, spillPath },
},
checkpoint ? { checkpoint: true } : undefined,
);
};
onUpdate({ content: [], details: undefined });
const result = await env.exec(
execution.command,
{
cwd: execution.cwd,
env: execution.env,
inheritEnv: execution.inheritEnv,
timeout,
capture: {
limits: { maxBytes: DEFAULT_MAX_BYTES, maxLines: DEFAULT_MAX_LINES, retain: "tail" },
intervalMs: BASH_UPDATE_INTERVAL_MS,
spill: true,
},
onUpdate: (update) => {
switch (update.kind) {
case "chunk":
view += update.text;
break;
case "snapshot":
view = update.output.text;
truncation = update.output.truncation;
spillPath = update.output.spillPath;
break;
case "counters":
truncation = update.truncation;
break;
}
// A snapshot is the durable-worthy state: it is the only kind that
// can express eviction, so it is where a crash loses least.
publish(update.kind === "snapshot");
},
},
context,
);
if (!result.ok) {
const error = result.error;
const status =
error.code === "timeout"
? `Command timed out after ${timeout} seconds`
: error.code === "aborted"
? "Command aborted"
: error.message;
throw new Error(view ? `${view}\n\n${status}` : status, { cause: error });
}
const captured = result.value.output;
view = captured.text;
truncation = captured.truncation;
spillPath = captured.spillPath;
let outputText = captured.text;
let details: BashToolDetails | undefined;
if (captured.truncation.truncated) {
details = { truncation: captured.truncation, spillPath };
const startLine = captured.truncation.totalLines - captured.truncation.outputLines + 1;
const endLine = captured.truncation.totalLines;
if (captured.truncation.lastLinePartial) {
// One line alone exceeded the byte cap. Line numbers would be
// misleading here, so report the line's own size instead — and only
// the env can measure it, since it holds the raw buffer.
const lastLineSize = formatSize(captured.lastLineBytes ?? captured.truncation.outputBytes);
outputText += `\n\n[Showing last ${formatSize(captured.truncation.outputBytes)} of line ${endLine} (line is ${lastLineSize}). Full output: ${spillPath}]`;
} else if (captured.truncation.truncatedBy === "lines") {
outputText += `\n\n[Showing lines ${startLine}-${endLine} of ${captured.truncation.totalLines}. Full output: ${spillPath}]`;
} else {
outputText += `\n\n[Showing lines ${startLine}-${endLine} of ${captured.truncation.totalLines} (${formatSize(DEFAULT_MAX_BYTES)} limit). Full output: ${spillPath}]`;
}
}
if (result.value.exitCode !== 0) {
throw new Error(
`${outputText ? `${outputText}\n\n` : ""}Command exited with code ${result.value.exitCode}`,
);
}
return { content: [{ type: "text", text: outputText || "(no output)" }], details };
},
};
}

148
packages/agent/docs/mobile-handoff/01-harness/03-execenv/execenv.md Normal file
View File

@@ -0,0 +1,148 @@
# ExecutionEnv: bounded shell output
**Status:** implemented in production source. The prototype files beside this document are historical evidence; production lives in:
- `packages/agent/src/harness/utils/adaptive-publisher.ts`
- `packages/agent/src/harness/utils/output-capture.ts`
- `packages/agent/src/harness/env/nodejs.ts`
- `packages/agent/src/harness/tools/bash.ts`
The generic `ToolOutput` use of the same publisher remains in `04-tool-output`.
## 1. Problem
The old `Shell.exec()` accumulated complete strings inside `NodeExecutionEnv`:
```ts
stdout += chunk;
stderr += chunk;
```
Bash truncated only after those strings had already been built. `cat 1gb.txt` therefore materialized a gigabyte in the worker before any tool-level bound could help.
Spilling also belongs where bytes originate. If execution is remote, a spill on the worker is inaccessible to the model's `read` and `grep` tools, and creating it after transport would first send the complete gigabyte over the connection.
## 2. Boundary
`ExecutionEnv` now owns:
- a bounded head or tail view;
- complete byte and line totals;
- lazy source-local spill after the view first crosses its limits;
- persistent source-local spill with bounded write-stream backpressure;
- adaptive publication of the latest bounded state;
- a forced final publication before settlement.
It does not return or retain separate `stdout` and `stderr` values. Both pipes feed one arrival-ordered model-visible text view, matching bash and `ToolOutput`; preserving stream styling through tail eviction would require a segmented retained state that the Harness does not expose. Text is folded from updates, and `ShellExecResult` contains only exit and truncation/spill metadata.
Bash owns only command semantics and its model-visible footer. Its old rolling buffer, spill creation, 100 ms throttle, and full-output accumulation are gone. The existing two-second durable checkpoint request remains temporarily until `ToolOutput` owns durable cadence.
## 3. Contract
```ts
interface ShellOutputLimits {
maxBytes: number;
maxLines: number;
retain?: "head" | "tail";
}
interface ShellOutputCaptureOptions {
limits: ShellOutputLimits;
spill?: boolean;
}
type ShellOutputTruncation = Omit<TruncationResult, "content">;
interface ShellOutputMetadata {
truncation: ShellOutputTruncation;
spillPath?: string;
lastLineBytes?: number;
}
interface ShellOutputView extends ShellOutputMetadata {
text: string;
}
type ShellOutputUpdate =
| { kind: "replace"; output: ShellOutputView }
| { kind: "append"; text: string; metadata: ShellOutputMetadata }
| { kind: "slide"; drop: number; text: string; metadata: ShellOutputMetadata }
| { kind: "metadata"; metadata: ShellOutputMetadata };
interface ShellExecResult extends ShellOutputMetadata {
exitCode: number;
}
```
`drop` counts JavaScript string code units, matching `slice()`. Updates are ordered. A consumer applies them with `applyShellOutputUpdate()`.
A complete replacement establishes the initial state or recovers when no verified overlap exists. An append carries only a growing suffix. A slide drops a prefix and appends the new suffix. Metadata updates move totals or the spill path without resending text.
The compatibility `executeShellWithCapture()` helper still returns one bounded final view. Its `onChunk` callback receives initial, append, and slide text only; metadata and post-turnover replacements are not mislabeled as new bytes.
## 4. Adaptive publication
`AdaptivePublisher` keeps only the latest dirty state. Intermediate process writes never become a queue of output updates.
Policy is harness-global rather than per tool:
```ts
minIntervalMs = 100;
targetBytesPerSecond = 100 * 1024;
nextDelayMs = max(minIntervalMs, encodedUpdateBytes * 1000 / targetBytesPerSecond);
```
The first dirty state after idle is immediate. Writes received before the deadline collapse into the latest bounded state. One trailing timer guarantees eventual publication. Finalization bypasses the deadline once, still bounded by the retained cap.
The publisher commits its baseline before invoking the consumer. If a consumer applies an update and then throws, finalization cannot emit the same delta twice. The command fails with `callback_error`.
### Workload behavior
| workload | result |
| --- | --- |
| Finishes below cap | immediate initial state, small appends, forced final state |
| Below-cap trickle | isolated writes are immediate; sustained writes are at most 100 ms apart |
| Full-force output after cap | complete turnovers are cap-sized replacements spaced by their encoded size |
| Post-cap trickle | small verified `slide` updates remain responsive; the complete window is not resent |
| Burst then silence | one leading update and one trailing update |
| Error, timeout, or abort | latest bounded state is forced before the error settles |
At a 50 KB cap and 100 KB/s target, repeated complete turnovers settle near two updates per second. Small post-cap slides still use the 100 ms floor.
The rate bound is amortized. An immediate-after-idle update and forced terminal update may each create one cap-bounded burst.
## 5. Capture and spill
`OutputCapture` trims its decoded line-aware working buffer back to twice the byte cap when it exceeds four times the cap. Tail mode drops old text; head mode preserves the original prefix. This amortizes UTF-8 trimming instead of rescanning the retained window for every process chunk.
Spill creation is lazy. Before crossing, the source keeps at most the bounded prefix needed to create a complete archive. On crossing it:
1. pauses stdout and stderr while it creates the file inside the execution environment;
2. opens one persistent append stream with a bounded 8 MB high-water mark;
3. writes the preserved raw prefix and subsequent raw chunks in arrival order;
4. resumes immediately while the writer accepts data;
5. pauses again only when `write()` reports backpressure, then resumes on `drain`.
This avoids both an unbounded promise chain and one async file-open/append cycle per process chunk. A spill create or stream-write failure kills the child and fails execution rather than publishing lossy success.
The spill path is force-published as metadata when it becomes available. Spill writes are awaited before the final output flush.
Node streams remain raw for spill throughput and exact archival bytes. `OutputCapture` uses one streaming `TextDecoder`, so a read boundary cannot split a code point; invalid display control characters are removed only from bounded snapshots, not by scanning the complete raw stream. Line totals count a final unterminated line, and `lastLineBytes` remains exact even when one line exceeds the working buffer.
## 6. Remote execution
When worker and execution environment are colocated, updates are in-process and this publisher primarily bounds capture work. `ToolOutput` remains the downstream event/durability limiter.
When an execution environment is physically remote from its worker, the same bounded updates cross that transport. Full-force output cannot transfer the complete stream: intermediate writes collapse at the source, while the complete stream remains in the source-local spill.
Each real costly boundary gets its own publisher instance. A colocated deployment may bypass transport serialization; a custom tool that bypasses `ExecutionEnv` still passes through the future `ToolOutput` publisher.
## 7. Remaining work
- Move generic custom-tool composition, text retention, event publication, and durable cadence into `ToolOutput`.
- Replace whole `AgentToolResult` progress with Chord operations at that downstream boundary.
- Make memo and output checkpoint persistence one atomic transaction.
- Seed replay from durable output rather than deleting it.
- Put spills in an environment-owned session directory and sweep them after session lifetime plus a crash-retention floor.
- Decide explicit limits for images and structured details; text is bounded, those values are not yet.
- Define raw binary-output behavior. Current shell output remains lossy UTF-8 text.

137
packages/agent/docs/mobile-handoff/01-harness/03-execenv/nodejs-env.diff Normal file
View File

@@ -0,0 +1,137 @@
diff --git a/packages/agent/src/harness/env/nodejs.ts b/packages/agent/src/harness/env/nodejs.ts
index 732dc84db..db9eb9a57 100644
--- a/packages/agent/src/harness/env/nodejs.ts
+++ b/packages/agent/src/harness/env/nodejs.ts
@@ -19,6 +19,7 @@ import { basename, isAbsolute, join, resolve } from "node:path";
import { createInterface } from "node:readline";
import { fileURLToPath } from "node:url";
import type { Context } from "../context.ts";
+import { OutputCapture } from "../utils/output-capture.ts";
import {
type ExecutionEnv,
ExecutionError,
@@ -28,7 +29,10 @@ import {
type FileKind,
ok,
type Result,
+ type ShellCapabilities,
type ShellExecOptions,
+ type ShellExecResult,
+ type ShellStream,
toError,
} from "../types.ts";
@@ -365,11 +369,15 @@ export class NodeExecutionEnv implements ExecutionEnv {
return ok(join(...parts));
}
+ capabilities(): ShellCapabilities {
+ return { capture: true, spill: true, liveUpdates: true };
+ }
+
async exec(
command: string,
options: ShellExecOptions | undefined,
context: Context,
- ): Promise<Result<{ stdout: string; stderr: string; exitCode: number }, ExecutionError>> {
+ ): Promise<Result<ShellExecResult, ExecutionError>> {
const signal = context.abortSignal;
if (signal?.aborted) return err(new ExecutionError("aborted", "aborted"));
const timeoutMsResult = resolveTimeoutMs(options?.timeout);
@@ -392,9 +400,28 @@ export class NodeExecutionEnv implements ExecutionEnv {
);
}
+ // Spill lives in the exec env, on the machine where the model's own `read`
+ // and `grep` run — the whole reason capping happens here rather than on the
+ // agent side (`executionenv.md` §1).
+ let spillHandle: string | undefined;
+ let spillChain: Promise<void> = Promise.resolve();
+ const wantSpill = options?.capture?.spill === true;
+
return await new Promise((resolvePromise) => {
- let stdout = "";
- let stderr = "";
+ const capture = new OutputCapture(options?.capture, options?.onUpdate);
+ const spillChunk = (chunk: string): Promise<void> => {
+ if (!wantSpill) return Promise.resolve();
+ spillChain = spillChain.then(async () => {
+ if (spillHandle === undefined) {
+ const created = await this.createTempFile({ prefix: "pi-output-", suffix: ".log" }, context);
+ if (!created.ok) return;
+ spillHandle = created.value;
+ capture.setSpillPath(spillHandle);
+ }
+ await this.appendFile(spillHandle, chunk, context);
+ });
+ return spillChain;
+ };
let settled = false;
let timedOut = false;
let callbackError: ExecutionError | undefined;
@@ -407,7 +434,7 @@ export class NodeExecutionEnv implements ExecutionEnv {
}
};
- const settle = (result: Result<{ stdout: string; stderr: string; exitCode: number }, ExecutionError>) => {
+ const settle = (result: Result<ShellExecResult, ExecutionError>) => {
if (timeoutId) clearTimeout(timeoutId);
if (signal) signal.removeEventListener("abort", onAbort);
if (child?.pid) this.activeChildPids.delete(child.pid);
@@ -458,31 +485,32 @@ export class NodeExecutionEnv implements ExecutionEnv {
}
}
+ // setEncoding("utf8") is what keeps a chunk boundary from splitting a
+ // codepoint; without it a multi-byte sequence straddling a read bakes in a
+ // replacement character (`executionenv.md` §8).
child.stdout?.setEncoding("utf8");
child.stderr?.setEncoding("utf8");
- child.stdout?.on("data", (chunk: string) => {
- stdout += chunk;
- try {
- options?.onStdout?.(chunk, context);
- } catch (error) {
- const cause = toError(error);
- callbackError = new ExecutionError("callback_error", cause.message, cause);
- onAbort();
- }
- });
- child.stderr?.on("data", (chunk: string) => {
- stderr += chunk;
+ const feed = (stream: ShellStream, chunk: string) => {
try {
- options?.onStderr?.(chunk, context);
+ capture.push(stream, chunk, context);
+ void spillChunk(chunk);
} catch (error) {
const cause = toError(error);
callbackError = new ExecutionError("callback_error", cause.message, cause);
onAbort();
}
- });
+ };
+ child.stdout?.on("data", (chunk: string) => feed("stdout", chunk));
+ child.stderr?.on("data", (chunk: string) => feed("stderr", chunk));
void waitForChildProcess(child).then(
- (code) => {
+ async (code) => {
+ // The spill chain is async and lazy: the file is created on the first
+ // chunk. Await it before snapshotting, or the returned CapturedOutput
+ // reports no spillPath for output that was in fact archived.
+ await spillChain;
+ capture.flush(context);
+ capture.dispose();
if (callbackError) {
settle(err(callbackError));
return;
@@ -495,7 +523,7 @@ export class NodeExecutionEnv implements ExecutionEnv {
settle(err(new ExecutionError("aborted", "aborted")));
return;
}
- settle(ok({ stdout, stderr, exitCode: code ?? 0 }));
+ settle(ok({ output: capture.snapshot(), exitCode: code ?? 0 }));
},
(error: Error) => settle(err(new ExecutionError("spawn_error", error.message, error))),
);

149
packages/agent/docs/mobile-handoff/01-harness/03-execenv/nodejs-env.test.diff Normal file
View File

@@ -0,0 +1,149 @@
diff --git a/packages/agent/test/harness/nodejs-env.test.ts b/packages/agent/test/harness/nodejs-env.test.ts
index 361040495..ee28ef506 100644
--- a/packages/agent/test/harness/nodejs-env.test.ts
+++ b/packages/agent/test/harness/nodejs-env.test.ts
@@ -8,7 +8,6 @@ import { afterEach, describe, expect, it } from "vitest";
import { BACKGROUND_CONTEXT, withAbortSignal } from "../../src/harness/context.ts";
import { NodeExecutionEnv } from "../../src/harness/env/nodejs.ts";
import { FileError, getOrThrow } from "../../src/harness/types.ts";
-import { executeShellWithCapture } from "../../src/harness/utils/shell-output.ts";
import { createTempDir } from "./session-test-utils.ts";
const chmodRestorePaths: string[] = [];
@@ -303,7 +302,8 @@ describe("NodeExecutionEnv", () => {
BACKGROUND_CONTEXT,
),
);
- expect(result).toEqual({ stdout: `${await realpath(root)}:ok`, stderr: "", exitCode: 0 });
+ expect(result.output.text).toContain(`${await realpath(root)}:ok`.trim());
+ expect(result.exitCode).toBe(0);
});
it.each([
@@ -334,7 +334,7 @@ describe("NodeExecutionEnv", () => {
),
);
- expect(result.stdout).toBe(`${expectedSessionFile}|true|preserved`);
+ expect(result.output.text).toBe(`${expectedSessionFile}|true|preserved`);
},
);
@@ -358,7 +358,7 @@ describe("NodeExecutionEnv", () => {
),
);
- expect(result.stdout).toBe("::explicit");
+ expect(result.output.text).toBe("::explicit");
} finally {
if (previousInherited === undefined) delete process.env[inheritedKey];
else process.env[inheritedKey] = previousInherited;
@@ -396,7 +396,9 @@ describe("NodeExecutionEnv", () => {
await wslEnv.exec(`name='World'; echo "Hello, ${nameExpansion}!"`, undefined, BACKGROUND_CONTEXT),
);
- expect(result).toEqual({ stdout: "Hello, World!\n", stderr: "args:-s\n", exitCode: 0 });
+ expect(result.output.text).toContain("Hello, World!");
+ expect(result.output.text).toContain("args:-s");
+ expect(result.exitCode).toBe(0);
} finally {
process.chdir(originalCwd);
process.env.PATH = originalPath;
@@ -425,7 +427,7 @@ describe("NodeExecutionEnv", () => {
() => controller.abort(),
),
);
- expect(result.stdout).toContain("child-exiting");
+ expect(result.output.text).toContain("child-exiting");
} finally {
controller.abort();
cleanupDetachedChild(pidFile);
@@ -448,25 +450,30 @@ describe("NodeExecutionEnv", () => {
it("streams stdout and stderr chunks", async () => {
const root = createTempDir();
const env = new NodeExecutionEnv({ cwd: root });
- let stdout = "";
- let stderr = "";
+ // Streams are aggregated in arrival order now; `chunk` carries `stream` so a
+ // renderer can still colour stderr (`executionenv.md` §6).
+ let seen = "";
+ const streams: string[] = [];
const result = getOrThrow(
await env.exec(
"printf out; printf err >&2",
{
- onStdout: (chunk) => {
- stdout += chunk;
- },
- onStderr: (chunk) => {
- stderr += chunk;
+ onUpdate: (update) => {
+ if (update.kind !== "chunk") return;
+ seen += update.text;
+ streams.push(update.stream);
},
},
BACKGROUND_CONTEXT,
),
);
- expect(result).toEqual({ stdout: "out", stderr: "err", exitCode: 0 });
- expect(stdout).toBe("out");
- expect(stderr).toBe("err");
+ expect(result.exitCode).toBe(0);
+ expect(result.output.text).toContain("out");
+ expect(result.output.text).toContain("err");
+ expect(seen).toContain("out");
+ expect(seen).toContain("err");
+ expect(streams).toContain("stdout");
+ expect(streams).toContain("stderr");
});
it("reports a missing working directory before spawning", async () => {
@@ -484,7 +491,8 @@ describe("NodeExecutionEnv", () => {
const root = createTempDir();
const env = new NodeExecutionEnv({ cwd: root });
const result = getOrThrow(await env.exec("exit 7", undefined, BACKGROUND_CONTEXT));
- expect(result).toEqual({ stdout: "", stderr: "", exitCode: 7 });
+ expect(result.output.text).toContain("".trim());
+ expect(result.exitCode).toBe(7);
});
it("returns timeout errors for commands exceeding the timeout", async () => {
@@ -501,7 +509,7 @@ describe("NodeExecutionEnv", () => {
const result = await env.exec(
"printf out",
{
- onStdout: () => {
+ onUpdate: () => {
throw new Error("callback failed");
},
},
@@ -542,12 +550,23 @@ describe("NodeExecutionEnv", () => {
const root = createTempDir();
const env = new NodeExecutionEnv({ cwd: root });
const result = getOrThrow(
- await executeShellWithCapture(env, "yes line | head -n 15000", undefined, BACKGROUND_CONTEXT),
+ await env.exec(
+ "yes line | head -n 15000",
+ {
+ capture: {
+ limits: { maxBytes: 50 * 1024, maxLines: 2000, retain: "tail" },
+ spill: true,
+ },
+ },
+ BACKGROUND_CONTEXT,
+ ),
);
- expect(result.truncated).toBe(true);
- expect(result.fullOutputPath).toBeDefined();
- const fullOutput = getOrThrow(await env.readTextFile(result.fullOutputPath!, BACKGROUND_CONTEXT));
+ expect(result.output.truncation.truncated).toBe(true);
+ // The spill lives inside the exec env, on the machine the command ran on —
+ // which is where the model's own read and grep run (`executionenv.md` §1).
+ expect(result.output.spillPath).toBeDefined();
+ const fullOutput = getOrThrow(await env.readTextFile(result.output.spillPath!, BACKGROUND_CONTEXT));
expect(fullOutput.split("\n").length).toBeGreaterThan(10000);
- expect(result.output.length).toBeLessThan(fullOutput.length);
+ expect(result.output.text.length).toBeLessThan(fullOutput.length);
});
});

194
packages/agent/docs/mobile-handoff/01-harness/03-execenv/output-capture.ts Normal file
View File

@@ -0,0 +1,194 @@
import type { Context } from "../context.ts";
import type { CaptureOptions, CapturedOutput, ShellOutputUpdate, ShellStream } from "../types.ts";
import {
DEFAULT_MAX_BYTES,
DEFAULT_MAX_LINES,
type TruncationResult,
truncateHead,
truncateTail,
utf8ByteLength,
} from "./truncate.ts";
/**
* Bounded capture for a Shell implementation.
*
* Owns the two-stage structure that `shell-output.ts` had inside `bash.ts`: a raw
* buffer trimmed to `maxBytes * 2` as a memory guard, and a retained view
* computed from it per snapshot. `truncateTail` is line-aware — it walks
* backwards line by line and takes a partial only when one line alone exceeds the
* cap — so eviction is not byte arithmetic and cannot be done incrementally.
*
* Emits the three update kinds from `executionenv.md` §4 and enforces the
* ordering invariant: once the cap is crossed, a `chunk` never follows without an
* intervening `snapshot`.
*/
export class OutputCapture {
readonly #maxBytes: number;
readonly #maxLines: number;
readonly #retain: "head" | "tail";
readonly #intervalMs: number;
readonly #onUpdate?: (update: ShellOutputUpdate, context: Context) => void;
/** Raw rolling buffer, held at twice the cap. */
#buffer = "";
/** Everything ever produced. */
#totalBytes = 0;
/** Newlines seen. A trailing partial line is counted separately — a command
* emitting one 60 KB line with no newline has produced one line, not zero. */
#newlines = 0;
#endsWithNewline = true;
/** True once the retained view has begun evicting (tail) or filled (head). */
#crossed = false;
/** Pending chunks for the current coalescing window. */
#pending: { stream: ShellStream; text: string }[] = [];
#lastFlush = 0;
#timer: ReturnType<typeof setTimeout> | undefined;
#spillPath: string | undefined;
constructor(
options: CaptureOptions | undefined,
onUpdate: ((update: ShellOutputUpdate, context: Context) => void) | undefined,
) {
this.#maxBytes = options?.limits.maxBytes ?? DEFAULT_MAX_BYTES;
this.#maxLines = options?.limits.maxLines ?? DEFAULT_MAX_LINES;
this.#retain = options?.limits.retain ?? "tail";
this.#intervalMs = options?.intervalMs ?? 0;
this.#onUpdate = onUpdate;
}
push(stream: ShellStream, text: string, context: Context): void {
if (text === "") return;
this.#totalBytes += utf8ByteLength(text);
this.#newlines += countNewlines(text);
this.#endsWithNewline = text.endsWith("\n");
this.#buffer += text;
// Memory guard. The retained view is computed from this, never equal to it.
const guard = this.#maxBytes * 2;
if (utf8ByteLength(this.#buffer) > guard) {
this.#buffer = trimToLastBytes(this.#buffer, guard);
}
this.#pending.push({ stream, text });
this.#schedule(context);
}
setSpillPath(path: string | undefined): void {
this.#spillPath = path;
}
/** The bounded view, recomputed from the raw buffer. */
snapshot(): CapturedOutput {
const truncation =
this.#retain === "head"
? truncateHead(this.#buffer, { maxBytes: this.#maxBytes, maxLines: this.#maxLines })
: truncateTail(this.#buffer, { maxBytes: this.#maxBytes, maxLines: this.#maxLines });
// The buffer already lost its front, so totals come from counters rather
// than from what survives.
const totalLines = this.#totalLines();
const truncated = this.#totalBytes > this.#maxBytes || totalLines > this.#maxLines;
return {
text: truncation.content,
truncation: {
...truncation,
lastLinePartial: truncation.lastLinePartial,
truncated,
truncatedBy: truncated ? (totalLines > this.#maxLines ? "lines" : "bytes") : null,
totalBytes: this.#totalBytes,
totalLines,
},
...(this.#spillPath === undefined ? {} : { spillPath: this.#spillPath }),
...(truncation.lastLinePartial ? { lastLineBytes: this.#lastLineBytes() } : {}),
};
}
#totalLines(): number {
return this.#newlines + (this.#endsWithNewline || this.#totalBytes === 0 ? 0 : 1);
}
/** Bytes in the final line of the raw buffer, for the single-oversized-line case. */
#lastLineBytes(): number {
const start = this.#buffer.lastIndexOf("\n");
return utf8ByteLength(start === -1 ? this.#buffer : this.#buffer.slice(start + 1));
}
#schedule(context: Context): void {
if (this.#onUpdate === undefined) return;
if (this.#intervalMs === 0) {
this.flush(context);
return;
}
if (this.#timer !== undefined) return;
const wait = Math.max(0, this.#lastFlush + this.#intervalMs - Date.now());
this.#timer = setTimeout(() => {
this.#timer = undefined;
this.flush(context);
}, wait);
}
/** Emit one update for everything buffered since the last flush. */
flush(context: Context): void {
if (this.#onUpdate === undefined) return;
this.#lastFlush = Date.now();
const pending = this.#pending;
this.#pending = [];
if (pending.length === 0) return;
const nowCrossed = this.#totalBytes > this.#maxBytes || this.#totalLines() > this.#maxLines;
if (this.#retain === "head") {
// Head never evicts. Below the cap the appended text is complete; above
// it, only totals move — resending maxBytes to change a number is the
// thing `counters` exists to avoid.
if (!nowCrossed) {
this.#onUpdate({ kind: "chunk", stream: pending[0]!.stream, text: joined(pending) }, context);
} else {
this.#onUpdate({ kind: "counters", truncation: this.snapshot().truncation }, context);
}
this.#crossed = nowCrossed;
return;
}
// Tail. Once the buffer is full every byte in evicts a byte out, so there is
// no cheap-trickle regime past the cap: chunks until full, snapshots after.
if (nowCrossed) {
this.#crossed = true;
this.#onUpdate({ kind: "snapshot", output: this.snapshot() }, context);
return;
}
if (this.#crossed) {
// Defensive: never let a chunk follow the crossing.
this.#onUpdate({ kind: "snapshot", output: this.snapshot() }, context);
return;
}
this.#onUpdate({ kind: "chunk", stream: pending[0]!.stream, text: joined(pending) }, context);
}
dispose(): void {
if (this.#timer !== undefined) clearTimeout(this.#timer);
this.#timer = undefined;
}
}
const joined = (pending: { text: string }[]): string => pending.map((p) => p.text).join("");
function countNewlines(text: string): number {
let count = 0;
for (let index = text.indexOf("\n"); index !== -1; index = text.indexOf("\n", index + 1)) count++;
return count;
}
/** Keep the last `bytes` UTF-8 bytes without splitting a codepoint. */
function trimToLastBytes(text: string, bytes: number): string {
if (utf8ByteLength(text) <= bytes) return text;
let low = 0;
let high = text.length;
while (low < high) {
const mid = Math.floor((low + high) / 2);
if (utf8ByteLength(text.slice(mid)) > bytes) low = mid + 1;
else high = mid;
}
return text.slice(low);
}
export type { TruncationResult };

211
packages/agent/docs/mobile-handoff/01-harness/04-tool-output/bash-worked-example.md Normal file
View File

@@ -0,0 +1,211 @@
# Worked Example: `bash` End to End
One tool through every layer. `execute` is simplified — timeout validation,
cancellation and exit-code branches are elided. Everything on the output path is
shown.
Layers: exec env → `ToolOutput` sink → harness events → durable storage → lane
state → facet → wire → consumer.
---
## 1. The tool
```ts
// packages/agent/src/harness/tools/bash.ts
export interface BashToolDetails { spillPath?: string; truncation?: ShellOutputTruncation }
export function createBashTool(): AgentHarnessTool<ExecutionToolContext, typeof bashSchema, BashToolDetails> {
return {
name: "bash",
parameters: bashSchema,
output: { retain: "tail", maxBytes: DEFAULT_MAX_BYTES, maxLines: DEFAULT_MAX_LINES },
async execute(_id, { command, timeout }, signal, out, context) {
const env = context.env;
let view: ShellOutputView | undefined;
const result = getOrThrow(await env.exec(command, {
cwd: env.cwd,
inheritEnv: true,
timeout,
capture: { limits: this.output, spill: true },
onUpdate: (u) => {
view = applyShellOutputUpdate(view, u);
if (u.kind === "append") out.write(u.text);
else out.replace(view.text);
out.details.truncation = view.truncation;
if (view.spillPath) out.details.spillPath = view.spillPath;
},
}, context));
if (result.spillPath) out.details.spillPath = result.spillPath;
if (result.truncation.truncated) out.write(`\n\n[${describe(result.truncation)}]`);
if (result.exitCode) throw new Error(`Command exited with code ${result.exitCode}`);
},
};
}
```
What is gone from today's implementation: the rolling `tailOutput` buffer,
`truncateTail`, `ensureFullOutputFile`, `createTempFile`,
`BASH_UPDATE_THROTTLE_MS`, `BASH_CHECKPOINT_INTERVAL_MS`, `updateDirty`,
`lastCheckpoint`, `scheduleOutputUpdate`, `emitOutputUpdate`,
`clearUpdateTimer`. About 40 lines of tool-local machinery, replaced by a capture
policy every tool gets.
`execute` returns `void`. `details` is one field, set once — and note the tool no
longer computes `truncation`, because the env owns the window and the tool no
longer knows what was dropped.
## 2. The exec env caps at the source
`env.exec` applies `ShellOutputLimits` where the bytes originate. For a sandbox host
that means `cat 1gb.txt` never ships 1 GB to the agent machine, and the spill
lands where the model's own `read` and `grep` run. See [`execenv.md`](../03-execenv/execenv.md).
The initial state is a bounded `replace`. Growth emits `append`; a moving tail emits `slide { drop, text }`; complete turnover falls back to a bounded `replace`; metadata moves totals and spill paths without resending text. The adaptive publisher keeps small slides responsive and spaces cap-sized turnovers according to their encoded size.
## 3. The sink
`ToolOutput` folds those into `ToolOutputState`:
```ts
{
content: [{ type: "text", text: "…the retained window…" }],
details: { spillPath: "/tmp/pi-session-x/bash-8f2.log" },
usage: undefined,
addedTools: undefined,
terminate: false,
truncation: { truncated: true, truncatedBy: "bytes", totalLines: 8123, totalBytes: 262144 },
}
```
`out.write(text)` appends to `content[0].text`; `out.replace(text)` assigns
the whole retained view when the env sends a `snapshot`, because an append cannot
express eviction. The tracker (`delta.md`) records intent either way: an append is
an `a`, and a window slide assigned wholesale becomes `t` + `a` via verified
overlap detection.
## 4. Ops
A 256 KB build over ~20 flushes, 50 KB window:
The first batch is a base batch — it begins with `r`, carrying the initial state. Everything
after is deltas. The repeated content path is interned on its second use, and
consecutive ops on it omit the id entirely:
```jsonc
[["r",{"content":[{"type":"text","text":""}],"details":{},"terminate":false,"truncation":{…}}]]
[["a",["content",0,"text"],"make: Entering directory …\n"]]
[["#",0,["content",0,"text"]],["a",0,"cc -c src/a.c …\n"]]
[["a",0,"cc -c src/b.c …\n"]]
…
[["s",["details","spillPath"],"/tmp/pi-session-x/bash-8f2.log"]]
…
[["t",0,4096],["a","cc -c src/z.c\n"],["s",["truncation","totalBytes"],262144]]
```
Details are one `s`, in one flush out of twenty. The `t` + `a` pair is the window slide. The current tracker recovers it by verified overlap; production remeasurement shows the generic path is already negligible, so no explicit append/truncate producer API is added ([decision](../01-delta/append-decision.md)).
## 5. Harness events
```ts
{ type: "tool_start", toolCallId: "call_7", toolName: "bash",
args: { command: "make -j8" } }
{ type: "tool_update", toolCallId: "call_7", ops: [ … ] }
{ type: "tool_end", toolCallId: "call_7", isError: false }
```
`tool_start` is identity only — no `caps`, no `initial`. The first batch is a base
batch, so the initial state arrives on the update channel, and the caps are
already inside it as `truncation.maxBytes` / `maxLines`.
`tool_end` carries no content and no details. Every byte already went out;
re-sending would duplicate any images.
## 6. Durable storage
`pendingToolOutput(operationId, "call_7")`, **ephemeral-scoped** to the operation,
so it lives in `<session>.op_….jsonl` and is retired on settle rather than
persisting in the main log forever. Retirement is a main-log `retireScope` record,
so it commits atomically with the settle writes; the unlink is a consequence of
replaying that record, not part of the transaction ([scopes.md §5](../02-scopes/scopes.md)).
A `list<WireOp[]>`, one encoded batch appended per flush, base batches tagged `"base"` so recovery reads backwards with `stopAtTag` and stops there ([scopes.md §11](../02-scopes/scopes.md#11-list-tags-and-stop-conditions)). The tracker emits structural ops; the producer calls `rebase()` periodically to write a capped root replacement and bound recovery replay. The durable interval is a sink policy. Shell capture controls only source-state and transport publication; memo, terminal, and recovery-base flushes are forced by `ToolOutput`.
Recovery seeds a fresh `ToolOutput` from that state — it does **not** delete it,
which is what `clearReplayCheckpoint` does today and is a bug
(`harness-tools.md` §7.4). If bash memoised "spilled to /tmp/…" and crashed, a
replay that discarded the state would create a second spill file and lose the
first.
The memo write and this checkpoint commit in the same transaction. Both are
ephemeral-scoped, so both land in the same sidecar — and the type system rejects a
commit that mixes scopes, so this cannot regress silently.
## 7. Lane state and the facet
```ts
export function reduceLaneSnapshot(view: LaneView, event: HarnessEvent): void {
switch (event.type) {
case "tool_start":
view.operation.tools.push({ id: event.toolCallId, name: event.toolName,
args: event.args, output: undefined });
return;
case "tool_update": {
const tool = view.operation.tools.find((t) => t.id === event.toolCallId);
if (tool === undefined) return; // host will send a base batch
tool.output = apply(tool.output, event.ops);
return;
}
case "tool_end": {
const i = view.operation.tools.findIndex(t => t.id === event.toolCallId);
if (i >= 0) view.operation.tools.splice(i, 1);
return;
}
}
}
```
Plain mutation, no Immer, no return value. An event for a tool the view has never
seen simply does nothing; the host sends a `replace`.
A lane facet runs this same function under the tracker, so the facet's own ops
fall out against **its** shape — which need not be `LaneView` and generally is
not. The facet never writes an op.
## 8. Wire and consumer
```jsonc
{ "seq": 0, "ops": [["r",{"transcript":[],"operation":null}]] }
{ "seq": 1, "ops": [ … ] }
```
One shape: a batch of ops. The first is a base batch — it begins with `r` —
and everything after is deltas. A gap, a reconnect, a provider
reload, or a fold that could not apply all take one path: send a fresh `replace`.
The consumer is `apply` — six verbs, no domain knowledge, no library, no tool
code, against a plain mutable object it owns.
## 9. What this run costs
256 KB of output, ~20 flushes, 50 KB window:
| | today | this design |
|---|---|---|
| durable writes | full `AgentToolResult` per checkpoint | structural ops plus explicit periodic capped base batches |
| durable location | main log, forever | sidecar, unlinked on settle |
| wire per flush | whole snapshot | one `truncate` + one `append` |
| details written | rebuilt whole, every flush | one `set`, once |
| truncation logic | in `bash.ts` | in the exec env, shared by all tools |
| spill location | `/tmp`, OS-cleaned | exec env, session-scoped |
The details row is the one worth dwelling on. Nothing about bash's details
changed — they were always small. What changed is that they stopped riding inside
a container replaced wholesale on every update, which is why no tool anywhere
currently bothers to mutate them incrementally.

462
packages/agent/docs/mobile-handoff/01-harness/04-tool-output/harness-tools.md Normal file
View File

@@ -0,0 +1,462 @@
# Tool Output and Progress
> **Scope:** harness-local. Nothing here depends on the facet system, RPC, or any
> presentation. Depends on [delta tracking](../01-delta/delta.md) (the landed Chord op vocabulary and tracker), [execution environments](../03-execenv/execenv.md) (where truncation happens), and [scoped storage](../02-scopes/scopes.md) (durability).
## 1. The problem
Three faults, one cause.
**Tools each implement their own truncation.** `bash.ts` owns a rolling buffer,
`truncateTail`, a spill file, an update throttle and a checkpoint interval.
`read.ts` has its own truncation. Every future tool producing bulk output
reimplements this, differently.
**Progress is a whole value.** `onUpdate(partialResult)` hands over a complete
`AgentToolResult` on every update, `tool_update` carries that whole result, and
`openToolProgress` persists it with `setValue`.
**`details: unknown` was believed to force this** — you cannot append to a value
whose shape you do not know. That premise is now false. A structural tracker
(`delta.md`) records ops against JSON without knowing the type, so details need
no special handling at all.
## 2. `ToolOutput`
The harness constructs a sink per invocation and passes it to `execute`. **Tools
return nothing**; the sink holds the result.
```ts
interface ToolOutput<TDetails extends JsonValue> {
/** Append to the text block. */
write(text: string): void;
/** Append an image block. Images are never windowed. */
image(image: ImageContent): void;
/** Replace the retained text wholesale. Chord still recovers a verified `t` + `a` slide when possible. */
replace(text: string): void;
/** Apply source-owned truncation totals and spill metadata without resending text. */
capture(metadata: ShellOutputMetadata): void;
/** The tool's own details object. Mutate it directly. */
readonly details: TDetails;
/** Accumulates. A subagent making several model calls adds to it. */
usage(usage: Usage): void;
/** Replaces. */
addTools(names: string[]): void;
/** Replaces — a tool that decides to terminate and then recovers can say so. */
terminate(value: boolean): void;
}
```
```ts
execute(
toolCallId: string,
params: Static<TParameters>,
signal: AbortSignal,
out: ToolOutput<TDetails>,
context: Context,
): Promise<void>;
```
`execute` returns `void`. Everything the tool produces flows through the sink —
including `usage` and `addTools`, which cannot be settle-time return values
because a replayed tool must be able to seed from durable state (§7.4).
**Failure is a thrown error.** `isError` is set by the harness when `execute`
rejects. A tool may throw anything, including errors from libraries it did not
write.
**`terminate` is orthogonal to how execution ended**, so a failing tool can ask
the loop to stop:
```ts
try { await thing(); } catch (error) { out.terminate(true); throw error; }
```
This closes a gap in the current implementation, where `executeToolCall`'s catch
hardcodes `isError: true` with no terminate, and `immediateError`'s terminate
parameter is only ever passed `true` by `applyBeforeToolDecision` for a hook
block.
### 2.1 Details are just an object
```ts
async execute(id, params, signal, out, context) {
out.details.total = 42;
out.details.passed += 1;
out.details.failures.push({ name, message });
out.details.current = undefined; // -> delete
}
```
Fully granular ops fall out, verified against the prototype:
```jsonc
["#",0,["details","passed"]]
["s",0,1]
["p",["details","failures"],0,0,[{…}]]
["d",["details","current"]]
```
No recipes, no mutation map, no `initialDetails` on `tool_start`, no Immer, and
no consumer ever runs tool code. `TDetails` remains the tool's own exported type
so a renderer can cast `call.details` to it — that is all it is for.
A survey of `packages/agent/src/harness/tools/` found **no tool mutating details
incrementally today**; only `bash.ts` writes them mid-stream and it rebuilds them
whole, because the container was replaced wholesale anyway. This design removes
that cause, and costs nothing if nobody takes it up.
**Caveat:** details are now unbounded. A tool pushing to `failures` in a loop
grows without limit, and unlike text there is no cap. Nothing shipped does this;
the door is open in a way it was not when details were a replace-only value.
### 2.2 Partial output survives failure
Today `executeToolCall` catches and returns `createErrorToolResult(message)`,
building a fresh result from the error string alone — a tool that streamed 8 KB
and then threw reports only the error.
With the sink, what was written is already the result. The harness appends the
error text as content and keeps the rest. The error text is model-visible
content, not presentation: the model needs to read why the call failed.
`abortedMessage` and `interruptedMessage` should be made consistent with this;
today they build replacement results via `syntheticMessage`, so a cancelled
long-running command loses its partial output too.
## 3. Content
Content is exactly **one text block followed by zero or more image blocks**. A
tool cannot interleave text between images — text written before and after an
image lands in the same block. Deliberate: truncation only ever touches a string,
and an image is never partially anything.
Images are **never windowed**. A byte or line cap over base64 payloads is
meaningless and `truncateTail` operates on text. `maxBytes` / `maxLines` govern
text only.
### 3.1 Retention mode
Declared on the tool definition:
```ts
output?: {
retain?: "head" | "tail"; // default "tail"
maxBytes?: number;
maxLines?: number;
}
```
**`head`** — append until the cap, then stop. Nothing is ever removed. Right for
output meaningful from the start: file reads, listings, greps.
**`tail`** — a rolling window. Right for anything whose interesting part is at
the end: builds, test runs.
Head+tail is **not** offered. `truncate.ts` exports `truncateHead` and
`truncateTail`; neither combines them and none is being added.
### 3.2 The tool does not truncate — the exec env does
For output originating in the execution environment, capping, coalescing and
spilling happen **at the source**. See [`execenv.md`](../03-execenv/execenv.md). The tool passes its
`ShellOutputLimits` into `env.exec` and pipes the resulting updates into the sink.
This is not a convenience. Reading a 1 GB file on a sandbox host must not ship
1 GB to the agent machine to be capped there, and the spill file must land where
the model's own `read` and `grep` run.
For output originating on the agent machine (subagents, in-process work),
`ToolOutput` applies the same logic locally. Same code, different location.
Note this reverses an earlier decision that there is no spill in the sink and a
temp-file path is tool-specific. The exec-env argument — model reachability
across a machine boundary — is what changed it.
## 4. Tool output state
`tool_update` carries `Op[]` (`delta.md` §6) targeting the invocation's
`ToolOutputState`:
```ts
interface ToolOutputState {
content: (TextContent | ImageContent)[];
details: JsonValue;
usage?: Usage;
addedTools?: string[];
terminate: boolean;
truncation: ShellOutputTruncation; // totals over everything ever written, without duplicate text
}
```
Ops rather than a typed variant union, for one decisive reason: **only ops give
details granularity without the harness knowing `TDetails`**. A typed union would
need per-tool recipes, which is the machinery §2.1 deletes.
Text still gets delta treatment, because the sink applies the cap *before*
mutating, so a rolling window produces `truncate` + `append` on
`content[0].text` rather than a whole-value set.
Interned ops also measure **smaller than a typed frame vocabulary** on every
workload, and 10x smaller on details, because a frame repeats `toolCallId` where
an interned op carries one integer. See `delta.md` §4.1.
There is no `drop` event. The sink knows what it evicted and expresses it as
`truncate`; a consumer needs no separate signal and derives nothing.
## 5. Harness events
```ts
| { type: "tool_start";
runId; turnId; toolCallId; toolName;
args: unknown }
| { type: "tool_update";
runId; turnId; toolCallId;
ops: Op[] } // a base batch begins with `r`
| { type: "tool_end";
runId; turnId; toolCallId;
isError: boolean }
```
**`tool_start` carries identity only.** An earlier draft added `caps` and
`initial`; both were redundant and are gone. The first batch is always a base batch
(`delta.md` §6), so the initial state arrives on the update channel — carrying it
twice means two ways to establish the base, which can disagree. And `caps` are
already inside the state: `ToolOutputState.truncation` carries `maxBytes` and
`maxLines`, which is what a renderer needs to say "50 KB limit". The draft itself
conceded a consumer "no longer has to" fold identically, since the producer's ops
encode eviction — that was the last reason for `caps` to travel, and it does not
hold.
`tool_update` carries ops. A base batch travels the same channel as a delta,
because a replacement is itself an op (`delta.md` §2) — there is no second shape. `message_update` has the identical shape
(`message-update.md` §5.1); a consumer folds tool output and assistant output with
one code path.
**`tool_end` carries no content, no details, no usage, no terminate.** Every byte
already went out. Re-sending would duplicate each base64 image at the moment
nothing new has happened.
> **The fold is the result.** Nothing a consumer folded is ever re-sent to
> confirm it.
This dissolves an earlier open question about settle-time truncation disagreeing
with the running fold. There is no separate settle-time truncation: the sink's
window *is* the truncation. Anything a tool wants to add at the end — bash's
`[Showing lines 8000-8123 of 8123]` footer — is `out.write(footer)`, one more
append.
The harness still assembles `AgentToolResult` in process for the model, and
`createToolResultMessage` still writes the settled `ToolResultMessage` to the
transcript with `content`, `details`, `usage`, `addedToolNames`, `isError`.
Neither is a wire event.
## 6. Lane reduction
```ts
export function reduceLaneSnapshot(view: LaneView, event: HarnessEvent): void;
```
Plain mutation on a plain object. No `Draft`, no `produce`, no Immer, and **no
`Rebase` return value** — a fold that cannot apply an event leaves state alone,
and the host sends a `replace`. (An earlier draft had `void | Rebase`; under
Immer that throws, and without Immer there is nothing to return to. Note
`reducer.ts:4` currently defines `LaneSnapshotReduction = LaneSnapshot | { rebase: true }`,
so this is a real change to existing code, and `navigation_end` is the event that
drives it.)
The event is the only input besides the view. No registry, no `resolve`, no tool
code — so a tool rewritten between crash and resume cannot make a persisted
stream unreadable.
When the harness is wrapped by a facet, the same mutation runs under the tracker
from `delta.md` and ops fall out. The harness itself does not know this.
## 7. Durability
### 7.1 What exists today
`pendingToolOutput(operationId, invocationId)` is a
`value<AgentToolResult<unknown>>`, and — importantly — `progress.write(partial)`
fires **only** when `options?.checkpoint === true` (`drive/tools.ts:325`). Not on
every update. bash checkpoints every 2 s with a `JSON.stringify` dedupe; every
other tool never checkpoints at all.
It is also **not a progress buffer**. It is the interruption checkpoint. On
resume, if the tool is not replay-safe, `readCheckpoint` turns it into a real
`ToolResultMessage` for the model: `[...checkpoint.content, INTERRUPTION_MARKER]`
plus `details` and `usage`.
Note the implication that misled an earlier draft: because the checkpoint must
*be* the current state, it was read as needing `value` semantics. It does not. It
needs to be *derivable*, and folding encoded batches from the last base batch derives it — which is the point of tagging base batches (§7.3). `checkpoint: true`
requests a durable write, not a replacement.
`pendingAssistantFrames` is a `list<AssistantMessageFrame>` appended per frame,
`deleteList` on settle in `response.ts:345`, `deferred.ts:157`, `terminal.ts:47`.
Note `operationCleanupWrites` (`terminal.ts:26`) does four `scanValues` calls to
enumerate what to delete at settle. Under scopes, the two covering
`operationToolMemoPrefix` and `pendingToolOutputPrefix` are replaced by a single
`retireScope(operationId)`.
### 7.2 Renaming
`pendingAssistantFrames` → **`pendingAssistantOutput`**, matching
`pendingToolOutput`. Frames stop being the durable unit; both addresses now hold
tracked state written as ops or snapshots.
### 7.3 What to write, and when
Both addresses are **ephemeral-scoped** ([scopes.md](../02-scopes/scopes.md)), so they live in a
sidecar retired on settle rather than in the main log forever.
Two independent wins, in order of importance:
- **Encoding.** Ops instead of whole values, plus address interning: 93.89 MB to
5.32 MB in a single file, no change to atomicity. Do this first.
- **Scopes.** Pending state leaves the main log entirely: 5.32 MB to 0.06 MB
surviving.
**Both are `list<WireOp[]>`**, not values. The sink owns one stateful Chord encoder/decoder pair per durable value stream. One encoded batch is appended per flush, and a logical batch whose first op is `r` is tagged `"base"` on the storage record.
Recovery reads backwards with `stopAtTag: "base"` and applies forward
([delta.md §9](../01-delta/delta.md#9-durable-form), [scopes.md §11](../02-scopes/scopes.md#11-list-tags-and-stop-conditions)).
Writing one flush:
```ts
const ops = out.flush();
if (ops.length === 0) return;
const wire = enc.encode(ops);
writes: [appendList(address, wire, isBase(ops) ? "base" : undefined)];
```
`isBase` comes from Chord. It checks for the root replacement op `r`; ordinary nested sets use `s`. Classification lives beside the vocabulary so the comparison is written once.
The landed tracker emits structural ops unconditionally. There is no serialized-size comparison or adaptive replacement heuristic. A producer requests a base batch explicitly with `rebase()`; the output sink's cap bounds that replacement, while periodic rebasing bounds recovery work. Production remeasurement rejects a text-specific append/truncate API: the generic path measured 2.43–2.46 µs per 50 KB rolling-window flush locally, below surrounding costs. Keep ordinary tracked string mutation; see the [decision record](../01-delta/append-decision.md).
**Checkpointing is not deleted.** `BASH_CHECKPOINT_INTERVAL_MS` remains only as an interim compatibility mechanism. The generic sink owns durable frequency because Shell cannot price a storage write or enforce memo/output atomicity. Forced memo, terminal, and recovery-base writes bypass ordinary pacing while remaining cap-bounded.
### 7.4 Replay must seed, not discard
`clearReplayCheckpoint` currently writes `deleteValue(pendingToolOutput(...))`
before re-executing a replay-safe tool (`drive/tools.ts:257`). **This is a bug.**
Replay-safe means the tool is re-executed, but memos exist precisely so it does
*not* redo work it already did — and skipped work emits nothing. Any output for
memoised work is lost today.
The fix: seed a fresh `ToolOutput` from the durable state, then re-execute. The
tool appends to a sink that already holds what it produced before the crash.
This is also why nothing may be settle-only. `usage` and `addTools` must survive
a seed, so they flow through the sink like everything else.
### 7.5 The memo invariant
> A tool's memo write and its output checkpoint must commit in the same
> transaction.
Otherwise a tool does work, sets a memo, crashes before the next checkpoint, and
on replay skips the work while the seeded output has no record of it.
**This does not hold today.** `setMemo` (`drive/tools.ts:112`) and `openProgress`
(`runtime/progress.ts:44`) are two separate `lane.command` calls, hence two
transactions. Making it hold means bundling the checkpoint into the memo's commit:
```ts
setMemo(name, value) {
validateMemoName(name);
if (!active) return Promise.reject(ended());
return lane.command<void>((state) => {
if (!ownsEffect(state)) return { kind: "reject", error: ended() };
const memo = operationToolMemo(drive.operationId, call.resultEntryId, name);
return {
kind: "commit",
writes: [
value === undefined ? deleteValue(memo) : setValue(memo, value),
setValue(pendingToolOutput(drive.operationId, call.resultEntryId), out.snapshot()),
],
next: state,
materialize: () => undefined,
};
}, drive.context);
}
```
Both addresses are **ephemeral-scoped**, so this is a single-file transaction and
statically enforced as one ([scopes.md §3 and §6](../02-scopes/scopes.md)). `operationToolMemo` is
placed in that scope precisely for this. Ordering on the Session line would not be
enough — two file writes are not atomic. The periodic checkpoint stays as it
is — best-effort, for the interruption path; this forces one where correctness
requires it.
The invariant holds because the tool does X, writes X's output to the sink, *then*
calls `setMemo("did X")` — so the sink's state at commit time already contains X's
output.
**Ordering alone does not work**, in case it looks tempting:
- memo first, checkpoint second → crash between → replay skips X and the seeded
output lacks it → silent loss;
- checkpoint first, memo second → crash between → replay redoes X and appends
again → duplicated output.
Duplication is the less bad failure, so ordered writes are a tolerable fallback,
but neither is correct.
**Cost:** `setMemo` now writes a full capped output state rather than a small
value. Memos are rare — a handful per invocation — so this is bounded by
`memo count x cap`, not by output volume.
### 7.6 `openProgress` has a write-ordering race
`commitWrite(item)` captures `item` when `write()` is called, and the write is
fire-and-forget with only `latest` tracked. A checkpoint captured at T1 can
therefore commit *after* a memo-bundled checkpoint at T2, overwriting newer state
with older — reintroducing exactly the loss §7.5 prevents.
Fix: resolve the sink's state inside the command planner rather than at call time.
```ts
commitWrite: () => setValue(address, out.snapshot()) // evaluated under the Session line
```
`lane.command` serializes on the Session line, so checkpoint writes become
monotonic by construction. This removes the race generally, not only against
memos.
### 7.7 Nothing on the durable path runs tool code
Ops are interpreted by a six-verb applier with no domain knowledge, so a tool
rewritten between crash and resume cannot make a persisted stream unreadable.
> **The durable path uses only harness-owned reducers.**
This also rules out persisting *facet* ops. The harness has no facet state,
facets come and go, and a recovering harness must rebuild its working values with
no facet present.
## 8. Open questions
- **Property tests for the tracker** (`delta.md` §3.3). Everything here rests on
producer and replica agreeing; nothing currently proves they do.
- Coalescing window: per tick, or a byte/time threshold.
- Whether image count needs a bound. Images are unwindowed, so a tool pushing
them in a loop grows `content` without limit. Currently treated as a tool bug.
- Whether details need a bound for the same reason (§2.1).
- Whether `retain: "head"` should keep emitting counter-only updates once capped
so a renderer can report how much was suppressed. [`execenv.md`](../03-execenv/execenv.md) answers this
for exec-originated output; agent-side output needs the same answer.
- Whether a failing tool *should* be able to terminate, or whether the current
inability is deliberate — there is a reasonable argument the model should
receive the error and decide.
- **Derived values.** `arguments` parsed from accumulated JSON should not be
replicated; derive it on demand. Safe because `parseStreamingJson` is total —
four fallbacks ending in `{}`, it cannot throw — so a replica derives with no
error path and no agreement protocol. Generalises to: no derived fields in
replicated state.

125
packages/agent/docs/mobile-handoff/01-harness/04-tool-output/rate-limiting.md Normal file
View File

@@ -0,0 +1,125 @@
# Bounded output publication
**Status:** the shared adaptive publisher and its `ExecutionEnv` use are implemented. Generic `ToolOutput` integration remains design work.
Depends on landed Chord delta tracking, source-bounded execution output, and scoped storage for durable batches.
## 1. Invariant
> The durable record, what the model sees, and what the UI shows are the same bounded view.
A spill file is not a second view. It is a file inside the execution environment that the model reaches through ordinary file tools.
Every uncontrolled producer boundary needs two independent bounds:
- **state size:** the latest retained text is capped;
- **publication:** both encoded bytes and event count are paced.
Delta encoding is complementary. It compresses a published change; it does not cap state or decide when publication occurs.
## 2. Boundaries
```text
remote process
-> optional ExecutionEnv publisher
-> worker ToolOutput publisher
-> events + durability + replication
```
A publisher instance is required only before a real costly boundary:
- a physically remote execution environment limits output before transport;
- `ToolOutput` limits custom tools and downstream event/storage traffic;
- a colocated environment can feed its bounded updates in process without another serialized transport;
- custom tools bypass `ExecutionEnv` but cannot bypass `ToolOutput`.
The control algorithm is shared. Payload vocabularies differ: Shell uses replace/append/slide/metadata; `ToolOutput` uses Chord operations.
## 3. Why size or cadence alone is insufficient
A fixed 50 KB snapshot is safe per event but not over time. At 100 ms it permits ten complete snapshots per second, approximately 500 KB/s plus envelopes.
A byte budget alone also permits excessive tiny events and durable transactions. A minimum interval bounds count; encoded-size debt bounds bandwidth.
The original handoff incorrectly treated `intervalMs = 100` as 100 emits/s. It is ten emits/s. The fixed interval was still non-adaptive: it delayed small trickles while allowing complete windows at the same frequency.
## 4. Landed adaptive algorithm
`packages/agent/src/harness/utils/adaptive-publisher.ts` implements:
```ts
nextDelayMs = max(globalMinEmitInterval, encodedUpdateBytes * 1000 / globalTargetBytesPerSecond);
```
Current harness-global policy:
```ts
minEmitInterval = 100 ms;
targetBytesPerSecond = 100 KB/s;
```
Behavior:
1. The first dirty state after idle publishes immediately.
2. Writes before the next deadline collapse into the latest state.
3. One trailing timer publishes held state after the deadline.
4. Completion and correctness boundaries force one bounded publication.
5. The publisher commits its baseline before consumer delivery, preventing duplicate deltas if a consumer applies and then throws.
This is an amortized token bucket with a cap-sized burst. A leading or forced terminal update may exceed the target over a short interval, but sustained encoded bytes converge to the target and sustained event count cannot exceed the minimum-interval floor except for explicit forced correctness writes.
## 5. Scenario traces
Assume a 50 KB cap, 100 KB/s target, and 100 ms floor.
### Below cap, completes
The initial state publishes immediately. Small appends publish no faster than the floor, and final dirty state is forced. Total encoded text is approximately the produced text.
### Below cap, trickling
Writes arriving more than 100 ms apart publish immediately because the preceding deadline has already passed. Faster writes collapse into one append per floor interval.
### Above cap, full force
The retained state never exceeds 50 KB. Complete window turnovers encode as bounded replacements. A roughly 50 KB update buys about 500 ms of silence, yielding approximately two updates and 100 KB per second regardless of raw producer throughput.
For shell execution, the complete stream goes to a source-local spill under backpressure rather than over the output-update channel.
### Above cap, trickling
A small tail movement encodes as truncate plus append (Chord) or `slide` (Shell). Its encoded size is small, so the 100 ms floor dominates and output remains responsive. Resending a complete 50 KB snapshot for each tiny slide would be both slower and larger.
### Burst then silence
The leading state is immediate. Held writes collapse, and one trailing timer publishes the latest residue. There is no polling timer.
### Huge single write
The producer may already have allocated its input, but the boundary retains and publishes only the configured cap. Shell spill keeps the complete source stream. Arbitrary custom-tool images and structured details still require separate limits.
## 6. Forced writes
These bypass pacing once, while remaining state-size bounded:
- command/tool completion;
- error or abort;
- a memo and output checkpoint committed atomically;
- an explicit recovery base/rebase.
A terminal flush cancels the trailing timer before `tool_end`, preventing a late update for a settled invocation.
## 7. `ToolOutput` application
The sink will own one Chord tracker and encoder per invocation. Mutations remain local while publication is blocked; flush-time dirty tracking means held writes collapse without retaining one op per write.
One sink flush feeds the live event and current model-visible state. Durable batches use the same logical flush, encoded as per-stream `WireOp[]`. Periodic `rebase()` bounds recovery replay; a memo correctness flush writes a tagged base batch in the same transaction as the memo.
Durable cadence belongs to the sink, not to Shell or bash. Bash's existing two-second checkpoint request remains only as an interim compatibility mechanism until the sink migration.
## 8. Remaining decisions
- Explicit byte/count rejection policy for images.
- Bounds for structured details; arbitrary JSON cannot be meaningfully tail-windowed.
- Whether ordinary durable writes ride every live sink publication initially or use a slower measured cadence. Memo and terminal correctness flushes are not optional.
- Exact global production values after profiling real remote transport and storage.

227
packages/agent/docs/mobile-handoff/01-harness/05-assistant-output/message-update.md Normal file
View File

@@ -0,0 +1,227 @@
# `message_update` Write Amplification
> **Scope:** harness-local. Depends on [delta tracking](../01-delta/delta.md) for the landed Chord `Op`/`WireOp` vocabulary and [scoped storage](../02-scopes/scopes.md) for durability. Chord delta tracking has landed; scoped storage and this Harness integration have not.
## 1. The problem
```ts
{ type: "message_update", runId, message, event, frame? }
```
Three representations of the same thing travel together:
- `message` — the full `AssistantMessage`;
- `event` — an `AssistantMessageEvent`, which itself carries `partial: AssistantMessage`,
a second full copy;
- `frame` — the actual delta, optional.
Per streamed token this is roughly two complete snapshots plus a delta, so bytes grow
quadratically over a response.
The reducer does not even use the delta:
```ts
case "message_update":
if (next.operation?.id === event.runId && event.message.role === "assistant") {
next.operation.streamingMessage = event.message;
}
```
A straight assignment. So on any wire, sending the event is *worse* than sending a
snapshot — it ships two where `replace` ships one.
The wire adapter is already halfway to the fix: it drops `event` and sends `message`
plus `frame`. That is the full snapshot together with the delta that produced it.
## 2. The compact form already exists
```ts
/**
* Compact, replayable assistant-message progress. Terminal settlement is
* intentionally excluded and must be persisted separately.
*/
export type AssistantMessageFrame =
| { type: "start"; partial: AssistantMessage }
| { type: "text_start" | "text_delta" | "text_end"; contentIndex: number; ... }
| { type: "thinking_start" | "thinking_delta" | "thinking_end"; ... }
| { type: "toolcall_start" | "toolcall_checkpoint" | "toolcall_delta" | "toolcall_end"; ... };
```
`AssistantMessageFrameEncoder` produces it, `reduceAssistantMessageFrames` folds it,
and `openFrameProgress` already persists it with `appendList`. The delta format is
built, used, and durable. It simply is not what `message_update` carries.
## 3. Precedent
pi already does wire-is-delta one layer down. `PiMessagesEvent` — the serialized form
a pi-messages backend sends — has no `partial`:
```ts
| { type: "text_delta"; contentIndex: number; delta: string }
| { type: "text_end"; contentIndex: number; content: string; contentSignature?: string }
```
`pi-messages.ts` then rehydrates: it holds a local `partial`, mutates it per event
(`partial.content[i].text += event.delta`), and returns the in-process
`AssistantMessageEvent` with `partial` attached.
So the convention is established. It just stops at the provider edge instead of
continuing through the harness.
## 4. The change
```ts
| { type: "message_update"; runId: string; entryId: string; frame: AssistantMessageFrame }
```
`message` and `event` are removed; `frame` stops being optional.
The blast radius is small because `HarnessEvent` is not what most streaming consumers
read. `AgentEvent` (`agent-loop.ts`) and `AgentSessionEvent` (`agent-session.ts`) are
separate unions that happen to share the tag name and build their own
`message_update` from the pi-ai event directly. They are out of scope here.
Real consumers and producers of `HarnessEvent.message_update`:
| site | change |
| --- | --- |
| `runtime/drive/response.ts` | emit the required semantic frame; stop attaching full snapshots |
| `runtime/reducer.ts` | fold the frame instead of assigning `event.message` |
| lane/facet state adapter | run that fold under the Chord tracker and emit `Op[]`/encoded `WireOp[]` |
| `experimental/harness-wire-adapter.ts` | stop treating raw `HarnessEvent` as the final replication format |
| `harness/telemetry.ts` | name list only |
| `protocol/harness.ts` | replication carries encoded `WireOp[]`, not the raw event |
`message_end` continues to carry the settled message, because frames deliberately
exclude terminal settlement. That is once per message, not once per token.
## 5. An incremental applier is needed
`reduceAssistantMessageFrames` is a whole-stream fold over an `Iterable`. The
reducer needs a step function:
```ts
export function applyAssistantMessageFrame(
state: AssistantFrameState,
frame: AssistantMessageFrame,
): void;
```
Plain mutation on a plain object. **No `Draft`, no Immer.** An earlier draft
argued for a draft-mutating signature so it would compose inside a `produce`
recipe; that motivation is gone ([delta.md §8](../01-delta/delta.md#8-what-this-removes-from-the-codebase)). The step function is still
needed — the whole-stream version becomes a loop over it — just for the simpler
reason that the reducer folds one frame at a time.
### 5.1 pi-ai frames stay at the pi-ai boundary; Chord ops cross replication boundaries
`AssistantMessageFrame` is pi-ai's semantic delta vocabulary (`text_delta`, `text_end`, …). [Delta tracking §6](../01-delta/delta.md#6-there-is-no-frame-type) defines no second frame wrapper: in-process replication carries `Op[]`, and a wire adapter carries encoded `WireOp[]`. Pi-ai frames stop at the fold; Chord ops cross the replication boundary.
`AssistantMessageFrame` is pi-ai's own delta vocabulary and stays. What changes is
that it is no longer the durable unit or the replication unit.
The harness folds frames into `LaneView` by plain mutation. Under the Chord tracker that yields:
```json
["a",["operation","streamingMessage","content",0,"text"],"Let me "]
```
Measured, interned ops are **smaller than frames** on this workload — 13.6 KB
against 21.5 KB raw for 200 deltas — because a frame carries three keys of its own
where an interned op carries one integer. So the size argument for putting frames
on the wire is dead.
What frames keep is semantic: `text_end` carries authoritative content plus a
signature in one atomic unit, where ops would need two with a weaker contract
about their relationship. That is why they remain the *input* vocabulary and are
folded before anything crosses a boundary.
### 5.2 Reducer state must live in the reduced value
Text and thinking fold purely: `block.text += frame.delta`, and `*_end` overwrites
with authoritative content plus signature. The `ended` flag in `ReducerBlockState`
is used only for validation and can be dropped.
Tool calls cannot. `toolcall_delta` does `state.json += frame.delta`, accumulating
a **raw JSON string that is never stored on the message** — the block holds
`arguments`, the parsed value. You cannot recover the accumulator from the
snapshot, and you cannot append a delta to a parsed object.
So the accumulator has to become part of the value being folded — e.g.
`operation.frameState[contentIndex].json` on `LaneView` — leaving
`AssistantMessage` clean. Generally:
> **A replicated reducer's state must be part of the replicated value.** Anything
> held beside it diverges on any consumer that did not run the producer's fold.
The corollary is that `arguments` itself should **not** be replicated. It is
derived from `json`, and a parsed object is a fresh reference on every parse, so
storing both ships two copies of the same information. Derive it on demand.
This is safe precisely because `parseStreamingJson` is **total** — four fallbacks
ending in `{}`, it cannot throw — so a replica derives unconditionally with no
error path and no agreement protocol. There is no parse-failure state to
represent, and no block needs an error slot.
### 5.3 Parse cost
`parseStreamingJson` on a growing string once per delta is quadratic per message.
Since `arguments` is now derived rather than replicated, this cost falls on whoever reads it rather than on every consumer. Presentation can refresh derived arguments at semantic checkpoints and `toolcall_end` instead of parsing on every delta; that policy is separate from why the encoder currently emits a checkpoint (§6).
## 6. What `toolcall_checkpoint` is for
`EncoderBlockState` carries `caughtUp` and `catchupJson` because a queued provider event's shared `partial` may already be ahead of that event's delta. The checkpoint catches the semantic frame stream up to the authoritative tool-call arguments visible at block start; it is not currently a general late-subscriber protocol.
After frames fold into tracked state, a Chord root replacement (`r`) is the replication and durable-recovery resync point. `toolcall_checkpoint` remains semantic input to that fold rather than carrying transport responsibility.
## 7. Write volume for pending output
`openFrameProgress` calls `appendList(pendingAssistantFrames(...))`, so every
frame is a line. A long response is thousands of writes.
**The address is renamed `pendingAssistantOutput`** and becomes a `list<WireOp[]>`, matching `pendingToolOutput` ([tool-output handoff §7.2](../04-tool-output/harness-tools.md#72-renaming)), and it stops being a list of frames. The progress sink encodes tracked `Op[]` with one stateful encoder per response before appending each durable `WireOp[]` batch; explicit `rebase()` calls produce bounded root-replacement batches for recovery. The list lives in an **ephemeral scope** so it is unlinked on settle rather than persisting in the main log ([scoped storage](../02-scopes/scopes.md)).
The important property is that this list is **not history**: `deleteList` runs on
settle in `response.ts`, `deferred.ts`, and `terminal.ts`. It exists so a crash
mid-response can recover a partial assistant message. The settled message is persisted
separately.
That means per-frame durability buys almost nothing, and the write rate can be traded
away directly:
- **Coalesce with a bounded, non-resetting window.** The first pending frame opens the
window; later frames join it *without* extending the deadline, so a continuously
streaming response cannot postpone the first write indefinitely — the failure mode a
naive debounce has. Frames admitted during an active write form the next batch.
- **Concatenate on flush.** A run of `text_delta` frames for one `contentIndex`
collapses to a single frame with the joined text. The fold result is identical, so
this is lossless for our purposes.
A crash then loses at most one window of in-flight streaming.
### 7.1 Contrast with DeepSeek Harness
DSH cannot make this trade. Its `assistant/chunk` events are canonical log entries, so
per-token durability is mandatory and it attacks size instead:
- the same bounded non-resetting write-behind window;
- **packed rows** — runs of consecutive chunk deltas stored as
`text-chunks` / `reasoning-chunks` / `tool-call-chunks`, lossless and roughly 60%
smaller on a real session, with reading unconditional so layout never depends on the
write switch;
- checksummed zstd frames by default, with recovery from a torn final frame.
Their packing must reconstruct exact event boundaries, sequence numbers and timestamps,
because `seq = log.length` and validation requires a contiguous logical log. Ours does
not, because the frames are discarded at settle — so plain concatenation is available
to us and packing machinery is not needed.
## 8. Consequence for ad-hoc listeners
`HarnessEvent.message_update` stops being self-describing. A listener that attaches
mid-stream sees a delta against a partial it does not hold.
Any correct listener already has the base, because `watch()` installs the subscription
and captures the snapshot inside one `readLane` critical section, buffering until
`start()`. But a listener that calls `on("message_update")` directly, without a watch,
is no longer viable — worth knowing before committing.

1855
packages/agent/docs/mobile-handoff/02-plugins/01-facets/facets.md Normal file
View File

File diff suppressed because it is too large Load Diff

162
packages/agent/docs/mobile-handoff/02-plugins/02-sandbox/README.md Normal file
View File

@@ -0,0 +1,162 @@
# Facet sandbox — isolated-vm
**Unmodified pi facet code running in a V8 isolate with no ambient authority.**
Full JIT, so ~1.8× native rather than the 8–17× a WASM interpreter costs.
```bash
npm install
npm run demo # a facet: contributions, components, commands, callbacks
npm run audit # escape audit — the interesting one
npm run bench # crossing cost, isolate cost, 300 components
npm test # 412 property assertions
```
Node 22+ (`--experimental-strip-types`). See `../facet-sandbox-poc` for the
QuickJS-WASM equivalent.
## The safety claim, and why it is structural
isolated-vm **can** be used unsafely, and that is what `facets.md` §14.2 objected
to. The footgun is handing the guest a live handle to a host object — via
`derefInto()`, or by returning a `Reference` from a host call — after which the
guest walks that object's prototype chain into the host realm.
This membrane makes that impossible **by construction, not by care**:
1. `encode()` is the only way a host value reaches the guest, and it is
`JSON.stringify` with a replacer. **Its output is a string.** A string cannot
carry a reference.
2. Host callables never cross. They become an integer id into `hostTable`, which
lives entirely host-side. The guest receives a **number**.
3. Exactly **one** `Reference` is ever given to the guest — `__invokeRef`, the
single call-in point.
4. `derefInto()` is used exactly once, on the guest's **own** global. No host
object is ever its argument.
The guest's entire view of the host is `{ number, string }`. There is no object
graph, so there is nothing to walk.
### Audit results (`npm run audit`)
```
Ambient authority:
require / process / fetch ["undefined","undefined","undefined"]
global names visible 64 globals
Classic escape ladders:
Function('return process')() undefined
constructor walk on host data undefined
The isolated-vm Reference footgun:
__invokeRef.deref() blocked: TypeError ("Cannot dereference
this from current isolate")
__invokeRef.copySync() blocked: TypeError
__invokeRef.getSync('constructor') blocked: TypeError
derefInto() is callable? inert (object)
invoke derefInto() result blocked: TypeError
host globals via derefInto() no host globals
Resource bounds:
spinning guest interrupted after 205ms
isolate usable afterwards true
```
One probe gave a **false positive** in a first draft: testing `f.deref === undefined`
to prove a returned host function is a Proxy rather than a Reference. The
membrane's proxy returns a proxy for *every* property, so `f.deref` is truthy.
That is not a leak — calling it routes to `hostFn["deref"]`, which does not exist
and throws host-side. The audit now calls it instead of checking for absence.
## Performance
| | isolated-vm | QuickJS | native |
| --- | --- | --- | --- |
| 300 markdown components, small | **70 ms** | 494 ms | ~40 ms |
| 300 markdown components, typical | **108 ms** | 1118 ms | 64 ms |
| 300 markdown components, large | **419 ms** | 4260 ms | ~150 ms |
| 146 KB bundle load | **35 ms** | 104 ms | — |
| membrane crossing | **4.5 µs** | 7–17 µs | — |
| per compartment | 1080 KB, 5.5 ms | **77 KB, 0.8 ms** | — |
**~1.8× native at typical size.** QuickJS is 8–17× and gets worse with input size.
Isolates cost **14× more memory than a QuickJS runtime** (1080 KB vs 77 KB) and
are slower to create. For a handful of facets that is irrelevant; for hundreds it
would not be.
`Intl.Segmenter` **exists** here, which is the single hard blocker for reusing
`packages/tui` components under QuickJS. That alone may decide it.
## Limitations
**Budgets do not nest.** A timeout bounds one `evalSync`. A guest function
re-entered *by the host* — a contributed callback, a component method — gets
`callGuestRef`'s own budget, not the outer one. Asserted explicitly in
`property-test.ts`; a runaway callback took 5005 ms under an outer budget of 50 ms.
Bounding total facet time needs separate accounting.
**Async is a settle-callback, not a native promise.** `applySync` is synchronous,
so a host Promise cannot cross. The host returns a token, the guest builds a real
Promise against it, and the host settles it by calling back in. Verified: two
sequential guest `await`s while the host event loop ticked 22 times.
**Native addon, and the ABI matrix is real.** `isolated-vm` ships prebuilds and
only falls back to `node-gyp rebuild` when none matches — which is why install
takes a second, not an hour. It is **not** building V8; V8 is already in the Node
binary. But coverage is narrow:
| version | engines | prebuilds |
| --- | --- | --- |
| **6.2.0** | `>=22.0.0` | linux-x64/arm64, darwin-arm64, win32-x64 — abi127, abi137 |
| 7.0.1 | `>=24.0.0` | — |
| 7.0.0 | `>=26.0.0` | — |
darwin-**x64** is absent everywhere. Installing `isolated-vm@7` on Node 22 falls
through to a source build and **fails without Python and a C++ toolchain** —
verified here. Note also that 6.2.0 is `latest` on npm despite 7.0.1 existing,
because it published one day later.
The project is **not abandonware** — 6.0.1 (Jul 2025) through 7.0.1 (Aug 2026),
actively released. §14.2's "maintenance mode" claim is out of date and should be
corrected. The real risk is different: adopting it couples your minimum Node
version to theirs, and they dropped Node 20 in 6.x and Node 22 in 7.x within a
year. Shipping SEA builds moves that from every user's install to your CI.
**Memory: references release, but lazily.** The `WeakRef` + `FinalizationRegistry`
interning table ports unchanged from the QuickJS membrane and works — 20 000
crossings went to **0 live refs**. But finalization is not prompt: 5 000 refs were
still live 200 ms after a forced GC. Under sustained churn refs accumulate faster
than the collector reclaims them, and a 32 MB isolate did hit its limit.
The consequence is an API rule, not a membrane fix: **do not create references in
a hot path.** `slots.claim(factory)` at construct is one reference forever; a
component method returning a fresh closure every frame is one per frame.
**Hitting `memoryLimit` kills the facet cleanly.** It throws a catchable
`"Isolate was disposed during execution due to memory limit"` and the host
survives — but the isolate is dead and unrecoverable, so teardown must tolerate
an already-disposed isolate. `dispose()` on it throws `"Isolate is already
disposed"`; the membrane guards this.
> **Do not call `isolate.getHeapStatisticsSync()` on a stressed isolate.** It
> **aborted the whole process** during testing — a hard crash, not an exception.
> A `heapMB()` helper was removed from the membrane rather than shipped.
**Still one process, one engine.** A separate isolate is a far stronger boundary
than SES, but Figma's argument for QuickJS was that a *different VM* cannot
confuse objects because the representations differ. Here the guarantee is V8's
isolate boundary plus the membrane's discipline — strong, and structurally
enforced above, but not the same category of claim.
## Files
- `src/membrane.ts` — the membrane. The safety argument is in the header comment.
- `src/facet-example.js` — unmodified facet code: `ctx.use()`, `slots.claim()`
with a closure, a class instance returned to the host, a subscribe callback.
- `src/demo-facet.ts` — loads it and exercises four boundary crossings.
- `src/escape-audit.ts` — the audit above.
- `src/property-test.ts` — 412 assertions: random `JsonValue` round-trips both
directions, identity across crossings, callables, error propagation, reference
release under GC, prototype pollution both ways, budgets, async, disposal.
- `src/bench.ts` — the numbers above.
- `src/markdown-bundle.js` — the real pi `Markdown` component, esbuild-bundled.

13
packages/agent/docs/mobile-handoff/02-plugins/02-sandbox/package.json Normal file
View File

@@ -0,0 +1,13 @@
{
"name": "facet-sandbox-ivm",
"private": true,
"type": "module",
"description": "Running pi facets in an isolated-vm isolate: full JIT, no ambient authority.",
"scripts": {
"demo": "node --experimental-strip-types src/demo-facet.ts",
"audit": "node --experimental-strip-types src/escape-audit.ts",
"bench": "node --experimental-strip-types src/bench.ts",
"test": "node --experimental-strip-types --expose-gc src/property-test.ts"
},
"dependencies": { "isolated-vm": "6.2.0" }
}

48
packages/agent/docs/mobile-handoff/02-plugins/02-sandbox/src/bench.ts Normal file
View File

@@ -0,0 +1,48 @@
/**
* Cost of the isolate, the membrane, and a real component at transcript scale.
* Compare against `../facet-sandbox-poc` (QuickJS) and native.
*/
import ivm from "isolated-vm";
import { readFileSync } from "node:fs";
import { createCompartment } from "./membrane.ts";
// ── membrane crossing ─────────────────────────────────────────────────────
{
const c = await createCompartment();
await c.endow("__h", { echo: (v: unknown) => v });
c.load(`globalThis.spin = function (n) { for (var i = 0; i < n; i++) __h.echo("x"); };`);
c.load(`spin(2000);`);
const N = 20000, t = Date.now();
c.load(`spin(${N});`, 60000);
console.log(`membrane crossing: ${((Date.now() - t) * 1000 / N).toFixed(1)} µs`);
c.dispose();
}
// ── isolate creation ──────────────────────────────────────────────────────
{
const before = process.memoryUsage().rss, t = Date.now();
const held = Array.from({ length: 50 }, () => {
const iso = new ivm.Isolate({ memoryLimit: 32 });
const ctx = iso.createContextSync();
ctx.evalSync(`var c = { render: function () { return ["x"]; } };`);
return iso;
});
console.log(`50 isolates: ${(((process.memoryUsage().rss - before) / 50) / 1024).toFixed(0)} KB each, ${Date.now() - t} ms total`);
for (const i of held) i.dispose();
}
// ── real Markdown at transcript scale ─────────────────────────────────────
{
const iso = new ivm.Isolate({ memoryLimit: 512 });
const ctx = iso.createContextSync();
ctx.global.setSync("global", ctx.global.derefInto());
const t = Date.now();
ctx.evalSync(readFileSync(new URL("./markdown-bundle.js", import.meta.url), "utf8"), { timeout: 60000 });
console.log(`146 KB bundle load: ${Date.now() - t} ms`);
for (const [label, pts] of [["small", 2], ["typical", 8], ["large", 35]] as [string, number][]) {
ctx.evalSync(`__bench(${pts}, 20, 50)`, { timeout: 120000 }); // warm the JIT
const ms = ctx.evalSync(`__bench(${pts}, 300, 3)`, { timeout: 300000 });
console.log(`300 markdown components, ${label.padEnd(8)} ${ms.toFixed(0)} ms per full repaint`);
}
iso.dispose();
}

22
packages/agent/docs/mobile-handoff/02-plugins/02-sandbox/src/demo-facet.ts Normal file
View File

@@ -0,0 +1,22 @@
import { readFileSync } from "node:fs";
import { createCompartment } from "./membrane.ts";
const mounted = new Map(), commands = new Map(), notifications = [], subs = [];
const Tui = {
slots: { claim: (s, f) => mounted.set(s, f) },
commands: { add: (n, h) => commands.set(n, h) },
notify: (m) => notifications.push(m),
};
const Transcript = { tail: { subscribe: (f) => subs.push(f) } };
const c = await createCompartment();
await c.endow("__host", { use: (t) => (t === "Tui" ? Tui : Transcript) });
c.load(readFileSync(new URL("./facet-example.js", import.meta.url), "utf8"));
c.load(`globalThis.__r = __facet.construct({ use: function (t) { return __host.use(t); } });`);
console.log("contributions:", { slots: [...mounted.keys()], commands: [...commands.keys()], subs: subs.length });
const comp = mounted.get("footer")({ label: "presses" });
console.log(comp.render(30).join("\n"));
comp.handleInput("abc");
console.log(comp.render(30).join("\n"));
console.log("command ->", commands.get("footer.reset")());
subs[0]({ entries: [1, 2, 3] });
console.log("notifications:", notifications);
c.dispose();

63
packages/agent/docs/mobile-handoff/02-plugins/02-sandbox/src/escape-audit.ts Normal file
View File

@@ -0,0 +1,63 @@
/**
* Escape audit.
*
* The claim the membrane makes is that the guest's entire view of the host is
* `{ number, string }` — no object graph, so nothing to walk. This tries to
* break that, including the specific `Reference` footgun that makes isolated-vm
* dangerous when used naively.
*/
import { createCompartment } from "./membrane.ts";
const c = await createCompartment();
const seen: string[] = [];
await c.endow("__h", {
sink: (v: string) => seen.push(v),
give: () => ({ nested: { a: 1 } }),
giveFn: () => () => "host function result",
});
const probe = (name: string, src: string) => {
c.load(`globalThis.__p = String(${src});`);
c.load(`__h.sink(__p);`);
const got = seen.pop();
const bad = got !== undefined && /GOT (?!undefined)|INVOKED|CALLABLE|^object$/.test(got) && !/blocked/.test(got);
console.log(` ${bad ? "!!" : "ok"} ${name.padEnd(34)} ${got}`);
return !bad;
};
console.log("Ambient authority:");
let ok = true;
ok = probe("require / process / fetch", `JSON.stringify([typeof require, typeof process, typeof fetch])`) && ok;
ok = probe("global names visible", `Object.getOwnPropertyNames(globalThis).length + " globals"`) && ok;
console.log("\nClassic escape ladders:");
ok = probe("Function('return process')()", `(function(){ try { return String((function(){}).constructor("return typeof process")()); } catch(e){ return "blocked: "+e.name; } })()`) && ok;
ok = probe("constructor walk on host data", `(function(){ try { var o = __h.give(); return String(o.constructor.constructor("return typeof process")()); } catch(e){ return "blocked: "+e.name; } })()`) && ok;
console.log("\nThe isolated-vm Reference footgun:");
ok = probe("__invokeRef.deref()", `(function(){ try { return "GOT " + typeof __invokeRef.deref(); } catch(e){ return "blocked: "+e.name; } })()`) && ok;
ok = probe("__invokeRef.copySync()", `(function(){ try { return "GOT " + typeof __invokeRef.copySync(); } catch(e){ return "blocked: "+e.name; } })()`) && ok;
ok = probe("__invokeRef.getSync('constructor')", `(function(){ try { return "GOT " + typeof __invokeRef.getSync("constructor"); } catch(e){ return "blocked: "+e.name; } })()`) && ok;
ok = probe("derefInto() is callable?", `(function(){ var d = __invokeRef.derefInto(); return typeof d === "function" ? "CALLABLE" : "inert (" + typeof d + ")"; })()`) && ok;
ok = probe("invoke derefInto() result", `(function(){ try { __invokeRef.derefInto()(); return "INVOKED"; } catch(e){ return "blocked: "+e.name; } })()`) && ok;
ok = probe("host globals via derefInto()", `(function(){ var d = __invokeRef.derefInto(); return String(d.process || d.global || "no host globals"); })()`) && ok;
console.log("\nWhat a host FUNCTION looks like to the guest:");
ok = probe("typeof a returned host fn", `(function(){ var f = __h.giveFn(); return typeof f; })()`) && ok;
// NOTE: `f.deref` is NOT undefined — the membrane's proxy returns a proxy for
// every property. That is not a leak: calling it routes to hostFn["deref"],
// which does not exist, and throws host-side. Testing for the property's
// absence gave a false positive in a first draft of this audit.
ok = probe("calling f.deref() reaches nothing", `(function(){ try { var f = __h.giveFn(); f.deref(); return "INVOKED"; } catch(e){ return "blocked: "+e.name; } })()`) && ok;
ok = probe("calling the host fn works", `(function(){ var f = __h.giveFn(); return String(f()); })()`) && ok;
console.log("\nResource bounds:");
const t0 = Date.now();
let interrupted = false;
try { c.load(`while(true){}`, 200); } catch { interrupted = true; }
console.log(` ${interrupted ? "ok" : "!!"} spinning guest interrupted after ${Date.now() - t0}ms`);
console.log(` ok isolate usable afterwards ${c.load(`1+1`) === 2}`);
c.dispose();
console.log(`\n${ok && interrupted ? "AUDIT PASSED" : "AUDIT FAILED"}`);
process.exit(ok && interrupted ? 0 : 1);

42
packages/agent/docs/mobile-handoff/02-plugins/02-sandbox/src/facet-example.js Normal file
View File

@@ -0,0 +1,42 @@
// ── REAL facet code. Nothing here knows it is sandboxed. ────────────────────
// No imports rewritten, no string marshalling, no __dispatch. It uses the
// host API exactly as facets.md §11.1 describes it.
class FooterComponent {
constructor(props) { this.label = props.label; this.count = 0; }
render(width) {
const text = ` ${this.label}: ${this.count} `;
const pad = Math.max(0, width - text.length - 2);
const bar = "\u2500".repeat(Math.max(0, width - 2));
return [`\u250c${bar}\u2510`, `\u2502${text}${" ".repeat(pad)}\u2502`, `\u2514${bar}\u2518`];
}
handleInput(data) { this.count += data.length; }
}
const facet = {
id: "@demo/footer:tui",
uses: ["Tui", "Transcript"],
construct(ctx) {
const tui = ctx.use("Tui");
const transcript = ctx.use("Transcript");
// A factory crossing guest → host, by reference.
tui.slots.claim("footer", (props) => new FooterComponent(props));
// A command handler, likewise.
tui.commands.add("footer.reset", () => {
tui.notify("footer reset");
return "ok";
});
// A subscription: host calls back into the guest.
transcript.tail.subscribe((tail) => {
tui.notify(`transcript now has ${tail.entries.length} entries`);
});
return [];
},
};
globalThis.__facet = facet;

34
packages/agent/docs/mobile-handoff/02-plugins/02-sandbox/src/iso-vs-ctx.ts Normal file
View File

@@ -0,0 +1,34 @@
import ivm from "isolated-vm";
const rss = () => process.memoryUsage().rss / 1024 / 1024;
// 1. Are two CONTEXTS in one isolate isolated from each other?
{
const iso = new ivm.Isolate({ memoryLimit: 64 });
const a = iso.createContextSync(), b = iso.createContextSync();
a.evalSync(`globalThis.secret = "from-A"`);
console.log("context B sees A's global:", b.evalSync(`typeof globalThis.secret`));
// Can a spin in A be timed out without harming B?
try { a.evalSync(`while(true){}`, { timeout: 100 }); } catch { /* expected */ }
console.log("B usable after A was interrupted:", b.evalSync(`1+1`) === 2);
// Does A's memory bomb kill B?
try { a.evalSync(`var x=[]; for(;;) x.push(new Array(50000).fill(1));`, { timeout: 15000 }); }
catch (e: any) { console.log("A bomb:", e.message.slice(0, 45)); }
try { console.log("B after A's bomb:", b.evalSync(`1+1`)); }
catch (e: any) { console.log("B after A's bomb: DEAD -", e.message.slice(0, 40)); }
try { iso.dispose(); } catch {}
}
// 2. Cost per facet, both ways.
for (const mode of ["one isolate, N contexts", "N isolates"]) {
const before = rss(), t = Date.now();
const held: any[] = [];
const shared = mode.startsWith("one") ? new ivm.Isolate({ memoryLimit: 512 }) : null;
for (let i = 0; i < 50; i++) {
const iso = shared ?? new ivm.Isolate({ memoryLimit: 32 });
const ctx = iso.createContextSync();
ctx.evalSync(`var c = { render: function () { return ["x"]; } };`);
held.push(iso);
}
console.log(`50 facets, ${mode.padEnd(24)}: ${(((rss() - before) / 50) * 1024).toFixed(0)} KB each, ${Date.now() - t} ms`);
if (shared) { try { shared.dispose(); } catch {} } else for (const i of held) { try { i.dispose(); } catch {} }
}

27
packages/agent/docs/mobile-handoff/02-plugins/02-sandbox/src/leak.ts Normal file
View File

@@ -0,0 +1,27 @@
import { createCompartment } from "./membrane.ts";
const c = await createCompartment();
let seen = 0;
await c.endow("__l", { drop: (_fn: unknown) => { seen++; } });
c.load(`globalThis.churn = function (n) { for (var i = 0; i < n; i++) __l.drop(function () { return i; }); };`);
const rss = () => process.memoryUsage().rss / 1024 / 1024;
const before = rss();
c.load(`churn(20000);`, 60000);
const peak = c.debugRefCount();
const afterChurn = rss();
globalThis.gc?.(); await new Promise((r) => setTimeout(r, 100));
globalThis.gc?.(); await new Promise((r) => setTimeout(r, 100));
const settled = c.debugRefCount();
console.log(`20000 crossings: host called ${seen}x`);
console.log(` guest refs: ${peak.guestRefs} -> ${settled.guestRefs}`);
console.log(` host table: ${peak.hostTable} -> ${settled.hostTable}`);
console.log(` rss: ${before.toFixed(0)} -> ${afterChurn.toFixed(0)} -> ${rss().toFixed(0)} MB`);
// Cost of the release path: one evalSync crossing per finalized ref.
const t = Date.now();
c.load(`churn(5000);`, 60000);
globalThis.gc?.(); await new Promise((r) => setTimeout(r, 200));
console.log(` 5000 more + gc: ${Date.now() - t} ms, refs now ${c.debugRefCount().guestRefs}`);
c.dispose();

3888
packages/agent/docs/mobile-handoff/02-plugins/02-sandbox/src/markdown-bundle.js Normal file
View File

File diff suppressed because it is too large Load Diff

197
packages/agent/docs/mobile-handoff/02-plugins/02-sandbox/src/membrane.ts Normal file
View File

@@ -0,0 +1,197 @@
import ivm from "isolated-vm";
/**
* A membrane between the host realm and an isolated-vm isolate.
*
* ── The safety argument, which is structural, not a convention ──────────────
*
* isolated-vm CAN be used unsafely. The documented footgun is handing the guest
* a live handle to a host object — via `derefInto()`, or by returning a
* `Reference` from a host call — after which the guest can walk that object's
* prototype chain into the host realm.
*
* This membrane makes that impossible by construction rather than by care:
*
* 1. `encode()` is the ONLY way a host value reaches the guest, and it is
* `JSON.stringify` with a replacer. Its output is a string. A string
* cannot carry a reference.
* 2. Host callables do not cross. They are replaced by an integer id into
* `hostTable`, which lives entirely host-side. The guest receives a number.
* 3. Exactly ONE `Reference` is ever given to the guest — `__invokeRef`, the
* single call-in point — and its `deref()` throws cross-isolate ("Cannot
* dereference this from current isolate"). Its `derefInto()` yields an
* inert `Dereference` marker: not callable, no host globals.
* 4. `derefInto()` is used exactly once, on the guest's OWN global. No host
* object is ever an argument to it.
*
* The consequence: the guest's entire view of the host is `{ number, string }`.
* There is no object graph to walk, so there is nothing to walk it into. This is
* the same property the QuickJS membrane has, and it is why the choice between
* them is about performance and packaging rather than about safety.
*
* Verified by `src/escape-audit.ts`.
*/
export async function createCompartment({ memoryLimit = 256 } = {}) {
const isolate = new ivm.Isolate({ memoryLimit });
const ctx = await isolate.createContext();
await ctx.global.set("global", ctx.global.derefInto()); // guest's own global only
const hostTable = new Map();
let nextHostId = 1;
const exposeHost = (o) => { const id = nextHostId++; hostTable.set(id, o); return id; };
const guestProxies = new Map();
const finalizer = new FinalizationRegistry((id) => {
guestProxies.delete(id);
if (!released) ctx.evalSync(`delete __refs[${JSON.stringify(String(id))}]`);
});
let released = false;
// Async: `applySync` is synchronous, so a host Promise cannot itself cross.
// Instead the host returns a token, the guest builds a real Promise against
// it, and the host settles that Promise by calling back in when it resolves.
let nextPending = 1;
function encodePending(promise) {
const token = nextPending++;
promise.then(
(v) => settle(token, true, v === undefined ? "" : encode(v)),
(e) => settle(token, false, JSON.stringify(String(e?.message ?? e))),
);
return JSON.stringify({ __pending: token });
}
function settle(token, ok, json) {
if (released) return;
ctx.evalSync(`__settle(${token}, ${ok}, ${JSON.stringify(json)})`);
}
function encode(v) {
if (v && typeof v.then === "function") return encodePending(v);
return JSON.stringify(v, (_k, val) => {
if (typeof val === "function") return { __ref: exposeHost(val) };
if (val && typeof val === "object" && !Array.isArray(val) && hasMethods(val))
return { __obj: exposeHost(val), keys: methodNames(val) };
return val;
});
}
const decode = (json) => JSON.parse(json, (_k, v) => reviveGuest(v));
function intern(id, build) {
const hit = guestProxies.get(id)?.deref();
if (hit !== undefined) return hit;
const p = build();
guestProxies.set(id, new WeakRef(p));
finalizer.register(p, id);
return p;
}
function reviveGuest(v) {
if (!v || typeof v !== "object") return v;
if (typeof v.__guestRef === "number") { const r = v.__guestRef; return intern(r, () => (...a) => callGuestRef(r, a)); }
if (typeof v.__guestObj === "number") {
const r = v.__guestObj;
return intern(r, () => { const o = {}; for (const k of v.keys) o[k] = (...a) => callGuestRef(r, [k, a]); return o; });
}
return v;
}
function callGuestRef(ref, args, timeout = 5000) {
const out = ctx.evalSync(
`__refs[${JSON.stringify(String(ref))}](${JSON.stringify(JSON.stringify(args))})`,
{ timeout },
);
return out === "" || out === undefined ? undefined : decode(out);
}
// The single host entry point. A Reference, applied synchronously with copied
// arguments — no host object graph is ever exposed.
const invoke = new ivm.Reference((idJson, pathJson, argsJson) => {
const path = JSON.parse(pathJson);
let target = hostTable.get(JSON.parse(idJson));
for (let i = 0; i < path.length - 1; i++) target = target[path[i]];
const result = path.length ? target[path[path.length - 1]](...decode(argsJson)) : target(...decode(argsJson));
return result === undefined ? "" : encode(result);
});
await ctx.global.set("__invokeRef", invoke);
await ctx.eval(`
globalThis.__refs = {}; globalThis.__nextRef = 1;
globalThis.__outgoing = new WeakMap();
globalThis.__methodNames = function (o) {
var out = [], p = o;
while (p && p !== Object.prototype) {
for (const k of Object.getOwnPropertyNames(p))
if (k !== "constructor" && typeof o[k] === "function" && out.indexOf(k) < 0) out.push(k);
p = Object.getPrototypeOf(p);
}
return out;
};
globalThis.__mkRef = function (fn) {
var prior = __outgoing.get(fn); if (prior !== undefined) return { __guestRef: prior };
var id = __nextRef++; __outgoing.set(fn, id);
__refs[String(id)] = function (j) { var r = fn.apply(null, JSON.parse(j)); return r === undefined ? "" : __encode(r); };
return { __guestRef: id };
};
globalThis.__mkObjRef = function (obj) {
var prior = __outgoing.get(obj);
if (prior !== undefined) return { __guestObj: prior, keys: __methodNames(obj) };
var id = __nextRef++; __outgoing.set(obj, id);
__refs[String(id)] = function (j) { var c = JSON.parse(j); var r = obj[c[0]].apply(obj, c[1]); return r === undefined ? "" : __encode(r); };
return { __guestObj: id, keys: __methodNames(obj) };
};
globalThis.__encode = function (v) {
return JSON.stringify(v, function (k, val) {
if (typeof val === "function") return __mkRef(val);
if (val && typeof val === "object" && !Array.isArray(val) && __methodNames(val).length) return __mkObjRef(val);
return val;
});
};
globalThis.__pending = {};
globalThis.__settle = function (token, ok, json) {
var d = __pending[token];
if (!d) return;
delete __pending[token];
var value = json === "" ? undefined : JSON.parse(json);
ok ? d.resolve(__revive(value)) : d.reject(new Error(value));
};
globalThis.__revive = function (v) {
if (v && typeof v === "object" && typeof v.__pending === "number") {
var token = v.__pending;
return new Promise(function (res, rej) { __pending[token] = { resolve: res, reject: rej }; });
}
if (v && typeof v === "object") {
if (typeof v.__ref === "number") return __wrap(v.__ref, []);
if (typeof v.__obj === "number") return __wrap(v.__obj, []);
for (var k in v) v[k] = __revive(v[k]);
}
return v;
};
globalThis.__wrap = function (id, path) {
return new Proxy(function () {}, {
get: function (t, prop) {
if (typeof prop !== "string") return undefined;
if (prop === "then" || prop === "toJSON" || prop === "inspect") return undefined;
if (prop === "length" || prop === "name") return t[prop];
return __wrap(id, path.concat([prop]));
},
apply: function (_t, _this, args) {
var out = __invokeRef.applySync(undefined, [JSON.stringify(id), JSON.stringify(path), __encode(args)]);
return out === "" ? undefined : __revive(JSON.parse(out));
},
});
};
`);
return {
async endow(name, obj) { await ctx.eval(`globalThis[${JSON.stringify(name)}] = __wrap(${exposeHost(obj)}, []);`); },
load: (src, timeout = 30000) => ctx.evalSync(src, { timeout }),
debugRefCount: () => ({ hostTable: hostTable.size, guestRefs: ctx.evalSync(`Object.keys(__refs).length`) }),
dispose() {
released = true;
// An isolate that hit its memory limit is already disposed, and calling
// dispose() again throws "Isolate is already disposed".
try { isolate.dispose(); } catch { /* already gone */ }
},
};
}
const RESERVED = new Set(["constructor"]);
const methodNames = (o) => { const out = []; for (let p = o; p && p !== Object.prototype; p = Object.getPrototypeOf(p)) for (const k of Object.getOwnPropertyNames(p)) if (!RESERVED.has(k) && typeof o[k] === "function" && !out.includes(k)) out.push(k); return out; };
const hasMethods = (o) => methodNames(o).length > 0;

137
packages/agent/docs/mobile-handoff/02-plugins/02-sandbox/src/property-test.ts Normal file
View File

@@ -0,0 +1,137 @@
/**
* Property tests for the membrane.
*
* Every bug found by hand while building it failed SILENTLY — wrong output, no
* throw. That is the worst possible failure mode for a boundary, and the reason
* these exist. Run with: node --experimental-strip-types --expose-gc property-test.ts
*/
import { createCompartment } from "./membrane.ts";
let pass = 0;
const failures: string[] = [];
const check = (name: string, ok: boolean, detail = "") => {
if (ok) pass++;
else failures.push(`${name}${detail ? ` — ${detail}` : ""}`);
};
/** Random JsonValue. The membrane claims to round-trip exactly these. */
function randomJson(depth = 0): unknown {
const r = Math.random();
if (depth > 3 || r < 0.25) return Math.floor(Math.random() * 1000) - 500;
if (r < 0.35) return Math.random() * 1e6 - 5e5;
if (r < 0.45) return Math.random() < 0.5;
if (r < 0.5) return null;
if (r < 0.65) {
const alphabet = ['a', 'z', '0', ' ', '"', '\\', '\n', '\u00e9', '\u4e2d', '\u{1f600}'];
let s = "";
const n = Math.floor(Math.random() * 12);
for (let i = 0; i < n; i++) s += alphabet[Math.floor(Math.random() * alphabet.length)];
return s;
}
if (r < 0.8) return Array.from({ length: Math.floor(Math.random() * 5) }, () => randomJson(depth + 1));
const o: Record<string, unknown> = {};
for (const k of ["a", "b", "c", "d", "__proto__x", "0", "length"]) {
if (Math.random() < 0.5) o[k] = randomJson(depth + 1);
}
return o;
}
const c = await createCompartment();
// ── 1. Value round-trip, both directions ──────────────────────────────────
const echoed: unknown[] = [];
await c.endow("__h", { echo: (v: unknown) => v, sink: (v: unknown) => { echoed.push(v); } });
c.load(`globalThis.roundTrip = function (v) { return __h.echo(v); };
globalThis.sendBack = function (v) { __h.sink(v); };`);
for (let i = 0; i < 400; i++) {
const value = randomJson();
const json = JSON.stringify(value);
if (json === undefined) continue;
// host -> guest -> host
c.load(`globalThis.__t = roundTrip(${json});`);
c.load(`sendBack(__t);`);
const got = echoed.pop();
check("value round-trip", JSON.stringify(got) === json, `${json} became ${JSON.stringify(got)}`);
}
// ── 2. Identity: the same guest object must map to the same host proxy ────
const seen: unknown[] = [];
await c.endow("__id", { take: (o: unknown) => { seen.push(o); } });
c.load(`globalThis.shared = { m: function () { return 1; } };
globalThis.sendShared = function () { __id.take(shared); __id.take(shared); };`);
c.load(`sendShared();`);
check("identity preserved across crossings", seen[0] === seen[1]);
c.load(`sendShared();`);
check("identity stable across separate calls", seen[0] === seen[2]);
// ── 3. Callables survive and remain callable ──────────────────────────────
let held: any;
await c.endow("__f", { keep: (fn: unknown) => { held = fn; } });
c.load(`__f.keep(function (a, b) { return a + b; });`);
check("guest function callable from host", held(2, 3) === 5, `got ${held?.(2, 3)}`);
// ── 4. Errors cross with name and message ─────────────────────────────────
c.load(`__f.keep(function () { throw new TypeError("boom"); });`);
let msg = "";
try { held(); } catch (e: any) { msg = e.message; }
check("guest throw reaches host", msg.includes("boom"), `got "${msg}"`);
// ── 5. Reference release: crossings must not leak ─────────────────────────
await c.endow("__l", { drop: (_fn: unknown) => {} });
c.load(`globalThis.churn = function () { for (var i = 0; i < 500; i++) __l.drop(function () { return i; }); };`);
c.load(`churn();`);
const beforeGc = c.debugRefCount().guestRefs;
globalThis.gc?.();
await new Promise((r) => setTimeout(r, 60));
globalThis.gc?.();
await new Promise((r) => setTimeout(r, 60));
const afterGc = c.debugRefCount().guestRefs;
check("dropped references are released", afterGc < beforeGc / 2, `${beforeGc} -> ${afterGc}`);
// ── 5b. Prototype pollution must not cross ────────────────────────────────
// JSON.parse is safe (it makes __proto__ an own property), but a membrane that
// assigns parsed keys onto a fresh object is not.
const before = ({} as any).polluted;
c.load(`__h.sink(JSON.parse('{"__proto__":{"polluted":true}}'));`);
check("guest cannot pollute host Object.prototype", ({} as any).polluted === before);
await c.endow("__p", { give: () => JSON.parse('{"__proto__":{"hostPolluted":true}}') });
await c.endow("__q", { show: (v: string) => check("host cannot pollute guest prototype", v === "undefined", v) });
c.load(`var got = __p.give(); __q.show(String(({}).hostPolluted));`);
// ── 6. Interrupt budget, and the VM survives it ───────────────────────────
// Direct: the budget on this call bounds it.
{
const t0 = Date.now();
try { c.load(`while (true) {}`, 100); } catch { /* expected */ }
check("runaway guest code is interrupted", Date.now() - t0 < 600, `took ${Date.now() - t0}ms`);
}
// Re-entrant: a guest function invoked BY THE HOST gets `callGuestRef`'s own
// budget, not the outer one. Budgets do not nest — see README "Limitations".
c.load(`globalThis.spin = function () { while (true) {} };`);
await c.endow("__s", { run: (fn: any) => { try { fn(); return "returned"; } catch { return "threw"; } } });
{
const t0 = Date.now();
try { c.load(`globalThis.__x = __s.run(spin);`, 50); } catch { /* expected */ }
const took = Date.now() - t0;
check("re-entrant call uses its OWN budget, not the outer one", took > 1000,
`took ${took}ms — if this is now <1s the nesting behaviour changed`);
}
c.load(`globalThis.__alive = 1 + 1;`);
check("VM usable after an interrupt", true);
// ── 7. Async, and clean teardown with a pending promise ───────────────────
await c.endow("__a", { never: () => new Promise(() => {}), soon: () => Promise.resolve("v") });
c.load(`globalThis.out = null; __a.soon().then(function (v) { out = v; });`);
await new Promise((r) => setTimeout(r, 30));
await c.endow("__o", { show: (v: string) => check("async host result reaches guest", v === '"v"', v) });
c.load(`__o.show(JSON.stringify(out));`);
c.load(`__a.never().then(function () {}, function () {});`); // deliberately unsettled
let disposeOk = true;
try { c.dispose(); } catch { disposeOk = false; }
check("dispose with a pending promise does not crash", disposeOk);
console.log(`\n${pass} passed, ${failures.length} failed`);
for (const f of failures.slice(0, 10)) console.log(" FAIL", f);
process.exit(failures.length === 0 ? 0 : 1);

81
packages/agent/docs/mobile-handoff/README.md Normal file
View File

@@ -0,0 +1,81 @@
# pi — design handoff
Work in numbered order. Each unit is self-contained and independently testable;
later units consume earlier ones.
```
01-harness/
01-delta/ op vocabulary, tracker, applier, codec [LANDED IN CHORD]
02-scopes/ storage scopes and list tags [STEP 1 ACTIONABLE]
03-execenv/ bounded Shell output, capture, spill [PRODUCTION CODE + TESTS]
04-tool-output/ the ToolOutput sink [SPEC ONLY]
05-assistant-output/ assistant partials, symmetric with 04 [SPEC ONLY]
02-plugins/
01-facets/ the facet system [SPEC ONLY]
02-sandbox/ isolated-vm membrane [CODE + 412 tests]
```
Base everything on a clean checkout of `origin/dev`.
## Read this first
**Three units ship working code. Four are specifications.** The table above says
which. Do not assume a doc describes something that exists.
**`01-delta/FINDINGS.md` is historical evidence, not an implementation queue.** Production lives in `packages/chord/src/delta/index.ts`: flush-time dirty tracking fixed D1, and production remeasurement closed D2. The explicit append/truncate API is rejected; see [`01-harness/01-delta/append-decision.md`](01-harness/01-delta/append-decision.md). The code beside the handoff remains prototype and benchmark evidence.
**If a doc and the code disagree, the code wins** — fix the doc and say so in the
commit.
**Port the tests before the implementation.** Each group's comment explains the
failure it guards against, and several of those failures are silent: wrong output,
no exception.
**Benchmark with `node --experimental-strip-types`, never through a transpiler.**
Measuring this module through `tsx` inflates results 2.6x. `FINDINGS.md` D5 lists
five more measurement traps, each of which produced a confident wrong conclusion.
## Unit status
| unit | ships | state |
| --- | --- | --- |
| **01-delta** | production implementation and tests in `packages/chord`; prototype evidence here | landed; D1 fixed, explicit text API rejected after production remeasurement |
| **02-scopes** | spec + [actionable Step 1 handoff](01-harness/02-scopes/implementation-handoff.md) + `scopes.variance.ts` | Step 1 scopes/list tags actionable, not implemented; JSONL Chord encoding/address interning deferred to separately approved Step 2 |
| **03-execenv** | production implementation in `packages/agent`; prototype evidence here | source-bounded adaptive output, lazy spill backpressure, and bash migration landed; bash's temporary checkpoint cadence moves to `ToolOutput` next |
| **04-tool-output** | spec + design notes | **not built.** The piece every measurement of the op encoding depends on |
| **05-assistant-output** | spec | not built. Same shape as 04; do it after |
| **02-plugins/01-facets** | spec, ~1800 lines | not built. §14 rewritten to match the sandbox PoC |
| **02-plugins/02-sandbox** | working PoC, 412 assertions | `npm install && npm run audit` |
## Suggested order
1. **`02-scopes` Step 1** — follow the [actionable implementation handoff](01-harness/02-scopes/implementation-handoff.md); stop for approval before its separate Step 2.
2. **`04-tool-output`** — reuse the landed adaptive publisher for generic tools, Chord event/durable batches, terminal flushes, and atomic memo checkpoints.
3. **05**, then **02-plugins**.
## Live bugs on `origin/dev`, independent of this design
- `drive/tools.ts:257` — `clearReplayCheckpoint` deletes `pendingToolOutput` before
re-executing a replay-safe tool. Memos exist so a replayed tool skips work, and
skipped work emits nothing, so output for memoised work is lost today. Seed from
it instead (`harness-tools.md` §7.4).
- `runtime/progress.ts:44` — `commitWrite(item)` captures the value at call time
and writes fire-and-forget, so an older checkpoint can land after a newer one.
- memo and checkpoint are two transactions (`drive/tools.ts:112` vs
`progress.ts:44`). They must be one (`harness-tools.md` §7.5).
**Expected test churn:** nine tests assert the old cleanup write set and will fail
once `retireScope` replaces the per-address deletes. That is the change landing.
## Environment
- Node 22+ for every shipped `.ts` file. They run under
`node --experimental-strip-types` with no build step and no dependencies.
- In the pi repo, tests run from the package:
`cd packages/agent && npx vitest run --config vitest.harness.config.ts`.
The root vitest config does **not** alias `@earendil-works/pi-ai`; the
per-package harness config does.
- Typecheck with `npx tsgo --noEmit` from the repo root. **Baseline is ~788
pre-existing errors**, almost all in `packages/ai/test`. Count only:
`grep "error TS" | grep -E "packages/(agent|session-backends)/src"`.
- `packages/ai` cannot be built offline — model data is fetched at build time.