chore(vendor): import pi 0.85.1
Imported as a linear vendor commit so forge rebases do not replay the upstream root tree. git-subtree-dir: third_party/pi git-subtree-split: d981de1229ef899957bbe968bc8dcda02a21f477
This commit is contained in:
alexion committed
1 parent
5ebc7018c3
commit
4def357d09
1669 files changed
+397627
No files matched your search
+214
@@ -0,0 +1,214 @@
|
||||
# 01-delta: known defects and measured findings
|
||||
|
||||
Everything here was found **after** `delta.md` and the prototype implementation beside it were written. None of it is applied to the code in this directory. Production now lives in `packages/chord/src/delta/index.ts`: flush-time dirty tracking resolves D1, and production remeasurement closes D2 without a new API. See [`append-decision.md`](append-decision.md). Each item below remains the original problem, evidence, and candidate-fix record.
|
||||
|
||||
Every number was measured with **`node --experimental-strip-types`**, not `tsx`.
|
||||
That matters: tsx transpiles through esbuild and inflated the same benchmark by
|
||||
**2.6x** (183 µs vs 63 µs per write). Do not benchmark this module through a
|
||||
transpiler.
|
||||
|
||||
---
|
||||
|
||||
## D1. Interleaved paths defeat coalescing (correctness-adjacent, unbounded memory)
|
||||
|
||||
**Severity: high.** This is the one to fix first.
|
||||
|
||||
### Problem
|
||||
|
||||
`flush()` coalesces only **adjacent** ops. A producer that alternates between two
|
||||
paths therefore never coalesces anything, and the ops array grows with the number
|
||||
of writes rather than with the size of the value.
|
||||
|
||||
This is not hypothetical. It is exactly what bash does: it updates the output text
|
||||
and a byte counter on every chunk.
|
||||
|
||||
```ts
|
||||
const next = t.state.content[0].text + chunk;
|
||||
t.state.content[0].text = next.length > CAP ? next.slice(next.length - CAP) : next;
|
||||
t.state.truncation.totalBytes += chunk.length; // <- the other path
|
||||
```
|
||||
|
||||
### Evidence
|
||||
|
||||
50 KB window, 200-byte chunks, no flush in between:
|
||||
|
||||
| held-back writes | ops | bytes |
|
||||
| --- | --- | --- |
|
||||
| 1 | 3 | 0.3 KB |
|
||||
| 100 | 201 | 26 KB |
|
||||
| **1000** | **2001** | **264 KB** |
|
||||
|
||||
Without interleaving the same workload gives 1 op and 51 KB. The window is
|
||||
bounded at 50 KB; the ops describing it are not.
|
||||
|
||||
### Why it matters beyond memory
|
||||
|
||||
Any design where the sink **holds back** — rate limiting, backpressure, a slow
|
||||
consumer — depends on held-back writes costing nothing. They currently cost
|
||||
everything. See `../../04-tool-output/rate-limiting.md`.
|
||||
|
||||
### Candidate fix
|
||||
|
||||
Fold at **record** time, keyed by path rather than by adjacency: one slot per
|
||||
path holding at most two ops (`[s]`, `[d]`, `[t]`, `[a]`, or `[t, a]`), emitted in
|
||||
first-touch order.
|
||||
|
||||
A prototype reached **2 ops / 51 KB** on the same workload. It is not included
|
||||
here because it was not verified to the standard the rest of this unit meets.
|
||||
|
||||
**Four folding rules, each a specific claim about the vocabulary:**
|
||||
|
||||
1. A later `s` supersedes an earlier `s` on the same path — but **not** an earlier
|
||||
`d`, which must survive or the key changes position on reinsertion (§5.1).
|
||||
2. Consecutive `a` merge. Once the merged append is **strictly longer** than the
|
||||
value, the window has been sliding and the append is the more expensive
|
||||
spelling, so collapse to `s`. Strictly: while a string is still *growing*
|
||||
toward its cap the append equals the value, and collapsing there would resend
|
||||
the whole string every batch.
|
||||
3. `t` and `a` on one path **commute** — the truncate drops from the front, the
|
||||
append adds to the end — so a rolling window's `t,a,t,a,...` folds to one of
|
||||
each. Without this, pairwise merging never fires at all.
|
||||
4. `t` or `a` following a pending `s` changes nothing: the `s` already means "this
|
||||
path becomes its current value".
|
||||
|
||||
**The trap that prototype hit.** Slots emit in first-touch order, which loses the
|
||||
fact that a child write happened *before* a later parent write:
|
||||
|
||||
```
|
||||
script : a={x:1}; a.b=99; a={c:2}
|
||||
producer: {"a":{"c":2}}
|
||||
replica : {"a":{"c":2,"b":99}} <- WRONG
|
||||
```
|
||||
|
||||
A set or delete must invalidate every pending slot **below** its path, at record
|
||||
time. The existing `verify-dead` style test passed 1190 sequences while this was
|
||||
broken, because its generator only nested one level. Any test for this must
|
||||
generate genuinely nested paths with parent overwrites interleaved between child
|
||||
writes.
|
||||
|
||||
---
|
||||
|
||||
## D2. `overlap()` was 94.5% of prototype tracker time — closed
|
||||
|
||||
**Historical severity: high. Production decision:** do not add an explicit append/truncate API. The prototype profile was dominated by a slow `startsWith` path; production uses a flattened-slice comparison and measured 17.8–18.7 µs per 200 KB assistant append flush and 2.43–2.46 µs per 50 KB rolling-window flush locally. See [`append-decision.md`](append-decision.md). The original evidence follows.
|
||||
|
||||
### Problem
|
||||
|
||||
The tracker infers "you appended and evicted" by searching two 50 KB strings.
|
||||
Each new value produced by `next.slice(-CAP)` is a V8 `SlicedString` — a lazy
|
||||
substring — and `indexOf`/`endsWith` force it to **flatten**, copying 50 KB per
|
||||
call.
|
||||
|
||||
### Evidence
|
||||
|
||||
CPU profile of a 20 000-write rolling-window loop, node, flush at the end:
|
||||
|
||||
```
|
||||
94.5% overlap
|
||||
2.7% (anon)
|
||||
1.2% (garbage collector)
|
||||
0.5% set
|
||||
0.3% emit
|
||||
0.0% fold
|
||||
```
|
||||
|
||||
And the flattening is directly observable:
|
||||
|
||||
| `overlap()` on a 50 KB window | µs |
|
||||
| --- | --- |
|
||||
| the same two strings reused | 37 |
|
||||
| **a fresh sliced string each call** | **149** |
|
||||
|
||||
An earlier benchmark reported 29 µs because it reused two already-flattened
|
||||
strings 2000 times — a case that never occurs in the real loop. **Do not benchmark
|
||||
string operations against a reused fixture.**
|
||||
|
||||
### Candidate fix
|
||||
|
||||
Do not infer what the producer already knows. `ToolOutput` performs the windowing,
|
||||
so it can emit the ops directly rather than assigning a new string and having the
|
||||
tracker search for the difference:
|
||||
|
||||
```ts
|
||||
out.append(chunk); // -> ["t", path, dropped], ["a", path, chunk]
|
||||
```
|
||||
|
||||
Two integers — chars appended, chars evicted — describe everything that happened
|
||||
between any two observations of a bounded window. A prototype of this shape held
|
||||
**exactly the window** in memory under every load, with no ops array, no overlap
|
||||
detection and no folding rules at all.
|
||||
|
||||
This does **not** remove `overlap`: it stays for the general case where a producer
|
||||
assigns a whole new string and the delta genuinely has to be discovered. It
|
||||
removes it from the hot path.
|
||||
|
||||
### Do not
|
||||
|
||||
Do not "optimise" `overlap` itself before doing the above. Its algorithm is
|
||||
already correct and bounded (`maxOverlapScan`); the cost is string flattening,
|
||||
which no probe strategy avoids.
|
||||
|
||||
---
|
||||
|
||||
## D3. Nested-path proxy overhead is next, and currently invisible
|
||||
|
||||
**Severity: unknown — measure before acting.**
|
||||
|
||||
`overlap` dominates so completely that nothing else shows in a profile. But two
|
||||
benchmarks disagree in a way that points at path depth:
|
||||
|
||||
| | µs per write |
|
||||
| --- | --- |
|
||||
| top-level path (`{ text }`) | 63 |
|
||||
| nested path (`{ content: [{ text }] }`) | 244 |
|
||||
|
||||
4x for depth alone. Each nested property access wraps a child proxy and builds a
|
||||
cache key by joining the path, so a read of `content[0].text` allocates on every
|
||||
access.
|
||||
|
||||
**Re-profile only if production delta cost becomes material.** This may be the next bottleneck or it may be
|
||||
nothing; the current numbers cannot distinguish them because `overlap` swamps both.
|
||||
|
||||
---
|
||||
|
||||
## D4. Things measured and found NOT worth doing
|
||||
|
||||
Recorded so nobody repeats the work. All were under **1.5% combined** in the
|
||||
profile above.
|
||||
|
||||
**A path trie replacing `JSON.stringify(path)` keys.** Genuinely faster in
|
||||
isolation — 6x on the `LaneSnapshot` shape, 93x at 500 distinct paths — because
|
||||
stringify cost scales with path depth while a trie walk is one `Map.get` per
|
||||
segment. But the whole slot layer is 0.5% of runtime. Build it only if a current production profile says so.
|
||||
|
||||
**Trie as storage vs trie as index.** If a trie is ever built, it should map path
|
||||
to a **position in the ops array**, not hold the ops. Holding them loses first-touch
|
||||
ordering and forces a sort at flush to rebuild it. Measured difference: none
|
||||
(1.4x / 0.9x / 1.0x). The argument is structural, not performance.
|
||||
|
||||
**A `maxDepth` heuristic to skip descendant scanning.** Rejected: the flat map was
|
||||
losing on the common case too, so the heuristic would have hidden a real cost
|
||||
rather than removed it.
|
||||
|
||||
**Eager string concatenation in the `a`+`a` merge.** Real (it rebuilds a growing
|
||||
string per write) but worth ~5%. Compare lengths and materialise only when
|
||||
emitting.
|
||||
|
||||
---
|
||||
|
||||
## D5. Measurement errors made while producing these numbers
|
||||
|
||||
Listed because each one produced a confident wrong conclusion, and the same traps
|
||||
are still there.
|
||||
|
||||
- **Benchmarking through `tsx`** — inflated everything 2.6x.
|
||||
- **Reusing a flattened string** in a string benchmark — hid the real cost 4x.
|
||||
- **Repetitive fixtures** (`"x".repeat(n)`) — overlap detection hits its candidate
|
||||
bound and gives up, so the benchmark measures the fallback path, not the real one.
|
||||
Use varied text.
|
||||
- **Measuring a cache hit as if it were work** — `Markdown.render()` returns
|
||||
`cachedLines` when text and width are unchanged, so a repeat-the-same-call loop
|
||||
measured 0.065 ms for something that costs 0.86 ms.
|
||||
- **Reasoning about complexity while the constant dominated** — twice.
|
||||
- **Measuring heap instead of retained size** — a fixture generating 800 x 1 MB
|
||||
strings reported 40 MB of "growth" that was its own garbage.
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
# Decision: no explicit text append/truncate API
|
||||
|
||||
**Status: closed.** Do not add `appendText`, explicit truncate, a text-specific dirty-node kind, or proxy-to-tracker lookup machinery.
|
||||
|
||||
## Why it was considered
|
||||
|
||||
The tracker marks dirty paths and compares them with its accepted baseline at `flush()`. For a growing or rolling string, confirming the relationship can scan the retained string even though the producer already knows it appended or evicted text.
|
||||
|
||||
The original benchmark made this look expensive because its append fast path used `after.startsWith(before)`. V8 walked the producer's cons string character by character. Production Chord now uses:
|
||||
|
||||
```ts
|
||||
if (after.length > before.length && after.slice(0, before.length) === before) {
|
||||
// emit append
|
||||
}
|
||||
```
|
||||
|
||||
The slice flattens once and comparison uses the native string path. Flush-time dirty tracking also removed the prototype's retained-op amplification.
|
||||
|
||||
## Local confirmation
|
||||
|
||||
Measured 2026-09-01 against `origin/dev` at `1a7bc80e7`, using `packages/chord/src/delta/index.ts` directly under Node 26.0.0 on an Apple M5 Max. Each workload used 3,000 warmups followed by 11 samples of 10,000 mutation-plus-flush iterations; a second process reproduced the result.
|
||||
|
||||
| Workload | Median µs/flush, run 1 | Median µs/flush, run 2 |
|
||||
| --- | ---: | ---: |
|
||||
| 200 KB assistant string, append 8 characters | 18.68 | 17.81 |
|
||||
| 50 KB rolling window, slide 32 varied characters | 2.46 | 2.43 |
|
||||
| Transcript push, one small entry | 0.74 | 0.78 |
|
||||
|
||||
The assistant tracker started with 200,000 varied characters, appended `" abcdef"`, and flushed each append. The rolling tracker started with 50,000 varied characters, then assigned `text.slice(32) + chunk` using a distinct 32-character `chunk:<base36 index>:durable-stream` value on every flush; every flush was asserted to emit `t` + `a`. The transcript tracker pushed `{ id: "e<index>", text: "message <index>" }` and asserted one `p` per flush. Setup and garbage collection were outside each timed loop.
|
||||
|
||||
At 100 assistant updates per second, the growing-string case consumes about 1.8 ms/s, approximately 0.18% of one core. The measured rolling-window cost is itself too small to justify the abandoned explicit API.
|
||||
|
||||
## Why the API is rejected
|
||||
|
||||
The prototype required a text-specific dirty-node state, proxy-to-tracker lookup, repeated-append/drop folding, and interaction rules for later whole-value replacement. It produced subtle silent failures: one implementation fell through to the ordinary differ while tests still passed, and another drifted when drops reached into earlier appends.
|
||||
|
||||
That complexity does not justify saving microseconds below the surrounding replication, isolation, and rendering costs. Keep ordinary string mutation and the generic Chord op vocabulary.
|
||||
|
||||
Reopen only when a production profile shows delta flush time is a meaningful fraction of a real workload. Re-measure the generic fast path first; a regression there is cheaper and safer to fix than adding a producer-specific API.
|
||||
+826
@@ -0,0 +1,826 @@
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// harness/delta — intent-recording change tracking over plain JSON.
|
||||
//
|
||||
// Depends on nothing else in the harness. Session storage, the runtime and the
|
||||
// facet host consume it; keep the arrows pointing that way.
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
export type JsonValue = null | boolean | number | string | JsonValue[] | { [k: string]: JsonValue };
|
||||
|
||||
export type Seg = string | number;
|
||||
export type Path = readonly Seg[];
|
||||
export type NonEmptyPath = readonly [Seg, ...Seg[]];
|
||||
|
||||
/** A path inline, or an id assigned by the encoder on second use. */
|
||||
export type PathRef<P extends Path = Path> = P | number;
|
||||
|
||||
/**
|
||||
* Tuples are the form — in memory, on the wire, on disk.
|
||||
*
|
||||
* `r` is the ONLY op that replaces a whole value. `s`/`d`/`a`/`t` cannot target
|
||||
* the root: the type forbids it. `p` may, and only because a tracked value can
|
||||
* itself be an array — but a `p` that replaces its entire target is normalised to
|
||||
* `r`/`s` at record time, so a root `p` is always a partial modification.
|
||||
*
|
||||
* `Op` knows nothing about the path dictionary. Interning, id references and
|
||||
* omitted paths live in `WireOp` and exist only between `encode` and `decode`.
|
||||
*/
|
||||
export type Op =
|
||||
| readonly ["r", JsonValue]
|
||||
| readonly ["s", NonEmptyPath, JsonValue]
|
||||
| readonly ["d", NonEmptyPath]
|
||||
| readonly ["a", NonEmptyPath, string]
|
||||
| readonly ["t", NonEmptyPath, number]
|
||||
| readonly ["p", Path, number, number, JsonValue[]];
|
||||
|
||||
/**
|
||||
* What crosses a boundary. Adds two compressions and nothing else:
|
||||
*
|
||||
* ["#", id, path] defines an id, emitted on a path's SECOND use
|
||||
* a numeric PathRef references a previously defined id
|
||||
* a shortened tuple reuses the previous op's path; arity disambiguates
|
||||
*
|
||||
* ["r", value] carries no path, so it encodes to itself — which is why isBase
|
||||
* works unchanged on either vocabulary.
|
||||
*/
|
||||
export type WireOp =
|
||||
| readonly ["r", JsonValue]
|
||||
| readonly ["s", PathRef<NonEmptyPath>, JsonValue] | readonly ["s", JsonValue]
|
||||
| readonly ["d", PathRef<NonEmptyPath>] | readonly ["d"]
|
||||
| readonly ["a", PathRef<NonEmptyPath>, string] | readonly ["a", string]
|
||||
| readonly ["t", PathRef<NonEmptyPath>, number] | readonly ["t", number]
|
||||
| readonly ["p", PathRef, number, number, JsonValue[]]
|
||||
| readonly ["p", number, number, JsonValue[]]
|
||||
| readonly ["#", number, Path];
|
||||
|
||||
// ─── Classification ──────────────────────────────────────────────────────────
|
||||
|
||||
export const isReplace = (op: Op | WireOp): boolean => op[0] === "r";
|
||||
|
||||
/**
|
||||
* A batch begins with a replacement. Flush guarantees `r` is at index 0 or absent,
|
||||
* so this is exact rather than a heuristic.
|
||||
*/
|
||||
export const isBase = (ops: readonly (Op | WireOp)[]): boolean =>
|
||||
ops.length > 0 && ops[0]![0] === "r";
|
||||
|
||||
// ─── Overlap ─────────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Longest suffix of `a` that is a prefix of `b`. Probes with indexOf and verifies
|
||||
* with endsWith, so the hot loops are native. A hand-written KMP is
|
||||
* asymptotically equivalent and ~47x slower in practice.
|
||||
*
|
||||
* Always correct: the returned n satisfies a.endsWith(b.slice(0, n)).
|
||||
*/
|
||||
export function overlap(a: string, b: string, scan: number, probe = 64, maxCandidates = 8): number {
|
||||
if (a.length === 0 || b.length === 0 || scan === 0) return 0;
|
||||
const tail = a.length > scan ? a.slice(a.length - scan) : a;
|
||||
|
||||
// A probe of length h can only find overlaps of at least h — the head must
|
||||
// actually occur in `a`. So try a long head first (few candidates, and it
|
||||
// catches the large overlaps a rolling window produces), then fall back to one
|
||||
// character, which finds any overlap at the cost of more candidates.
|
||||
//
|
||||
// Candidates are bounded because repetitive output — a build log, or any run of
|
||||
// one character — makes a long head match at thousands of positions, each
|
||||
// costing a full endsWith. Giving up returns 0, which emits a set: larger, never
|
||||
// wrong. Unbounded, a 50 KB window of one repeated character took 4.7 s per
|
||||
// 2000 slides against 90 ms bounded.
|
||||
for (const h of [Math.min(probe, b.length), 1]) {
|
||||
const head = b.slice(0, h);
|
||||
let tried = 0;
|
||||
for (let k = tail.indexOf(head); k !== -1; k = tail.indexOf(head, k + 1)) {
|
||||
if (++tried > maxCandidates) break;
|
||||
const n = tail.length - k;
|
||||
if (n <= b.length && tail.endsWith(b.slice(0, n))) return n;
|
||||
}
|
||||
if (h === 1) break;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
// ─── Tracker ─────────────────────────────────────────────────────────────────
|
||||
|
||||
export interface TrackerOptions {
|
||||
maxOverlapScan?: number;
|
||||
coalesce?: boolean;
|
||||
}
|
||||
|
||||
export interface Tracker<T extends object> {
|
||||
/**
|
||||
* Mutate this. Plain TS: assignment, push, splice, delete, all of it.
|
||||
*
|
||||
* Assigning to it replaces the whole value: emits `["r", next]` and discards
|
||||
* ops recorded before it, which describe a value that no longer exists.
|
||||
*
|
||||
* `tracker.state = tracker.target` is therefore a checkpoint — it forces a base
|
||||
* batch without changing anything. Discarding the pending ops is correct
|
||||
* because the proxy mutates the target directly, so the value already carries
|
||||
* them.
|
||||
*/
|
||||
state: T;
|
||||
/** The untracked backing object. Reads only — writing here emits nothing. */
|
||||
readonly target: T;
|
||||
flush(): Op[];
|
||||
/**
|
||||
* Clear pending ops and make the next flush a base batch, without changing
|
||||
* the value.
|
||||
*
|
||||
* Recovery replays from the last base batch, so a producer that wants "at most
|
||||
* N batches to replay" calls this every N. The pending ops are already
|
||||
* reflected in the value — the proxy mutates the target directly — so
|
||||
* discarding them loses nothing.
|
||||
*/
|
||||
rebase(): void;
|
||||
/** Discard without emitting. */
|
||||
discard(): void;
|
||||
readonly dirty: boolean;
|
||||
}
|
||||
|
||||
const isObj = (v: unknown): v is object => v !== null && typeof v === "object";
|
||||
const clone = <T>(v: T): T => (isObj(v) ? (structuredClone(v) as T) : v);
|
||||
const INDEX = /^(?:0|[1-9]\d*)$/;
|
||||
const norm = (t: object, k: string | symbol): Seg | symbol =>
|
||||
typeof k === "symbol" ? k : Array.isArray(t) && INDEX.test(k) ? Number(k) : k;
|
||||
const MUTATORS = new Set(["push", "pop", "shift", "unshift", "splice", "sort", "reverse", "fill", "copyWithin"]);
|
||||
|
||||
export function track<T extends object>(root: T, options: TrackerOptions = {}): Tracker<T> {
|
||||
const scan = options.maxOverlapScan ?? 65_536;
|
||||
const doCoalesce = options.coalesce ?? true;
|
||||
|
||||
let ops: Op[] = [];
|
||||
|
||||
|
||||
const emit = (op: Op) => {
|
||||
ops.push(op);
|
||||
};
|
||||
|
||||
/**
|
||||
* Reject reserved keys at record time, not just on apply.
|
||||
*
|
||||
* `JSON.parse('{"__proto__":{}}')` yields an OWN property, so a producer can
|
||||
* hold state containing one without doing anything unusual. The tracker would
|
||||
* then emit a path the applier refuses — state that cannot be replicated. Fail
|
||||
* where the mistake is, at the write.
|
||||
*/
|
||||
const guard = (seg: Seg | symbol): Seg => {
|
||||
if (typeof seg === "symbol") throw new UnsafePathError(String(seg));
|
||||
if (typeof seg === "string" && FORBIDDEN.has(seg)) throw new UnsafePathError(seg);
|
||||
return seg;
|
||||
};
|
||||
|
||||
/**
|
||||
* A splice covering the whole target is a replacement written the long way.
|
||||
* Only the interception knows the pre-splice length, so this is a record-time
|
||||
* rewrite, not something flush can recover.
|
||||
*/
|
||||
const spliceOrReplace = (path: Path, before: number, index: number, remove: number, items: JsonValue[]) => {
|
||||
if (index === 0 && remove === before) {
|
||||
if (path.length === 0) emit(["r", items]);
|
||||
else emit(["s", path as NonEmptyPath, items]);
|
||||
return;
|
||||
}
|
||||
emit(["p", [...path], index, remove, items]);
|
||||
};
|
||||
|
||||
const cache = new WeakMap<object, Map<string, object>>();
|
||||
|
||||
const wrap = <V extends object>(obj: V, path: Path): V => {
|
||||
let byPath = cache.get(obj);
|
||||
if (byPath === undefined) cache.set(obj, (byPath = new Map()));
|
||||
const cacheKey = path.join("\u0000");
|
||||
const hit = byPath.get(cacheKey);
|
||||
if (hit !== undefined) return hit as V;
|
||||
|
||||
const proxy = new Proxy(obj, {
|
||||
get(t, k, r) {
|
||||
if (Array.isArray(t) && typeof k === "string" && MUTATORS.has(k)) {
|
||||
return (...args: unknown[]) => {
|
||||
const before = t.length;
|
||||
const result = (Array.prototype[k as "push"] as Function).apply(t, args);
|
||||
const items = (xs: unknown[]) => xs.map(clone) as JsonValue[];
|
||||
switch (k) {
|
||||
case "push": spliceOrReplace(path, before, before, 0, items(args)); break;
|
||||
case "unshift": spliceOrReplace(path, before, 0, 0, items(args)); break;
|
||||
case "pop": if (before > 0) spliceOrReplace(path, before, before - 1, 1, []); break;
|
||||
case "shift": if (before > 0) spliceOrReplace(path, before, 0, 1, []); break;
|
||||
case "splice": {
|
||||
const raw = Number(args[0] ?? 0);
|
||||
const index = raw < 0 ? Math.max(0, before + raw) : Math.min(raw, before);
|
||||
const remove = args.length < 2 ? before - index : Math.max(0, Math.min(Number(args[1]), before - index));
|
||||
spliceOrReplace(path, before, index, remove, items(args.slice(2)));
|
||||
break;
|
||||
}
|
||||
// sort/reverse/fill/copyWithin permute wholesale; intent is not a splice.
|
||||
default:
|
||||
if (path.length === 0) emit(["r", clone(t) as JsonValue]);
|
||||
else emit(["s", path as NonEmptyPath, clone(t) as JsonValue]);
|
||||
}
|
||||
return result;
|
||||
};
|
||||
}
|
||||
const v = Reflect.get(t, k, r);
|
||||
return isObj(v) ? wrap(v, [...path, guard(norm(t, k))]) : v;
|
||||
},
|
||||
|
||||
set(t, k, v, r) {
|
||||
// `length` is a real mutation but not a document location.
|
||||
if (Array.isArray(t) && k === "length") {
|
||||
const before = t.length;
|
||||
const next = Number(v);
|
||||
if (next < before) spliceOrReplace(path, before, next, before - next, []);
|
||||
else if (next > before) emit(["p", [...path], before, 0, Array(next - before).fill(null)]);
|
||||
return Reflect.set(t, k, v, r);
|
||||
}
|
||||
|
||||
const at = [...path, guard(norm(t, k))] as unknown as NonEmptyPath;
|
||||
|
||||
if (v === undefined) { // JSON has no undefined
|
||||
emit(["d", at]);
|
||||
return Reflect.deleteProperty(t, k);
|
||||
}
|
||||
|
||||
const prev = (t as Record<string | symbol, unknown>)[k];
|
||||
if (typeof prev === "string" && typeof v === "string") {
|
||||
if (prev === v) return true;
|
||||
if (v.length > prev.length && v.startsWith(prev)) {
|
||||
emit(["a", at, v.slice(prev.length)]); // fast path, no probe
|
||||
} else {
|
||||
const ov = overlap(prev, v, scan);
|
||||
if (ov === 0) emit(["s", at, v]);
|
||||
else {
|
||||
emit(["t", at, prev.length - ov]);
|
||||
if (v.length > ov) emit(["a", at, v.slice(ov)]);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
assertJsonValue(v);
|
||||
emit(["s", at, clone(v) as JsonValue]);
|
||||
}
|
||||
return Reflect.set(t, k, v, r);
|
||||
},
|
||||
|
||||
deleteProperty(t, k) {
|
||||
emit(["d", [...path, guard(norm(t, k))] as unknown as NonEmptyPath]);
|
||||
return Reflect.deleteProperty(t, k);
|
||||
},
|
||||
});
|
||||
|
||||
byPath.set(cacheKey, proxy);
|
||||
return proxy as V;
|
||||
};
|
||||
|
||||
let state = wrap(root, []);
|
||||
// The first flush is always a base batch. A consumer starts with nothing, so
|
||||
// a stream that opens with deltas has nothing to apply them to — and that
|
||||
// fails at runtime, in the consumer, rather than where the mistake is.
|
||||
let forceBase = true;
|
||||
|
||||
return {
|
||||
get state() { return state; },
|
||||
// A setter, so the obvious thing works. Without it `tracker.state = next`
|
||||
// silently swaps the proxy for a plain object and stops tracking.
|
||||
set state(next: T) {
|
||||
ops = [];
|
||||
root = next;
|
||||
state = wrap(root, []);
|
||||
forceBase = true;
|
||||
},
|
||||
get target() { return root; },
|
||||
|
||||
// Same effect as assigning the current value back, minus the re-wrap: the
|
||||
// value has not changed, so the proxy cache is still valid.
|
||||
rebase() { ops = []; forceBase = true; },
|
||||
|
||||
|
||||
get dirty() { return ops.length > 0; },
|
||||
discard() { ops = []; },
|
||||
flush() {
|
||||
if (forceBase) {
|
||||
forceBase = false;
|
||||
ops = [];
|
||||
return [["r", clone(root) as JsonValue]];
|
||||
}
|
||||
if (ops.length === 0) return [];
|
||||
let out = ops;
|
||||
ops = [];
|
||||
|
||||
// Everything before a replacement is dead.
|
||||
const lastReplace = out.findLastIndex(isReplace);
|
||||
if (lastReplace > 0) out = out.slice(lastReplace);
|
||||
|
||||
if (doCoalesce) out = coalesce(dropDead(out));
|
||||
|
||||
return out;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
const samePath = (a: Op, b: Op): boolean => {
|
||||
const pa = pathOf(a);
|
||||
const pb = pathOf(b);
|
||||
if (pa === undefined || pb === undefined) return false;
|
||||
return JSON.stringify(pa) === JSON.stringify(pb);
|
||||
};
|
||||
|
||||
function pathOf(op: Op): Path | undefined {
|
||||
return op[0] === "r" ? undefined : op[1];
|
||||
}
|
||||
|
||||
/**
|
||||
* Drop ops that a later op makes unreachable.
|
||||
*
|
||||
* Ops apply in order, so an op is dead if a later one overwrites the whole
|
||||
* subtree it lives in. Only `s`, `d` and `r` do that — `a`, `t` and `p` modify
|
||||
* what is already there, so they never dominate.
|
||||
*
|
||||
* O(n * d): walk backwards keeping a map of overwritten paths, and for each op
|
||||
* test its own prefixes against that map. `d` is the path depth, which is small
|
||||
* and bounded. The obvious formulation — compare each op against every
|
||||
* dominator — is O(n^2 * d) and degrades on exactly the wide-flush input this
|
||||
* pass exists to clean up. Measured: 12-14 microseconds per op, flat from 100 to
|
||||
* 5000 ops, and 10001 redundant writes collapse to 2 emitted ops.
|
||||
*
|
||||
* OBJECT KEY ORDER IS NOT PRESERVED. A producer that does
|
||||
* `set c; set a; delete c; set c` leaves `c` last, because deleting and
|
||||
* reinserting moves it. The replica receives only the surviving `set c`, applies
|
||||
* it to a base where `c` never existed, and `c` lands first. Values are
|
||||
* identical — verified over 1823 randomised sequences with zero value
|
||||
* mismatches — but the serialised forms can differ.
|
||||
*
|
||||
* This is fine because nothing compares a replica against its producer: the
|
||||
* consumer holds the state and nobody diffs it back. The one visible case is a
|
||||
* consumer that iterates keys for display, where two views could order rows
|
||||
* differently. It requires a delete-and-recreate of the same key within one
|
||||
* flush. Do not hash or content-address a replicated value.
|
||||
*/
|
||||
function dropDead(ops: Op[]): Op[] {
|
||||
const overwritten = new Map<string, "s" | "d">();
|
||||
const kept: Op[] = [];
|
||||
|
||||
for (let i = ops.length - 1; i >= 0; i--) {
|
||||
const op = ops[i]!;
|
||||
if (op[0] === "r") {
|
||||
kept.push(op);
|
||||
overwritten.clear();
|
||||
overwritten.set("", "s"); // the root is now overwritten
|
||||
continue;
|
||||
}
|
||||
|
||||
const path = opPath(op);
|
||||
if (path === undefined) { kept.push(op); continue; } // arity form, or an interned id
|
||||
|
||||
let dead = overwritten.has("");
|
||||
let key = "";
|
||||
for (let k = 0; k < path.length && !dead; k++) {
|
||||
key += `\u0000${String(path[k])}`;
|
||||
const kind = overwritten.get(key);
|
||||
if (kind === undefined) continue;
|
||||
// A `d` at exactly a later `s`'s path is NOT dead. Dropping it leaves the
|
||||
// key in its original slot, while the producer's delete-then-set moved it
|
||||
// to the end: same value, different serialisation.
|
||||
if (k === path.length - 1 && op[0] === "d" && kind === "s") break;
|
||||
dead = true;
|
||||
}
|
||||
if (dead) continue;
|
||||
|
||||
kept.push(op);
|
||||
if (op[0] === "s" || op[0] === "d") overwritten.set(key, op[0]);
|
||||
}
|
||||
return kept.reverse();
|
||||
}
|
||||
|
||||
/** Total, because `Op` paths are always inline. */
|
||||
function opPath(op: Op): Path | undefined {
|
||||
return op[0] === "r" ? undefined : op[1];
|
||||
}
|
||||
|
||||
/** Merge adjacent ops describing the same path. Lossless. */
|
||||
function coalesce(ops: Op[]): Op[] {
|
||||
const out: Op[] = [];
|
||||
for (const op of ops) {
|
||||
const last = out[out.length - 1];
|
||||
if (last !== undefined && samePath(last, op)) {
|
||||
if (last[0] === "a" && op[0] === "a") {
|
||||
out[out.length - 1] = ["a", last[1], `${last[2]}${op[2]}`];
|
||||
continue;
|
||||
}
|
||||
if (last[0] === "s" && op[0] === "s") { // later write wins
|
||||
out[out.length - 1] = op;
|
||||
continue;
|
||||
}
|
||||
if (last[0] === "s" && op[0] === "a" && typeof last[2] === "string") {
|
||||
out[out.length - 1] = ["s", last[1], `${last[2]}${op[2]}`];
|
||||
continue;
|
||||
}
|
||||
}
|
||||
out.push(op);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// ─── Path safety ─────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Segments that reach the prototype chain.
|
||||
*
|
||||
* `JSON.parse` is safe on its own — it makes `__proto__` an own property. What is
|
||||
* not safe is `parent[key] = value`, which is exactly what an applier does, and
|
||||
* paths are data: `["s", ["__proto__", "isAdmin"], true]` pollutes
|
||||
* `Object.prototype` for the whole process.
|
||||
*
|
||||
* Ops arrive from a facet, a plugin compartment, or a tool whose details may echo
|
||||
* model output, so none of it is trusted input.
|
||||
*/
|
||||
const FORBIDDEN = new Set(["__proto__", "constructor", "prototype"]);
|
||||
|
||||
export class UnsafePathError extends Error {
|
||||
// Not a parameter property: Node's --experimental-strip-types rejects those,
|
||||
// and these files are meant to run under it directly.
|
||||
readonly segment: Seg;
|
||||
constructor(segment: Seg) {
|
||||
super(`unsafe path segment: ${String(segment)}`);
|
||||
this.segment = segment;
|
||||
this.name = "UnsafePathError";
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Rejects anything JSON.stringify would not round-trip.
|
||||
*
|
||||
* structuredClone is NOT a JSON check: it clones a Map happily, and the op then
|
||||
* carries `{}` while the producer keeps a real Map — producer and replica diverge
|
||||
* silently, forever.
|
||||
*/
|
||||
export function assertJsonValue(value: unknown, seen = new Set<unknown>()): void {
|
||||
if (value === null) return;
|
||||
const t = typeof value;
|
||||
if (t === "string" || t === "boolean") return;
|
||||
if (t === "number") {
|
||||
if (!Number.isFinite(value as number)) throw new TypeError("non-finite number is not JSON");
|
||||
return;
|
||||
}
|
||||
if (t !== "object") throw new TypeError(`${t} is not a JsonValue`);
|
||||
if (seen.has(value)) throw new TypeError("cyclic value is not JSON");
|
||||
seen.add(value);
|
||||
if (Array.isArray(value)) {
|
||||
for (const v of value) assertJsonValue(v, seen);
|
||||
} else {
|
||||
const proto = Object.getPrototypeOf(value);
|
||||
if (proto !== Object.prototype && proto !== null) {
|
||||
throw new TypeError(`${(value as object).constructor?.name ?? "object"} is not a JsonValue`);
|
||||
}
|
||||
if (Object.getOwnPropertySymbols(value).length > 0) {
|
||||
throw new TypeError("symbol keys are not JSON");
|
||||
}
|
||||
for (const v of Object.values(value as object)) assertJsonValue(v, seen);
|
||||
}
|
||||
seen.delete(value);
|
||||
}
|
||||
|
||||
/**
|
||||
* Verb, arity and payload shape for a **decoded** op: paths inline, no `#`, no
|
||||
* short forms. `apply` uses this.
|
||||
*
|
||||
* Validating `Op` against the wire grammar would be laxer than the type: a
|
||||
* two-element `["s", value]` would pass, and `apply` would then read the value as
|
||||
* a path. Each vocabulary gets the validator that matches it.
|
||||
*/
|
||||
export function assertValidOp(op: unknown): void {
|
||||
if (!Array.isArray(op) || op.length === 0) throw new TypeError("op is not a tuple");
|
||||
switch (op[0]) {
|
||||
case "r": if (op.length !== 2) throw new TypeError("r arity"); assertJsonValue(op[1]); return;
|
||||
case "s": if (op.length !== 3) throw new TypeError("s arity"); assertPathArg(op[1]); assertJsonValue(op[2]); return;
|
||||
case "d": if (op.length !== 2) throw new TypeError("d arity"); assertPathArg(op[1]); return;
|
||||
case "a": if (op.length !== 3 || typeof op[2] !== "string") throw new TypeError("a shape"); assertPathArg(op[1]); return;
|
||||
case "t": if (op.length !== 3 || !Number.isInteger(op[2])) throw new TypeError("t shape"); assertPathArg(op[1]); return;
|
||||
case "p": {
|
||||
if (op.length !== 5) throw new TypeError("p arity");
|
||||
assertPathArg(op[1]);
|
||||
if (!Number.isInteger(op[2]) || op[2] < 0) throw new TypeError("p index");
|
||||
if (!Number.isInteger(op[3]) || op[3] < 0) throw new TypeError("p remove");
|
||||
if (!Array.isArray(op[4])) throw new TypeError("p items");
|
||||
for (const v of op[4]) assertJsonValue(v);
|
||||
return;
|
||||
}
|
||||
// Silently skipping an unknown verb is how a newer producer's op vanishes.
|
||||
default: throw new TypeError(`unknown op verb: ${String(op[0])}`);
|
||||
}
|
||||
}
|
||||
|
||||
function assertPathArg(p: unknown): void {
|
||||
if (!Array.isArray(p)) throw new TypeError("path is not an array");
|
||||
assertSafePath(p as Path);
|
||||
}
|
||||
|
||||
/** The same, for the wire grammar: ids and short forms are legal here. */
|
||||
export function assertValidWireOp(op: unknown): void {
|
||||
if (!Array.isArray(op) || op.length === 0) throw new TypeError("op is not a tuple");
|
||||
const [verb] = op as unknown[];
|
||||
const okRef = (r: unknown): void => {
|
||||
if (typeof r === "number") {
|
||||
if (!Number.isInteger(r) || r < 0) throw new TypeError("bad path id");
|
||||
return;
|
||||
}
|
||||
// A string is not a path. Unchecked, `"a".slice(0, -1)` is `""`, so it
|
||||
// resolves to the ROOT and writes there — a path that is not a path, accepted.
|
||||
if (!Array.isArray(r)) throw new TypeError("path is not an array");
|
||||
assertSafePath(r as Path);
|
||||
};
|
||||
switch (verb) {
|
||||
case "r": if (op.length !== 2) throw new TypeError("r arity"); assertJsonValue(op[1]); return;
|
||||
case "s": if (op.length === 3) { okRef(op[1]); assertJsonValue(op[2]); }
|
||||
else if (op.length === 2) assertJsonValue(op[1]);
|
||||
else throw new TypeError("s arity"); return;
|
||||
case "d": if (op.length === 2) okRef(op[1]); else if (op.length !== 1) throw new TypeError("d arity"); return;
|
||||
case "a": if (op.length === 3) { okRef(op[1]); if (typeof op[2] !== "string") throw new TypeError("a value"); }
|
||||
else if (op.length === 2) { if (typeof op[1] !== "string") throw new TypeError("a value"); }
|
||||
else throw new TypeError("a arity"); return;
|
||||
case "t": if (op.length === 3) { okRef(op[1]); if (!Number.isInteger(op[2])) throw new TypeError("t count"); }
|
||||
else if (op.length === 2) { if (!Number.isInteger(op[1])) throw new TypeError("t count"); }
|
||||
else throw new TypeError("t arity"); return;
|
||||
case "p": {
|
||||
const [i, r, items] = op.length === 5 ? [op[2], op[3], op[4]] : op.length === 4 ? [op[1], op[2], op[3]] : [];
|
||||
if (items === undefined) throw new TypeError("p arity");
|
||||
if (op.length === 5) okRef(op[1]);
|
||||
if (!Number.isInteger(i) || (i as number) < 0) throw new TypeError("p index");
|
||||
if (!Number.isInteger(r) || (r as number) < 0) throw new TypeError("p remove");
|
||||
if (!Array.isArray(items)) throw new TypeError("p items");
|
||||
for (const v of items) assertJsonValue(v);
|
||||
return;
|
||||
}
|
||||
case "#": {
|
||||
if (op.length !== 3 || !Number.isInteger(op[1]) || !Array.isArray(op[2])) throw new TypeError("# shape");
|
||||
assertSafePath(op[2] as Path);
|
||||
return;
|
||||
}
|
||||
// Silently skipping an unknown verb is how a newer producer's op vanishes.
|
||||
default: throw new TypeError(`unknown op verb: ${String(verb)}`);
|
||||
}
|
||||
}
|
||||
|
||||
export function assertSafePath(path: Path): void {
|
||||
for (const seg of path) {
|
||||
if (typeof seg === "string") {
|
||||
if (FORBIDDEN.has(seg)) throw new UnsafePathError(seg);
|
||||
} else if (!Number.isInteger(seg) || seg < 0) {
|
||||
throw new UnsafePathError(seg);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* An index may address an existing element or append exactly one past the end.
|
||||
*
|
||||
* This is not an arbitrary cap — it is what keeps the value a `JsonValue`. A
|
||||
* sparse array does not survive a JSON round trip: holes serialise to `null` and
|
||||
* return as real properties, so `arr[7] = x` on a length-3 array already produces
|
||||
* state a replica cannot match. Rejecting the write is more honest than silently
|
||||
* diverging.
|
||||
*
|
||||
* It also removes the denial of service it would otherwise permit:
|
||||
* `["s", ["xs", 4294967290], 1]` allocates a 4.29-billion-entry array from one op.
|
||||
* Growth stays possible and stays proportional — the tracker already emits
|
||||
* `arr.length = n` as a splice of explicit nulls, whose op size grows with the
|
||||
* gap, so a large growth costs a large op rather than a small one.
|
||||
*/
|
||||
function assertIndexInRange(parent: readonly unknown[], index: number): void {
|
||||
if (index > parent.length) throw new UnsafePathError(index);
|
||||
}
|
||||
|
||||
// ─── Applier ─────────────────────────────────────────────────────────────────
|
||||
|
||||
export class PathError extends Error {
|
||||
readonly path: Path | number;
|
||||
constructor(path: Path | number) {
|
||||
super(`unresolvable path: ${JSON.stringify(path)}`);
|
||||
this.path = path;
|
||||
this.name = "PathError";
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply ops to a plain mutable value. Returns the value, because `r` replaces it
|
||||
* outright and cannot be done in place.
|
||||
*
|
||||
* Takes decoded ops. Path ids and omitted paths are a wire concern — run
|
||||
* `decode` first if the ops came from a boundary.
|
||||
*/
|
||||
export function apply<T>(target: T, ops: readonly Op[]): T {
|
||||
let root = target as unknown as JsonValue;
|
||||
|
||||
for (const op of ops) {
|
||||
assertValidOp(op);
|
||||
|
||||
if (op[0] === "r") {
|
||||
// Adopted, not copied. The consumer owns the batch it was handed.
|
||||
//
|
||||
// Fanning one batch out to several consumers in-process therefore makes
|
||||
// their replicas alias each other. That is an ownership rule, not a
|
||||
// defect: copy the batch at the fan-out point, or let each consumer
|
||||
// decode its own. A batch that crosses a real boundary is already
|
||||
// distinct, because serialisation produces fresh objects.
|
||||
root = op[1];
|
||||
continue;
|
||||
}
|
||||
|
||||
const path = op[1];
|
||||
assertSafePath(path);
|
||||
|
||||
if (op[0] === "p") {
|
||||
const target_ = path.length === 0 ? root : resolve(root, path);
|
||||
if (!Array.isArray(target_)) throw new PathError(path);
|
||||
target_.splice(op[2], op[3], ...op[4]);
|
||||
continue;
|
||||
}
|
||||
|
||||
// s/d/a/t can never target the root — the type forbids it.
|
||||
const parent = resolve(root, path.slice(0, -1)) as Record<Seg, JsonValue>;
|
||||
const key = path[path.length - 1]!;
|
||||
if (Array.isArray(parent) && typeof key === "number") assertIndexInRange(parent, key);
|
||||
// defineProperty rather than assignment: a setter inherited from the prototype
|
||||
// chain would otherwise run on write.
|
||||
const write = (value: JsonValue) => {
|
||||
Object.defineProperty(parent, key, { value, writable: true, enumerable: true, configurable: true });
|
||||
};
|
||||
const read = (): unknown => (Object.hasOwn(parent, key) ? parent[key] : undefined);
|
||||
switch (op[0]) {
|
||||
case "s": write(op[2]); break;
|
||||
case "d": Array.isArray(parent) ? (parent as unknown as JsonValue[]).splice(Number(key), 1) : delete parent[key]; break;
|
||||
case "a": write(`${String(read() ?? "")}${op[2]}`); break;
|
||||
case "t": write(String(read() ?? "").slice(op[2])); break;
|
||||
}
|
||||
}
|
||||
return root as unknown as T;
|
||||
}
|
||||
|
||||
function resolve(root: JsonValue, path: Path): JsonValue {
|
||||
let node: JsonValue = root;
|
||||
for (const seg of path) {
|
||||
if (!isObj(node)) throw new PathError(path);
|
||||
// Own properties only: an inherited getter must not run, and a walk must not
|
||||
// escape the value into the prototype chain.
|
||||
if (!Object.hasOwn(node, seg as PropertyKey)) throw new PathError(path);
|
||||
node = (node as Record<Seg, JsonValue>)[seg]!;
|
||||
}
|
||||
if (!isObj(node)) throw new PathError(path);
|
||||
return node;
|
||||
}
|
||||
|
||||
// ─── Codec ───────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// Path interning and arity omission, per `delta.md` §4. These live between the
|
||||
// tracker and a boundary; `Op` and `apply` know nothing about them.
|
||||
//
|
||||
// ONE PAIR PER STREAM. The table spans an entire subscription or file: a path
|
||||
// interned in batch 3 is referenced in batch 40. A second consumer that
|
||||
// subscribes at batch 40 has never seen the definition, so it needs its own
|
||||
// encoder — sharing one across consumers hands the late subscriber ids it cannot
|
||||
// resolve. A base batch does not rescue it, because `["r", value]` carries no
|
||||
// path refs and leaves the table empty.
|
||||
|
||||
const pathKey = (path: Path): string => path.join("\u0000");
|
||||
|
||||
export interface Encoder {
|
||||
encode(ops: readonly Op[]): WireOp[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Intern on SECOND use. A definition costs more than the path it replaces, so
|
||||
* interning on first use loses on the many paths written exactly once —
|
||||
* measured worse than no interning at all (`delta.md` §4.1).
|
||||
*/
|
||||
export function encoder(): Encoder {
|
||||
const seen = new Set<string>();
|
||||
const ids = new Map<string, number>();
|
||||
let nextId = 0;
|
||||
let previous: string | undefined; // last path in THIS batch
|
||||
|
||||
return {
|
||||
encode(ops) {
|
||||
// Arity omission is scoped to a batch. Letting it span batches would make
|
||||
// a batch's first op depend on the previous batch's last one, so a reader
|
||||
// that skips or reorders a batch decodes into the wrong path. Ids are the
|
||||
// only cross-batch state, and the dictionary makes those explicit.
|
||||
previous = undefined;
|
||||
const out: WireOp[] = [];
|
||||
for (const op of ops) {
|
||||
if (op[0] === "r") {
|
||||
out.push(op);
|
||||
// A base batch is a RECOVERY POINT: a reader replays from the last one
|
||||
// with a fresh decoder. So everything after it must be self-contained.
|
||||
// Keeping ids across a replacement emits references to definitions the
|
||||
// reader never saw — recovery fails with an unresolvable path id.
|
||||
seen.clear();
|
||||
ids.clear();
|
||||
nextId = 0;
|
||||
previous = undefined;
|
||||
continue;
|
||||
}
|
||||
const path = op[1];
|
||||
const key = pathKey(path);
|
||||
|
||||
// Same path as the previous op: drop the ref entirely.
|
||||
if (key === previous) {
|
||||
switch (op[0]) {
|
||||
case "s": out.push(["s", op[2]]); break;
|
||||
case "d": out.push(["d"]); break;
|
||||
case "a": out.push(["a", op[2]]); break;
|
||||
case "t": out.push(["t", op[2]]); break;
|
||||
case "p": out.push(["p", op[2], op[3], op[4]]); break;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
let ref: PathRef = path;
|
||||
const existing = ids.get(key);
|
||||
if (existing !== undefined) {
|
||||
ref = existing;
|
||||
} else if (seen.has(key)) {
|
||||
const id = nextId++;
|
||||
ids.set(key, id);
|
||||
out.push(["#", id, path]); // second use: define, then reference
|
||||
ref = id;
|
||||
} else {
|
||||
seen.add(key); // first use: inline
|
||||
}
|
||||
|
||||
switch (op[0]) {
|
||||
case "s": out.push(["s", ref as PathRef<NonEmptyPath>, op[2]]); break;
|
||||
case "d": out.push(["d", ref as PathRef<NonEmptyPath>]); break;
|
||||
case "a": out.push(["a", ref as PathRef<NonEmptyPath>, op[2]]); break;
|
||||
case "t": out.push(["t", ref as PathRef<NonEmptyPath>, op[2]]); break;
|
||||
case "p": out.push(["p", ref, op[2], op[3], op[4]]); break;
|
||||
}
|
||||
previous = key;
|
||||
}
|
||||
return out;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export interface Decoder {
|
||||
decode(wire: readonly WireOp[]): Op[];
|
||||
}
|
||||
|
||||
export function decoder(): Decoder {
|
||||
const paths = new Map<number, Path>();
|
||||
|
||||
return {
|
||||
decode(wire) {
|
||||
let previous: Path | undefined; // scoped to the batch, as in encode
|
||||
const out: Op[] = [];
|
||||
for (const op of wire) {
|
||||
assertValidWireOp(op);
|
||||
if (op[0] === "#") {
|
||||
assertSafePath(op[2]);
|
||||
paths.set(op[1], op[2]);
|
||||
continue;
|
||||
}
|
||||
if (op[0] === "r") {
|
||||
out.push(op);
|
||||
previous = undefined;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Arity tells us whether a ref is present: the short forms omit it.
|
||||
const short =
|
||||
(op[0] === "d" && op.length === 1) ||
|
||||
(op[0] !== "d" && op[0] !== "p" && op.length === 2) ||
|
||||
(op[0] === "p" && op.length === 4);
|
||||
|
||||
let path: Path;
|
||||
if (short) {
|
||||
if (previous === undefined) throw new PathError([]);
|
||||
path = previous;
|
||||
} else {
|
||||
const ref = op[1] as PathRef;
|
||||
if (typeof ref === "number") {
|
||||
const resolved = paths.get(ref);
|
||||
if (resolved === undefined) throw new PathError(ref);
|
||||
path = resolved;
|
||||
} else {
|
||||
path = ref;
|
||||
}
|
||||
previous = path;
|
||||
}
|
||||
|
||||
switch (op[0]) {
|
||||
case "s": out.push(["s", path as NonEmptyPath, (short ? op[1] : op[2]) as JsonValue]); break;
|
||||
case "d": out.push(["d", path as NonEmptyPath]); break;
|
||||
case "a": out.push(["a", path as NonEmptyPath, (short ? op[1] : op[2]) as string]); break;
|
||||
case "t": out.push(["t", path as NonEmptyPath, (short ? op[1] : op[2]) as number]); break;
|
||||
case "p": {
|
||||
const [i, r, items] = short
|
||||
? [op[1] as number, op[2] as number, op[3] as JsonValue[]]
|
||||
: [op[2] as number, op[3] as number, op[4] as JsonValue[]];
|
||||
out.push(["p", path, i, r, items]);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
return out;
|
||||
},
|
||||
};
|
||||
}
|
||||
+113
@@ -0,0 +1,113 @@
|
||||
/**
|
||||
* Not a test — a guard against the cliffs found during implementation. Reports
|
||||
* units you can reason about: cost per WRITE, and bytes on the wire relative to
|
||||
* bytes consumed.
|
||||
*
|
||||
* npx vitest bench delta.bench.ts
|
||||
*
|
||||
* The headline finding: cost is per write, not per byte. A 100-byte chunk costs
|
||||
* about the same as a 1 KB chunk, because the fixed cost is the window slice and
|
||||
* the overlap probe. So chunk size — set by the exec env's coalescing interval,
|
||||
* not by this package — decides both throughput and wire size.
|
||||
*/
|
||||
import { bench, describe } from "vitest";
|
||||
import { decoder, encoder, track } from "./delta-impl.ts";
|
||||
|
||||
const WORDS = "the quick brown fox jumps over lazy dog while considering strategies".split(" ");
|
||||
const text = (n: number): string => {
|
||||
let s = "";
|
||||
while (s.length < n) s += `${WORDS[Math.floor(Math.random() * WORDS.length)]} `;
|
||||
return s.slice(0, n);
|
||||
};
|
||||
|
||||
const CAP = 50_000;
|
||||
const WRITES = 500;
|
||||
const FLUSH_EVERY = 20;
|
||||
|
||||
/** One write = one chunk handed to the sink. Returns nothing; bench times it. */
|
||||
function drive(initial: string, chunkSize: number, cap = CAP): void {
|
||||
const t = track({ out: initial });
|
||||
for (let i = 0; i < WRITES; i++) {
|
||||
const next = t.state.out + text(chunkSize);
|
||||
t.state.out = next.length > cap ? next.slice(next.length - cap) : next;
|
||||
if (i % FLUSH_EVERY === FLUSH_EVERY - 1) t.flush();
|
||||
}
|
||||
t.flush();
|
||||
}
|
||||
|
||||
/**
|
||||
* Measured on the reference machine, 500 writes each. Reproduced here so a
|
||||
* regression is legible rather than just "slower".
|
||||
*
|
||||
* append only, 100 B 127 µs/write 8k writes/s 101% of input on wire
|
||||
* rolling, 100 B 176 µs/write 6k writes/s 135% of input on wire
|
||||
* rolling, 1 KB 68 µs/write 15k writes/s 104% of input on wire
|
||||
* rolling, 8 KB 366 µs/write 3k writes/s 31% of input on wire
|
||||
* pathological, 100 B 307 µs/write 3k writes/s 727% of input on wire
|
||||
*
|
||||
* Two things to notice.
|
||||
*
|
||||
* 135% at 100-byte chunks: past the cap every write emits `truncate` + `append`,
|
||||
* and two op envelopes exceed a 100-byte payload — we put MORE on the wire than we
|
||||
* consumed. At 8 KB it is 31%. Flush granularity is therefore a size lever, not
|
||||
* just an update-rate lever, and the exec env's coalescing is what keeps chunks
|
||||
* off the bad end of that curve.
|
||||
*
|
||||
* The pathological row is bounded only because the overlap candidate scan is
|
||||
* capped. Unbounded it was 4.7 s for 2000 writes against 93 ms; the cap makes it
|
||||
* give up and emit a set — larger, never wrong.
|
||||
*/
|
||||
describe("rolling window, 50 KB cap", () => {
|
||||
bench("100 B chunks", () => { drive(text(CAP), 100); });
|
||||
bench("1 KB chunks", () => { drive(text(CAP), 1024); });
|
||||
bench("8 KB chunks", () => { drive(text(CAP), 8192); });
|
||||
bench("pathological: one repeated character", () => { drive("x".repeat(CAP), 100); });
|
||||
});
|
||||
|
||||
describe("below the cap", () => {
|
||||
// The dominant case for most tools: startsWith fast path, no probe at all,
|
||||
// and one append per flush rather than a truncate/append pair.
|
||||
bench("append only, never fills", () => { drive("", 100, Number.MAX_SAFE_INTEGER); });
|
||||
});
|
||||
|
||||
// ── Codec: what path interning and arity omission actually save ────────────
|
||||
//
|
||||
// Reported as wire bytes relative to inline paths, on three shapes:
|
||||
// - one hot path (a rolling tool-output window): omission does the work
|
||||
// - a few alternating paths (lane state): interning does the work
|
||||
// - many distinct paths: neither helps, and definitions cost a little
|
||||
describe("codec", () => {
|
||||
const streams = {
|
||||
"one hot path": () => {
|
||||
const t = track({ content: [{ type: "text", text: "" }] });
|
||||
return Array.from({ length: 200 }, (_, i) => {
|
||||
(t.state.content[0] as { text: string }).text += `line ${i} of build output\n`;
|
||||
return t.flush();
|
||||
});
|
||||
},
|
||||
"four alternating paths": () => {
|
||||
const t = track({ a: "", b: "", c: "", d: "" });
|
||||
return Array.from({ length: 200 }, (_, i) => {
|
||||
for (const k of ["a", "b", "c", "d"] as const) t.state[k] += `${i}`;
|
||||
return t.flush();
|
||||
});
|
||||
},
|
||||
"200 distinct paths": () => {
|
||||
const root: Record<string, number> = {};
|
||||
for (let i = 0; i < 200; i++) root[`f${i}`] = 0;
|
||||
const t = track(root);
|
||||
return Array.from({ length: 5 }, (_, r) => {
|
||||
for (let i = 0; i < 200; i++) t.state[`f${i}`] = r + 1;
|
||||
return t.flush();
|
||||
});
|
||||
},
|
||||
};
|
||||
|
||||
for (const [label, build] of Object.entries(streams)) {
|
||||
bench(label, () => {
|
||||
const enc = encoder();
|
||||
const dec = decoder();
|
||||
for (const ops of build()) dec.decode(enc.encode(ops));
|
||||
});
|
||||
}
|
||||
});
|
||||
+133
@@ -0,0 +1,133 @@
|
||||
/**
|
||||
* Runnable illustrations of the two things that are easy to get wrong.
|
||||
*
|
||||
* node --experimental-strip-types delta.examples.ts
|
||||
*
|
||||
* Nothing here is a test — `delta.test.ts` covers the same ground with
|
||||
* assertions. This exists to be read.
|
||||
*/
|
||||
import { apply, decoder, encoder, isBase, track, type Op } from "./delta-impl.ts";
|
||||
|
||||
const show = (label: string, ops: Op[]) => {
|
||||
console.log(` ${label.padEnd(22)} ${ops.length} op(s), base=${isBase(ops)}`);
|
||||
for (const op of ops) console.log(` ${trunc(JSON.stringify(op), 66)}`);
|
||||
};
|
||||
const trunc = (s: string, n: number) => (s.length <= n ? s : `${s.slice(0, n)}…`);
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
// 1. Replacing the whole value
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
console.log("\n1. REPLACING THE WHOLE VALUE\n");
|
||||
|
||||
const before = { user: { id: "u1", name: "ada" }, items: ["a", "b"], note: "n".repeat(120) };
|
||||
const after = { user: { id: "u2", name: "bob" }, items: ["x"], note: "m".repeat(120) };
|
||||
|
||||
{
|
||||
// The wrong way. `state` is a property of the TRACKER, not of the tracked
|
||||
// object. Without a setter this swaps the proxy for a plain object and every
|
||||
// later mutation is silently untracked — no error, no ops.
|
||||
//
|
||||
// The implementation defines `state` as a setter precisely so this works.
|
||||
const t = track<Record<string, unknown>>(structuredClone(before));
|
||||
t.flush(); // drain the opening base batch
|
||||
t.state.items = ["changed first"]; // recorded…
|
||||
t.state = structuredClone(after); // …then discarded: it described a dead value
|
||||
show("assign to .state", t.flush());
|
||||
|
||||
t.state.user = { id: "u3" }; // still tracked afterwards
|
||||
show(" and still tracked", t.flush());
|
||||
}
|
||||
|
||||
{
|
||||
// Dead-op elimination: a field written three times emits one op, and a parent
|
||||
// replaced after its child emits one. Object key order is not preserved by
|
||||
// this pass — see delta.md §5.1.
|
||||
const t = track<Record<string, any>>(structuredClone(before));
|
||||
t.flush(); // drain the opening base batch
|
||||
t.state.items = ["first"];
|
||||
t.state.items = ["second"];
|
||||
t.state.user.name = "dropped";
|
||||
t.state.user = { id: "u9" };
|
||||
show("4 writes, 2 survive", t.flush());
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
// 2. Why checkpointing exists
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
console.log("\n2. RECOVERY LENGTH\n");
|
||||
|
||||
/**
|
||||
* A bash-shaped workload: output arrives continuously and the sink writes
|
||||
* durably on an interval. Each flush is one batch appended to a list.
|
||||
* On resume the reader folds back to the last BASE batch, so that distance is
|
||||
* the cost of recovery.
|
||||
*/
|
||||
function simulate(writes: number, checkpointEvery: number | null) {
|
||||
const CAP = 50_000;
|
||||
const t = track({ out: "" });
|
||||
const isBaseBatch: boolean[] = [];
|
||||
|
||||
for (let i = 0; i < writes; i++) {
|
||||
const next = `${t.state.out}[${i}] cc -c src/file${i}.c -o build/file${i}.o\n`;
|
||||
t.state.out = next.length > CAP ? next.slice(next.length - CAP) : next;
|
||||
|
||||
if (checkpointEvery !== null && i % checkpointEvery === checkpointEvery - 1) t.rebase();
|
||||
|
||||
const ops = t.flush();
|
||||
if (ops.length > 0) isBaseBatch.push(isBase(ops));
|
||||
}
|
||||
|
||||
const lastBase = isBaseBatch.lastIndexOf(true);
|
||||
return { written: isBaseBatch.length, replay: isBaseBatch.length - lastBase - 1 };
|
||||
}
|
||||
|
||||
for (const [label, every] of [["never", null], ["rebase() every 50", 50]] as const) {
|
||||
const r = simulate(500, every);
|
||||
console.log(` ${label.padEnd(18)} ${r.written} batches written → ${r.replay} to replay on recovery`);
|
||||
}
|
||||
|
||||
console.log(`
|
||||
Nothing produces a base batch on its own: flush() emits ops, and a replacement
|
||||
happens only when the producer asks for one. So a stream of appends stays a
|
||||
stream of deltas indefinitely, and recovery grows without bound.
|
||||
|
||||
Two callers need rebase():
|
||||
- pendingToolOutput, during a long-running command. Without it a ten-minute
|
||||
build leaves hundreds of batches to fold on resume.
|
||||
- a facet host on resubscribe. facets.md §9.2 requires the first batch of a
|
||||
subscription to be a base batch, and the host must be able to produce one
|
||||
on demand.
|
||||
`);
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
// 3. A replica always agrees
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
console.log("3. PRODUCER AND REPLICA AGREE\n");
|
||||
{
|
||||
// The whole loop: tracker -> encode -> [boundary] -> decode -> apply.
|
||||
const t = track<Record<string, unknown>>(structuredClone(before));
|
||||
const enc = encoder();
|
||||
const dec = decoder();
|
||||
let replica: unknown;
|
||||
|
||||
const step = (label: string, mutate: () => void) => {
|
||||
mutate();
|
||||
const ops = dec.decode(enc.encode(t.flush()));
|
||||
if (ops.length === 0) return;
|
||||
// No base-batch branch: apply handles `r` by replacing, and tolerates an
|
||||
// undefined target because a root op never reads it.
|
||||
replica = apply(replica, ops);
|
||||
const same = JSON.stringify(replica) === JSON.stringify(t.target);
|
||||
console.log(` ${label.padEnd(22)} ${ops.length} op(s) ${isBase(ops) ? "BASE " : "delta"} matches: ${same}`);
|
||||
};
|
||||
|
||||
step("opening base batch", () => {});
|
||||
|
||||
step("append to a string", () => { (t.state.note as string); t.state.note = `${t.state.note}!`; });
|
||||
step("push", () => { (t.state.items as string[]).push("c"); });
|
||||
step("nested set", () => { (t.state.user as Record<string, unknown>).name = "cy"; });
|
||||
step("delete", () => { delete t.state.note; });
|
||||
step("whole replacement", () => { t.state = structuredClone(after); });
|
||||
step("rebase", () => { t.rebase(); });
|
||||
}
|
||||
console.log();
|
||||
+617
@@ -0,0 +1,617 @@
|
||||
# Delta Tracking and the Op Vocabulary
|
||||
|
||||
> **Production status:** landed in `packages/chord/src/delta/index.ts`, with tests in `packages/chord/test/delta.test.ts`. Chord owns the dependency-free `Op`/`WireOp`, tracker, applier, codec, and validation boundary; Session storage, the Harness, and facets consume it. The implementation and tests beside this document are historical prototype and benchmark evidence, not production source.
|
||||
>
|
||||
> The landed tracker computes deltas at `flush()` from a dirty tree and baseline rather than retaining one op per mutation. That resolves FINDINGS D1. Production remeasurement also closes FINDINGS D2: the generic string path is below surrounding replication/rendering costs, so an explicit append/truncate API is rejected ([decision](append-decision.md)).
|
||||
|
||||
One mechanism covers assistant partials, tool output, tool details, lane state, and arbitrary facet state — on the wire and in durable storage.
|
||||
|
||||
```bash
|
||||
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run test/delta.test.ts
|
||||
# run from packages/chord
|
||||
```
|
||||
|
||||
## 1. Why not an existing library
|
||||
|
||||
Immer, Valtio, Mutative and Colyseus all **record effects, not intent**. A string
|
||||
is a leaf, so `text += chunk` is one write of the whole new string; there is no
|
||||
representation in which the library could know you appended. `unshift` is the
|
||||
same story — the engine shifts every index, so each is a write.
|
||||
|
||||
Measured, all four:
|
||||
|
||||
| operation | what they emit |
|
||||
| --- | --- |
|
||||
| `text += "x"` on 40 KB | `replace` carrying 40 KB |
|
||||
| `arr.unshift(x)` on 100 items | ~100 index writes |
|
||||
| `arr.pop()` | a write to a `length` path, which is not a document location |
|
||||
|
||||
Immer's own pitfalls page states the guarantee: patches are correct and
|
||||
**explicitly not minimal**. Colyseus documents the same array weakness — removing
|
||||
the first of 20 items costs 38 extra bytes.
|
||||
|
||||
Diffing base against result ourselves does not fix it either. A differ sees two
|
||||
values, not intent. Prefix comparison catches pure growth, but a rolling window
|
||||
that drops from the front *and* appends is neither a prefix nor a suffix of the
|
||||
old value; measured on 5000 elements the fallback was
|
||||
`{index: 0, remove: 5000, items: 5000}` — a full replace, in exactly the case the
|
||||
optimisation existed for.
|
||||
|
||||
So the win is not a blind whole-value differ. The tracker records which paths and array operations became dirty, then compares only those subtrees against its accepted baseline at `flush()`.
|
||||
|
||||
## 2. Ops
|
||||
|
||||
**Tuples are the form. Everywhere. There is no keyed variant.**
|
||||
|
||||
```ts
|
||||
export type Seg = string | number;
|
||||
export type Path = readonly Seg[];
|
||||
/** Non-empty: `s`/`d`/`a`/`t` cannot target the root. Enforced by the type. */
|
||||
export type NonEmptyPath = readonly [Seg, ...Seg[]];
|
||||
|
||||
export type Op =
|
||||
| readonly ["r", JsonValue] // replace the value
|
||||
| readonly ["s", NonEmptyPath, JsonValue] // set
|
||||
| readonly ["d", NonEmptyPath] // delete
|
||||
| readonly ["a", NonEmptyPath, string] // append
|
||||
| readonly ["t", NonEmptyPath, number] // truncate, in chars
|
||||
| readonly ["p", Path, number, number, JsonValue[]] // splice: index, remove, items
|
||||
```
|
||||
|
||||
Interning, id references and omitted paths are **not** here — they live in
|
||||
`WireOp` and exist only between `encode` and `decode` (§4).
|
||||
|
||||
**`r` is the only op that replaces a whole value.** It is an op rather than a
|
||||
frame kind for the same reason `a` and `t` exist: they are specialisations of `s`
|
||||
that are smaller and state intent.
|
||||
|
||||
Do not encode a replacement as a set at the root path. Base-batch detection is a
|
||||
**correctness boundary** — recovery stops there (§9) — so it must be a token
|
||||
comparison, `ops[0]?.[0] === "r"`, and not a path inspection. A path-inspecting
|
||||
predicate has to handle the two-element arity form, where `op[1]` is a value
|
||||
rather than a path, and misclassifies `["s", []]`.
|
||||
|
||||
**Only `p` may target the root**, and only because a tracked value can itself be
|
||||
an array — `entries.push(x)` on a root array is `["p", [], 3, 0, [x]]`. A `p`
|
||||
covering its entire target is normalised at flush time to `r` (root) or `s`
|
||||
(nested), so a root `p` is always a partial modification. The other four verbs
|
||||
take a `NonEmptyPath`: `["s", [], v]` and `["d", []]` do not typecheck.
|
||||
|
||||
**Do not add a keyed in-memory shape with a codec at the serialization boundary.**
|
||||
Two representations of one thing means two appliers, two size estimates, and a
|
||||
conversion nobody needs. Ops are tuples in memory, on the wire, and on disk;
|
||||
readability is what a debug formatter is for.
|
||||
|
||||
Six verbs. No `move`, `copy` or `test`: the first two are size optimisations for
|
||||
a case we do not have, and `test` belongs to a conflict model we do not have,
|
||||
since there is exactly one authoritative writer.
|
||||
|
||||
`chars` counts UTF-16 code units, matching `String.prototype.slice`. **Not bytes.**
|
||||
Byte caps are a producer concern and never cross a boundary.
|
||||
|
||||
The format is JSON-Patch-*shaped*, not RFC 6902 conformant. That costs nothing:
|
||||
RFC 6902 has had no successor since 2013, still has the same six ops, and nothing
|
||||
anywhere adds a string splice. Immer was never conformant either.
|
||||
|
||||
## 3. The tracker
|
||||
|
||||
State is **plain TypeScript**. No containers, no handles, no schema, no
|
||||
decorators. You mutate normally.
|
||||
|
||||
```ts
|
||||
const t = track(laneView);
|
||||
t.state.operation.streamingMessage.content[0].text += delta; // -> append
|
||||
t.state.transcript.push(entry); // -> splice
|
||||
t.state.tools[0].details.failures.push({ name, msg }); // -> splice
|
||||
delete t.state.config.model; // -> delete
|
||||
const ops = t.flush();
|
||||
```
|
||||
|
||||
Three mechanisms, one per case:
|
||||
|
||||
**Objects** — ordinary `set` / `deleteProperty` traps.
|
||||
|
||||
**Arrays — mutator methods intercepted in the `get` trap.** `arr.push(x)` passes
|
||||
through `get(arr, "push")` first, so the tracker returns its own function that
|
||||
records `splice(len, 0, [x])` and then delegates. Intent is captured *before* the
|
||||
engine performs its index writes, which is why `unshift` is one op rather than
|
||||
O(n). This is the thing no other library does.
|
||||
|
||||
**Strings — dirty marking on the `set` trap, overlap detection at `flush()`.** Strings are primitives, so there is nothing to intercept. Given the accepted `prev` and final `next`, find the longest suffix of `prev` that is a prefix of `next`:
|
||||
|
||||
```
|
||||
overlap === prev.length -> append(next.slice(overlap))
|
||||
overlap > 0 -> truncate(prev.length - overlap) + append(rest)
|
||||
overlap === 0 -> set(next)
|
||||
```
|
||||
|
||||
Always correct, because the overlap is **verified**, not guessed. The rolling
|
||||
window works:
|
||||
|
||||
```ts
|
||||
s.out = s.out.slice(4) + "!!!";
|
||||
// ["truncate", ["out"], 4]
|
||||
// ["append", ["out"], "!!!"]
|
||||
```
|
||||
|
||||
### 3.1 Overlap must not use a hand-written KMP
|
||||
|
||||
A KMP failure function is asymptotically right and **47x slower in practice**,
|
||||
because it runs character by character in JS. Measured on 2000 window slides over
|
||||
a 50 KB string: 2870 ms for KMP, 61 ms for a native probe.
|
||||
|
||||
The probe: `indexOf` a short head of `next` within `prev`, verify each candidate
|
||||
with `endsWith`. Both are native. Verified identical to KMP across 200 window
|
||||
slides.
|
||||
|
||||
A `startsWith` fast path runs first, so a pure append — the dominant case — skips
|
||||
the probe entirely.
|
||||
|
||||
### 3.2 Rules the implementation must hold
|
||||
|
||||
**Inserted values are adopted; emitted payloads are cloned at flush.** Callers may retain read-only references but must not mutate adopted objects outside the tracker. `flush()` clones payloads and advances a cloned accepted baseline, so emitted ops never alias producer state.
|
||||
|
||||
**`x = undefined` normalises to `delete`.** JSON has no `undefined`, and a `set`
|
||||
with a missing value is indistinguishable from a lost value after a
|
||||
`JSON.stringify` round trip.
|
||||
|
||||
**`arr.length = n` is translated, never emitted as a path.** Shrink becomes a
|
||||
splice, grow becomes a splice of nulls. `length` is a real mutation but not a
|
||||
document location — this is the bug Immer ships (issue 208). Dropping it silently,
|
||||
as an early prototype did, means `arr.length = 0` never reaches the replica.
|
||||
|
||||
**Numeric array keys are normalised to numbers.** Proxy traps deliver `"0"`;
|
||||
paths carry `0`. Valtio ships the string form and it is worse for both size and
|
||||
comparison.
|
||||
|
||||
**Proxies are cached per object and path**, so `s.a === s.a` and proxies are not
|
||||
rebuilt on every access.
|
||||
|
||||
**Scope is `JsonValue` only, and it must be CHECKED, not assumed.**
|
||||
`structuredClone` is not a JSON check — it clones `Map`, `Set`, `Date` and
|
||||
`RegExp` happily, and the result then differs from what `JSON.stringify` produces,
|
||||
so producer and replica diverge silently. Assert the value at record time and
|
||||
reject symbol keys. See §7.4 for what leaks without it.
|
||||
|
||||
**Exactly two ops reach the root**, and the applier's root branch must handle
|
||||
both: `r` always, and `p` when the tracked value is itself an array.
|
||||
`s`/`d`/`a`/`t` cannot, by type, which makes the branch's fallthrough provably
|
||||
unreachable. Handling only `r` there passes every hand-written test and fails on
|
||||
the first randomised sequence that produces a root splice.
|
||||
|
||||
**Two writes to one address inside a transaction chain.** When several writes to
|
||||
the same address are emitted in one commit, each must be recorded against the
|
||||
previous one's result, not against pre-transaction state. The applier replays them
|
||||
in order; anything else applies the second against a stale base. This did not fire
|
||||
in the workload that surfaced it — it would have corrupted silently when it did.
|
||||
|
||||
### 3.2.1 Replacing the whole value
|
||||
|
||||
Assigning to `state` replaces it:
|
||||
|
||||
```ts
|
||||
tracker.state = next; // emits ["r", next]; discards ops recorded before it
|
||||
```
|
||||
|
||||
`state` **must be a setter on the tracker**, not a plain property. Without one,
|
||||
`tracker.state = next` swaps the proxy for a plain object and every later
|
||||
mutation is silently untracked — no error, no ops, `target` unchanged. It is also
|
||||
the first thing anyone tries.
|
||||
|
||||
Prior ops are discarded because they describe a value that no longer exists.
|
||||
|
||||
Only a *full* replacement collapses to `r`. Rewrite part of the value and the ops
|
||||
survive, because they are genuinely cheaper:
|
||||
|
||||
```
|
||||
partial rewrite 2 op(s), base=false
|
||||
["s",["user"],{"id":"u2","name":"bob"}]
|
||||
["s",["items"],["x"]]
|
||||
```
|
||||
|
||||
### 3.2.2 The first flush is a base batch
|
||||
|
||||
`track()` opens a stream, and a consumer starts with nothing, so the first flush
|
||||
always emits `["r", value]` — carrying any mutations made before it:
|
||||
|
||||
```ts
|
||||
const t = track({ x: 0, l: [] });
|
||||
t.state.x = 100;
|
||||
t.state.l.push("xyz");
|
||||
t.flush(); // [["r", { x: 100, l: ["xyz"] }]]
|
||||
t.state.x = 101;
|
||||
t.flush(); // [["s", ["x"], 101]]
|
||||
```
|
||||
|
||||
Requiring the producer to remember a `rebase()` first would fail at runtime, in
|
||||
the consumer, far from the mistake.
|
||||
|
||||
### 3.2.3 Forcing a base batch later
|
||||
|
||||
```ts
|
||||
tracker.rebase(); // next flush is ["r", value]; value unchanged
|
||||
```
|
||||
|
||||
Discarding the pending ops is correct: the proxy mutates the target directly, so
|
||||
the value already carries them. `tracker.state = tracker.state` has the same effect in the landed implementation, but `rebase()` states the intent directly.
|
||||
|
||||
**Nothing produces a base batch on its own.** `flush()` emits ops; a replacement
|
||||
happens only when the producer asks for one. So a stream of appends stays a
|
||||
stream of deltas indefinitely.
|
||||
|
||||
Recovery replays from the last base batch (§9). From `delta.examples.ts`, a
|
||||
bash-shaped workload of 500 durable writes into a 50 KB rolling window:
|
||||
|
||||
| | batches written | to replay on recovery |
|
||||
| --- | --- | --- |
|
||||
| never | 500 | **499** |
|
||||
| `rebase()` every 50 | 500 | 0 |
|
||||
|
||||
Two callers need this:
|
||||
|
||||
- **A durable sink.** `pendingToolOutput` accumulates one batch per checkpoint for
|
||||
as long as a command runs. A `make -j8` running for ten minutes otherwise leaves
|
||||
hundreds of batches to fold on resume. Checkpointing every N makes "at most N
|
||||
batches to replay" a policy rather than an accident.
|
||||
- **A facet host on resubscribe.** `facets.md` §9.2 requires the first batch of
|
||||
every subscription to be a base batch, and a consumer that sees a gap
|
||||
resubscribes to get one. The host has to be able to produce one on demand rather
|
||||
than wait for one to happen by chance.
|
||||
|
||||
### 3.3 Known gaps
|
||||
|
||||
- `sort` / `reverse` / `fill` / `copyWithin` mark the array dirty and emit the resulting structural/index changes rather than preserving the producer's method intent. Add a dedicated op only if a measured workload needs one.
|
||||
- A manual index-shift loop (`for (…) a[i] = a[i+1]`) can still cost O(n) sets. Correct, not minimal, and unavoidable — the producer genuinely wrote every element.
|
||||
- Rolling-window string assignment still runs overlap discovery at flush, deliberately: production measurements do not justify text-specific tracker state or API surface ([decision](append-decision.md)).
|
||||
|
||||
### 3.4 Constraints the string algorithm must hold
|
||||
|
||||
Each of these is cheap to get wrong in a way that passes hand-written tests.
|
||||
|
||||
**A fixed-length probe cannot find an overlap shorter than the probe.** The head
|
||||
must actually occur in `a`: `"abcdefgh"` -> `"defghxyz"` overlaps by 5, and a
|
||||
64-character head cannot occur in an 8-character string. Try a long head first,
|
||||
then fall back to a one-character head. A test using only 50 KB strings, where
|
||||
overlaps are huge, will pass against a broken implementation.
|
||||
|
||||
**Bound the candidate scan.** Repetitive output — a build log, or a run of one
|
||||
character — makes a long head match at thousands of positions, each costing a full
|
||||
`endsWith`. Unbounded, 2000 slides over a 50 KB window take 4.7 s against 93 ms
|
||||
bounded. Exceeding the bound returns 0, which emits a set: larger, never wrong.
|
||||
|
||||
**Property tests before anything depends on this.** 3000 random mutation
|
||||
round-trips find the root-splice hole in §3.2 on the first run; the hand-written
|
||||
cases do not. `delta.test.ts` is the suite to port.
|
||||
|
||||
## 4. The codec
|
||||
|
||||
Two vocabularies, not one.
|
||||
|
||||
`Op` is what the tracker produces and `apply` consumes. **Paths are always
|
||||
inline.** It knows nothing about a dictionary.
|
||||
|
||||
`WireOp` is what crosses a boundary. It adds exactly two compressions:
|
||||
|
||||
```
|
||||
["#", id, path] defines an id, emitted on a path's SECOND use
|
||||
a numeric PathRef references a previously defined id
|
||||
a shortened tuple reuses the previous op's path; arity disambiguates
|
||||
```
|
||||
|
||||
`encoder().encode(ops): WireOp[]` and `decoder().decode(wire): Op[]` are the only places either exists. Keeping them out of `Op` means `apply` has no id resolution, no `#`
|
||||
case, and no previous-path state — three branches removed from the hot path —
|
||||
and dirty-tree generation always works with inline paths before encoding.
|
||||
|
||||
`["r", value]` carries no path, so it encodes to itself. That is why `isBase`
|
||||
works unchanged on either vocabulary.
|
||||
|
||||
### 4.1 Two rules that are easy to get wrong
|
||||
|
||||
**One encoder/decoder pair per stream.** The id table spans a whole subscription
|
||||
or file: a path interned in batch 3 is referenced in batch 40. A second consumer
|
||||
that subscribes at batch 40 has never seen the definition, so it needs its own
|
||||
encoder. Sharing one across consumers hands the late subscriber ids it cannot
|
||||
resolve, and a base batch does not rescue it — `["r", value]` carries no refs and
|
||||
leaves the table empty.
|
||||
|
||||
**Reset the table on a base batch.** A reader replays from the *last* base batch
|
||||
with a fresh decoder, so everything after one must be self-contained. Carrying
|
||||
ids across a replacement emits references to definitions the reader never saw:
|
||||
|
||||
```
|
||||
4 : [["a",0,"x4"],["a",1,"y4"]]
|
||||
5 BASE: [["r",{…}]]
|
||||
6 : [["a",0,"x6"]] <- id 0 was defined in batch 1
|
||||
-> RECOVERY FAILED: PathError - unresolvable path: 0
|
||||
```
|
||||
|
||||
**Arity omission is scoped to a batch.** Letting it span batches makes a batch's
|
||||
first op depend on the previous batch's last one, so a reader that skips or
|
||||
reorders a batch decodes into the wrong path. Ids are the only cross-batch state,
|
||||
and the dictionary makes those explicit.
|
||||
|
||||
### 4.2 Intern on second use, not first
|
||||
|
||||
A definition costs more than the path it replaces, so interning on first use
|
||||
loses on every path written exactly once — and most are. Measured on a bulk
|
||||
workload: **255.5 KB first-use versus 179.6 KB with no interning at all.**
|
||||
|
||||
### 4.3 Measured
|
||||
|
||||
Wire bytes against inline paths, round-trip verified:
|
||||
|
||||
| stream | inline | wire | saved |
|
||||
| --- | --- | --- | --- |
|
||||
| one hot path (a rolling tool-output window) | 11,290 B | 7,538 B | **33.2%** |
|
||||
| four alternating paths (lane state) | 14,160 B | 11,032 B | **22.1%** |
|
||||
| 200 distinct paths | 16,455 B | 16,035 B | 2.6% |
|
||||
|
||||
Omission does the work in the first case, interning in the second. In the third
|
||||
neither helps and the definitions cost a little — which is the case second-use
|
||||
interning exists to bound.
|
||||
|
||||
## 5. Flush emits ops, and drops the dead ones
|
||||
|
||||
`flush()` computes ops for the dirty paths against the last accepted baseline. It does not retain one op per mutation, so repeated and interleaved writes stay bounded by changed state rather than write count. There is no size comparison or replacement heuristic: a replacement is something the producer asks for by assigning `state` or calling `rebase()`.
|
||||
|
||||
An earlier design compared op bytes against the value's serialised size and
|
||||
replaced when ops were larger. It was removed. Measured against emitting ops
|
||||
unconditionally, on six workloads, it changed the output in two — both requiring
|
||||
hundreds of distinct paths to change in a single flush. Neither the harness nor
|
||||
the replication path does that: `LaneSnapshot` is folded one event at a time, and
|
||||
the widest case (`run_end`) touches four fields. The rule cost a per-flush
|
||||
comparison, a running size estimate, and an invalidation rule for the ops that
|
||||
could not maintain it.
|
||||
|
||||
### 5.1 Dirty-tree collapse
|
||||
|
||||
The landed tracker records only which subtrees are dirty. At flush it compares each dirty subtree's accepted baseline with its final value and emits the surviving structural change. Repeated writes to one field collapse naturally; alternating writes to two fields retain two dirty paths rather than one op per mutation; a parent replacement subsumes dirty descendants where the final comparison permits it. `packages/chord/test/delta.test.ts` pins the interleaved rolling-window case at at most three ops after 1,000 alternating writes.
|
||||
|
||||
This replaces the prototype's backwards dead-op pass and adjacent-only coalescer. Do not port those algorithms into Chord: flush-time generation is the D1 fix.
|
||||
|
||||
> **Object key order is not a replicated invariant.** Values round-trip, but delete-and-reinsert activity within one flush can produce a different insertion order on a replica. Do not hash or content-address a replicated value, and sort explicitly where display order matters.
|
||||
|
||||
## 6. There is no frame type
|
||||
|
||||
The logical batch is `Op[]`; transport and durable storage carry the statefully encoded `WireOp[]`. Do not wrap either batch.
|
||||
|
||||
**`seq` does not belong in the payload.** It would defend against a lossy
|
||||
transport we do not have: a durable list element already carries `seq` from
|
||||
storage, an in-process callback cannot skip, and SSE either delivers in order or
|
||||
breaks — and a break means resubscribe, which means a base batch. A second copy of
|
||||
storage's `seq` could only disagree with the first. The SSE binding stamps `id:`,
|
||||
which is where transport metadata belongs.
|
||||
|
||||
**Nothing else needs wrapping either.** A `kind` discriminator is redundant once a
|
||||
replacement is an op (§2), and the address already says which value a batch
|
||||
belongs to. What would remain is a struct around an array.
|
||||
|
||||
A producer batch is just `Op[]`, and its encoded boundary form is `WireOp[]`. A **base batch** is one whose first op is `r`;
|
||||
`isBase(ops)` is `ops[0]?.[0] === "r"`, exact rather than heuristic because flush
|
||||
guarantees `r` appears at index 0 or not at all (§5).
|
||||
|
||||
Everything a frame used to do is now done by something that already existed:
|
||||
|
||||
| was on the frame | now |
|
||||
| --- | --- |
|
||||
| `seq` | the list element's `seq`, or the SSE `id:` |
|
||||
| `kind: "replace"` | the `r` op |
|
||||
| which value it belongs to | the address |
|
||||
| "this is a snapshot" | the `"base"` storage tag |
|
||||
|
||||
**Resubscription is a base batch plus buffered batches** — the same thing the lane
|
||||
adapter already does for the harness: snapshot first, then whatever accumulated
|
||||
while the client caught up. There is no `Last-Event-ID` and no cross-subscription
|
||||
resume; that would need a retained op log, which §5 deliberately does not keep.
|
||||
|
||||
A consumer that sees a gap, fails to resolve a path, or connects cold takes one
|
||||
path: ask for a base batch. That is why there is no `Rebase` type.
|
||||
|
||||
The cost, stated plainly: **a consumer cannot detect staleness from the payload
|
||||
alone.** It relies on the transport reporting a break. For SSE and in-process that
|
||||
is sound. If a lossy or multiplexed transport ever appears, sequencing goes on that
|
||||
transport's envelope — not back into the ops.
|
||||
|
||||
## 7. Safety
|
||||
|
||||
Ops arrive from a facet, a plugin compartment, or a tool whose details may echo
|
||||
model output. **None of it is trusted input**, and the boundary where untrusted
|
||||
data meets trusted machinery is where this design has to hold.
|
||||
|
||||
The reference point is CVE-2025-55182 — RCE in React Server Components, CVSS 10.0,
|
||||
exploited in the wild. Their Flight protocol is a compact tagged wire format that
|
||||
reconstructs structure, so the resemblance is real. The failure was *"fails to
|
||||
validate the structure correctly... treats the fake object as genuine"*: a forged
|
||||
Chunk resolved as a Promise and exposed internal state containing gadgets to reach
|
||||
`Function`.
|
||||
|
||||
**We are structurally safer, and not because of diligence:**
|
||||
|
||||
| | Flight | ops |
|
||||
| --- | --- | --- |
|
||||
| can describe runtime objects | yes — Chunks resolve as Promises | no |
|
||||
| can reference code or modules | yes — client components | no |
|
||||
| values | arbitrary object graphs | `JsonValue` |
|
||||
| worst case from a forged payload | RCE | corrupted replica state |
|
||||
|
||||
Flight *must* reference code; that is its job. An op can only put a `JsonValue` at
|
||||
a path, so there is no first link in a gadget chain. This is why an attacker cannot
|
||||
plant a non-resolving `then` on `Object.prototype`: a data-only `then` is not
|
||||
callable, and `await` resolves normally when `then` is not callable.
|
||||
|
||||
> **Rule: no op may name something the host resolves.** Not a mutation name, not a
|
||||
> component id, not a module reference. Mutation names were removed for an
|
||||
> unrelated reason (§8); this is the second and better reason never to reintroduce
|
||||
> them, because it is the ingredient that made Flight exploitable.
|
||||
|
||||
### 7.1 Paths are the dangerous part
|
||||
|
||||
`JSON.parse` is safe on its own — `{"__proto__":{}}` becomes an *own* property.
|
||||
The hazard is `parent[key]`, which is exactly what applying a path does.
|
||||
|
||||
A single key is not enough. `x["__proto__"] = v` swaps *x's own* parent, which is
|
||||
local. It takes a **walk** — `x["__proto__"]["polluted"] = v` — to reach the shared
|
||||
prototype, and a path is precisely a walk.
|
||||
|
||||
`constructor` is worse than `__proto__`, because `({}).constructor.constructor` is
|
||||
`Function`. That ladder is closed here only because op values cannot be functions;
|
||||
it is closed properly by rejecting the segment.
|
||||
|
||||
**Reserved path segments: `__proto__`, `constructor`, `prototype`.** Rejected at
|
||||
record time *and* at apply time, including through an interned path id.
|
||||
|
||||
They are reserved as *segments*, not as values. An object with a literal
|
||||
`"__proto__"` key replicates fine as a whole value; only a path walking *through*
|
||||
it is refused. This is a genuine restriction on what is mutable — document it, do
|
||||
not pretend it away.
|
||||
|
||||
Two further measures on the same hazard:
|
||||
|
||||
- **Write with `Object.defineProperty`**, never assignment, so an inherited setter
|
||||
cannot run.
|
||||
- **Resolve own properties only** (`Object.hasOwn`), so a walk cannot escape into
|
||||
the prototype chain and an inherited getter cannot fire. This blocks
|
||||
`constructor` a second time, since it is inherited.
|
||||
|
||||
What prototype pollution actually buys an attacker here is narrower than it first
|
||||
appears: **it flips defaults, it does not overwrite explicit values.** An own
|
||||
property shadows. So `{name:"bob", isAdmin:false}` is unaffected; an option bag read
|
||||
as `opts.skipSandbox` is not. Our own `ShellExecOptions`, `ShellOutputCaptureOptions`,
|
||||
`ListReadOptions` and `TrackerOptions` are exactly that shape. Note also that
|
||||
`"x" in {}` and `const {x = false} = {}` both lie under pollution — which is why
|
||||
the applier uses `Object.hasOwn`.
|
||||
|
||||
### 7.2 Array indices
|
||||
|
||||
An index may address an existing element or append exactly one past the end.
|
||||
|
||||
This is not an arbitrary cap; it is what keeps the value a `JsonValue`. A sparse
|
||||
array does not survive a JSON round trip — holes serialise to `null` and return as
|
||||
real properties — so `arr[7] = x` on a length-3 array already produces state a
|
||||
replica cannot match. Rejecting the write is more honest than diverging.
|
||||
|
||||
It removes a denial of service as a side effect rather than as its purpose:
|
||||
`["s",["xs",4294967290],1]` would otherwise allocate a 4.29-billion-entry array
|
||||
from one op. Growth stays available and stays proportional, because
|
||||
`arr.length = n` is emitted as a splice of explicit nulls whose op size grows with
|
||||
the gap.
|
||||
|
||||
### 7.3 Validate op structure on decode
|
||||
|
||||
This is the RSC lesson applied to us. A decoder must not trust tuple shape.
|
||||
Measured against an unvalidated applier:
|
||||
|
||||
| malformed op | result |
|
||||
| --- | --- |
|
||||
| `["p",["xs"],0,0,"not-an-array"]` | string spread into the array: `["n","o","t",…]` |
|
||||
| `["s","a",9]` — path is a string | accepted; `"a".slice(0,-1)` is `""`, so it wrote at the root |
|
||||
| `["ZZZ",["a"],9]` | silently ignored, replica diverges with no error |
|
||||
|
||||
Validate: verb is known, arity matches the verb, a path is an array of strings and
|
||||
non-negative integers, `p` carries integer index and count plus an array of items,
|
||||
`#` defines an array path. An unknown verb is an **error**, not a no-op — silently
|
||||
skipping it is how a newer producer's op disappears and a replica drifts.
|
||||
|
||||
**One validator per vocabulary.** `assertValidOp` guards `apply` and rejects ids,
|
||||
short forms and `#`; `assertValidWireOp` guards `decode` and permits them.
|
||||
Validating a decoded op against the wire grammar is laxer than its own type: a
|
||||
two-element `["s", value]` would pass, and `apply` would then read the value as a
|
||||
path.
|
||||
|
||||
### 7.4 The tracker's cage leaks on types, not shape
|
||||
|
||||
A facet cannot forge an op: it mutates plain objects and the tracker builds the
|
||||
tuples, so shape is well-formed by construction. Values and keys are another
|
||||
matter, and `structuredClone` is **not** a JSON check.
|
||||
|
||||
| what a facet writes | what happens |
|
||||
| --- | --- |
|
||||
| a function | throws (`DataCloneError`) |
|
||||
| a BigInt, a cycle | throws |
|
||||
| `new Map([[1,2]])` | op carries `{}`, producer keeps a real Map — **silent divergence** |
|
||||
| `new Date(0)` | op carries an ISO string, producer keeps a Date |
|
||||
| `state[Symbol("s")] = 1` | emits `["s",[null],1]` — **a malformed op from our own tracker** |
|
||||
|
||||
So the tracker must check what it is given, at record time, where the mistake is:
|
||||
|
||||
- **Reject symbol keys.** They are not path segments.
|
||||
- **Assert `JsonValue` on every recorded value.** Reject `Map`, `Set`, `Date`,
|
||||
`RegExp`, typed arrays and class instances explicitly. The scope restriction in
|
||||
§3.2 was a comment; it has to be a check.
|
||||
|
||||
A getter on state is safe — the trap records the computed result — and a facet
|
||||
value that *looks* like an op is nested inside `["s", path, value]` and can never
|
||||
be read as a top-level op, because nothing flattens.
|
||||
|
||||
### 7.5 Applier
|
||||
|
||||
```ts
|
||||
export function apply<T>(target: T | undefined, ops: readonly Op[]): T;
|
||||
```
|
||||
|
||||
Six verbs, no domain knowledge, no library, no tool code, no registry lookup, and
|
||||
**no path table** — ids and omitted paths are resolved by `decode` before `apply`
|
||||
sees anything (§4).
|
||||
|
||||
It returns the value rather than mutating in place, because `r` replaces it
|
||||
outright.
|
||||
Runs against a **plain mutable object the consumer owns**. It must not be pointed
|
||||
at a value produced by Immer, which deep-freezes and would throw.
|
||||
|
||||
A page of code in any language, which is the property that keeps non-JS consumers
|
||||
viable. Colyseus ships decoders for C#, Lua and Haxe on the same basis.
|
||||
|
||||
## 8. What this removes from the codebase
|
||||
|
||||
Concrete deletions, not simplifications in principle:
|
||||
|
||||
- **Immer**, from the harness and from the facet layer. See §1 for why a
|
||||
patch-producing library cannot serve this purpose.
|
||||
- **Per-type reducers.** There is one applier, `apply(target, ops)`, with no
|
||||
domain knowledge and no registry. Nobody writes a fold for `ToolOutputState` or
|
||||
for a plugin's shape.
|
||||
- **`detailMutations` and `initialDetails`**, and with them the argument that
|
||||
`details: unknown` forces special handling. A structural tracker never needs
|
||||
the type.
|
||||
- **`Rebase` as a reducer return value.** A fold that cannot apply leaves state
|
||||
alone and the host sends a base batch.
|
||||
- **Mutation names on the wire**, and therefore mutation-name version skew. Names
|
||||
never cross a boundary, so adding or renaming one is not a breaking change.
|
||||
|
||||
And three things not to build, each of which looks reasonable until §1:
|
||||
|
||||
- **A blind whole-value differ.** The landed tracker compares only dirty subtrees against its accepted baseline; it does not scan unrelated state.
|
||||
- **A keyed op shape plus a codec.** Tuples are the form everywhere (§2).
|
||||
- **A frame wrapper.** What travels is `Op[]` (§6).
|
||||
|
||||
## 9. Durable form
|
||||
|
||||
A tracked value is stored as a **list of encoded `WireOp[]` batches**, one appended per flush. Base batches carry the storage tag `"base"`; one stateful decoder per value decodes them before `apply`.
|
||||
|
||||
Recovery reads backwards to the last base batch and applies forward:
|
||||
|
||||
```ts
|
||||
readList(address, { order: "desc", stopAtTag: "base", limit: 100 })
|
||||
```
|
||||
|
||||
`stopAtTag` is a stop condition *within a page*: if no base batch is in the page,
|
||||
the consumer pages again with the cursor. The tag lives on the storage record
|
||||
beside `seq`, never inside the value, so storage never parses ops. See
|
||||
[scopes.md](../02-scopes/scopes.md) §11.
|
||||
|
||||
This is what makes "a replacement truncates recovery" (§5) real rather than
|
||||
aspirational — a reader stops at the last base batch instead of replaying from
|
||||
the beginning.
|
||||
|
||||
## 10. Open questions
|
||||
|
||||
- Prefix interning (a trie over path heads) if a workload emerges with many
|
||||
distinct paths sharing long prefixes. Second-use interning (§4) removes the
|
||||
pathological case; this would go further.
|
||||
- Whether array reordering needs a dedicated op; the landed tracker currently marks the array dirty and emits the resulting structural/index changes.
|
||||
- Cross-language replicas: the applier is a page of code in any language, but the
|
||||
interning tables and the wire framing are not specified for a non-JS consumer.
|
||||
|
||||
Address interning in the JSONL log is the same trick as path interning, applied
|
||||
one layer up over `namespace` + `key`. It is **not** an open question — it is
|
||||
specified in [scopes.md](../02-scopes/scopes.md) §12 and is a separate dictionary from this one.
|
||||
+563
@@ -0,0 +1,563 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { apply, assertJsonValue, assertValidOp, assertValidWireOp, decoder, encoder, isBase, overlap, track, type Op, type WireOp } from "./delta-impl.ts";
|
||||
|
||||
const PAD = "p".repeat(400); // see "adaptive emission": a delta must be able to win
|
||||
|
||||
describe("tracker: intent", () => {
|
||||
it("records an append, not a replacement", () => {
|
||||
const t = track({ s: "", pad: PAD });
|
||||
t.flush(); // drain the base batch
|
||||
t.state.s += "ab";
|
||||
t.state.s += "cd";
|
||||
const ops = t.flush();
|
||||
expect(ops).toEqual([["a", ["s"], "abcd"]]);
|
||||
expect(apply({ s: "", pad: PAD }, ops)).toEqual({ s: "abcd", pad: PAD });
|
||||
});
|
||||
|
||||
it("recovers truncate+append from a rolling window", () => {
|
||||
// The case every effect-recording library degrades to a whole-value set.
|
||||
const t = track({ s: "abcdefgh", pad: PAD });
|
||||
t.flush(); // drain the base batch
|
||||
t.state.s = t.state.s.slice(3) + "xyz";
|
||||
const ops = t.flush();
|
||||
expect(ops).toEqual([["t", ["s"], 3], ["a", ["s"], "xyz"]]);
|
||||
expect(apply({ s: "abcdefgh", pad: PAD }, ops)).toEqual({ s: "defghxyz", pad: PAD });
|
||||
});
|
||||
|
||||
it("records array intent as one splice", () => {
|
||||
const t = track({ xs: [1, 2], pad: PAD });
|
||||
t.flush(); // drain the base batch
|
||||
t.state.xs.push(3);
|
||||
expect(t.flush()).toEqual([["p", ["xs"], 2, 0, [3]]]);
|
||||
});
|
||||
|
||||
it("normalises undefined to a delete", () => {
|
||||
const t = track({ a: 1, pad: PAD });
|
||||
t.flush(); // drain the base batch
|
||||
(t.state as Record<string, unknown>).a = undefined;
|
||||
expect(t.flush()).toEqual([["d", ["a"]]]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("tracker: root ops", () => {
|
||||
it("splices a value that is itself an array", () => {
|
||||
const t = track([1, 2, 3, PAD]);
|
||||
t.flush(); // drain the base batch
|
||||
t.state.push(4);
|
||||
const ops = t.flush();
|
||||
expect(ops).toEqual([["p", [], 4, 0, [4]]]);
|
||||
expect(apply([1, 2, 3, PAD], ops)).toEqual([1, 2, 3, PAD, 4]);
|
||||
});
|
||||
|
||||
it("normalises a splice covering the whole root to a replacement", () => {
|
||||
const t = track([1, 2, 3]);
|
||||
t.flush(); // drain the base batch
|
||||
t.state.splice(0, 3, 9);
|
||||
const ops = t.flush();
|
||||
expect(ops).toEqual([["r", [9]]]);
|
||||
expect(isBase(ops)).toBe(true);
|
||||
});
|
||||
|
||||
it("normalises a nested splice-all to a set, not a replacement", () => {
|
||||
const t = track({ xs: [1, 2, 3], pad: PAD });
|
||||
t.flush(); // drain the base batch
|
||||
t.state.xs.splice(0, 3, 9);
|
||||
expect(t.flush()).toEqual([["s", ["xs"], [9]]]);
|
||||
});
|
||||
|
||||
it("normalises length = 0 on the root", () => {
|
||||
const t = track([1, 2, 3]);
|
||||
t.flush(); // drain the base batch
|
||||
t.state.length = 0;
|
||||
const ops = t.flush();
|
||||
expect(ops).toEqual([["r", []]]);
|
||||
expect(apply([1, 2, 3], ops)).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("replacing the whole value", () => {
|
||||
// The proxy cannot observe `tracker.state = next`: `state` belongs to the
|
||||
// tracker, not to the tracked object, so assigning to it swaps the proxy for a
|
||||
// plain object and silently stops tracking. Replacement must be a method.
|
||||
it("assigning to state emits a base batch and keeps tracking", () => {
|
||||
const t = track<Record<string, unknown>>({ p: 1, q: PAD });
|
||||
t.flush(); // drain the base batch
|
||||
t.state.p = 2;
|
||||
t.state = { r: 9, s: "new" };
|
||||
const ops = t.flush();
|
||||
expect(ops).toEqual([["r", { r: 9, s: "new" }]]);
|
||||
expect(isBase(ops)).toBe(true);
|
||||
expect(apply({}, ops)).toEqual(t.target);
|
||||
|
||||
t.state.r = 10; // the new value is tracked
|
||||
expect(t.flush()).toEqual([["s", ["r"], 10]]);
|
||||
});
|
||||
|
||||
it("assigning to state discards ops recorded before it", () => {
|
||||
// They describe a value that no longer exists.
|
||||
const t = track<Record<string, unknown>>({ p: 1, q: PAD });
|
||||
t.flush(); // drain the base batch
|
||||
t.state.p = 2;
|
||||
t.state = { z: 1 };
|
||||
expect(t.flush()).toEqual([["r", { z: 1 }]]);
|
||||
});
|
||||
|
||||
it("a partial rewrite keeps its ops", () => {
|
||||
const t = track<Record<string, unknown>>({ a: "x".repeat(200), b: "y".repeat(200) });
|
||||
t.flush(); // drain the base batch
|
||||
t.state.a = "p";
|
||||
const ops = t.flush();
|
||||
expect(isBase(ops)).toBe(false);
|
||||
expect(ops).toEqual([["s", ["a"], "p"]]);
|
||||
});
|
||||
|
||||
it("rebase() forces a base batch without changing the value", () => {
|
||||
// This is the checkpoint. Recovery replays from the last base batch, so a
|
||||
// producer must be able to bound that rather than depend on the adaptive
|
||||
// size rule, which compares each flush independently.
|
||||
const t = track<Record<string, unknown>>({ p: 1, pad: PAD });
|
||||
t.flush(); // drain the base batch
|
||||
t.rebase();
|
||||
const ops = t.flush();
|
||||
expect(isBase(ops)).toBe(true);
|
||||
expect(apply({}, ops)).toEqual(t.target);
|
||||
});
|
||||
|
||||
it("rebase() applies once, not to every later flush", () => {
|
||||
const t = track<Record<string, unknown>>({ p: 1, pad: PAD });
|
||||
t.flush(); // drain the base batch
|
||||
t.rebase();
|
||||
t.flush();
|
||||
t.state.p = 2;
|
||||
expect(t.flush()).toEqual([["s", ["p"], 2]]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("the first flush", () => {
|
||||
it("is always a base batch", () => {
|
||||
// A consumer starts with nothing, so a stream opening with deltas has
|
||||
// nothing to apply them to. Requiring the producer to remember would fail
|
||||
// at runtime, in the consumer, rather than where the mistake is.
|
||||
const t = track<Record<string, unknown>>({ x: 0 });
|
||||
t.state.x = 100;
|
||||
const ops = t.flush();
|
||||
expect(isBase(ops)).toBe(true);
|
||||
expect(ops).toEqual([["r", { x: 100 }]]); // carries mutations made before it
|
||||
});
|
||||
|
||||
it("is followed by deltas", () => {
|
||||
const t = track<Record<string, unknown>>({ x: 0, pad: PAD });
|
||||
t.flush();
|
||||
t.state.x = 1;
|
||||
expect(t.flush()).toEqual([["s", ["x"], 1]]);
|
||||
});
|
||||
|
||||
it("emits nothing when untouched", () => {
|
||||
// An unmutated tracker has nothing to say; the base batch is owed to a
|
||||
// consumer, and there is no consumer until someone flushes.
|
||||
const t = track<Record<string, unknown>>({ x: 0 });
|
||||
expect(t.flush()).toEqual([["r", { x: 0 }]]);
|
||||
expect(t.flush()).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("apply and fan-out", () => {
|
||||
it("adopts an `r` payload rather than copying it", () => {
|
||||
// The consumer owns the batch it was handed. Copying every base batch
|
||||
// defensively doubles the memory of the one op carrying the whole value.
|
||||
//
|
||||
// The rule that follows: do not hand one batch to two in-process
|
||||
// consumers. This test pins the aliasing so nobody "fixes" it by adding a
|
||||
// clone back.
|
||||
const t = track<Record<string, unknown>>({ n: 0 });
|
||||
const batch = t.flush();
|
||||
const a = apply(undefined, batch) as Record<string, unknown>;
|
||||
const b = apply(undefined, batch) as Record<string, unknown>;
|
||||
a.n = 1;
|
||||
expect(b.n).toBe(1);
|
||||
});
|
||||
|
||||
it("does not alias the producer", () => {
|
||||
const t = track<Record<string, unknown>>({ x: 0 });
|
||||
const replica = apply(undefined, t.flush()) as Record<string, unknown>;
|
||||
t.state.x = 999;
|
||||
t.flush();
|
||||
expect(replica.x).toBe(0);
|
||||
});
|
||||
|
||||
it("folds a whole stream without a base-batch branch", () => {
|
||||
// `apply` handles `r` by replacing, and tolerates an undefined target,
|
||||
// so a consumer needs no isBase check and no clone of its own.
|
||||
const t = track<Record<string, any>>({ x: 0, l: [] as string[] });
|
||||
const enc = encoder();
|
||||
const dec = decoder();
|
||||
let replica: unknown;
|
||||
const send = () => { replica = apply(replica, dec.decode(enc.encode(t.flush()))); };
|
||||
|
||||
t.state.x = 100;
|
||||
t.state.l.push("xyz");
|
||||
send();
|
||||
t.state.x = 101;
|
||||
send();
|
||||
expect(replica).toEqual(t.target);
|
||||
});
|
||||
});
|
||||
|
||||
describe("dead-op elimination", () => {
|
||||
// Ops apply in order, so an op is dead if a later one overwrites the whole
|
||||
// subtree it lives in. Only s/d/r dominate; a/t/p modify what is there.
|
||||
const run = (mutate: (s: any) => void, init: any = { a: { b: 1 }, x: 1, y: 2 }) => {
|
||||
const t = track<any>(structuredClone(init));
|
||||
t.flush(); // drain the base batch
|
||||
mutate(t.state);
|
||||
return t.flush();
|
||||
};
|
||||
|
||||
it("collapses repeated writes to one field", () => {
|
||||
expect(run((s) => { s.x = 1; s.x = 2; s.x = 3; })).toEqual([["s", ["x"], 3]]);
|
||||
});
|
||||
|
||||
it("drops a write superseded by a later, non-adjacent one", () => {
|
||||
expect(run((s) => { s.x = 10; s.y = 20; s.x = 30; }))
|
||||
.toEqual([["s", ["y"], 20], ["s", ["x"], 30]]);
|
||||
});
|
||||
|
||||
it("drops a child write when the parent is replaced after it", () => {
|
||||
expect(run((s) => { s.a.b = 99; s.a = { c: 5 }; })).toEqual([["s", ["a"], { c: 5 }]]);
|
||||
});
|
||||
|
||||
it("keeps a child write that follows the parent replacement", () => {
|
||||
expect(run((s) => { s.a = { b: 1 }; s.a.b = 7; }))
|
||||
.toEqual([["s", ["a"], { b: 1 }], ["s", ["a", "b"], 7]]);
|
||||
});
|
||||
|
||||
it("collapses set-then-delete to the delete", () => {
|
||||
expect(run((s) => { s.x = 5; delete s.x; })).toEqual([["d", ["x"]]]);
|
||||
});
|
||||
|
||||
it("keeps delete-then-set, to preserve key position", () => {
|
||||
// Dropping the delete would leave the key in its original slot while the
|
||||
// producer moved it to the end. Same value, different serialisation.
|
||||
expect(run((s) => { delete s.x; s.x = 5; })).toEqual([["d", ["x"]], ["s", ["x"], 5]]);
|
||||
});
|
||||
|
||||
it("is linear in the number of ops", () => {
|
||||
// The naive formulation compares every op against every dominator, which
|
||||
// is quadratic and degrades on exactly the wide flush this pass cleans up.
|
||||
const wide = (n: number) => {
|
||||
const root: Record<string, number> = {};
|
||||
for (let i = 0; i < n; i++) root[`f${i}`] = i;
|
||||
const t = track(root);
|
||||
t.flush(); // drain the base batch
|
||||
const started = performance.now();
|
||||
for (let i = 0; i < n; i++) t.state[`f${i}`] = i + 1;
|
||||
t.flush();
|
||||
return performance.now() - started;
|
||||
};
|
||||
wide(200); // warm
|
||||
const small = Math.max(wide(250), 0.1);
|
||||
const large = wide(2500);
|
||||
expect(large / small).toBeLessThan(40); // linear would be ~10x
|
||||
});
|
||||
|
||||
it("collapses a pathological redundant producer", () => {
|
||||
const t = track<any>({ a: { b: 1 }, x: 1 });
|
||||
t.flush(); // drain the base batch
|
||||
for (let i = 0; i < 2000; i++) { t.state.x = i; t.state.a.b = i; }
|
||||
t.state.a = { done: true };
|
||||
expect(t.flush()).toHaveLength(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe("flush", () => {
|
||||
it("drops everything before a replacement", () => {
|
||||
const t = track([1, 2]);
|
||||
t.flush(); // drain the base batch
|
||||
t.state.push(3);
|
||||
t.state.splice(0, 3, 7);
|
||||
expect(t.flush()).toEqual([["r", [7]]]);
|
||||
});
|
||||
|
||||
it("keeps the prefix when the replacement is nested", () => {
|
||||
// `s` on a subtree is not a root replacement, so earlier ops stay live.
|
||||
const t = track({ a: 1, xs: [1, 2, 3], pad: PAD });
|
||||
t.flush(); // drain the base batch
|
||||
t.state.a = 2;
|
||||
t.state.xs.splice(0, 3, 9);
|
||||
expect(t.flush()).toEqual([["s", ["a"], 2], ["s", ["xs"], [9]]]);
|
||||
});
|
||||
|
||||
it("rejects a constructor walk", () => {
|
||||
// ({}).constructor.constructor is Function — the classic escape ladder.
|
||||
expect(() => apply({}, JSON.parse('[["s",["constructor","prototype","gadget"],true]]'))).toThrow();
|
||||
expect(({} as Record<string, unknown>).gadget).toBeUndefined();
|
||||
});
|
||||
|
||||
it("rejects a forbidden path reached through an interned id", () => {
|
||||
const dict = new Map<number, readonly (string | number)[]>([[0, ["__proto__", "w"]]]);
|
||||
expect(() => apply({}, [["s", 0, true]], (id) => dict.get(id)!)).toThrow();
|
||||
});
|
||||
|
||||
it("does not run an inherited setter", () => {
|
||||
Object.defineProperty(Object.prototype, "trap", {
|
||||
get() { throw new Error("inherited getter ran"); },
|
||||
configurable: true,
|
||||
});
|
||||
try {
|
||||
expect(() => apply({}, [["s", ["trap"], 1]])).not.toThrow();
|
||||
} finally {
|
||||
delete (Object.prototype as Record<string, unknown>).trap;
|
||||
}
|
||||
});
|
||||
|
||||
it("allows a reserved name as a VALUE key", () => {
|
||||
// Reserved as segments, not as values.
|
||||
const out = apply({}, [["s", ["a"], JSON.parse('{"__proto__":{"z":1}}')]]);
|
||||
expect(({} as Record<string, unknown>).z).toBeUndefined();
|
||||
expect(out).toBeDefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("safety: array indices", () => {
|
||||
// An index may address an existing element or append exactly one past the end.
|
||||
// Not an arbitrary cap: a sparse array does not survive a JSON round trip, so
|
||||
// arr[7] = x on a length-3 array already produces unreplicable state.
|
||||
it("writes an existing index", () => {
|
||||
expect(apply({ xs: [1, 2, 3] }, [["s", ["xs", 1], 9]])).toEqual({ xs: [1, 9, 3] });
|
||||
});
|
||||
it("appends one past the end", () => {
|
||||
expect(apply({ xs: [1, 2, 3] }, [["s", ["xs", 3], 9]])).toEqual({ xs: [1, 2, 3, 9] });
|
||||
});
|
||||
it("rejects a gap", () => {
|
||||
expect(() => apply({ xs: [1, 2, 3] }, [["s", ["xs", 5], 9]])).toThrow();
|
||||
});
|
||||
it("rejects a huge index", () => {
|
||||
// Would otherwise allocate 4.29 billion entries from one op.
|
||||
expect(() => apply({ xs: [] }, [["s", ["xs", 4_294_967_290], 1]])).toThrow();
|
||||
});
|
||||
it("allows explicit growth with nulls", () => {
|
||||
expect(apply({ xs: [1] }, [["p", ["xs"], 1, 0, [null, null, 9]]])).toEqual({ xs: [1, null, null, 9] });
|
||||
});
|
||||
});
|
||||
|
||||
describe("safety: op structure", () => {
|
||||
// The CVE-2025-55182 lesson: a decoder must not trust tuple shape.
|
||||
it("rejects an unknown verb rather than skipping it", () => {
|
||||
// Silently skipping is how a newer producer's op vanishes and a replica drifts.
|
||||
expect(() => apply({ a: 1 }, [["ZZZ", ["a"], 9] as never])).toThrow();
|
||||
});
|
||||
it("rejects non-array splice items", () => {
|
||||
// Unvalidated, this spreads a string: ["n","o","t","-","a",…]
|
||||
expect(() => apply({ xs: [1] }, [["p", ["xs"], 0, 0, "not-an-array"] as never])).toThrow();
|
||||
});
|
||||
it("rejects a string path", () => {
|
||||
// Unvalidated, "a".slice(0,-1) is "", so it resolved to the ROOT and wrote.
|
||||
expect(() => apply({ a: 1 }, [["s", "a", 9] as never])).toThrow();
|
||||
});
|
||||
it("rejects a non-tuple op", () => {
|
||||
expect(() => apply({ a: 1 }, [{ op: "s" } as never])).toThrow();
|
||||
expect(() => apply({ a: 1 }, [null as never])).toThrow();
|
||||
});
|
||||
it("clamps a splice remove past the end, as Array.prototype.splice does", () => {
|
||||
// Deterministic and identical on both sides — not a hole.
|
||||
expect(apply({ xs: [1, 2] }, [["p", ["xs"], 0, 1e9, []]])).toEqual({ xs: [] });
|
||||
});
|
||||
});
|
||||
|
||||
describe("safety: what a facet can put on the wire", () => {
|
||||
// A facet cannot forge an op — it mutates plain objects and the tracker builds
|
||||
// the tuples. Values and keys are the leak, and structuredClone is NOT a JSON
|
||||
// check: it clones a Map happily, and the op then carries {} while the producer
|
||||
// keeps a real Map.
|
||||
const rejects = (name: string, mutate: (s: Record<string, unknown>) => void) =>
|
||||
it(`rejects ${name}`, () => {
|
||||
const t = track<Record<string, unknown>>({ pad: PAD });
|
||||
t.flush(); // drain the base batch
|
||||
expect(() => mutate(t.state)).toThrow();
|
||||
});
|
||||
|
||||
rejects("a function", (s) => { s.f = () => {}; });
|
||||
rejects("a Map", (s) => { s.m = new Map([[1, 2]]); });
|
||||
rejects("a Date", (s) => { s.d = new Date(0); });
|
||||
rejects("a BigInt", (s) => { s.b = 1n; });
|
||||
rejects("a cycle", (s) => { const a: Record<string, unknown> = {}; a.self = a; s.c = a; });
|
||||
rejects("a symbol key", (s) => { s[Symbol("k") as unknown as string] = 1; });
|
||||
|
||||
it("allows a getter on state, recording the computed result", () => {
|
||||
const t = track<Record<string, unknown>>({ pad: PAD });
|
||||
t.flush(); // drain the base batch
|
||||
Object.defineProperty(t.target, "g", { get: () => 42, enumerable: true });
|
||||
t.state.x = 1;
|
||||
expect(t.flush()).toEqual([["s", ["x"], 1]]);
|
||||
});
|
||||
|
||||
it("accepts a value that merely looks like an op", () => {
|
||||
// Nested inside ["s", path, value]; nothing flattens, so it cannot be read
|
||||
// as a top-level op.
|
||||
const t = track<Record<string, unknown>>({ pad: PAD });
|
||||
t.flush(); // drain the base batch
|
||||
t.state.x = ["r", { evil: true }];
|
||||
expect(t.flush()).toEqual([["s", ["x"], ["r", { evil: true }]]]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("assertions", () => {
|
||||
it("assertJsonValue rejects non-JSON", () => {
|
||||
for (const bad of [new Map(), new Set(), new Date(), /re/, () => {}, Number.NaN, Infinity]) {
|
||||
expect(() => assertJsonValue(bad)).toThrow();
|
||||
}
|
||||
for (const good of [null, true, 1, "s", [1, "a"], { a: { b: [null] } }]) {
|
||||
expect(() => assertJsonValue(good)).not.toThrow();
|
||||
}
|
||||
});
|
||||
it("assertValidOp accepts decoded ops and rejects wire forms", () => {
|
||||
// Validating Op against the wire grammar would be laxer than the type: a
|
||||
// two-element ["s", value] would pass, and apply would read the value as a
|
||||
// path. Each vocabulary gets the validator that matches it.
|
||||
for (const op of [
|
||||
["r", { a: 1 }], ["s", ["a"], 1], ["d", ["a"]],
|
||||
["a", ["a"], "x"], ["t", ["a"], 2], ["p", ["a"], 0, 0, []],
|
||||
] as never[]) {
|
||||
expect(() => assertValidOp(op)).not.toThrow();
|
||||
}
|
||||
for (const wireOnly of [["s", 1], ["d"], ["a", "x"], ["t", 2], ["p", 0, 0, []], ["#", 0, ["a"]], ["s", 0, 1]] as never[]) {
|
||||
expect(() => assertValidOp(wireOnly)).toThrow();
|
||||
expect(() => assertValidWireOp(wireOnly)).not.toThrow();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("codec: path interning and arity omission", () => {
|
||||
// One pair per stream. The table spans a whole subscription or file, so a
|
||||
// second consumer joining later needs its own encoder.
|
||||
const roundTrip = (batches: Op[][]) => {
|
||||
const enc = encoder();
|
||||
const dec = decoder();
|
||||
return batches.map((ops) => dec.decode(enc.encode(ops)));
|
||||
};
|
||||
|
||||
it("round-trips a stream exactly", () => {
|
||||
const t = track<Record<string, any>>({ a: { deep: "" }, b: { deep: "" } });
|
||||
t.flush(); // drain the base batch
|
||||
const batches: Op[][] = [];
|
||||
for (let i = 0; i < 6; i++) {
|
||||
t.state.a.deep += `x${i}`;
|
||||
t.state.b.deep += `y${i}`;
|
||||
batches.push(t.flush());
|
||||
}
|
||||
expect(roundTrip(batches)).toEqual(batches);
|
||||
});
|
||||
|
||||
it("interns on second use, not first", () => {
|
||||
const enc = encoder();
|
||||
const p = ["a", "deep"] as const;
|
||||
const first = enc.encode([["a", p, "1"]]);
|
||||
const second = enc.encode([["a", p, "2"]]);
|
||||
expect(first).toEqual([["a", p, "1"]]); // inline
|
||||
expect(second).toEqual([["#", 0, p], ["a", 0, "2"]]); // define, then use
|
||||
});
|
||||
|
||||
it("omits the path when it repeats", () => {
|
||||
const enc = encoder();
|
||||
const p = ["a"] as const;
|
||||
expect(enc.encode([["s", p, 1], ["s", p, 2]])).toEqual([["s", p, 1], ["s", 2]]);
|
||||
});
|
||||
|
||||
it("resets the table on a base batch, so recovery is self-contained", () => {
|
||||
// A reader replays from the LAST base batch with a fresh decoder. Ids
|
||||
// defined before it were never seen, so keeping them breaks recovery with
|
||||
// an unresolvable path id.
|
||||
const enc = encoder();
|
||||
const p = ["a", "deep"] as const;
|
||||
enc.encode([["a", p, "1"]]);
|
||||
enc.encode([["a", p, "2"]]); // id 0 assigned
|
||||
const base = enc.encode([["r", { a: { deep: "x" } }]]);
|
||||
const after = enc.encode([["a", p, "3"]]);
|
||||
expect(base).toEqual([["r", { a: { deep: "x" } }]]);
|
||||
expect(after).toEqual([["a", p, "3"]]); // inline again
|
||||
|
||||
const dec = decoder();
|
||||
expect(() => dec.decode(base)).not.toThrow();
|
||||
expect(dec.decode(after)).toEqual([["a", p, "3"]]);
|
||||
});
|
||||
|
||||
it("survives recovery from the last base batch", () => {
|
||||
const enc = encoder();
|
||||
const t = track<Record<string, any>>({ a: { deep: "" }, b: { deep: "" } });
|
||||
t.flush(); // drain the base batch
|
||||
const wire: WireOp[][] = [];
|
||||
for (let i = 0; i < 8; i++) {
|
||||
t.state.a.deep += `x${i}`;
|
||||
t.state.b.deep += `y${i}`;
|
||||
if (i === 5) t.rebase();
|
||||
wire.push(enc.encode(t.flush()));
|
||||
}
|
||||
const lastBase = wire.map(isBase).lastIndexOf(true);
|
||||
const dec = decoder();
|
||||
let replica: unknown;
|
||||
for (const w of wire.slice(lastBase)) {
|
||||
const ops = dec.decode(w);
|
||||
replica = replica === undefined
|
||||
? structuredClone((ops[0] as ["r", unknown])[1])
|
||||
: apply(replica, ops);
|
||||
}
|
||||
expect(replica).toEqual(t.target);
|
||||
});
|
||||
|
||||
it("round-trips random streams", () => {
|
||||
for (let round = 0; round < 300; round++) {
|
||||
const t = track<Record<string, any>>({ a: { p: "", q: "" }, b: [] as unknown[], c: 0 });
|
||||
t.flush(); // drain the base batch
|
||||
const batches: Op[][] = [];
|
||||
for (let i = 0; i < 8; i++) {
|
||||
const r = Math.random();
|
||||
if (r < 0.3) t.state.a.p += "x";
|
||||
else if (r < 0.5) t.state.a.q += "y";
|
||||
else if (r < 0.65) t.state.b.push(i);
|
||||
else if (r < 0.8) t.state.c = i;
|
||||
else if (r < 0.9) delete t.state.c;
|
||||
else t.rebase();
|
||||
const ops = t.flush();
|
||||
if (ops.length > 0) batches.push(ops);
|
||||
}
|
||||
expect(roundTrip(batches)).toEqual(batches);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("property: random round-trip", () => {
|
||||
// 3000 random sequences found the root-splice hole on the first run, before any
|
||||
// hand-written case did.
|
||||
const rnd = (d = 0): unknown => {
|
||||
const r = Math.random();
|
||||
if (d > 2 || r < 0.3) return Math.floor(Math.random() * 5);
|
||||
if (r < 0.45) return ["x", "y", null, true][Math.floor(Math.random() * 4)];
|
||||
if (r < 0.7) return Array.from({ length: Math.floor(Math.random() * 4) }, () => rnd(d + 1));
|
||||
const o: Record<string, unknown> = {};
|
||||
for (const k of ["a", "b", "c"]) if (Math.random() < 0.6) o[k] = rnd(d + 1);
|
||||
return o;
|
||||
};
|
||||
|
||||
it("a replica always matches the producer", () => {
|
||||
let checked = 0;
|
||||
for (let i = 0; i < 3000; i++) {
|
||||
const base = rnd();
|
||||
if (typeof base !== "object" || base === null) continue;
|
||||
const t = track(structuredClone(base) as object);
|
||||
t.flush(); // drain the base batch
|
||||
const next = rnd();
|
||||
if (Array.isArray(t.state) && Array.isArray(next)) {
|
||||
(t.state as unknown[]).splice(0, (t.state as unknown[]).length, ...next);
|
||||
} else if (!Array.isArray(t.state) && typeof next === "object" && next !== null && !Array.isArray(next)) {
|
||||
const s = t.state as Record<string, unknown>;
|
||||
for (const k of Object.keys(s)) if (!(k in next)) delete s[k];
|
||||
for (const [k, v] of Object.entries(next)) s[k] = v;
|
||||
} else continue;
|
||||
expect(apply(structuredClone(base), t.flush())).toEqual(t.target);
|
||||
checked++;
|
||||
}
|
||||
// Most random pairs are shape-incompatible and skipped; this only guards
|
||||
// against the loop silently checking nothing.
|
||||
expect(checked).toBeGreaterThan(200);
|
||||
});
|
||||
});
|
||||
+285
@@ -0,0 +1,285 @@
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// harness/delta — intent-recording change tracking over plain JSON.
|
||||
//
|
||||
// Depends on nothing else in the harness. Session storage, the runtime and the
|
||||
// facet host consume it; keep the arrows pointing that way.
|
||||
//
|
||||
// State is plain TS. You mutate it normally. Ops come out carrying what you
|
||||
// *did*, not what changed — `text += chunk` yields an append, not a 50 KB
|
||||
// replacement; `arr.unshift(x)` yields one splice, not O(n) index writes.
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
export type JsonValue = null | boolean | number | string | JsonValue[] | { [k: string]: JsonValue };
|
||||
|
||||
export type Seg = string | number;
|
||||
export type Path = readonly Seg[];
|
||||
|
||||
/** Non-empty: `s`/`d`/`a`/`t` cannot target the root. Enforced by the type. */
|
||||
export type NonEmptyPath = readonly [Seg, ...Seg[]];
|
||||
|
||||
/** A path inline on first use, an interned id thereafter. */
|
||||
export type PathRef<P extends Path = Path> = P | number;
|
||||
|
||||
/**
|
||||
* Tuples are the form — in memory, on the wire, on disk. There is no keyed
|
||||
* variant and no codec between them.
|
||||
*
|
||||
* `r` is the ONLY op that replaces a whole value. It is an op rather than a
|
||||
* frame kind for the same reason `a` and `t` exist: they are specialisations of
|
||||
* `s` that are smaller and state intent. It also makes base detection a token
|
||||
* comparison rather than a path inspection, which matters because recovery stops
|
||||
* at the last base batch — a correctness boundary.
|
||||
*
|
||||
* Only `p` may target the root, and only because a tracked value can itself be
|
||||
* an array. A `p` covering its entire target is normalised at RECORD time to `r`
|
||||
* or `s`, so a root `p` is always a partial modification.
|
||||
*
|
||||
* The two-element forms reuse the previous op's path; arity disambiguates.
|
||||
*/
|
||||
export type Op =
|
||||
| readonly ["r", JsonValue]
|
||||
| readonly ["s", NonEmptyPath, JsonValue]
|
||||
| readonly ["d", NonEmptyPath]
|
||||
| readonly ["a", NonEmptyPath, string]
|
||||
| readonly ["t", NonEmptyPath, number]
|
||||
| readonly ["p", Path, number, number, JsonValue[]];
|
||||
|
||||
/**
|
||||
* What crosses a boundary. Adds path interning and arity omission, and nothing
|
||||
* else. `["r", value]` carries no path, so it encodes to itself — which is why
|
||||
* `isBase` works on either vocabulary.
|
||||
*/
|
||||
export type WireOp =
|
||||
| readonly ["r", JsonValue]
|
||||
| readonly ["s", PathRef<NonEmptyPath>, JsonValue] | readonly ["s", JsonValue]
|
||||
| readonly ["d", PathRef<NonEmptyPath>] | readonly ["d"]
|
||||
| readonly ["a", PathRef<NonEmptyPath>, string] | readonly ["a", string]
|
||||
| readonly ["t", PathRef<NonEmptyPath>, number] | readonly ["t", number]
|
||||
| readonly ["p", PathRef, number, number, JsonValue[]]
|
||||
| readonly ["p", number, number, JsonValue[]]
|
||||
| readonly ["#", number, Path];
|
||||
|
||||
/**
|
||||
* There is no frame type. What travels is `Op[]`.
|
||||
*
|
||||
* A **base batch** is one whose first op is `r`; the first batch of a stream is
|
||||
* always one, and storage tags it "base" so recovery can stop there. Sequencing
|
||||
* lives on the transport — a list element's `seq`, or an SSE `id:` — never in the
|
||||
* payload. See `delta.md` §6.
|
||||
*/
|
||||
|
||||
// ─── Safety ──────────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Path segments that reach the prototype chain. Rejected at record time AND at
|
||||
* apply time, including through an interned id.
|
||||
*
|
||||
* Reserved as SEGMENTS, not as values: an object with a literal `__proto__` key
|
||||
* replicates fine as a whole value; only a path walking through it is refused.
|
||||
* That is a real restriction on what is mutable — see `delta.md` §7.1.
|
||||
*/
|
||||
export declare const RESERVED_SEGMENTS: ReadonlySet<string>;
|
||||
|
||||
export declare class UnsafePathError extends Error {
|
||||
readonly segment: Seg;
|
||||
}
|
||||
|
||||
/** Rejects reserved segments, symbol keys, and negative or non-integer indices. */
|
||||
export declare function assertSafePath(path: Path): void;
|
||||
|
||||
/**
|
||||
* Rejects anything `JSON.stringify` would not round-trip: Map, Set, Date, RegExp,
|
||||
* typed arrays, class instances, functions, BigInt, cycles.
|
||||
*
|
||||
* `structuredClone` is NOT a JSON check — it clones a Map happily, and the op then
|
||||
* carries `{}` while the producer keeps a real Map. Silent divergence.
|
||||
*/
|
||||
export declare function assertJsonValue(value: unknown): asserts value is JsonValue;
|
||||
|
||||
/**
|
||||
* Validate an op's structure before applying it. A decoder must not trust tuple
|
||||
* shape: an unvalidated applier spreads a string into an array given
|
||||
* `["p",["xs"],0,0,"not-an-array"]`, and writes at the root given a string path.
|
||||
*
|
||||
* An unknown verb is an error, not a no-op — skipping it is how a newer producer's
|
||||
* op vanishes and a replica drifts. This is the CVE-2025-55182 lesson
|
||||
* (`delta.md` §7.3).
|
||||
*/
|
||||
export declare function assertValidOp(op: unknown): asserts op is Op;
|
||||
|
||||
/**
|
||||
* The same, for the wire grammar, where ids and short forms are legal.
|
||||
* `decode` uses this; `apply` uses `assertValidOp`. Validating a decoded op
|
||||
* against the wire grammar is laxer than its type — a two-element `["s", value]`
|
||||
* would pass and then be read as a path.
|
||||
*/
|
||||
export declare function assertValidWireOp(op: unknown): asserts op is WireOp;
|
||||
|
||||
// ─── Classification ─────────────────────────────────────────────────────────
|
||||
//
|
||||
// Ops are tuples, so classification is easy to hand-roll and easy to get wrong.
|
||||
// `op[0] === "s"` is not enough — a set on any path is "s"; what makes a base is
|
||||
// that it targets the ROOT. And because a tuple may omit the PathRef (arity
|
||||
// disambiguates), `op[1]` is a PathRef or a value depending on length. These
|
||||
// exist so nobody writes that comparison twice.
|
||||
|
||||
/** True for the `r` op. */
|
||||
export declare function isReplace(op: Op): boolean;
|
||||
|
||||
/**
|
||||
* True when this batch begins with `r` — a **base batch**. Exact rather than a
|
||||
* heuristic, because flush guarantees `r` appears at index 0 or not at all.
|
||||
* Storage tags such elements `"base"` so recovery can stop at the last one.
|
||||
*/
|
||||
export declare function isBase(ops: readonly (Op | WireOp)[]): boolean;
|
||||
|
||||
// ─── Codec ───────────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Path interning and arity omission. ONE PAIR PER STREAM — the id table spans a
|
||||
* whole subscription or file, so a consumer joining later needs its own.
|
||||
*
|
||||
* The table resets on a base batch, because a reader replays from the last one
|
||||
* with a fresh decoder and everything after it must be self-contained.
|
||||
*/
|
||||
export declare function encoder(): { encode(ops: readonly Op[]): WireOp[] };
|
||||
export declare function decoder(): { decode(wire: readonly WireOp[]): Op[] };
|
||||
|
||||
// ─── Tracker ─────────────────────────────────────────────────────────────────
|
||||
|
||||
export interface TrackerOptions {
|
||||
/**
|
||||
* Cap on characters compared when deriving append/truncate from a string
|
||||
* write. A window slide's overlap lies near the end, so a cap bounds cost
|
||||
* without changing the result. Default 65536; 0 disables detection.
|
||||
*/
|
||||
maxOverlapScan?: number;
|
||||
/** Merge compatible adjacent ops on flush. Default true. */
|
||||
coalesce?: boolean;
|
||||
}
|
||||
|
||||
export interface Tracker<T extends object> {
|
||||
/**
|
||||
* Mutate this. Plain TS: assignment, push, splice, delete, all of it.
|
||||
*
|
||||
* Assigning to it replaces the whole value: emits `["r", next]` and discards
|
||||
* ops recorded before it. It must be a SETTER on the tracker — a plain
|
||||
* property would be overwritten, swapping the proxy for a plain object and
|
||||
* silently ending tracking.
|
||||
*
|
||||
* To force a base batch without changing the value, call `rebase()`.
|
||||
*/
|
||||
state: T;
|
||||
/** The untracked backing object. Reads only — writing here emits nothing. */
|
||||
readonly target: T;
|
||||
/**
|
||||
* Drain. Empty when nothing changed.
|
||||
*
|
||||
* The FIRST flush is always a base batch, carrying the value including any
|
||||
* mutations made before it — a consumer starts with nothing, so a stream must
|
||||
* open with something to apply deltas to.
|
||||
*
|
||||
* Ops that a later op makes unreachable are dropped first — a field written
|
||||
* three times emits one op, and a parent replaced after its child emits one.
|
||||
* Object key order is not preserved by that pass; see `delta.md` §5.
|
||||
*/
|
||||
flush(): Op[];
|
||||
/** Discard without emitting. For a producer that just sent a base batch. */
|
||||
discard(): void;
|
||||
get dirty(): boolean;
|
||||
}
|
||||
|
||||
export declare function track<T extends object>(root: T, options?: TrackerOptions): Tracker<T>;
|
||||
|
||||
// ─── Applier ─────────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Apply ops to a plain mutable object. Five verbs plus the dictionary entry, no
|
||||
* domain knowledge, no library, no registry lookup.
|
||||
*
|
||||
* Returns the value, because a root set replaces it outright and cannot be done
|
||||
* in place. Throws `PathError` when a path does not resolve, which a consumer
|
||||
* treats exactly like a sequence gap: request a fresh base frame.
|
||||
*
|
||||
* Must not be pointed at a value produced by Immer, which deep-freezes.
|
||||
*/
|
||||
/**
|
||||
* Fold ops into a value and return it — returned rather than mutated in place,
|
||||
* because `r` replaces the value outright. Tolerates an `undefined` target,
|
||||
* since a root op never reads it, so a consumer needs no base-batch branch.
|
||||
*
|
||||
* An `r` payload is **adopted, not copied**: the consumer owns the batch it was
|
||||
* handed. Do not hand one batch to two in-process consumers, or their replicas
|
||||
* will alias.
|
||||
*/
|
||||
export declare function apply<T>(target: T, ops: readonly Op[]): T;
|
||||
|
||||
export declare class PathError extends Error {
|
||||
readonly path: Path;
|
||||
}
|
||||
|
||||
// ─── Replica ─────────────────────────────────────────────────────────────────
|
||||
|
||||
export interface Replica<T> {
|
||||
/** Current value. `undefined` until the first base frame. */
|
||||
readonly value: T | undefined;
|
||||
/**
|
||||
* Feed one batch, with the sequence number the transport supplied. Returns
|
||||
* "gap" when `seq` is not contiguous or a path failed to resolve; the caller
|
||||
* then requests a fresh base batch.
|
||||
*/
|
||||
receive(ops: readonly Op[], seq: number): "ok" | "gap";
|
||||
}
|
||||
|
||||
export declare function replica<T>(): Replica<T>;
|
||||
|
||||
// ─── Semantics ───────────────────────────────────────────────────────────────
|
||||
//
|
||||
// ROOT OPS
|
||||
// `["s", [], v]` replaces the value; `["p", [], …]` splices it when the value
|
||||
// is an array. The applier must special-case `path.length === 0` before
|
||||
// resolving a parent — a prototype handling only a root set threw on the first
|
||||
// randomised sequence that produced a root splice.
|
||||
//
|
||||
// STRINGS
|
||||
// A write of `next` over `prev` derives ops from the longest suffix of `prev`
|
||||
// that is a prefix of `next`:
|
||||
// overlap === prev.length -> append(next.slice(overlap))
|
||||
// overlap > 0 -> truncate(prev.length - overlap) + append(rest)
|
||||
// overlap === 0 -> set(next)
|
||||
// Always correct: the overlap is verified, not guessed. This is what makes a
|
||||
// rolling window cheap — the case every effect-recording library degrades on.
|
||||
//
|
||||
// Use a native probe, not a hand-written KMP: `indexOf` a short head of `next`
|
||||
// in `prev`, verify with `endsWith`. Measured 47x faster over 2000 slides on a
|
||||
// 50 KB string, verified identical across 200.
|
||||
//
|
||||
// `t` counts UTF-16 code units, matching String.prototype.slice. Not bytes.
|
||||
//
|
||||
// ARRAYS
|
||||
// Mutator methods are intercepted in the `get` trap, so intent is recorded
|
||||
// before the engine performs index writes:
|
||||
// push/unshift/pop/shift/splice -> one `p`
|
||||
// sort/reverse/fill/copyWithin -> `s` of the whole array (rare)
|
||||
// Direct index writes are honest `s` ops. `a.length = n` is translated —
|
||||
// shrink to a splice, grow to a splice of nulls — never emitted as a path,
|
||||
// because `length` is not a document location.
|
||||
//
|
||||
// TRANSACTIONS
|
||||
// Several writes to one address in one commit must each be recorded against
|
||||
// the previous one's result, not against pre-transaction state. The applier
|
||||
// replays in order; anything else applies the second against a stale base.
|
||||
//
|
||||
// ALIASING
|
||||
// Op payloads are structured-cloned at record time. Without this a later
|
||||
// mutation retroactively changes an already-emitted op and a local replay
|
||||
// aliases the source — an observed bug, not a hypothetical.
|
||||
//
|
||||
// UNDEFINED
|
||||
// `x = undefined` normalises to `d`. JSON has no undefined, and a set with a
|
||||
// missing value is indistinguishable from a lost value after a round trip.
|
||||
//
|
||||
// SCOPE
|
||||
// JsonValue only. No Map, Set, Date, RegExp, class instances, getters, symbol
|
||||
// keys, or cycles. That keeps this to five traps and one string algorithm, and
|
||||
// makes structuredClone on payloads always safe.
|
||||
Reference in new issue
Block a user