Squashed 'third_party/pi/' content from commit d981de122
git-subtree-dir: third_party/pi git-subtree-split: d981de1229ef899957bbe968bc8dcda02a21f477
This commit is contained in:
83
packages/agent/docs/work-packages/00-runtime1-removal.md
Normal file
83
packages/agent/docs/work-packages/00-runtime1-removal.md
Normal file
@@ -0,0 +1,83 @@
|
||||
# WP00 — Runtime1 removal
|
||||
|
||||
## Status
|
||||
|
||||
Complete. No tag. Runtime2 is the sole public implementation. Do not release while its execution path is incomplete.
|
||||
|
||||
## Goal
|
||||
|
||||
Make runtime2 the sole public harness implementation, delete runtime1 and its obsolete tests, then stop before adding runtime2 behavior.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- The approved acceptance/hook redesign and durability handoffs are present in `harness.md`, `values.md`, `assistant-durability.md`, and `tool-durability.md`.
|
||||
- Existing tests are evidence, not authority.
|
||||
|
||||
## Work, in order
|
||||
|
||||
1. **Reconcile the contract.** Fold the approved acceptance/hook redesign into `harness.md`, including durable `starting`, atomic hook-free acceptance, driver-owned `before_run`, `before_drive`, request-local system-prompt transformation, trusted restore, and removal of process-origin activation semantics. Audit §§0.4, 1.2, 3.1–3.6, 4.1–4.2, 4.5, 5.1–5.2, 5.5–5.6, and Parts 8–9; remove every stale `BeforeResumePrepared`, `before_resume`, `resumeData`, `systemPromptOverride`, stable-ID routing, reservation, and `fresh | continue | resume` activation reference. Delete obsolete runtime planning documents once `harness.md` and linked handoffs own the active contract.
|
||||
2. **Harvest before deletion.** Inspect `agent-harness-runtime.test.ts`, `agent-harness-r2/r3/r4.test.ts`, and old `restore.test.ts`. Preserve unique scenarios in the detailed future rows or a temporary categorized inventory; explicitly discard old reservation, `before_resume`, `resumeData`, persisted hook prompt override, semantic restore audit, and pre-`outcome_ready` tool-crash behavior.
|
||||
3. **Remove runtime1-only public members.** Delete `before_resume`, `BeforeResumePrepared`, `resumeData`, `systemPromptOverride`, and stable hook-ID routing. Add the approved `before_drive` and `transform_context` shapes. Update telemetry schema source and regenerate its document. Do not implement `starting` or acceptance behavior here.
|
||||
4. **Switch the factory.** Add `packages/agent/src/harness/runtime2/index.ts`, point `agent-harness.ts` at it, and add a constructor-selection regression. Verify the experimental coding-agent worker still creates the harness, subscribes to events, and closes it.
|
||||
5. **Delete runtime1 source and tests** listed below.
|
||||
6. **Update `[Unreleased]` on `main` or a pull-request branch** for the public breaking removals and temporarily incomplete factory. Repository policy forbids changelog edits on `dev`, so WP00 records but does not perform that release-facing step here.
|
||||
7. Run the retained tests and checks. Fix every failure; do not restore compatibility shims.
|
||||
|
||||
## Delete
|
||||
|
||||
```text
|
||||
packages/agent/src/harness/runtime/**
|
||||
packages/agent/src/harness/restore.ts
|
||||
packages/agent/test/harness/agent-harness-runtime.test.ts
|
||||
packages/agent/test/harness/agent-harness-r2.test.ts
|
||||
packages/agent/test/harness/agent-harness-r3.test.ts
|
||||
packages/agent/test/harness/agent-harness-r4.test.ts
|
||||
packages/agent/test/harness/restore.test.ts
|
||||
packages/agent/test/harness/scratch/r1.ts
|
||||
packages/agent/test/harness/scratch/r2.ts
|
||||
packages/agent/test/harness/scratch/r3.ts
|
||||
packages/agent/test/harness/scratch/r4.ts
|
||||
```
|
||||
|
||||
## Retain
|
||||
|
||||
- all `test/harness/runtime2/**` tests;
|
||||
- Session, Branch, storage, repository, backend-conformance, and instrumentation tests;
|
||||
- execution assistant/tool/primitives tests;
|
||||
- config, hooks, events, telemetry, compaction, and branch-summary code/tests;
|
||||
- `types.test.ts`, updated for the reduced public contract;
|
||||
- `packages/agent/src/agent-loop.ts` unchanged.
|
||||
|
||||
Do not parameterize obsolete runtime1 suites against runtime2 and do not keep a runtime1 smoke suite.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- No source import references `harness/runtime/*`.
|
||||
- Public `AgentHarness.create()` selects runtime2.
|
||||
- Runtime2 creation, events, inspection, close, and fault tests pass.
|
||||
- These coding-agent tests pass:
|
||||
- `experimental-remote-runtime.test.ts`
|
||||
- `experimental-session-worker-manager.test.ts` (the upstream replacement for the removed `experimental-session-worker.test.ts`)
|
||||
- `experimental-session-worker-lifecycle.test.ts`
|
||||
- Every modified test passes individually.
|
||||
- Agent and root TypeScript pass.
|
||||
- `git diff --check` and `npm run check` pass.
|
||||
|
||||
## Outcome
|
||||
|
||||
- The accepted hook/drive contract is normative in `harness.md`; stale acceptance/resume contracts are absent.
|
||||
- Runtime1 source, validating restore, obsolete suites, and R1–R4 scratch scenarios are deleted.
|
||||
- `AgentHarness.create()` resolves through `runtime2/index.ts`; a constructor-selection regression proves it.
|
||||
- The scenario harvest added missing tool-close, identity-preflight, recovery-ordering, turn-bracket, and telemetry cases to future rows.
|
||||
- Upstream added two real remote prompt tests after this handoff was drafted. They remain present but skipped with an R2 re-enable requirement because runtime2 execution is intentionally incomplete. Worker creation, attachment, lifecycle, operation correlation, and close coverage passes.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No bound-value/list implementation.
|
||||
- No implementation of `starting` or atomic acceptance.
|
||||
- No drive owner, provider, retry, deferred, or tool execution.
|
||||
- No runtime1 parity work, compatibility layer, archaeology tag, or release.
|
||||
|
||||
## Stop condition
|
||||
|
||||
Stop once runtime1 is absent, runtime2 is the public factory, retained coverage is green, and all checks pass. Report the deletion and harvested scenarios; do not begin another work package.
|
||||
327
packages/agent/docs/work-packages/01-bound-values-lists.md
Normal file
327
packages/agent/docs/work-packages/01-bound-values-lists.md
Normal file
@@ -0,0 +1,327 @@
|
||||
# WP01 — Bound values and lists
|
||||
|
||||
## Status
|
||||
|
||||
Complete. `harness.md` is normative. [`values.md`](../values.md) supplies the detailed address, backend, and conformance design.
|
||||
|
||||
## Goal
|
||||
|
||||
Replace the retained register/custom-state storage surface with bound `Value<T>` and `ValueList<T>` addresses across Session, Memory, JSONL, SQLite, instrumentation, tests, and public application access. Stop before runtime execution consumers.
|
||||
|
||||
## Decisions fixed for this package
|
||||
|
||||
1. **Core and application namespaces.** Every core address uses the exact documented `pi.*` namespace. Applications use their own non-reserved namespaces through the same public `value()` / `list()` constructors. `fact.custom` and its API are deleted, not renamed to a built-in custom namespace.
|
||||
2. **Forks.** Generic forks copy only explicitly handled core addresses: Branch tip/lane configuration plus fresh lane state, session name, and labels whose targets copy. They copy no `pi.op.*`, `pi.pending.*`, list, ledger, or arbitrary application address. A later application feature must add an address-specific fork policy before relying on copied application state.
|
||||
3. **Trusted kind discipline.** Using one `(namespace, key)` as both a value and a list is a trusted-programming defect. Backends do not add cross-kind collision checks, triggers, registries, or catalogs.
|
||||
4. **Operation names.** Keep source `OperationMeta` for immutable acceptance metadata and `Operation` for the process-local `{ meta: OperationMeta, state: OperationState }` projection. `operationMeta(id)` binds `Value<OperationMeta>`; the composite is never persisted as one value.
|
||||
5. **Query ordering and bounds.** `scanValues()` returns key-ascending results. `readList()` limits only one query page, never total list length or bytes: reject non-positive or non-safe limits, default to 1,000, and clamp larger values to 10,000.
|
||||
6. **No WIP compatibility.** Replace the unfinished format-4 storage schema in place. JSONL remains format 4/storage version 1 but accepts only the new value/list records. SQLite keeps `SQLITE_STORAGE_VERSION = 1`, edits `001_initial.sql` in place, renames `registers` to `scalar_values`, and adds `list_values`. Pre-WP01 format-4 JSONL and SQLite files are unsupported. Add no migration runner or legacy decoder.
|
||||
7. **Generic infrastructure only.** WP01 defines every built-in value/list constructor, including future assistant/tool addresses, but does not implement their runtime consumers.
|
||||
|
||||
## Public and storage contract
|
||||
|
||||
Add `packages/agent/src/harness/session/values.ts` with:
|
||||
|
||||
- invariant `Value<T>` and `ValueList<T>` address types;
|
||||
- universal `value<T>(namespace, key?)` and `list<T>(namespace, key?)` constructors;
|
||||
- namespace/key validation only: non-empty namespace and no `\u0000` component;
|
||||
- `StoredValue<T>`, `ListElement<T>`, `ListCursor`, and `ListReadOptions`;
|
||||
- typed `setValue`, `deleteValue`, `appendList`, and `deleteList` write helpers using `NoInfer<T>`;
|
||||
- every exact built-in constructor and the five scan-prefix constructors from `values.md`.
|
||||
|
||||
Replace the old API throughout:
|
||||
|
||||
```ts
|
||||
getRegister(namespace, key) -> getValue(address)
|
||||
listRegisters(namespace, prefix) -> scanValues(prefixAddress)
|
||||
register set/delete writes -> typed value helpers
|
||||
```
|
||||
|
||||
Storage and the historical Session reader, mutator, tree-view, and repository surfaces expose the same bound-address reads:
|
||||
|
||||
```ts
|
||||
getValue<T>(address: Value<T>): Promise<StoredValue<T> | undefined>;
|
||||
scanValues<T>(prefix: Value<T>): Promise<StoredValue<T>[]>;
|
||||
readList<T>(address: ValueList<T>, options?: ListReadOptions): Promise<ListElement<T>[]>;
|
||||
```
|
||||
|
||||
The historical tree-view and Session surfaces additionally expose one-commit direct writes:
|
||||
|
||||
```ts
|
||||
setValue<T>(address: Value<T>, next: NoInfer<T>): Promise<void>;
|
||||
deleteValue<T>(address: Value<T>): Promise<void>;
|
||||
appendList<T>(address: ValueList<T>, element: NoInfer<T>): Promise<void>;
|
||||
deleteList<T>(address: ValueList<T>): Promise<void>;
|
||||
```
|
||||
|
||||
`SessionMutator` retains one explicit `commit(writes)` and does not gain direct committing methods. Each write array composes helper-constructed value/list writes with entries and usage.
|
||||
|
||||
Keep `getName` / `setName` and `getLabel` / `setLabel` as wrappers over `sessionName` and `entryLabel(id)`. Delete `getCustomFact` / `setCustomFact`. Rename the public passive metadata event from `fact_update` to the `value_update` shape already specified by `harness.md`; it covers only session-name and entry-label wrappers, not arbitrary application writes.
|
||||
|
||||
## Files
|
||||
|
||||
### Add
|
||||
|
||||
- `packages/agent/src/harness/session/values.ts`
|
||||
- `packages/agent/test/harness/values.test.ts`
|
||||
- `packages/session-backends/sqlite-node/src/sqlite/session/values.ts`
|
||||
|
||||
### Delete or rename
|
||||
|
||||
- delete `packages/session-backends/sqlite-node/src/sqlite/session/registers.ts` after moving its scalar behavior to `values.ts`;
|
||||
- remove all register/global-map/custom-fact declarations from `packages/agent/src/harness/session/types.ts`.
|
||||
|
||||
### Agent source
|
||||
|
||||
- `packages/agent/src/harness/agent-harness.ts`
|
||||
- `packages/agent/src/harness/session/types.ts`
|
||||
- `packages/agent/src/harness/session/commit.ts`
|
||||
- `packages/agent/src/harness/session/storage-state.ts`
|
||||
- `packages/agent/src/harness/session/memory.ts`
|
||||
- `packages/agent/src/harness/session/session.ts`
|
||||
- `packages/agent/src/harness/session/fork.ts`
|
||||
- `packages/agent/src/harness/session/index.ts`
|
||||
- `packages/agent/src/harness/session/jsonl/storage.ts`
|
||||
- `packages/agent/src/harness/session/jsonl/repo.ts`
|
||||
- `packages/agent/src/harness/session/testing/storage-decorator.ts`
|
||||
- `packages/agent/src/harness/session/testing/instrumented-storage.ts`
|
||||
- `packages/agent/src/harness/session/testing/types.ts`
|
||||
- `packages/agent/src/harness/session/testing/conformance/storage.ts`
|
||||
- `packages/agent/src/harness/session/testing/conformance/session-repo.ts`
|
||||
- `packages/agent/src/harness/session/testing/benchmark/storage.ts`
|
||||
- `packages/agent/src/harness/session/testing/benchmark/session-repo.ts`
|
||||
- `packages/agent/src/harness/session/testing/index.ts`
|
||||
- `packages/agent/src/harness/runtime2/restore.ts`
|
||||
- `packages/agent/src/harness/runtime2/harness.ts`
|
||||
- `packages/agent/src/harness/runtime2/lane.ts`
|
||||
- `packages/agent/src/harness/telemetry.ts`
|
||||
- `packages/agent/src/index.ts` and `packages/agent/src/node.ts` only as needed to verify the new public exports; do not add a second export path.
|
||||
|
||||
### Agent tests and generated documentation
|
||||
|
||||
- `packages/agent/test/harness/memory-storage.test.ts`
|
||||
- `packages/agent/test/harness/memory-conformance.test.ts`
|
||||
- `packages/agent/test/harness/memory-session-repo.test.ts`
|
||||
- `packages/agent/test/harness/jsonl-storage.test.ts`
|
||||
- `packages/agent/test/harness/jsonl-storage-conformance.test.ts`
|
||||
- `packages/agent/test/harness/jsonl-session-repo.test.ts`
|
||||
- `packages/agent/test/harness/jsonl-session-repo-conformance.test.ts`
|
||||
- `packages/agent/test/harness/storage-backed-session.test.ts`
|
||||
- `packages/agent/test/harness/session-tree.test.ts`
|
||||
- `packages/agent/test/harness/session-create-lane.test.ts`
|
||||
- `packages/agent/test/harness/instrumented-storage.test.ts`
|
||||
- `packages/agent/test/harness/types.test.ts`
|
||||
- `packages/agent/test/harness/telemetry.test.ts`
|
||||
- `packages/agent/test/harness/runtime2/harness.test.ts`
|
||||
- `packages/agent/test/harness/runtime2/lane.test.ts`
|
||||
- `packages/agent/test/harness/runtime2/restore.test.ts`
|
||||
- regenerated `packages/agent/docs/telemetry-schema.md`
|
||||
|
||||
### SQLite backend
|
||||
|
||||
- `packages/session-backends/sqlite-node/src/sqlite/migrations/001_initial.sql`
|
||||
- `packages/session-backends/sqlite-node/src/sqlite/repo.ts`
|
||||
- `packages/session-backends/sqlite-node/src/sqlite/session.ts`
|
||||
- `packages/session-backends/sqlite-node/src/sqlite/storage.ts`
|
||||
- `packages/session-backends/sqlite-node/src/sqlite/index.ts` if needed for the renamed module
|
||||
- `packages/session-backends/sqlite-node/test/storage.test.ts`
|
||||
- `packages/session-backends/sqlite-node/test/storage-conformance.test.ts`
|
||||
- `packages/session-backends/sqlite-node/test/repo.test.ts`
|
||||
- `packages/session-backends/sqlite-node/test/adapter.test.ts`
|
||||
- `packages/session-backends/sqlite-node/test/sql.test.ts`
|
||||
|
||||
### Coding-agent consumer
|
||||
|
||||
- `packages/coding-agent/test/experimental-session-support.ts`
|
||||
- verify `packages/coding-agent/test/experimental-remote-runtime.test.ts`
|
||||
- verify `packages/coding-agent/test/experimental-server-replacement.test.ts`
|
||||
|
||||
If the final old-API grep identifies another retained source/test call site, it belongs to WP01; do not add a compatibility shim to avoid touching it.
|
||||
|
||||
## Work, in order
|
||||
|
||||
1. **Add the address vocabulary.** Implement `values.ts`, export it through the existing session/root path, and add focused compile-time/runtime address tests. Include exact built-in namespace/key/kind tests and prefix-constructor tests before migrating callers.
|
||||
2. **Cut the shared API once.** Replace register types and writes in `types.ts`/`commit.ts`; split `StorageState` into current scalar values and surviving list elements; implement Memory reads/writes, ordered prefix scans, paged list reads, transaction validation/application, snapshots, and direct Session methods. Remove custom-fact APIs and migrate name/label wrappers.
|
||||
3. **Migrate JSONL and generic fork/snapshot code.** Encode only `kind:"value"` and `kind:"list"`; replay set/delete/append/delete; preserve transaction-line torn-tail atomicity; serialize surviving list elements with original `seq` merged in global sequence order; preserve the sequence high-water mark. This extends existing snapshot serialization only—do not add a new compaction trigger or precise-rewrite feature. Forks copy the fixed core set from Decisions item 2, re-sequence destination scalar values after copied entries as today, and copy no lists.
|
||||
4. **Migrate instrumentation, conformance, and benchmarks.** The storage decorator exposes all three reads; instrumented storage records erased value/list writes in exact order without content telemetry. Extend shared conformance before backend-specific assertions.
|
||||
5. **Migrate runtime2 shell call sites.** Replace lane/harness raw writes with built-in constructors/helpers. `restore.ts` uses `scanValues(branchTipInventoryPrefix())` plus exact `getValue` lookups and performs no `readList()` call. Do not add acceptance, drive, hydration, or cleanup behavior.
|
||||
6. **Replace the SQLite WIP schema and adapter.** Edit `001_initial.sql` in place, implement scalar operations and indexed list append/delete/paging in `session/values.ts`, keep every write inside the existing `BEGIN IMMEDIATE` writer-lease transaction, update both fork snapshot paths, and retain all entry/usage/branch/lease behavior from current `dev`.
|
||||
7. **Migrate public events, tests, and the coding-agent helper.** Remove old type assertions and raw namespaces. Change `fact_update` to `value_update`. Keep the two remote prompt tests skipped for the existing WP00 reason; WP01 must not alter runtime execution.
|
||||
8. **Update telemetry and documentation.** Change `pi.session.write` item kinds from `register` to `value` and `list`, regenerate `telemetry-schema.md`, run the old-API sweeps, and record any branch-policy-deferred changelog requirement. Do not edit a changelog on `gramps` unless it becomes a pull-request branch or the user requests it.
|
||||
|
||||
## Backend requirements
|
||||
|
||||
### Memory
|
||||
|
||||
- prepare and validate the complete transaction before mutating entries, values, lists, usage, or stats;
|
||||
- current scalar replacement stores only the latest value and set `seq`;
|
||||
- list append performs no list read and stores each global write `seq`;
|
||||
- list delete removes the whole exact key;
|
||||
- snapshots include current scalar values and surviving list elements.
|
||||
|
||||
### JSONL
|
||||
|
||||
- keep format 4/storage version 1 with no legacy register decode;
|
||||
- one single-write object or multi-write array remains one atomic line;
|
||||
- replay produces the same logical state as Memory;
|
||||
- torn final lines expose no partial transaction;
|
||||
- snapshot serialization preserves surviving list-element sequences and the next-sequence high-water mark.
|
||||
|
||||
### SQLite
|
||||
|
||||
Use:
|
||||
|
||||
```sql
|
||||
CREATE TABLE scalar_values (
|
||||
namespace TEXT NOT NULL,
|
||||
key TEXT NOT NULL,
|
||||
seq INTEGER NOT NULL,
|
||||
value TEXT NOT NULL,
|
||||
PRIMARY KEY (namespace, key)
|
||||
) WITHOUT ROWID;
|
||||
|
||||
CREATE TABLE list_values (
|
||||
namespace TEXT NOT NULL,
|
||||
key TEXT NOT NULL,
|
||||
seq INTEGER NOT NULL,
|
||||
value TEXT NOT NULL,
|
||||
PRIMARY KEY (namespace, key, seq)
|
||||
) WITHOUT ROWID;
|
||||
```
|
||||
|
||||
Ascending and descending list queries use the primary key with an exclusive sequence predicate and `LIMIT`. Add `EXPLAIN QUERY PLAN` assertions proving no table scan or temporary ordering b-tree. Do not change storage version, add migrations, or weaken the current lease/fence/fork behavior.
|
||||
|
||||
## Required coverage
|
||||
|
||||
### Address and type tests
|
||||
|
||||
- invariant address typing and inferred scalar/list result types;
|
||||
- `NoInfer` rejects incompatible set/append values;
|
||||
- scalar helpers reject list addresses and list helpers reject scalar addresses;
|
||||
- independently constructed equal addresses resolve the same location;
|
||||
- empty key works; empty namespace and `\u0000` components reject;
|
||||
- exact built-in namespaces/key grammars and exactly five prefix constructors;
|
||||
- application-wide and dynamic non-reserved addresses require no second operation-time key;
|
||||
- no registry, catalog, privilege constructor, global value map, or runtime `pi.*` gate.
|
||||
|
||||
### Shared scalar/list conformance
|
||||
|
||||
- scalar set/get/delete/delete-absent/recreate and latest set `seq`;
|
||||
- namespace-scoped, key-ascending prefix scans;
|
||||
- append one and several elements, including several appends in one transaction;
|
||||
- appends separated by unrelated writes preserve per-list order and global element sequences;
|
||||
- ascending/descending exclusive cursors;
|
||||
- default, explicit, invalid, and clamped query-page limits;
|
||||
- absent list, whole-list delete, delete-absent, and delete-then-append;
|
||||
- atomic entry + usage + value + list transactions;
|
||||
- rollback when any sibling write is invalid;
|
||||
- no list read on append;
|
||||
- close rejects later reads while already-admitted commits drain.
|
||||
|
||||
Do not add a value/list collision test: cross-kind misuse is intentionally an unenforced trusted-programming defect.
|
||||
|
||||
### Backend and repository tests
|
||||
|
||||
- identical Memory/JSONL/SQLite pages and cursors;
|
||||
- JSONL single/multi-write replay, torn-tail behavior, and sequence-preserving snapshot output;
|
||||
- SQLite query plans and writer-lease transaction behavior;
|
||||
- branch/tree forks copy session name, eligible labels, and lane configuration/Branch tip with fresh lane state;
|
||||
- forks exclude operation/pending values, all lists, application addresses, last results, queues, and ledger rows;
|
||||
- repository parent metadata, entry IDs, stats, branch indexes, v3 normalization, UUIDv7/follower IDs, and current SQLite lease/fork scenarios remain unchanged;
|
||||
- runtime2 restore enumerates lanes through the one prefix constructor and reads no list.
|
||||
|
||||
## Deferred consumers
|
||||
|
||||
The following `values.md` requirements are explicitly not WP01 coverage:
|
||||
|
||||
- assistant frame conversion, append scheduling, settlement, recovery, cancellation, snapshot hydration, and byte-growth tests (R2/R3/R6/R12);
|
||||
- invocation `getMemo` / `setMemo`, tool-output checkpoint writes, outcome cleanup, and prefix-driven operation cleanup (R4/R6);
|
||||
- any consumption-time list hydration beyond proving base restore reads no list;
|
||||
- runtime acceptance, driving, provider/tool effects, or operation-state redesign.
|
||||
|
||||
WP01 still exports `pendingAssistantFrames`, `operationToolMemo`, `pendingToolOutput`, and every cleanup prefix so later packages do not redesign storage.
|
||||
|
||||
## Removal checks
|
||||
|
||||
These must have zero matches in retained source/tests, excluding immutable released changelog history and archived prose:
|
||||
|
||||
```bash
|
||||
rg -n 'getRegister\(|listRegisters\(|RegisterValues|RegisterNamespace|RegisterSetWrite|\bRegister<' \
|
||||
packages/agent packages/session-backends/sqlite-node packages/coding-agent \
|
||||
--glob '!**/dist/**' --glob '!**/CHANGELOG.md' --glob '!**/docs/**'
|
||||
|
||||
rg -n 'kind: "register"|getCustomFact\(|setCustomFact\(|fact\.(name|label|custom)|fact_update' \
|
||||
packages/agent packages/session-backends/sqlite-node packages/coding-agent \
|
||||
--glob '!**/dist/**' --glob '!**/CHANGELOG.md' --glob '!**/docs/**'
|
||||
|
||||
rg -n '"(branch\.tip|lane\.(config|state|lastResult)|op\.(meta|state|tool_args|preparation)|pending\.entry)"' \
|
||||
packages/agent packages/session-backends/sqlite-node packages/coding-agent \
|
||||
--glob '!**/dist/**' --glob '!**/CHANGELOG.md' --glob '!**/docs/**'
|
||||
|
||||
rg -n '\bregisters\b' packages/agent/src/harness/session packages/session-backends/sqlite-node/src \
|
||||
--glob '*.ts' --glob '*.sql'
|
||||
|
||||
rg -n '"register"' \
|
||||
packages/agent/src/harness/telemetry.ts \
|
||||
packages/agent/test/harness/telemetry.test.ts \
|
||||
packages/agent/docs/telemetry-schema.md
|
||||
```
|
||||
|
||||
Do not treat unrelated model/provider/hook registration terminology as storage API residue.
|
||||
|
||||
## Validation
|
||||
|
||||
Run each created or modified test file directly and iterate until green. At minimum:
|
||||
|
||||
```bash
|
||||
# From packages/agent
|
||||
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run \
|
||||
test/harness/values.test.ts \
|
||||
test/harness/memory-storage.test.ts \
|
||||
test/harness/memory-conformance.test.ts \
|
||||
test/harness/memory-session-repo.test.ts \
|
||||
test/harness/jsonl-storage.test.ts \
|
||||
test/harness/jsonl-storage-conformance.test.ts \
|
||||
test/harness/jsonl-session-repo.test.ts \
|
||||
test/harness/jsonl-session-repo-conformance.test.ts \
|
||||
test/harness/storage-backed-session.test.ts \
|
||||
test/harness/session-tree.test.ts \
|
||||
test/harness/session-create-lane.test.ts \
|
||||
test/harness/instrumented-storage.test.ts \
|
||||
test/harness/types.test.ts \
|
||||
test/harness/telemetry.test.ts \
|
||||
test/harness/runtime2/harness.test.ts \
|
||||
test/harness/runtime2/lane.test.ts \
|
||||
test/harness/runtime2/restore.test.ts
|
||||
|
||||
# From packages/session-backends/sqlite-node
|
||||
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run \
|
||||
test/adapter.test.ts \
|
||||
test/repo.test.ts \
|
||||
test/sql.test.ts \
|
||||
test/storage.test.ts \
|
||||
test/storage-conformance.test.ts
|
||||
|
||||
# From packages/coding-agent
|
||||
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run \
|
||||
test/experimental-remote-runtime.test.ts \
|
||||
test/experimental-server-replacement.test.ts
|
||||
```
|
||||
|
||||
Then run from the repository root:
|
||||
|
||||
```bash
|
||||
cd packages/agent && npm run check:telemetry-docs
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
node_modules/.bin/tsgo --noEmit -p packages/agent/tsconfig.build.json
|
||||
node_modules/.bin/tsgo --noEmit
|
||||
git diff --check
|
||||
npm run check
|
||||
./test.sh
|
||||
```
|
||||
|
||||
Never run unrestricted Vitest, `npm test`, paid-provider tests, or `npm run build`.
|
||||
|
||||
## Stop condition
|
||||
|
||||
Stop when every retained backend and Session surface uses bound values/lists; all core addresses use the exact `pi.*` grammar; arbitrary application addresses work but generic forks exclude them; old register/fact/custom-state APIs and physical names are absent; base restore performs no list read; schema/compatibility decisions above are implemented; focused, conformance, TypeScript, telemetry-doc, diff, and repository checks pass. Report the final schema and fork behavior. Do not begin runtime acceptance, assistant/tool consumers, or any later work package.
|
||||
542
packages/agent/docs/work-packages/02-atomic-run-acceptance.md
Normal file
542
packages/agent/docs/work-packages/02-atomic-run-acceptance.md
Normal file
@@ -0,0 +1,542 @@
|
||||
# WP02 — Atomic acceptance and coherent lane observation
|
||||
|
||||
## Status
|
||||
|
||||
Complete. Phase A established minimal attachment, open-operation inventory, Session-line inspection/watch capture, commit-continuation recipient binding, and in-band identity-failure vocabulary. Phase B landed in `beac75ecc` with focused, monorepo-check, full-suite, and final Fable review passing.
|
||||
|
||||
Implementation also updated downstream protocol/coding-agent wire projections required by the changed public contract. No execution owner, provider/tool effect, timer, retry, polling, cancellation procedure, manual action, or terminal settlement was added.
|
||||
|
||||
## Goal
|
||||
|
||||
Deliver two effect-free boundaries:
|
||||
|
||||
```text
|
||||
idle lane
|
||||
→ atomic prompt/skill/template acceptance
|
||||
→ durable open operation in payload-free starting
|
||||
|
||||
open or running lane
|
||||
→ Session-line watch capture
|
||||
→ complete snapshot plus gap-free subsequent events
|
||||
```
|
||||
|
||||
After `AgentHarness.create(options, context)` succeeds:
|
||||
|
||||
- every configured lane has a complete small process-local projection;
|
||||
- `open` inventories every durable current operation without predicting model/tool availability;
|
||||
- attachment starts no hook, provider, tool, timer, breakpoint, drive owner, or application callback;
|
||||
- `inspectExecution(context)` observes the small projection and local owner on the Session line;
|
||||
- `watch(context)` registers buffering, clones live presentation, and performs bounded snapshot reads in one no-write lane job;
|
||||
- every committing lane job publishes memory and synchronously binds its event batch in the commit continuation;
|
||||
- the mutation does not await listener delivery, while the public operation does.
|
||||
|
||||
WP02 does not implement drive, provider generation, hooks, tools, retries, deferred polling, cancellation procedures, manual action execution, or terminal settlement.
|
||||
|
||||
## Decisions
|
||||
|
||||
### 1. Attachment restores projection, not presentation
|
||||
|
||||
Passing the open Session to `AgentHarness.create()` transfers orchestration ownership until create rejects or the harness closes. Direct `Session.mutate`, `Session.createLane`, reserved-address writes, and second-harness construction are prohibited during that interval, so lane inventory cannot race an out-of-band lane creation.
|
||||
|
||||
Attachment reads only:
|
||||
|
||||
- `branchTip`, `laneConfig`, `laneState`, optional `laneLastResult`;
|
||||
- `operationMeta` and `operationState` for a current operation.
|
||||
|
||||
It validates required existence, operation id/lane ownership, and intent/state kind compatibility. Projection corruption faults `create()`.
|
||||
|
||||
Attachment does not read transcript, queues, pending writes, drained payloads, deferred sources, frames, tool calls, arguments, checkpoints, preparations, memos, or staged outcomes. Those references are checked by `watch()` or drive when consumed. Missing or contradictory required payload data is terminal storage corruption and faults that consumer. Optional frame/checkpoint absence is legal.
|
||||
|
||||
### 2. Watch owns detailed snapshot reads
|
||||
|
||||
One no-write Session mutation job defines the watch boundary:
|
||||
|
||||
```text
|
||||
enter after all earlier lane jobs
|
||||
→ synchronously register buffering watcher
|
||||
→ synchronously clone live presentation state
|
||||
→ perform bounded durable reads while later lane jobs are excluded
|
||||
→ assemble snapshot
|
||||
→ release Session line
|
||||
→ return handle
|
||||
```
|
||||
|
||||
There is no special first-watch cache. The same path handles immediate post-attachment watch, reconnect, and watch during live execution.
|
||||
|
||||
The bounded read set is:
|
||||
|
||||
- one compaction-bounded branch scan from the current tip;
|
||||
- exact `pendingEntry(id)` reads for next-run, steer, follow-up, writes, and abort drains;
|
||||
- exact deferred source entry when represented;
|
||||
- tools-phase assistant entry and exact args for represented `effect_pending` calls, using `batch.turnId` as the args step id;
|
||||
- bounded exact assistant-frame pages and exact optional tool checkpoints.
|
||||
|
||||
`ToolCall.sourceIndex` is the index in the assistant message's full content array, not a filtered tool-call ordinal. A represented call must index a tool-call block.
|
||||
|
||||
### 3. Recipient binding belongs in the commit continuation
|
||||
|
||||
A successful committing lane job performs:
|
||||
|
||||
```text
|
||||
commit
|
||||
→ publish small owned projection
|
||||
→ synchronously bind recipients and append the complete `{ event, context }` batch
|
||||
→ return from the mutation without awaiting delivery
|
||||
→ await delivery before the public operation resolves
|
||||
```
|
||||
|
||||
WP02 initially implemented this with `enqueue()` plus caller-operated `start()`. WP04 replaces that gate with one immediate `emitBatch()` call in the same commit-observation continuation. Recipient binding and delivery awaiting remain unchanged. A listener or watcher registered after `emitBatch()` cannot receive that historical event.
|
||||
|
||||
The only watcher/publication orders are:
|
||||
|
||||
```text
|
||||
watcher first
|
||||
→ snapshot-before + complete buffered event batch
|
||||
|
||||
publication/`emitBatch` first
|
||||
→ snapshot-after + no old event
|
||||
```
|
||||
|
||||
A live provider/tool presentation update follows the same synchronous publish-plus-`emitBatch` discipline. Frame/checkpoint commits are lane jobs and queue behind watch capture.
|
||||
|
||||
### 4. Inspection is a no-write Session-line observation
|
||||
|
||||
`inspectExecution(context)` observes:
|
||||
|
||||
- lane and tip;
|
||||
- configured model identity as unresolved `{ provider, modelId }` strings;
|
||||
- current operation id/kind/start time;
|
||||
- process status `running`, `open`, or durable `aborting`;
|
||||
- captured model identity when the current durable phase contains one;
|
||||
- optional latest result.
|
||||
|
||||
It does not resolve model/tool registries and does not read transcript or presentation payloads.
|
||||
|
||||
### 5. Missing implementations are in-band outcomes
|
||||
|
||||
There is no `blocked`, missing-identity suspension, predictive classifier, or acceptance registry preflight.
|
||||
|
||||
At the actual execution boundary:
|
||||
|
||||
- unavailable captured model or configured active-tool definition before provider intent becomes a non-retryable configuration failure;
|
||||
- pre-intent configuration failure reserves no response/usage ids and fabricates no assistant response or usage row;
|
||||
- restored `effect_pending` settles uncertainty under its existing reserved ids before any later configuration failure;
|
||||
- unavailable deferred model durably abandons redemption through configuration failure; R7 implements and tests restored `effect_pending` abandonment, including deletion of its exact old assistant-frame list while dropping the reserved response/usage strings without fabricating settlement;
|
||||
- missing requested tool stages a direct `isError` `ToolResultMessage` and continues;
|
||||
- missing/no-longer-safe replay implementation synthesizes interruption instead of waiting.
|
||||
|
||||
Synthetic harness tool results omit `details`. A tool owns the type of its details contract; the harness must not invent `{}` or a diagnostic object. `isError` and human-readable content carry the tool-level diagnosis. Run-level configuration failure remains machine-readable through `OperationError` and `laneLastResult`.
|
||||
|
||||
Stable configuration error codes:
|
||||
|
||||
- `model_unavailable`, details `{ provider, modelId }`;
|
||||
- `configured_tools_unavailable`, details `{ tools: string[] }`.
|
||||
|
||||
`failure_drain` gains `{ kind: "configuration" }` provenance. Actual transitions land with their owning execution packages; WP02 lands the normative/source vocabulary only.
|
||||
|
||||
### 6. Acceptance is independent of process registries
|
||||
|
||||
Acceptance validates durable caller input and lane state, not current model/tool registrations. This avoids a time-of-check/time-of-use check and permits acceptance in one process followed by execution in another.
|
||||
|
||||
A misconfigured convenience prompt eventually returns a durable failed run from drive. Explicit hosted acceptance remains durable even before an execution worker loads implementations.
|
||||
|
||||
### 7. Invocation Context remains explicit
|
||||
|
||||
Every current public harness/lane operation receives trailing `context: Context`. Acceptance, attachment, watch capture, Session reads/commit, faults, and event publication preserve it. Shared harness/lane/Session receivers retain no caller Context. Context and its signal/telemetry values are never durable business data.
|
||||
|
||||
Buffered events retain the exact emitting Context. Invocation cancellation remains distinct from durable `requestAbort()`.
|
||||
|
||||
## End-state public contract
|
||||
|
||||
```ts
|
||||
export interface ModelIdentity {
|
||||
provider: string;
|
||||
modelId: string;
|
||||
}
|
||||
|
||||
export type OperationStatus = "running" | "open" | "aborting";
|
||||
|
||||
export interface OpenOperation {
|
||||
lane: string;
|
||||
operationId: string;
|
||||
kind: "run" | "compaction" | "navigation";
|
||||
startedAt: number;
|
||||
aborting?: true;
|
||||
}
|
||||
|
||||
export interface CurrentOperationInfo {
|
||||
id: string;
|
||||
kind: "run" | "compaction" | "navigation";
|
||||
startedAt: number;
|
||||
status: OperationStatus;
|
||||
capturedModel?: ModelIdentity;
|
||||
}
|
||||
|
||||
export interface LaneExecutionInfo {
|
||||
lane: string;
|
||||
tipId: string | null;
|
||||
configuredModel: ModelIdentity;
|
||||
current: CurrentOperationInfo | null;
|
||||
lastResult?: LaneLastResult;
|
||||
}
|
||||
|
||||
export interface LaneInfo {
|
||||
name: string;
|
||||
tipId: string | null;
|
||||
operation: CurrentOperationInfo | null;
|
||||
}
|
||||
|
||||
export interface AgentHarnessConstructor {
|
||||
create<TContext extends object | undefined = object | undefined>(
|
||||
options: AgentHarnessOptions<TContext>,
|
||||
context: Context,
|
||||
): Promise<{ harness: AgentHarness<TContext>; open: OpenOperation[] }>;
|
||||
}
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- `open` has exactly one item per durable current operation and omits idle lanes;
|
||||
- `aborting:true` comes only from durable `cancel_requested`;
|
||||
- `open` is inventory, not scheduling or identity advice;
|
||||
- normal applications establish watch and call `resume(context)`;
|
||||
- hosted schedulers retain expected-id `drive` fencing;
|
||||
- configured/captured identity fields are durable strings and may not resolve.
|
||||
|
||||
### Outcomes
|
||||
|
||||
Delete `MissingIdentitySuspension`, `MissingIdentities`, missing-identity drive waiting, and missing-identity suspension events.
|
||||
|
||||
Keep deferred suspension as provider semantics:
|
||||
|
||||
```ts
|
||||
{ kind: "suspended"; reason: "deferred"; ... }
|
||||
```
|
||||
|
||||
WP05 removes the withdrawn action outcome before execution is enabled. Convenience operation outcomes remain operation-tagged branches of `ResumeOutcome`.
|
||||
|
||||
### Snapshot
|
||||
|
||||
```ts
|
||||
export interface LaneSnapshot {
|
||||
lane: string;
|
||||
transcript: Entry[];
|
||||
tipId: string | null;
|
||||
lastResult?: LaneLastResult;
|
||||
operation: null | {
|
||||
id: string;
|
||||
kind: "run" | "compaction" | "navigation";
|
||||
startedAt: number;
|
||||
status: OperationStatus;
|
||||
action?: ActionInfo;
|
||||
retry?: { attempt: number; maxAttempts: number; nextAttemptAt: number };
|
||||
deferred?: { handle: DeferredHandle; poll: number };
|
||||
drained?: { steer: QueuedItem[]; followUp: QueuedItem[] };
|
||||
streamingMessage?: AssistantMessage;
|
||||
runningTools: {
|
||||
toolCallId: string;
|
||||
toolName: string;
|
||||
args: unknown;
|
||||
partialResult?: AgentToolResult<unknown>;
|
||||
}[];
|
||||
};
|
||||
queues: { steer: QueuedItem[]; followUp: QueuedItem[]; nextRun: QueuedItem[] };
|
||||
pendingWrites: {
|
||||
entryId: string;
|
||||
type: EntryType;
|
||||
customType?: string;
|
||||
message?: AgentMessage;
|
||||
data?: JsonValue;
|
||||
}[];
|
||||
faulted: boolean;
|
||||
}
|
||||
```
|
||||
|
||||
Configuration is not duplicated in snapshots. `inspectExecution()` exposes configured/captured model identities; getters expose current configuration.
|
||||
|
||||
## Atomic run acceptance
|
||||
|
||||
WP02 implements `accept()` for prompt, skill, and prompt-template requests. Compaction/navigation acceptance remains with their execution packages.
|
||||
|
||||
State-independent normalization occurs before `Lane.command(plan, context)`:
|
||||
|
||||
- prompt strings/images;
|
||||
- supplied message or message array;
|
||||
- explicit skill formatting;
|
||||
- prompt-template formatting;
|
||||
- pending assistant rejection;
|
||||
- unknown skill/template errors;
|
||||
- supplied or minted operation and prompt-entry ids.
|
||||
|
||||
Public prompt convenience overloads remain exactly `[text, images | undefined, context]` and `[messageOrMessages, context]`, but convenience implementations remain `SliceNotImplemented` until R2.
|
||||
|
||||
Inside one lane command:
|
||||
|
||||
1. reject busy;
|
||||
2. capture current `pendingNextRun` ids;
|
||||
3. read and validate captured pending messages;
|
||||
4. reject zero placed messages;
|
||||
5. parent captured next-run entries before request prompt entries;
|
||||
6. commit exactly once;
|
||||
7. publish the small owned projection;
|
||||
8. synchronously call `emitBatch` with the acceptance event batch and accepting Context;
|
||||
9. return from the mutation callback without awaiting delivery;
|
||||
10. await event delivery before `accept` resolves.
|
||||
|
||||
Exact writes:
|
||||
|
||||
```text
|
||||
insert captured nextRun message entries
|
||||
insert request prompt entries
|
||||
delete captured pendingEntry values
|
||||
set branchTip
|
||||
set operationMeta
|
||||
set operationState(run starting)
|
||||
set laneState(current operation, pendingNextRun=[])
|
||||
```
|
||||
|
||||
Exact event order:
|
||||
|
||||
```text
|
||||
run_start
|
||||
for each placed message:
|
||||
message_start
|
||||
message_end
|
||||
entry_added
|
||||
queue_update if nextRun was captured
|
||||
```
|
||||
|
||||
Acceptance starts no drive or effect and writes no Context.
|
||||
|
||||
## Phase A — normative rewrite
|
||||
|
||||
Update `harness.md` before runtime source:
|
||||
|
||||
- replace eager attachment hydration with minimal projection restore;
|
||||
- replace predictive status/classifier with open inventory;
|
||||
- specify configured/captured identity inspection without resolution;
|
||||
- move detailed reads to ad-hoc Session-line watch capture;
|
||||
- require recipient binding in the exact commit-observation continuation;
|
||||
- remove identity preflight/suspension/error/event types;
|
||||
- specify in-band model/tool unavailability and configuration provenance;
|
||||
- require direct detail-free synthetic tool-result messages;
|
||||
- update invariants, races, roadmap, glossary, and Appendix C.
|
||||
|
||||
Review stop:
|
||||
|
||||
1. `git diff --check`;
|
||||
2. fresh Terra contradiction/source-feasibility audit;
|
||||
3. full-context Fable review against complete docs and source;
|
||||
4. resolve all findings and repeat until no findings;
|
||||
5. obtain user approval before runtime source.
|
||||
|
||||
## Phase B — implementation
|
||||
|
||||
### Public and durable types
|
||||
|
||||
Modify `agent-harness.ts`:
|
||||
|
||||
- remove `SuspendedOperation`, `MissingIdentityInfo`, `MissingIdentities`, and missing-identity outcome/event branches;
|
||||
- add `ModelIdentity`, `OperationStatus`, `OpenOperation`, and corrected inspection/snapshot types;
|
||||
- change create result from `suspended` to `open`;
|
||||
- land operation-tagged action-required outcome families;
|
||||
- correct `executeAction`/`runToCompletion` signatures;
|
||||
- retain trailing Context everywhere.
|
||||
|
||||
Modify session types:
|
||||
|
||||
- add `failure_drain` configuration provenance;
|
||||
- add the already-normative `ToolCall.outcome_ready` vocabulary without producers;
|
||||
- document `sourceIndex` as full assistant-content index;
|
||||
- add callback-scoped `scanBranch` to `SessionReader`; inheritance by `SessionMutator` and `Session` is intentional;
|
||||
- implement it in `StorageBackedSession` and its mutator and `MemorySessionFacade`; Session mutation authority remains process-local and has no remote Session facade.
|
||||
|
||||
### Event publication
|
||||
|
||||
WP02 initially added synchronous recipient binding with a reserved delivery gate. WP04 supersedes only that mechanism:
|
||||
|
||||
- `HarnessEventBus.emitBatch()` snapshots ordinary and watcher recipients synchronously;
|
||||
- delivery uses only that bound list;
|
||||
- watcher buffers retain `{ event, context }`;
|
||||
- a watcher registered after `emitBatch` receives nothing from that event.
|
||||
|
||||
`LaneCommand` commit decisions retain a synchronous post-commit event batch. After commit succeeds, `Lane.command` publishes `next`, calls `emitBatch` as its final mutation action, and carries the delivery promise outside `Session.mutate` before awaiting it. Every existing event-producing commit uses this path, including direct idle/pending appends, lane configuration setters, and session-name/entry-label setters.
|
||||
|
||||
WP04 also moves harness lane publication into `Session.createLane`'s committed-publication callback. Session commits, Harness publishes `lanesByName` and calls `emitBatch(lane_created)` in that continuation, and Session awaits the retained delivery promise after releasing the line.
|
||||
|
||||
Do not execute listeners on the line.
|
||||
|
||||
### Attachment
|
||||
|
||||
Keep `restore.ts` projection-only. Validate lane/operation ownership and kind compatibility. Remove `describeSuspension` and all payload hydration from `createAgentHarness`. Construct lanes and return `open` inventory without resolving registries.
|
||||
|
||||
### Inspection
|
||||
|
||||
Implement `inspectExecution(context)` as a no-write `Lane.command` observation. Derive captured model identity from the current durable phase. Read no storage payloads and resolve no registry identities.
|
||||
|
||||
### Watch
|
||||
|
||||
Implement `watch(context)` as one no-write lane job:
|
||||
|
||||
- register watcher and clone live presentation synchronously;
|
||||
- perform the bounded read matrix above through callback-scoped readers;
|
||||
- assemble isolated snapshot payloads;
|
||||
- fault and unsubscribe on required corruption;
|
||||
- retain legal optional absence;
|
||||
- release the line before returning.
|
||||
|
||||
A focused internal snapshot helper is allowed. Do not create a persistent hydrated presentation cache or generic reducer.
|
||||
|
||||
### Out of scope
|
||||
|
||||
`drive`, `resume`, prompt convenience, compaction/navigation acceptance, abort/queues, `executeAction`, and `runToCompletion` remain `SliceNotImplemented` where not already implemented.
|
||||
|
||||
Provider/tool/configuration-failure transitions are specified now but implemented by R2/R3/R4/R7/R8. WP02 adds no effect, active operation, timer, hook, provider request, tool execution, retry, deferred fetch, cancellation reconciliation, or terminal transaction.
|
||||
|
||||
## Required tests
|
||||
|
||||
### Public types
|
||||
|
||||
- `SuspendedOperation`, `MissingIdentities`, and missing-identity status/outcomes/events are absent;
|
||||
- open/current/status unions narrow exhaustively;
|
||||
- configured and captured model identities are unresolved strings;
|
||||
- action-required outcome families and method signatures match;
|
||||
- prompt overload tuples remain exact;
|
||||
- `AgentHarnessOptions` has no receiver telemetry default.
|
||||
|
||||
### Acceptance
|
||||
|
||||
- text-only, images-only, text-plus-images, supplied arrays;
|
||||
- pending assistant rejection;
|
||||
- skills/templates and unknown resources;
|
||||
- empty input writes nothing unless captured nextRun supplies input;
|
||||
- no identity registry preflight;
|
||||
- supplied/minted operation ids;
|
||||
- exact writes, parent chain, metadata, starting state, settings, commit materialization;
|
||||
- pending-next-run capture/deletion;
|
||||
- busy run/structural operation;
|
||||
- one concurrent accept winner;
|
||||
- commit failure and close races;
|
||||
- exact event order and object-identical accepting Context;
|
||||
- no hook, provider, tool, timer, drive-owner, or option callback invocation; passive event-listener delivery remains required.
|
||||
|
||||
### Attachment and inspection
|
||||
|
||||
- idle omitted and every open operation inventoried;
|
||||
- durable cancellation sets only `aborting:true`;
|
||||
- no transcript/pending/frame/tool reads at create;
|
||||
- configured and captured model identities may differ and remain visible when unresolved;
|
||||
- inspection runs on the Session line and performs no payload reads;
|
||||
- projection corruption faults create;
|
||||
- no option callback/effect starts.
|
||||
|
||||
### Event publication and watch
|
||||
|
||||
- recipient set binds at `emitBatch`: a watcher registered after publication but before delivery receives nothing;
|
||||
- state/event batch publication happens before Session-line release;
|
||||
- direct append, lane configuration, session-name/entry-label, acceptance, and lane-creation commits all use that publication path;
|
||||
- `value_update` and `lane_created` bind recipients in their commit continuation, so later listeners receive neither historical event;
|
||||
- watcher-first gives snapshot-before plus complete events;
|
||||
- publication-first gives snapshot-after without old events;
|
||||
- live update during awaited capture appears only as a buffered event after the cloned live snapshot;
|
||||
- frame/checkpoint mutations queue behind capture;
|
||||
- first watch and reconnect use the same path;
|
||||
- exact transcript, queues, writes, drain, deferred, frame, tool args/checkpoint fields;
|
||||
- required payload corruption faults watch and removes watcher;
|
||||
- absent frames/checkpoints omit optional partials;
|
||||
- pre-registration lifecycle is not replayed;
|
||||
- buffered events retain object-identical emitting Context;
|
||||
- payload mutation cannot affect later state/listeners;
|
||||
- close/fault lifecycle matches the event bus.
|
||||
|
||||
### In-band identity vocabulary
|
||||
|
||||
Type/direct-state tests prove:
|
||||
|
||||
- acceptance has no `MissingIdentities` path;
|
||||
- configuration failure provenance and stable error codes are representable;
|
||||
- deferred configuration abandonment remains assigned to R7 rather than adding an execution transition here;
|
||||
- missing-tool synthetic `ToolResultMessage` may omit `details`;
|
||||
- sourceIndex uses full assistant-content indexing;
|
||||
- outcome-ready calls are not rendered as running.
|
||||
|
||||
No execution transition is added in WP02.
|
||||
|
||||
## Files
|
||||
|
||||
### Add
|
||||
|
||||
- `packages/agent/test/harness/runtime2/accept.test.ts`
|
||||
- focused watch tests if existing files become oversized.
|
||||
|
||||
### Modify
|
||||
|
||||
- `packages/agent/docs/harness.md`
|
||||
- `packages/agent/docs/work-packages/02-atomic-run-acceptance.md`
|
||||
- `packages/agent/src/harness/agent-harness.ts`
|
||||
- `packages/agent/src/harness/events.ts`
|
||||
- `packages/agent/src/harness/session/types.ts`
|
||||
- `packages/agent/src/harness/session/session.ts`
|
||||
- `packages/agent/src/harness/session/memory.ts`
|
||||
- `packages/agent/src/harness/session/remote.ts`
|
||||
- `packages/server/src/remote-session-manager.ts`
|
||||
- `packages/agent/src/harness/runtime2/harness.ts`
|
||||
- `packages/agent/src/harness/runtime2/lane.ts`
|
||||
- `packages/agent/src/harness/runtime2/restore.ts`
|
||||
- `packages/agent/src/harness/runtime2/types.ts`
|
||||
- `packages/agent/test/harness/runtime2/harness.test.ts`
|
||||
- `packages/agent/test/harness/runtime2/lane.test.ts`
|
||||
- `packages/agent/test/harness/runtime2/restore.test.ts`
|
||||
- `packages/agent/test/harness/types.test.ts`
|
||||
- `packages/agent/test/harness/storage-backed-session.test.ts`
|
||||
- `packages/agent/test/harness/memory-session-repo.test.ts`
|
||||
- `packages/server/test/conformance.test.ts`
|
||||
- event/session test files required by recipient binding and callback-scoped branch reads.
|
||||
|
||||
No backend schema, telemetry schema, coding-agent, or changelog change is expected. Stop for boundary review if one becomes necessary. On `dev`, defer changelog entries.
|
||||
|
||||
## Validation
|
||||
|
||||
After Phase A:
|
||||
|
||||
```bash
|
||||
git diff --check -- \
|
||||
packages/agent/docs/harness.md \
|
||||
packages/agent/docs/work-packages/02-atomic-run-acceptance.md
|
||||
```
|
||||
|
||||
After Phase B:
|
||||
|
||||
```bash
|
||||
cd packages/agent
|
||||
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run \
|
||||
test/harness/runtime2/accept.test.ts \
|
||||
test/harness/runtime2/harness.test.ts \
|
||||
test/harness/runtime2/lane.test.ts \
|
||||
test/harness/runtime2/restore.test.ts \
|
||||
test/harness/types.test.ts
|
||||
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
git diff --check
|
||||
npm run check
|
||||
./test.sh
|
||||
```
|
||||
|
||||
Report runtime2 source line counts. The synchronized pre-WP02 runtime2 baseline is 967 lines; treat growth above 1,900 source lines as a design review trigger, not a target.
|
||||
|
||||
## Stop condition
|
||||
|
||||
Stop when:
|
||||
|
||||
- acceptance commits exactly once into payload-free `starting` without registry preflight;
|
||||
- attachment returns minimal complete projections and open inventory;
|
||||
- inspection is a coherent Session-line no-write observation;
|
||||
- watch captures detailed state ad hoc on the Session line;
|
||||
- state publication and `emitBatch` recipient binding occur in the commit continuation while the mutation never awaits delivery;
|
||||
- snapshots and buffered events have no gap or duplicate;
|
||||
- required payload corruption faults its consumer;
|
||||
- no execution effect or owner is introduced;
|
||||
- focused tests, `npm run check`, and full tests pass;
|
||||
- final Fable review has no findings.
|
||||
|
||||
Do not begin the first real drive package.
|
||||
183
packages/agent/docs/work-packages/03-remove-drive-deadlines.md
Normal file
183
packages/agent/docs/work-packages/03-remove-drive-deadlines.md
Normal file
@@ -0,0 +1,183 @@
|
||||
# WP03 — Remove drive deadlines
|
||||
|
||||
## Status
|
||||
|
||||
Complete. `DriveOptions.deadline` and the `DriveOutcome` `yielded` branch are removed from public types, active documentation, future package requirements, invariants, races, and tests. Obsolete `runtime2.md` is deleted. Focused tests, `npm run check`, and `./test.sh` pass.
|
||||
|
||||
WP02 is complete at `beac75ecc`. Preserve unrelated concurrent source work, especially current `packages/agent/src/harness/runtime2/lane.ts` changes, JSONL/fork work, plugins, RPC, and experimental directories.
|
||||
|
||||
## Problem
|
||||
|
||||
`DriveOptions.deadline` and the corresponding `DriveOutcome { kind: "yielded" }` do not provide a correctness boundary.
|
||||
|
||||
A deadline is checked only before starting another transition or effect. An admitted provider/tool/hook may run beyond it and the host may terminate the process anyway:
|
||||
|
||||
```text
|
||||
check deadline
|
||||
→ admit provider or tool
|
||||
→ host limit expires while the effect is running
|
||||
→ process dies with durable effect_pending
|
||||
```
|
||||
|
||||
Unknown-outcome recovery remains mandatory. The deadline therefore does not prevent process loss, bound admitted work, make effects exactly once, or simplify recovery. Flue-style tool memoization depends on stable invocation ids and durable memos, not drive deadlines.
|
||||
|
||||
Deadline handling instead adds wall-clock policy to the durable core:
|
||||
|
||||
- a `yielded` public outcome unrelated to durable state;
|
||||
- safe-boundary checks before hooks/effects/transitions;
|
||||
- deadline-versus-retry-timer arbitration;
|
||||
- deadline-versus-effect-admission races;
|
||||
- convenience loops and event-bracket behavior for yields.
|
||||
|
||||
The host already owns scheduling and termination. Process loss is a controlled crash boundary recovered from durable operation state.
|
||||
|
||||
## Decision
|
||||
|
||||
Remove drive deadlines completely:
|
||||
|
||||
```ts
|
||||
interface DriveOptions {
|
||||
operationId: string;
|
||||
waitForRetry?: boolean;
|
||||
pollDeferred?: boolean;
|
||||
}
|
||||
|
||||
type DriveOutcome =
|
||||
| { kind: "settled"; operationId: string; outcome: TerminalOperationOutcome }
|
||||
| { kind: "waiting"; operationId: string; reason: "retry"; notBefore: number }
|
||||
| { kind: "waiting"; operationId: string; reason: "deferred"; deferred: DeferredHandle }
|
||||
| { kind: "action_required"; operationId: string; action: ActionInfo };
|
||||
```
|
||||
|
||||
There is no deprecated alias, ignored `deadline` field, compatibility overload, alternate timestamp option, or replacement pause flag in WP03.
|
||||
|
||||
The next drive package uses direct durable transitions. Deterministic tests gate commits and control hooks, providers, tools, and timers without adding production execution barriers.
|
||||
|
||||
## Host behavior
|
||||
|
||||
Hosts own execution budgets:
|
||||
|
||||
```text
|
||||
invoke drive
|
||||
→ terminal or durable waiting result: schedule normally
|
||||
→ planned shutdown: stop routing/releasing work and close session processes
|
||||
→ forced termination: replacement attaches and recovers durable open operations
|
||||
```
|
||||
|
||||
A process-per-session host may stop routing work and close before exit, and use process termination as the hard fence for non-cooperative providers, tools, hooks, storage, or event listeners. This operational policy does not require a wall-clock field in `DriveOptions`.
|
||||
|
||||
WP03 does not add `stopAfterCheckpoint`, `pause`, `quiesce`, or in-process crash simulation. Those ideas remain outside the direct durable-drive design. In-process process-loss simulation is not a public core primitive: an old continuation requires fencing, for which process isolation is the reliable mechanism.
|
||||
|
||||
## Work
|
||||
|
||||
### Public types
|
||||
|
||||
In `packages/agent/src/harness/agent-harness.ts`:
|
||||
|
||||
- delete `DriveOptions.deadline`;
|
||||
- delete the `DriveOutcome` `yielded` branch;
|
||||
- preserve expected-id fencing, retry waiting, deferred waiting, and manual action branches unchanged.
|
||||
|
||||
No current runtime2 drive implementation exists, so this package adds no execution behavior or owner.
|
||||
|
||||
### Normative documentation
|
||||
|
||||
Update `packages/agent/docs/harness.md` completely:
|
||||
|
||||
- remove safe-yield scheduling language from non-goals/orientation;
|
||||
- remove only the deadline half of cancellation/deadline prerequisite wording in §3.6, §5.6's `before_drive` row, and invariant 22; the cancellation prerequisite remains;
|
||||
- remove deadline checks and yielded returns from the drive-pass pseudocode;
|
||||
- remove deadline policy from pass joining, retry waiting, convenience composition, recovery, and public method prose;
|
||||
- remove `deadline` from `DriveOptions` and `yielded` from `DriveOutcome`;
|
||||
- remove deadline-specific event/turn requirements;
|
||||
- remove deadline language from `before_drive` hook timing;
|
||||
- remove deadline/yield requirements from future work rows;
|
||||
- replace invariant 25 with the explicit no-wall-clock-policy invariant while retaining the rule that an admitted effect settles normally or is recovered after task loss;
|
||||
- remove deadline races from the race catalog;
|
||||
- update the drive-pass glossary.
|
||||
|
||||
Do not alter generic RPC timeout/deadline documentation or unrelated process/model-catalog timeouts. Those are invocation/transport policies, not `AgentLane.drive`.
|
||||
|
||||
Delete obsolete `packages/agent/docs/runtime2.md`; `harness.md` plus linked handoffs are the sole implementation plan and history needed for active work.
|
||||
|
||||
Mark WP02 complete in its handoff and Part 8. Add WP03 as the concrete cleanup package. Leave the former R2/R3 drive rows as historical future candidates, minus deadline/yield requirements, until the reviewed direct-drive handoff replaces them.
|
||||
|
||||
### Delete
|
||||
|
||||
- `packages/agent/docs/runtime2.md`
|
||||
|
||||
### Type tests
|
||||
|
||||
Extend `packages/agent/test/harness/types.test.ts`:
|
||||
|
||||
- assert `keyof DriveOptions` is exactly `"operationId" | "waitForRetry" | "pollDeferred"`;
|
||||
- assert `DriveOutcome["kind"]` excludes `"yielded"`;
|
||||
- add `@ts-expect-error` coverage proving callers cannot supply `deadline`;
|
||||
- retain existing drive/result signatures.
|
||||
|
||||
Run the focused type test after editing it.
|
||||
|
||||
### Downstream compatibility
|
||||
|
||||
Search protocol, server, coding-agent, examples, and tests for structural mirrors or exhaustive `DriveOutcome` switches. Update only compile-time/type compatibility forced by the public removal. Coding-agent experimental worker/remote-runtime behavior and tests are out of scope.
|
||||
|
||||
The current protocol harness schemas expose prompt/run/watch DTOs, not `DriveOptions` or `DriveOutcome`; no protocol edit is expected unless the final search proves otherwise.
|
||||
|
||||
## Non-goals
|
||||
|
||||
WP03 does not implement or redesign:
|
||||
|
||||
- `drive`, `resume`, prompt conveniences, operation ownership, or latest-result lookup;
|
||||
- manual actions or automatic barrier release;
|
||||
- provider/tool execution, recovery, retry timers, deferred polling, abort, or terminal settlement;
|
||||
- checkpoint pause/quiesce;
|
||||
- close admission changes;
|
||||
- worker/RPC cancellation or experimental remote prompting;
|
||||
- storage/schema/migration behavior.
|
||||
|
||||
## Required checks
|
||||
|
||||
```bash
|
||||
# No drive deadline/yield contract remains in active harness docs or source.
|
||||
rg -n 'deadline|yield' \
|
||||
packages/agent/src/harness \
|
||||
packages/agent/test/harness \
|
||||
packages/agent/docs/harness.md \
|
||||
packages/agent/docs/work-packages
|
||||
|
||||
cd packages/agent
|
||||
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run \
|
||||
test/harness/types.test.ts
|
||||
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
git diff --check
|
||||
npm run check
|
||||
./test.sh
|
||||
```
|
||||
|
||||
The removal grep may still match this handoff's historical problem explanation. Every remaining match must be reviewed; no active API, normative behavior, future acceptance criterion, or test expectation may retain the removed contract.
|
||||
|
||||
## Review
|
||||
|
||||
Before implementation:
|
||||
|
||||
1. Fable reviews this handoff against complete docs/source.
|
||||
2. As explicitly requested by the user, `openai-codex/gpt-5.6-sol` reviews it with thinking level high.
|
||||
3. Resolve all findings and repeat until no findings.
|
||||
|
||||
After implementation, repeat both reviews over the final documentation/type diff.
|
||||
|
||||
## Stop condition
|
||||
|
||||
Stop when:
|
||||
|
||||
- `DriveOptions` has no wall-clock budget;
|
||||
- `DriveOutcome` has no `yielded` branch;
|
||||
- active normative docs contain no deadline/yield behavior;
|
||||
- future drive rows contain no hidden deadline requirements;
|
||||
- type tests prove removal;
|
||||
- unrelated timeout/deadline APIs remain untouched;
|
||||
- focused tests, `npm run check`, and `./test.sh` pass;
|
||||
- final Fable and Codex reviews have no findings.
|
||||
|
||||
Do not begin the direct durable-drive implementation in this package.
|
||||
337
packages/agent/docs/work-packages/04-mutation-publication.md
Normal file
337
packages/agent/docs/work-packages/04-mutation-publication.md
Normal file
@@ -0,0 +1,337 @@
|
||||
# WP04 — Mutation publication and event delivery
|
||||
|
||||
## Status
|
||||
|
||||
Complete. Phase A and the final implementation rereview passed Fable with no findings. Focused agent/server/SQLite tests, `npm run build`, `npm run check`, and `./test.sh` pass. `packages/agent/docs/harness.md` remains normative.
|
||||
|
||||
> Historical note: WP06 later replaced the lane-creation API and keyed line described in this completed handoff with atomic `AgentHarness.lane()` acquisition on one Session line. The event-publication guarantees remain current.
|
||||
|
||||
WP02 established atomic acceptance, recipient binding, and coherent lane watches. WP03 removed drive deadlines. WP04 removes the caller-operated event-delivery gate without weakening those guarantees and makes the historical `Session.createLane()` own lane creation end to end. The direct durable-drive package follows as WP05.
|
||||
|
||||
## Problem
|
||||
|
||||
A committing lane job currently uses a two-part event API:
|
||||
|
||||
```text
|
||||
inside Session.mutate:
|
||||
commit
|
||||
publish process-local state
|
||||
delivery = events.enqueue(batch, context)
|
||||
|
||||
outside Session.mutate:
|
||||
await delivery.start()
|
||||
```
|
||||
|
||||
The split preserves the correct boundary, but it is a footgun: calling `emit()` too early, calling `start()` too early, or dropping `start()` can violate observation semantics or stall the global event tail. The historical `Harness.createLane()` repeats the choreography manually.
|
||||
|
||||
Lane creation has a second one-off boundary. The historical `Session.createLane()` owns validation and the durable transaction, but Harness cannot publish its process-local `Lane` and bind `lane_created` recipients from that same commit continuation. Harness therefore opens `Session.mutate()` itself and calls exported `createLaneWithMutator()`.
|
||||
|
||||
WP04 removes both caller-operated seams while preserving current direct-listener and hook barriers.
|
||||
|
||||
## Required semantics
|
||||
|
||||
### Direct events remain awaited observations
|
||||
|
||||
Direct `events.on()` listeners remain passive but causally ordered:
|
||||
|
||||
```text
|
||||
hook or preparation
|
||||
→ commit
|
||||
→ publish process-local state
|
||||
→ bind and append event batch
|
||||
→ release lane mutation line
|
||||
→ await direct listeners
|
||||
→ resolve operation
|
||||
→ later hook or transition
|
||||
```
|
||||
|
||||
Passive means a listener cannot transform the in-flight operation and listener failures are isolated as `handler_error`. It does not mean fire-and-forget. An extension may update process-local state in an event listener and inspect it from a later hook. Awaiting also supplies producer backpressure.
|
||||
|
||||
A direct listener must not call a state-mutating harness API: an emitted mutation would queue a later event behind the event currently awaiting that listener. Read-only lane calls remain legal.
|
||||
|
||||
Deliberate exceptions remain unchanged:
|
||||
|
||||
- watchers and RPC/watch consumers use their own buffered FIFO and are not awaited by operations;
|
||||
- fault publication is fire-and-forget, and close never waits for listener completion;
|
||||
- high-frequency tool updates enqueue every event and retain only the latest delivery promise; tool settlement awaits that promise before `after_tool` and outcome publication, which drains all earlier updates through global FIFO without per-update backpressure.
|
||||
|
||||
### Commit and recipient binding remain one continuation
|
||||
|
||||
Every event-producing committing lane job performs, in the exact continuation that observes successful commit:
|
||||
|
||||
```text
|
||||
commit succeeds
|
||||
→ publish complete process-local state
|
||||
→ synchronously call emitBatch(batch, context)
|
||||
- clone payloads
|
||||
- bind current ordinary listeners and watchers
|
||||
- append the complete batch to the global delivery tail
|
||||
→ return from the mutation callback
|
||||
```
|
||||
|
||||
There is no scheduler-owned after-release publication phase. Moving recipient binding into a later promise continuation creates a gap in which another task may observe committed state and register a listener before the old event binds recipients.
|
||||
|
||||
`emitBatch()` starts asynchronous delivery immediately and returns its completion promise. The mutation callback never awaits that promise. Listener code may begin after publication/binding but before the mutation line technically releases; a reentrant lane read queues behind the current job. The command carries the promise out of `Session.mutate()` and awaits it before resolving publicly.
|
||||
|
||||
This preserves the two legal watcher races:
|
||||
|
||||
```text
|
||||
watcher registration first
|
||||
→ snapshot-before + complete buffered batch
|
||||
|
||||
commit publication first
|
||||
→ snapshot-after + no old event
|
||||
```
|
||||
|
||||
### Ordering
|
||||
|
||||
- One non-empty `emitBatch()` call publishes one contiguous batch.
|
||||
- Batches enter the existing global event tail in `emitBatch()` invocation order, including across lanes.
|
||||
- Events within a batch retain source order.
|
||||
- Direct listeners run serially in registration order.
|
||||
- Same-lane committing jobs bind batches in lane mutation order.
|
||||
- A lane procedure awaits each command's delivery before invoking its next hook or transition.
|
||||
- Hooks on unrelated lanes may overlap; there is no total cross-lane hook order.
|
||||
- Context remains the exact emitting invocation Context and is always the final parameter.
|
||||
|
||||
## Event bus contract
|
||||
|
||||
Replace the public/internal delivery split with:
|
||||
|
||||
```ts
|
||||
class HarnessEventBus implements Events {
|
||||
emit(event: HarnessEvent, context: Context): Promise<void>;
|
||||
emitBatch(events: readonly HarnessEvent[], context: Context): Promise<void>;
|
||||
}
|
||||
```
|
||||
|
||||
`emitBatch()`:
|
||||
|
||||
1. returns an already-resolved promise for an empty batch or a closed bus;
|
||||
2. synchronously structured-clones every payload and binds its current recipients;
|
||||
3. appends one contiguous delivery to the existing global tail;
|
||||
4. delivers cloned payloads serially with the emitting Context;
|
||||
5. isolates listener failures and publishes non-recursive `handler_error` as today;
|
||||
6. returns a promise that resolves after eligible direct listeners settle and never rejects because a listener failed.
|
||||
|
||||
Synchronous publication defects such as an uncloneable internal payload still throw in the caller's commit continuation and follow the existing harness-fault path.
|
||||
|
||||
Delete:
|
||||
|
||||
- `HarnessEventDelivery`;
|
||||
- `HarnessEventBus.enqueue()`;
|
||||
- caller-operated `start()`;
|
||||
- delivery gates and `pendingStarts`;
|
||||
- close-time forced gate release.
|
||||
|
||||
`close(error)` seals listener/watch registration and future publication immediately. Already appended batches retain their bound recipients and drain through the existing tail; listener completion still does not block Harness close.
|
||||
|
||||
## Lane command integration
|
||||
|
||||
The commit branch of `Lane.command()` returns a plain internal outcome containing the caller result and optional delivery promise:
|
||||
|
||||
```ts
|
||||
const events = decision.events?.(commit) ?? [];
|
||||
const delivery = events.length === 0
|
||||
? undefined
|
||||
: this.onEvent(events, context);
|
||||
|
||||
return {
|
||||
kind: "return",
|
||||
result,
|
||||
...(delivery === undefined ? {} : { delivery }),
|
||||
};
|
||||
```
|
||||
|
||||
`onEvent` returns `Promise<void>` and delegates to `emitBatch()`. It is called only after commit, complete process-local state publication, and synchronous result materialization, as the final action before returning the mutation outcome.
|
||||
|
||||
After `Session.mutate()` returns:
|
||||
|
||||
```ts
|
||||
if (outcome.kind === "reject") throw outcome.error;
|
||||
await outcome.delivery;
|
||||
return outcome.result;
|
||||
```
|
||||
|
||||
Expected no-commit rejections publish no events. Commit/materialization/publication errors retain the existing harness-fault semantics.
|
||||
|
||||
## Session lane creation contract
|
||||
|
||||
The historical `Session.createLane()` owns validation, commit, and the synchronous committed-publication callback. Context remains last:
|
||||
|
||||
```ts
|
||||
createLane(
|
||||
name: string,
|
||||
at: string | null,
|
||||
configuration: LaneConfiguration,
|
||||
onCommitted: ((context: Context) => void | Promise<void>) | undefined,
|
||||
context: Context,
|
||||
): Promise<SessionTree>;
|
||||
```
|
||||
|
||||
The implementation performs:
|
||||
|
||||
```text
|
||||
enter the prospective lane's mutation line
|
||||
→ validate name, absence, complete lane shape, and target
|
||||
→ commit lane configuration + leaf + idle lane state
|
||||
→ synchronously invoke onCommitted(context) in that same commit continuation
|
||||
→ retain its returned promise inside a non-thenable outcome object
|
||||
→ return from the mutation callback and release the line
|
||||
→ await the retained promise
|
||||
→ return Session.view(name)
|
||||
```
|
||||
|
||||
The callback receives neither `SessionTree` nor `SessionMutator`. It is only the process-local publication point. Its synchronous prefix must complete the publication needed before another same-lane job can run. Ordinary Session callers pass `undefined`.
|
||||
|
||||
If validation or commit fails, the callback is not invoked. If the callback throws after commit or its retained promise rejects after line release, the durable lane remains and the caller rejects; Harness converts that committed-publication defect to its existing fault path. A promise returned by the Harness callback is event delivery, whose listener failures are isolated by the bus.
|
||||
|
||||
The current lane validation/transaction implementation becomes private to Session. Delete exported `createLaneWithMutator()` and its direct tests.
|
||||
|
||||
Harness preconstructs a detached `Lane`, then calls Session:
|
||||
|
||||
```ts
|
||||
const lane = this.buildLane(name, state);
|
||||
|
||||
await this.session.createLane(
|
||||
name,
|
||||
at,
|
||||
this.seed,
|
||||
(context) => {
|
||||
if (this.closedError !== undefined) lane.seal(this.closedError);
|
||||
this.lanesByName.set(name, lane);
|
||||
return this.events.emitBatch(
|
||||
[{ type: "lane_created", lane: name, at }],
|
||||
context,
|
||||
);
|
||||
},
|
||||
context,
|
||||
);
|
||||
|
||||
return Result.ok(lane);
|
||||
```
|
||||
|
||||
The callback's synchronous prefix publishes `lanesByName` and binds `lane_created` before the first creation job releases its line. A queued duplicate therefore cannot report `LaneExists` before the winner is visible through `harness.lane(name)`.
|
||||
|
||||
If close or fault wins while commit is admitted, the callback publishes the new Lane sealed. Emission on the already-closed bus is a resolved no-op, matching the existing admitted-creation race. The successful admitted creation still returns its sealed Lane.
|
||||
|
||||
## Scope
|
||||
|
||||
### Source
|
||||
|
||||
Modify:
|
||||
|
||||
- `packages/agent/src/harness/events.ts`;
|
||||
- `packages/agent/src/harness/runtime2/lane.ts`;
|
||||
- `packages/agent/src/harness/runtime2/harness.ts`;
|
||||
- `packages/agent/src/harness/session/types.ts`;
|
||||
- local Session implementations and tests required by the historical `createLane` signature;
|
||||
- direct event primitive, acceptance, watch, lane, and harness tests.
|
||||
|
||||
Remote/experimental runtime behavior is not a design constraint for WP04. Session mutation authority remains process-local; do not add remote Session callback transport, protocol machinery, compatibility abstractions, or boundary tests.
|
||||
|
||||
### Documentation
|
||||
|
||||
Update normative `harness.md`:
|
||||
|
||||
- replace enqueue/start language with synchronous `emitBatch` binding and post-mutation awaiting;
|
||||
- state accurately that listener execution may begin after publication/binding but before technical line release;
|
||||
- retain awaited direct-listener, event/hook, watcher, Context, close, and ordering semantics;
|
||||
- replace shared exported mutator-procedure lane creation with the historical `Session.createLane(onCommitted, context)`;
|
||||
- update invariants, races, tests, glossary, and Part 8;
|
||||
- link WP04 and move direct durable drive to WP05.
|
||||
|
||||
Update the historical WP02 handoff only where it points forward or claims the old mechanism remains current. Do not rewrite its completed-package record as though WP04 behavior landed in WP02.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- fire-and-forget direct events or a public flush API;
|
||||
- sequence/watermark delivery redesign;
|
||||
- per-extension event queues;
|
||||
- changing hook aggregation or event/hook causal barriers;
|
||||
- changing watcher/RPC buffering;
|
||||
- changing tool-update settlement barriers;
|
||||
- making event-listener mutation safe;
|
||||
- drive, breakpoints, providers, tools, recovery, retries, polling, abort, or terminal settlement;
|
||||
- generic Session post-commit/after-release task APIs;
|
||||
- remote callback execution or remote-runtime redesign.
|
||||
|
||||
## Required tests
|
||||
|
||||
### Event primitives
|
||||
|
||||
- `emitBatch` binds ordinary listeners and watchers synchronously;
|
||||
- a listener registered after `emitBatch` but before delayed delivery receives nothing;
|
||||
- a complete batch is contiguous and preserves event order;
|
||||
- concurrent batch publication preserves invocation order;
|
||||
- listener payload mutation remains isolated;
|
||||
- listener rejection emits one non-recursive `handler_error` and does not reject delivery;
|
||||
- empty and post-close batches resolve without delivery;
|
||||
- already appended batches drain after close;
|
||||
- no gate/start tests remain.
|
||||
|
||||
### Lane commands and watch
|
||||
|
||||
- direct listeners may perform reentrant read-only lane inspection without deadlock;
|
||||
- a command does not resolve until its direct listeners finish;
|
||||
- committed memory is visible to listeners;
|
||||
- commit failure publishes nothing;
|
||||
- a late listener that observes durable state receives no historical event;
|
||||
- watcher-first yields snapshot-before plus the complete buffered batch;
|
||||
- publication-first yields snapshot-after without replay;
|
||||
- source Context identity survives delayed and buffered delivery.
|
||||
|
||||
### Lane creation
|
||||
|
||||
- Session callback runs exactly once after successful commit and never on validation/commit failure;
|
||||
- callback synchronous publication occurs before a queued duplicate reports `LaneExists`;
|
||||
- Harness Lane and durable configuration are visible to `lane_created` listeners;
|
||||
- Harness waits for asynchronous `lane_created` listeners before resolving;
|
||||
- admitted creation racing close publishes a sealed Lane and does not require event delivery;
|
||||
- callback throw and retained-promise rejection after commit follow the documented committed-publication failure path;
|
||||
- ordinary Session creation with `undefined` returns its view;
|
||||
- no exported `createLaneWithMutator` remains.
|
||||
|
||||
### Ordering barriers
|
||||
|
||||
- acceptance events finish before `accept()` resolves;
|
||||
- command resolution cannot overtake its awaited direct event delivery; WP05 tests the first subsequent procedure hook;
|
||||
- event batches are globally ordered across concurrent lane publication;
|
||||
- tool-update latest-delivery settlement behavior remains unchanged where implemented.
|
||||
|
||||
## Validation
|
||||
|
||||
After documentation:
|
||||
|
||||
```bash
|
||||
git diff --check -- \
|
||||
packages/agent/docs/harness.md \
|
||||
packages/agent/docs/work-packages/02-atomic-run-acceptance.md \
|
||||
packages/agent/docs/work-packages/04-mutation-publication.md \
|
||||
packages/agent/docs/work-packages/05-direct-durable-drive.md
|
||||
```
|
||||
|
||||
After implementation, run every modified focused test, then:
|
||||
|
||||
```bash
|
||||
git diff --check
|
||||
npm run check
|
||||
./test.sh
|
||||
```
|
||||
|
||||
Review the final implementation with Fable before declaring WP04 complete. Do not commit without explicit user approval.
|
||||
|
||||
## Stop condition
|
||||
|
||||
WP04 is complete when:
|
||||
|
||||
- event publication has one `emitBatch()` operation and no caller-operated gate;
|
||||
- recipient binding remains in the exact commit-observation continuation;
|
||||
- direct event delivery remains globally FIFO and awaited before public operation resolution;
|
||||
- existing event/hook causal barriers remain intact;
|
||||
- lane watches retain exactly the two coherent race outcomes;
|
||||
- Session owns lane creation and invokes Harness publication before releasing the creation job;
|
||||
- `createLaneWithMutator()` is gone;
|
||||
- Context is trailing and source-identical throughout;
|
||||
- focused tests, `npm run check`, and `./test.sh` pass;
|
||||
- final Fable review reports no findings.
|
||||
606
packages/agent/docs/work-packages/05-direct-durable-drive.md
Normal file
606
packages/agent/docs/work-packages/05-direct-durable-drive.md
Normal file
@@ -0,0 +1,606 @@
|
||||
# WP05 — Direct durable drive
|
||||
|
||||
**Status: WP05 complete through M10: lane-owned inbox, immutable result records, 13 family-neutral leaves, atomic boundary planning, total cancellation/dispatch, public and replicated lane surfaces, documentation reconciliation, and lane-safe provider cache identity. Remaining assistant-output work is owned by the [mobile assistant-output handoff](../mobile-handoff/01-harness/05-assistant-output/message-update.md), outside the public-drive gate.**
|
||||
|
||||
WP06's Session/Branch/Lane separation is part of the foundation. Public drive is enabled; `watchSession` is the sole deferred Harness method.
|
||||
|
||||
Format 4 remains work in progress. Every durable type replacement in this package requires no migration and no compatibility decoder for the pre-redesign shapes.
|
||||
|
||||
The standalone-compaction inbox/promotion design previously in this document is **withdrawn** (§5). R1–R3 and M7/M8 replace it. M9 reconciled `docs/harness.md`, which is normative on its own again.
|
||||
|
||||
## 0. Mandatory reading
|
||||
|
||||
Read completely before implementation work:
|
||||
|
||||
1. `packages/agent/docs/harness.md` (normative; reconciled by M9)
|
||||
2. `packages/agent/docs/runtime-simplification.md` (its 22-leaf list is replaced by R3)
|
||||
3. `packages/agent/src/harness/session/types.ts`
|
||||
4. `packages/agent/src/harness/session/values.ts`
|
||||
5. `packages/agent/src/harness/runtime/lane.ts`
|
||||
6. `packages/agent/src/harness/runtime/types.ts`
|
||||
7. every existing `packages/agent/src/harness/runtime/drive/*.ts`
|
||||
8. `packages/agent/src/harness/runtime/progress.ts`
|
||||
9. `packages/agent/src/harness/runtime/restore.ts`
|
||||
10. `packages/agent/src/harness/execution/{effect-gate,assistant,tools}.ts`
|
||||
11. relevant focused runtime tests
|
||||
|
||||
Do not inspect Git history or removed runtime implementations. The current source and these documents are the only implementation inputs.
|
||||
|
||||
## 1. Runtime model (target)
|
||||
|
||||
### Lane authority
|
||||
|
||||
`Lane.state` is authoritative while a Harness owns its Session. It contains all orchestration state required to dispatch:
|
||||
|
||||
- tip;
|
||||
- lane configuration;
|
||||
- the lane inbox: tagged queued-input ids (R1);
|
||||
- last operation id (R2);
|
||||
- operation metadata;
|
||||
- flat operation state, including control.
|
||||
|
||||
Every supported mutation commits on the one Session mutation line and publishes the matching `Lane.state` before releasing the line. Drive procedures never reread `laneState`, `operationMeta`, `operationState`, `branchTip`, `laneConfig`, or `operationResult` from storage.
|
||||
|
||||
`SessionReader` is used only to dereference content named by in-memory state and to enumerate operation-owned cleanup addresses:
|
||||
|
||||
- tree entries and branch context;
|
||||
- pending entry payloads;
|
||||
- assistant frame lists;
|
||||
- tool arguments, checkpoints, and memos;
|
||||
- structural preparations;
|
||||
- staged tool outcomes;
|
||||
- cleanup-prefix scans.
|
||||
|
||||
### One lane-owned inbox (R1)
|
||||
|
||||
Queued input is lane-owned, never operation-owned. `LaneState` (durable and process-local) carries one ordered inbox of `{ entryId, kind }` items with `kind: "steer" | "followUp" | "nextRun" | "write"`. Payload staging is unchanged: enqueue mints the entry id and writes `pendingEntry(id)`; the inbox holds ids only. Item lifecycle is admission → consumption | cancellation; terminal cleanup never touches queue payloads.
|
||||
|
||||
Tags are consumption-eligibility markers, not ownership. Enqueue always succeeds — during runs, structural operations, cancellation, and idle. Every drain point selects eligible items per tag, applying queue modes during selection, but places all selected items in the inbox's single global admission order. Tag grouping must never reorder user input. At acceptance, request prompt entries follow the selected inbox items because the request is the newest admission.
|
||||
|
||||
| Drain point | Eligibility and decision order |
|
||||
| --- | --- |
|
||||
| acceptance (idle lane) | write + nextRun (all), steer (`steeringMode`: all or oldest), and followUp (`followUpMode`) are eligible. Place selected items in global admission order, then request prompt entries. The acceptance transaction places them and removes their ids; `starting` drains nothing. followUp is eligible because an idle lane vacuously satisfies its "after current work" condition. |
|
||||
| turn-end boundary pass (run) | write + steer are eligible before threshold/continuation planning and are placed in global admission order. followUp becomes eligible only at `may_finish`, before `before_run_end` and finish. nextRun is never eligible mid-run and never blocks finish. |
|
||||
| idle direct append | queued write items are placed in admission order, then the new entry, in one commit |
|
||||
| abort (M7) | steer + followUp are removed from the inbox, payload values deleted, and payloads returned; nextRun and write items stay |
|
||||
|
||||
**One decision, at most one commit (R1b).** A boundary pass may enter the mutation line several times for bounded reads and for `before_run_end` mediation, but it performs **at most one commit**, and that commit always lands in a state that neither drains nor rechecks (`assistant.ready`, `summary.deciding`, placed entries + `assistant.ready`, or a terminal transaction). No boundary decision ever commits back into `checkpoint`. A crash before the commit re-runs the whole decision with nothing consumed; a crash after it lands past the decision point. Consequently `skipInboxOnce` and `thresholdCheckedTriggerEntryId` are deleted: drains cannot re-fire because their target states do not drain, and the threshold marker is replaced by a guard derived from the branch itself — threshold fires only when `shouldCompact` holds **and** the branch's newest compaction entry is older than the trigger entry, so a committed threshold compaction is its own durable marker. Mode remainders stay queued and are consumed at later boundaries, giving the per-turn steer cadence and per-run-end followUp cadence.
|
||||
|
||||
There is no terminal drain. `cancelQueued` triage is unchanged and single-location. Queued input survives operation termination (including abort, for nextRun/write) until consumed or cancelled; this is a deliberate product decision.
|
||||
|
||||
### Flat state (R3)
|
||||
|
||||
`OperationState` has one `at` discriminator with **13** direct leaves:
|
||||
|
||||
- `starting`
|
||||
- `checkpoint`
|
||||
- `assistant.ready`
|
||||
- `assistant.effect_pending`
|
||||
- `assistant.retry_wait`
|
||||
- `tools`
|
||||
- `deferred.suspended`
|
||||
- `deferred.effect_pending`
|
||||
- `summary.deciding`
|
||||
- `summary.ready`
|
||||
- `summary.effect_pending`
|
||||
- `summary.retry_wait`
|
||||
- `navigation.ready_to_commit`
|
||||
|
||||
Every leaf carries one uniform scope `{ control, settings, latestAssistantEntryId }`. There are no `run.`/`compaction.`/`navigation.` leaf prefixes, no per-family scope types, and no intersection zoo. The four `summary.*` leaves carry a `SummaryTask` whose `boundary` datum (a closed three-arm union, §8) decides what happens at the result boundary. `starting` and `navigation.ready_to_commit` are intent-locked entry leaves. `ToolBatch`/`ToolCall` remain the nested child collection state machine. `Control` stays orthogonal and loses its drained fields when M7 installs drain-and-return abort.
|
||||
|
||||
### Durable operation results (R2)
|
||||
|
||||
Every terminal transaction writes one small immutable result record at `operationResult(operationId)` (namespace `pi.result`, operation id key). The record is lane-lived; keeping it outside `pi.op.*` keeps that namespace's "deleted no later than the terminal transaction" grammar total and exception-free. `LaneState.lastOperationId` points at the newest record. `laneLastResult` (value, address constructor, and type union) is deleted.
|
||||
|
||||
- `drive(id)` becomes total for settled operations: current id → install/join; record exists → return the record itself; neither → `OperationMismatch`.
|
||||
- Recovery and attachment never read result records; they are observation only.
|
||||
- Terminal-control invariant: a terminal transaction that executes under durable `cancel_requested` always records `status: "aborted"`; equivalently, any other status implies running terminal control. (Every non-aborted terminal path goes through `continueOperation`, which diverts under cancellation, and the mutation line serializes the marker against the terminal commit.)
|
||||
- There is no listing, filtering, pagination, or retention API for records — and no hydration layer. `getResult(operationId)` and the drive settled arm are the entire read surface; both return the record with no entry dereference, so observation can never fault on missing entries.
|
||||
|
||||
### One lane-owned Drive
|
||||
|
||||
Unchanged: one process-local Drive pass per lane; first matching caller installs, later matching callers observe the same completion; no caller owns it after installation; invocation cancellation rejects only that caller's observation; an installed Drive is never abandoned or replaced in-process; `requestAbort(operationId)` is the only durable operation cancellation; a stale operation id returns `OperationMismatch` and affects nothing. `Drive.context` removes the installing invocation signal.
|
||||
|
||||
### Close
|
||||
|
||||
Unchanged: close is not abort. It seals mutation admission, rejects local observations with `HarnessClosed`, observes detached pass failures, drains mutations admitted before the seal, and closes the Session. It writes no cancellation marker or synthetic terminal state and never replaces a Drive.
|
||||
|
||||
### No external finalization
|
||||
|
||||
Unchanged: no live-process external-finalization path, `OperationEnded`, `finalizedOutcome`, ownership-loss result, or exact-Drive ABA fence.
|
||||
|
||||
## 2. Procedure shape
|
||||
|
||||
Live procedures are ordinary straight-line async functions:
|
||||
|
||||
```text
|
||||
prepare
|
||||
→ commit intent
|
||||
→ admit and await effect
|
||||
→ commit settlement
|
||||
```
|
||||
|
||||
A procedure is the sole writer that changes its top-level `at` leaf. `requestAbort` changes only `control` (and the lane inbox); inbox admission changes only the lane inbox. Operation state has exactly two supported writers: the procedure and `requestAbort`.
|
||||
|
||||
The Lane supplies `continueOperation` and `settleOperation` as today. A terminal decision appends the universal suffix:
|
||||
|
||||
1. procedure-specific publication and cleanup writes;
|
||||
2. `setValue(operationResult(operationId), record)`;
|
||||
3. idle `laneState` with `currentOperationId: null`, `lastOperationId: operationId`, and the preserved inbox;
|
||||
4. idle process-local projection;
|
||||
5. terminal result and event materialization.
|
||||
|
||||
The state argument is a type capability established by dispatcher control flow, never compared with current state at runtime.
|
||||
|
||||
## 3. Remaining concurrency checks
|
||||
|
||||
Keep checks only where another supported writer can change the relevant fact:
|
||||
|
||||
1. `requestAbort` versus effect admission and settlement;
|
||||
2. lane-inbox arrival before a drain point, a summary result boundary, or terminal finish;
|
||||
3. parallel tool-call statuses and source-ready placement;
|
||||
4. queued frame/checkpoint writes versus settlement;
|
||||
5. invocation memo/checkpoint calls versus effect completion;
|
||||
6. retry timers versus cancellation/close;
|
||||
7. deferred permit consumption;
|
||||
8. accept/claim serialization;
|
||||
9. external/provider/content validation.
|
||||
|
||||
Do not reintroduce operation identity, expected-`at`, exact-Drive, or ownership-loss checks into ordinary transitions.
|
||||
|
||||
`requestAbort` keeps the two-step gate order: `beginAbort` before the cancellation mutation, commit `cancel_requested`, `signalAbort` after the commit.
|
||||
|
||||
## 4. Completed milestones
|
||||
|
||||
### M0 — Withdrawn execution-step controls
|
||||
|
||||
Complete. Breakpoints, manual drive, and drive deadlines are absent.
|
||||
|
||||
### M1–M2 — Foundations and terminal mechanics
|
||||
|
||||
Complete. Includes the split effect gate, deterministic gated storage, authoritative process-local Lane projection, commit-result usage totals, progress channels, terminal cleanup, and immutable operation-result observation.
|
||||
|
||||
### M3 — Assistant generation and recovery
|
||||
|
||||
Complete. Includes `run.starting → run.checkpoint`, assistant ready/intent/effect/settlement, frame persistence and cleanup, configuration failure, unknown-outcome assistant recovery, and response/usage/state atomicity.
|
||||
|
||||
### M4 — Retry and deferred
|
||||
|
||||
Complete. Includes durable retry waits, local wait policy, one deferred poll permit per pass, unknown-poll replacement under fresh ids, and event-stream-preserving `Models.streamDeferred`.
|
||||
|
||||
### M5 — Durable tools
|
||||
|
||||
Complete. Includes planned/effect-pending/outcome-ready/completed calls, safe replay, unsafe interruption, invocation memos, bounded checkpoints, completion-order staging, source-order placement, sequential and parallel modes, and tool-reported usage.
|
||||
|
||||
### Pre-M6 simplification
|
||||
|
||||
Complete. Canonical flat `at` types, `continueOperation`/`settleOperation`, removal of the installer-owned Drive model, and conversion of M3–M5 procedures.
|
||||
|
||||
### M6 structural foundation
|
||||
|
||||
Committed (`9b23c6583`): `runtime/drive/structural.ts`, `test/harness/runtime/drive-structural.test.ts`, the one-provider-request seams in the compaction modules, threshold routing, overflow preparation, per-request structural intent/usage settlement, hook decisions, structural retry/recovery, and unsummarized navigation commit. Its family cross-product layer (triplicated quadruples, three-armed publishers) is rebuilt by R3; its effect seam, preparation plumbing, threshold/overflow logic, and navigation commit are retained.
|
||||
|
||||
## 5. Withdrawn: standalone-compaction inbox and run-continuation promotion
|
||||
|
||||
The design that gave `compaction.*` leaves a `RunScope` inbox and a one-way cross-family move into `run.*` at the result boundary is withdrawn. Its problems: a relaxed intent/state invariant with bespoke restore rules, `compact()` resolving with a run outcome that does not fit `CompactionResult`, suppressed `run_start` producing unbalanced event brackets, fused-consumption rules to avoid `cancelQueued` stranding, and promotion-specific arms in reconciliation, cleanup, watch, and admission.
|
||||
|
||||
Replacement, spread across this package:
|
||||
|
||||
- queued input during any operation is lane-owned (R1) — no operation inbox exists to promote;
|
||||
- continuation after standalone compaction/navigation is a **second ordinary run operation** with a fresh id, composed in the convenience layer (M8);
|
||||
- each operation delivers exactly one result; both results are returned by the convenience call and both are durable records (R2).
|
||||
|
||||
No code implements promotion. M9 removed the withdrawn design from `harness.md`; this section remains only as the historical decision record.
|
||||
|
||||
## 6. R1 — Lane-owned tagged inbox
|
||||
|
||||
### Goal
|
||||
|
||||
Move all queued input to one lane-owned tagged inbox first (R1a, implemented), then — after R3 — replace the transitional stepwise checkpoint drains with one shared atomic boundary planner (R1b). `Control.drained*` remains until M7 introduces drain-and-return abort; it is not part of inbox ownership.
|
||||
|
||||
### R1a — Ownership relocation
|
||||
|
||||
- `src/harness/session/types.ts` — durable `LaneState { currentOperationId, inbox }` with `InboxItem { entryId, kind }` replaces `pendingNextRun`; delete `Inbox` and `RunScope.inbox`. Keep `Control.drained*`, `skipInboxOnce`, and `thresholdCheckedTriggerEntryId` temporarily.
|
||||
- `src/harness/runtime/types.ts` — process-local `LaneState.inbox` replaces `pendingNextRun`; `LanePatch.inbox` pairs durable lane-state replacement with process-local publication.
|
||||
- `src/harness/runtime/lane.ts` — acceptance selects per tag and mode but places selected items in global admission order, then request prompt entries; it deletes selected pending values and removes their ids atomically with the `LaneBusy` check. A lone queued write never validates acceptance. `append` during any operation enqueues a write-tagged lane item without touching operation state; idle `append` places queued writes first, then the new entry, in one commit. `watch` derives queues and pending writes from the lane inbox.
|
||||
- `src/harness/runtime/drive/checkpoint.ts` — transitional stepwise drains read and replace the lane inbox rather than operation state. `skipInboxOnce` and `thresholdCheckedTriggerEntryId` continue to protect those temporary multi-commit boundaries.
|
||||
- `src/harness/runtime/drive/terminal.ts` — operation cleanup never deletes lane-inbox payloads. Staged `outcome_ready` results, drained abort payloads, and other operation-owned addresses are unchanged.
|
||||
- `src/harness/runtime/drive/{generation,response,deferred,tools,structural}.ts` — `runScopeOf` and scope copies drop `inbox`; ordinary operation-state transitions cannot clobber concurrent input.
|
||||
- `src/harness/runtime/restore.ts`, fork/legacy normalization, and focused tests use the tagged lane inbox.
|
||||
|
||||
### R1b — Shared atomic boundary planner
|
||||
|
||||
**Lands after R3**, because pre-R3 the structural result boundary is spread across the triplicated publishers: fusing the planner into three arms and then collapsing them in R3 would be double work, while post-R3 the single `ResultBoundary` switch is exactly one planner call site. The two `CheckpointData` patch fields ride through R3's mechanical rename as inert baggage.
|
||||
|
||||
**Core invariant: no boundary decision ever commits back into `checkpoint`.** `checkpoint` is the durable "boundary pending" resting leaf, entered only from settlements, run-start consumption, and deferred redemption. Every exit of its pass is `assistant.ready` (with any placed entries), `summary.deciding`, a placed follow-up + `assistant.ready`, or the terminal transaction.
|
||||
|
||||
One shared boundary planner with a process-local `threshold: "check" | "skip"` parameter has exactly two callers:
|
||||
|
||||
- the `checkpoint` pass (`threshold: "check"`): select write + steer (modes, global admission order) → threshold guard → continuation → followUp (at `may_finish`) → `before_run_end` → finish; one commit;
|
||||
- the structural result boundary for `resume_checkpoint` (`threshold: "skip"`): the **same planner runs inside the publication commit** — compaction entry (on success) first, then selected write/steer items, then the routed continuation, all in one transaction. Any placed conversational item routes to `assistant.ready` with that item as trigger, overriding a `may_finish` resume continuation (steer semantics). With nothing queued: `need_assistant` resumes commit `assistant.ready` directly; `may_finish` resumes commit `checkpoint{may_finish}` as the resting leaf and the live pass continues into the finish phases.
|
||||
|
||||
**Threshold without the durable marker.** The guard is derived from the branch: threshold fires only when `shouldCompact` holds and the newest compaction entry is older than the trigger entry. A committed threshold compaction is therefore its own marker — any crash re-entry sees the newer compaction and skips. A **declined** threshold compaction publishes no entry and needs no marker: decline never lands on `checkpoint`. The decline decision stays process-local at `summary.deciding` until its single continuation commit (`assistant.ready`, follow-up + ready, or the terminal after `before_run_end`); a crash in that window restores `deciding` and re-runs `before_compaction` under its documented repetition contract. A live decline/recheck loop is structurally impossible because no decline path re-enters the threshold-checking pass.
|
||||
|
||||
**`before_run_end` without hooks on the mutation line.** The finish arm is the one multi-phase case: (1) on-line verdict, no commit — reached only at `may_finish` with no eligible write/steer/followUp; (2) hook runs off the line; (3) on-line **replan from current state** — if the inbox gained eligible items or control changed, the stale hook result is dropped and the planner takes the new decision; otherwise the single commit is the follow-up injection (entry + `assistant.ready`) or the terminal transaction. Stale-dropping is replanning, not a version check.
|
||||
|
||||
Concrete control trace (post-R3 names; threshold compaction with `may_finish` resume and a steer arriving mid-compaction):
|
||||
|
||||
```text
|
||||
TX1 settlement: response n7 + usage + tip, S=checkpoint{may_finish, trigger n7}
|
||||
pass (check): shouldCompact && newestCompaction < n7 → prepare
|
||||
TX2: preparation, S=summary.deciding{boundary: resume_checkpoint{may_finish, n7}}
|
||||
TX3 (any time): steer s1 → pendingEntry(s1), laneState.inbox += s1
|
||||
TX4..n: deciding → ready → effect_pending → nested request/usage settlements
|
||||
TX-pub (skip): insert compaction c1, tip=c1, insert s1 entry (parent c1),
|
||||
delete pendingEntry(s1), laneState.inbox -= s1,
|
||||
S=assistant.ready{trigger s1} ← one commit, checkpoint never re-entered
|
||||
…turn answers s1; next settlement → checkpoint{may_finish, trigger n9}
|
||||
pass (check): newestCompaction c1 > … tokens small → no threshold; inbox empty; verdict finish
|
||||
→ before_run_end off-line → replan: still empty, still running
|
||||
TX-final: record + cleanup + laneState{current: null, last: op}
|
||||
```
|
||||
|
||||
Crash between TX4..n and TX-pub → structural recovery (attempt unknown, per existing rules); crash during the hook window → `checkpoint{may_finish}` restored, pass re-runs (threshold skipped by the recency guard), hook re-runs per contract.
|
||||
|
||||
Slices, in order (the order is load-bearing — deleting the marker while decline still re-entered `checkpoint` would live-loop):
|
||||
|
||||
1. **R1b-1**: fuse the future shared planner into the structural `resume_checkpoint` boundary (`threshold: "skip"`); structural success/decline route directly except for a `may_finish` resting checkpoint, which remains safe for either outcome while the durable marker still exists. Keep the route inline until R1b-2 creates its second caller. Both patch fields are still written and checked by the old checkpoint pass — harmless.
|
||||
2. **R1b-2**: convert the `checkpoint` pass to single-commit exits with the recency guard, implement the three-phase finish arm, delete `skipInboxOnce` and `thresholdCheckedTriggerEntryId`.
|
||||
|
||||
### Rules
|
||||
|
||||
- The §1 eligibility table, global placement order, and one-decision-one-commit target are normative.
|
||||
- Queue modes apply at selection time. Remainders preserve global admission order and are consumed only at later eligible boundaries.
|
||||
- nextRun items are never consumed mid-run and never block run finish.
|
||||
- Deferred-suspension enqueue always admits; consumption happens at the post-redemption boundary.
|
||||
|
||||
### Focused validation
|
||||
|
||||
Enqueue during idle/run/structural/cancelled states always admits; acceptance preserves global order with modes applied and remainders retained; `one-at-a-time` cadence gives each remaining steer its own later boundary; followUp is captured at idle acceptance and only at `may_finish` during a run; a boundary crash before its commit consumes nothing; threshold is checked at most once per boundary by construction; idle append places queued writes first; items survive every terminal status and process loss; `cancelQueued` works at every boundary; watch groups by tag without changing order; no terminal transaction deletes a lane-inbox payload; a lone queued write never validates acceptance.
|
||||
|
||||
## 7. R2 — Neutral operation outcome and durable result records
|
||||
|
||||
### Goal
|
||||
|
||||
Replace the three per-family outcome unions and `laneLastResult` with one immutable per-operation result record. The record **is** the public settled outcome; there is no separate outcome type, no embedded entries, and no hydration.
|
||||
|
||||
### Types
|
||||
|
||||
```ts
|
||||
type TerminalStatus = "completed" | "declined" | "aborted" | "failed";
|
||||
|
||||
/** Stored at operationResult(operationId) — namespace "pi.result" — by the terminal transaction. Immutable, lane-lived. */
|
||||
interface OperationResultRecord {
|
||||
operationId: string;
|
||||
kind: "run" | "compaction" | "navigation"; // meta.intent.kind; matches OperationAdmission.kind
|
||||
status: TerminalStatus;
|
||||
error?: OperationError; // status "failed"
|
||||
fromTipId: string | null; // meta.sourceTipId — start of the transcript segment
|
||||
tipId: string | null; // lane tip at terminal — end of the segment
|
||||
startedAt: number; // Unix ms, from meta
|
||||
endedAt: number; // Unix ms, Date.now() at terminal planning
|
||||
}
|
||||
|
||||
/** Convenience-only suspension observation for prompt()/resume() (M8). Never stored. */
|
||||
interface SuspendedRun { operationId: string; status: "suspended"; deferred: DeferredHandle }
|
||||
```
|
||||
|
||||
The record is a disposition plus a pointer to the transcript segment `(fromTipId, tipId]`. It never lists intermediate work (compactions, turns, tools) and never embeds entries. There is **no `tipEntry` and no `OperationOutcome` union**: a caller that wants the payload dereferences `tipId` with the already-public `getEntry`/`findEntry` (for runs, the final message was additionally delivered by `message_end`/`entry_added`). This buys three things: terminal planners perform zero outcome reads, observation cannot fault on a missing entry (a `getEntries` throw inside hydration was a harness-fault path for a read-only convenience), and the future protocol never serializes `Entry` inside results — a result frame is eight flat fields. `SuspendedRun` is the one non-terminal observation, exists only on convenience returns, and carries nothing derivable elsewhere.
|
||||
|
||||
### Files
|
||||
|
||||
- `src/harness/session/types.ts` — add the record; delete the `LaneLastResult` union; `LaneState` gains `lastOperationId: string | null`; delete `failure_drain`, `RunFailureDrainOperation`, and `FailureProvenance`.
|
||||
- `src/harness/session/values.ts` — `operationResult(operationId) = value<OperationResultRecord>("pi.result", operationId)`; delete `laneLastResult`. The `pi.op.*` lifetime grammar stays total: no operation-lived namespace survives its terminal transaction.
|
||||
- `src/harness/agent-harness.ts` — delete `RunOutcome`, `CompactionOutcome`, `NavigationOutcome`, `OptionalFinalAssistant`, `TerminalOperationOutcome`, `ResumeOutcome`, and the transitional `OperationOutcome` union; `DriveOutcome.settled` carries `OperationResultRecord` directly (no duplicate `operationId` field); result aliases become
|
||||
`RunResult = Result<OperationResultRecord | SuspendedRun, …>`,
|
||||
`CompactionResult = Result<{ compaction: OperationResultRecord; run?: OperationResultRecord | SuspendedRun }, …>`,
|
||||
`NavigationResult = Result<{ navigation: OperationResultRecord; run?: OperationResultRecord | SuspendedRun }, …>`,
|
||||
`ResumeResult = Result<OperationResultRecord | SuspendedRun, …>`;
|
||||
`QueueResult` loses `NoActiveRun` and absorbs `NextRunResult`. `SuspendedRun` is declared here but constructed only by M8 convenience paths.
|
||||
- End-event payloads, one shape per event type: `run_end` and `navigation_end` carry `{ runId, status, error?, fromTipId, tipId }`. `compaction_end` is a segment event, not a terminal event — it also closes in-run threshold/overflow brackets — and always carries `{ runId, reason, status: "completed" | "declined" | "failed" | "aborted", error?, entryId? }`, where `entryId` names the compaction entry on success. Embedded `entry`/`summaryEntry`/final-assistant fields are dropped everywhere (`entry_added`/`message_end` already delivered payloads).
|
||||
- `src/harness/runtime/types.ts` — `FinishDecision` carries the record instead of `lastResult`.
|
||||
- `src/harness/runtime/lane.ts` — terminal suffix per §2; `getLastResult` is replaced by `getResult(operationId, context): Promise<OperationResultRecord | undefined>` — one `getValue`, nothing else; `inspectExecution` reports `lastOperationId` only.
|
||||
- `src/harness/runtime/drive/terminal.ts` — `hydrateTerminalOutcome`/`hydrateOperationOutcome` are deleted outright; the file keeps only `operationCleanupWrites` and the `operationResultRecord` constructor. Finish decisions materialize `{ kind: "settled", outcome: record }` from the record they already built — no reader dereference inside any terminal planner.
|
||||
- `src/harness/runtime/drive/{checkpoint,response,structural,recovery,deferred,tools}.ts` — every finish decision constructs the record; `runCompletion`, per-family last-result construction, and final-assistant plumbing in results are deleted. Every former `failure_drain` producer instead commits terminal failure directly: response publication and cleanup stay in the same transaction, in-run structural failure closes `compaction_end` then `run_end`, and queued lane input remains untouched for a later ordinary run. (`latestAssistantEntryId` stays in scope for cancellation classification and checkpoint logic; it just no longer feeds results.)
|
||||
- `src/harness/runtime/restore.ts` — restore `lastOperationId`; attachment never reads records.
|
||||
- Focused tests.
|
||||
|
||||
### Rules
|
||||
|
||||
- Records are written exactly once, by the terminal transaction, and never deleted, updated, or read by recovery.
|
||||
- Terminal-control invariant (§1): a terminal transaction under durable `cancel_requested` records `aborted`; no path commits any other status under cancellation. M7 adds the enforcing test; M9 adds the Part 9 invariant.
|
||||
- Forks exclude `pi.result` values.
|
||||
- `drive(id)` arms: current → install/join; record → returned directly; neither → `OperationMismatch`.
|
||||
|
||||
### Focused validation
|
||||
|
||||
Record written for every terminal path of every operation kind and status; `drive(id)` returns records for arbitrarily old ids across reopen; `getResult` is a plain value read returning the record or `undefined`; no terminal planner or observation path reads entries for the result; `lastOperationId` restore; suspension never stored; fork exclusion; segment pointers correct (`fromTipId` = pre-acceptance tip, including navigation); `compaction_end` closes both standalone and in-run brackets, including `aborted` via reconciliation; each former failure-drain producer finishes in one transaction and preserves every lane-inbox item for a later run.
|
||||
|
||||
## 8. R3 — Family-neutral leaves and structural rebuild
|
||||
|
||||
### Goal
|
||||
|
||||
Collapse 22 leaves to the 13 in §1; rebuild `structural.ts`'s state-shape layer on one summary quadruple with an explicit result-boundary datum.
|
||||
|
||||
### Types
|
||||
|
||||
```ts
|
||||
interface OperationScope { control: Control; settings: RunSettings; latestAssistantEntryId: string | null }
|
||||
|
||||
type ResultBoundary = // closed; do not extend
|
||||
| { kind: "resume_checkpoint"; resumeAfter: CheckpointData } // in-run threshold/overflow
|
||||
| { kind: "finish" } // standalone compaction
|
||||
| { kind: "commit_navigation"; targetId: string; label?: string };
|
||||
|
||||
interface SummaryTask {
|
||||
taskId: string;
|
||||
reason?: "manual" | "threshold" | "overflow";
|
||||
customInstructions?: string;
|
||||
boundary: ResultBoundary;
|
||||
}
|
||||
```
|
||||
|
||||
The summary algorithm kind is **derived from the boundary** (`resume_checkpoint`/`finish` → compaction; `commit_navigation` → branch summary); it is not stored, so no contradictory kind/boundary combination is representable. `summary.ready/effect_pending/retry_wait` additionally carry the generation snapshot (result entry id, configuration, stream options, retry policy, attempt counters); the snapshot drops the `taskId`/`kind`/`reason` fields it previously duplicated. Canonical declarations land in `session/types.ts`. Uniform scope means compaction/navigation acceptance captures `settings` exactly like run acceptance; `latestAssistantEntryId` stays null for structural intents.
|
||||
|
||||
### Reachability (restore check; replaces the intent-prefix check)
|
||||
|
||||
| Intent | Admissible leaves |
|
||||
| --- | --- |
|
||||
| run | `starting`, `checkpoint`, `assistant.*`, `tools`, `deferred.*`, `summary.*` with boundary `resume_checkpoint` |
|
||||
| compaction | `summary.*` with boundary `finish` |
|
||||
| navigation | `navigation.ready_to_commit`; `summary.*` with boundary `commit_navigation` |
|
||||
|
||||
Forbidden and unreachable by construction (assert in tests, never implement): any edge into `starting` or `navigation.ready_to_commit` other than acceptance; `summary.*` → `tools`/`assistant.*` directly; `navigation.ready_to_commit` → `summary.*`; terminal → anything.
|
||||
|
||||
### Result-boundary rule
|
||||
|
||||
Every summary boundary (hook decline, hook-supplied result, generated success, terminal generation failure, model unavailability) plans on the mutation line and switches on `boundary` in one visible place:
|
||||
|
||||
- `resume_checkpoint` → publish result and restore the marked checkpoint on success or threshold decline; overflow decline or structural failure closes the compaction bracket and terminal-fails the run in the same commit;
|
||||
- `finish` → publish the compaction entry and terminal-complete (or terminal declined/failed) in one commit;
|
||||
- `commit_navigation` → the single move/summary/label/terminal commit (or terminal declined/failed).
|
||||
|
||||
A `cancel_requested` boundary defers to reconciliation and never takes its continuation.
|
||||
|
||||
### Files
|
||||
|
||||
- `src/harness/session/types.ts` — the 13-leaf union, `OperationScope`, `SummaryTask`, `ResultBoundary`; delete `RunScope`, `CompactionScope`, `NavigationScope`, `NavigationSummaryScope`, `RunCompactionScope`, `StructuralTask`, per-family `Extract` aliases, `isRunOperationState`.
|
||||
- `src/harness/runtime/drive/structural.ts` — rebuild: keep `durableCompactionPreparation`/readers, the nested request seam, `performStructuralAttempt`, `runCompactionThreshold`, `prepareOverflowCompaction`, `commitNavigation`; delete the seven union aliases, the `startsWith` guards, and the three-armed `effectPendingFromReady`/`retryWaitFromEffect`/`readyFromRetryWait`/`publishStructuralReady`/`publishStructuralDecline`/`publishStructuralFailure`/`publishCompactionResult`-vs-`publishNavigationSummary` split; one quadruple of converters plus one boundary switch replace them.
|
||||
- `src/harness/runtime/drive/{checkpoint,generation,response,recovery,deferred,tools,tool-placement,terminal}.ts` — leaf literal renames (`run.checkpoint` → `checkpoint`, …); `response.ts` overflow arm constructs `summary.deciding` with `resume_checkpoint`; no other behavior change.
|
||||
- `src/harness/runtime/progress.ts` — frame/checkpoint fences currently match the leaf literals `"run.assistant.effect_pending"`, `"run.deferred.effect_pending"`, and `"run.tools"`; retarget them to the neutral leaves.
|
||||
- `src/harness/runtime/restore.ts` — reachability predicate above.
|
||||
- `src/harness/runtime/lane.ts` — `capturedModel` and `watch` switch over the neutral leaves; acceptance writes neutral initial leaves.
|
||||
- `docs/runtime-simplification.md` — replace the 22-leaf list and status.
|
||||
- `test/harness/runtime/drive-structural.test.ts` and other focused tests — retarget; behavioral assertions survive.
|
||||
|
||||
### Focused validation
|
||||
|
||||
Everything the M6 foundation covered, re-expressed over neutral leaves, plus: reachability accept/reject matrix (including corrupted boundary/intent combinations faulting restore); each boundary arm × {success, hook result, decline, failure, model absence} × {running, cancelled}; a post-terminal leak scan asserting every `pi.op.*` address is gone for every leaf while the `pi.result` record is present; crash/reopen at every leaf under every intent that can reach it.
|
||||
|
||||
## 9. M7 — Cancellation reconciliation and total switch
|
||||
|
||||
### Goal
|
||||
|
||||
**Implemented and reviewed.** Make the internal graph total without public wiring.
|
||||
|
||||
Create `src/harness/runtime/drive/reconcile.ts`, `src/harness/runtime/drive.ts`, and focused reconciliation and switch tests.
|
||||
|
||||
### `requestOperationAbort`
|
||||
|
||||
Package-private expected-id primitive:
|
||||
|
||||
1. reject with `OperationMismatch` when the expected id is not the current durable operation, including when that id already has a settled result record;
|
||||
2. synchronously `beginAbort` on a matching live Drive;
|
||||
3. one commit on the Session line: `control = cancel_requested` (with no drained fields) **plus** removal of steer/followUp-tagged ids from the lane inbox and deletion of their `pendingEntry` values; payloads are read in the same mutation and returned; nextRun/write items stay;
|
||||
4. `signalAbort` after the commit;
|
||||
5. after a newly requested abort commits, publish `{ type: "operation_abort", operationId, steer, followUp }` and return once cancellation is durable. This family-neutral event replaces `run_abort`/`runId`; the lane snapshot already identifies the operation kind. Repeat calls against the same current cancelled operation publish nothing, drain nothing further, and report `newlyRequested: false`.
|
||||
|
||||
With no Drive but a matching current durable operation, it commits the same marker and starts no pass. There is no `control.drained*`; the drained payloads exist only in the returned result (accepted loss: a crash inside the abort window loses their content).
|
||||
|
||||
### Reconciliation
|
||||
|
||||
Checked before `before_drive` and ordinary dispatch; a single switch over the 13 leaves; never starts new ordinary work. It must handle:
|
||||
|
||||
- assistant and deferred effects with live or reconstructed results;
|
||||
- planned/effect-pending/outcome-ready tool calls;
|
||||
- structural process-local results (discarded unless already atomically published);
|
||||
- cancelled summary boundaries finishing `aborted` without taking their `ResultBoundary` continuation;
|
||||
- retry waits, checkpoints, and suspended deferred work;
|
||||
- best-effort deferred-provider cancellation using the Drive's close-only signal, never its already-triggered operation-abort gate;
|
||||
- the aborted terminal transaction (result record `status: "aborted"`);
|
||||
- the terminal-control invariant test: no terminal path commits a non-aborted status under cancelled control (this keeps §10's continuation rule decidable from the record alone).
|
||||
|
||||
Queued lane-inbox items are **not** applied or deleted by reconciliation; nextRun/write items simply remain queued.
|
||||
|
||||
Each Drive owns a private close controller and exposes its signal to mandatory cleanup. `closeGate()` aborts that controller on harness close or fault; operation abort does not. Deferred-provider cancellation uses this close-only signal, so the cleanup request can start after durable operation cancellation but cannot outlive harness shutdown. Do not add a harness-global cancellation controller.
|
||||
|
||||
### Total switch
|
||||
|
||||
`runtime/drive.ts` owns one direct `state.at` switch over 13 leaves. It imports only complete procedure modules. No graph table, action interpreter, ownership-loss arm, external-finalization arm, or storage-state reload. A `continue` result must correspond to a replaced `Lane.state` projection **or** to currently observable `cancel_requested` control, which the switch routes to reconciliation before ordinary dispatch; any other unchanged continuation is an invariant defect.
|
||||
|
||||
Public methods remain guarded through M7.
|
||||
|
||||
## 10. M8 — Public surfaces
|
||||
|
||||
**Implemented.** Execution guards were removed only after every leaf and reconciliation path became total.
|
||||
|
||||
Order:
|
||||
|
||||
1. admit compaction/navigation requests (acceptance captures `settings`; writes the R3 entry leaves);
|
||||
2. implement `drive` install/join/record lookup (three arms, §7);
|
||||
3. expose `requestAbort`;
|
||||
4. add convenience compositions;
|
||||
5. add queues/configuration/usage/idle surfaces;
|
||||
6. retain `watchSession` as the sole `SliceNotImplemented` method.
|
||||
|
||||
### Queues
|
||||
|
||||
`steer`/`followUp`/`nextRun` are tag sugar over one enqueue and always admit. `queue_update` and `LaneSnapshot.queues` expose the same tagged ordered inbox, including pending writes; clients group by tag without reordering it.
|
||||
|
||||
### Client replication surface
|
||||
|
||||
Findings from a working RPC-shaped client replica (mini TUI exercise) fold in here. The goal is one testable contract: a remote client renders exclusively from a replicated `LaneSnapshot` and the event stream, with no side-channel getters.
|
||||
|
||||
- **Normative reducer.** Export `reduceLaneSnapshot(snapshot, event): LaneSnapshot | { rebase: true }` from `packages/agent`. It is the single supported event fold: in-run `compaction_start`/`compaction_end` are segments and must not clear `operation` (the fold knows the open operation's kind, so no event field is needed); `run_suspend` keeps `operation` non-null with `deferred` set; only `run_end`, `navigation_end`, and `compaction_end` under a compaction-kind operation are operation-terminal. `navigation_end` returns `{ rebase: true }` because the moved tip may lie outside the replica. Conformance assertion: for every non-navigation flow, folding a `watch()` snapshot over its own event stream equals a later `watch()` snapshot — this equivalence test is what keeps the event vocabulary complete.
|
||||
- **Snapshot completeness.** `LaneSnapshot` gains `configuration: LaneConfiguration`, `lastResult?: OperationResultRecord` (replacing the bare `lastOperationId` field), a real `faulted` flag (the current hardcoded `false` is a defect), and session `stats: SessionStats` as the usage baseline (usage events already carry `totals`; the snapshot supplies the value before the first event).
|
||||
- **Replicable configuration events.** Every `config_update` variant whose value is data carries `value`/`previous` (`streamOptions`, `retryPolicy`, `compactionSettings`, `steeringMode`, `followUpMode`, plus the existing lane variants); `tools`/`resources` remain notification-only because registries are code — clients re-fetch names.
|
||||
- **Identity-based `setModel`.** `setModel` accepts `ModelIdentity` (`{ provider, modelId }`); a live `Model` object is a process-local registry concern, and an unregistered identity fails in-band at generation exactly like any registry absence.
|
||||
- **Re-basing.** `WatchHandle.resnapshot(context): Promise<LaneSnapshot>` captures a fresh snapshot on the mutation line, serialized against the same stream — the recovery path after `{ rebase: true }` with no teardown/re-subscribe choreography.
|
||||
- **Start-event timestamps.** `run_start`/`compaction_start`/`navigation_start` carry `startedAt` (operation starts: `meta.startedAt`; in-run segment starts: the commit timestamp), so folds never invent times.
|
||||
|
||||
### Convenience compositions and continuation
|
||||
|
||||
- `prompt`/`skill`/`promptFromTemplate`: accept + drive; return `Result<OperationResultRecord | SuspendedRun, …>` — the record for terminal outcomes, `SuspendedRun` when the run defers.
|
||||
- `compact`/`navigateTree`: accept A + drive A. If A settled with status `completed`, `declined`, or `failed` and eligible steer/followUp/nextRun items exist, accept a continuation run B with the **public empty-prompt request** (`{ kind: "prompt", prompt: "" }`): acceptance places no request messages and is legal exactly when its capture places at least one conversational item; `OperationMeta.intent` is an ordinary run intent with empty `promptEntryIds`. Then drive B and return `{ compaction|navigation: A, run?: B }`. Never after `aborted`: abort already drained steer/followUp, and the terminal-control invariant (§7) makes `status ∈ {completed, declined, failed}` imply the operation was never durably cancelled, so the rule is decidable from the record alone. B is an ordinary run — it emits `run_start`, runs `before_run` (with `prompt: []`, the existing captured-only acceptance shape), and owns the full run graph with no continuation-aware special case.
|
||||
- Continuation races: a competing accept that wins the idle window captures the queued input into its own run; the continuation accept then fails empty (`InvalidMessage`) or `LaneBusy`, and the convenience returns A without `run`. Both histories are valid. A crash between A's terminal and B's acceptance leaves the items queued; no auto-start on reopen.
|
||||
- `resume`: inspect + drive current id, one deferred-poll permit; returns `OperationResultRecord | SuspendedRun`.
|
||||
- `abort`: inspect + `requestAbort` + ensure a reconciliation pass; returns the drained steer/followUp payloads.
|
||||
- `getResult(operationId)`: public.
|
||||
- Equivalence: every convenience call ≡ its primitive composition using only public request kinds (`compact()` with continuation ≡ `accept(A); drive(A); accept({ kind: "prompt", prompt: "" }); drive(B)`), byte-identical writes and events — externally reproducible by any scheduler.
|
||||
|
||||
### Focused validation
|
||||
|
||||
One install and same-id joins; stale-id isolation; record lookup for old ids; caller cancellation before/after installation; close/fault during cooperative and non-cooperative effects; accept/drive vs convenience equivalence including continuation via the public empty-prompt request; steer-only, followUp-only, and nextRun-only continuation each start run B; both continuation race orders; abort drain-and-return including the both-orders abort/settlement race; enqueue during every state through public surfaces; full crash matrix across all 13 leaves; no unhandled detached rejection; no `SliceNotImplemented` except `watchSession`; reducer fold-equivalence across every non-navigation flow (including in-run compaction segments, suspend/resume, retry waits, and reattach mid-stream); `navigation_end` fold returns rebase and `resnapshot` restores equivalence; snapshot `configuration`/`lastResult`/`faulted`/`stats` replicate through events alone.
|
||||
|
||||
## 11. M9 — Documentation reconciliation
|
||||
|
||||
**Implemented.** `docs/harness.md` is reconciled to the runtime and normative on its own. Updated sections include:
|
||||
|
||||
- §1.3 address table and lifetime grammar: new lane-lived `pi.result` namespace, `laneLastResult` removed, `LaneState.inbox`/`lastOperationId`; `pi.op.*` stays strictly operation-lived;
|
||||
- §1.7 JSONL/SQLite examples referencing `pi.lane.lastResult`; state the bounded growth tradeoff — one small immutable record per operation, retained forever, carried through JSONL snapshot compaction;
|
||||
- §2.9 precise rewrite: decide and document the policy for records whose `tipId` the rewrite removes (retain-dangling or delete);
|
||||
- §3.1–§3.2 state shapes: 13 neutral leaves, `OperationScope`, `SummaryTask`/`ResultBoundary` (kind derived from boundary), `Control` without drained fields, `CheckpointData` without `skipInboxOnce`/`thresholdCheckedTriggerEntryId`;
|
||||
- §3.5 graph (one summary quadruple, boundary arms);
|
||||
- §3.6 acceptance: per-tag mode-respecting selection, global admission-order placement, prompt entries last, idle followUp eligibility, and empty-prompt continuation acceptance;
|
||||
- §3.9/§3.10 summary machinery expressed once over the boundary datum;
|
||||
- §3.11 complete rewrite: one lane inbox, the selection table, one-decision-one-commit boundaries, silent deferral of late steer (an unconsumed steer becomes future-run input rather than an error), and unbounded write pendency during structural operations with the `waitForIdle`-then-append escape hatch;
|
||||
- §3.12 checkpoint procedure: the one-commit boundary decision replaces the stepwise algorithm; failures terminalize while queued lane input remains for a later ordinary run;
|
||||
- §3.13 terminal transactions: result records, universal suffix, observation contract (`drive(id)` total, `getResult`);
|
||||
- §4.6 abort: drain-and-return, no drained control, and the client-visible consequence — drained steer/followUp payloads exist only in the returned result, so a crash or lost response in that window loses their content permanently;
|
||||
- §5.1 lane surface and all result types; §5.5 `queue_update` and end-event payloads (`compaction_end` as a segment event with `aborted`);
|
||||
- Part 9: invariants 12–16, 21, 26 (result-record lifetime, observation via records, continuation equivalence), the new terminal-control invariant (a non-aborted terminal status implies running terminal control), and the race catalog rows touching drained items, `cancelQueued`, and continuation;
|
||||
- Appendix A glossary (Inbox, Result record, Continuation run, Boundary pass; remove Drained);
|
||||
- §5.5 event taxonomy: state explicitly which events are operation-terminal (`run_end`, `navigation_end`, compaction-kind `compaction_end`) versus segment brackets (in-run `compaction_start`/`compaction_end`) versus non-terminal lifecycle (`run_suspend` leaves the operation durably open), and name `reduceLaneSnapshot` as the normative fold;
|
||||
- §2.5 branch-scan sharp edge: `stopAtType` applies after ordering, so `oldestFirst` + `stopAtType: "compaction"` returns the oldest segment; document the canonical context read as `newestFirst` + reverse;
|
||||
- serving-layer boundary statement: branch-relative reads/appends remain on `AgentLane`, while whole-tree browsing, forks, label inventory, and session listing live beside it in Session/repository services composed by an RPC facade.
|
||||
|
||||
Also update `docs/runtime-simplification.md` status and remove every remaining promotion reference in the repository's docs. Grep gate:
|
||||
|
||||
```bash
|
||||
rg -i 'promotion|drainedSteer|drainedFollowUp|laneLastResult|lastResult|skipInboxOnce|thresholdChecked' packages/agent/src packages/agent/docs
|
||||
```
|
||||
|
||||
(matches must be zero in `src`; docs may retain historical notes only inside completed-milestone records.)
|
||||
|
||||
## 12. M10 — Provider KV-cache identity review
|
||||
|
||||
### Goal
|
||||
|
||||
**Implemented and reviewed.** Core Session identity alone is not a valid provider cache lineage: several lanes may issue concurrent requests over divergent transcripts. Ordinary assistant requests therefore derive identity from Session metadata id plus lane name; structural requests remain isolated.
|
||||
|
||||
### Review scope
|
||||
|
||||
- Inventory every harness and pi-ai path that sets, preserves, derives, or consumes `SimpleStreamOptions.sessionId`, including prompt-cache keys, affinity headers, WebSocket/session-resource caches, deferred requests, and structural summary requests. Ordinary harness generation now forwards the derived lane identity; legacy `src/agent.ts` keeps its independent conversation id; structural summary requests mint fresh ids.
|
||||
- Define a provider-request cache lineage distinct from the durable Session id and operation id. Concurrent lanes must never share one lineage merely because they belong to the same Session.
|
||||
- Derive ordinary assistant lineage as `Session metadata id + ":" + lane name`; do not store another durable identifier. It remains stable for the lane's lifetime and differs across lanes in one Session.
|
||||
- Preserve same-lane reuse across ordinary assistant turns and retries. Compaction, navigation, branch replacement, and model changes may miss an old prefix, but cannot incorrectly reuse it; do not add rotation machinery.
|
||||
- Keep structural summary requests isolated: `cacheRetention: "none"` and fresh request identity per nested request remain the baseline unless the review proves a safe, useful alternative.
|
||||
- Distinguish cache identity from observability correlation. Session, lane, operation, task, and request ids remain available to telemetry without being reused blindly as provider affinity/cache keys.
|
||||
- Review provider-specific semantics rather than assuming `sessionId` means only KV caching; some adapters also use it for headers, WebSocket reuse, fallback state, or resource cleanup.
|
||||
|
||||
### Focused validation
|
||||
|
||||
- Two active lanes in one Session issue concurrent, divergent prompts and receive distinct provider cache/affinity identities.
|
||||
- One lane keeps a stable lineage across append-only assistant/tool turns where reuse is valid.
|
||||
- Context-prefix discontinuities safely miss old cached prefixes without lineage rotation.
|
||||
- Assistant retries retain the lane identity; deferred handle polling needs no cache identity.
|
||||
- Structural split-turn requests remain isolated from transcript assistant caching.
|
||||
- Faux-provider tests assert stable same-lane and distinct cross-lane request identities.
|
||||
|
||||
### Exit condition
|
||||
|
||||
Ordinary assistant requests use the derived lane identity, structural requests keep fresh identities with `cacheRetention: "none"`, and deferred handle polling sends no cache identity. The policy is documented in `docs/harness.md` and independently reviewed before WP05 completion.
|
||||
|
||||
## 13. Mobile assistant-output handoff
|
||||
|
||||
**Tracked follow-up; not part of the M8/M10 public-drive gate.** A trivial mini coding-agent Session produced an approximately 300 KB JSONL file because live assistant streaming persists many `pi.pending.assistant_frame` list appends. Logical frame cleanup does not reclaim bytes already appended to the JSONL history, so short conversations can have disproportionate durable storage and replay cost.
|
||||
|
||||
The authoritative follow-up is the [mobile assistant-output handoff](../mobile-handoff/01-harness/05-assistant-output/message-update.md) and its numbered prerequisites in the [mobile handoff README](../mobile-handoff/README.md). It addresses the complete path rather than only batching frames: Chord op tracking, scoped pending-output durability, tool/assistant output reduction, and `message_update` replication amplification. Preserve the existing crash contract: an admitted assistant effect remains reconstructible, progress observation remains useful, and settlement retires operation-owned pending state.
|
||||
|
||||
Exit checks:
|
||||
|
||||
- representative short streamed responses do not create hundreds of kilobytes of durable frame history;
|
||||
- crash/reopen at every assistant and deferred effect boundary reconstructs the same pending message;
|
||||
- frame/checkpoint writers remain fenced to the owning durable effect;
|
||||
- backend conformance covers both reconstruction and bounded write amplification.
|
||||
|
||||
## 14. Module boundaries
|
||||
|
||||
```text
|
||||
session/** durable storage; imports no runtime module
|
||||
execution/** neutral provider/tool/gate mechanics
|
||||
runtime/types.ts LaneState, Drive, command decisions
|
||||
runtime/progress.ts frame/tool progress channels
|
||||
runtime/drive/*.ts direct procedures
|
||||
runtime/drive.ts total flat switch, created only in M7
|
||||
runtime/lane.ts Lane actor and public surfaces
|
||||
runtime/harness.ts Harness lifecycle and Lane composition
|
||||
```
|
||||
|
||||
Procedure modules import concrete `Lane<TContext>` type-only. `TContext` remains `object | undefined` invariant. No `any`, `Lane<any>`, `as unknown as`, `@ts-expect-error`, inline imports, parameter properties, enums, or other non-erasable TypeScript syntax.
|
||||
|
||||
## 15. Exclusions
|
||||
|
||||
Do not introduce:
|
||||
|
||||
- generic scheduler, graph, action interpreter, or effect-plan DSL — `ResultBoundary` stays a closed three-arm data union switched in one visible place;
|
||||
- per-family outcome unions, `OptionalFinalAssistant`, or embedded result entries in stored records;
|
||||
- result-record listing, filtering, pagination, status queries, or retention machinery;
|
||||
- operation-owned queues, drained-control fields, or terminal queue cleanup;
|
||||
- automatic continuation anywhere below the convenience layer (never inside `drive`);
|
||||
- a second mutation line or transaction framework;
|
||||
- expected-`at` runtime checks for the sole top-level writer;
|
||||
- process-local Drive replacement or caller-owned lifetime;
|
||||
- external finalization;
|
||||
- storage rereads of authoritative control state;
|
||||
- read caches, read budgets, or generic `getValues` batching;
|
||||
- compatibility aliases for any pre-redesign durable shape;
|
||||
- a `phase`/segment discriminator field on structural events — the reducer derives segment-vs-terminal from the open operation's kind;
|
||||
- whole-tree, fork, label-inventory, or repository methods on `AgentLane`; branch-relative reads/appends remain part of the lane facade, while broader administration lives beside it.
|
||||
|
||||
Procedure-specific writes, effect admission, settlement classification, and event construction remain visible at their call sites.
|
||||
|
||||
## 16. Validation and reviews
|
||||
|
||||
After every code stage:
|
||||
|
||||
```bash
|
||||
npm run check
|
||||
```
|
||||
|
||||
Run each modified focused test file from the package root. Do not invoke the full Vitest suite directly. Run `./test.sh` only for final package validation or when explicitly requested.
|
||||
|
||||
Review checkpoints are mandatory at the end of R3, R1b, M7, M10, and final completion. Delegated reviews use provider `anthropic` and model `claude-fable-5`.
|
||||
|
||||
Final greps:
|
||||
|
||||
```bash
|
||||
rg 'lost_ownership|LostOwnership|DriveAbandoned|commandDriveOwned|installerSignal|finalizedOutcome|OperationEnded' \
|
||||
packages/agent/src/harness packages/agent/test/harness
|
||||
rg 'drainedSteer|drainedFollowUp|laneLastResult|RunOutcome|CompactionOutcome|NavigationOutcome|TerminalOperationOutcome|skipInboxOnce|thresholdCheckedTriggerEntryId' \
|
||||
packages/agent/src/harness
|
||||
rg 'SliceNotImplemented' packages/agent/src/harness/runtime
|
||||
rg ': any\b|<any>|as unknown as|@ts-expect-error' packages/agent/src/harness/runtime
|
||||
```
|
||||
|
||||
Final exit conditions:
|
||||
|
||||
- every flat leaf is driveable and reconcilable;
|
||||
- public primitive/convenience behavior is equivalent, including continuation;
|
||||
- every focused test and backend conformance path passes;
|
||||
- public drive exposes no partial graph;
|
||||
- `watchSession` is the only deferred public method;
|
||||
- `drive(id)` answers every settled operation id in the session;
|
||||
- harness.md is self-consistent with the implementation (M9);
|
||||
- provider cache/affinity identity is lane-safe across concurrent and context-reset histories (M10);
|
||||
- independent final review reports no blocker.
|
||||
@@ -0,0 +1,734 @@
|
||||
# WP06 — Session, Branch, Lane separation
|
||||
|
||||
**Status: implemented before WP05 M4.** WP05 M3 remains complete. Retry/deferred work resumes on the composition, mutation, and ownership boundaries landed here.
|
||||
|
||||
This package replaces the mixed `SessionTree`/implicit-main inheritance design with four explicit concepts:
|
||||
|
||||
```text
|
||||
Session global durable data + one mutation line
|
||||
Branch one path through the entry tree, with a movable tip
|
||||
AgentLane Branch data surface + agent operations/configuration
|
||||
AgentHarness manager of AgentLanes; never a lane itself
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 0. Mandatory reading
|
||||
|
||||
Read completely before editing:
|
||||
|
||||
1. `packages/agent/docs/harness.md`.
|
||||
2. `packages/agent/docs/work-packages/05-direct-durable-drive.md`.
|
||||
3. `packages/agent/src/harness/session/types.ts`.
|
||||
4. `packages/agent/src/harness/session/session.ts`.
|
||||
5. `packages/agent/src/harness/session/memory.ts`.
|
||||
6. `packages/agent/src/harness/session/jsonl/repo.ts` and `jsonl/storage.ts`.
|
||||
7. `packages/session-backends/sqlite-node/src/sqlite/session.ts`, `storage.ts`, and repo implementation.
|
||||
8. `packages/agent/src/harness/runtime/lane.ts`, `harness.ts`, `restore.ts`, and `types.ts`.
|
||||
9. `packages/agent/src/harness/agent-harness.ts`.
|
||||
10. `packages/agent/src/harness/session/fork.ts` and repository conformance tests.
|
||||
11. Every test named in §8 before modifying it.
|
||||
|
||||
Do not inspect deleted runtime implementations or Git history. Current source, `harness.md`, WP05, and this package are the only sources of truth.
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem
|
||||
|
||||
### 1.1 `SessionTree` mixes unrelated ownership
|
||||
|
||||
`SessionTree` currently contains:
|
||||
|
||||
```ts
|
||||
interface SessionTree {
|
||||
// Lane/path data.
|
||||
getLeafId(...): Promise<string | null>;
|
||||
findEntriesOnBranch(...): Promise<Entry[]>;
|
||||
findEntryOnBranch(...): Promise<Entry | undefined>;
|
||||
appendMessage(...): Promise<string>;
|
||||
appendCustomEntry(...): Promise<string>;
|
||||
|
||||
// Session-global data that is not lane- or branch-owned.
|
||||
getEntry(...): Promise<Entry | undefined>;
|
||||
getStats(...): Promise<SessionStats>;
|
||||
findEntries(...): Promise<Entry[]>;
|
||||
findEntry(...): Promise<Entry | undefined>;
|
||||
getValue(...): Promise<StoredValue<unknown> | undefined>;
|
||||
setValue(...): Promise<void>;
|
||||
readList(...): Promise<ListElement<unknown>[]>;
|
||||
appendList(...): Promise<void>;
|
||||
getName(...): Promise<string | undefined>;
|
||||
setName(...): Promise<void>;
|
||||
getLabel(...): Promise<string | undefined>;
|
||||
setLabel(...): Promise<void>;
|
||||
}
|
||||
```
|
||||
|
||||
A selected tree view changes only which mutation queue a global write enters. It does not change the durable address. Two views can therefore read the same global value under different lane lines and commit a lost update.
|
||||
|
||||
### 1.2 `Session` silently means `main`
|
||||
|
||||
`Session extends SessionTree`. Its inherited branch methods and high-level writes silently delegate to the `main` lane. `session.setValue(...)` is actually `setValueForLane("main", ...)`. A fresh repository session creates a partial implicit main lane before any harness exists.
|
||||
|
||||
### 1.3 `AgentHarness` silently means `main`
|
||||
|
||||
`AgentHarness extends AgentLane`, and the runtime `Harness extends Lane`. The manager object is inserted into its own lane map as `main`. Calls such as `harness.prompt(...)`, `harness.watch(...)`, or `harness.getModel(...)` silently target main.
|
||||
|
||||
### 1.4 Per-lane mutation queues solve the wrong problem
|
||||
|
||||
Storage already serializes atomic commits and assigns one session-wide `seq` per write. Per-lane mutation queues allow useful preparation overlap, but this runtime does not need that complexity now: all harness mutation callbacks perform bounded storage reads, prepare one write set, commit at most once, publish process-local state, and return. Providers, tools, hooks, timers, and asynchronous event delivery are outside mutation callbacks.
|
||||
|
||||
The approved first implementation uses one Session mutation line. Keyed lines may be added later if profiling proves the global line is a bottleneck; that future change requires a mutable-ownership audit but no public API redesign.
|
||||
|
||||
---
|
||||
|
||||
## 2. Approved terminology and ownership
|
||||
|
||||
### Session
|
||||
|
||||
Owns:
|
||||
|
||||
- session metadata;
|
||||
- global entry and usage queries;
|
||||
- application values and lists;
|
||||
- session name and entry labels;
|
||||
- Branch discovery/creation;
|
||||
- one process-local mutation line for every supported mutation;
|
||||
- one open storage/backend lifecycle.
|
||||
|
||||
A Session does not implement Branch and has no implicit branch.
|
||||
|
||||
### Branch
|
||||
|
||||
Data-only capability describing one named path through the immutable entry tree.
|
||||
|
||||
Owns only:
|
||||
|
||||
- its current tip;
|
||||
- branch-relative entry queries;
|
||||
- direct extension of its tip with message or custom entries.
|
||||
|
||||
A Branch has no model configuration, queues, operation state, drive, hooks, or agent policy.
|
||||
|
||||
### AgentLane
|
||||
|
||||
One Branch plus agent configuration and operations. It exposes Branch methods directly rather than exposing a nested `Branch`, tree, store, view, or access object.
|
||||
|
||||
When idle, `AgentLane.appendMessage` / `appendCustomEntry` extend the tip directly. During an active run, they retain the existing deferred-write semantics: reserve the entry id, persist `pendingEntry(id)`, and enqueue the id in the operation inbox for checkpoint placement. A raw Branch append is always a direct data append; raw Branch mutation while a Harness owns the corresponding lane is a trusted-programming defect.
|
||||
|
||||
### AgentHarness
|
||||
|
||||
Owns global registries/configuration, hooks, events, lifecycle, and a map of AgentLanes. It is not an AgentLane and exposes no implicit-main operation methods.
|
||||
|
||||
---
|
||||
|
||||
## 3. Target public types
|
||||
|
||||
### 3.1 Session reader and mutation
|
||||
|
||||
```ts
|
||||
export interface SessionReader {
|
||||
getEntries(ids: string[], context: Context): Promise<Map<string, Entry>>;
|
||||
getValue<T>(
|
||||
address: Value<T>,
|
||||
context: Context,
|
||||
): Promise<StoredValue<T> | undefined>;
|
||||
scanValues<T>(prefix: Value<T>, context: Context): Promise<StoredValue<T>[]>;
|
||||
readList<T>(
|
||||
address: ValueList<T>,
|
||||
options: ListReadOptions | undefined,
|
||||
context: Context,
|
||||
): Promise<ListElement<T>[]>;
|
||||
scanBranch(query: StorageBranchScan, context: Context): Promise<Entry[]>;
|
||||
}
|
||||
|
||||
export interface SessionMutation extends SessionReader {
|
||||
/** Exactly zero or one attempt. A second call rejects, including after failure. */
|
||||
commit(writes: Write[], context: Context): Promise<CommitResult>;
|
||||
/** Wait for any admitted commit, invalidate the capability, and release the Session line. */
|
||||
end(context: Context): Promise<void>;
|
||||
}
|
||||
|
||||
export type SessionMutator = Omit<SessionMutation, "end">;
|
||||
|
||||
export type SessionMutationCallback<TResult> = (
|
||||
mutator: SessionMutator,
|
||||
context: Context,
|
||||
) => TResult | Promise<TResult>;
|
||||
```
|
||||
|
||||
`beginMutation()`/`SessionMutation.end()` remain the transportable scope used by the current remote Session protocol. They are keyless and carry no lane field. Normal local/harness code uses `mutate()`; no caller selects a mutation key.
|
||||
|
||||
### 3.2 Branch
|
||||
|
||||
```ts
|
||||
export interface Branch {
|
||||
readonly name: string;
|
||||
getTipId(context: Context): Promise<string | null>;
|
||||
findEntries(
|
||||
query: BranchScan | undefined,
|
||||
context: Context,
|
||||
): Promise<Entry[]>;
|
||||
findEntry(
|
||||
query: BranchScan | undefined,
|
||||
context: Context,
|
||||
): Promise<Entry | undefined>;
|
||||
appendMessage(message: AgentMessage, context: Context): Promise<string>;
|
||||
appendCustomEntry(
|
||||
customType: string,
|
||||
data: JsonValue | undefined,
|
||||
context: Context,
|
||||
): Promise<string>;
|
||||
}
|
||||
```
|
||||
|
||||
Because the receiver is already a Branch, the public names are `findEntries` and `findEntry`, not `findEntriesOnBranch` and `findEntryOnBranch`.
|
||||
|
||||
### 3.3 Session
|
||||
|
||||
```ts
|
||||
export interface Session<
|
||||
TMetadata extends SessionMetadata = SessionMetadata,
|
||||
> extends SessionReader {
|
||||
readonly metadata: TMetadata;
|
||||
readonly idGenerator: IdGenerator;
|
||||
|
||||
// Direct reads. No mutation-line acquisition.
|
||||
getEntry(id: string, context: Context): Promise<Entry | undefined>;
|
||||
getStats(context: Context): Promise<SessionStats>;
|
||||
findEntries(
|
||||
query: EntryQuery | undefined,
|
||||
context: Context,
|
||||
): Promise<Entry[]>;
|
||||
findEntry(
|
||||
query: EntryQuery | undefined,
|
||||
context: Context,
|
||||
): Promise<Entry | undefined>;
|
||||
getName(context: Context): Promise<string | undefined>;
|
||||
getLabel(targetId: string, context: Context): Promise<string | undefined>;
|
||||
|
||||
// Existing Branch acquisition performs durable I/O and therefore receives Context.
|
||||
branch(name: string, context: Context): Promise<Branch | undefined>;
|
||||
createBranch(
|
||||
name: string,
|
||||
at: string | null,
|
||||
context: Context,
|
||||
): Promise<Branch>;
|
||||
|
||||
// Transportable explicit scope; RemoteSession maps begin/read/commit/end over RPC.
|
||||
beginMutation(context: Context): Promise<SessionMutation>;
|
||||
|
||||
// Trusted sharp edge. The callback holds the sole Session mutation line.
|
||||
mutate<TResult>(
|
||||
mutation: SessionMutationCallback<TResult>,
|
||||
context: Context,
|
||||
): Promise<TResult>;
|
||||
|
||||
// One-write conveniences implemented through mutate().
|
||||
setValue<T>(
|
||||
address: Value<T>,
|
||||
next: NoInfer<T>,
|
||||
context: Context,
|
||||
): Promise<void>;
|
||||
deleteValue<T>(address: Value<T>, context: Context): Promise<void>;
|
||||
appendList<T>(
|
||||
address: ValueList<T>,
|
||||
element: NoInfer<T>,
|
||||
context: Context,
|
||||
): Promise<void>;
|
||||
deleteList<T>(address: ValueList<T>, context: Context): Promise<void>;
|
||||
setName(name: string | undefined, context: Context): Promise<void>;
|
||||
setLabel(
|
||||
targetId: string,
|
||||
label: string | undefined,
|
||||
context: Context,
|
||||
): Promise<void>;
|
||||
|
||||
close(context: Context): Promise<void>;
|
||||
}
|
||||
```
|
||||
|
||||
`mutate()` remains public. Plugins are trusted not to retain the mutator, invoke nested public writers, perform effects, or hold the line across unbounded work. Misuse may block every mutation in that Session and is a plugin defect. `beginMutation()` exists for transport/lifecycle integration rather than ordinary plugin work; every direct caller must call `end()` in `finally`.
|
||||
|
||||
### 3.4 AgentLane
|
||||
|
||||
`AgentLane` keeps its operation/configuration/observation methods and directly adds the Branch surface:
|
||||
|
||||
```ts
|
||||
export interface AgentLane {
|
||||
readonly name: string;
|
||||
|
||||
getTipId(context: Context): Promise<string | null>;
|
||||
findEntries(
|
||||
query: BranchScan | undefined,
|
||||
context: Context,
|
||||
): Promise<Entry[]>;
|
||||
findEntry(
|
||||
query: BranchScan | undefined,
|
||||
context: Context,
|
||||
): Promise<Entry | undefined>;
|
||||
appendMessage(message: AgentMessage, context: Context): Promise<string>;
|
||||
appendCustomEntry(
|
||||
customType: string,
|
||||
data: JsonValue | undefined,
|
||||
context: Context,
|
||||
): Promise<string>;
|
||||
|
||||
getLastResult(context: Context): Promise<LaneLastResult | undefined>;
|
||||
accept(
|
||||
request: OperationRequest,
|
||||
context: Context,
|
||||
): Promise<OperationAdmissionResult>;
|
||||
drive(options: DriveOptions, context: Context): Promise<DriveResult>;
|
||||
requestAbort(
|
||||
operationId: string,
|
||||
context: Context,
|
||||
): Promise<AbortRequestResult>;
|
||||
inspectExecution(context: Context): Promise<LaneExecutionInfo>;
|
||||
// Existing convenience, queue, configuration, idle, and watch methods remain.
|
||||
}
|
||||
```
|
||||
|
||||
Delete `AgentLane.sessionTree`.
|
||||
|
||||
### 3.5 AgentHarness
|
||||
|
||||
```ts
|
||||
export interface AcquireLaneOptions {
|
||||
/** Used only when the AgentLane does not exist. Defaults to null. */
|
||||
createAt?: string | null;
|
||||
}
|
||||
|
||||
export interface AgentHarness<
|
||||
TContext extends object | undefined = object | undefined,
|
||||
> {
|
||||
lane(name: string, context: Context): Promise<AgentLane>;
|
||||
lane(
|
||||
name: string,
|
||||
options: AcquireLaneOptions,
|
||||
context: Context,
|
||||
): Promise<AgentLane>;
|
||||
lanes(context: Context): Promise<LaneInfo[]>;
|
||||
|
||||
// Session-global metadata wrappers preserve existing value_update events.
|
||||
getName(context: Context): Promise<string | undefined>;
|
||||
setName(name: string | undefined, context: Context): Promise<void>;
|
||||
getLabel(targetId: string, context: Context): Promise<string | undefined>;
|
||||
setLabel(
|
||||
targetId: string,
|
||||
label: string | undefined,
|
||||
context: Context,
|
||||
): Promise<void>;
|
||||
|
||||
// Existing global tools/resources/options/settings/hooks/events/watchSession/close surface.
|
||||
}
|
||||
```
|
||||
|
||||
`AgentHarness` does not extend `AgentLane`. Delete `createLane`; `lane()` is atomic get-or-create. Existing lanes ignore `createAt`. A missing lane uses `createAt ?? null`. Concurrent acquisitions return the same published AgentLane. Invalid names and unknown non-null targets reject with the existing tagged errors; close/fault reject with their existing lifecycle errors.
|
||||
|
||||
A fresh Session and fresh Harness contain no implicit main Branch or AgentLane. `await harness.lane("main", context)` creates main completely. `lanes()` may return `[]`.
|
||||
|
||||
---
|
||||
|
||||
## 4. Mutation and read semantics
|
||||
|
||||
### 4.1 One Session mutation line
|
||||
|
||||
Replace `LaneMutationLine` with a single `MutationLine`:
|
||||
|
||||
```ts
|
||||
export class MutationLine {
|
||||
private tail: Promise<void> = Promise.resolve();
|
||||
private sealedError: Error | undefined;
|
||||
|
||||
run<TResult>(operation: () => TResult | Promise<TResult>): Promise<TResult>;
|
||||
seal(error: Error): Promise<void>;
|
||||
}
|
||||
```
|
||||
|
||||
`StorageBackedSession.beginMutation()` acquires that line and returns one explicit keyless capability; only `end()` releases it. `mutate()` is the callback convenience built from begin/end and always ends in `finally`. The callback may read, prepare, commit once, publish process-local state, synchronously bind event recipients, and return. `close()` seals admission and waits for every acquired scope to end before closing Storage.
|
||||
|
||||
High-level Session writes, Branch creation/appends, every `Lane.command`, progress writes, restore snapshots requiring coherence, and Harness lane acquisition all call the same `Session.mutate()`.
|
||||
|
||||
### 4.2 Reads bypass the line
|
||||
|
||||
Ordinary Session and Branch reads call Storage directly. They observe the latest fully applied atomic storage commit at the time each read executes:
|
||||
|
||||
- a queued/planning/unapplied mutation is invisible;
|
||||
- no partial commit is visible;
|
||||
- once Storage commit resolves, direct reads may observe it even while the mutation callback is still publishing process-local state;
|
||||
- multiple reads outside one `mutate()` are not a snapshot;
|
||||
- coherent read-decide-write/CAS uses `Session.mutate()`.
|
||||
|
||||
### 4.3 Effects stay outside
|
||||
|
||||
A mutation callback must not perform or await:
|
||||
|
||||
- providers or deferred fetch/cancel;
|
||||
- tools;
|
||||
- hooks;
|
||||
- timers;
|
||||
- asynchronous event delivery;
|
||||
- idle callbacks or Drive completion;
|
||||
- nested Session/Branch/AgentLane mutators.
|
||||
|
||||
The callback may synchronously call `emitBatch` after publication to bind recipients and retain its delivery promise. The public operation awaits delivery after `mutate()` returns.
|
||||
|
||||
### 4.4 Storage remains independently atomic
|
||||
|
||||
Storage retains its one-session commit serializer and assigns one global `seq` per write. The Session mutation line protects read-decide-commit-publication procedures; the Storage queue protects atomic transaction application, sequence assignment, fork snapshots, and backend callers. Do not merge the two abstractions.
|
||||
|
||||
---
|
||||
|
||||
## 5. Branch and lane durable shape
|
||||
|
||||
### 5.1 No implicit main
|
||||
|
||||
Repository `create()` writes only session metadata/header/catalog state. It writes no branch tip, lane configuration, or lane state. Remove main seeding from Memory and JSONL creation and from SQLite initialization.
|
||||
|
||||
Legacy coding-agent v3 normalization may still produce a main Branch because the imported transcript has one selected path.
|
||||
|
||||
### 5.2 Branch completeness
|
||||
|
||||
A Branch exists exactly when its required tip value exists. `createBranch(name, at, context)` validates name, absence, and non-null target, then writes the tip in one mutation. It writes no model configuration or operation state.
|
||||
|
||||
Use Branch terminology in source. This package renames the typed constructor and public concepts to `branchTip`/`tipId`. The persisted namespace and durable field spelling decision must be consistent across all backends and docs:
|
||||
|
||||
- use `pi.branch.tip` and rename format-4 fields from leaf to tip rather than retain misleading new-code aliases;
|
||||
- format 4 and the new harness are WIP, so replace their schema and field names in place: no storage-version bump, migration, compatibility decoder, or old-format rejection path;
|
||||
- legacy coding-agent v3 import remains supported: it maps its selected main leaf to a main Branch tip and walks that selected physical ancestry to reconstruct the nearest `model_change`, `thinking_level_change`, and `active_tools_change` independently; unsupported nearest values do not fall back to older history;
|
||||
- when the importer can reconstruct a total configuration, it writes ordinary `laneConfig("main")` plus fresh idle `laneState("main")` before returning the Session; missing active-tools history normalizes to `[]` because v3 did not persist the initial tool inventory;
|
||||
- if required model or thinking configuration is missing or unsupported, the importer leaves a data-only main Branch rather than persisting partial compatibility state;
|
||||
- update the legacy active-tools record type to read its encoded `activeToolNames` array;
|
||||
- update protocol schemas and experimental adapters for public `tipId` fields in the same package.
|
||||
|
||||
Inventory uses `branchTipInventoryPrefix()`. Legacy import emits only ordinary Branch/Lane values; there is no temporary compatibility address or attachment-time migration.
|
||||
|
||||
### 5.3 AgentLane completeness
|
||||
|
||||
An AgentLane adds total `laneConfig`, `laneState`, optional `laneLastResult`, and optional current operation values to an existing Branch.
|
||||
|
||||
`harness.lane(name, options?, context)` executes one Session mutation and handles exactly these cases:
|
||||
|
||||
| Durable state | Result |
|
||||
| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Branch absent; lane values absent | validate `createAt`, commit Branch tip + the immutable `AgentHarnessOptions` seed config + idle lane state, publish one AgentLane and `lane_created { at: createAt ?? null }` |
|
||||
| Branch present; lane config/state absent and no last result | commit the immutable seed config + idle lane state at the existing tip, publish one AgentLane and `lane_created { at: existingTip }` |
|
||||
| Branch and complete lane values present | return the restored/published AgentLane; no commit/event |
|
||||
| Any partial or contradictory combination | fault as storage corruption |
|
||||
|
||||
The committing callback publishes the new Branch/AgentLane into process-local maps and synchronously calls `emitBatch(lane_created, context)` before returning from `Session.mutate()`. Event delivery is awaited after line release.
|
||||
|
||||
`AgentHarness.create()` restores every complete durable AgentLane and open operation, but creates nothing and requires no main. A data-only Branch remains a Branch until `harness.lane(name, ...)` attaches agent state.
|
||||
|
||||
### 5.4 Append behavior
|
||||
|
||||
`Branch.appendMessage` / `appendCustomEntry` always commit an immutable entry parented to the current tip and move the tip in the same transaction.
|
||||
|
||||
`AgentLane.appendMessage` / `appendCustomEntry` retain harness semantics:
|
||||
|
||||
- idle: append and move tip immediately;
|
||||
- active run: stage `pendingEntry(id)` and enqueue `inbox.writes`;
|
||||
- active structural operation: retain the existing wait/re-evaluate contract when that surface lands;
|
||||
- pending assistant messages reject before commit.
|
||||
|
||||
Both return the entry id reserved before their mutation.
|
||||
|
||||
---
|
||||
|
||||
## 6. Runtime composition
|
||||
|
||||
### 6.1 Harness
|
||||
|
||||
Replace inheritance with composition:
|
||||
|
||||
```ts
|
||||
export class Harness<
|
||||
TContext extends object | undefined,
|
||||
> implements AgentHarness<TContext> {
|
||||
readonly session: Session;
|
||||
readonly models: Models;
|
||||
readonly hooks: HookRegistry;
|
||||
readonly events: HarnessEventBus;
|
||||
readonly lanesByName = new Map<string, Lane<TContext>>();
|
||||
// global config/lifecycle fields
|
||||
}
|
||||
```
|
||||
|
||||
The constructor builds every restored Lane as an ordinary object. It never calls `super("main", ...)` and never inserts `this` into `lanesByName`.
|
||||
|
||||
Fault and close iterate ordinary Lane objects. Global getters/setters live only on Harness. Coding-agent experimental services and workers must acquire/cache `main` explicitly before invoking lane operations.
|
||||
|
||||
### 6.2 Lane
|
||||
|
||||
`Lane.command` and `commandDriveOwned` call keyless `session.mutate(plan, context)`. The live `Lane.state` remains the authoritative process projection. Exact Drive fencing remains adjacent to commit admission.
|
||||
|
||||
Lane directly implements Branch query/append names. It may hold a package-private Branch implementation for direct data reads, but no nested Branch is exposed publicly.
|
||||
|
||||
### 6.3 Restore
|
||||
|
||||
Restore no longer enters a named mutation line. `AgentHarness.create()` owns the Session attachment interval and performs one bounded keyless `Session.mutate()` callback to inventory/restore every complete AgentLane before publishing the Harness; it commits only when an intentional attachment normalization is required. Coherent live watch/inspection likewise uses `Session.mutate()` as a no-commit callback.
|
||||
|
||||
Restore inventories complete AgentLanes separately from data-only Branches. Missing main is legal.
|
||||
|
||||
---
|
||||
|
||||
## 7. Repository, backend, and fork requirements
|
||||
|
||||
### Memory/JSONL/SQLite
|
||||
|
||||
All Session facades:
|
||||
|
||||
- remove lane arguments and lane fields from begin/mutate capabilities;
|
||||
- retain explicit `beginMutation(context)` / `SessionMutation.end(context)` forwarding for local lifecycle and remote transport;
|
||||
- keep an explicit scope admitted until `end()` and keep callback `mutate()` admitted through its implicit finally/end;
|
||||
- expose Branch acquisition/creation;
|
||||
- keep direct reads outside the line;
|
||||
- keep Storage commit sequencing unchanged.
|
||||
|
||||
SQLite and future SQL backends still serialize sequence-allocating commits in Storage. The Session mutation line intentionally serializes complete callbacks only within one open Session owner; separate sessions remain concurrent.
|
||||
|
||||
### Forks
|
||||
|
||||
Preserve one coherent source Storage snapshot. Update terms from lane leaf to Branch tip.
|
||||
|
||||
- branch-scope fork creates destination Branch `main` at the copied path;
|
||||
- if no explicit source entry is provided, source `main` must exist or fork rejects;
|
||||
- branch scope copies source main's configuration and writes idle lane state **together iff** source main is a complete configured AgentLane; an unconfigured/data-only source main produces only a data-only destination main Branch;
|
||||
- tree scope copies every Branch tip; each complete configured source AgentLane copies its configuration plus fresh idle lane state together, while each data-only Branch remains data-only;
|
||||
- operation values, pending values/lists, last results, and usage rows remain excluded;
|
||||
- destination sessions do not gain an unrelated implicit main;
|
||||
- retain the explicit begin/commit/end fork-ordering seam: code that must admit a commit before starting a fork snapshot calls keyless `beginMutation()`, invokes `commit()`, starts the repository snapshot only after commit admission, and calls `end()` in `finally`; Storage queues the source snapshot against commits to choose one coherent boundary.
|
||||
|
||||
---
|
||||
|
||||
## 8. Implementation phases and file manifest
|
||||
|
||||
Public drive remains disabled. Complete this package before resuming WP05 M4.
|
||||
|
||||
### Phase A — Session mutation and Branch
|
||||
|
||||
**Rename**
|
||||
|
||||
- `src/harness/session/lane-mutations.ts` → `mutation-line.ts`.
|
||||
|
||||
**Modify**
|
||||
|
||||
- `src/harness/session/types.ts` — `Branch`, keyless `Session.mutate` and keyless begin/end transport scope, no `SessionTree`, Session-global methods.
|
||||
- `src/harness/session/session.ts` — one line, Branch implementation, no implicit-main delegates, direct Branch append; rename lane-creation validation errors to Branch terminology.
|
||||
- Memory/JSONL/SQLite format-4 schema and codecs — replace WIP leaf/lane spellings in place; do not add a version gate or migration.
|
||||
- `src/harness/session/memory.ts`.
|
||||
- `src/harness/session/jsonl/repo.ts`, `jsonl/legacy-v3.ts`, and related open/create facade files.
|
||||
- `packages/session-backends/sqlite-node/src/sqlite/session.ts` and repo creation.
|
||||
- `src/harness/session/fork.ts`.
|
||||
- `src/harness/session/index.ts` and package exports.
|
||||
- storage/repository benchmarks and conformance call sites.
|
||||
|
||||
**Rename tests**
|
||||
|
||||
- `test/harness/session-tree.test.ts` → `branch.test.ts`.
|
||||
- `test/harness/session-create-lane.test.ts` → `session-create-branch.test.ts`.
|
||||
|
||||
**Modify tests**
|
||||
|
||||
- `test/harness/storage-backed-session.test.ts`.
|
||||
- `test/harness/memory-session-repo.test.ts`.
|
||||
- `test/harness/jsonl-session-repo.test.ts`.
|
||||
- `test/harness/memory-conformance.test.ts`.
|
||||
- `src/harness/session/testing/conformance/session-repo.ts`.
|
||||
- SQLite repo/storage tests.
|
||||
- compaction/branch-summarization type fixtures.
|
||||
|
||||
### Phase B — Harness/Lane composition
|
||||
|
||||
**Modify**
|
||||
|
||||
- `src/harness/agent-harness.ts`.
|
||||
- `src/harness/runtime/harness.ts`.
|
||||
- `src/harness/runtime/lane.ts`.
|
||||
- `src/harness/runtime/restore.ts`.
|
||||
- `src/harness/runtime/types.ts` where `leafId` becomes `tipId`.
|
||||
- `src/harness/session/values.ts` and durable state types for Branch tip naming.
|
||||
- all runtime drive modules already present (`checkpoint`, `generation`, `recovery`, `terminal`, `progress`) only where names/signatures change.
|
||||
- `src/harness/compaction/branch-summarization.ts`.
|
||||
- `packages/protocol/src/harness.ts`.
|
||||
- `packages/coding-agent/src/experimental/services/agent-controller-provider.ts`.
|
||||
- `packages/coding-agent/src/experimental/services/models-provider.ts`.
|
||||
- `packages/coding-agent/src/experimental/session-worker.ts`.
|
||||
- experimental harness wire/session worker tests.
|
||||
|
||||
**Modify focused tests**
|
||||
|
||||
- every `test/harness/runtime/*.test.ts` helper/call site using keyed mutate, implicit Harness-as-main, `sessionTree`, `leafId`, or `createLane`;
|
||||
- `test/harness/types.test.ts`;
|
||||
- `test/harness/branch-summarization.test.ts`;
|
||||
- protocol and coding-agent experimental tests.
|
||||
|
||||
Phases A and B are one atomic landing. Removing `SessionTree`, keyed mutate, and Harness inheritance cannot compile as separately committed compatibility phases, and this package intentionally adds no temporary aliases.
|
||||
|
||||
### Phase C — Normative documentation
|
||||
|
||||
Update `packages/agent/docs/harness.md` completely and consistently:
|
||||
|
||||
- orientation/system model and worked examples;
|
||||
- bound Branch tip addresses;
|
||||
- Branch, Session metadata, queries, forks, and repository boundary;
|
||||
- operation metadata/result/snapshot `tipId` terminology;
|
||||
- one Session mutation line in Parts 3–5;
|
||||
- attachment with no required main;
|
||||
- public Branch, AgentLane, AgentHarness, and Session surfaces;
|
||||
- event/watcher ordering under the Session line;
|
||||
- remove the old second harness-settings-line lock-order narrative: harness-global settings and every durable lane mutation now serialize through the sole Session line when a coherent snapshot is required; pure synchronous registry reads remain direct;
|
||||
- work-package table, invariants, races, backend conformance, and glossary.
|
||||
|
||||
Update WP05 before M4:
|
||||
|
||||
- replace `SessionTree`/lane-line/inheritance assumptions;
|
||||
- replace `leafId` source examples where the new type names require it;
|
||||
- keep all M0–M3 historical behavior and M4–M8 durable requirements;
|
||||
- state that WP06 is the foundation between M3 and M4.
|
||||
|
||||
Update every current supporting document whose public names or ordering statements change:
|
||||
|
||||
- `docs/assistant-durability.md` and `docs/tool-durability.md` — Session-line FIFO and `branchTip` terminology;
|
||||
- `docs/values.md` — Session-global value/list surface, Branch tip addresses, and no `SessionTree`;
|
||||
- `docs/telemetry.md` — receiver inventory and Session mutation spans;
|
||||
- `docs/plugins.md` — remove `sessionTree`; the plugin's AgentLane directly supplies Branch methods, while a scoped Session-data facet supplies global value/list/name/label/query methods and excludes raw `mutate`, `idGenerator`, close, and backend authority;
|
||||
- `docs/extensions/pi-extensions-v2.md` and `docs/extensions/pi-server-artifact/index.md` where examples/types use the changed surfaces;
|
||||
- completed WP00–WP04 only where a forward-looking/current-state statement would otherwise claim the removed API still exists.
|
||||
|
||||
Retain the current remote Session mutation contract and update it from a named lane line to the sole Session line: worker `RemoteSession.mutate()` performs keyless begin RPC → local callback with remote reads/one remote commit → local post-commit publication → end RPC. The server holds the Session line through commit and publication until end acknowledgment. Disconnect/timeout terminates the scope under the existing hosting policy. Update the current remote protocol/vertical-slice documentation and every implementation/test present on dev; do not delete or defer this behavior.
|
||||
|
||||
Do not rewrite released changelog sections. Add no changelog entry on a non-main/non-PR development branch.
|
||||
|
||||
---
|
||||
|
||||
## 9. Required tests
|
||||
|
||||
### Session mutation line
|
||||
|
||||
- concurrent `mutate()` callbacks serialize globally, including callbacks invoked from different AgentLanes;
|
||||
- a second callback does not enter until the first callback returns after commit/publication;
|
||||
- direct reads do not wait for a callback that has not committed;
|
||||
- direct reads observe a fully landed commit even while that mutation callback remains open after commit;
|
||||
- no direct read observes a partial multi-write commit;
|
||||
- two read-modify-write counter mutations produce `1`, then `2`;
|
||||
- separate direct `getValue` + `setValue` calls remain deliberately non-atomic;
|
||||
- zero-commit callback is legal;
|
||||
- keyless `beginMutation()` excludes every other mutation until `end()`, commit does not release it, end-without-commit is legal, repeated end is idempotent, and close waits for end;
|
||||
- RemoteSession begin/read/commit/publication/end preserves that same scope;
|
||||
- second commit attempt rejects, including after a failed first attempt;
|
||||
- nested public write from a mutation callback is documented as invalid and deterministically blocks/rejects according to the chosen guard;
|
||||
- close-first rejects mutation; mutation-first completes and close waits; a trusted callback that never returns can block close;
|
||||
- Storage commit `seq` and stats behavior is unchanged.
|
||||
|
||||
### Session and Branch
|
||||
|
||||
- fresh Session has zero Branches and no main tip value;
|
||||
- no storage-version bump, migration, compatibility decoder, or rejection path is added for the replaced WIP format-4 schema;
|
||||
- legacy coding-agent v3 import reconstructs ordinary total main-lane configuration plus idle state before returning when valid model/thinking history exists; otherwise it returns a data-only main Branch;
|
||||
- legacy config tests cover complete and incomplete histories, invalid legacy fields, and branched histories where only changes on the selected main path apply;
|
||||
- `branch(name, context)` returns undefined for absence and receives the exact Context on reads;
|
||||
- `createBranch` validates name, target, and duplicate atomically;
|
||||
- two concurrent creates have one winner;
|
||||
- Branch queries default to its tip and preserve scan/filter/cursor behavior;
|
||||
- direct Branch message/custom append extends its tip atomically;
|
||||
- custom entry with absent data remains valid;
|
||||
- pending assistant append rejects;
|
||||
- Session global values/lists/name/labels/global queries no longer depend on a branch;
|
||||
- all pre-close Branch objects reject after close;
|
||||
- branch-scope fork copies config + idle state together iff source main is configured; tree scope does the same independently per configured lane; data-only Branches remain data-only;
|
||||
- explicit keyless begin/commit/end preserves commit-before-fork-snapshot ordering and captures one Storage-serialized snapshot boundary.
|
||||
|
||||
### AgentHarness and AgentLane
|
||||
|
||||
- Harness has no AgentLane methods at type or runtime;
|
||||
- Session has no Branch methods at type or runtime;
|
||||
- AgentLane has Branch methods directly and no `sessionTree`/nested Branch property;
|
||||
- fresh Harness `lanes()` is empty;
|
||||
- `lane("main")` creates complete main atomically;
|
||||
- missing named lane defaults to null tip; `createAt` anchors creation;
|
||||
- existing data-only Branch becomes one complete AgentLane without moving its tip;
|
||||
- existing complete AgentLane acquisition commits nothing and emits no creation event;
|
||||
- concurrent acquisitions return the same object and emit exactly one `lane_created`;
|
||||
- creation commit publishes the Lane and binds recipients before releasing `Session.mutate`;
|
||||
- invalid name/unknown anchor commit nothing;
|
||||
- restored complete lanes/open operations are inventoried without creating main;
|
||||
- partial Branch/config/lane-state combinations fault;
|
||||
- Harness global metadata wrappers retain commit → publication → `value_update` delivery;
|
||||
- AgentLane idle append moves the tip; active-run append stages one deferred write and preserves authoritative state;
|
||||
- close/fault seal every ordinary Lane without relying on Harness inheritance.
|
||||
|
||||
### Regression
|
||||
|
||||
- M2 exact-Drive ABA fence still checks immediately before commit admission;
|
||||
- M3 generation intent/frame/settlement writes are byte-identical apart from renamed durable/public fields;
|
||||
- frame FIFO remains correct under the one Session line;
|
||||
- watch has only snapshot-first or publication-first outcomes;
|
||||
- usage totals remain commit-boundary exact across lanes;
|
||||
- Context remains trailing and source-identical throughout;
|
||||
- Memory, JSONL, and SQLite repository/storage conformance pass.
|
||||
|
||||
---
|
||||
|
||||
## 10. Exclusions
|
||||
|
||||
Do not add:
|
||||
|
||||
- keyed mutation lines, arbitrary lock names, resource locks, multi-lock ordering, versions, or optimistic retries;
|
||||
- a scheduler, transaction framework, action interpreter, or generic post-commit task system;
|
||||
- a nested Branch/tree/store/access property on AgentLane;
|
||||
- compatibility aliases for `SessionTree`, `view`, implicit Session main methods, Harness lane methods, or `createLane`;
|
||||
- keyed/named begin/end RPC scopes or removal of the current keyless RemoteSession mutation transport;
|
||||
- effects inside `Session.mutate()`;
|
||||
- a second Storage commit/sequence mechanism;
|
||||
- automatic work start from `AgentHarness.create()` or `harness.lane()`;
|
||||
- public drive before WP05 M8.
|
||||
|
||||
Do not modify provider/tool behavior, durable execution phases, retry/deferred policy, or assistant-frame semantics beyond signature/name propagation required by this package.
|
||||
|
||||
---
|
||||
|
||||
## 11. Validation
|
||||
|
||||
Run every modified focused test, then:
|
||||
|
||||
```bash
|
||||
git diff --check
|
||||
npm run check
|
||||
./test.sh
|
||||
|
||||
rg -n "SessionTree|sessionTree|\.view\(|LaneMutationLine|extends AgentLane|extends Lane" \
|
||||
packages/agent/src packages/agent/test packages/agent/docs \
|
||||
packages/session-backends packages/protocol packages/coding-agent/src/experimental \
|
||||
packages/coding-agent/test/experimental*
|
||||
|
||||
rg -n "beginMutation\([^)]*,|mutate\([^)]*,[^)]*," \
|
||||
packages/agent/src packages/agent/test packages/session-backends packages/coding-agent/src/experimental
|
||||
|
||||
rg -n "session\.mutate\([^)]*\"|\.mutate\(\s*[a-zA-Z_$][a-zA-Z0-9_$]*\s*," \
|
||||
packages/agent/src packages/agent/test packages/session-backends
|
||||
```
|
||||
|
||||
Expected first grep matches only this package's problem statement, explicitly historical work-package API descriptions, negative type assertions, and unrelated coding-agent tree-widget names such as `SessionTreeNode`; no current harness `SessionTree` concept remains. The keyed-mutate grep must be empty after manually checking false positives.
|
||||
|
||||
Review the complete implementation and normative doc update with Fable before committing. Do not commit without explicit user approval.
|
||||
|
||||
---
|
||||
|
||||
## 12. Stop condition
|
||||
|
||||
WP06 is complete when:
|
||||
|
||||
- Session has one keyless mutation line; callback `mutate()` and explicit begin/commit/end remote transport share it, and neither accepts a lane key;
|
||||
- direct reads bypass that line and expose only fully applied Storage commits;
|
||||
- `SessionTree` and implicit-main Session behavior are gone;
|
||||
- Branch is the data-only path/tip abstraction;
|
||||
- AgentLane exposes Branch methods directly and retains operation-aware append semantics;
|
||||
- AgentHarness is composition-only, has no AgentLane methods, and atomically gets/creates lanes;
|
||||
- fresh Session/Harness require no main;
|
||||
- all process-local publication and event-binding boundaries remain correct;
|
||||
- all three backends and focused race tests pass;
|
||||
- `harness.md` and WP05 describe the new model consistently;
|
||||
- final Fable review reports no findings;
|
||||
- WP05 M4 may resume.
|
||||
@@ -0,0 +1,372 @@
|
||||
# WP07 — SQLite host ownership and live-source forks
|
||||
|
||||
**Status: implemented.**
|
||||
|
||||
The delivered backend has no writer lease or replacement ownership primitive. It provides no-create read-write/read-only opens, queued same-repository snapshots, independent read-only WAL snapshots for live external sources, canonical physical identity, path-safe IDs, repository-local deletion reservation, and all-settled close. The tests cover both per-file and shared-container layouts, including a writer commit completed after a read snapshot boundary but before that reader closes.
|
||||
|
||||
This package aligns `packages/session-backends/sqlite-node` with the product ownership model: the server owns Session records and worker lifecycle, and exactly one host-assigned process owns writable Session authority at a time. Normally that process is the Session worker. The server may temporarily own a newly created or forked destination, but it closes that Session before handing its metadata to a worker.
|
||||
|
||||
Storage does not implement writer ownership. Remove the SQLite writer lease; do not repair or replace it.
|
||||
|
||||
A server-side fork is intentionally different from a second writer: it may open a live worker-owned source concurrently for one coherent read-only snapshot while the worker keeps committing. Shared SQLite containers remain supported.
|
||||
|
||||
## 0. Mandatory reading
|
||||
|
||||
Read completely before editing:
|
||||
|
||||
1. `packages/agent/docs/plugins.md` ownership, replacement, and removal sections.
|
||||
2. `packages/server/README.md` and relevant Session routing/removal source and tests.
|
||||
3. `packages/agent/docs/harness.md` §§0.6, 1.4–1.7, 2.7–2.8, 4.3, and Part 9.
|
||||
4. `packages/agent/docs/post-wp05-roadmap.md`.
|
||||
5. completed WP06 §7 and repository/fork conformance.
|
||||
6. every source file under `packages/session-backends/sqlite-node/src`.
|
||||
7. every test and benchmark under `packages/session-backends/sqlite-node`.
|
||||
8. `packages/session-backends/sqlite-node/README.md` and `CHANGELOG.md`.
|
||||
|
||||
Do not use stale files under `dist/` as implementation input. Completed WP01/WP06 documents are historical; do not rewrite them to hide the earlier lease implementation.
|
||||
|
||||
## 1. Fixed architecture
|
||||
|
||||
### 1.1 Writable authority belongs to the host
|
||||
|
||||
Exactly one host-assigned process owns a writable Session. The Session worker is the normal owner. Worker replacement closes the old owner before the new worker opens the Session. Server management serializes creation, forking, removal, and attachment lifecycle around that ownership transfer.
|
||||
|
||||
Memory, JSONL, and SQLite do not detect a second process opening the same Session for writes. Bypassing the server/worker lifecycle is a trusted-host defect, not a storage race to repair. A repository still rejects duplicate writable handles it owns in one process.
|
||||
|
||||
Do not add a storage lease, filesystem lock, fencing token, heartbeat, timeout-based takeover, deletion tombstone, quarantine protocol, or generic lock manager.
|
||||
|
||||
### 1.2 Read-only fork access may overlap the worker
|
||||
|
||||
The server owns repository administration and may fork a source while its Session worker continues writing it. The source side of that fork:
|
||||
|
||||
- opens the exact source container without creating it;
|
||||
- is read-only;
|
||||
- uses one deferred `BEGIN` transaction;
|
||||
- reads the version-gated Session row, scalar values, and selected entries/branch index in that transaction;
|
||||
- never upgrades to a write or claims writable authority;
|
||||
- closes after `COMMIT`/`ROLLBACK`.
|
||||
|
||||
SQLite WAL permits later worker commits while the read transaction remains open. The fork sees each source commit entirely before or entirely after its snapshot boundary, never a mixture.
|
||||
|
||||
### 1.3 Same-repository fork ordering remains distinct
|
||||
|
||||
A source already open in the same repository uses its active `SqliteStorage.snapshot()` path. That snapshot is queued on the source `commitQueue`, preserving WP06's admitted-commit ordering seam and the existing conformance case.
|
||||
|
||||
Do not replace this path with an independent connection. Instead, bind active storage lookup to exact physical identity plus Session ID so metadata for another container cannot select it accidentally.
|
||||
|
||||
### 1.4 Destination ownership does not overlap
|
||||
|
||||
`SessionRepo.create()` and `fork()` continue returning an open Session. A server may temporarily own that new destination, capture its metadata, and close it before launching a worker. This is a valid ownership transfer, not a reason to redesign `SessionRepo` into a record-only API.
|
||||
|
||||
## 2. Problems in current source
|
||||
|
||||
### 2.1 SQLite duplicates host ownership
|
||||
|
||||
Current source contains:
|
||||
|
||||
- `writer_lease` schema state;
|
||||
- claim, renew, and release helpers;
|
||||
- lease claims in create/open/fork;
|
||||
- an idle renewal timer and lease-loss path in `SqliteOpenSession`;
|
||||
- a pre-commit renewal callback in `SqliteStorage`;
|
||||
- lease-based deletion checks.
|
||||
|
||||
This is a second, incomplete ownership system. Its pre-commit renewal is not atomic with the following data transaction, but the correct fix is deletion of the ownership mechanism, not transaction-local fencing.
|
||||
|
||||
### 2.2 Non-creation access can create files
|
||||
|
||||
The database factory exposes only `open(path)`, which creates a missing SQLite file. Metadata open, listing probes, deletion, and fork-source reads must not turn a removed path into an empty database.
|
||||
|
||||
Fork-source reads also configure `PRAGMA journal_mode = WAL`, which is a write-oriented setup step and must not run on a read-only connection.
|
||||
|
||||
### 2.3 Delete does not reserve its local critical section
|
||||
|
||||
`delete()` checks `pendingIds` but does not reserve the ID. A same-repository create/open/fork destination can enter while asynchronous destructive work is in progress. Host lifecycle owns cross-process ordering; the repository still must serialize its own local operations.
|
||||
|
||||
### 2.4 Physical identity, paths, and close need correction
|
||||
|
||||
- `openStorages` is keyed only by Session ID, so the same ID at another physical path may select the wrong active source.
|
||||
- `create()` makes `options.directory`, not the parent of an explicit `databasePath`.
|
||||
- Per-session filenames interpolate arbitrary caller IDs directly; `/`, `\`, `..`, `%`, and platform separators must not escape `directory`.
|
||||
- `repo.close()` uses fail-fast `Promise.all`, so it may return before every open Session has attempted to drain and close.
|
||||
|
||||
Deterministic list ordering, bind-variable limits, branch-copy cost, fork scalar filtering, prepared statements, and VACUUM policy remain separate.
|
||||
|
||||
## 3. Required result
|
||||
|
||||
### 3.1 Remove storage-layer writer ownership
|
||||
|
||||
Delete all runtime lease behavior:
|
||||
|
||||
- delete `src/sqlite/session/writer-lease.ts`;
|
||||
- remove `writer_lease` from WIP `001_initial.sql`;
|
||||
- remove lease deletion from `deleteSessionRows()`;
|
||||
- remove claim/renew/release code from `SqliteSessionRepo`;
|
||||
- remove `beforeCommit` from `SqliteStorage`;
|
||||
- remove renewal/release options, timer, and `leaseError` from `SqliteOpenSession`;
|
||||
- remove lease-specific tests and replace them with host-authority and live-fork coverage.
|
||||
|
||||
Keep:
|
||||
|
||||
- the Storage `commitQueue`;
|
||||
- one `BEGIN IMMEDIATE` transaction per commit;
|
||||
- in-transaction `next_seq` allocation;
|
||||
- entry/usage uniqueness and parent triggers;
|
||||
- Session mutation admission and close draining;
|
||||
- process-local duplicate-open rejection.
|
||||
|
||||
Format 4 remains WIP. Remove the table from new schema in place; an old file containing an unused `writer_lease` table remains readable and the table and stale rows are ignored forever. Post-WP07 code does not delete them because doing so serves no runtime behavior. A pre-WP07 binary cannot open a new post-WP07 database without that table; backward compatibility for this WIP format is not required. Add no migration, compatibility path, or storage-version bump.
|
||||
|
||||
### 3.2 Add explicit database open modes
|
||||
|
||||
Extend `SqliteDatabaseFactory` with narrow operations:
|
||||
|
||||
- `open(path)` — intentional creation or create-if-missing;
|
||||
- `openExisting(path)` — read-write open that fails if the file does not exist;
|
||||
- `openReadOnly(path)` — read-only open that fails if the file does not exist.
|
||||
|
||||
The Node adapter uses `DatabaseSync(path, { readOnly: true })` for read-only access. Implement and test an actual no-create read-write mode for `openExisting`; do not rely on `access()` followed by a create-capable open.
|
||||
|
||||
Split connection setup:
|
||||
|
||||
- writable connections establish WAL mode and `busy_timeout`;
|
||||
- read-only connections set only read-safe options such as `busy_timeout` and never attempt to change journal mode.
|
||||
|
||||
Use no-create modes for metadata open, listing probes, deletion, and fork-source reads.
|
||||
|
||||
### 3.3 Preserve both fork-source paths
|
||||
|
||||
**Source open in this repository:** retain `SqliteStorage.snapshot()` and queue it after prior admitted commits. Replace the ID-only active map key with canonical `(containerPath, sessionId)` identity and use the same helper for publish, lookup, and removal.
|
||||
|
||||
**Source not open in this repository:** this includes a closed source and a source currently owned by a worker in another process. Open the exact source through `openReadOnly`, then capture it in one deferred read transaction. Validate the Session row and storage version inside that transaction. Do not consult destination reservations, claim source ownership, or block the worker's later commits.
|
||||
|
||||
The destination remains a normal writable create/fork transaction after source capture. In shared-container mode the source worker and destination transaction may use the same file; SQLite serializes destination writes while preserving Session row isolation.
|
||||
|
||||
### 3.4 Make deletion locally exclusive
|
||||
|
||||
The host must close the Session worker before calling `repo.delete()`. Direct cross-process deletion of a live Session is unsupported.
|
||||
|
||||
Within one `SqliteSessionRepo`, deletion must reserve the Session ID from entry through completion and release it in `finally`:
|
||||
|
||||
- an already open or reserved Session rejects deletion;
|
||||
- create/open/fork destination for that ID rejects while deletion runs;
|
||||
- shared-container deletion removes only the target Session's rows in one `BEGIN IMMEDIATE` transaction on one connection;
|
||||
- per-file deletion removes the database and its WAL/SHM sidecars after a no-create open/existence check;
|
||||
- a missing Session rejects without creating a file.
|
||||
|
||||
Do not add a lease check, tombstone, quarantine rename, or stale-deleter protocol. Cross-process removal ordering is the server's responsibility.
|
||||
|
||||
### 3.5 Bind metadata to physical identity and make paths safe
|
||||
|
||||
- Canonical identity is `(canonical container path, sessionId)`.
|
||||
- In per-file mode metadata must identify the repository-affine encoded path for its durable ID.
|
||||
- In shared-container mode metadata must identify the configured canonical container and Session ID.
|
||||
- A foreign or mismatched path must never alias a local active source. Pin in focused tests and the SQLite README whether foreign source metadata is rejected or read only from its exact path; do not silently substitute a local storage by ID.
|
||||
- Create `dirname(databasePath)` when `databasePath` is configured.
|
||||
- Encode arbitrary explicit IDs into safe per-session filenames without changing the durable ID. The encoding must prevent path escape and round-trip through metadata/list/open/fork.
|
||||
- Two different Session IDs in one shared container remain independently addressable.
|
||||
|
||||
### 3.6 Drain all repository-owned closes
|
||||
|
||||
`SqliteSessionRepo.close(context)` must:
|
||||
|
||||
1. seal repository admission once;
|
||||
2. start close on every currently open Session;
|
||||
3. wait for every close to settle;
|
||||
4. resolve when all succeed;
|
||||
5. otherwise reject only after all cleanup attempts, returning the one error or an `AggregateError` containing all failures;
|
||||
6. return the same promise on repeated close.
|
||||
|
||||
This is backend-local resource cleanup. Do not change the shared `SessionRepo` interface or JSONL lifecycle in this package.
|
||||
|
||||
## 4. Implementation slices
|
||||
|
||||
### Slice A — remove writer leases
|
||||
|
||||
Files:
|
||||
|
||||
- delete `src/sqlite/session/writer-lease.ts`;
|
||||
- `src/sqlite/migrations/001_initial.sql`;
|
||||
- `src/sqlite/session/session-row.ts`;
|
||||
- `src/sqlite/storage.ts`;
|
||||
- `src/sqlite/session.ts`;
|
||||
- `src/sqlite/repo.ts`;
|
||||
- lease-focused repository tests.
|
||||
|
||||
Tasks:
|
||||
|
||||
1. Remove schema/runtime lease state and timer behavior.
|
||||
2. Preserve commit serialization, one write transaction, mutation draining, and process-local duplicate-open behavior.
|
||||
3. Prove ordinary commits now use one write transaction rather than renewal plus write.
|
||||
|
||||
### Slice B — no-create opens and deletion reservation
|
||||
|
||||
Files:
|
||||
|
||||
- `src/index.ts`;
|
||||
- `src/sqlite/types.ts`;
|
||||
- `src/sqlite/repo.ts`;
|
||||
- focused adapter/repository tests.
|
||||
|
||||
Tasks:
|
||||
|
||||
1. Add `openExisting` and `openReadOnly` with tested no-create behavior.
|
||||
2. Separate writable and read-only connection configuration.
|
||||
3. Reserve deletion locally for its full critical section.
|
||||
4. Delete one shared-container Session in one transaction; preserve unrelated Sessions.
|
||||
|
||||
### Slice C — live-source read-only forks and identity
|
||||
|
||||
Files:
|
||||
|
||||
- `src/sqlite/repo.ts`;
|
||||
- repository/conformance tests.
|
||||
|
||||
Tasks:
|
||||
|
||||
1. Key active storage by canonical container plus Session ID; keep same-repository queue ordering.
|
||||
2. Use one independent read-only deferred transaction for non-open/live-worker sources.
|
||||
3. Validate source metadata/version inside the snapshot.
|
||||
4. Preserve shared-container destination behavior.
|
||||
|
||||
### Slice D — paths and close draining
|
||||
|
||||
Files:
|
||||
|
||||
- `src/sqlite/repo.ts`;
|
||||
- focused repository/conformance tests.
|
||||
|
||||
Tasks:
|
||||
|
||||
1. Create the actual custom container parent.
|
||||
2. Encode arbitrary IDs safely.
|
||||
3. Reject or exactly handle foreign metadata without active-source aliasing.
|
||||
4. Make repository close all-settled and error-complete.
|
||||
|
||||
### Slice E — documentation
|
||||
|
||||
Files:
|
||||
|
||||
- `packages/agent/docs/harness.md`;
|
||||
- `packages/agent/docs/post-wp05-roadmap.md`;
|
||||
- `packages/agent/docs/values.md`;
|
||||
- `packages/session-backends/sqlite-node/README.md`;
|
||||
- changelog only under normal branch rules.
|
||||
|
||||
Document host-owned writable authority, the two fork-source paths, no-create opens, local deletion reservation, and absence of storage-layer ownership.
|
||||
|
||||
## 5. Required tests
|
||||
|
||||
Use real independent `node:sqlite` connections. Test-only wrappers may expose deterministic transaction boundaries; production code gets no sleeps or race flags.
|
||||
|
||||
### Lease removal
|
||||
|
||||
- new schema has no `writer_lease` table;
|
||||
- create/open/fork/commit/close perform no lease reads or writes and start no renewal timer;
|
||||
- same-repository duplicate writable open still rejects through process-local reservation;
|
||||
- ordinary commit remains one `BEGIN IMMEDIATE` transaction.
|
||||
|
||||
### Live fork source
|
||||
|
||||
For both per-file and shared-container layouts:
|
||||
|
||||
- a server repository forks a source held open by a separate repository/connection representing its worker;
|
||||
- source capture uses a distinct read-only connection and claims no writable ownership;
|
||||
- a source commit completed before the snapshot boundary appears wholly in the fork;
|
||||
- a commit after the read snapshot is established can complete before the reader closes and is wholly absent from that fork;
|
||||
- no fork contains an entry without the same commit's Branch tip/value/stats changes;
|
||||
- a later fork includes the later commit;
|
||||
- same-repository admitted-commit fork conformance remains unchanged.
|
||||
|
||||
### Deletion
|
||||
|
||||
- open/reserved Session first → same-repository delete rejects;
|
||||
- delete reservation first → same-repository create/open/fork destination for that ID rejects;
|
||||
- shared deletion removes only the target rows;
|
||||
- per-file deletion removes database/WAL/SHM files;
|
||||
- missing-path open/list/fork/delete does not create an empty database;
|
||||
- tests state the host precondition: worker close precedes deletion; no cross-process bypass safety is promised.
|
||||
|
||||
### Identity and paths
|
||||
|
||||
- same Session ID at two physical paths cannot cross-select active source storage;
|
||||
- two Session IDs in one shared container remain independent;
|
||||
- `databasePath` succeeds when its parent does not exist;
|
||||
- explicit IDs containing `../`, `/`, `\`, `%`, dots, and Unicode stay inside `directory` and preserve the metadata ID;
|
||||
- metadata returned by create/list/open/fork names the actual container.
|
||||
|
||||
### Close
|
||||
|
||||
- one Session close failure does not prevent every other Session cleanup attempt;
|
||||
- multiple failures are reported after all settle;
|
||||
- repeated repository close returns the same promise;
|
||||
- every successfully closed Session releases its connection.
|
||||
|
||||
### Regression
|
||||
|
||||
- existing storage and repository conformance remains semantically unchanged;
|
||||
- fork destination reservation and same-repository source ordering remain intact;
|
||||
- fork snapshots exclude operation/pending/result/usage/application state exactly as before;
|
||||
- shared-container create/list/open/fork/delete remains supported;
|
||||
- every write transaction still uses `BEGIN IMMEDIATE`;
|
||||
- no migration, storage-version bump, compatibility layer, or replacement ownership primitive appears.
|
||||
|
||||
## 6. Validation and review
|
||||
|
||||
After each code slice:
|
||||
|
||||
```bash
|
||||
npm run check
|
||||
```
|
||||
|
||||
Run each modified focused test from `packages/session-backends/sqlite-node` with the repository Vitest binary. Final validation:
|
||||
|
||||
```bash
|
||||
./test.sh
|
||||
```
|
||||
|
||||
Review checkpoints:
|
||||
|
||||
1. Fable after Slice A: verify no storage ownership mechanism remains and close draining is intact.
|
||||
2. Fable after Slice C: verify same-repository ordering and live-worker read-only overlap both hold.
|
||||
3. Final Fable review of source, tests, docs, and exclusions.
|
||||
|
||||
Delegated reviews use provider `anthropic` and model `claude-fable-5`.
|
||||
|
||||
## 7. Exclusions
|
||||
|
||||
Do not include:
|
||||
|
||||
- any storage lease, lock, fence, heartbeat, takeover, tombstone, or quarantine;
|
||||
- a record-only `SessionRepo` redesign or compatibility facade;
|
||||
- server/router/worker-manager redesign; record any backend-independent lifecycle race separately;
|
||||
- SQLite branch-segment redesign or uncompacted-divergence optimization;
|
||||
- fork scalar filtering/indexing;
|
||||
- `getEntries` bind-limit chunking or general query-limit normalization;
|
||||
- statement caches or stats aggregation optimization;
|
||||
- catalog redesign, async database replacement, or VACUUM policy;
|
||||
- search/FTS;
|
||||
- R11 migration machinery or a storage-version bump;
|
||||
- [mobile assistant-output handoff](../mobile-handoff/01-harness/05-assistant-output/message-update.md) changes or JSONL compaction;
|
||||
- repository-wide `SessionRepo.close()` contract changes;
|
||||
- removal of shared-container support;
|
||||
- transaction DSLs, schedulers, generic lock managers, or compatibility layers.
|
||||
|
||||
If implementation requires an excluded item, stop and revise the handoff rather than expanding silently.
|
||||
|
||||
## 8. Exit condition
|
||||
|
||||
WP07 is complete when:
|
||||
|
||||
- SQLite contains no active or schema-defined writer lease and implements no replacement ownership mechanism;
|
||||
- host ownership is the documented single-writer authority;
|
||||
- same-repository active-source forks retain their queued commit boundary;
|
||||
- live worker-owned sources fork through an independent read-only snapshot while later WAL commits continue;
|
||||
- deletion reserves its same-repository critical section and assumes worker-first host removal;
|
||||
- non-creation paths cannot create empty databases;
|
||||
- active source identity includes physical container plus Session ID;
|
||||
- explicit IDs cannot escape the directory and custom database parents are created;
|
||||
- repository close waits for every cleanup attempt;
|
||||
- shared-container mode remains fully covered;
|
||||
- focused tests, `npm run check`, and `./test.sh` pass;
|
||||
- final Fable review reports no blocker.
|
||||
@@ -0,0 +1,240 @@
|
||||
# WP08 — Named-branch and tree forks with streaming copies
|
||||
|
||||
**Status: in progress — implementing Slice A.**
|
||||
|
||||
This package replaces the fork contract: `ForkOptions` gains a mandatory scope and a mandatory named source branch, branch forks validate a complete configured source AgentLane and ancestry membership, tree forks copy the complete immutable tree plus current application values/lists, and all three backends replace materialized source snapshot arrays with bounded-memory streaming copies. A JSONL fork never repairs or mutates its source. One closed core classifier owns every namespace's fork disposition.
|
||||
|
||||
WP07 is a hard dependency and is preserved unchanged: no-create database modes, canonical `(containerPath, sessionId)` identity, independent read-only WAL reader for external/live-worker sources, repository-local deletion reservation, and all-settled close. This package supersedes only the roadmap's "SQLite fork cost" performance item.
|
||||
|
||||
## 0. Mandatory reading
|
||||
|
||||
Read completely before editing:
|
||||
|
||||
1. `packages/agent/docs/harness.md` §§0.6, 1.3–1.7, 2.3, 2.7–2.9, Part 9 (invariants 3–5, 13, 16; ledger completeness).
|
||||
2. `packages/agent/docs/values.md`, especially "Forks and rewrites" and the backend sections.
|
||||
3. `packages/agent/docs/post-wp05-roadmap.md` (SQLite fork cost, repository lifecycle context).
|
||||
4. Completed WP06 §7 and WP07 (historical; do not edit them).
|
||||
5. `packages/agent/src/harness/session/fork.ts`, `types.ts` (`ForkOptions`, `SessionRepo`), `values.ts`, `in-memory-storage-state.ts`, `memory.ts`, `session/index.ts`.
|
||||
6. `packages/agent/src/harness/session/jsonl/repo.ts`, `jsonl/storage.ts`, `jsonl/codec.ts`, `jsonl/legacy-v3.ts`, `jsonl/types.ts`.
|
||||
7. `packages/session-backends/sqlite-node/src/sqlite/repo.ts`, `storage.ts`, `session/values.ts`, `session/entries.ts`, `session/branch-entries.ts`, `types.ts`.
|
||||
8. `packages/agent/src/harness/session/testing/conformance/session-repo.ts` and every test named in §5.
|
||||
9. `packages/agent/src/harness/session/testing/benchmark/session-repo.ts` and both `session-repo.bench.ts` files.
|
||||
|
||||
Do not use `dist/` output as implementation input. WP00–WP07 documents and released changelog sections are immutable in this package.
|
||||
|
||||
## 1. Fixed architecture
|
||||
|
||||
### 1.1 Public contract
|
||||
|
||||
```ts
|
||||
export type ForkOptions =
|
||||
| { scope: "branch"; branch: string; entryId?: string; position?: "before" | "at"; id?: string }
|
||||
| { scope: "tree"; id?: string };
|
||||
```
|
||||
|
||||
`scope` is required. `branch` is required for branch scope. There is no default scope, no implicit `main`, and no compatibility alias.
|
||||
|
||||
**Branch scope** requires a complete configured source AgentLane: `pi.branch.tip/{branch}`, `pi.lane.config/{branch}`, and `pi.lane.state/{branch}` must all exist. A missing tip rejects (`unknown branch`); a data-only Branch (tip without config/state) rejects; a partial config/state pair, or `laneConfig`/`laneState` present without a `branchTip`, is corruption and rejects. `entryId`, when supplied, must be an entry on that Branch's current tip ancestry (inclusive); an entry elsewhere in the tree rejects. Omitted `entryId` means the current tip. `position` defaults to `"at"`; `"before"` selects the target's parent and may yield a `null` destination tip (before the root entry, or a `null` source tip with no `entryId`) — legal. The destination contains exactly one Branch with the same name, the selected tip, the copied `LaneConfiguration`, and fresh idle lane state `{ currentOperationId: null, lastOperationId: null, inbox: [] }`. No other Branch or lane exists in the destination.
|
||||
|
||||
This deliberately makes branch forks of data-only Branches — including legacy v3 imports whose main lacked reconstructable model/thinking history — unavailable. Tree scope remains available for those sources.
|
||||
|
||||
**Tree scope** copies: every immutable entry, including entries unreachable from every current Branch tip; every Branch tip verbatim; every configured lane's config plus fresh idle lane state under the same name; data-only Branches (tip present, config/state absent) as data-only Branches. A partial config/state pair, or `laneConfig`/`laneState` without a `branchTip`, is corruption in both scopes: tree scope **rejects** it rather than dropping it.
|
||||
|
||||
**Both scopes**: copy `pi.session.name`; copy `pi.entry.label` values only for copied entries; exclude the usage ledger, `pi.result`, all `pi.op.*`, all `pi.pending.*` (entries, tool checkpoints, assistant frames), and every trace of open-operation state. Destination usage totals start at zero; destination `messageCount` equals the number of copied message entries, exactly as current conformance already proves. Destination metadata records `parentSessionId = source.id`. Copied entries, values, and list elements preserve their original `seq`; the destination `nextSeq` high-water mark is the source's, so no sequence is reused. Transformed built-ins keep coherent seqs: the rewritten `branchTip` and each fresh idle `laneState` reuse the seq of the source's current corresponding value row, and copied config, name, labels, application values, and list elements retain theirs — destination writes stay in original seq order under an unchanged high-water mark.
|
||||
|
||||
**Application values/lists (outside the reserved `pi`/`pi.*` namespaces)**: tree scope copies every current scalar value and every surviving list element with original seqs; branch scope copies none. A `seq <= tipSeq` cutoff is forbidden as "historical reconstruction" — it is provably not one:
|
||||
|
||||
```text
|
||||
Scalar: TX[seq 10: set my-app.state = v1] · TX[seq 12: insert e1] ·
|
||||
TX[seq 50: set my-app.state = v2] (v2 describes work after e1)
|
||||
Branch fork at e1, cutoff seq <= 12: only v2 exists (v1 was replaced, no
|
||||
history is retained); 50 > 12 excludes it → state absent, though the app
|
||||
demonstrably had state v1 at the fork point. The cutoff cannot recover v1.
|
||||
|
||||
List: append seq 5 · append seq 20 · deleteList seq 40 · append seq 60
|
||||
Cutoff seq <= 30: elements 5 and 20 no longer exist (whole-list delete
|
||||
destroyed them); only 60 survives and is excluded → empty list, though the
|
||||
list held {5, 20} at seq-30 time.
|
||||
```
|
||||
|
||||
Any cutoff filters *survivors*, silently mixing rewound intent with post-deletion reality. Branch scope therefore copies nothing application-owned, matching its fresh idle lane state and zero ledger; applications own their own re-derivation.
|
||||
|
||||
### 1.2 One closed fork classifier
|
||||
|
||||
All namespace fork knowledge lives in one core module beside `session/values.ts` and `session/fork.ts` (e.g. `session/fork-policy.ts`). It is a closed switch, not a registry, plugin policy, or DSL:
|
||||
|
||||
- `pi.op.*`, `pi.pending.*`, `pi.result` → exclude, both scopes.
|
||||
- `pi.session.name` → copy.
|
||||
- `pi.entry.label` → copy iff the keyed entry is copied.
|
||||
- `pi.branch.tip`, `pi.lane.config`, `pi.lane.state` → structured lane actions; the scope-specific rules (branch keeps only the named lane and rewrites its tip; lane state is always replaced with fresh idle state; tree keeps all) live in one shared driver consumed by all backends.
|
||||
- The exact namespace `pi` and any other `pi.*` namespace → the fork **fails**, but only when current surviving state exists at fork time (a current scalar row or a surviving list element). Historical writes later replaced or deleted are absent from current state on every backend and must not alone fail a JSONL fork — behavior is backend-equivalent. Introducing a new built-in namespace without declaring its fork semantics must break fork tests, not silently copy or drop state.
|
||||
- Neither `pi` nor `pi.*` → application: copy on tree, exclude on branch. The only built-in list namespace (`pi.pending.assistant_frame`) excludes; application lists follow the application rule.
|
||||
|
||||
The driver exposes a streaming shape — accept one committed value/list write (or current row), emit zero or more destination writes, `finish()` emits fresh idle lane states and the rewritten branch tip — holding only bounded state (lane names, destination tip). Entry-copy membership is a backend-supplied predicate so each backend uses its own index. `createForkSnapshot`, `forkSnapshotWrites`, `ForkSourceSnapshot`, `ForkDestinationSnapshot`, and the `entriesComplete` escape hatch are deleted.
|
||||
|
||||
### 1.3 Bounded-memory backend procedures
|
||||
|
||||
Auxiliary memory during a fork must stay bounded regardless of source size. The unavoidable exceptions are Memory's destination state and the returned open destination Session (JSONL destination state after open). Source copy paths must not call array-returning full reads: no `snapshotEntriesAndValues()`, `captureForkSource()`, `readAllScalarValueRows()`, `readAllEntryRows()`, whole-file `readTextFile`, or SQLite `.all()` over unbounded row sets.
|
||||
|
||||
**Memory.** At a source `commitQueue` boundary, build the destination `InMemoryStorageState` directly by iterating source maps once through the classifier/driver. Branch scope walks `parentId` from the branch tip toward the root to compute the ancestry id set and verify `entryId` membership; copy exactly that set. No intermediate snapshot arrays; `MemoryStorage.fromSnapshot` and `captureForkSource` are removed.
|
||||
|
||||
**JSONL.** The source is read through a read-only path that captures a fixed file prefix — open a read handle, record the current length at capture time, and read only that prefix — and never writes the source: no torn-tail truncation/rewrite, no v3 normalization persistence, no `.tmp` beside the source. A torn or incomplete final line inside the captured prefix is discarded in memory. The source `nextSeq` high-water mark is derived as `max(header.nextSeq when present, highest complete captured write seq + 1)` — ordinary headers are not rewritten on every append, and a torn incomplete final write never advances it.
|
||||
|
||||
- *Both scopes* use two passes over the captured prefix with one temporary disk-backed index (deleted in `finally` on success and failure). Pass 1 streams every line and folds value/list writes into the index: per scalar address, the seq of the current surviving `set` (absent after a trailing `delete`); per list address, the set of surviving element seqs (whole-list delete clears it); plus entry `id → parentId` (parents always precede children) where branch scope needs ancestry. The fold applies the same current-state semantics replay does, without whole-state RAM.
|
||||
- Pass 2 streams the prefix again in order and emits a write only when the index proves it current and the classifier selects it: an entry write in the selected set; a scalar `set` whose seq equals the index's current-row seq for its address; a list `append` whose element seq is in the survivor set. Deletes, superseded sets, and dead application value/list history are never emitted as destination state — the destination file contains only current selected rows, already in original seq order because source lines are seq-ordered.
|
||||
- Transformed built-ins emit in place: when pass 2 reaches the seq of the source's current `branchTip` row for a kept branch it emits the (possibly rewritten) tip under that same seq, and when it reaches the seq of the current `laneState` row for a kept configured lane it emits fresh idle lane state under that same seq. No tail appends exist, so replay stays seq-monotonic and the high-water mark is untouched.
|
||||
- *Branch scope* walks tip→root through the disk index to materialize the ancestry set on disk and verify `entryId` membership before pass 2; labels emit only for member entries.
|
||||
- The destination is staged in a temp file and atomically renamed (existing `publishFileAtomically`); failure removes the temp file and the index. Legacy v3 splits by source state. Forking an **open** legacy-v3 `JsonlStorage` **rejects with a clear error** until a normal non-empty commit has upgraded and persisted its normalized format-4 ids: v3 normalization mints nondeterministic UUIDv7 tails, so an independent disk reparse cannot reproduce the ids the open Session exposes or validate a caller `entryId`; the fork must not mutate/upgrade the source itself. A **closed** legacy-v3 source uses the bounded fixed-prefix disk-backed parser/normalizer and never touches the file: tree forks succeed; branch forks may use an omitted `entryId` (the default normalized tip) when the import reconstructs a complete configured lane, while any caller-supplied process-local id from an earlier open is not stable across reparses and rejects normally; a data-only reconstructed lane rejects as everywhere. Ordinary writable `JsonlStorage.open` may retain in-memory normalization, but resident state is never a fork source.
|
||||
- There is **no resident-state path**: an open same-repository JSONL source enqueues a short boundary callback on the source `commitQueue` whose only job is to capture the fixed read-only file-handle prefix (open handle, record length), then releases the queue; the fork runs the same two-pass disk-backed procedure over that prefix while later source appends proceed. Committed writes are durable in the file before they are applied to resident state, so the prefix is authoritative at the boundary, and file order preserves seq order without resident-Map ordering ambiguity.
|
||||
|
||||
**SQLite.** Iterator-based row transfer through a bounded-memory temporary on-disk SQLite **staging database**, used uniformly for per-file and shared-container layouts. The source reader never streams directly into the destination transaction — a WAL reader and one writer can coexist, but in shared-container mode the destination writer would hold the container's sole write lock and block the post-boundary source writer while capture is still streaming; staging into a separate file removes that coupling. Use prepared-statement iteration (`iterate`/stepwise), never source-sized `.all()` snapshot arrays:
|
||||
|
||||
- *External/closed/live-worker source:* WP07 path — `openReadOnly` on the exact canonical path, one deferred read transaction, session row and storage version validated inside it.
|
||||
- *Same-repository open source:* open the independent read-only connection **first**, then enqueue a short boundary callback on the source Storage `commitQueue` whose only job is to `BEGIN` and establish the independent reader's snapshot (issue a trivial read) before releasing the queue. Do not begin a read transaction on the source writer connection and then release its queue, and do not hold the queue for the copy duration.
|
||||
- *Stage:* while the source reader remains open, stream selected entries `ORDER BY seq` and classifier-selected current scalar/list rows (SQL-level namespace prefilters matching the classifier — enumerated built-in namespaces plus the application predicate excluding exact `pi` and `pi.%` — with each row still passing the classifier) into the temporary staging database in bounded batches. Branch scope enumerates the ancestry through the `branch_entries` segment chain up to the selected entry and answers `entryId` and label membership through that index, not an in-RAM id set. Later source commits use the original writer connection and may complete while staging streams, in **both** layouts, because stage writes target another file.
|
||||
- *Publish:* close/commit the source read transaction, then stream the stage into one destination `BEGIN IMMEDIATE` transaction — entries in seq order maintaining the destination branch index and `message_count` incrementally, then values/list elements — and delete the staging database in `finally` on success and failure. Destination `next_seq` is the source's. Shared-container destinations write only the new session's rows.
|
||||
|
||||
### 1.4 Preserved WP07 behavior
|
||||
|
||||
Destination id reservation across create/open/fork/delete, no-create opens, foreign-metadata rejection, per-file and shared-container layouts, WAL commit-boundary wholeness (a source commit is wholly inside or wholly outside one fork), and all-settled repository close are unchanged.
|
||||
|
||||
### 1.5 Coding-agent status (record only)
|
||||
|
||||
`/fork`, `/clone`, and `--fork` run entirely on the legacy `SessionManager` (`createBranchedSession`, `forkFrom`) and are not migrated here. Future mapping when coding-agent adopts `SessionRepo`: `/fork` → `{ scope: "branch", branch: "main", entryId, position: "before" }`; `/clone` → `{ scope: "branch", branch: "main" }`; `--fork` → `{ scope: "tree" }`.
|
||||
|
||||
## 2. Problems in current source
|
||||
|
||||
- `ForkOptions` defaults `scope` to `"branch"` and hard-codes source/destination `main` (`fork.ts`, both backend branch readers).
|
||||
- No ancestry membership check: `selectForkContents` walks parents from any supplied `entryId` anywhere in the tree and labels the result `main`.
|
||||
- Every backend materializes the full source: `snapshotEntriesAndValues()` (Memory/JSONL), `readAllScalarValueRows` + `readAllEntryRows`/`scanBranchEntries` arrays (SQLite), all funneled through in-memory `createForkSnapshot`.
|
||||
- `createForkSnapshot` renumbers copied scalar values with fresh seqs and computes `nextSeq` as `max entry seq + 1`, losing original value seqs.
|
||||
- A JSONL fork of a closed source uses `JsonlStorage.open()`, which rewrites the source file on a torn tail — a fork can mutate its source today.
|
||||
- Lists are never copied; application values are excluded with no declared tree policy (`values.md` explicitly defers it).
|
||||
- Namespace fork knowledge is repeated across `fork.ts`, `values.md` prose, and conformance assertions; nothing forces a new `pi.*` namespace to declare fork semantics.
|
||||
- `entriesComplete?: false` exists only to let SQLite branch snapshots skip tip validation for other branches.
|
||||
|
||||
## 3. Required result
|
||||
|
||||
1. `ForkOptions` and validation exactly as §1.1; all rejection paths create no destination file, database rows, or reserved-but-leaked ids.
|
||||
2. The closed classifier/driver of §1.2, exported from core and consumed by all three backends; unknown reserved namespaces (exact `pi` or undeclared `pi.*`) fail the fork only when current surviving state exists, identically on every backend.
|
||||
3. Streaming procedures of §1.3 on all three backends; `createForkSnapshot`/`captureForkSource`/`snapshot()` fork plumbing and their exports removed from `session/index.ts` and the sqlite-node import surface.
|
||||
4. Seq preservation (including reused seqs for rewritten tips and fresh idle lane states) and source `nextSeq` high-water on every backend (JSONL derives it per §1.3); identical logical destination state across backends for identical sources (conformance).
|
||||
5. JSONL source non-mutation, including torn-tail sources and legacy v3 sources.
|
||||
6. Documentation: `harness.md` §2.7 (and the §1.7 fork-related sentences), `values.md` "Forks and rewrites" and backend snapshot mentions, `post-wp05-roadmap.md` (retire the SQLite fork-cost item, add/point to this package), `packages/session-backends/sqlite-node/README.md` fork paragraphs, `packages/agent/benchmark/session/README.md` if dataset wording changes. Historical WP docs and released changelogs untouched. Format/storage versions unchanged; no migration.
|
||||
|
||||
## 4. Implementation slices
|
||||
|
||||
### Slice A — contract and classifier (core, Memory reference)
|
||||
|
||||
Files: `session/types.ts`, new `session/fork-policy.ts`, `session/fork.ts` (rewritten or deleted), `session/values.ts` (only if new helpers are needed), `session/index.ts`, `session/memory.ts`, `session/in-memory-storage-state.ts`, conformance `testing/conformance/session-repo.ts`, `test/harness/memory-conformance.test.ts`, `test/harness/memory-session-repo.test.ts`.
|
||||
|
||||
1. Replace `ForkOptions`; implement validation and the classifier/driver.
|
||||
2. Rewrite Memory fork as direct destination construction at a commit-queue boundary.
|
||||
3. Rewrite the shared fork conformance for the new contract (§5) and pass it on Memory.
|
||||
|
||||
### Slice B — JSONL streaming
|
||||
|
||||
Files: `session/jsonl/storage.ts`, `session/jsonl/repo.ts`, `session/jsonl/types.ts`, streaming reader support in the `FileSystem` capability (`harness/types.ts`, `harness/env/*`) if required, `test/harness/jsonl-session-repo.test.ts`, `jsonl-session-repo-conformance.test.ts`, `jsonl-storage.test.ts`, `jsonl-v3-migration.test.ts`.
|
||||
|
||||
1. Read-only fixed-prefix source capture with in-memory torn-tail discard; the same capture at the source `commitQueue` boundary for open sources; no source writes on any fork path.
|
||||
2. Two-pass disk-backed current-state fold and emission for both scopes; branch ancestry membership through the disk index; in-place transformed built-ins; atomic destination publish.
|
||||
3. Open-v3 fork rejection and the closed-v3 disk-backed parser/normalizer path; update v3 fork tests per §5.
|
||||
|
||||
|
||||
### Slice C — SQLite streaming
|
||||
|
||||
Files: `sqlite/repo.ts`, `sqlite/storage.ts`, `sqlite/session/values.ts`, `sqlite/session/entries.ts`, `sqlite/session/branch-entries.ts`, `sqlite/types.ts`, `test/repo.test.ts`, `test/repo-conformance.test.ts`.
|
||||
|
||||
1. Replace `SqliteStorage.snapshot()`/array snapshot helpers with the independent-reader-plus-boundary-callback design and iterator transfer into the temporary staging database, then stage-to-destination publication with `finally` cleanup.
|
||||
2. Branch ancestry/membership/labels through the branch index; classifier-matching SQL prefilters.
|
||||
3. Preserve WP07 identity, reservation, no-create, and close coverage; both layouts.
|
||||
|
||||
### Slice D — benchmarks and documentation
|
||||
|
||||
Files: `testing/benchmark/session-repo.ts`, both `session-repo.bench.ts` files, `benchmark/session/README.md`, `harness.md`, `values.md`, `post-wp05-roadmap.md`, sqlite-node `README.md`, changelogs only under normal branch rules.
|
||||
|
||||
Update fork option literals, add large-source fork benchmarks (tree and branch), and land the §3.6 documentation set.
|
||||
|
||||
## 5. Required tests
|
||||
|
||||
### Contract and validation
|
||||
|
||||
- branch scope rejects: unknown branch name; data-only Branch (tip only); partial config/state pair (corruption); `entryId` not on the named Branch's tip ancestry (present elsewhere in the tree); `entryId` unknown; `entryId` with a `null` tip. Each rejection leaves no destination artifact and releases its reserved id.
|
||||
- branch scope accepts: omitted `entryId` (tip), explicit tip, mid-ancestry entry, `position: "before"` at a mid entry and at the root entry (`null` destination tip), `null` source tip with no `entryId`.
|
||||
- destination shape: exactly one Branch, same name, copied config, fresh idle lane state, no other lane values, `parentSessionId` set.
|
||||
- tree scope: unreachable entries copied; every tip copied; configured lanes get config plus fresh idle state; data-only Branches stay data-only; partial pairs fault.
|
||||
- both scopes: session name copied; labels only for copied entries (branch fork excludes labels of non-ancestry entries); `pi.result`, `pi.op.*`, `pi.pending.*` (pending entries, tool checkpoints, assistant frame lists), and usage rows absent; exact stats expectation — a fork copying N message entries reports `getStats()` = `{ messageCount: N, usage: all-zero }`, matching current conformance; the ledger-completeness invariant ("a fork's ledger starts at zero") retained.
|
||||
|
||||
### Application values and lists
|
||||
|
||||
- tree fork copies every non-`pi.*` scalar and every surviving list element with original seqs; list cursors from the source page identically in the destination; a list deleted-then-reappended in the source reproduces only survivors.
|
||||
- branch fork copies no application values/lists (pin the §1.1 scalar and list traces as regression cases: post-fork-point overwrite and delete-destroyed elements must not resurface under any implementation).
|
||||
- current surviving state in an unknown reserved namespace (exact `pi` or an undeclared `pi.*` scalar or surviving list element) fails the fork on every backend; the same namespace **set then deleted** before the fork fails no backend — including JSONL, where the dead history remains as physical lines and the disk fold must classify it as absent. Construct both cases via raw committed writes.
|
||||
|
||||
### Sequence preservation
|
||||
|
||||
- copied entries, values, and list elements keep source seqs; rewritten branch tips and fresh idle lane states occupy the seqs of the source's current corresponding rows; destination `nextSeq` equals the source high-water mark and the first post-fork commit allocates above it on every backend (JSONL header `nextSeq`, SQLite `next_seq`).
|
||||
- JSONL derives the high-water mark as `max(header.nextSeq when present, highest complete captured write seq + 1)`; a torn incomplete final write does not advance it.
|
||||
|
||||
### Bounded memory and source non-mutation
|
||||
|
||||
- instrumented source readers (test decorators/spies) assert fork paths never call `snapshotEntriesAndValues`, `captureForkSource`, `readAllScalarValueRows`, `readAllEntryRows`, whole-file `readTextFile`, or source-sized `.all()` arrays on entries/values/lists — bounded iterator steps and staging-database operations are permitted; deterministic large fixtures (extend the existing benchmark dataset generators) exercise tree and branch forks over multi-thousand-entry sources.
|
||||
- SQLite staging databases are removed on success and failure; a failed fork leaves neither a stage file nor destination rows.
|
||||
- JSONL: fork of a closed torn-tail source succeeds, the destination excludes the torn transaction wholly, and the source file bytes are unchanged (byte-compare before/after); fork of a legacy v3 source leaves the source file unchanged; no `.tmp` appears beside the source; temporary ancestry index files are removed on success and failure.
|
||||
- JSONL fixed-prefix capture: a source append completed after capture is wholly absent from the fork.
|
||||
- JSONL destination content: the destination file contains only current selected scalar rows and surviving list elements in original seq order — no delete records, superseded sets, or dead application history; the rewritten tip and fresh idle lane state sit at the source current rows' seqs.
|
||||
- legacy v3: forking an **open** v3 source rejects with the clear pre-upgrade error; after one normal non-empty commit upgrades it to format 4, a fork succeeds and preserves the persisted ids; a **closed** v3 tree fork succeeds through the disk-backed parser (instrumented: no whole-state in-memory normalization on the fork path) and leaves the source byte-identical; a closed v3 branch fork with omitted `entryId` succeeds when a complete configured lane is reconstructed, a caller-supplied id from an earlier open rejects, and a data-only reconstructed main rejects.
|
||||
|
||||
### Coordination and ordering
|
||||
|
||||
- Memory open-source forks keep the queue-boundary conformance case: a commit admitted before the fork appears wholly; one admitted after does not.
|
||||
- JSONL open-source forks capture the fixed prefix at the `commitQueue` boundary: a commit admitted before the boundary appears wholly; a source append completed after boundary capture proceeds while the fork is still streaming and is wholly absent from that fork.
|
||||
- SQLite same-repo: the independent reader opens first, the boundary callback on the source `commitQueue` establishes its snapshot and releases the queue, and a subsequent source commit on the writer connection **completes while the reader is still streaming into the stage** and is wholly absent from that fork; a later fork includes it wholly. Assert this in **both** per-file and shared-container layouts (satisfiable in shared containers because stage writes target another file), plus the WP07 live external-worker case unchanged.
|
||||
- destination-reservation races (create vs fork on one id, both orders) unchanged.
|
||||
|
||||
### Backend equivalence
|
||||
|
||||
- one shared conformance source (entries, unreachable entries, multiple lanes, data-only Branch, labels, app scalars/lists with deletes, open operation with pending/frame/checkpoint state, usage rows) forks to identical logical destination state on Memory, JSONL, and SQLite for both scopes.
|
||||
|
||||
## 6. Validation and review
|
||||
|
||||
After each slice: `npm run check`, then the modified focused tests via the repository Vitest binary from the owning package; `./test.sh` before final review. Grep guards: no remaining `createForkSnapshot|captureForkSource|ForkSourceSnapshot|entriesComplete` outside historical docs and this handoff; no **production fork source path** referencing `"main"` as a literal (this document and the recorded coding-agent mapping in §1.5 intentionally mention main).
|
||||
|
||||
Review checkpoints (delegated reviews use provider `anthropic`, model `claude-fable-5`):
|
||||
|
||||
1. After Slice A: contract, classifier closure, conformance shape.
|
||||
2. After Slice C: same-repo ordering, reader independence, bounded-memory evidence.
|
||||
3. Final review of source, tests, docs, and exclusions.
|
||||
|
||||
## 7. Exclusions
|
||||
|
||||
Do not include:
|
||||
|
||||
- compatibility aliases, a default scope, or implicit `main`;
|
||||
- any `seq <= tip` cutoff or other historical-state reconstruction;
|
||||
- a fork-policy registry, plugin hook, or DSL; per-address application opt-ins;
|
||||
- copying usage rows, `pi.result`, or any open-operation state under any option;
|
||||
- coding-agent `/fork`, `/clone`, `--fork`, or RPC migration;
|
||||
- storage-version bumps, migrations, or format changes (formats stay WIP-in-place);
|
||||
- J1 snapshot compaction (the streaming reader may be built shareable, nothing more);
|
||||
- SQLite branch-segment redesign or the uncompacted-divergence fix;
|
||||
- `SessionRepo` interface changes beyond `ForkOptions` (repository `close()` belongs to the lifecycle package);
|
||||
- precise-rewrite tooling; search; edits to WP00–WP07 documents or released changelog sections.
|
||||
|
||||
If implementation requires an excluded item, stop and revise the handoff.
|
||||
|
||||
## 8. Exit condition
|
||||
|
||||
WP08 is complete when:
|
||||
|
||||
- `ForkOptions` is exactly the §1.1 union with the stated validation, on all three backends;
|
||||
- branch forks require a complete configured source AgentLane and enforce ancestry membership; tree forks copy the complete immutable tree, every tip, configured and data-only Branches;
|
||||
- tree forks carry all current application values and surviving list elements with original seqs; branch forks carry none;
|
||||
- one closed core classifier owns every namespace disposition; unknown reserved namespaces (exact `pi` or undeclared `pi.*`) fail forks exactly when current surviving state exists, equivalently on all backends;
|
||||
- all supported fork paths — including closed legacy v3 sources — are streaming and bounded in auxiliary memory per §1.3, proven by instrumented-reader tests over large fixtures; forking an open legacy-v3 source is explicitly unsupported (clear rejection) until an ordinary commit upgrades it, not handled by an in-memory exception;
|
||||
- JSONL forks never mutate their source, including torn-tail and legacy v3 sources, and JSONL destinations contain only current selected rows in original seq order;
|
||||
- SQLite forks stage through a temporary on-disk database (source reader → stage while open, stage → one destination `BEGIN IMMEDIATE` after reader close, stage deleted in `finally`), with the independent-reader boundary design and concurrent later writer commits proven in both layouts, preserving all WP07 behavior;
|
||||
- copied seqs and `nextSeq` high-water marks are preserved and backends produce identical logical destinations;
|
||||
- shared conformance, focused backend tests, benchmarks, `npm run check`, and `./test.sh` pass;
|
||||
- normative docs and the roadmap reflect the new contract with historical documents untouched;
|
||||
- final Fable review reports no blocker.
|
||||
@@ -0,0 +1,517 @@
|
||||
# Work package 09 — LaneSnapshot settled-but-unplaced tools
|
||||
|
||||
## Status and baseline
|
||||
|
||||
- Repository: `earendil-works/pi`
|
||||
- Branch at handoff creation: `dev`
|
||||
- Baseline commit: `d14d6b22327d545d6a253f932165b63e48d7f9c8`
|
||||
- The user reported the worktree clean immediately before this handoff.
|
||||
- This document began as an implementation handoff after conversation compaction and now records the implemented design. It is not itself the normative harness specification; `packages/agent/docs/harness.md` remains normative.
|
||||
|
||||
## Goal
|
||||
|
||||
Keep every started or settled-but-unplaced tool call visible in `LaneSnapshot.operation.runningTools` until its immutable `toolResult` entry is placed in the transcript.
|
||||
|
||||
The intended projection is:
|
||||
|
||||
```text
|
||||
planned → not yet represented in runningTools
|
||||
effect_pending → runningTools(status: "running")
|
||||
outcome_ready → runningTools(status: "settled")
|
||||
completed → transcript toolResult entry
|
||||
```
|
||||
|
||||
For each call after it becomes presentation-active, `runningTools` and placed transcript entries must not have a gap or overlap. Placement is a source-prefix flush, not an all-tools barrier. Remove a call from `runningTools` on its own `entry_added`, never on `turn_end`.
|
||||
|
||||
## Original bug
|
||||
|
||||
A call disappeared between real-effect completion and source-ordered tree placement:
|
||||
|
||||
1. `packages/agent/src/harness/runtime/reducer.ts`: `tool_end` spliced the call out of `runningTools`.
|
||||
2. `packages/agent/src/harness/runtime/lane.ts`: `captureLaneSnapshot()`, `case "tools"`, projected only `effect_pending` calls and skipped `outcome_ready` calls.
|
||||
3. The finalized result was already durable at `pendingEntry(resultEntryId)`, but a fresh/reconnected snapshot could not display it.
|
||||
4. It reappeared only after `entry_added` placed the immutable `toolResult` entry.
|
||||
|
||||
For a parallel batch `[A, B, C]`, if B settles while A remains pending, B may stay `outcome_ready` until A is ready. If A is already placed, B can place without waiting for C. Therefore clearing everything at `turn_end` is wrong: early-placed results would temporarily exist in both `transcript` and `runningTools`.
|
||||
|
||||
## Confirmed current architecture
|
||||
|
||||
Mini does **not** replicate structural object deltas.
|
||||
|
||||
- `packages/coding-agent/src/experimental/mini/worker/lane-service.ts` sends one initial full snapshot and then forwards individual `HarnessEvent` objects.
|
||||
- `packages/coding-agent/src/experimental/mini/tui/session.ts` folds those events through `reduceLaneSnapshot()`.
|
||||
- Reconnect/rebase fetches another complete snapshot.
|
||||
- `tool_update` currently carries a complete replacement progress result, not a nested diff.
|
||||
|
||||
Implemented durable tool flow in `packages/agent/src/harness/runtime/drive/tools.ts`:
|
||||
|
||||
```text
|
||||
prepare
|
||||
→ before_tool
|
||||
→ intent commit (effect_pending + effective args), then tool_start
|
||||
→ execute/update/checkpoint
|
||||
→ after_tool
|
||||
→ finalize
|
||||
→ publishToolOutcome staging commit (pendingEntry + outcome_ready), then tool_end
|
||||
→ materializeReady prefix placement
|
||||
→ entry_added
|
||||
```
|
||||
|
||||
All real, immediate synthetic, cancellation, and recovery outcomes converge through `publishToolOutcome()`.
|
||||
|
||||
`Lane.settleOperation()` supports commit-bound events. It commits, publishes process-local state, constructs the event batch, and the public operation awaits delivery. In parallel execution, `materializeReady()` is scheduled only after the outcome-completion promise resolves. Consequently, `tool_end` is delivered after staging and before placement.
|
||||
|
||||
## Agreed event contract change
|
||||
|
||||
Do **not** add `tool_result_ready` or `tool_outcome_ready`.
|
||||
|
||||
Instead, redefine harness `tool_start`/`tool_end` as tool-call processing/result lifecycle events rather than exclusively real external-effect lifecycle events.
|
||||
|
||||
### Fresh executed call
|
||||
|
||||
```text
|
||||
intent commit
|
||||
→ tool_start
|
||||
→ tool_update*
|
||||
→ execute/finalize
|
||||
→ TX[pendingEntry + outcome_ready + cleanup]
|
||||
→ tool_end
|
||||
→ source-ordered placement
|
||||
→ entry_added
|
||||
```
|
||||
|
||||
### Fresh synthetic call
|
||||
|
||||
```text
|
||||
TX[pendingEntry + outcome_ready]
|
||||
→ tool_start
|
||||
→ tool_end
|
||||
→ source-ordered placement
|
||||
→ entry_added
|
||||
```
|
||||
|
||||
The staging transaction's post-commit event batch contains `tool_start` followed by `tool_end`, so a watcher cannot observe either lifecycle event without the authoritative staged state.
|
||||
|
||||
Fresh synthetic calls include:
|
||||
|
||||
- unknown tool;
|
||||
- argument preparation or validation failure;
|
||||
- `before_tool` denial or invalid replacement arguments;
|
||||
- genuine assistant `length`/truncated call handling;
|
||||
- cancellation while still `planned` or after intent but before effect admission.
|
||||
|
||||
### Recovery
|
||||
|
||||
Historical lifecycle events are not replayed.
|
||||
|
||||
- A restored `effect_pending` call is already represented by the initial snapshot.
|
||||
- Safe replay emits recovery-tagged `tool_start` from the checkpoint-clear commit and `tool_end` from the later outcome-staging commit.
|
||||
- Unsafe interruption synthesis may emit a recovery-tagged `tool_end` without a newly emitted `tool_start`; the initial snapshot supplied the running row.
|
||||
- A call already restored as `outcome_ready` appears as settled in the initial snapshot and needs no replayed end event before placement.
|
||||
|
||||
### Meaning of `tool_end`
|
||||
|
||||
After this change, `tool_end` means:
|
||||
|
||||
> The complete final tool result is durably staged and the call is `outcome_ready`.
|
||||
|
||||
It becomes the authoritative reducer transition from running to settled. It must be emitted **after** the staging commit, not before it.
|
||||
|
||||
The old distinction between actually executed and synthetic results was encoded by omitting lifecycle events. There is no in-repo runtime consumer requiring that distinction. If preserving it is desired, discuss adding an explicit field such as `execution: "executed" | "synthetic"`; this field was discussed but **not agreed**, so do not add it silently.
|
||||
|
||||
## LaneSnapshot type
|
||||
|
||||
Change `LaneSnapshot.operation.runningTools` in `packages/agent/src/harness/agent-harness.ts` to use one `result` field for both progress and final output. Do not retain a separate `partialResult` field in the snapshot.
|
||||
|
||||
Prefer a discriminated union so invalid combinations are unrepresentable:
|
||||
|
||||
```ts
|
||||
type SnapshotTool =
|
||||
| {
|
||||
status: "running";
|
||||
toolCallId: string;
|
||||
toolName: string;
|
||||
args: unknown;
|
||||
result?: AgentToolResult<unknown>; // latest complete progress snapshot
|
||||
}
|
||||
| {
|
||||
status: "settled";
|
||||
toolCallId: string;
|
||||
toolName: string;
|
||||
args: unknown;
|
||||
result: AgentToolResult<unknown>; // complete finalized result
|
||||
isError: boolean;
|
||||
};
|
||||
```
|
||||
|
||||
The user explicitly agreed to the discriminated union.
|
||||
|
||||
`tool_update.partialResult` may remain named `partialResult` in the event API; the reducer assigns it to the snapshot row's unified `result` field.
|
||||
|
||||
The current mini transport sends semantic events, not structural deltas, so `tool_end` still carries the complete final result even if it equals the latest update. Unifying the snapshot field is still the correct state model.
|
||||
|
||||
## Exact reducer behavior
|
||||
|
||||
File: `packages/agent/src/harness/runtime/reducer.ts`
|
||||
|
||||
### `tool_start`
|
||||
|
||||
- Resolve the operation with `matchingOperation(snapshot, event.runId)`.
|
||||
- Upsert by `toolCallId`; do not blindly push.
|
||||
- Set `status: "running"`, `toolName`, `args`.
|
||||
- Clear stale settled-only fields if replacing an existing row.
|
||||
- Preserve no stale final result. A newly started/replayed call may receive later `tool_update` values.
|
||||
|
||||
Upsert is required because a watch may capture durable `effect_pending` state before the buffered `tool_start` event is delivered.
|
||||
|
||||
### `tool_update`
|
||||
|
||||
- Use `matchingOperation(snapshot, event.runId)`, not `snapshot.operation` directly.
|
||||
- Find the matching row.
|
||||
- For a running row, replace `result` with `event.partialResult`.
|
||||
- Ignore wrong-operation or missing rows.
|
||||
|
||||
### `tool_end`
|
||||
|
||||
- Resolve with `matchingOperation(snapshot, event.runId)`.
|
||||
- Find the existing row by batch-local `toolCallId`; only one tool batch is presentation-active at a time.
|
||||
- Replace it with `status: "settled"`, preserving its arguments and using `event.result` and `event.isError`.
|
||||
- This naturally removes the provisional interpretation of the old `result`; there is no separate `partialResult` to delete.
|
||||
- The finalized result remains displayed until placement.
|
||||
|
||||
`tool_end` does not carry arguments and cannot create a row. Fresh synthetic `tool_start` and `tool_end` are emitted together after the staging commit, eliminating the old capture/event gap. Unsafe recovery relies on the initial snapshot's running row.
|
||||
|
||||
### `entry_added`
|
||||
|
||||
If `event.entry` is a message whose role is `toolResult`, remove the matching batch-local `toolCallId` from `snapshot.operation?.runningTools`, then apply the transcript update. Harness events are serialized, trusted, emitted exactly once, and not historically replayed, so neither duplicate-entry handling nor cross-batch identity is needed.
|
||||
|
||||
Do not clear tool rows on `turn_end`.
|
||||
|
||||
## Exact authoritative capture behavior
|
||||
|
||||
File: `packages/agent/src/harness/runtime/lane.ts`, `captureLaneSnapshot()`, `case "tools"`.
|
||||
|
||||
The assistant entry is already loaded once. For each batch call:
|
||||
|
||||
### `planned`
|
||||
|
||||
Skip. It has not become presentation-active yet.
|
||||
|
||||
### `completed`
|
||||
|
||||
Skip. Its `toolResult` entry must already be in the captured transcript.
|
||||
|
||||
### `effect_pending`
|
||||
|
||||
- Validate that `assistant.message.content[sourceIndex]` is the matching `toolCall` block.
|
||||
- Read `operationToolArgs(operationId, turnId, sourceIndex)`; it is required for effect-pending calls.
|
||||
- Read optional `pendingToolOutput(operationId, resultEntryId)`.
|
||||
- Project:
|
||||
|
||||
```ts
|
||||
{
|
||||
status: "running",
|
||||
toolCallId: block.id,
|
||||
toolName: block.name,
|
||||
args: persistedArgs,
|
||||
...(checkpoint === undefined ? {} : { result: checkpoint })
|
||||
}
|
||||
```
|
||||
|
||||
### `outcome_ready`
|
||||
|
||||
- Validate the source tool-call block.
|
||||
- Read `pendingEntry(call.resultEntryId)`.
|
||||
- Require a message payload with role `toolResult`.
|
||||
- Validate staged `toolCallId` and `toolName` against the source block.
|
||||
- Read `operationToolArgs(...)` when present.
|
||||
- Use `persistedArgs ?? block.arguments`. Immediate synthetic calls may never have written `operationToolArgs`, and this absence is legal only for the outcome-ready projection.
|
||||
- Reconstruct the canonical `AgentToolResult` from the staged `ToolResultMessage` and the durable call termination flag as needed.
|
||||
- Project `status: "settled"`, `result`, and `isError`.
|
||||
|
||||
The event and capture representations must normalize the final result identically so folding through `tool_end` equals a later authoritative snapshot. Pay attention to optional `details`, `usage`, `addedToolNames`, and `terminate`; do not rely on incidental object-property presence differences.
|
||||
|
||||
A missing or mismatched staged result for `outcome_ready` is presentation corruption and must fault snapshot capture.
|
||||
|
||||
## Runtime event production changes
|
||||
|
||||
Primary file: `packages/agent/src/harness/runtime/drive/tools.ts`
|
||||
|
||||
Related helpers: `packages/agent/src/harness/execution/tools.ts` and `packages/agent/src/harness/runtime/drive/tool-placement.ts`.
|
||||
|
||||
### Internal outcome shape
|
||||
|
||||
Current:
|
||||
|
||||
```ts
|
||||
type ToolOutcome = { message: ToolResultMessage<unknown>; terminate: boolean };
|
||||
```
|
||||
|
||||
Extend/refactor it so post-commit event production has the complete canonical final result and `isError`, without lossy reconstruction. It must retain enough data for:
|
||||
|
||||
- staged `ToolResultMessage`;
|
||||
- `tool_end.result`;
|
||||
- `tool_end.isError`;
|
||||
- durable/effective `terminate` after cancellation normalization.
|
||||
|
||||
Synthetic helpers currently return `ToolResultMessage` directly. Refactor carefully so synthetic outcomes also carry the canonical result data. Do not invent `details` in the transcript: existing unknown/invalid synthetic results deliberately omit message details.
|
||||
|
||||
### Commit-bound `tool_start`
|
||||
|
||||
For fresh execution, `publishToolIntent()` attaches `tool_start` to the commit that persists effective arguments and changes the call to `effect_pending`. The public operation awaits delivery before admitting `executeToolCall()`, preserving `tool_start → tool_update*` without requiring each update callback to await delivery.
|
||||
|
||||
For a fresh synthetic call that never writes effect intent, `publishToolOutcome()` attaches `tool_start` before `tool_end` in the outcome-staging commit's event batch. It reports the source block arguments.
|
||||
|
||||
For safe recovery, the checkpoint-clear commit emits recovery-tagged `tool_start` using persisted effective arguments. Do not emit a fresh start for an already-restored unsafe `effect_pending` call; its initial snapshot is the baseline.
|
||||
|
||||
### Post-commit `tool_end`
|
||||
|
||||
Remove the current pre-staging `tool_end` emission from `performToolInvocation()`.
|
||||
|
||||
`publishToolOutcome()` attaches `tool_end` to the same staging command's `events` callback. The event carries:
|
||||
|
||||
- `runId`, `turnId`, `toolCallId`, `toolName`;
|
||||
- canonical final `result`;
|
||||
- `isError`;
|
||||
- cancellation-normalized durable `terminate`;
|
||||
- `recovery: true` where applicable.
|
||||
|
||||
Arguments belong to `tool_start` and are not repeated on `tool_end`. Event data describes the state actually committed, especially cancellation forcing `terminate: false`.
|
||||
|
||||
Because `Lane.command()` awaits retained event delivery and `runParallel()` schedules materialization from the outcome-completion promise, the required order is:
|
||||
|
||||
```text
|
||||
staging commit
|
||||
→ tool_end delivery
|
||||
→ source-ready message lifecycle
|
||||
→ placement commit
|
||||
→ entry_added
|
||||
```
|
||||
|
||||
### Call sites to audit
|
||||
|
||||
Every `publishToolOutcome()` call must supply the source tool call and recovery context correctly:
|
||||
|
||||
- immediate outcome in `startToolInvocation()`;
|
||||
- cancellation after intent but before execution;
|
||||
- normal `performToolInvocation()` completion;
|
||||
- safe replay completion;
|
||||
- unsafe recovery interruption;
|
||||
- sequential cancellation of `planned`;
|
||||
- sequential cancellation of `effect_pending`.
|
||||
|
||||
Also audit the synchronous `AbortRequested` path inside `performToolInvocation()`: a call with durable intent must still have coherent start/end presentation even if effect admission fails immediately.
|
||||
|
||||
## Mini and other presentation consumers
|
||||
|
||||
### Mini
|
||||
|
||||
File: `packages/coding-agent/src/experimental/mini/tui/view.ts`, `MiniTui.apply()`.
|
||||
|
||||
For each `runningTools` row:
|
||||
|
||||
```text
|
||||
status running:
|
||||
markExecutionStarted()
|
||||
if result exists: updateResult({...result, isError:false}, true)
|
||||
|
||||
status settled:
|
||||
do not call markExecutionStarted()
|
||||
updateResult({...result, isError}, false)
|
||||
```
|
||||
|
||||
The final result remains visible while awaiting placement. After `entry_added`, transcript synchronization supplies the immutable `ToolResultMessage` and the row is no longer in `runningTools`.
|
||||
|
||||
### Other shared consumer
|
||||
|
||||
Apply equivalent handling in:
|
||||
|
||||
- `packages/coding-agent/src/experimental/client-tui-chat.ts`
|
||||
|
||||
Read `packages/coding-agent/src/modes/interactive/components/tool-execution.ts` before editing to confirm `updateResult(result, isPartial)` semantics.
|
||||
|
||||
## Tests
|
||||
|
||||
### Reducer tests
|
||||
|
||||
File: `packages/agent/test/harness/runtime/reducer.test.ts`
|
||||
|
||||
Add a parallel batch event-fold test with calls 0, 1, 2:
|
||||
|
||||
1. Establish all three as running.
|
||||
2. Call 2 settles before call 0.
|
||||
3. Call 0 settles and is placed while call 1 still runs.
|
||||
4. Call 1 settles.
|
||||
5. Placement flushes calls 1 and 2 in source order.
|
||||
6. After every settlement and placement event, assert each presentation-active call appears in exactly one of:
|
||||
- `operation.runningTools`; or
|
||||
- transcript `toolResult` entries.
|
||||
7. Assert call 2 remains present with `status:"settled"` and its final result while blocked by earlier calls.
|
||||
8. Assert each `entry_added` removes only its matching active row.
|
||||
|
||||
Add focused coverage for:
|
||||
|
||||
- `tool_update` from a stale/wrong `runId` does not mutate the current operation;
|
||||
- `tool_start` upserts rather than duplicates a row captured from durable intent;
|
||||
- `tool_end` settles the existing batch-local row.
|
||||
|
||||
### Capture/watch tests
|
||||
|
||||
File: `packages/agent/test/harness/runtime/watch.test.ts`
|
||||
|
||||
Construct a durable tools state containing:
|
||||
|
||||
- `planned` (omitted);
|
||||
- `effect_pending` with checkpoint (`status:"running"`, checkpoint exposed as `result`);
|
||||
- `effect_pending` without checkpoint;
|
||||
- real `outcome_ready` with persisted effective args;
|
||||
- synthetic `outcome_ready` without `operationToolArgs`, falling back to source block arguments;
|
||||
- completed call represented only by a transcript entry where practical.
|
||||
|
||||
Assert staged settled content, `isError`, arguments, and absence of duplicates. Add missing/mismatched `pendingEntry` corruption assertions.
|
||||
|
||||
### Runtime tool tests
|
||||
|
||||
File: `packages/agent/test/harness/runtime/drive-tools.test.ts`
|
||||
|
||||
Update/add ordering assertions proving:
|
||||
|
||||
- real: `intent commit < tool_start < tool_update* < staging commit < tool_end < entry_added`;
|
||||
- immediate synthetic: `staging commit < tool_start < tool_end < entry_added`, no tool effect and no `after_tool`;
|
||||
- planned cancellation gets coherent start/end;
|
||||
- unsafe recovery uses initial snapshot plus recovery-tagged end without replaying effects;
|
||||
- B can emit post-commit end and remain settled while A blocks placement;
|
||||
- source-order placement remains unchanged;
|
||||
- a crash after staging cannot replay the call.
|
||||
|
||||
The baseline normative tests/documentation required `tool_end` before staging; the implementation reverses those expectations deliberately.
|
||||
|
||||
### Type/event catalog tests
|
||||
|
||||
Audit:
|
||||
|
||||
- `packages/agent/test/harness/types.test.ts`
|
||||
- `packages/agent/src/harness/telemetry.ts`
|
||||
|
||||
No new event name is added. `tool_end` omits arguments and its semantics change.
|
||||
|
||||
### Mini regression
|
||||
|
||||
After unit tests, run the real mini abort smoke test used previously:
|
||||
|
||||
- execute exactly `sleep 20`;
|
||||
- send Escape after about two seconds;
|
||||
- assert `Command aborted` and elapsed time appear;
|
||||
- assert raw ANSI contains `toolErrorBg` (`48;2;60;40;40`);
|
||||
- verify the durable session contains an `isError:true` tool result and operation status `aborted`.
|
||||
|
||||
Also exercise a parallel batch where a later tool finishes first and verify its final result remains visible until in-order placement.
|
||||
|
||||
## Documentation changes
|
||||
|
||||
Read both documents completely before editing:
|
||||
|
||||
- `packages/agent/docs/harness.md` (normative)
|
||||
- `packages/agent/docs/tool-durability.md`
|
||||
|
||||
The implementation updates baseline statements that required:
|
||||
|
||||
- `tool_end` before staging;
|
||||
- `tool_start`/`tool_end` only for real effects;
|
||||
- synthetic outcomes emitting no lifecycle;
|
||||
- `outcome_ready` being omitted from `runningTools`;
|
||||
- snapshot field `partialResult`.
|
||||
|
||||
Important known locations from the baseline:
|
||||
|
||||
- `harness.md` §3.8 around lines 784–796;
|
||||
- `harness.md` §5.4 `LaneSnapshot` around lines 1101–1134;
|
||||
- `harness.md` §5.5 events around lines 1140–1158;
|
||||
- `harness.md` tool phases around lines 1214–1224;
|
||||
- `harness.md` conformance requirements around lines 1386–1400;
|
||||
- `tool-durability.md` finalization around lines 255–280;
|
||||
- `tool-durability.md` snapshots/events around lines 568–584;
|
||||
- `tool-durability.md` test requirements around lines 671–679.
|
||||
|
||||
The revised docs must state:
|
||||
|
||||
- `tool_end` is post-staging durability evidence for a final result;
|
||||
- fresh synthetic outcomes receive start/end lifecycle;
|
||||
- recovery events are not historically replayed;
|
||||
- `outcome_ready` remains projected as settled until placement;
|
||||
- `entry_added` moves settled presentation into transcript;
|
||||
- the unified snapshot `result` is provisional when running and final when settled.
|
||||
|
||||
Do not modify the separate `response.ts`/`tool-placement.ts` recovery `turn_end` discrepancy unless separately requested.
|
||||
|
||||
## Files to reread completely after compaction
|
||||
|
||||
Core/specification:
|
||||
|
||||
1. `packages/agent/docs/harness.md`
|
||||
2. `packages/agent/docs/tool-durability.md`
|
||||
3. `packages/agent/src/harness/agent-harness.ts`
|
||||
4. `packages/agent/src/harness/runtime/reducer.ts`
|
||||
5. `packages/agent/src/harness/runtime/lane.ts`
|
||||
6. `packages/agent/src/harness/runtime/drive/tools.ts`
|
||||
7. `packages/agent/src/harness/runtime/drive/tool-placement.ts`
|
||||
8. `packages/agent/src/harness/execution/tools.ts`
|
||||
9. `packages/agent/src/harness/runtime/types.ts`
|
||||
10. `packages/agent/src/harness/session/types.ts`
|
||||
11. `packages/agent/src/harness/events.ts`
|
||||
12. `packages/agent/src/harness/telemetry.ts`
|
||||
|
||||
Tests:
|
||||
|
||||
13. `packages/agent/test/harness/runtime/reducer.test.ts`
|
||||
14. `packages/agent/test/harness/runtime/watch.test.ts`
|
||||
15. `packages/agent/test/harness/runtime/drive-tools.test.ts`
|
||||
16. `packages/agent/test/harness/types.test.ts`
|
||||
17. Relevant test helpers imported by those files.
|
||||
|
||||
Mini/presentation:
|
||||
|
||||
18. `packages/coding-agent/src/experimental/mini/tui/session.ts`
|
||||
19. `packages/coding-agent/src/experimental/mini/worker/lane-service.ts`
|
||||
20. `packages/coding-agent/src/experimental/mini/shared/protocol.ts`
|
||||
21. `packages/coding-agent/src/experimental/mini/tui/view.ts`
|
||||
22. `packages/coding-agent/src/experimental/client-tui-chat.ts`
|
||||
23. `packages/coding-agent/src/modes/interactive/components/tool-execution.ts`
|
||||
|
||||
Before editing, run `git status --short` and inspect current diffs because other Pi sessions may share the worktree.
|
||||
|
||||
## Validation commands
|
||||
|
||||
From repository root, after changes:
|
||||
|
||||
```bash
|
||||
cd packages/agent
|
||||
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run test/harness/runtime/reducer.test.ts
|
||||
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run test/harness/runtime/watch.test.ts
|
||||
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run test/harness/runtime/drive-tools.test.ts
|
||||
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run test/harness/types.test.ts
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
npm run check
|
||||
```
|
||||
|
||||
Do not run `npm test`, the full Vitest suite, or `npm run build` unless requested.
|
||||
|
||||
If a delegated review is used, repository policy requires:
|
||||
|
||||
```text
|
||||
--provider anthropic --model claude-fable-5
|
||||
```
|
||||
|
||||
Keep extensions enabled.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No generic structural-delta transport for mini.
|
||||
- No optimization to avoid the final result crossing the wire once in `tool_end` and again in `entry_added`.
|
||||
- No change to source-prefix placement semantics.
|
||||
- No clearing on `turn_end`.
|
||||
- No change to tool effect replay/durability rules.
|
||||
- No change to `after_tool`: it still runs only for actual fresh/safely replayed effects under its existing cancellation contract.
|
||||
- No work on the separate recovery `turn_end` discrepancy between `response.ts` and `tool-placement.ts`.
|
||||
- No backward-compatibility layer unless the user explicitly asks for one.
|
||||
- Do not commit unless the user asks.
|
||||
Reference in New Issue
Block a user