alexion/dev
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore(vendor): import pi 1.3.0

Browse Source 
Imported as a linear vendor commit so forge rebases do not replay the upstream root tree.

git-subtree-dir: third_party/pi

git-subtree-split: 71023aec1546d6d677a65669c5665c0927ce0dd0
This commit is contained in:
alexion
2026-10-15 17:58:48 -04:00
parent 719370c8f7
commit 6081e3e325
1743 changed files with 205460 additions and 131705 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
third_party/pi/packages/env/CHANGELOG.md vendored Normal file
View File

@@ -0,0 +1,22 @@
# Changelog
## [1.3.0] - 2026-10-15
## [1.2.0] - 2026-10-12
### Added
- The daemon's `listFiles` operation: a bounded, cancellable, gitignore-aware listing of a directory tree (files, directories and unfollowed symlinks); `hello` reports it in `features`, and older daemons answer `not_supported` without a request.
- `RemoteExecutionEnv.listFiles` and `LocalExecutionEnv`, a local `ExecutionEnv` on a pi-env daemon connection with the same operations.
- `nodePlatform(info)` maps the daemon's platform and architecture names to Node's.
- Pseudo-terminals for interactive programs: the daemon's `pty`, `ptyWrite`, and `ptyResize` operations stream raw terminal output as events, write input in arrival order, send `SIGWINCH` on resize, and kill the process group on cancel or a lost connection. `openTerminal()` returns a `RemoteTerminal`, and `localDaemon()` starts the daemon for this machine. Unix only; Windows answers `ENOTSUP`.
- `RemoteTerminal.foreground()` (daemon `ptyForeground`) reports the process group in front of a terminal when it is not the program's own, such as a job its shell runs, with the leader's name on macOS and Linux (`TerminalForeground`).
## [1.1.0] - 2026-10-07
## [1.0.4] - 2026-10-05
### Added
- Initial release of `@earendil-works/pi-env`: `RemoteExecutionEnv`, a Durable `ExecutionEnv` on another machine, and `Connection` to the `pi-env` daemon that runs there. Results match `NodeExecutionEnv` on that machine on Linux, macOS, Android and Windows. The daemon watches files natively (polling on Windows and network file systems), honors output windows so dropped command output is not transferred, and keeps control replies ahead of bulk output; watchers report `overflow` and resume after a lost connection.
- SSH bootstrap: `connectSsh` detects the remote system, deploys the daemon this package ships for it (named by its SHA-256 and verified before every start), and connects; `sshConnection` does the same lazily at the first operation, so building an environment never needs the network without forwarding, shared connections or configured commands; `loginShell` starts it through the user's login shell, and Termux hosts get setup warnings. `scanHostKey`, `acceptHostKey` and `forgetHostKey` manage trusted host keys; a changed key is never accepted without removing the old one.

66
third_party/pi/packages/env/README.md vendored Normal file
View File

@@ -0,0 +1,66 @@
# @earendil-works/pi-env
Remote execution environments for [Pi Durable](../durable): an agent's tools run on another machine, usually over SSH,
while the Durable worker, its storage and credentials stay local.
- `pi-env` (`daemon/`): a small Rust program that runs on the remote machine. It speaks a framed protocol on stdin and
stdout ([docs/protocol.md](docs/protocol.md)) and performs file operations and commands there.
- `RemoteExecutionEnv`: a Durable `ExecutionEnv` that talks to the daemon through a `Connection`. Its results match
`NodeExecutionEnv` running on the remote machine; only error messages may differ
([docs/semantics.md](docs/semantics.md)).
```ts
import { acceptHostKey, connectSsh, RemoteExecutionEnv, scanHostKey, sshConnection } from "@earendil-works/pi-env";
const target = { host: "gpu-box", knownHostsFile: "/data/ssh/known_hosts", hostKeyAlias: "pi-env-gpu" };
// Once: show the host's key fingerprint to the owner, who compares it out of band and accepts it.
const { lines, fingerprints } = await scanHostKey(target);
await acceptHostKey(target, lines);
// Detects the remote system, deploys the daemon if missing (named by its SHA-256 and verified before every start),
// and returns a connection that starts it over ssh.
const { connection } = await connectSsh(target);
const env = new RemoteExecutionEnv({ connection, id: "pi-env:gpu", cwd: "/home/me/project" });
// Or lazily: nothing happens until the first operation, which detects, deploys and connects. A failure (no network,
// untrusted host key) is that operation's error, and the next operation tries again.
const lazy = sshConnection(target);
const lazyEnv = new RemoteExecutionEnv({ connection: lazy.connection, id: "pi-env:gpu", cwd: "/home/me/project" });
```
The package ships the daemon for every supported remote system in `bin/`. `ssh` runs with `BatchMode`, strict host-key
checking against the application's own known-hosts file under a fixed alias, no forwarding of any kind, no shared
connections or configured commands, and without forwarding the local locale. On Windows, detection and deployment go
through PowerShell, and the daemon starts through the server's default shell (cmd.exe or PowerShell).
`env.listFiles(path, options, context)` lists the tree below a directory in the daemon, as `fd --hidden --exclude .git`
does without following symbolic links: files, directories and links as relative `/` paths, skipping what git ignores
inside work trees, bounded by `maxEntries`, `maxBytes` and `timeoutMs` with a `truncated` flag, and stopped by aborting
the context. It is Durable's optional `ExecutionEnv.listFiles`. A daemon released before it reports `not_supported`
(`hello.features` lacks `listFiles`), and callers fall back to their own walk. For local sessions,
`LocalExecutionEnv` is `NodeExecutionEnv` plus `listFiles` through a daemon on the same machine:
```ts
const local = new Connection({ command: [localDaemon()] }); // owned by the caller; close() it at shutdown
const env = new LocalExecutionEnv({ cwd: "/home/me/project", connection: local });
const listing = await env.listFiles(".", { exclude: ["node_modules"] }, context);
// { ok: true, value: { entries: [{ path: "src", kind: "directory" }, { path: "src/a.ts", kind: "file" }], truncated } }
```
`nodePlatform(await connection.info())` gives the remote system in Node's names (`win32`, `darwin`, `x64`, `arm64`),
as `process.platform` and `process.arch` spell them.
`openTerminal(connection, options)` runs a program, by default the user's login shell, in a pseudo-terminal of the
daemon: raw output, input, resizing and its exit code, for interactive terminals. `localDaemon()` is the daemon for this
machine, for terminals without SSH (`new Connection({ command: [localDaemon()] })`). Terminals are not supported on
Windows yet.
Supported remote systems: Linux, macOS, Android (Termux) and Windows, on x86-64 and arm64. On Windows, string commands
run through Git Bash as `NodeExecutionEnv` runs them there; argv commands run directly.
## Development
`npm run build:daemon` builds the daemon with Cargo; the tests talk to `daemon/target/debug/pi-env` over a pipe, or to
the binary named by `PI_ENV_DAEMON` (CI tests the release builds this way). They run Durable's env conformance suite
against `RemoteExecutionEnv` and compare random operation sequences and Durable's tools against `NodeExecutionEnv` on
the same machine. `test/ssh-external.test.ts` runs the suite over a real SSH server when `PI_ENV_SSH_HOST` is set.

479
third_party/pi/packages/env/daemon/Cargo.lock generated vendored Normal file
View File

@@ -0,0 +1,479 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "aho-corasick"
version = "1.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba"
dependencies = [
"memchr",
]
[[package]]
name = "bitflags"
version = "2.13.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3ded4057c258ba199e2d26386d3af3780957ecaee6c4ef4041c6b4b8b97c0b06"
[[package]]
name = "bstr"
version = "1.13.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6bb31b46c14244e20ee9984b11bf5c992b91fb6939fea616e3512c8baecdbe5f"
dependencies = [
"memchr",
"serde_core",
]
[[package]]
name = "cfg-if"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600"
[[package]]
name = "core_detect"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7f8f80099a98041a3d1622845c271458a2d73e688351bf3cb999266764b81d48"
[[package]]
name = "crossbeam-deque"
version = "0.8.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "622f3fc73690be383c7214310406f28a90e6edeadc3cea882f9d71e495b9711a"
dependencies = [
"crossbeam-epoch",
"crossbeam-utils",
]
[[package]]
name = "crossbeam-epoch"
version = "0.9.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dc74980687109a3b14c72fd458107bf0baa1da1a1a805e178d15501ba9b86d9d"
dependencies = [
"crossbeam-utils",
]
[[package]]
name = "crossbeam-utils"
version = "0.8.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a31eee39dddec8330830986fcd7625edb5a24ec90ea038215273bbc3adb08ac6"
[[package]]
name = "encoding_rs"
version = "0.8.42"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e985e0451871ad22fb8d2b6b076e2028a502a0d3950998c2c5c0a4f9b5d9679"
dependencies = [
"cfg-if",
"core_detect",
"multiversion_no_op",
"rustversion",
"scopeguard",
"simdutf8",
]
[[package]]
name = "fsevent-sys"
version = "4.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "76ee7a02da4d231650c7cea31349b889be2f45ddb3ef3032d2ec8185f6313fd2"
dependencies = [
"libc",
]
[[package]]
name = "globset"
version = "0.4.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "07c34a9410465b45bd9787443bc7370f37735bad04b0f0cd57ff1a3186c98988"
dependencies = [
"aho-corasick",
"bstr",
"log",
"regex-automata",
"regex-syntax",
]
[[package]]
name = "ignore"
version = "0.4.33"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "00b69833ed729dc5aa7d19541d96d6cf8e9137194207a04916d658e43168402f"
dependencies = [
"crossbeam-deque",
"globset",
"log",
"memchr",
"regex-automata",
"same-file",
"walkdir",
"winapi-util",
]
[[package]]
name = "inotify"
version = "0.11.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4cc00ea907cab49550b7da656f80ebb97be1b997d931fbcd28d39734e17ce592"
dependencies = [
"bitflags",
"inotify-sys",
"libc",
]
[[package]]
name = "inotify-sys"
version = "0.1.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c033f80b2c113cdf91ab7a33faa9cbc014726dcad99880c8609af2a370edf37d"
dependencies = [
"libc",
]
[[package]]
name = "itoa"
version = "1.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
[[package]]
name = "kqueue"
version = "1.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8d763e5b24120b4ddf50de6c92308156765aabfbbccebf401da7cff2d70a41ea"
dependencies = [
"kqueue-sys",
"libc",
]
[[package]]
name = "kqueue-sys"
version = "1.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "07293a4e297ac234359b510362495713f75ea345d5307140414f20c69ffeb087"
dependencies = [
"bitflags",
"libc",
]
[[package]]
name = "libc"
version = "0.2.190"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ce5d3ddc6d3fa000eb1536d85e147bfe31aacaba692ed6a876f95cb7c855be78"
[[package]]
name = "log"
version = "0.4.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6"
[[package]]
name = "memchr"
version = "2.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
[[package]]
name = "mio"
version = "1.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1788edb87fdc09c7e26304471e2f5be8cdefb1b6930d6e3985fc02ff53bf86ee"
dependencies = [
"libc",
"log",
"wasi",
"windows-sys 0.61.2",
]
[[package]]
name = "multiversion_no_op"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "743fb55ba31b18fb1ecef6bdc9aa2743314978ac084044301a7eee33fb99a20d"
[[package]]
name = "notify"
version = "8.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4d3d07927151ff8575b7087f245456e549fea62edf0ec4e565a5ee50c8402bc3"
dependencies = [
"bitflags",
"fsevent-sys",
"inotify",
"kqueue",
"libc",
"log",
"mio",
"notify-types",
"walkdir",
"windows-sys 0.60.2",
]
[[package]]
name = "notify-types"
version = "2.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "42b8cfee0e339a0337359f3c88165702ac6e600dc01c0cc9579a92d62b08477a"
dependencies = [
"bitflags",
]
[[package]]
name = "pi-env"
version = "0.1.0"
dependencies = [
"encoding_rs",
"ignore",
"libc",
"notify",
"serde_json",
"windows-sys 0.61.2",
]
[[package]]
name = "proc-macro2"
version = "1.0.107"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9"
dependencies = [
"unicode-ident",
]
[[package]]
name = "quote"
version = "1.0.47"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001"
dependencies = [
"proc-macro2",
]
[[package]]
name = "regex-automata"
version = "0.4.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2"
dependencies = [
"aho-corasick",
"memchr",
"regex-syntax",
]
[[package]]
name = "regex-syntax"
version = "0.8.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4"
[[package]]
name = "rustversion"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f"
[[package]]
name = "same-file"
version = "1.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502"
dependencies = [
"winapi-util",
]
[[package]]
name = "scopeguard"
version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49"
[[package]]
name = "serde"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"
dependencies = [
"serde_core",
]
[[package]]
name = "serde_core"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48"
dependencies = [
"serde_derive",
]
[[package]]
name = "serde_derive"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "serde_json"
version = "1.0.151"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14"
dependencies = [
"itoa",
"memchr",
"serde",
"serde_core",
"zmij",
]
[[package]]
name = "simdutf8"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e"
[[package]]
name = "syn"
version = "3.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8593e8e72159ed2257d083c7a454a85cbf854f37a0966d8d483aff8c8a3ebcee"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "unicode-ident"
version = "1.0.26"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d245f478577f809a851594d02313b640fb437e0bb33866753cff937863096954"
[[package]]
name = "walkdir"
version = "2.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b"
dependencies = [
"same-file",
"winapi-util",
]
[[package]]
name = "wasi"
version = "0.11.1+wasi-snapshot-preview1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b"
[[package]]
name = "winapi-util"
version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "windows-link"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
[[package]]
name = "windows-sys"
version = "0.60.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb"
dependencies = [
"windows-targets",
]
[[package]]
name = "windows-sys"
version = "0.61.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc"
dependencies = [
"windows-link",
]
[[package]]
name = "windows-targets"
version = "0.53.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3"
dependencies = [
"windows-link",
"windows_aarch64_gnullvm",
"windows_aarch64_msvc",
"windows_i686_gnu",
"windows_i686_gnullvm",
"windows_i686_msvc",
"windows_x86_64_gnu",
"windows_x86_64_gnullvm",
"windows_x86_64_msvc",
]
[[package]]
name = "windows_aarch64_gnullvm"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53"
[[package]]
name = "windows_aarch64_msvc"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006"
[[package]]
name = "windows_i686_gnu"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3"
[[package]]
name = "windows_i686_gnullvm"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c"
[[package]]
name = "windows_i686_msvc"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2"
[[package]]
name = "windows_x86_64_gnu"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499"
[[package]]
name = "windows_x86_64_gnullvm"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1"
[[package]]
name = "windows_x86_64_msvc"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650"
[[package]]
name = "zmij"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"

30
third_party/pi/packages/env/daemon/Cargo.toml vendored Normal file
View File

@@ -0,0 +1,30 @@
[package]
name = "pi-env"
version = "0.1.0"
edition = "2024"
description = "Remote execution environment daemon for Pi Durable"
publish = false
[dependencies]
encoding_rs = "=0.8.42"
ignore = "=0.4.33"
notify = "=8.2.0"
serde_json = "=1.0.151"
[target.'cfg(unix)'.dependencies]
libc = "=0.2.190"
[target.'cfg(windows)'.dependencies]
windows-sys = { version = "=0.61.2", features = [
"Win32_Foundation",
"Win32_Security",
"Win32_Storage_FileSystem",
"Win32_System_IO",
"Win32_System_JobObjects",
"Win32_System_Threading",
] }
[profile.release]
lto = true
codegen-units = 1
strip = true

12
third_party/pi/packages/env/daemon/build.rs vendored Normal file
View File

@@ -0,0 +1,12 @@
//! Exposes the npm package version as `PI_ENV_VERSION`, so `hello` reports the release the daemon shipped with.
fn main() {
println!("cargo:rerun-if-changed=../package.json");
let package = std::fs::read_to_string("../package.json").expect("read ../package.json");
let version = package
.split("\"version\"")
.nth(1)
.and_then(|rest| rest.split('"').nth(1))
.expect("version in ../package.json");
println!("cargo:rustc-env=PI_ENV_VERSION={version}");
}

3
third_party/pi/packages/env/daemon/rust-toolchain.toml vendored Normal file
View File

@@ -0,0 +1,3 @@
[toolchain]
channel = "1.96.0"
components = ["rustfmt", "clippy"]

83
third_party/pi/packages/env/daemon/src/decode.rs vendored Normal file
View File

@@ -0,0 +1,83 @@
//! WHATWG UTF-8 decoding that matches Node's `new TextDecoder().decode(bytes)` of a whole input when the input arrives
//! in pieces (Durable `env/decode.ts`): invalid sequences become U+FFFD by the maximal-subpart rule, and only a
//! byte-order mark at the very start is dropped.
use encoding_rs::{Decoder, UTF_8};
/// Decodes one stream chunk by chunk, dropping a leading byte-order mark.
pub struct StreamDecoder {
decoder: Decoder,
}
impl StreamDecoder {
pub fn new() -> StreamDecoder {
// Removes a UTF-8 BOM only; plain `new_decoder()` would also sniff UTF-16 BOMs and switch encodings.
StreamDecoder {
decoder: UTF_8.new_decoder_with_bom_removal(),
}
}
/// Text for `bytes`, holding back an incomplete character; with `last`, the end of the stream.
pub fn decode(&mut self, bytes: &[u8], last: bool) -> String {
decode_into(&mut self.decoder, bytes, last)
}
}
/// A decoder for a byte range that does not start at the beginning of its input: no BOM handling at all.
pub fn range_decoder() -> Decoder {
UTF_8.new_decoder_without_bom_handling()
}
pub fn decode_into(decoder: &mut Decoder, bytes: &[u8], last: bool) -> String {
let capacity = decoder
.max_utf8_buffer_length(bytes.len())
.unwrap_or(bytes.len() * 3 + 16);
let mut text = String::with_capacity(capacity);
let (_result, read, _replaced) = decoder.decode_to_string(bytes, &mut text, last);
debug_assert_eq!(read, bytes.len());
text
}
/// Whether decoding the whole input drops its first three bytes as a byte-order mark.
pub fn starts_with_bom(bytes: &[u8]) -> bool {
bytes.starts_with(&[0xef, 0xbb, 0xbf])
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn keeps_a_feff_that_does_not_start_the_stream() {
let mut decoder = StreamDecoder::new();
let bytes = [0xe2, 0x82, 0xef, 0xbb, 0xbf, 0x61];
let text: String = bytes
.iter()
.map(|byte| decoder.decode(&[*byte], false))
.collect::<String>()
+ &decoder.decode(&[], true);
assert_eq!(text, "\u{fffd}\u{feff}a");
}
#[test]
fn drops_only_a_leading_mark_and_never_sniffs_utf16() {
let mut decoder = StreamDecoder::new();
assert_eq!(
decoder.decode(&[0xef, 0xbb], false) + &decoder.decode(&[0xbf, 0x61], true),
"a"
);
let mut decoder = StreamDecoder::new();
assert_eq!(
decoder.decode(&[0xff, 0xfe, 0x41, 0x00], true),
"\u{fffd}\u{fffd}A\u{0}"
);
}
#[test]
fn replaces_an_incomplete_sequence_once() {
let mut decoder = StreamDecoder::new();
assert_eq!(decoder.decode(&[0xe2, 0x82, 0x41], true), "\u{fffd}A");
let mut decoder = StreamDecoder::new();
assert_eq!(decoder.decode(&[0xe2, 0x82], true), "\u{fffd}");
}
}

126
third_party/pi/packages/env/daemon/src/errors.rs vendored Normal file
View File

@@ -0,0 +1,126 @@
//! Errors as Node reports them: a code such as `ENOENT` plus a diagnostic message.
use serde_json::{Value, json};
use std::io;
/// A failure with a Node-style code; messages are diagnostic only.
#[derive(Debug)]
pub struct Failure {
pub code: String,
pub message: String,
details: Option<Box<Details>>,
}
#[derive(Debug, Default)]
struct Details {
syscall: Option<String>,
path: Option<String>,
/// Extra fields for the client, such as `spillPath`.
extra: Option<Value>,
}
impl Failure {
pub fn new(code: &str, message: impl Into<String>) -> Failure {
Failure {
code: code.to_string(),
message: message.into(),
details: None,
}
}
fn details(&mut self) -> &mut Details {
self.details.get_or_insert_with(Default::default)
}
/// Add fields to the error JSON, such as `spillPath`.
pub fn extra(mut self, extra: Value) -> Failure {
self.details().extra = Some(extra);
self
}
/// An I/O error of `syscall` on `path`, with libuv's error name as code.
pub fn io(error: &io::Error, syscall: &str, path: &str) -> Failure {
Failure::io_between(error, syscall, path, None)
}
/// Node's message for an error of a two-path call: `CODE: description, syscall 'from' -> 'to'`.
pub fn io_between(
error: &io::Error,
syscall: &str,
path: &str,
destination: Option<&str>,
) -> Failure {
let code = crate::sys::error_name(error).to_string();
let description = describe(&code).map(str::to_string).unwrap_or_else(|| {
// Unknown to libuv's table: the OS text without Rust's " (os error N)".
let text = error.to_string();
text.split(" (os error")
.next()
.unwrap_or(&text)
.to_lowercase()
});
let target = match destination {
Some(destination) => format!("'{path}' -> '{destination}'"),
None => format!("'{path}'"),
};
let mut failure = Failure::new(&code, format!("{code}: {description}, {syscall} {target}"));
failure.details().syscall = Some(syscall.to_string());
failure.path(path)
}
pub fn path(mut self, path: &str) -> Failure {
self.details().path = Some(path.to_string());
self
}
pub fn to_json(&self) -> Value {
let mut value = json!({ "code": self.code, "message": self.message });
let Some(details) = &self.details else {
return value;
};
if let Some(syscall) = &details.syscall {
value["syscall"] = json!(syscall);
}
if let Some(path) = &details.path {
value["path"] = json!(path);
}
if let Some(Value::Object(extra)) = &details.extra {
for (key, field) in extra {
value[key] = field.clone();
}
}
value
}
}
/// libuv's `uv_strerror` texts, which Node uses on every system.
fn describe(code: &str) -> Option<&'static str> {
Some(match code {
"EACCES" => "permission denied",
"EAGAIN" => "resource temporarily unavailable",
"EBADF" => "bad file descriptor",
"EBUSY" => "resource busy or locked",
"EEXIST" => "file already exists",
"EFBIG" => "file too large",
"EINVAL" => "invalid argument",
"EIO" => "i/o error",
"EISDIR" => "illegal operation on a directory",
"ELOOP" => "too many symbolic links encountered",
"EMFILE" => "too many open files",
"EMLINK" => "too many links",
"ENAMETOOLONG" => "name too long",
"ENFILE" => "file table overflow",
"ENOENT" => "no such file or directory",
"ENOMEM" => "not enough memory",
"ENOSPC" => "no space left on device",
"ENOSYS" => "function not implemented",
"ENOTDIR" => "not a directory",
"ENOTEMPTY" => "directory not empty",
"ENOTSUP" => "operation not supported on socket",
"EPERM" => "operation not permitted",
"EROFS" => "read-only file system",
"ETXTBSY" => "text file is busy",
"EXDEV" => "cross-device link not permitted",
_ => return None,
})
}

483
third_party/pi/packages/env/daemon/src/exec.rs vendored Normal file
View File

@@ -0,0 +1,483 @@
//! `exec`: Durable's `NodeExecutionEnv.exec` semantics (docs/semantics.md), output streamed as `output` events.
use crate::decode::StreamDecoder;
use crate::errors::Failure;
use crate::frame::{EVENT, Frame};
use crate::output::Output;
use crate::sys;
use crate::window::{Pending, Window};
use serde_json::{Map, Value, json};
use std::collections::HashSet;
use std::fs::{File, OpenOptions};
use std::io::Write;
use std::path::Path;
use std::process::{Child, Command, ExitStatus, Stdio};
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::sync::mpsc::{self, RecvTimeoutError, Sender};
use std::sync::{Arc, Mutex};
use std::thread::{self, JoinHandle};
use std::time::{Duration, Instant};
const EXIT_STDIO_GRACE: Duration = Duration::from_millis(100);
/// How often a delivery waiting for the link to drain looks again.
const UNSENT_POLL: Duration = Duration::from_millis(20);
/// Commands this daemon started and has not reaped; killed when the connection ends.
pub type Groups = Arc<Mutex<HashSet<u32>>>;
pub enum Message {
Data(usize, Vec<u8>),
Eof(usize),
Exit(ExitStatus),
WaitFailed(String),
/// Abort: the command settles as `aborted`.
Cancel,
/// Kill without aborting, as `cleanup()` does: the command settles with the killed process's status.
Kill,
}
/// Cancellation of one request, registered before the request runs so a cancel that arrives first is not lost.
#[derive(Default)]
pub struct Control {
pub aborted: AtomicBool,
pub killed: AtomicBool,
sender: Mutex<Option<Sender<Message>>>,
}
impl Control {
pub fn cancel(&self, kill: bool) {
let sender = self.sender.lock().unwrap();
if kill {
self.killed.store(true, Ordering::SeqCst);
} else {
self.aborted.store(true, Ordering::SeqCst);
}
if let Some(sender) = sender.as_ref() {
let _ = sender.send(if kill { Message::Kill } else { Message::Cancel });
}
}
pub fn attach(&self, sender: Sender<Message>) {
*self.sender.lock().unwrap() = Some(sender);
}
}
pub struct ExecRequest {
pub command: Option<String>,
pub argv: Option<Vec<String>>,
pub cwd: String,
pub env: Map<String, Value>,
pub inherit_env: bool,
pub shell_path: Option<String>,
pub timeout: Option<Duration>,
pub spill: Option<(u64, u64)>,
pub window: Option<Window>,
}
impl ExecRequest {
pub fn from_json(json: &Value) -> Result<ExecRequest, Failure> {
let strings = |value: &Value| -> Option<Vec<String>> {
value
.as_array()?
.iter()
.map(|item| item.as_str().map(str::to_string))
.collect()
};
Ok(ExecRequest {
command: json["command"].as_str().map(str::to_string),
argv: strings(&json["argv"]),
cwd: json["cwd"]
.as_str()
.ok_or_else(|| Failure::new("EINVAL", "exec needs cwd"))?
.to_string(),
env: json["env"].as_object().cloned().unwrap_or_default(),
inherit_env: json["inheritEnv"].as_bool().unwrap_or(true),
shell_path: json["shellPath"].as_str().map(str::to_string),
timeout: json["timeoutMs"]
.as_f64()
.map(|ms| Duration::from_secs_f64(ms / 1000.0)),
spill: json["spill"].as_object().and_then(|spill| {
Some((
spill.get("afterBytes")?.as_u64()?,
spill.get("afterLines")?.as_u64()?,
))
}),
window: Window::from_json(&json["window"]),
})
}
}
pub fn kill_group(pid: u32) {
sys::kill_tree(pid);
}
fn random_uuid() -> String {
let mut bytes = sys::random_bytes();
bytes[6] = (bytes[6] & 0x0f) | 0x40;
bytes[8] = (bytes[8] & 0x3f) | 0x80;
let hex: String = bytes.iter().map(|byte| format!("{byte:02x}")).collect();
format!(
"{}-{}-{}-{}-{}",
&hex[0..8],
&hex[8..12],
&hex[12..16],
&hex[16..20],
&hex[20..32]
)
}
/// Spill of the complete raw output, started once it crosses either threshold, like Node's.
struct Spill {
after_bytes: u64,
after_lines: u64,
seen_bytes: u64,
seen_newlines: u64,
prefix: Vec<Vec<u8>>,
file: Option<File>,
path: Option<String>,
failed: Option<String>,
}
impl Spill {
fn push(&mut self, chunk: &[u8], tmpdir: &str) {
if self.failed.is_some() || chunk.is_empty() {
return;
}
if self.file.is_none() {
self.seen_bytes += chunk.len() as u64;
self.seen_newlines += chunk.iter().filter(|byte| **byte == b'\n').count() as u64;
let lines = self.seen_newlines + u64::from(chunk.last() != Some(&b'\n'));
if self.seen_bytes <= self.after_bytes && lines <= self.after_lines {
self.prefix.push(chunk.to_vec());
return;
}
if let Err(error) = self.start(tmpdir) {
self.failed = Some(error);
return;
}
}
if let Some(file) = &mut self.file
&& let Err(error) = file.write_all(chunk)
{
self.failed = Some(error.to_string());
}
}
/// `createTempFile({ prefix: "pi-output-", suffix: ".log" })`: a fresh `tmp-` directory holding the file.
fn start(&mut self, tmpdir: &str) -> Result<(), String> {
let prefix = Path::new(tmpdir)
.join("tmp-")
.to_string_lossy()
.into_owned();
let directory = sys::mkdtemp(&prefix).map_err(|error| error.to_string())?;
let path = Path::new(&directory)
.join(format!("pi-output-{}.log", random_uuid()))
.to_string_lossy()
.into_owned();
let mut file = OpenOptions::new()
.append(true)
.create(true)
.open(&path)
.map_err(|error| error.to_string())?;
for chunk in self.prefix.drain(..) {
file.write_all(&chunk).map_err(|error| error.to_string())?;
}
self.file = Some(file);
self.path = Some(path);
Ok(())
}
}
fn spawn(
request: &ExecRequest,
program: &str,
args: &[String],
piped_stdin: bool,
) -> std::io::Result<Child> {
let mut command = Command::new(program);
command
.current_dir(&request.cwd)
.stdin(if piped_stdin {
Stdio::piped()
} else {
Stdio::null()
})
.stdout(Stdio::piped())
.stderr(Stdio::piped());
sys::configure(&mut command, args, &request.env, request.inherit_env);
let child = command.spawn()?;
sys::after_spawn(&child);
Ok(child)
}
/// Read one pipe on its own thread. Without an output window, reading waits while too much output is unsent.
fn spawn_reader(
stream: usize,
pipe: impl sys::Pipe + Send + 'static,
sender: Sender<Message>,
stop: Arc<AtomicBool>,
output: Option<Arc<Output>>,
) -> JoinHandle<()> {
thread::spawn(move || {
sys::read_pipe(pipe, &stop, |bytes| {
let delivered = sender.send(Message::Data(stream, bytes.to_vec())).is_ok();
if let Some(output) = &output {
output.wait_for_room(&stop);
}
delivered
});
let _ = sender.send(Message::Eof(stream));
})
}
fn output_frame(id: u32, stream: &str, text: String, skipped: Option<Value>) -> Frame {
let mut json = json!({ "kind": "output", "stream": stream });
if let Some(skipped) = skipped {
json["skipped"] = skipped;
}
Frame::with_payload(EVENT, id, json, text.into_bytes())
}
/// Run one command, sending output events for request `id`, until it settles.
pub fn run(
id: u32,
request: ExecRequest,
tmpdir: &str,
output: &Arc<Output>,
control: &Control,
groups: &Groups,
) -> Result<Value, Failure> {
let (sender, receiver) = mpsc::channel();
control.attach(sender.clone());
if control.aborted.load(Ordering::SeqCst) {
return Err(Failure::new("aborted", "aborted"));
}
// A string runs through the shell (as its last argument, or on stdin for legacy WSL bash); argv runs directly.
let (program, args, stdin_command) = match (&request.command, &request.argv) {
(Some(command), _) => {
let shell = sys::shell_config(request.shell_path.as_deref())?;
let mut args = shell.args;
if shell.command_on_stdin {
(shell.program, args, Some(command.clone()))
} else {
args.push(command.clone());
(shell.program, args, None)
}
}
(None, Some(argv)) => match argv.split_first() {
Some((first, rest)) => (
sys::resolve_program(first, &request.cwd, &request.env)?,
rest.to_vec(),
None,
),
None => return Err(Failure::new("spawn_error", "Empty argv: no program to run")),
},
(None, None) => return Err(Failure::new("EINVAL", "exec needs command or argv")),
};
if !Path::new(&request.cwd).exists() {
return Err(Failure::new(
"spawn_error",
format!(
"Working directory does not exist: {}\nCannot execute bash commands.",
request.cwd
),
));
}
let mut child = spawn(&request, &program, &args, stdin_command.is_some())
.map_err(|error| Failure::new("spawn_error", error.to_string()))?;
if let (Some(command), Some(mut stdin)) = (stdin_command, child.stdin.take()) {
let _ = stdin.write_all(command.as_bytes());
}
let pid = child.id();
groups.lock().unwrap().insert(pid);
let deadline = request.timeout.map(|timeout| Instant::now() + timeout);
let stop = Arc::new(AtomicBool::new(false));
// Without a window, output is not coalesced; reading then waits for the link.
let gate = if request.window.is_none() {
Some(output.clone())
} else {
None
};
let readers = [
spawn_reader(
0,
child.stdout.take().unwrap(),
sender.clone(),
stop.clone(),
gate.clone(),
),
spawn_reader(
1,
child.stderr.take().unwrap(),
sender.clone(),
stop.clone(),
gate,
),
];
let waiter = sender.clone();
thread::spawn(move || {
let _ = waiter.send(match child.wait() {
Ok(status) => Message::Exit(status),
Err(error) => Message::WaitFailed(error.to_string()),
});
});
drop(sender);
let mut decoders = [StreamDecoder::new(), StreamDecoder::new()];
let mut spill = request.spill.map(|(after_bytes, after_lines)| Spill {
after_bytes,
after_lines,
seen_bytes: 0,
seen_newlines: 0,
prefix: Vec::new(),
file: None,
path: None,
failed: None,
});
let mut pending = request.window.map(Pending::new);
let unsent = Arc::new(AtomicUsize::new(0));
let names = ["stdout", "stderr"];
let emit = |pending: &mut Option<Pending>, stream: usize, text: String| {
if text.is_empty() {
return;
}
match pending {
Some(pending) => pending.push(names[stream], text),
None => output.bulk(output_frame(id, names[stream], text, None), None),
}
};
let deliver = |pending: &mut Pending| {
for event in pending.take() {
output.bulk(
output_frame(id, event.stream, event.text, event.skipped),
Some(unsent.clone()),
);
}
};
let mut ended = [false, false];
let mut status: Option<Result<ExitStatus, String>> = None;
let mut idle_until: Option<Instant> = None;
let mut timed_out = false;
let mut aborted = false;
let mut killed = false;
let kill = |killed: &mut bool| {
if !*killed {
*killed = true;
kill_group(pid);
}
};
if control.killed.load(Ordering::SeqCst) {
kill(&mut killed);
}
loop {
let now = Instant::now();
// Node's timer stays armed until the command settles, including the grace period after exit.
if !timed_out && deadline.is_some_and(|deadline| now >= deadline) {
timed_out = true;
kill(&mut killed);
}
if status.is_some() && ended[0] && ended[1] {
break;
}
if idle_until.is_some_and(|idle| now >= idle) {
// A descendant may hold the pipes open after the process exited; stop waiting for it.
break;
}
if let Some(pending) = &mut pending
&& !pending.is_empty()
&& now >= pending.next_send()
&& unsent.load(Ordering::SeqCst) == 0
{
deliver(pending);
}
let mut wait = Duration::from_secs(3600);
if let Some(deadline) = deadline.filter(|_| !timed_out) {
wait = wait.min(deadline.saturating_duration_since(now));
}
if let Some(idle) = idle_until {
wait = wait.min(idle.saturating_duration_since(now));
}
if let Some(pending) = pending.as_ref().filter(|pending| !pending.is_empty()) {
wait = wait.min(if unsent.load(Ordering::SeqCst) > 0 {
UNSENT_POLL
} else {
pending.next_send().saturating_duration_since(now)
});
}
match receiver.recv_timeout(wait) {
Ok(Message::Data(stream, bytes)) => {
emit(&mut pending, stream, decoders[stream].decode(&bytes, false));
if let Some(spill) = &mut spill {
spill.push(&bytes, tmpdir);
if spill.failed.is_some() {
kill(&mut killed);
}
}
if status.is_some() {
idle_until = Some(Instant::now() + EXIT_STDIO_GRACE);
}
}
Ok(Message::Eof(stream)) => ended[stream] = true,
Ok(Message::Exit(exit)) => {
status = Some(Ok(exit));
idle_until = Some(Instant::now() + EXIT_STDIO_GRACE);
}
Ok(Message::WaitFailed(error)) => {
status = Some(Err(error));
idle_until = Some(Instant::now());
}
Ok(Message::Cancel) => {
aborted = true;
kill(&mut killed);
}
Ok(Message::Kill) => kill(&mut killed),
Err(RecvTimeoutError::Timeout) => {}
Err(RecvTimeoutError::Disconnected) => break,
}
}
// Like Node destroying its streams: stop reading, so the pipes close even if a descendant still holds them.
stop.store(true, Ordering::SeqCst);
for reader in &readers {
sys::interrupt_reader(reader);
}
groups.lock().unwrap().remove(&pid);
emit(&mut pending, 0, decoders[0].decode(&[], true));
emit(&mut pending, 1, decoders[1].decode(&[], true));
if let Some(pending) = &mut pending
&& !pending.is_empty()
{
deliver(pending);
}
let spill_path = spill.as_ref().and_then(|spill| spill.path.clone());
let with_spill = |failure: Failure| match &spill_path {
Some(path) => failure.extra(json!({ "spillPath": path })),
None => failure,
};
if timed_out {
return Err(with_spill(Failure::new("timeout", "timeout")));
}
if aborted {
return Err(with_spill(Failure::new("aborted", "aborted")));
}
if let Some(error) = spill.as_ref().and_then(|spill| spill.failed.clone()) {
return Err(Failure::new(
"unknown",
format!("Failed to preserve complete shell output: {error}"),
));
}
let status = match status {
Some(Ok(status)) => status,
Some(Err(error)) => return Err(Failure::new("spawn_error", error)),
None => {
return Err(Failure::new(
"spawn_error",
"process did not report its exit",
));
}
};
let mut result = json!({ "exitCode": sys::exit_code(status) });
if let Some(path) = spill_path {
result["spillPath"] = json!(path);
}
Ok(result)
}

120
third_party/pi/packages/env/daemon/src/frame.rs vendored Normal file
View File

@@ -0,0 +1,120 @@
//! Length-prefixed frames: `u32 length | u8 type | u32 id | u32 jsonLength | json | payload` (docs/protocol.md).
use serde_json::Value;
use std::io::{self, Read, Write};
pub const REQUEST: u8 = 1;
pub const RESULT: u8 = 2;
pub const ERROR: u8 = 3;
pub const EVENT: u8 = 4;
pub const CANCEL: u8 = 5;
pub const PING: u8 = 6;
/// Largest frame either side accepts.
pub const MAX_FRAME: usize = 16 * 1024 * 1024;
/// Largest payload a response carries, leaving room for its JSON.
pub const MAX_PAYLOAD: usize = MAX_FRAME - 64 * 1024;
pub struct Frame {
pub kind: u8,
pub id: u32,
pub json: Value,
pub payload: Vec<u8>,
}
impl Frame {
pub fn new(kind: u8, id: u32, json: Value) -> Frame {
Frame {
kind,
id,
json,
payload: Vec::new(),
}
}
pub fn with_payload(kind: u8, id: u32, json: Value, payload: Vec<u8>) -> Frame {
Frame {
kind,
id,
json,
payload,
}
}
}
fn invalid(message: &str) -> io::Error {
io::Error::new(io::ErrorKind::InvalidData, message.to_string())
}
/// Read one frame; `None` at a clean end of input.
pub fn read_frame(input: &mut impl Read) -> io::Result<Option<Frame>> {
let mut length = [0u8; 4];
match input.read_exact(&mut length) {
Ok(()) => {}
Err(error) if error.kind() == io::ErrorKind::UnexpectedEof => return Ok(None),
Err(error) => return Err(error),
}
let length = u32::from_be_bytes(length) as usize;
if !(9..=MAX_FRAME).contains(&length) {
return Err(invalid("frame length out of range"));
}
let mut body = vec![0u8; length];
input.read_exact(&mut body)?;
let kind = body[0];
let id = u32::from_be_bytes([body[1], body[2], body[3], body[4]]);
let json_length = u32::from_be_bytes([body[5], body[6], body[7], body[8]]) as usize;
if 9 + json_length > length {
return Err(invalid("JSON length out of range"));
}
// Unparseable JSON leaves the framing intact: the frame arrives with `Value::Null` and gets an error reply.
let json = if json_length == 0 {
Value::Object(Default::default())
} else {
serde_json::from_slice(&body[9..9 + json_length]).unwrap_or(Value::Null)
};
let payload = body[9 + json_length..].to_vec();
Ok(Some(Frame {
kind,
id,
json,
payload,
}))
}
pub fn write_frame(output: &mut impl Write, frame: &Frame) -> io::Result<()> {
let json = serde_json::to_vec(&frame.json).map_err(|_| invalid("unserializable JSON"))?;
let length = 9 + json.len() + frame.payload.len();
if length > MAX_FRAME {
return Err(invalid("frame too large"));
}
let mut header = Vec::with_capacity(13);
header.extend_from_slice(&(length as u32).to_be_bytes());
header.push(frame.kind);
header.extend_from_slice(&frame.id.to_be_bytes());
header.extend_from_slice(&(json.len() as u32).to_be_bytes());
output.write_all(&header)?;
output.write_all(&json)?;
output.write_all(&frame.payload)?;
output.flush()
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
#[test]
fn round_trips_json_and_payload() {
let mut buffer = Vec::new();
write_frame(
&mut buffer,
&Frame::with_payload(RESULT, 7, json!({ "a": 1 }), vec![0, 255, 10]),
)
.unwrap();
let frame = read_frame(&mut buffer.as_slice()).unwrap().unwrap();
assert_eq!((frame.kind, frame.id), (RESULT, 7));
assert_eq!(frame.json, json!({ "a": 1 }));
assert_eq!(frame.payload, vec![0, 255, 10]);
assert!(read_frame(&mut &[][..]).unwrap().is_none());
}
}

258
third_party/pi/packages/env/daemon/src/fs.rs vendored Normal file
View File

@@ -0,0 +1,258 @@
//! File operations with Node's semantics where the Rust standard library differs (docs/semantics.md).
use crate::errors::Failure;
use crate::sys;
use serde_json::{Value, json};
use std::ffi::OsStr;
use std::fs::{self, File, Metadata};
use std::io::{self, Read, Write};
use std::path::Path;
pub type Outcome<T> = Result<T, Failure>;
/// `{ name, kind, size, mtimeSec, mtimeNsec, dev, ino }` of lstat or fstat metadata.
pub fn info(path: &str, metadata: &Metadata) -> Value {
let file_type = metadata.file_type();
let kind = if file_type.is_file() {
"file"
} else if file_type.is_dir() {
"directory"
} else if file_type.is_symlink() {
"symlink"
} else {
"other"
};
let name = Path::new(path)
.file_name()
.map(|name| name.to_string_lossy().into_owned())
.unwrap_or_default();
let (mtime_sec, mtime_nsec) = sys::mtime(metadata);
let (dev, ino) = sys::identity(metadata);
json!({
"name": name,
"kind": kind,
"size": metadata.len(),
"mtimeSec": mtime_sec,
"mtimeNsec": mtime_nsec,
"dev": dev,
"ino": ino,
})
}
pub fn lstat(path: &str) -> Outcome<Value> {
fs::symlink_metadata(path)
.map(|metadata| info(path, &metadata))
.map_err(|error| Failure::io(&error, "lstat", path))
}
pub fn realpath(path: &str) -> Outcome<Value> {
sys::realpath(path)
.map(|resolved| json!({ "path": resolved }))
.map_err(|error| Failure::io(&error, "realpath", path))
}
/// Node's recursive `mkdir`: an existing directory is fine, an existing file is `EEXIST`, a file in the way of a
/// parent is `ENOTDIR`.
pub fn mkdir_recursive(path: &Path) -> io::Result<()> {
let is_dir = |path: &Path| {
fs::metadata(path)
.map(|metadata| metadata.is_dir())
.unwrap_or(false)
};
match fs::create_dir(path) {
Ok(()) => Ok(()),
Err(error) if error.kind() == io::ErrorKind::AlreadyExists => {
if is_dir(path) {
Ok(())
} else {
Err(error)
}
}
Err(error) if error.kind() == io::ErrorKind::NotFound => {
let Some(parent) = path
.parent()
.filter(|parent| !parent.as_os_str().is_empty())
else {
return Err(error);
};
match mkdir_recursive(parent) {
Err(parent_error) if parent_error.kind() == io::ErrorKind::AlreadyExists => {
return Err(sys::synthetic("ENOTDIR"));
}
other => other?,
}
match fs::create_dir(path) {
Err(error) if error.kind() == io::ErrorKind::AlreadyExists && is_dir(path) => {
Ok(())
}
other => other,
}
}
Err(error) => Err(error),
}
}
pub fn mkdir(path: &str, recursive: bool) -> Outcome<Value> {
let result = if recursive {
mkdir_recursive(Path::new(path))
} else {
fs::create_dir(path)
};
result
.map(|()| json!({}))
.map_err(|error| Failure::io(&error, "mkdir", path))
}
/// `writeFile`/`appendFile`: create missing parents like Node's recursive `mkdir`, then write. Returns the open file
/// for further chunks.
pub fn write(path: &str, append: bool, parents: bool, content: &[u8]) -> Outcome<File> {
if let Some(parent) = Path::new(path)
.parent()
.filter(|_| parents)
.filter(|parent| !parent.as_os_str().is_empty())
{
mkdir_recursive(parent)
.map_err(|error| Failure::io(&error, "mkdir", &parent.to_string_lossy()))?;
}
let mode = if append {
sys::OpenMode::Append
} else {
sys::OpenMode::Write
};
let mut file = sys::open(path, mode).map_err(|error| Failure::io(&error, "open", path))?;
file.write_all(content)
.map_err(|error| Failure::io(&error, "write", path))?;
Ok(file)
}
fn open_existing_for_write(path: &str) -> Outcome<File> {
sys::open(path, sys::OpenMode::ReadWrite).map_err(|error| Failure::io(&error, "open", path))
}
pub fn truncate(path: &str, size: u64) -> Outcome<Value> {
let file = open_existing_for_write(path)?;
file.set_len(size)
.map_err(|error| Failure::io(&error, "ftruncate", path))?;
Ok(json!({}))
}
pub fn fsync(path: &str) -> Outcome<Value> {
// POSIX refuses to open a directory for writing; check first so Windows reports the same.
if fs::metadata(path).is_ok_and(|metadata| metadata.is_dir()) {
return Err(Failure::new(
"EISDIR",
format!("EISDIR: illegal operation on a directory, open '{path}'"),
)
.path(path));
}
let file = open_existing_for_write(path)?;
file.sync_all()
.map_err(|error| Failure::io(&error, "fsync", path))?;
Ok(json!({}))
}
pub fn rename(from: &str, to: &str) -> Outcome<Value> {
sys::rename(from, to)
.map(|()| json!({}))
.map_err(|error| Failure::io_between(&error, "rename", from, Some(to)))
}
/// Node's `fs.rm`: missing paths fail unless `force`; a directory without `recursive` fails with `ERR_FS_EISDIR`.
pub fn rm(path: &str, recursive: bool, force: bool) -> Outcome<Value> {
let metadata = match fs::symlink_metadata(path) {
Ok(metadata) => metadata,
Err(error) if force && error.kind() == io::ErrorKind::NotFound => return Ok(json!({})),
Err(error) => return Err(Failure::io(&error, "lstat", path)),
};
let result = if metadata.is_dir() {
if !recursive {
return Err(Failure::new(
"ERR_FS_EISDIR",
format!("Path is a directory: rm returned EISDIR (is a directory) {path}"),
)
.path(path));
}
fs::remove_dir_all(path)
} else {
sys::remove_file(path)
};
result
.map(|()| json!({}))
.map_err(|error| Failure::io(&error, "rm", path))
}
/// A new directory named `prefix` plus six random characters, as libuv's `mkdtemp` creates it.
pub fn mkdtemp(prefix: &str) -> Outcome<Value> {
sys::mkdtemp(prefix)
.map(|path| json!({ "path": path }))
.map_err(|error| Failure::io(&error, "mkdtemp", &format!("{prefix}XXXXXX")))
}
/// Open a regular file for positional reads; directories fail with `EISDIR`, other non-regular files with
/// `NOT_REGULAR`, and with `no_follow` a final symbolic link with `SYMLINK`.
pub fn open_reader(path: &str, no_follow: bool) -> Outcome<(File, Value)> {
let file = sys::open_reader(path, no_follow)?;
let metadata = file
.metadata()
.map_err(|error| Failure::io(&error, "fstat", path))?;
if metadata.is_dir() {
return Err(
Failure::new("EISDIR", "EISDIR: illegal operation on a directory, read").path(path),
);
}
if !metadata.is_file() {
return Err(Failure::new("NOT_REGULAR", "Not a regular file").path(path));
}
let info = info(path, &metadata);
Ok((file, info))
}
/// Up to `length` bytes at `offset`; fewer only at the end of the file.
pub fn pread(file: &File, path: &str, offset: u64, length: usize) -> Outcome<Vec<u8>> {
let mut buffer = vec![0u8; length];
let mut filled = 0;
while filled < length {
match sys::read_at(file, &mut buffer[filled..], offset + filled as u64) {
Ok(0) => break,
Ok(read) => filled += read,
Err(error) if error.kind() == io::ErrorKind::Interrupted => {}
Err(error) => return Err(Failure::io(&error, "read", path)),
}
}
buffer.truncate(filled);
Ok(buffer)
}
/// Up to `length` bytes from the current position: one `read` call, as Node's `readFile` makes for files of unknown
/// size (FIFOs, devices).
pub fn read(mut file: &File, path: &str, length: usize) -> Outcome<Vec<u8>> {
let mut buffer = vec![0u8; length];
loop {
match file.read(&mut buffer) {
Ok(read) => {
buffer.truncate(read);
return Ok(buffer);
}
Err(error) if error.kind() == io::ErrorKind::Interrupted => {}
Err(error) => return Err(Failure::io(&error, "read", path)),
}
}
}
/// One directory entry as Node lists it: the name decoded as UTF-8 (invalid bytes replaced), and lstat of the
/// directory joined with that decoded name, so a name that is not valid UTF-8 reports `ENOENT` as in Node. `raw`
/// carries the original bytes of such a name, which libuv sorts by.
pub fn dir_entry(dir: &str, file_name: &OsStr) -> Value {
let name = file_name.to_string_lossy().into_owned();
let entry_path = Path::new(dir).join(&name).to_string_lossy().into_owned();
let mut entry = match fs::symlink_metadata(&entry_path) {
Ok(metadata) => json!({ "name": name, "info": info(&entry_path, &metadata) }),
Err(error) => {
json!({ "name": name, "error": Failure::io(&error, "lstat", &entry_path).to_json() })
}
};
if let Some(raw) = sys::raw_name(file_name).filter(|raw| *raw != name.as_bytes()) {
entry["raw"] = json!(raw);
}
entry
}

368
third_party/pi/packages/env/daemon/src/list.rs vendored Normal file
View File

@@ -0,0 +1,368 @@
//! `listFiles`: the files and directories below a directory, as `fd --hidden --exclude .git` lists them (without
//! following symbolic links), skipping what git ignores. Uses the `ignore` crate, which fd and ripgrep use.
use crate::errors::Failure;
use crate::frame::MAX_PAYLOAD;
use ignore::WalkBuilder;
use serde_json::{Value, json};
use std::ffi::OsString;
use std::fs;
use std::path::{Component, Path};
use std::time::{Duration, Instant};
const DEFAULT_MAX_ENTRIES: u64 = 50_000;
const DEFAULT_MAX_BYTES: u64 = 4 * 1024 * 1024;
const DEFAULT_TIMEOUT_MS: u64 = 10_000;
pub struct Options {
/// Include names starting with `.` (and, on Windows, hidden files).
pub hidden: bool,
/// Skip what git ignores inside git work trees: `.gitignore` files, `.git/info/exclude`, the global excludes file.
pub gitignore: bool,
/// Names skipped with everything below them, in addition to `.git`.
pub exclude: Vec<OsString>,
pub max_entries: u64,
/// Bytes of the listing payload: per entry one kind byte, the path, and a NUL.
pub max_bytes: u64,
pub timeout: Duration,
}
impl Options {
pub fn from_json(json: &Value) -> Result<Options, Failure> {
let count = |key: &str, default: u64| -> Result<u64, Failure> {
match &json[key] {
Value::Null => Ok(default),
value => value
.as_u64()
.ok_or_else(|| Failure::new("EINVAL", format!("invalid field {key}"))),
}
};
let exclude = match &json["exclude"] {
Value::Null => Vec::new(),
Value::Array(names) => names
.iter()
.map(|name| name.as_str().map(OsString::from))
.collect::<Option<Vec<_>>>()
.ok_or_else(|| Failure::new("EINVAL", "invalid field exclude"))?,
_ => return Err(Failure::new("EINVAL", "invalid field exclude")),
};
Ok(Options {
hidden: json["hidden"].as_bool().unwrap_or(true),
gitignore: json["gitignore"].as_bool().unwrap_or(true),
exclude,
max_entries: count("maxEntries", DEFAULT_MAX_ENTRIES)?,
max_bytes: count("maxBytes", DEFAULT_MAX_BYTES)?.min(MAX_PAYLOAD as u64),
timeout: Duration::from_millis(count("timeoutMs", DEFAULT_TIMEOUT_MS)?),
})
}
}
pub struct Listing {
/// Records of a kind byte (`f`, `d`, `l`), the path relative to the root with `/` separators, and a NUL.
pub payload: Vec<u8>,
pub count: u64,
pub truncated: bool,
}
/// The path of `path` below `root`, its components joined with `/`; names that are not UTF-8 get replacement
/// characters, as `readdir` reports them.
fn relative(root: &Path, path: &Path) -> Option<String> {
let rest = path.strip_prefix(root).ok()?;
let mut text = String::new();
for component in rest.components() {
let Component::Normal(name) = component else {
return None;
};
if !text.is_empty() {
text.push('/');
}
text.push_str(&name.to_string_lossy());
}
(!text.is_empty()).then_some(text)
}
/// List `root` depth first, siblings in byte order of their names. Stops with `truncated` at the first entry beyond
/// a bound; fails with `aborted` once `aborted()` is true. Entries that cannot be read are skipped.
pub fn list(root: &str, options: &Options, aborted: &dyn Fn() -> bool) -> Result<Listing, Failure> {
let metadata = fs::metadata(root).map_err(|error| Failure::io(&error, "scandir", root))?;
if !metadata.is_dir() {
return Err(Failure::new(
"ENOTDIR",
format!("ENOTDIR: not a directory, scandir '{root}'"),
)
.path(root));
}
let root_path = Path::new(root);
let exclude = options.exclude.clone();
let walk = WalkBuilder::new(root_path)
.hidden(!options.hidden)
.ignore(false)
.parents(options.gitignore)
.git_ignore(options.gitignore)
.git_exclude(options.gitignore)
.git_global(options.gitignore)
.require_git(true)
.follow_links(false)
// The global excludes file is found relative to the listed directory, not the daemon's.
.current_dir(root_path)
.sort_by_file_name(|a, b| a.cmp(b))
.filter_entry(move |entry| {
let name = entry.file_name();
name != ".git" && !exclude.iter().any(|excluded| excluded == name)
})
.build();
let started = Instant::now();
let mut listing = Listing {
payload: Vec::new(),
count: 0,
truncated: false,
};
for entry in walk {
if aborted() {
return Err(Failure::new("aborted", "aborted"));
}
let Ok(entry) = entry else { continue };
if entry.depth() == 0 {
continue;
}
let Some(file_type) = entry.file_type() else {
continue;
};
let kind = if file_type.is_symlink() {
b'l'
} else if file_type.is_dir() {
b'd'
} else if file_type.is_file() {
b'f'
} else {
continue;
};
let Some(path) = relative(root_path, entry.path()) else {
continue;
};
let size = path.len() as u64 + 2;
if listing.count >= options.max_entries
|| listing.payload.len() as u64 + size > options.max_bytes
|| started.elapsed() > options.timeout
{
listing.truncated = true;
break;
}
listing.payload.push(kind);
listing.payload.extend_from_slice(path.as_bytes());
listing.payload.push(0);
listing.count += 1;
}
Ok(listing)
}
pub fn run(json: &Value, aborted: &dyn Fn() -> bool) -> Result<(Value, Vec<u8>), Failure> {
let root = json["path"]
.as_str()
.ok_or_else(|| Failure::new("EINVAL", "missing field path"))?;
let listing = list(root, &Options::from_json(json)?, aborted)?;
Ok((
json!({ "count": listing.count, "truncated": listing.truncated }),
listing.payload,
))
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::atomic::{AtomicU64, Ordering};
struct Tree(std::path::PathBuf);
impl Tree {
fn new(files: &[&str]) -> Tree {
static NEXT: AtomicU64 = AtomicU64::new(0);
let root = std::env::temp_dir().join(format!(
"pi-env-list-{}-{}",
std::process::id(),
NEXT.fetch_add(1, Ordering::Relaxed)
));
let tree = Tree(root);
for file in files {
tree.write(file, "");
}
tree
}
fn write(&self, path: &str, content: &str) {
let path = self.0.join(path);
if path.to_string_lossy().ends_with('/') {
fs::create_dir_all(&path).unwrap();
return;
}
fs::create_dir_all(path.parent().unwrap()).unwrap();
fs::write(path, content).unwrap();
}
fn list(&self, options: Options) -> (Vec<String>, bool) {
let listing = list(&self.0.to_string_lossy(), &options, &|| false).unwrap();
let entries: Vec<String> = listing
.payload
.split(|byte| *byte == 0)
.filter(|record| !record.is_empty())
.map(|record| {
format!(
"{} {}",
record[0] as char,
String::from_utf8_lossy(&record[1..])
)
})
.collect();
assert_eq!(entries.len() as u64, listing.count);
(entries, listing.truncated)
}
}
impl Drop for Tree {
fn drop(&mut self) {
let _ = fs::remove_dir_all(&self.0);
}
}
fn options() -> Options {
Options::from_json(&json!({})).unwrap()
}
#[test]
fn lists_files_and_directories_depth_first_in_name_order() {
let tree = Tree::new(&[
"b.txt",
"a/z.txt",
"a/b/c.txt",
"empty/",
".hidden/x",
"A.txt",
]);
let (entries, truncated) = tree.list(options());
assert_eq!(
entries,
[
"d .hidden",
"f .hidden/x",
"f A.txt",
"d a",
"d a/b",
"f a/b/c.txt",
"f a/z.txt",
"f b.txt",
"d empty"
]
);
assert!(!truncated);
let (entries, _) = tree.list(Options {
hidden: false,
..options()
});
assert!(!entries.iter().any(|entry| entry.contains(".hidden")));
}
#[test]
fn skips_what_git_ignores_only_in_work_trees() {
let tree = Tree::new(&[
"keep.txt",
"out/x.js",
"sub/keep.txt",
"sub/drop.log",
"node_modules/m/i.js",
]);
tree.write(".gitignore", "out/\n*.log\n");
tree.write("sub/.gitignore", "!keep.txt\n");
// Not a work tree yet: everything is listed.
let (entries, _) = tree.list(options());
assert!(entries.contains(&"f out/x.js".to_string()));
tree.write(".git/HEAD", "ref: refs/heads/main\n");
tree.write(".git/info/exclude", "keep.txt\n");
let (entries, _) = tree.list(Options {
exclude: vec!["node_modules".into()],
..options()
});
assert_eq!(
entries,
[
"f .gitignore",
"d sub",
"f sub/.gitignore",
"f sub/keep.txt"
]
);
let (entries, _) = tree.list(Options {
gitignore: false,
..options()
});
assert!(entries.contains(&"f out/x.js".to_string()));
assert!(!entries.iter().any(|entry| entry.contains(".git/")));
// A subdirectory of a work tree applies the rules of its parents.
let sub = list(&tree.0.join("sub").to_string_lossy(), &options(), &|| false).unwrap();
assert_eq!(sub.payload, b"f.gitignore\0fkeep.txt\0");
}
#[cfg(unix)]
#[test]
fn lists_symbolic_links_without_following_them() {
let tree = Tree::new(&["dir/inner.txt"]);
std::os::unix::fs::symlink(tree.0.join("dir"), tree.0.join("link")).unwrap();
std::os::unix::fs::symlink("missing", tree.0.join("dangling")).unwrap();
std::os::unix::fs::symlink(&tree.0, tree.0.join("dir/loop")).unwrap();
let (entries, _) = tree.list(options());
assert_eq!(
entries,
[
"l dangling",
"d dir",
"f dir/inner.txt",
"l dir/loop",
"l link"
]
);
}
#[test]
fn stops_at_bounds_with_truncated() {
let tree = Tree::new(&["a", "b", "c"]);
let (entries, truncated) = tree.list(Options {
max_entries: 2,
..options()
});
assert_eq!((entries.len(), truncated), (2, true));
let (entries, truncated) = tree.list(Options {
max_entries: 3,
..options()
});
assert_eq!((entries.len(), truncated), (3, false));
let (entries, truncated) = tree.list(Options {
max_bytes: 6,
..options()
});
assert_eq!(
(entries, truncated),
(vec!["f a".to_string(), "f b".to_string()], true)
);
let (entries, truncated) = tree.list(Options {
timeout: Duration::ZERO,
..options()
});
assert!(truncated && entries.is_empty());
}
#[test]
fn fails_on_missing_roots_files_and_aborts() {
let tree = Tree::new(&["file"]);
let missing = list(
&tree.0.join("missing").to_string_lossy(),
&options(),
&|| false,
);
assert_eq!(missing.err().unwrap().code, "ENOENT");
let file = list(&tree.0.join("file").to_string_lossy(), &options(), &|| {
false
});
assert_eq!(file.err().unwrap().code, "ENOTDIR");
let aborted = list(&tree.0.to_string_lossy(), &options(), &|| true);
assert_eq!(aborted.err().unwrap().code, "aborted");
}
}

551
third_party/pi/packages/env/daemon/src/main.rs vendored Normal file
View File

@@ -0,0 +1,551 @@
//! pi-env: a small daemon that gives a Pi Durable host an execution environment on this machine over stdin/stdout
//! (docs/protocol.md). Started as `pi-env serve --token <hex>`, usually through `ssh`.
mod decode;
mod errors;
mod exec;
mod frame;
mod fs;
mod list;
mod output;
mod pty;
mod scan;
mod sys;
mod watch;
mod window;
use errors::Failure;
use exec::Control;
use frame::{CANCEL, ERROR, Frame, PING, REQUEST, RESULT};
use output::Output;
use serde_json::{Value, json};
use std::collections::{HashMap, VecDeque};
use std::env;
use std::fs::{File, ReadDir};
use std::io::{self, BufReader, BufWriter, Read, Write};
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::mpsc::{self, Receiver, Sender};
use std::sync::{Arc, Mutex};
use std::thread;
use std::time::{Duration, SystemTime, UNIX_EPOCH};
const PROTOCOL: u64 = 1;
/// Operations added after protocol 1's first release; clients check `hello.features` before using them.
const FEATURES: &[&str] = &["listFiles"];
/// The npm package version this daemon ships with (build.rs).
const VERSION: &str = env!("PI_ENV_VERSION");
const PING_INTERVAL: Duration = Duration::from_secs(5);
const SILENCE_LIMIT_MS: u64 = 30_000;
const SCAN_CHUNK: usize = 64 * 1024;
/// Threads for file operations; commands run on their own threads.
const WORKERS: usize = 16;
/// Open handles per connection, beyond which opening fails with `EMFILE`.
const MAX_HANDLES: usize = 4096;
/// Replies with larger payloads queue behind command output instead of going first.
const CONTROL_PAYLOAD: usize = 64 * 1024;
struct FileHandle {
file: File,
path: String,
/// A failed chunk write poisons a write handle, so later chunks cannot leave a gap.
failed: AtomicBool,
/// Chunk writes run one at a time in arrival order, though the client sends several at once.
serial: Mutex<Serial>,
}
#[derive(Default)]
struct Serial {
jobs: VecDeque<Job>,
running: bool,
}
struct DirHandle {
entries: ReadDir,
path: String,
}
enum Handle {
File(Arc<FileHandle>),
Dir(Arc<Mutex<DirHandle>>),
}
type Job = Box<dyn FnOnce() + Send>;
struct Server {
output: Arc<Output>,
tmpdir: String,
handles: Mutex<HashMap<u64, Handle>>,
next_handle: AtomicU64,
controls: Mutex<HashMap<u32, Arc<Control>>>,
groups: exec::Groups,
terminals: pty::Terminals,
}
fn now_ms() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|elapsed| elapsed.as_millis() as u64)
.unwrap_or(0)
}
fn field<'a>(json: &'a Value, key: &str) -> Result<&'a str, Failure> {
json[key]
.as_str()
.ok_or_else(|| Failure::new("EINVAL", format!("missing field {key}")))
}
fn number(json: &Value, key: &str) -> Result<u64, Failure> {
json[key]
.as_u64()
.ok_or_else(|| Failure::new("EINVAL", format!("missing or invalid field {key}")))
}
fn flag(json: &Value, key: &str) -> bool {
json[key].as_bool().unwrap_or(false)
}
impl Server {
fn file(&self, json: &Value) -> Result<Arc<FileHandle>, Failure> {
match self.handles.lock().unwrap().get(&number(json, "handle")?) {
Some(Handle::File(file)) => Ok(file.clone()),
Some(Handle::Dir(_)) => Err(Failure::new("EBADF", "not a file handle")),
None => Err(Failure::new("EBADF", "unknown handle")),
}
}
fn insert(&self, handle: Handle) -> Result<u64, Failure> {
let mut handles = self.handles.lock().unwrap();
if handles.len() >= MAX_HANDLES {
return Err(Failure::new("EMFILE", "EMFILE: too many open files"));
}
let id = self.next_handle.fetch_add(1, Ordering::Relaxed);
handles.insert(id, handle);
Ok(id)
}
fn insert_file(&self, file: File, path: &str) -> Result<u64, Failure> {
self.insert(Handle::File(Arc::new(FileHandle {
file,
path: path.to_string(),
failed: AtomicBool::new(false),
serial: Mutex::new(Serial::default()),
})))
}
fn dispatch(
&self,
id: u32,
json: &Value,
payload: Vec<u8>,
control: &Control,
) -> Result<(Value, Vec<u8>), Failure> {
if json.is_null() {
return Err(Failure::new("EINVAL", "invalid JSON"));
}
let op = json["op"].as_str().unwrap_or_default();
let plain = |value: Result<Value, Failure>| value.map(|value| (value, Vec::new()));
match op {
"hello" => plain(Ok(json!({
"protocol": PROTOCOL,
"version": VERSION,
"os": env::consts::OS,
"arch": env::consts::ARCH,
"home": sys::home(),
"separator": sys::PATH_SEPARATOR,
"tmpdir": self.tmpdir,
"cwd": sys::cwd(),
"driveCwds": sys::drive_cwds(),
"pid": std::process::id(),
"features": FEATURES,
}))),
"lstat" => plain(fs::lstat(field(json, "path")?)),
"realpath" => plain(fs::realpath(field(json, "path")?)),
"write" => {
let path = field(json, "path")?;
let parents = json["parents"].as_bool().unwrap_or(true);
let file = fs::write(path, flag(json, "append"), parents, &payload)?;
if !flag(json, "keep") {
return plain(Ok(json!({})));
}
plain(Ok(json!({ "handle": self.insert_file(file, path)? })))
}
"writeChunk" => {
let handle = self.file(json)?;
if handle.failed.load(Ordering::SeqCst) {
return Err(Failure::new("EBADF", "an earlier write failed"));
}
(&handle.file).write_all(&payload).map_err(|error| {
handle.failed.store(true, Ordering::SeqCst);
Failure::io(&error, "write", &handle.path)
})?;
plain(Ok(json!({})))
}
"truncate" => plain(fs::truncate(field(json, "path")?, number(json, "size")?)),
"fsync" => plain(fs::fsync(field(json, "path")?)),
"rename" => plain(fs::rename(field(json, "path")?, field(json, "to")?)),
"mkdir" => plain(fs::mkdir(field(json, "path")?, flag(json, "recursive"))),
"rm" => plain(fs::rm(
field(json, "path")?,
flag(json, "recursive"),
flag(json, "force"),
)),
"mkdtemp" => plain(fs::mkdtemp(field(json, "path")?)),
"open" => {
let path = field(json, "path")?;
if json["mode"].as_str() == Some("read") {
// Node's `open(path, "r")`: any kind of file, blocking; reads report what the file is.
let file = sys::open(path, sys::OpenMode::Read)
.map_err(|error| Failure::io(&error, "open", path))?;
let stat = match file.metadata() {
Ok(metadata) => json!({ "stat": fs::info(path, &metadata) }),
Err(error) => {
json!({ "statError": Failure::io(&error, "fstat", path).to_json() })
}
};
let mut result = stat;
result["handle"] = json!(self.insert_file(file, path)?);
return plain(Ok(result));
}
let (file, info) = fs::open_reader(path, flag(json, "noFollow"))?;
plain(Ok(
json!({ "handle": self.insert_file(file, path)?, "info": info }),
))
}
"pread" => {
let handle = self.file(json)?;
let length = (number(json, "length")? as usize).min(frame::MAX_PAYLOAD);
let bytes = match json["offset"].as_u64() {
Some(offset) => fs::pread(&handle.file, &handle.path, offset, length)?,
None => fs::read(&handle.file, &handle.path, length)?,
};
Ok((json!({}), bytes))
}
"fstat" => {
let handle = self.file(json)?;
let metadata = handle
.file
.metadata()
.map_err(|error| Failure::io(&error, "fstat", &handle.path))?;
plain(Ok(fs::info(&handle.path, &metadata)))
}
"scanLines" => {
let handle = self.file(json)?;
plain(scan_lines(&handle.file, &handle.path, json, control))
}
"opendir" => {
let path = field(json, "path")?;
let entries =
sys::read_dir(path).map_err(|error| Failure::io(&error, "opendir", path))?;
let handle = self.insert(Handle::Dir(Arc::new(Mutex::new(DirHandle {
entries,
path: path.to_string(),
}))))?;
plain(Ok(json!({ "handle": handle })))
}
"readdir" => plain(self.read_dir(json)),
"listFiles" => list::run(json, &|| control.aborted.load(Ordering::SeqCst)),
"close" => {
self.handles
.lock()
.unwrap()
.remove(&number(json, "handle")?);
plain(Ok(json!({})))
}
"exec" => plain(exec::run(
id,
exec::ExecRequest::from_json(json)?,
&self.tmpdir,
&self.output,
control,
&self.groups,
)),
"watch" => plain(watch::run(id, json, &self.output, control)),
"pty" => plain(pty::run(
id,
pty::PtyRequest::from_json(json)?,
&self.output,
control,
&self.groups,
&self.terminals,
)),
_ => Err(Failure::new("EINVAL", format!("unknown operation {op}"))),
}
}
fn read_dir(&self, json: &Value) -> Result<Value, Failure> {
let max = number(json, "max")?.max(1) as usize;
// Lock only this directory, so other requests are not held up by a slow listing.
let dir = match self.handles.lock().unwrap().get(&number(json, "handle")?) {
Some(Handle::Dir(dir)) => dir.clone(),
_ => return Err(Failure::new("EBADF", "not a directory handle")),
};
let mut dir = dir.lock().unwrap();
let mut entries = Vec::new();
let mut done = false;
while entries.len() < max {
match dir.entries.next() {
None => {
done = true;
break;
}
Some(Err(error)) => return Err(Failure::io(&error, "readdir", &dir.path)),
Some(Ok(entry)) => entries.push(fs::dir_entry(&dir.path, &entry.file_name())),
}
}
Ok(json!({ "entries": entries, "done": done }))
}
}
fn scan_lines(file: &File, path: &str, json: &Value, control: &Control) -> Result<Value, Failure> {
let start_line = number(json, "startLine")?;
let end_line = json["endLine"].as_u64();
if end_line.is_some_and(|end| end <= start_line) {
return Err(Failure::new("EINVAL", "Invalid line range"));
}
let mut scanner = scan::LineScanner::new(start_line, end_line);
let mut buffer = vec![0u8; SCAN_CHUNK];
let mut position = 0u64;
loop {
if control.aborted.load(Ordering::SeqCst) {
return Err(Failure::new("aborted", "aborted"));
}
let read = sys::read_at(file, &mut buffer, position)
.map_err(|error| Failure::io(&error, "read", path))?;
if read == 0 {
return Ok(scanner.finish());
}
scanner.push(&buffer[..read]);
position += read as u64;
}
}
fn kill_all(groups: &exec::Groups) {
for pid in groups.lock().unwrap().iter() {
exec::kill_group(*pid);
}
}
/// Stdin that records when bytes last arrived, so a large frame arriving slowly counts as a live client.
struct Seen<R> {
inner: R,
last_seen: Arc<AtomicU64>,
}
impl<R: Read> Read for Seen<R> {
fn read(&mut self, buffer: &mut [u8]) -> io::Result<usize> {
let read = self.inner.read(buffer)?;
if read > 0 {
self.last_seen.store(now_ms(), Ordering::Relaxed);
}
Ok(read)
}
}
fn start_workers() -> Sender<Job> {
let (jobs, queue) = mpsc::channel::<Job>();
let queue = Arc::new(Mutex::new(queue));
for _ in 0..WORKERS {
let queue: Arc<Mutex<Receiver<Job>>> = queue.clone();
thread::spawn(move || {
loop {
let job = queue.lock().unwrap().recv();
match job {
Ok(job) => job(),
Err(_) => return,
}
}
});
}
jobs
}
fn respond(server: &Server, id: u32, request: Frame, control: Arc<Control>) {
let is_exec = matches!(request.json["op"].as_str(), Some("exec" | "pty"));
let reply = match server.dispatch(id, &request.json, request.payload, &control) {
Ok((json, payload)) => Frame::with_payload(RESULT, id, json, payload),
Err(failure) => Frame::new(ERROR, id, failure.to_json()),
};
server.controls.lock().unwrap().remove(&id);
// A command's result follows its output; large payloads do not hold up pings and small replies.
if is_exec || reply.payload.len() > CONTROL_PAYLOAD {
server.output.bulk(reply, None);
} else {
server.output.control(reply);
}
}
/// Queue `job` behind the handle's earlier chunk writes; one pool job drains the queue.
fn run_in_order(jobs: &Sender<Job>, handle: Arc<FileHandle>, job: Job) {
let mut serial = handle.serial.lock().unwrap();
serial.jobs.push_back(job);
if serial.running {
return;
}
serial.running = true;
drop(serial);
let _ = jobs.send(Box::new(move || {
loop {
let next = {
let mut serial = handle.serial.lock().unwrap();
let next = serial.jobs.pop_front();
if next.is_none() {
serial.running = false;
}
next
};
match next {
Some(job) => job(),
None => return,
}
}
}));
}
fn serve_frames(server: &Arc<Server>, input: &mut impl Read) -> io::Result<()> {
let jobs = start_workers();
while let Some(request) = frame::read_frame(input)? {
match request.kind {
REQUEST
if matches!(
request.json["op"].as_str(),
Some("ptyWrite" | "ptyResize" | "ptyForeground")
) =>
{
// Answered here, in arrival order: through the worker pool, two writes could swap.
let reply = match request.json["op"].as_str() {
Some("ptyWrite") => {
pty::write(&server.terminals, &request.json, request.payload)
}
Some("ptyResize") => pty::resize(&server.terminals, &request.json),
_ => pty::foreground(&server.terminals, &request.json),
};
server.output.control(match reply {
Ok(json) => Frame::new(RESULT, request.id, json),
Err(failure) => Frame::new(ERROR, request.id, failure.to_json()),
});
}
REQUEST => {
let id = request.id;
// Registered before the request runs, so a cancel right behind it is not lost.
let control = Arc::new(Control::default());
server.controls.lock().unwrap().insert(id, control.clone());
// Commands and watchers last; they get their own threads (watchers recurse through trees).
if matches!(request.json["op"].as_str(), Some("exec" | "watch" | "pty")) {
let worker = server.clone();
let spawned = thread::Builder::new()
.stack_size(16 * 1024 * 1024)
.spawn(move || respond(&worker, id, request, control));
if let Err(error) = spawned {
server.controls.lock().unwrap().remove(&id);
let failure = Failure::new("spawn_error", error.to_string());
server
.output
.bulk(Frame::new(ERROR, id, failure.to_json()), None);
}
} else {
let worker = server.clone();
let serial = match request.json["op"].as_str() {
Some("writeChunk") => server.file(&request.json).ok(),
_ => None,
};
let job: Job = Box::new(move || respond(&worker, id, request, control));
match serial {
Some(handle) => run_in_order(&jobs, handle, job),
None => {
let _ = jobs.send(job);
}
}
}
}
CANCEL => {
let control = server.controls.lock().unwrap().get(&request.id).cloned();
if let Some(control) = control {
control.cancel(request.json["mode"].as_str() == Some("kill"));
}
}
_ => {}
}
}
Ok(())
}
fn serve(token: &str) -> io::Result<()> {
{
let mut stdout = io::stdout().lock();
writeln!(stdout, "PI-ENV {token}")?;
stdout.flush()?;
}
let output = Arc::new(Output::default());
let writer = {
let output = output.clone();
thread::spawn(move || {
output.run(&mut BufWriter::with_capacity(1 << 20, io::stdout().lock()))
})
};
let server = Arc::new(Server {
output: output.clone(),
tmpdir: sys::tmpdir(),
handles: Mutex::new(HashMap::new()),
next_handle: AtomicU64::new(1),
controls: Mutex::new(HashMap::new()),
groups: Arc::new(Mutex::new(Default::default())),
terminals: Arc::new(Mutex::new(HashMap::new())),
});
let last_seen = Arc::new(AtomicU64::new(now_ms()));
{
let output = output.clone();
let last_seen = last_seen.clone();
let groups = server.groups.clone();
thread::spawn(move || {
loop {
thread::sleep(PING_INTERVAL);
output.control(Frame::new(PING, 0, json!({})));
// A client that went silent (a phone that lost its network) leaves nothing running.
if now_ms().saturating_sub(last_seen.load(Ordering::Relaxed)) > SILENCE_LIMIT_MS {
kill_all(&groups);
std::process::exit(0);
}
}
});
}
let mut input = BufReader::with_capacity(
1 << 20,
Seen {
inner: io::stdin().lock(),
last_seen,
},
);
let result = serve_frames(&server, &mut input);
// However input ended, the client is gone: stop everything it started.
kill_all(&server.groups);
output.close();
let _ = writer.join();
result
}
fn main() {
let args: Vec<String> = env::args().collect();
match args.get(1).map(String::as_str) {
Some("serve") => {
let token = args
.iter()
.position(|arg| arg == "--token")
.and_then(|index| args.get(index + 1));
let Some(token) = token else {
eprintln!("usage: pi-env serve --token <hex>");
std::process::exit(2);
};
if let Err(error) = serve(token) {
eprintln!("pi-env: {error}");
std::process::exit(1);
}
// Request threads may still be blocked (a FIFO open); nothing is left to answer them.
std::process::exit(0);
}
Some("--version") => println!("pi-env {VERSION}"),
_ => {
eprintln!("usage: pi-env serve --token <hex>");
std::process::exit(2);
}
}
}

100
third_party/pi/packages/env/daemon/src/output.rs vendored Normal file
View File

@@ -0,0 +1,100 @@
//! The single writer of stdout. Control frames (results, errors, pings, watch events) always go before bulk command
//! output, so a slow link delays output but never a cancel's result or a ping. Bulk output is bounded: readers of
//! commands without an output window wait while too much of it is unsent, which slows the command down like a full
//! pipe would.
use crate::frame::{self, Frame};
use std::collections::VecDeque;
use std::io::Write;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::sync::{Arc, Condvar, Mutex};
use std::time::Duration;
/// Unsent bulk bytes above which unwindowed command output waits.
const BULK_LIMIT: usize = 4 * 1024 * 1024;
#[derive(Default)]
struct Queues {
control: VecDeque<Frame>,
bulk: VecDeque<(Frame, Option<Arc<AtomicUsize>>)>,
bulk_bytes: usize,
closed: bool,
}
#[derive(Default)]
pub struct Output {
queues: Mutex<Queues>,
/// Signalled when a frame is queued or the writer stops.
queued: Condvar,
/// Signalled when bulk bytes were written.
drained: Condvar,
}
impl Output {
pub fn control(&self, frame: Frame) {
let mut queues = self.queues.lock().unwrap();
queues.control.push_back(frame);
self.queued.notify_one();
}
/// Queue bulk output; `unsent` counts this sender's frames until they are written.
pub fn bulk(&self, frame: Frame, unsent: Option<Arc<AtomicUsize>>) {
let mut queues = self.queues.lock().unwrap();
queues.bulk_bytes += frame.payload.len();
if let Some(counter) = &unsent {
counter.fetch_add(1, Ordering::SeqCst);
}
queues.bulk.push_back((frame, unsent));
self.queued.notify_one();
}
/// Wait while more than the bulk limit is unsent, unless `stop` is set.
pub fn wait_for_room(&self, stop: &AtomicBool) {
let mut queues = self.queues.lock().unwrap();
while queues.bulk_bytes > BULK_LIMIT && !queues.closed && !stop.load(Ordering::SeqCst) {
queues = self
.drained
.wait_timeout(queues, Duration::from_millis(100))
.unwrap()
.0;
}
}
pub fn close(&self) {
let mut queues = self.queues.lock().unwrap();
queues.closed = true;
self.queued.notify_all();
self.drained.notify_all();
}
/// Write frames to `writer` until closed or a write fails.
pub fn run(&self, writer: &mut impl Write) {
loop {
let (frame, unsent) = {
let mut queues = self.queues.lock().unwrap();
loop {
if let Some(frame) = queues.control.pop_front() {
break (frame, None);
}
if let Some((frame, unsent)) = queues.bulk.pop_front() {
queues.bulk_bytes -= frame.payload.len();
break (frame, unsent);
}
if queues.closed {
return;
}
queues = self.queued.wait(queues).unwrap();
}
};
let failed = frame::write_frame(writer, &frame).is_err();
if let Some(counter) = unsent {
counter.fetch_sub(1, Ordering::SeqCst);
}
self.drained.notify_all();
if failed {
self.close();
return;
}
}
}
}

556
third_party/pi/packages/env/daemon/src/pty.rs vendored Normal file
View File

@@ -0,0 +1,556 @@
//! `pty`: a program in a pseudo-terminal, for interactive terminals. Unlike `exec`, output is the terminal's raw
//! bytes, and the program reads input: `ptyWrite`, `ptyResize` and `ptyForeground` address the running `pty` request by
//! its id.
use crate::errors::Failure;
use crate::exec::{Control, Groups};
#[cfg(unix)]
use crate::exec::{Message, kill_group};
#[cfg(unix)]
use crate::frame::{EVENT, Frame};
use crate::output::Output;
use serde_json::{Map, Value, json};
use std::collections::HashMap;
use std::sync::mpsc::Sender;
use std::sync::{Arc, Mutex};
/// After the program exits, output still arriving is collected until this much silence.
#[cfg(unix)]
const EXIT_GRACE: std::time::Duration = std::time::Duration::from_millis(100);
/// Largest terminal size accepted, in cells.
const MAX_CELLS: u64 = 4096;
/// What `ptyWrite`, `ptyResize` and `ptyForeground` reach: the terminal of one running `pty` request.
pub struct Terminal {
input: Sender<Vec<u8>>,
#[cfg(unix)]
master: Arc<std::fs::File>,
/// The program's process id, which is also its process group's (it leads the terminal's session).
#[cfg(unix)]
pid: u32,
}
/// Running terminals by the id of their `pty` request.
pub type Terminals = Arc<Mutex<HashMap<u32, Arc<Terminal>>>>;
#[cfg_attr(not(unix), allow(dead_code))]
pub struct PtyRequest {
argv: Option<Vec<String>>,
cwd: String,
env: Map<String, Value>,
inherit_env: bool,
cols: u16,
rows: u16,
}
fn size(json: &Value, key: &str) -> Result<u16, Failure> {
match json[key].as_u64() {
Some(value) if (1..=MAX_CELLS).contains(&value) => Ok(value as u16),
_ => Err(Failure::new(
"EINVAL",
format!("{key} must be between 1 and {MAX_CELLS}"),
)),
}
}
impl PtyRequest {
pub fn from_json(json: &Value) -> Result<PtyRequest, Failure> {
let argv = match &json["argv"] {
Value::Null => None,
value => Some(
value
.as_array()
.and_then(|items| {
items
.iter()
.map(|item| item.as_str().map(str::to_string))
.collect()
})
.filter(|argv: &Vec<String>| !argv.is_empty())
.ok_or_else(|| {
Failure::new("EINVAL", "argv must be a non-empty array of strings")
})?,
),
};
Ok(PtyRequest {
argv,
cwd: json["cwd"]
.as_str()
.ok_or_else(|| Failure::new("EINVAL", "pty needs cwd"))?
.to_string(),
env: json["env"].as_object().cloned().unwrap_or_default(),
inherit_env: json["inheritEnv"].as_bool().unwrap_or(true),
cols: size(json, "cols")?,
rows: size(json, "rows")?,
})
}
}
fn find(terminals: &Terminals, json: &Value) -> Result<Arc<Terminal>, Failure> {
let target = json["target"]
.as_u64()
.ok_or_else(|| Failure::new("EINVAL", "missing field target"))?;
terminals
.lock()
.unwrap()
.get(&(target as u32))
.cloned()
.ok_or_else(|| Failure::new("EBADF", "no running terminal with this id"))
}
/// Queue input for the terminal; it is written in the order it arrives, without holding up other requests.
pub fn write(terminals: &Terminals, json: &Value, payload: Vec<u8>) -> Result<Value, Failure> {
let terminal = find(terminals, json)?;
terminal
.input
.send(payload)
.map_err(|_| Failure::new("EBADF", "the terminal has ended"))?;
Ok(json!({}))
}
/// Change the terminal's size; the program receives `SIGWINCH`.
pub fn resize(terminals: &Terminals, json: &Value) -> Result<Value, Failure> {
let terminal = find(terminals, json)?;
let (cols, rows) = (size(json, "cols")?, size(json, "rows")?);
#[cfg(unix)]
unix::set_size(&terminal.master, cols, rows)
.map_err(|error| Failure::io(&error, "ioctl", "pty"))?;
#[cfg(not(unix))]
let _ = (terminal, cols, rows);
Ok(json!({}))
}
/// The process group in the terminal's foreground when it is not the program's own: a job a shell started, such as an
/// editor. `{ foreground: null }` while the program itself is in front (a shell at its prompt).
pub fn foreground(terminals: &Terminals, json: &Value) -> Result<Value, Failure> {
let terminal = find(terminals, json)?;
#[cfg(unix)]
{
let group = unix::foreground(&terminal.master)
.map_err(|error| Failure::io(&error, "tcgetpgrp", "pty"))?;
if group <= 0 || group as u32 == terminal.pid {
return Ok(json!({ "foreground": null }));
}
let name = unix::process_name(group);
Ok(json!({ "foreground": { "pid": group, "name": name } }))
}
#[cfg(not(unix))]
{
let _ = terminal;
Ok(json!({ "foreground": null }))
}
}
#[cfg(not(unix))]
pub fn run(
_id: u32,
_request: PtyRequest,
_output: &Arc<Output>,
_control: &Control,
_groups: &Groups,
_terminals: &Terminals,
) -> Result<Value, Failure> {
Err(Failure::new(
"ENOTSUP",
"Terminals are not supported on this system yet",
))
}
/// Run the program in a new terminal, sending a `started` event and then its output, until it exits.
#[cfg(unix)]
pub fn run(
id: u32,
request: PtyRequest,
output: &Arc<Output>,
control: &Control,
groups: &Groups,
terminals: &Terminals,
) -> Result<Value, Failure> {
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::mpsc::{self, RecvTimeoutError};
use std::thread;
use std::time::{Duration, Instant};
let (sender, receiver) = mpsc::channel();
control.attach(sender.clone());
if control.aborted.load(Ordering::SeqCst) {
return Err(Failure::new("aborted", "aborted"));
}
if !std::path::Path::new(&request.cwd).exists() {
return Err(Failure::new(
"spawn_error",
format!("Working directory does not exist: {}", request.cwd),
));
}
let (master, mut child) =
unix::spawn(&request).map_err(|error| Failure::new("spawn_error", error.to_string()))?;
let master = Arc::new(master);
let pid = child.id();
groups.lock().unwrap().insert(pid);
let (input, inputs) = mpsc::channel::<Vec<u8>>();
terminals.lock().unwrap().insert(
id,
Arc::new(Terminal {
input,
master: master.clone(),
pid,
}),
);
output.control(Frame::new(
EVENT,
id,
json!({ "kind": "started", "pid": pid }),
));
let stop = Arc::new(AtomicBool::new(false));
let reader = {
let master = master.clone();
let sender = sender.clone();
let stop = stop.clone();
let output = output.clone();
thread::spawn(move || {
unix::read(&master, &stop, |bytes| {
let delivered = sender.send(Message::Data(0, bytes.to_vec())).is_ok();
// Like a full pipe: a program that writes faster than the link waits.
output.wait_for_room(&stop);
delivered
});
let _ = sender.send(Message::Eof(0));
})
};
{
let master = master.clone();
let stop = stop.clone();
thread::spawn(move || {
for bytes in inputs {
if !unix::write_all(&master, &bytes, &stop) {
return;
}
}
});
}
let waiter = sender.clone();
thread::spawn(move || {
let _ = waiter.send(match child.wait() {
Ok(status) => Message::Exit(status),
Err(error) => Message::WaitFailed(error.to_string()),
});
});
drop(sender);
let mut status = None;
let mut ended = false;
let mut idle_until: Option<Instant> = None;
let mut aborted = false;
let mut killed = false;
let kill = |killed: &mut bool| {
if !*killed {
*killed = true;
kill_group(pid);
}
};
if control.killed.load(Ordering::SeqCst) {
kill(&mut killed);
}
loop {
if status.is_some() && ended {
break;
}
let wait = match idle_until {
Some(idle) => {
let now = Instant::now();
if now >= idle {
// A background job may keep the terminal open after the program exited.
break;
}
idle - now
}
None => Duration::from_secs(3600),
};
match receiver.recv_timeout(wait) {
Ok(Message::Data(_, bytes)) => {
output.bulk(
Frame::with_payload(EVENT, id, json!({ "kind": "output" }), bytes),
None,
);
if status.is_some() {
idle_until = Some(Instant::now() + EXIT_GRACE);
}
}
Ok(Message::Eof(_)) => ended = true,
Ok(Message::Exit(exit)) => {
status = Some(Ok(exit));
idle_until = Some(Instant::now() + EXIT_GRACE);
}
Ok(Message::WaitFailed(error)) => {
status = Some(Err(error));
idle_until = Some(Instant::now());
}
Ok(Message::Cancel) => {
aborted = true;
kill(&mut killed);
}
Ok(Message::Kill) => kill(&mut killed),
Err(RecvTimeoutError::Timeout) => {}
Err(RecvTimeoutError::Disconnected) => break,
}
}
stop.store(true, Ordering::SeqCst);
// Ending the input channel ends the writer thread.
terminals.lock().unwrap().remove(&id);
groups.lock().unwrap().remove(&pid);
let _ = reader.join();
if aborted {
return Err(Failure::new("aborted", "aborted"));
}
match status {
Some(Ok(status)) => Ok(json!({ "exitCode": crate::sys::exit_code(status) })),
Some(Err(error)) => Err(Failure::new("spawn_error", error)),
None => Err(Failure::new(
"spawn_error",
"process did not report its exit",
)),
}
}
#[cfg(unix)]
mod unix {
use super::PtyRequest;
use std::fs::File;
use std::io::{self, Read, Write};
use std::os::fd::{AsRawFd, FromRawFd, OwnedFd};
use std::os::unix::process::CommandExt;
use std::process::{Child, Command, Stdio};
use std::sync::atomic::{AtomicBool, Ordering};
use serde_json::Value;
fn check(result: libc::c_int) -> io::Result<libc::c_int> {
if result < 0 {
Err(io::Error::last_os_error())
} else {
Ok(result)
}
}
fn winsize(cols: u16, rows: u16) -> libc::winsize {
libc::winsize {
ws_row: rows,
ws_col: cols,
ws_xpixel: 0,
ws_ypixel: 0,
}
}
pub fn set_size(master: &File, cols: u16, rows: u16) -> io::Result<()> {
let size = winsize(cols, rows);
// SAFETY: TIOCSWINSZ reads one winsize from a valid pointer.
check(unsafe { libc::ioctl(master.as_raw_fd(), libc::TIOCSWINSZ as _, &size) })?;
Ok(())
}
/// The terminal's foreground process group.
pub fn foreground(master: &File) -> io::Result<libc::pid_t> {
// SAFETY: tcgetpgrp on a descriptor the terminal owns.
let group = unsafe { libc::tcgetpgrp(master.as_raw_fd()) };
if group < 0 {
Err(io::Error::last_os_error())
} else {
Ok(group)
}
}
/// A process's name, if it still runs and the system tells.
#[cfg(target_vendor = "apple")]
pub fn process_name(pid: libc::pid_t) -> Option<String> {
let mut buffer = [0u8; 256];
// SAFETY: proc_name writes at most `buffer.len()` bytes and returns how many.
let length =
unsafe { libc::proc_name(pid, buffer.as_mut_ptr().cast(), buffer.len() as u32) };
(length > 0).then(|| String::from_utf8_lossy(&buffer[..length as usize]).into_owned())
}
#[cfg(any(target_os = "linux", target_os = "android"))]
pub fn process_name(pid: libc::pid_t) -> Option<String> {
let name = std::fs::read_to_string(format!("/proc/{pid}/comm")).ok()?;
Some(name.trim_end().to_string()).filter(|name| !name.is_empty())
}
#[cfg(not(any(target_vendor = "apple", target_os = "linux", target_os = "android")))]
pub fn process_name(_pid: libc::pid_t) -> Option<String> {
None
}
fn cloexec(fd: libc::c_int) -> io::Result<()> {
// SAFETY: fcntl on a descriptor this function's caller owns.
unsafe {
let flags = check(libc::fcntl(fd, libc::F_GETFD))?;
check(libc::fcntl(fd, libc::F_SETFD, flags | libc::FD_CLOEXEC))?;
}
Ok(())
}
/// The program to run: `argv`, or the user's shell as a login shell (`-` before its name, as `login` does).
fn command(request: &PtyRequest) -> Command {
match &request.argv {
Some(argv) => {
let mut command = Command::new(&argv[0]);
command.args(&argv[1..]);
command
}
None => {
let shell = request
.env
.get("SHELL")
.and_then(Value::as_str)
.map(str::to_string)
.or_else(|| std::env::var("SHELL").ok())
.filter(|shell| !shell.is_empty() && crate::sys::path_exists(shell))
.unwrap_or_else(|| "/bin/sh".into());
let name = shell.rsplit('/').next().unwrap_or("sh").to_string();
let mut command = Command::new(&shell);
command.arg0(format!("-{name}"));
command
}
}
}
pub fn spawn(request: &PtyRequest) -> io::Result<(File, Child)> {
let mut master: libc::c_int = -1;
let mut slave: libc::c_int = -1;
let mut size = winsize(request.cols, request.rows);
// SAFETY: openpty writes two descriptors; the name and termios are not requested.
check(unsafe {
libc::openpty(
&mut master,
&mut slave,
std::ptr::null_mut(),
std::ptr::null_mut(),
&raw mut size,
)
})?;
// SAFETY: openpty returned two new descriptors this function now owns.
let (master, slave) =
unsafe { (OwnedFd::from_raw_fd(master), OwnedFd::from_raw_fd(slave)) };
cloexec(master.as_raw_fd())?;
cloexec(slave.as_raw_fd())?;
let slave = File::from(slave);
let mut command = command(request);
command
.current_dir(&request.cwd)
.stdin(Stdio::from(slave.try_clone()?))
.stdout(Stdio::from(slave.try_clone()?))
.stderr(Stdio::from(slave.try_clone()?));
if !request.inherit_env {
command.env_clear();
}
for (key, value) in &request.env {
if let Some(value) = value.as_str() {
command.env(key, value);
}
}
// SAFETY: only async-signal-safe calls between fork and exec.
unsafe {
command.pre_exec(|| {
// A new session whose controlling terminal is the pty (stdin), so job control and SIGHUP work; default
// signal handling and an empty mask, as for `exec`.
if libc::setsid() < 0 {
return Err(io::Error::last_os_error());
}
if libc::ioctl(0, libc::TIOCSCTTY as _, 0) < 0 {
return Err(io::Error::last_os_error());
}
for signal in 1..32 {
if signal != libc::SIGKILL && signal != libc::SIGSTOP {
libc::signal(signal, libc::SIG_DFL);
}
}
let mut mask: libc::sigset_t = std::mem::zeroed();
libc::sigemptyset(&mut mask);
libc::sigprocmask(libc::SIG_SETMASK, &mask, std::ptr::null_mut());
Ok(())
});
}
let child = command.spawn()?;
// Only the program holds the terminal now, so reads end once it and its descendants close it.
drop(slave);
let master = File::from(master);
// SAFETY: fcntl on the descriptor owned above; reads and writes poll instead of blocking.
unsafe {
let flags = check(libc::fcntl(master.as_raw_fd(), libc::F_GETFL))?;
check(libc::fcntl(
master.as_raw_fd(),
libc::F_SETFL,
flags | libc::O_NONBLOCK,
))?;
}
Ok((master, child))
}
fn poll(fd: libc::c_int, events: libc::c_short) -> io::Result<bool> {
let mut poll = libc::pollfd {
fd,
events,
revents: 0,
};
// SAFETY: one valid pollfd.
let ready = unsafe { libc::poll(&mut poll, 1, 100) };
if ready < 0 {
let error = io::Error::last_os_error();
return if error.kind() == io::ErrorKind::Interrupted {
Ok(false)
} else {
Err(error)
};
}
Ok(ready > 0)
}
/// Read until the terminal closes (Linux reports `EIO` once no program holds it), `on_data` returns false, or
/// `stop` is set.
pub fn read(master: &File, stop: &AtomicBool, mut on_data: impl FnMut(&[u8]) -> bool) {
let mut buffer = vec![0u8; 64 * 1024];
while !stop.load(Ordering::SeqCst) {
match poll(master.as_raw_fd(), libc::POLLIN) {
Ok(false) => continue,
Ok(true) => {}
Err(_) => return,
}
match (&*master).read(&mut buffer) {
Ok(0) => return,
Ok(read) => {
if !on_data(&buffer[..read]) {
return;
}
}
Err(error)
if matches!(
error.kind(),
io::ErrorKind::WouldBlock | io::ErrorKind::Interrupted
) => {}
Err(_) => return,
}
}
}
/// Write all of `bytes`, waiting while the program does not read its input; false once the terminal is gone.
pub fn write_all(master: &File, mut bytes: &[u8], stop: &AtomicBool) -> bool {
while !bytes.is_empty() {
if stop.load(Ordering::SeqCst) {
return false;
}
match (&*master).write(bytes) {
Ok(0) => return false,
Ok(written) => bytes = &bytes[written..],
Err(error) if error.kind() == io::ErrorKind::WouldBlock => {
if poll(master.as_raw_fd(), libc::POLLOUT).is_err() {
return false;
}
}
Err(error) if error.kind() == io::ErrorKind::Interrupted => {}
Err(_) => return false,
}
}
true
}
}

215
third_party/pi/packages/env/daemon/src/scan.rs vendored Normal file
View File

@@ -0,0 +1,215 @@
//! Port of Durable's `LineScanner` (`env/line-scan.ts`): locates lines `[start_line, end_line)` of a file fed in order
//! and measures their decoded UTF-8 size as `new TextDecoder().decode(file)` would produce it.
use crate::decode::{decode_into, range_decoder, starts_with_bom};
use encoding_rs::Decoder;
use serde_json::{Value, json};
const NEWLINE: u8 = 0x0a;
pub struct LineScanner {
start_line: u64,
end_line: Option<u64>,
position: u64,
newlines: u64,
line_start: u64,
start: Option<u64>,
end: Option<u64>,
first_line_end: Option<u64>,
last_line_start: Option<u64>,
selected_bytes: u64,
first_line_bytes: u64,
selection: Option<Decoder>,
first_line: Option<Decoder>,
head: Option<Vec<u8>>,
bom: bool,
}
impl LineScanner {
/// `end_line`, when given, must be greater than `start_line`.
pub fn new(start_line: u64, end_line: Option<u64>) -> LineScanner {
let mut scanner = LineScanner {
start_line,
end_line,
position: 0,
newlines: 0,
line_start: 0,
start: None,
end: None,
first_line_end: None,
last_line_start: None,
selected_bytes: 0,
first_line_bytes: 0,
selection: None,
first_line: None,
head: Some(Vec::with_capacity(3)),
bom: false,
};
if start_line == 0 {
scanner.begin(0);
}
scanner
}
fn is_last_line(&self, line: u64) -> bool {
self.end_line.is_some_and(|end| line + 1 == end)
}
pub fn push(&mut self, mut chunk: &[u8]) {
if let Some(head) = &mut self.head {
let take = (3 - head.len()).min(chunk.len());
head.extend_from_slice(&chunk[..take]);
if head.len() < 3 {
return;
}
chunk = &chunk[take..];
self.release_head();
}
self.process(chunk);
}
fn release_head(&mut self) {
let head = self.head.take().unwrap_or_default();
self.bom = starts_with_bom(&head);
self.process(&head);
}
fn process(&mut self, chunk: &[u8]) {
let base = self.position;
let mut from = 0;
let mut index = 0;
while let Some(found) = chunk[index..].iter().position(|byte| *byte == NEWLINE) {
let newline = index + found;
// The newline ends line `self.newlines`; it belongs to the selection between selected lines only.
self.feed(chunk, base, from, newline);
let line = self.newlines;
let position = base + newline as u64;
if line == self.start_line {
self.end_first_line(position);
}
if self.is_last_line(line) {
self.end_selection(position);
}
self.feed(chunk, base, newline, newline + 1);
from = newline + 1;
self.newlines += 1;
self.line_start = position + 1;
if self.newlines == self.start_line {
self.begin(self.line_start);
}
if self.is_last_line(self.newlines) {
self.last_line_start = Some(self.line_start);
}
index = newline + 1;
}
self.feed(chunk, base, from, chunk.len());
self.position += chunk.len() as u64;
}
pub fn finish(mut self) -> Value {
if self.head.is_some() {
self.release_head();
}
let size = self.position;
let Some(start) = self.start else {
return json!({
"newlines": self.newlines, "start": size, "end": size, "firstLineEnd": size,
"lastLineStart": size, "selectedBytes": 0, "firstLineBytes": 0,
});
};
if self.first_line_end.is_none() {
self.end_first_line(size);
}
if self.end.is_none() {
self.end_selection(size);
}
json!({
"newlines": self.newlines,
"start": start,
"end": self.end.unwrap_or(size),
"firstLineEnd": self.first_line_end.unwrap_or(size),
// A selection that reaches past the last line ends with the last line.
"lastLineStart": self.last_line_start.unwrap_or(self.line_start),
"selectedBytes": self.selected_bytes,
"firstLineBytes": self.first_line_bytes,
})
}
fn begin(&mut self, start: u64) {
self.start = Some(start);
if self.is_last_line(self.start_line) {
self.last_line_start = Some(start);
}
self.selection = Some(range_decoder());
self.first_line = Some(range_decoder());
}
fn end_first_line(&mut self, position: u64) {
self.first_line_end = Some(position);
if let Some(mut decoder) = self.first_line.take() {
self.first_line_bytes += decode_into(&mut decoder, &[], true).len() as u64;
}
}
fn end_selection(&mut self, position: u64) {
self.end = Some(position);
if let Some(mut decoder) = self.selection.take() {
self.selected_bytes += decode_into(&mut decoder, &[], true).len() as u64;
}
}
fn feed(&mut self, chunk: &[u8], base: u64, mut from: usize, to: usize) {
// Decoding the whole file drops a leading byte-order mark.
if self.bom && base + (from as u64) < 3 {
from = to.min((3 - base) as usize);
}
if to <= from {
return;
}
let bytes = &chunk[from..to];
if let Some(decoder) = &mut self.selection {
self.selected_bytes += decode_into(decoder, bytes, false).len() as u64;
}
if let Some(decoder) = &mut self.first_line {
self.first_line_bytes += decode_into(decoder, bytes, false).len() as u64;
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn scan(file: &[u8], start: u64, end: Option<u64>, chunk: usize) -> Value {
let mut scanner = LineScanner::new(start, end);
for piece in file.chunks(chunk.max(1)) {
scanner.push(piece);
}
scanner.finish()
}
#[test]
fn locates_and_measures_lines_like_whole_file_decoding() {
// BOM, "a", invalid sequence, empty line, a later U+FEFF, and no final newline.
let file = [
0xef, 0xbb, 0xbf, 0x61, 0x0a, 0xe2, 0x82, 0x0a, 0x0a, 0xef, 0xbb, 0xbf, 0x62, 0x0a,
0xc3, 0xa9,
];
for chunk in 1..6 {
let all = scan(&file, 0, None, chunk);
assert_eq!(all["newlines"], 4);
// "a\n\u{fffd}\n\n\u{feff}b\né" is 1+1+3+1+1+3+1+1+2 = 14 bytes.
assert_eq!(all["selectedBytes"], 14);
assert_eq!(all["firstLineBytes"], 1);
let middle = scan(&file, 1, Some(3), chunk);
assert_eq!(
(middle["start"].as_u64(), middle["end"].as_u64()),
(Some(5), Some(8))
);
assert_eq!(middle["selectedBytes"], 4);
assert_eq!(middle["lastLineStart"], 8);
let past = scan(&file, 9, None, chunk);
assert_eq!(past["start"], 16);
}
}
}

91
third_party/pi/packages/env/daemon/src/sys/mod.rs vendored Normal file
View File

@@ -0,0 +1,91 @@
//! What differs between operating systems: error names, positional reads, temporary directories, opening readers,
//! and starting and killing commands. Each implementation follows what Node (libuv) does on that system.
#[cfg(unix)]
mod unix;
#[cfg(unix)]
pub use unix::*;
#[cfg(windows)]
mod windows;
#[cfg(windows)]
pub use windows::*;
/// How a string command runs: the shell, its arguments before the command, and whether the command goes to its
/// stdin instead of its arguments (legacy WSL `bash.exe`).
pub struct ShellConfig {
pub program: String,
pub args: Vec<String>,
pub command_on_stdin: bool,
}
/// How Node opens files: `r`, `writeFile` (`w`), `appendFile` (`a`), and `r+` for truncate and fsync.
#[derive(Clone, Copy)]
pub enum OpenMode {
/// Node's `open(path, "r")`: any kind of file, blocking reads.
Read,
Write,
Append,
ReadWrite,
}
/// An error code libuv reports where the system call itself reported another one, such as `ENOTDIR` from a recursive
/// `mkdir` through a file, or `EISDIR` when Windows creates a file where a directory exists.
#[derive(Debug)]
pub struct Synthetic(pub &'static str);
impl std::fmt::Display for Synthetic {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter.write_str(self.0)
}
}
impl std::error::Error for Synthetic {}
pub fn synthetic(code: &'static str) -> std::io::Error {
std::io::Error::other(Synthetic(code))
}
/// The libuv name of an error: a synthetic code, or the system's code translated per platform.
pub fn error_name(error: &std::io::Error) -> &'static str {
if let Some(Synthetic(code)) = error
.get_ref()
.and_then(|inner| inner.downcast_ref::<Synthetic>())
{
return code;
}
os_error_name(error)
}
/// Whether a path exists, following symbolic links, like Node's `access(path, F_OK)`.
pub fn path_exists(path: &str) -> bool {
std::path::Path::new(path).metadata().is_ok()
}
/// Sixteen random bytes for temporary names (not for secrets).
pub fn random_bytes() -> [u8; 16] {
use std::hash::{BuildHasher, Hasher};
use std::sync::atomic::{AtomicU64, Ordering};
static COUNTER: AtomicU64 = AtomicU64::new(0);
let state = std::collections::hash_map::RandomState::new();
let mut bytes = [0u8; 16];
for half in 0..2 {
let mut hasher = state.build_hasher();
hasher.write_u64(COUNTER.fetch_add(1, Ordering::Relaxed));
hasher.write_u128(
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|time| time.as_nanos())
.unwrap_or(0),
);
bytes[half * 8..half * 8 + 8].copy_from_slice(&hasher.finish().to_le_bytes());
}
bytes
}
/// The daemon's working directory, the `process.cwd()` Node's `path.resolve` falls back to.
pub fn cwd() -> String {
std::env::current_dir()
.map(|cwd| cwd.to_string_lossy().into_owned())
.unwrap_or_default()
}

323
third_party/pi/packages/env/daemon/src/sys/unix.rs vendored Normal file
View File

@@ -0,0 +1,323 @@
//! Linux, macOS and Android.
use super::{OpenMode, ShellConfig, path_exists};
use crate::errors::Failure;
use serde_json::{Map, Value};
use std::ffi::{CString, OsStr};
use std::fs::{File, Metadata, OpenOptions};
use std::io;
use std::os::unix::ffi::OsStrExt;
use std::os::unix::fs::{FileExt, MetadataExt, OpenOptionsExt};
use std::os::unix::process::{CommandExt, ExitStatusExt};
use std::process::{Child, Command, ExitStatus, Stdio};
use std::sync::atomic::{AtomicBool, Ordering};
/// libuv's name for an OS error.
pub fn os_error_name(error: &io::Error) -> &'static str {
let Some(errno) = error.raw_os_error() else {
return "UNKNOWN";
};
match errno {
libc::EPERM => "EPERM",
libc::ENOENT => "ENOENT",
libc::EIO => "EIO",
libc::EBADF => "EBADF",
libc::EAGAIN => "EAGAIN",
libc::ENOMEM => "ENOMEM",
libc::EACCES => "EACCES",
libc::EBUSY => "EBUSY",
libc::EEXIST => "EEXIST",
libc::EXDEV => "EXDEV",
libc::ENOTDIR => "ENOTDIR",
libc::EISDIR => "EISDIR",
libc::EINVAL => "EINVAL",
libc::ENFILE => "ENFILE",
libc::EMFILE => "EMFILE",
libc::ETXTBSY => "ETXTBSY",
libc::EFBIG => "EFBIG",
libc::ENOSPC => "ENOSPC",
libc::EROFS => "EROFS",
libc::EMLINK => "EMLINK",
libc::ENAMETOOLONG => "ENAMETOOLONG",
libc::ENOTEMPTY => "ENOTEMPTY",
libc::ELOOP => "ELOOP",
libc::ENOSYS => "ENOSYS",
libc::ENOTSUP => "ENOTSUP",
_ => "UNKNOWN",
}
}
pub const PATH_SEPARATOR: &str = "/";
pub fn read_at(file: &File, buffer: &mut [u8], offset: u64) -> io::Result<usize> {
file.read_at(buffer, offset)
}
/// Modification time as seconds and nanoseconds since the epoch.
pub fn mtime(metadata: &Metadata) -> (i64, i64) {
(metadata.mtime(), metadata.mtime_nsec())
}
/// Device and inode, which identify a file across renames.
pub fn identity(metadata: &Metadata) -> (u64, u64) {
(metadata.dev(), metadata.ino())
}
/// Node's `os.tmpdir()`; Termux's Node falls back to `$PREFIX/tmp`.
pub fn tmpdir() -> String {
let configured = ["TMPDIR", "TMP", "TEMP"]
.iter()
.find_map(|key| std::env::var(key).ok().filter(|value| !value.is_empty()));
let mut path = configured.unwrap_or_else(|| {
if cfg!(target_os = "android") {
format!(
"{}/tmp",
std::env::var("PREFIX")
.unwrap_or_else(|_| "/data/data/com.termux/files/usr".into())
)
} else {
"/tmp".into()
}
});
if path.len() > 1 && path.ends_with('/') {
path.pop();
}
path
}
pub fn home() -> String {
std::env::var("HOME").unwrap_or_default()
}
/// `mkdtemp(3)` of `prefix`, as libuv calls it: six random characters from `[A-Za-z0-9]`.
pub fn mkdtemp(prefix: &str) -> io::Result<String> {
let mut template = prefix.as_bytes().to_vec();
template.extend_from_slice(b"XXXXXX");
let template =
CString::new(template).map_err(|_| io::Error::from_raw_os_error(libc::EINVAL))?;
let mut buffer = template.into_bytes_with_nul();
// SAFETY: `buffer` is a NUL-terminated, writable template that mkdtemp fills in place.
let created = unsafe { libc::mkdtemp(buffer.as_mut_ptr().cast()) };
if created.is_null() {
return Err(io::Error::last_os_error());
}
buffer.pop();
Ok(OsStr::from_bytes(&buffer).to_string_lossy().into_owned())
}
/// Open a regular file for positional reads, without blocking on FIFOs and, with `no_follow`, without following a
/// final symbolic link.
pub fn open_reader(path: &str, no_follow: bool) -> Result<File, Failure> {
let mut flags = libc::O_NONBLOCK;
if no_follow {
flags |= libc::O_NOFOLLOW;
}
OpenOptions::new()
.read(true)
.custom_flags(flags)
.open(path)
.map_err(|error| {
if no_follow && matches!(error.raw_os_error(), Some(libc::ELOOP) | Some(libc::EMLINK)) {
Failure::new("SYMLINK", "Refusing to follow a symbolic link").path(path)
} else {
Failure::io(&error, "open", path)
}
})
}
/// Open a file in one of Node's write modes.
pub fn open(path: &str, mode: OpenMode) -> io::Result<File> {
let mut options = OpenOptions::new();
match mode {
OpenMode::Read => options.read(true),
OpenMode::Write => options.write(true).create(true).truncate(true),
OpenMode::Append => options.append(true).create(true),
OpenMode::ReadWrite => options.read(true).write(true),
};
options.open(path)
}
pub fn realpath(path: &str) -> io::Result<String> {
std::fs::canonicalize(path).map(|resolved| resolved.to_string_lossy().into_owned())
}
pub fn rename(from: &str, to: &str) -> io::Result<()> {
std::fs::rename(from, to)
}
pub fn remove_file(path: &str) -> io::Result<()> {
std::fs::remove_file(path)
}
/// A pipe a reader thread can poll, so that it stops when the command settles.
pub trait Pipe: io::Read + std::os::fd::AsRawFd {}
impl<T: io::Read + std::os::fd::AsRawFd> Pipe for T {}
/// Read `pipe` until it ends, `on_data` returns false, or `stop` is set; the pipe closes when this returns.
pub fn read_pipe(mut pipe: impl Pipe, stop: &AtomicBool, mut on_data: impl FnMut(&[u8]) -> bool) {
let fd = pipe.as_raw_fd();
// SAFETY: fcntl on a descriptor this thread owns.
unsafe {
let flags = libc::fcntl(fd, libc::F_GETFL);
libc::fcntl(fd, libc::F_SETFL, flags | libc::O_NONBLOCK);
}
let mut buffer = vec![0u8; 64 * 1024];
while !stop.load(Ordering::SeqCst) {
let mut poll = libc::pollfd {
fd,
events: libc::POLLIN,
revents: 0,
};
// SAFETY: one valid pollfd.
let ready = unsafe { libc::poll(&mut poll, 1, 100) };
if ready == 0
|| (ready < 0 && io::Error::last_os_error().kind() == io::ErrorKind::Interrupted)
{
continue;
}
if ready < 0 {
return;
}
match pipe.read(&mut buffer) {
Ok(0) => return,
Ok(read) => {
if !on_data(&buffer[..read]) {
return;
}
}
Err(error)
if matches!(
error.kind(),
io::ErrorKind::WouldBlock | io::ErrorKind::Interrupted
) => {}
Err(_) => return,
}
}
}
/// Readers poll `stop` every 100 ms.
pub fn interrupt_reader(_reader: &std::thread::JoinHandle<()>) {}
/// The program an argv command runs; `Command` searches `PATH` like `execvp`, as libuv does.
pub fn resolve_program(
program: &str,
_cwd: &str,
_env: &Map<String, Value>,
) -> Result<String, Failure> {
Ok(program.to_string())
}
fn which_bash() -> Option<String> {
let output = Command::new("which")
.arg("bash")
.stdin(Stdio::null())
.stderr(Stdio::null())
.output()
.ok()?;
if !output.status.success() {
return None;
}
let first = String::from_utf8_lossy(&output.stdout)
.trim()
.lines()
.next()?
.to_string();
path_exists(&first).then_some(first)
}
/// Node's shell resolution: a configured shell must exist; otherwise `/bin/bash`, `which bash`, then `sh`.
pub fn shell_config(shell_path: Option<&str>) -> Result<ShellConfig, Failure> {
let shell = |program: &str| ShellConfig {
program: program.into(),
args: vec!["-c".into()],
command_on_stdin: false,
};
if let Some(configured) = shell_path {
if path_exists(configured) {
return Ok(shell(configured));
}
return Err(Failure::new(
"shell_unavailable",
format!("Custom shell path not found: {configured}"),
));
}
if path_exists("/bin/bash") {
return Ok(shell("/bin/bash"));
}
Ok(shell(&which_bash().unwrap_or_else(|| "sh".into())))
}
/// Arguments, environment and process setup as libuv applies them.
pub fn configure(
command: &mut Command,
args: &[String],
env: &Map<String, Value>,
inherit_env: bool,
) {
command.args(args);
if !inherit_env {
command.env_clear();
}
for (key, value) in env {
if let Some(value) = value.as_str() {
command.env(key, value);
}
}
// SAFETY: only async-signal-safe calls between fork and exec.
unsafe {
command.pre_exec(|| {
// A new session, so a kill reaches every descendant; default signal handling and an empty mask, as libuv sets.
libc::setsid();
for signal in 1..32 {
if signal != libc::SIGKILL && signal != libc::SIGSTOP {
libc::signal(signal, libc::SIG_DFL);
}
}
let mut mask: libc::sigset_t = std::mem::zeroed();
libc::sigemptyset(&mut mask);
libc::sigprocmask(libc::SIG_SETMASK, &mask, std::ptr::null_mut());
Ok(())
});
}
}
pub fn after_spawn(_child: &Child) {}
/// Kill the command's process group, or the process if the group is gone.
pub fn kill_tree(pid: u32) {
let pid = pid as i32;
// SAFETY: plain signal delivery.
unsafe {
if libc::kill(-pid, libc::SIGKILL) != 0 {
libc::kill(pid, libc::SIGKILL);
}
}
}
/// A process killed by a signal has no exit code; report 128 + the signal, as the shell does.
pub fn exit_code(status: ExitStatus) -> i64 {
i64::from(
status
.code()
.unwrap_or_else(|| 128 + status.signal().unwrap_or(0)),
)
}
/// The bytes of a file name, which libuv sorts directory listings by.
pub fn raw_name(name: &OsStr) -> Option<Vec<u8>> {
Some(name.as_bytes().to_vec())
}
/// Windows only: the per-drive working directories of the `=C:` variables.
pub fn drive_cwds() -> Map<String, Value> {
Map::new()
}
pub fn read_dir(path: &str) -> io::Result<std::fs::ReadDir> {
std::fs::read_dir(path)
}
/// Device and inode of `path`; the metadata already has them.
pub fn path_identity(_path: &str, metadata: &Metadata, _follow: bool) -> (u64, u64) {
identity(metadata)
}

665
third_party/pi/packages/env/daemon/src/sys/windows.rs vendored Normal file
View File

@@ -0,0 +1,665 @@
//! Windows, following libuv (`src/win/error.c`, `src/win/process.c`, `src/win/fs.c`) and Node's `child_process`.
use super::{OpenMode, ShellConfig, path_exists, synthetic};
use crate::errors::Failure;
use serde_json::{Map, Value};
use std::collections::hash_map::RandomState;
use std::ffi::OsStr;
use std::fs::{File, Metadata};
use std::hash::{BuildHasher, Hasher};
use std::io;
use std::os::windows::ffi::OsStrExt;
use std::os::windows::fs::FileExt;
use std::os::windows::io::{AsRawHandle, FromRawHandle};
use std::os::windows::process::CommandExt;
use std::path::Path;
use std::process::{Child, Command, ExitStatus, Stdio};
use std::sync::OnceLock;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::time::UNIX_EPOCH;
use windows_sys::Win32::Foundation::{CloseHandle, INVALID_HANDLE_VALUE};
use windows_sys::Win32::Storage::FileSystem::{
BY_HANDLE_FILE_INFORMATION, CREATE_ALWAYS, CreateFileW, FILE_APPEND_DATA,
FILE_ATTRIBUTE_NORMAL, FILE_FLAG_BACKUP_SEMANTICS, FILE_FLAG_OPEN_REPARSE_POINT,
FILE_GENERIC_READ, FILE_GENERIC_WRITE, FILE_SHARE_DELETE, FILE_SHARE_READ, FILE_SHARE_WRITE,
FILE_WRITE_DATA, GetFileInformationByHandle, MOVEFILE_REPLACE_EXISTING, MoveFileExW,
OPEN_ALWAYS, OPEN_EXISTING,
};
use windows_sys::Win32::System::IO::CancelSynchronousIo;
use windows_sys::Win32::System::JobObjects::{
AssignProcessToJobObject, CreateJobObjectW, JOB_OBJECT_LIMIT_BREAKAWAY_OK,
JOB_OBJECT_LIMIT_DIE_ON_UNHANDLED_EXCEPTION, JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE,
JOB_OBJECT_LIMIT_SILENT_BREAKAWAY_OK, JOBOBJECT_EXTENDED_LIMIT_INFORMATION,
JobObjectExtendedLimitInformation, SetInformationJobObject,
};
use windows_sys::Win32::System::Threading::CREATE_NO_WINDOW;
/// libuv's `uv_translate_sys_error` for the errors file and process operations produce.
pub fn os_error_name(error: &io::Error) -> &'static str {
let Some(code) = error.raw_os_error() else {
return "UNKNOWN";
};
match code {
1 => "EISDIR", // ERROR_INVALID_FUNCTION
2 | 3 | 15 | 123 | 161 | 4392 => "ENOENT", // FILE_NOT_FOUND, PATH_NOT_FOUND, INVALID_DRIVE, INVALID_NAME, BAD_PATHNAME, INVALID_REPARSE_DATA
4 => "EMFILE", // ERROR_TOO_MANY_OPEN_FILES
5 | 1314 => "EPERM", // ACCESS_DENIED, PRIVILEGE_NOT_HELD
6 => "EBADF", // ERROR_INVALID_HANDLE
8 | 14 => "ENOMEM", // NOT_ENOUGH_MEMORY, OUTOFMEMORY
17 => "EXDEV", // ERROR_NOT_SAME_DEVICE
19 => "EROFS", // ERROR_WRITE_PROTECT
32 | 33 => "EBUSY", // SHARING_VIOLATION, LOCK_VIOLATION
39 | 82 | 112 => "ENOSPC", // HANDLE_DISK_FULL, CANNOT_MAKE, DISK_FULL
50 => "ENOTSUP", // ERROR_NOT_SUPPORTED
80 | 183 => "EEXIST", // FILE_EXISTS, ALREADY_EXISTS
13 | 87 | 122 | 131 | 1464 => "EINVAL", // INVALID_DATA, INVALID_PARAMETER, INSUFFICIENT_BUFFER, NEGATIVE_SEEK, SYMLINK_NOT_SUPPORTED
126 => "ENOENT", // ERROR_MOD_NOT_FOUND
145 => "ENOTEMPTY", // ERROR_DIR_NOT_EMPTY
206 => "ENAMETOOLONG", // ERROR_FILENAME_EXCED_RANGE
267 => "ENOENT", // ERROR_DIRECTORY (libuv maps it to ENOENT)
998 => "EFAULT", // ERROR_NOACCESS
740 | 1920 => "EACCES", // ELEVATION_REQUIRED, CANT_ACCESS_FILE
1921 => "ELOOP", // ERROR_CANT_RESOLVE_FILENAME
_ => "UNKNOWN",
}
}
pub const PATH_SEPARATOR: &str = "\\";
pub fn read_at(file: &File, buffer: &mut [u8], offset: u64) -> io::Result<usize> {
file.seek_read(buffer, offset)
}
/// Modification time as seconds and nanoseconds since the epoch; before 1970 the seconds are negative.
pub fn mtime(metadata: &Metadata) -> (i64, i64) {
let Ok(time) = metadata.modified() else {
return (0, 0);
};
match time.duration_since(UNIX_EPOCH) {
Ok(after) => (after.as_secs() as i64, i64::from(after.subsec_nanos())),
Err(before) => {
let before = before.duration();
let nanos = i64::from(before.subsec_nanos());
if nanos == 0 {
(-(before.as_secs() as i64), 0)
} else {
(-(before.as_secs() as i64) - 1, 1_000_000_000 - nanos)
}
}
}
}
/// File index numbers need a handle per file; the creation time tells a replaced file or directory apart instead.
pub fn identity(metadata: &Metadata) -> (u64, u64) {
use std::os::windows::fs::MetadataExt;
(0, metadata.creation_time())
}
/// Node's `os.tmpdir()` on Windows.
pub fn tmpdir() -> String {
let variable = |key: &str| std::env::var(key).ok().filter(|value| !value.is_empty());
let mut path = variable("TEMP")
.or_else(|| variable("TMP"))
.unwrap_or_else(|| {
format!(
"{}\\temp",
variable("SystemRoot")
.or_else(|| variable("windir"))
.unwrap_or_default()
)
});
if path.len() > 1 && path.ends_with('\\') && !path.ends_with(":\\") {
path.pop();
}
path
}
/// libuv's `uv_os_homedir`: `USERPROFILE` first.
pub fn home() -> String {
std::env::var("USERPROFILE").unwrap_or_default()
}
/// A NUL-terminated wide path; an embedded NUL fails like the standard library's path conversion (Node rejects such
/// paths before any system call, which the client reports with the same code).
fn wide(text: &str) -> io::Result<Vec<u16>> {
if text.contains('\0') {
return Err(io::Error::new(
io::ErrorKind::InvalidInput,
"path contains a NUL byte",
));
}
Ok(OsStr::new(text)
.encode_wide()
.chain(std::iter::once(0))
.collect())
}
/// libuv's `fs__mkdtemp`: `prefix` plus six characters from `[a-zA-Z0-9]`, retried while the name exists.
pub fn mkdtemp(prefix: &str) -> io::Result<String> {
const CHARACTERS: &[u8] = b"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
static COUNTER: AtomicU64 = AtomicU64::new(0);
let state = RandomState::new();
for _ in 0..1000 {
let mut hasher = state.build_hasher();
hasher.write_u64(COUNTER.fetch_add(1, Ordering::Relaxed));
let mut random = hasher.finish();
let mut path = prefix.to_string();
for _ in 0..6 {
path.push(CHARACTERS[(random % CHARACTERS.len() as u64) as usize] as char);
random /= CHARACTERS.len() as u64;
}
match std::fs::create_dir(&path) {
Ok(()) => return Ok(path),
Err(error) if error.kind() == io::ErrorKind::AlreadyExists => continue,
Err(error) => return Err(error),
}
}
Err(io::Error::from_raw_os_error(80))
}
/// Windows cannot open a directory as a file and has no `O_NOFOLLOW`: check the path first, as `NodeExecutionEnv` does
/// there (not race-free).
pub fn open_reader(path: &str, no_follow: bool) -> Result<File, Failure> {
if no_follow
&& std::fs::symlink_metadata(path).is_ok_and(|metadata| metadata.file_type().is_symlink())
{
return Err(Failure::new("SYMLINK", "Refusing to follow a symbolic link").path(path));
}
if std::fs::metadata(path).is_ok_and(|metadata| metadata.is_dir()) {
return Err(
Failure::new("EISDIR", "EISDIR: illegal operation on a directory, read").path(path),
);
}
File::open(path).map_err(|error| Failure::io(&error, "open", path))
}
/// libuv's `fs__open` for Node's write modes: `CreateFileW` with its access, sharing and disposition, always with
/// `FILE_FLAG_BACKUP_SEMANTICS`, so a directory opens and later calls on it fail as in Node; creating a file where a
/// directory exists reports `EISDIR`. (Rust's `File::create` opens with `OPEN_ALWAYS` and truncates afterwards.)
pub fn open(path: &str, mode: OpenMode) -> io::Result<File> {
let (access, disposition) = match mode {
OpenMode::Read => (FILE_GENERIC_READ, OPEN_EXISTING),
OpenMode::Write => (FILE_GENERIC_WRITE, CREATE_ALWAYS),
OpenMode::Append => (
(FILE_GENERIC_WRITE & !FILE_WRITE_DATA) | FILE_APPEND_DATA,
OPEN_ALWAYS,
),
OpenMode::ReadWrite => (FILE_GENERIC_READ | FILE_GENERIC_WRITE, OPEN_EXISTING),
};
let name = wide(path)?;
// SAFETY: a NUL-terminated wide path; the returned handle is owned by the File on success.
let handle = unsafe {
CreateFileW(
name.as_ptr(),
access,
FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
std::ptr::null(),
disposition,
FILE_ATTRIBUTE_NORMAL | FILE_FLAG_BACKUP_SEMANTICS,
std::ptr::null_mut(),
)
};
if handle == INVALID_HANDLE_VALUE {
let error = io::Error::last_os_error();
// ERROR_FILE_EXISTS while creating means the path is a directory.
if error.raw_os_error() == Some(80) && matches!(mode, OpenMode::Write | OpenMode::Append) {
return Err(synthetic("EISDIR"));
}
return Err(error);
}
// SAFETY: `handle` is a valid, owned file handle.
Ok(unsafe { File::from_raw_handle(handle as _) })
}
/// libuv's `uv_fs_realpath`: the final path without the `\\?\` prefix.
pub fn realpath(path: &str) -> io::Result<String> {
let resolved = std::fs::canonicalize(path)?.to_string_lossy().into_owned();
Ok(if let Some(unc) = resolved.strip_prefix("\\\\?\\UNC\\") {
format!("\\\\{unc}")
} else {
resolved
.strip_prefix("\\\\?\\")
.map(str::to_string)
.unwrap_or(resolved)
})
}
/// libuv's `fs__rename`: `MoveFileExW` replacing an existing destination.
pub fn rename(from: &str, to: &str) -> io::Result<()> {
let (from, to) = (wide(from)?, wide(to)?);
// SAFETY: both arguments are NUL-terminated wide strings that outlive the call.
if unsafe { MoveFileExW(from.as_ptr(), to.as_ptr(), MOVEFILE_REPLACE_EXISTING) } == 0 {
return Err(io::Error::last_os_error());
}
Ok(())
}
/// Node's `rm` on Windows: a symbolic link or junction to a directory is removed as a directory entry (libuv's
/// `fs__unlink`), and a read-only file is retried after making it writable.
pub fn remove_file(path: &str) -> io::Result<()> {
use std::os::windows::fs::FileTypeExt;
if std::fs::symlink_metadata(path).is_ok_and(|metadata| metadata.file_type().is_symlink_dir()) {
return std::fs::remove_dir(path);
}
match std::fs::remove_file(path) {
Err(error) if error.kind() == io::ErrorKind::PermissionDenied => {
let mut permissions = std::fs::metadata(path)?.permissions();
#[allow(clippy::permissions_set_readonly_false)]
permissions.set_readonly(false);
std::fs::set_permissions(path, permissions)?;
std::fs::remove_file(path)
}
other => other,
}
}
/// A pipe a reader thread reads with blocking calls; `interrupt_reader` cancels a blocked read.
pub trait Pipe: io::Read {}
impl<T: io::Read> Pipe for T {}
/// Read `pipe` until it ends, `on_data` returns false, or `stop` is set; the pipe closes when this returns.
pub fn read_pipe(mut pipe: impl Pipe, stop: &AtomicBool, mut on_data: impl FnMut(&[u8]) -> bool) {
let mut buffer = vec![0u8; 64 * 1024];
while !stop.load(Ordering::SeqCst) {
match pipe.read(&mut buffer) {
Ok(0) | Err(_) => return,
Ok(read) => {
if !on_data(&buffer[..read]) {
return;
}
}
}
}
}
/// Cancel a reader blocked in `ReadFile`, so it sees `stop` and closes its pipe.
pub fn interrupt_reader(reader: &std::thread::JoinHandle<()>) {
// SAFETY: the handle of a live or finished thread; cancelling no pending I/O is harmless.
unsafe {
CancelSynchronousIo(reader.as_raw_handle() as _);
}
}
/// libuv's `search_path` for an argv command: a name with a directory is tried as given, otherwise the working
/// directory and then each `PATH` entry; a name without an extension also tries `.com` and `.exe`. Node refuses
/// batch files without a shell.
pub fn resolve_program(
program: &str,
cwd: &str,
env: &Map<String, Value>,
) -> Result<String, Failure> {
let not_found = || Failure::new("spawn_error", format!("spawn {program} ENOENT"));
let has_extension = Path::new(program).extension().is_some();
let candidates = |base: &Path| -> Vec<std::path::PathBuf> {
let mut names = Vec::new();
if has_extension {
names.push(base.to_path_buf());
}
for extension in ["com", "exe"] {
let mut name = base.as_os_str().to_owned();
name.push(".");
name.push(extension);
names.push(name.into());
}
names
};
let found = if program.contains(['\\', '/', ':']) {
candidates(&Path::new(cwd).join(program))
.into_iter()
.find(|path| path.is_file())
} else {
let path_variable = env
.iter()
.find(|(key, _)| key.eq_ignore_ascii_case("PATH"))
.and_then(|(_, value)| value.as_str().map(str::to_string))
.or_else(|| std::env::var("PATH").ok())
.unwrap_or_default();
std::iter::once(std::path::PathBuf::from(cwd))
.chain(std::env::split_paths(&path_variable))
.flat_map(|directory| candidates(&directory.join(program)))
.find(|path| path.is_file())
};
let found = found.ok_or_else(not_found)?;
let lower = found.to_string_lossy().to_lowercase();
if lower.ends_with(".bat") || lower.ends_with(".cmd") {
return Err(Failure::new(
"spawn_error",
format!("spawn {program} EINVAL"),
));
}
Ok(found.to_string_lossy().into_owned())
}
/// Node's `isLegacyWslBashPath`: `^[a-z]:\\windows\\(system32|sysnative)\\bash\.exe$`, case-insensitive.
fn is_legacy_wsl_bash(path: &str) -> bool {
let normalized = path.replace('/', "\\").to_lowercase();
let mut characters = normalized.chars();
characters
.next()
.is_some_and(|drive| drive.is_ascii_lowercase())
&& matches!(
characters.as_str(),
":\\windows\\system32\\bash.exe" | ":\\windows\\sysnative\\bash.exe"
)
}
fn bash_config(program: &str) -> ShellConfig {
if is_legacy_wsl_bash(program) {
ShellConfig {
program: program.into(),
args: vec!["-s".into()],
command_on_stdin: true,
}
} else {
ShellConfig {
program: program.into(),
args: vec!["-c".into()],
command_on_stdin: false,
}
}
}
fn where_bash() -> Option<String> {
let output = Command::new("where")
.arg("bash.exe")
.creation_flags(CREATE_NO_WINDOW)
.stdin(Stdio::null())
.stderr(Stdio::null())
.output()
.ok()?;
if !output.status.success() {
return None;
}
let first = String::from_utf8_lossy(&output.stdout)
.trim()
.lines()
.next()?
.trim()
.to_string();
path_exists(&first).then_some(first)
}
/// Node's shell resolution on Windows: a configured shell, Git Bash, `where bash.exe`, or `shell_unavailable`.
pub fn shell_config(shell_path: Option<&str>) -> Result<ShellConfig, Failure> {
if let Some(configured) = shell_path {
if path_exists(configured) {
return Ok(bash_config(configured));
}
return Err(Failure::new(
"shell_unavailable",
format!("Custom shell path not found: {configured}"),
));
}
let mut candidates = Vec::new();
for variable in ["ProgramFiles", "ProgramFiles(x86)"] {
if let Ok(directory) = std::env::var(variable) {
candidates.push(format!("{directory}\\Git\\bin\\bash.exe"));
}
}
if let Some(found) = candidates.iter().find(|candidate| path_exists(candidate)) {
return Ok(bash_config(found));
}
if let Some(found) = where_bash() {
return Ok(bash_config(&found));
}
let searched: String = candidates
.iter()
.map(|candidate| format!(" {candidate}\n"))
.collect();
Err(Failure::new(
"shell_unavailable",
format!(
"No bash shell found. Options:\n 1. Install Git for Windows: https://git-scm.com/download/win\n 2. Add your bash to PATH (Cygwin, MSYS2, etc.)\n 3. Configure an explicit shellPath\n\nSearched Git Bash in:\n{}",
searched.trim_end()
),
))
}
/// libuv's `quote_cmd_arg`: how one argument is written into the command line the child splits again.
pub fn quote_argument(argument: &str) -> String {
if argument.is_empty() {
return "\"\"".into();
}
if !argument.contains([' ', '\t', '"']) {
return argument.into();
}
if !argument.contains(['"', '\\']) {
return format!("\"{argument}\"");
}
// Walk backwards: backslashes before a quote (or the closing quote) are doubled, and quotes are escaped.
let mut reversed = Vec::new();
let mut quote_hit = true;
for character in argument.chars().rev() {
reversed.push(character);
if quote_hit && character == '\\' {
reversed.push('\\');
} else if character == '"' {
quote_hit = true;
reversed.push('\\');
} else {
quote_hit = false;
}
}
let mut quoted = String::from("\"");
quoted.extend(reversed.into_iter().rev());
quoted.push('"');
quoted
}
/// Variables libuv copies from the parent into every child environment that lacks them.
const REQUIRED_VARIABLES: [&str; 11] = [
"HOMEDRIVE",
"HOMEPATH",
"LOGONSERVER",
"PATH",
"SYSTEMDRIVE",
"SYSTEMROOT",
"TEMP",
"USERDOMAIN",
"USERNAME",
"USERPROFILE",
"WINDIR",
];
/// The child environment as Node builds it on Windows: the variables in object order, sorted by UTF-16 code units,
/// keeping the first of names that differ only in case; then libuv adds required variables the result lacks.
fn child_environment(env: &Map<String, Value>, inherit_env: bool) -> Vec<(String, String)> {
let mut entries: Vec<(String, String)> = Vec::new();
let mut set =
|key: String, value: String| match entries.iter_mut().find(|(name, _)| *name == key) {
Some(entry) => entry.1 = value,
None => entries.push((key, value)),
};
if inherit_env {
for (key, value) in std::env::vars_os() {
set(
key.to_string_lossy().into_owned(),
value.to_string_lossy().into_owned(),
);
}
}
for (key, value) in env {
if let Some(value) = value.as_str() {
set(key.clone(), value.to_string());
}
}
entries.sort_by(|(a, _), (b, _)| a.encode_utf16().cmp(b.encode_utf16()));
let mut seen = std::collections::HashSet::new();
entries.retain(|(name, _)| seen.insert(name.to_uppercase()));
for required in REQUIRED_VARIABLES {
if !entries
.iter()
.any(|(name, _)| name.eq_ignore_ascii_case(required))
&& let Ok(value) = std::env::var(required)
{
entries.push((required.to_string(), value));
}
}
entries
}
/// Arguments quoted as libuv quotes them, Node's environment, and a hidden console.
pub fn configure(
command: &mut Command,
args: &[String],
env: &Map<String, Value>,
inherit_env: bool,
) {
for argument in args {
command.raw_arg(quote_argument(argument));
}
command.env_clear();
for (key, value) in child_environment(env, inherit_env) {
command.env(key, value);
}
command.creation_flags(CREATE_NO_WINDOW);
}
/// libuv puts every child it starts into one job that ends its processes when the daemon exits.
fn job() -> Option<isize> {
static JOB: OnceLock<Option<isize>> = OnceLock::new();
*JOB.get_or_init(|| {
// SAFETY: plain Win32 calls with a zeroed, correctly sized information struct.
unsafe {
let job = CreateJobObjectW(std::ptr::null(), std::ptr::null());
if job.is_null() {
return None;
}
let mut information: JOBOBJECT_EXTENDED_LIMIT_INFORMATION = std::mem::zeroed();
information.BasicLimitInformation.LimitFlags = JOB_OBJECT_LIMIT_BREAKAWAY_OK
| JOB_OBJECT_LIMIT_SILENT_BREAKAWAY_OK
| JOB_OBJECT_LIMIT_DIE_ON_UNHANDLED_EXCEPTION
| JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE;
SetInformationJobObject(
job,
JobObjectExtendedLimitInformation,
(&information as *const JOBOBJECT_EXTENDED_LIMIT_INFORMATION).cast(),
std::mem::size_of::<JOBOBJECT_EXTENDED_LIMIT_INFORMATION>() as u32,
);
Some(job as isize)
}
})
}
pub fn after_spawn(child: &Child) {
if let Some(job) = job() {
// SAFETY: both handles are valid for the duration of the call.
unsafe {
AssignProcessToJobObject(job as _, child.as_raw_handle() as _);
}
}
}
/// Node's `killProcessTree` on Windows: `taskkill /F /T`, not waited for.
pub fn kill_tree(pid: u32) {
let system_root = std::env::var("SystemRoot").unwrap_or_else(|_| "C:\\Windows".into());
let _ = Command::new(format!("{system_root}\\System32\\taskkill.exe"))
.args(["/F", "/T", "/PID", &pid.to_string()])
.creation_flags(CREATE_NO_WINDOW)
.stdin(Stdio::null())
.stdout(Stdio::null())
.stderr(Stdio::null())
.spawn();
}
/// Exit codes are unsigned 32-bit values on Windows (an access violation is 3221225477).
pub fn exit_code(status: ExitStatus) -> i64 {
i64::from(status.code().unwrap_or(1) as u32)
}
/// Windows listings keep the file system's order; nothing sorts by raw names.
pub fn raw_name(_name: &OsStr) -> Option<Vec<u8>> {
None
}
/// The per-drive working directories of the `=C:` variables, which Node's `path.resolve` uses for drive-relative paths.
pub fn drive_cwds() -> Map<String, Value> {
std::env::vars_os()
.filter_map(|(key, value)| {
let key = key.to_str()?;
let bytes = key.as_bytes();
(bytes.len() == 3
&& bytes[0] == b'='
&& bytes[1].is_ascii_alphabetic()
&& bytes[2] == b':')
.then(|| {
(
key[1..].to_ascii_uppercase(),
Value::String(value.to_string_lossy().into_owned()),
)
})
})
.collect()
}
/// libuv's `scandir` and `opendir` report a file as `ENOTDIR` (elsewhere `ERROR_DIRECTORY` is `ENOENT`).
pub fn read_dir(path: &str) -> io::Result<std::fs::ReadDir> {
std::fs::read_dir(path).map_err(|error| {
if error.raw_os_error() == Some(267) {
synthetic("ENOTDIR")
} else {
error
}
})
}
/// Volume serial number and file index of `path`, as libuv's `stat` reports `dev` and `ino`: the creation time is not
/// enough, because NTFS gives a file created under a recently removed name the old one's creation time ("tunneling").
/// Opens a handle, like libuv; falls back to the metadata when the path cannot be opened.
pub fn path_identity(path: &str, metadata: &Metadata, follow: bool) -> (u64, u64) {
let Ok(name) = wide(path) else {
return identity(metadata);
};
let flags = FILE_FLAG_BACKUP_SEMANTICS
| if follow {
0
} else {
FILE_FLAG_OPEN_REPARSE_POINT
};
// SAFETY: a NUL-terminated wide path; the handle is closed below.
let handle = unsafe {
CreateFileW(
name.as_ptr(),
0,
FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
std::ptr::null(),
OPEN_EXISTING,
flags,
std::ptr::null_mut(),
)
};
if handle == INVALID_HANDLE_VALUE {
return identity(metadata);
}
// SAFETY: a valid handle and a zeroed buffer for the call to fill.
let mut information: BY_HANDLE_FILE_INFORMATION = unsafe { std::mem::zeroed() };
let found = unsafe { GetFileInformationByHandle(handle, &mut information) } != 0;
// SAFETY: the handle opened above.
unsafe { CloseHandle(handle) };
if !found {
return identity(metadata);
}
(
u64::from(information.dwVolumeSerialNumber),
(u64::from(information.nFileIndexHigh) << 32) | u64::from(information.nFileIndexLow),
)
}
#[cfg(test)]
mod tests {
use super::quote_argument;
#[test]
fn quotes_arguments_like_libuv() {
assert_eq!(quote_argument(""), "\"\"");
assert_eq!(quote_argument("plain"), "plain");
assert_eq!(quote_argument("a b"), "\"a b\"");
assert_eq!(quote_argument("*.txt"), "*.txt");
assert_eq!(quote_argument("say \"hi\""), "\"say \\\"hi\\\"\"");
assert_eq!(
quote_argument("C:\\dir with space\\"),
"\"C:\\dir with space\\\\\""
);
assert_eq!(quote_argument("a\\\\b c"), "\"a\\\\b c\"");
}
}

778
third_party/pi/packages/env/daemon/src/watch.rs vendored Normal file
View File

@@ -0,0 +1,778 @@
//! `watch`: Durable's `NodeFileWatcher` (packages/durable/src/env/node-watch.ts) running next to the files, so a scan
//! costs no round trips. Native events only trigger a debounced rescan; changes are the difference between snapshots
//! plus the event paths. Events go out as `change` events of the request, which lasts until it is cancelled.
use crate::errors::Failure;
use crate::exec::Control;
use crate::frame::{EVENT, Frame};
use crate::output::Output;
use crate::sys;
use notify::{Event, EventKind, RecommendedWatcher, RecursiveMode, Watcher};
use serde_json::{Value, json};
use std::collections::{HashMap, HashSet};
use std::fs::{self, Metadata};
use std::hash::Hasher;
use std::io;
use std::path::Path;
use std::sync::atomic::Ordering;
use std::sync::mpsc::{self, RecvTimeoutError, Sender};
use std::time::{Duration, Instant, SystemTime};
const DEBOUNCE: Duration = Duration::from_millis(50);
/// macOS: FSEvents streams go live after `watch` returns; a rescan this long after installing watchers catches changes
/// made in between.
const FSEVENTS_SETTLE: Duration = Duration::from_millis(500);
const DEFAULT_POLL_MS: u64 = 2000;
const DEFAULT_MAX_DIRECTORIES: usize = 10_000;
/// In polling mode, recently modified small files are also compared by content.
const HASH_MAX_BYTES: u64 = 256 * 1024;
const HASH_RECENT: Duration = Duration::from_secs(5);
/// How often a waiting watcher looks for a cancel.
const CANCEL_POLL: Duration = Duration::from_millis(100);
/// Linux `statfs` magic numbers of file systems that accept watches but do not report changes made elsewhere.
#[cfg(any(target_os = "linux", target_os = "android"))]
const UNRELIABLE_FILE_SYSTEMS: [u64; 12] = [
0x6969, // NFS
0x517b, // SMB
0xff534d42, // CIFS
0xfe534d42, // SMB2
0x65735546, // FUSE (sshfs, Android shared storage)
0x01021997, // 9P (WSL2 Windows drives)
0x0bd00bd0, // Lustre
0x47504653, // GPFS
0x00c36400, // Ceph
0x5346414f, // OpenAFS
0x6b414653, // kAFS
0x5dca2df5, // sdcardfs
];
struct Target {
path: String,
recursive: bool,
hidden: bool,
names: HashSet<String>,
}
#[derive(Clone, Copy, PartialEq, Eq)]
enum Kind {
File,
Directory,
Symlink,
Other,
}
/// What a snapshot remembers of one path. Directories and ancestors are compared by identity only.
#[derive(Clone, PartialEq, Eq)]
struct Entry {
kind: Kind,
dev: u64,
ino: u64,
size: u64,
mtime: (i64, i64),
hash: Option<u64>,
}
type Snapshot = HashMap<String, Entry>;
#[derive(Clone, Copy, PartialEq, Eq)]
enum Mode {
Native,
Polling,
}
impl Mode {
fn name(self) -> &'static str {
match self {
Mode::Native => "native",
Mode::Polling => "polling",
}
}
}
enum ScanError {
Budget(usize),
Io(io::Error, &'static str, String),
}
impl ScanError {
/// The error `NodeExecutionEnv.watch` returns when opening fails.
fn failure(&self) -> Failure {
match self {
ScanError::Budget(max) => {
Failure::new("EINVAL", format!("Watched paths exceed {max} directories"))
}
ScanError::Io(error, syscall, path) => Failure::io(error, syscall, path),
}
}
/// The `FileError` code `NodeFileWatcher` reports when a later rescan fails.
fn code(&self) -> &'static str {
match self {
ScanError::Io(error, ..) if denied(error) => "permission_denied",
_ => "invalid",
}
}
}
fn denied(error: &io::Error) -> bool {
error.kind() == io::ErrorKind::PermissionDenied
}
fn kind_of(metadata: &Metadata) -> Kind {
let file_type = metadata.file_type();
if file_type.is_file() {
Kind::File
} else if file_type.is_dir() {
Kind::Directory
} else if file_type.is_symlink() {
Kind::Symlink
} else {
Kind::Other
}
}
/// What a snapshot remembers of `path`, whose `metadata` came from `stat` (`follow`) or `lstat`.
fn entry_of(path: &str, metadata: &Metadata, follow: bool, hash: Option<u64>) -> Entry {
let kind = kind_of(metadata);
let (dev, ino) = sys::path_identity(path, metadata, follow);
let directory = kind == Kind::Directory;
Entry {
kind,
dev,
ino,
size: if directory { 0 } else { metadata.len() },
mtime: if directory {
(0, 0)
} else {
sys::mtime(metadata)
},
hash,
}
}
/// Ancestors of `path`, nearest first, up to the root.
fn ancestors(path: &str) -> Vec<String> {
Path::new(path)
.ancestors()
.skip(1)
.map(|ancestor| ancestor.to_string_lossy().into_owned())
.collect()
}
fn with_separator(directory: &str) -> String {
if directory.ends_with(sys::PATH_SEPARATOR) {
directory.to_string()
} else {
format!("{directory}{}", sys::PATH_SEPARATOR)
}
}
/// Whether `path` is `ancestor` or below it.
fn is_within(path: &str, ancestor: &str) -> bool {
path == ancestor || path.starts_with(&with_separator(ancestor))
}
fn excluded(target: &Target, name: &str) -> bool {
(target.hidden && name.starts_with('.')) || target.names.contains(name)
}
/// JavaScript's default sort: by UTF-16 code units.
fn sort_like_javascript(paths: &mut [String]) {
paths.sort_by(|a, b| a.encode_utf16().cmp(b.encode_utf16()));
}
/// Whether any path, or its nearest existing ancestor, is on a file system that does not report remote changes.
#[cfg(any(target_os = "linux", target_os = "android"))]
fn any_unreliable(targets: &[Target]) -> bool {
use std::ffi::CString;
for target in targets {
for candidate in std::iter::once(target.path.clone()).chain(ancestors(&target.path)) {
let Ok(name) = CString::new(candidate) else {
break;
};
// SAFETY: a NUL-terminated path and a zeroed statfs buffer for the call to fill.
let mut info: libc::statfs = unsafe { std::mem::zeroed() };
if unsafe { libc::statfs(name.as_ptr(), &mut info) } != 0 {
continue;
}
#[allow(clippy::unnecessary_cast)]
let kind = (info.f_type as u64) & 0xffff_ffff;
if UNRELIABLE_FILE_SYSTEMS.contains(&kind) {
return true;
}
break;
}
}
false
}
#[cfg(not(any(target_os = "linux", target_os = "android")))]
fn any_unreliable(_targets: &[Target]) -> bool {
false
}
/// One scan: the snapshot, and targets that are symbolic links to files, whose files need their own watchers.
struct Scan {
snapshot: Snapshot,
linked_files: HashSet<String>,
/// Directories counted against the limit, and directories traversed per target.
counted: HashSet<String>,
traversed: HashSet<(usize, String)>,
/// Kinds of listed entries without following links.
listed: HashMap<String, Kind>,
}
struct Installed {
dev: u64,
ino: u64,
/// The watched path with links resolved, as FSEvents spells event paths (`/private/var` for `/var`).
canonical: Option<String>,
}
struct FileWatcher<'a> {
id: u32,
output: &'a Output,
control: &'a Control,
targets: Vec<Target>,
mode: Mode,
poll_interval: Duration,
max_directories: usize,
snapshot: Snapshot,
native: Option<RecommendedWatcher>,
events_sender: Sender<notify::Result<Event>>,
installed: HashMap<String, Installed>,
events: HashSet<String>,
settle_at: Option<Instant>,
}
impl FileWatcher<'_> {
fn send(&self, json: Value) {
self.output.control(Frame::new(EVENT, self.id, json));
}
fn cancelled(&self) -> bool {
self.control.aborted.load(Ordering::SeqCst)
}
fn record(&self, scan: &mut Scan, path: &str, metadata: &Metadata, follow: bool) {
let mut hash = None;
let recent = metadata
.modified()
.map(|modified| {
SystemTime::now()
.duration_since(modified)
.map_or(true, |age| age < HASH_RECENT)
})
.unwrap_or(false);
if self.mode == Mode::Polling
&& metadata.is_file()
&& metadata.len() <= HASH_MAX_BYTES
&& recent
{
hash = fs::read(path).ok().map(|content| {
let mut hasher = std::collections::hash_map::DefaultHasher::new();
hasher.write(&content);
hasher.finish()
});
}
scan.snapshot
.insert(path.to_string(), entry_of(path, metadata, follow, hash));
}
fn count(&self, scan: &mut Scan, path: &str) -> Result<(), ScanError> {
scan.counted.insert(path.to_string());
if scan.counted.len() > self.max_directories {
return Err(ScanError::Budget(self.max_directories));
}
Ok(())
}
fn scan_directory(
&self,
scan: &mut Scan,
index: usize,
directory: &str,
) -> Result<(), ScanError> {
let target = &self.targets[index];
if !scan.traversed.insert((index, directory.to_string())) {
return Ok(());
}
let skip = |error: io::Error| -> Result<(), ScanError> {
// The watched directory itself must be readable; below it, unreadable directories are skipped.
if directory == target.path && denied(&error) {
return Err(ScanError::Io(error, "scandir", directory.to_string()));
}
match error.kind() {
io::ErrorKind::NotFound
| io::ErrorKind::PermissionDenied
| io::ErrorKind::NotADirectory => Ok(()),
_ => Err(ScanError::Io(error, "scandir", directory.to_string())),
}
};
let mut names = Vec::new();
match fs::read_dir(directory) {
Err(error) => return skip(error),
Ok(entries) => {
for entry in entries {
match entry {
Ok(entry) => names.push(entry.file_name().to_string_lossy().into_owned()),
Err(error) => return skip(error),
}
}
}
}
let prefix = with_separator(directory);
for name in names {
if excluded(target, &name) {
continue;
}
let path = format!("{prefix}{name}");
let kind = match scan.listed.get(&path) {
Some(kind) => *kind,
None => {
let Ok(metadata) = fs::symlink_metadata(&path) else {
continue;
};
let kind = kind_of(&metadata);
scan.listed.insert(path.clone(), kind);
// A target's own entry (following links) wins over its listing by another target.
if !scan.snapshot.contains_key(&path) {
self.record(scan, &path, &metadata, false);
}
kind
}
};
if target.recursive && kind == Kind::Directory {
self.count(scan, &path)?;
self.scan_directory(scan, index, &path)?;
}
}
Ok(())
}
fn scan(&self) -> Result<Scan, ScanError> {
let mut scan = Scan {
snapshot: Snapshot::new(),
linked_files: HashSet::new(),
counted: HashSet::new(),
traversed: HashSet::new(),
listed: HashMap::new(),
};
for (index, target) in self.targets.iter().enumerate() {
for ancestor in ancestors(&target.path) {
if scan.snapshot.contains_key(&ancestor) {
continue;
}
// Identity only: an ancestor's own timestamps change with every unrelated sibling.
if let Ok(metadata) = fs::symlink_metadata(&ancestor) {
let entry = Entry {
size: 0,
mtime: (0, 0),
..entry_of(&ancestor, &metadata, false, None)
};
scan.snapshot.insert(ancestor, entry);
}
}
// The target may be a symbolic link to what is watched; follow it. A missing target is watched for its
// creation; one that cannot be reached for lack of permission fails.
let metadata = match fs::metadata(&target.path) {
Ok(metadata) => metadata,
Err(error) if denied(&error) => {
return Err(ScanError::Io(error, "stat", target.path.clone()));
}
Err(_) => continue,
};
self.record(&mut scan, &target.path, &metadata, true);
if metadata.is_file()
&& fs::symlink_metadata(&target.path)
.is_ok_and(|link| link.file_type().is_symlink())
{
scan.linked_files.insert(target.path.clone());
}
if metadata.is_dir() {
self.count(&mut scan, &target.path)?;
self.scan_directory(&mut scan, index, &target.path)?;
}
}
Ok(scan)
}
/// An ancestor that changed identity moved every target below it; report those targets.
fn reported(&self, path: &str) -> String {
if self
.targets
.iter()
.any(|target| is_within(path, &target.path))
{
return path.to_string();
}
self.targets
.iter()
.find(|target| is_within(&target.path, path))
.map_or_else(|| path.to_string(), |target| target.path.clone())
}
fn diff(&self, previous: &Snapshot, next: &Snapshot) -> Vec<String> {
let mut changed = Vec::new();
for (path, entry) in next {
if previous.get(path) != Some(entry) {
changed.push(self.reported(path));
}
}
for path in previous.keys() {
if !next.contains_key(path) {
changed.push(self.reported(path));
}
}
changed
}
/// Rescan, report differences, and install watchers for new directories, rescanning until none are new.
fn sync(&mut self, mut report: bool) -> Result<HashSet<String>, ScanError> {
let mut changed = HashSet::new();
for _ in 0..10 {
if self.cancelled() {
break;
}
let scan = self.scan()?;
if report {
changed.extend(self.diff(&self.snapshot, &scan.snapshot));
}
self.snapshot = scan.snapshot;
if self.mode == Mode::Polling || !self.reconcile(&scan.linked_files) {
break;
}
if cfg!(target_os = "macos") {
self.settle_at = Some(Instant::now() + FSEVENTS_SETTLE);
}
// Something written into a new directory before its watcher existed shows up in the next round.
report = true;
}
Ok(changed)
}
/// Watch every existing ancestor of each target, each target directory, each target that links to a file, and on
/// Linux each directory below a recursive target (elsewhere one recursive watcher per target). Returns whether a
/// watcher was added.
fn reconcile(&mut self, linked_files: &HashSet<String>) -> bool {
let per_directory = cfg!(any(target_os = "linux", target_os = "android"));
let mut wanted: HashMap<String, bool> = linked_files
.iter()
.map(|path| (path.clone(), false))
.collect();
for target in &self.targets {
for ancestor in ancestors(&target.path) {
if self
.snapshot
.get(&ancestor)
.is_some_and(|entry| entry.kind == Kind::Directory)
{
wanted.entry(ancestor).or_insert(false);
}
}
if self
.snapshot
.get(&target.path)
.is_none_or(|entry| entry.kind != Kind::Directory)
{
continue;
}
wanted.insert(target.path.clone(), target.recursive && !per_directory);
if target.recursive && per_directory {
for (path, entry) in &self.snapshot {
if entry.kind == Kind::Directory
&& *path != target.path
&& is_within(path, &target.path)
{
wanted.insert(path.clone(), false);
}
}
}
}
// Gone, or replaced: a watcher follows the directory it was installed on, not the path.
let stale: Vec<String> = self
.installed
.iter()
.filter(|(path, installed)| {
!wanted.contains_key(*path)
|| self.snapshot.get(*path).is_none_or(|entry| {
entry.dev != installed.dev || entry.ino != installed.ino
})
})
.map(|(path, _)| path.clone())
.collect();
for path in stale {
self.installed.remove(&path);
if let Some(native) = &mut self.native {
let _ = native.unwatch(Path::new(&path));
}
}
let mut added = false;
for (path, recursive) in wanted {
let Some(entry) = self.snapshot.get(&path) else {
continue;
};
if self.installed.contains_key(&path) {
continue;
}
let (dev, ino) = (entry.dev, entry.ino);
if self.native.is_none() {
let sender = self.events_sender.clone();
match RecommendedWatcher::new(
move |event| drop(sender.send(event)),
notify::Config::default(),
) {
Ok(native) => self.native = Some(native),
Err(_) => {
self.switch_to_polling();
return false;
}
}
}
let mode = if recursive {
RecursiveMode::Recursive
} else {
RecursiveMode::NonRecursive
};
match self.native.as_mut().unwrap().watch(Path::new(&path), mode) {
Ok(()) => {
let canonical = fs::canonicalize(&path)
.ok()
.map(|canonical| canonical.to_string_lossy().into_owned())
.filter(|canonical| *canonical != path);
self.installed.insert(
path,
Installed {
dev,
ino,
canonical,
},
);
added = true;
}
Err(error) => {
let gone = match &error.kind {
notify::ErrorKind::PathNotFound => true,
notify::ErrorKind::Io(io) => {
matches!(
io.kind(),
io::ErrorKind::NotFound | io::ErrorKind::PermissionDenied
)
}
_ => false,
};
// Out of watches or unsupported: compare snapshots from now on, and say coverage was uncertain.
if !gone {
self.switch_to_polling();
return false;
}
}
}
}
added
}
fn switch_to_polling(&mut self) {
if self.mode == Mode::Polling {
return;
}
self.mode = Mode::Polling;
self.native = None;
self.installed.clear();
self.send(json!({ "kind": "change", "overflow": true, "mode": "polling" }));
}
fn in_scope(&self, path: &str) -> bool {
for target in &self.targets {
if is_within(&target.path, path) {
return true;
}
if !is_within(path, &target.path) || path == target.path {
continue;
}
let relative = &path[with_separator(&target.path).len()..];
let components: Vec<&str> = relative.split(sys::PATH_SEPARATOR).collect();
if !target.recursive && components.len() > 1 {
continue;
}
if components.iter().any(|name| excluded(target, name)) {
continue;
}
return true;
}
false
}
/// An event path in the spelling of the watched path it is under.
fn watched_spelling(&self, path: &str) -> String {
let mut best: Option<(&str, &str)> = None;
for (watched, installed) in &self.installed {
if let Some(canonical) = &installed.canonical
&& is_within(path, canonical)
&& best.is_none_or(|(_, longest)| canonical.len() > longest.len())
{
best = Some((watched, canonical));
}
}
match best {
Some((watched, canonical)) => format!("{watched}{}", &path[canonical.len()..]),
None => path.to_string(),
}
}
/// Note a native event; returns whether it calls for a rescan.
fn on_event(&mut self, event: notify::Result<Event>) -> bool {
let event = match event {
Ok(event) => event,
Err(error) => {
// A watched directory that disappears or fails: rescan, which also replaces its watcher.
for path in &error.paths {
self.installed.remove(&*path.to_string_lossy());
}
return true;
}
};
// Reads are not changes (libuv does not ask for them).
if matches!(event.kind, EventKind::Access(_)) {
return false;
}
if event.need_rescan() || event.paths.is_empty() {
return true;
}
let mut flush = false;
for path in &event.paths {
let path = self.watched_spelling(&path.to_string_lossy());
// Events about unrelated siblings of an ancestor, or about excluded entries, are ignored.
if self.in_scope(&path) {
let reported = self.reported(&path);
self.events.insert(reported);
flush = true;
}
}
flush
}
/// Rescan and report what changed; an error ends the watcher.
fn flush(&mut self) -> bool {
let events = std::mem::take(&mut self.events);
match self.sync(true) {
Ok(mut changed) => {
changed.extend(events);
if !changed.is_empty() && !self.cancelled() {
let mut paths: Vec<String> = changed.into_iter().collect();
sort_like_javascript(&mut paths);
self.send(json!({ "kind": "change", "paths": paths }));
}
true
}
Err(error) => {
let message = error.failure().message;
self.send(json!({ "kind": "error", "code": error.code(), "message": message }));
false
}
}
}
}
fn parse_targets(json: &Value) -> Result<Vec<Target>, Failure> {
let invalid = || Failure::new("EINVAL", "watch needs targets");
json["targets"]
.as_array()
.ok_or_else(invalid)?
.iter()
.map(|target| {
Ok(Target {
path: target["path"].as_str().ok_or_else(invalid)?.to_string(),
recursive: target["recursive"].as_bool().unwrap_or(false),
hidden: target["exclude"]["hidden"].as_bool().unwrap_or(false),
names: target["exclude"]["names"]
.as_array()
.map(|names| {
names
.iter()
.filter_map(|name| name.as_str().map(str::to_string))
.collect()
})
.unwrap_or_default(),
})
})
.collect()
}
/// Watch until cancelled: a `ready` event once coverage is established, then `change` events, or an `error` event
/// after which nothing follows.
pub fn run(id: u32, json: &Value, output: &Output, control: &Control) -> Result<Value, Failure> {
let targets = parse_targets(json)?;
// Windows refuses to rename a directory while a directory below it is open, and native watchers keep watched
// directories open, so Windows polls, as do file systems that do not report remote changes.
let mode = match json["mode"].as_str() {
Some("native") => Mode::Native,
Some("polling") => Mode::Polling,
_ if cfg!(windows) || any_unreliable(&targets) => Mode::Polling,
_ => Mode::Native,
};
let (events_sender, events) = mpsc::channel();
let mut watcher = FileWatcher {
id,
output,
control,
targets,
mode,
poll_interval: Duration::from_millis(
json["pollIntervalMs"].as_u64().unwrap_or(DEFAULT_POLL_MS),
),
max_directories: json["maxDirectories"]
.as_u64()
.map_or(DEFAULT_MAX_DIRECTORIES, |max| max as usize),
snapshot: Snapshot::new(),
native: None,
events_sender,
installed: HashMap::new(),
events: HashSet::new(),
settle_at: None,
};
watcher.sync(false).map_err(|error| error.failure())?;
watcher.send(json!({ "kind": "ready", "mode": watcher.mode.name() }));
let mut flush_at: Option<Instant> = None;
let mut poll_at = Instant::now() + watcher.poll_interval;
loop {
if watcher.cancelled() {
return Ok(json!({}));
}
let now = Instant::now();
let mut wake = now + CANCEL_POLL;
for at in [flush_at, watcher.settle_at].into_iter().flatten() {
wake = wake.min(at);
}
if watcher.mode == Mode::Polling {
wake = wake.min(poll_at);
}
match events.recv_timeout(wake.saturating_duration_since(now)) {
Ok(event) => {
if watcher.on_event(event) && flush_at.is_none() && watcher.mode == Mode::Native {
flush_at = Some(Instant::now() + DEBOUNCE);
}
}
Err(RecvTimeoutError::Timeout) => {}
Err(RecvTimeoutError::Disconnected) => unreachable!("the watcher holds a sender"),
}
let now = Instant::now();
let due = |at: Option<Instant>| at.is_some_and(|at| at <= now);
let polling_due = watcher.mode == Mode::Polling && poll_at <= now;
if due(flush_at) || due(watcher.settle_at) || polling_due {
if due(flush_at) {
flush_at = None;
}
if due(watcher.settle_at) {
watcher.settle_at = None;
}
if !watcher.flush() {
return Ok(json!({}));
}
if polling_due {
poll_at = Instant::now() + watcher.poll_interval;
}
}
}
}

269
third_party/pi/packages/env/daemon/src/window.rs vendored Normal file
View File

@@ -0,0 +1,269 @@
//! Durable's output window (`ShellExecOptions.window`, `ShellOutputInfo.skipped`): undelivered command output is held
//! and coalesced; text followed by more than the window (by a byte or a line) can never be in the caller's tail and is
//! replaced by counts. Delivery follows the caller's commit pace, and waits while an earlier frame is still unsent.
use serde_json::{Value, json};
use std::time::{Duration, Instant};
#[derive(Clone, Copy)]
pub struct Window {
pub max_bytes: usize,
pub max_lines: usize,
pub min_interval: Duration,
pub bytes_per_second: f64,
}
impl Window {
pub fn from_json(json: &Value) -> Option<Window> {
let object = json.as_object()?;
Some(Window {
max_bytes: object.get("maxBytes")?.as_u64()? as usize,
max_lines: object.get("maxLines")?.as_u64()? as usize,
min_interval: Duration::from_secs_f64(
object.get("minIntervalMs")?.as_f64()?.max(0.0) / 1000.0,
),
bytes_per_second: object.get("bytesPerSecond")?.as_f64()?.max(1.0),
})
}
}
#[derive(Default)]
struct Skipped {
bytes: u64,
newlines: u64,
ends_with_newline: bool,
}
/// One output event: its stream, text, and the output omitted right before it.
pub struct Event {
pub stream: &'static str,
pub text: String,
pub skipped: Option<Value>,
}
pub struct Pending {
window: Window,
/// Undelivered text in arrival order, by stream.
segments: Vec<(&'static str, String)>,
bytes: usize,
newlines: usize,
skipped: Option<Skipped>,
next_send: Instant,
}
fn newline_count(text: &str) -> usize {
text.bytes().filter(|byte| *byte == b'\n').count()
}
impl Pending {
pub fn new(window: Window) -> Pending {
Pending {
window,
segments: Vec::new(),
bytes: 0,
newlines: 0,
skipped: None,
next_send: Instant::now(),
}
}
pub fn is_empty(&self) -> bool {
self.segments.is_empty()
}
/// When the next delivery may happen.
pub fn next_send(&self) -> Instant {
self.next_send
}
pub fn push(&mut self, stream: &'static str, text: String) {
if text.is_empty() {
return;
}
self.bytes += text.len();
self.newlines += newline_count(&text);
match self.segments.last_mut() {
Some((last, existing)) if *last == stream => existing.push_str(&text),
_ => self.segments.push((stream, text)),
}
// Trimming costs a pass over the held text; do it once a few windows have accumulated.
if self.bytes > 4 * (self.window.max_bytes + 1)
|| self.newlines > 4 * (self.window.max_lines + 1)
{
self.trim();
}
}
/// Whether the text after a cut still proves the cut text is outside the window.
fn proves(&self, bytes: usize, newlines: usize) -> bool {
bytes > self.window.max_bytes || newlines > self.window.max_lines
}
/// Replace the longest prefix whose remainder still proves exclusion by counts.
fn trim(&mut self) {
let mut removed = String::new();
// Whole segments first.
while self.segments.len() > 1 {
let (bytes, newlines) = {
let first = &self.segments[0].1;
(first.len(), newline_count(first))
};
if !self.proves(self.bytes - bytes, self.newlines - newlines) {
break;
}
let (_, text) = self.segments.remove(0);
self.bytes -= bytes;
self.newlines -= newlines;
removed.push_str(&text);
}
// Then a prefix of the first remaining segment, at a character boundary.
let rest_bytes = self.bytes - self.segments[0].1.len();
let rest_newlines = self.newlines - newline_count(&self.segments[0].1);
let first = &self.segments[0].1;
let mut cut = 0;
// By bytes: keep more than `max_bytes` overall.
let keep_bytes = (self.window.max_bytes + 1).saturating_sub(rest_bytes);
if first.len() > keep_bytes {
let mut index = first.len() - keep_bytes;
while index > 0 && !first.is_char_boundary(index) {
index -= 1;
}
cut = cut.max(index);
}
// By lines: keep more than `max_lines` newlines overall, starting at a newline of this segment.
let keep_newlines = (self.window.max_lines + 1).saturating_sub(rest_newlines);
if keep_newlines > 0 {
let positions: Vec<usize> = first
.bytes()
.enumerate()
.filter(|(_, byte)| *byte == b'\n')
.map(|(index, _)| index)
.collect();
if positions.len() >= keep_newlines {
cut = cut.max(positions[positions.len() - keep_newlines]);
}
} else {
cut = first.len();
}
if cut > 0 {
let first = &mut self.segments[0].1;
let prefix: String = first.drain(..cut).collect();
self.bytes -= prefix.len();
self.newlines -= newline_count(&prefix);
removed.push_str(&prefix);
if first.is_empty() {
self.segments.remove(0);
}
}
if !removed.is_empty() {
let skipped = self.skipped.get_or_insert_with(Skipped::default);
skipped.bytes += removed.len() as u64;
skipped.newlines += newline_count(&removed) as u64;
skipped.ends_with_newline = removed.ends_with('\n');
}
}
/// Deliver everything held: one event carrying the skip and all text after it, or one event per segment.
pub fn take(&mut self) -> Vec<Event> {
let segments = std::mem::take(&mut self.segments);
let delivered = self.bytes;
self.bytes = 0;
self.newlines = 0;
// The caller writes at most the window per sample, and paces itself by what it wrote.
let pace = Duration::from_secs_f64(
delivered.min(self.window.max_bytes) as f64 / self.window.bytes_per_second,
);
self.next_send = Instant::now() + self.window.min_interval.max(pace);
match self.skipped.take() {
Some(skipped) => {
let stream = segments
.last()
.map(|(stream, _)| *stream)
.unwrap_or("stdout");
let text: String = segments.into_iter().map(|(_, text)| text).collect();
vec![Event {
stream,
text,
skipped: Some(json!({
"bytes": skipped.bytes,
"newlines": skipped.newlines,
"endsWithNewline": skipped.ends_with_newline,
})),
}]
}
None => segments
.into_iter()
.map(|(stream, text)| Event {
stream,
text,
skipped: None,
})
.collect(),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn window(max_bytes: usize, max_lines: usize) -> Window {
Window {
max_bytes,
max_lines,
min_interval: Duration::ZERO,
bytes_per_second: 1e12,
}
}
/// Bytes and newlines delivered or skipped, and the text after the last skip.
fn total(events: &[Event]) -> (u64, u64, String) {
let mut bytes = 0;
let mut newlines = 0;
let mut after = String::new();
for event in events {
if let Some(skipped) = &event.skipped {
bytes += skipped["bytes"].as_u64().unwrap();
newlines += skipped["newlines"].as_u64().unwrap();
after.clear();
}
bytes += event.text.len() as u64;
newlines += newline_count(&event.text) as u64;
after.push_str(&event.text);
}
(bytes, newlines, after)
}
#[test]
fn counts_everything_and_keeps_more_than_the_window_after_a_skip() {
let mut pending = Pending::new(window(10, 2));
let mut full = String::new();
let mut events = Vec::new();
for index in 0..500 {
let text = format!("line {index} é😀\n");
full.push_str(&text);
pending.push(if index % 7 == 0 { "stderr" } else { "stdout" }, text);
if index % 50 == 49 {
events.extend(pending.take());
}
}
events.extend(pending.take());
let (bytes, newlines, after) = total(&events);
assert_eq!(bytes, full.len() as u64);
assert_eq!(newlines, newline_count(&full) as u64);
assert!(full.ends_with(&after));
assert!(after.len() > 10 || newline_count(&after) > 2);
assert!(events.iter().any(|event| event.skipped.is_some()));
}
#[test]
fn delivers_streams_apart_without_skips() {
let mut pending = Pending::new(window(1000, 1000));
pending.push("stdout", "a".into());
pending.push("stderr", "b".into());
pending.push("stdout", "c".into());
let events = pending.take();
let streams: Vec<_> = events.iter().map(|event| event.stream).collect();
assert_eq!(streams, ["stdout", "stderr", "stdout"]);
}
}

151
third_party/pi/packages/env/docs/protocol.md vendored Normal file
View File

@@ -0,0 +1,151 @@
# pi-env wire protocol
Version 1. The client starts `pi-env serve --token <hex>` (over `ssh`, or directly in tests) and talks to it over the
process's stdin and stdout. stderr is diagnostic text for logs only.
## Sync
Shell startup files may print to stdout before the daemon runs. The daemon's first output is the line
`PI-ENV <token>\n`, with the token from its command line. The client discards everything before that line. After it,
stdout carries only frames.
## Frames
Every frame, in both directions:
```text
u32 length bytes after this field
u8 type
u32 id request id; 0 for frames that belong to no request
u32 jsonLength
json UTF-8 JSON object, jsonLength bytes
bytes raw payload, the remaining length - 9 - jsonLength bytes
```
All integers are big-endian. A frame is at most 16 MiB; larger transfers are split by the client. A request whose JSON
does not parse gets an `EINVAL` error; framing errors end the connection.
| type | name | direction | meaning |
|---|---|---|---|
| 1 | request | client → daemon | `json.op` names the operation |
| 2 | result | daemon → client | success of request `id`; the request is finished |
| 3 | error | daemon → client | failure of request `id`; the request is finished |
| 4 | event | daemon → client | progress of a running request (`exec` and `pty` output, `watch` changes), not finishing it |
| 5 | cancel | client → daemon | cancel request `id`; see below |
| 6 | ping | both | liveness; the receiver ignores it |
Errors are `{ code, message, syscall?, path? }`. `code` is a Node-style error code (`ENOENT`, `EISDIR`,
`ERR_FS_EISDIR`, ...) or one of the execution codes `shell_unavailable`, `spawn_error`, `timeout`, `aborted`, `unknown`.
Messages are diagnostic; only codes are part of the contract.
Paths are absolute JSON strings; the client resolves relative paths, `~` and `file://` URLs before sending. Strings are
sent well-formed (lone surrogates become U+FFFD) and encoded as UTF-8, as Node encodes string paths and arguments. File
contents and output travel as raw payload bytes.
## Cancellation
A cancel names one request. The daemon registers each request before running it, so a cancel sent right after its
request is never lost. `exec` and `pty` kill the command's process group and finish with `aborted`; `{ mode: "kill" }` kills
without aborting, so the command finishes with the killed process's status, as `cleanup()` does. `scanLines` stops
between chunks and `listFiles` between entries, with `aborted`. `watch` stops and finishes with `{}`. Other operations
are short and finish normally.
## Scheduling
File operations run on a fixed pool of worker threads; `exec`, `pty` and `watch` run on their own threads. `ptyWrite`,
`ptyResize` and `ptyForeground` are answered in arrival order by the thread that reads frames, so keystrokes never
swap. Chunk writes to one handle run in arrival order. Results, errors, pings and `watch` events are sent before queued
command output, and results with payloads above 64 KiB queue behind output too, so a slow link delays bulk data, not
replies. Output of a command without an output window waits while more than 4 MiB of output is unsent, which slows the
command as a full pipe would.
## Liveness
Each side sends `ping` every 5 seconds. The daemon kills every process group it started and exits after 30 seconds
without receiving any bytes, or when stdin ends. A dropped connection therefore stops remote commands. The client
fails requests of a lost connection with `{ code: "unknown", lost: true }` and starts a new daemon on the next request;
handles belong to the daemon that opened them, so requests on them fail instead of reaching the new one.
## Operations
`hello { protocol }` → `{ protocol, version, os, arch, home, tmpdir, cwd, driveCwds, pid, features }`. The first
request. `version` is the npm package version the daemon shipped with; `os` and `arch` are Rust's `std::env::consts`
values (`linux`, `android`, `macos`, `windows`; `x86_64`, `aarch64`), which the client's `nodePlatform` maps to Node's
names; `tmpdir` follows the remote Node's `os.tmpdir()` rules; `cwd` and `driveCwds` (Windows' `=C:` variables) are
what Node's `path.resolve` falls back to for drive-relative paths. `features` lists operations added to protocol 1
after its first release (`listFiles`); older daemons omit the field and refuse those operations as unknown, so clients
check it first.
File operations take `{ path }` (and the listed fields) and return `{}` unless noted:
| op | fields | result |
|---|---|---|
| `lstat` | | `info` |
| `realpath` | | `{ path }` |
| `write` | `append`, `parents?` (default true), `keep?`, payload | creates missing parents like Node's recursive `mkdir`, opens like `writeFile` (`w`) or `appendFile` (`a`), writes the payload; with `keep`, `{ handle }` for `writeChunk` |
| `writeChunk` | `handle`, payload | appends to a kept write handle; after a failed chunk, later chunks fail with `EBADF` |
| `truncate` | `size` | |
| `fsync` | | |
| `rename` | `to` | |
| `mkdir` | `recursive` | |
| `rm` | `recursive`, `force` | Node `fs.rm` semantics |
| `mkdtemp` | (`path` is the prefix) | `{ path }` |
| `open` | `noFollow`, `mode?` | `{ handle, info }` for `openBinaryReader`: nonblocking, regular files only. With `mode: "read"`, Node's `open(path, "r")` of any file, blocking: `{ handle, stat }` or `{ handle, statError }` |
| `pread` | `handle`, `offset?`, `length` | payload; without `offset`, one read at the current position |
| `fstat` | `handle` | `info` |
| `scanLines` | `handle`, `startLine`, `endLine?` | `LineScan` |
| `opendir` | | `{ handle }` |
| `readdir` | `handle`, `max` | `{ entries, done }` |
| `close` | `handle` | |
`listFiles { path, hidden?, gitignore?, exclude?, maxEntries?, maxBytes?, timeoutMs? }` lists the tree below the
directory `path` (feature `listFiles`): files, directories and symbolic links, which are not followed; other kinds of
files are left out, and entries that cannot be read are skipped. The payload holds one record per entry: a kind byte
(`f` file, `d` directory, `l` symbolic link), the path relative to `path` with `/` separators as UTF-8 (names that are
not UTF-8 get replacement characters), and a NUL byte. Records come depth first, siblings in byte order of their
names. The result is `{ count, truncated }`. Entries named `.git` and the names in `exclude` are skipped with
everything below them. `hidden` (default true) includes names starting with `.` (and Windows' hidden files).
`gitignore` (default true) skips, inside git work trees, what the `.gitignore` files of `path`, its subdirectories and
its parents, `.git/info/exclude` and the global excludes file ignore, using the `ignore` crate of `fd` and ripgrep. The
listing stops with `truncated: true` at the first entry beyond `maxEntries` (default 50 000), beyond `maxBytes` of
payload (default 4 MiB, at most the frame limit less 64 KiB), or after `timeoutMs` (default 10 000). A missing `path`
fails with `ENOENT`, a file with `ENOTDIR`.
`info` is `{ name, kind, size, mtimeSec, mtimeNsec, dev, ino }` with `kind` one of `file`, `directory`, `symlink`, `other`.
`readdir` entries are `{ name, raw?, info }` or `{ name, raw?, error }` when the entry's `lstat` failed. Names are
decoded as UTF-8 with replacement characters, and lstat'ed under that name, as Node does; `raw` holds the original
bytes of a name that is not valid UTF-8, for sorting. Handles are numbers, valid until `close` or the end of the
connection; opening more than 4096 fails with `EMFILE`.
`exec { command | argv, cwd, env, inheritEnv, shellPath?, timeoutMs?, spill?, window? }` runs a shell string or an
argv array. While it runs, `event` frames `{ kind: "output", stream, skipped? }` carry decoded output text as UTF-8
payload. With `window { maxBytes, maxLines, minIntervalMs, bytesPerSecond }`, output is coalesced and paced like the
caller's commits, at most one output frame per command is unsent, and output followed by more than the window is
replaced by `skipped { bytes, newlines, endsWithNewline }` on the next event, which carries all output after it
(`ShellOutputInfo.skipped`). The result is `{ exitCode, spillPath? }`; failures are errors with codes `timeout` or
`aborted` (carrying `spillPath`), `shell_unavailable`, `spawn_error`, or `unknown` (spill failure).
`watch { targets, mode?, pollIntervalMs?, maxDirectories? }` runs Durable's `NodeFileWatcher` in the daemon. `targets`
are `{ path, recursive, exclude: { hidden, names } }`. Once coverage is established, an event `{ kind: "ready", mode }`
follows; then `{ kind: "change", paths }` or `{ kind: "change", overflow: true, mode: "polling" }` (native watching
failed), or `{ kind: "error", code, message }`, after which nothing follows. The request lasts until cancelled.
Failing to establish coverage is an error of the request.
`pty { argv?, cwd, env, inheritEnv, cols, rows }` runs a program in a new pseudo-terminal of `cols` × `rows` cells
(1 to 4096), as the leader of a new session with the terminal as its controlling terminal. Without `argv`, it runs
`SHELL` (from `env`, else the daemon's environment, else `/bin/sh`) as a login shell (`-` before its name). The
environment is built as for `exec`; set `TERM` in `env`. Once the program runs, an event `{ kind: "started", pid }`
comes first; then `{ kind: "output" }` events carry the terminal's raw output bytes as payload (not decoded, unlike
`exec`). Output waits while more than 4 MiB of output is unsent, like a full pipe. After the program exits, output is
collected until the terminal closes or 100 ms pass without output. The result is `{ exitCode }`. Not supported on
Windows yet (`ENOTSUP`).
While a `pty` request runs, `ptyWrite { target }` writes its payload to the terminal's input, and
`ptyResize { target, cols, rows }` changes its size (the program gets `SIGWINCH`); `target` is the id of the `pty`
request. Both return `{}` at once; input is queued and written in order, waiting while the program does not read it.
Once the terminal has ended, they fail with `EBADF`.
`ptyForeground { target }` reports the terminal's foreground process group when it is not the program's own:
`{ foreground: { pid, name } }`, where `pid` is the group's id and `name` its leader's process name (`null` if unknown),
or `{ foreground: null }` while the program itself is in front. For a shell this tells a running job (an editor, a
build) from the prompt; background jobs are not reported. It fails with `EBADF` once the terminal has ended.

68
third_party/pi/packages/env/docs/semantics.md vendored Normal file
View File

@@ -0,0 +1,68 @@
# Semantics
The reference is Durable's `NodeExecutionEnv` (`packages/durable/src/env/node.ts`) on the remote machine. Durable's
env conformance suite and `test/differential.test.ts` check each rule below.
## Split of work
| Concern | Where |
|---|---|
| Path resolution: `~` (remote home), `file://`, relative to `cwd`; on Windows drive-relative paths against the remote's working directories | client, the remote system's `path` rules |
| Abort checkpoints of each method | client, as `NodeExecutionEnv` |
| Error code mapping (`ENOENT` → `not_found`, ...) | client, as `NodeExecutionEnv`'s `toFileError` |
| `readFile`'s read sizes, size limit and decoding (`StringDecoder`, byte-order mark kept) over the daemon's `open(path, "r")` | client |
| `writeFile` in 512 KiB writes with an abort check before each; `appendFile` without checks | client, over one open file |
| Line reader | client, positional reads and Durable's `StreamDecoder` |
| Watching: snapshots, native events, polling | daemon, a port of Durable's `NodeFileWatcher` |
| Timeout validation, result precedence (callback error, timeout, abort, spill failure, exit code) | client |
| System calls, processes, spill files, line scans | daemon |
| Command output decoding: per stream, WHATWG UTF-8, only a leading byte-order mark dropped | daemon (`encoding_rs`) |
## Rules the standard libraries do not follow
- Recursive `mkdir`: an existing directory is fine, an existing file fails with `EEXIST`, a file in the way of a parent
fails with `ENOTDIR`. Writes create missing parents this way.
- `rm`: a missing path fails unless `force`; a directory without `recursive` fails with `ERR_FS_EISDIR`
(`FileError` code `unknown`).
- `mkdtemp`: `mkdtemp(3)` of `<tmpdir>/<prefix>XXXXXX`; the prefix is joined with POSIX `path.join`, so `../x` works.
- `tmpdir`: `TMPDIR`, `TMP`, `TEMP`, then `/tmp` (Termux: `$PREFIX/tmp`), trailing slash removed.
- `listDir`: names in byte order, like libuv's `scandir`; fails if any entry cannot be lstat'ed. `openDirReader` keeps
directory order, skips entries removed meanwhile, and fills each page after skipping. Names that are not UTF-8 are
decoded with replacement characters and lstat'ed under that name, so they fail with `ENOENT` as in Node.
- `openBinaryReader`: nonblocking open; directories fail with `is_directory`, other non-regular files with `invalid`;
`noFollow` refuses a final symbolic link (`O_NOFOLLOW`).
- `readBinaryFile`, `readTextFile` and `openTextLineReader` open any file like `open(path, "r")`: `/dev/null` reads
empty, a FIFO blocks until it has a writer, and a directory fails when read (a line reader at `readLine`). A regular
file is read up to its size at open; other files to their end.
- `listFiles` has no `NodeExecutionEnv` counterpart: the daemon implements it for remote and local environments
alike (`LocalExecutionEnv` runs a daemon on the same machine), so both list with the same code. Unlike `fd`,
symbolic links are not followed; unlike `git ls-files`, a tracked file that matches an ignore rule is left out.
- `readBinaryFile` returns a `Buffer`, reader reads a plain `Uint8Array`, as Node does.
- Windows: error codes are libuv's translations; `rm` of a symbolic link or junction to a directory removes the link;
creating a file where a directory exists fails with `EISDIR`; modification times before 1970 are negative.
## Commands
- A string runs through the shell: a configured `shellPath` that does not exist (following links) fails with
`shell_unavailable`; otherwise `/bin/bash`, `which bash`, then `sh`, with `-c`. An argv array runs its program
directly; on Windows it is searched like libuv does (working directory, then `PATH`, adding `.com` and `.exe`), and
batch files are refused.
- The working directory must exist (`spawn_error`).
- Environment: with `inheritEnv` (default), the daemon's environment, then `shellEnv`, then `env`; without it, only
`env`.
- Each command gets a new session (process group), default signal dispositions and an empty signal mask. Timeout,
abort and `cleanup()` kill the group with `SIGKILL`.
- After the process exits, output is still collected until both pipes end, or 100 ms pass without output; the timeout
still applies meanwhile. The pipes are closed when the command settles, even if a descendant keeps them open.
- An abort settles as `aborted`; `cleanup()` kills without aborting, so the command settles with exit code 137.
- A process killed by a signal reports `128 + signal`.
- Spill: once the output crosses `spill.afterBytes` or `spill.afterLines` (counted over raw chunks in arrival order, as
Node counts), the complete raw output goes to `pi-output-<uuid>.log` in a fresh `tmp-` directory.
## Watching
- The daemon runs Durable's `NodeFileWatcher` next to the files: native events (inotify, FSEvents) only trigger a
debounced rescan, and changes are the difference between snapshots plus the event paths. Reads are not events.
- Windows and network or FUSE file systems poll, as `NodeExecutionEnv` does; running out of native watches switches to
polling and reports `overflow`.
- When the connection is lost, the client opens the watcher again once a new daemon starts and reports `overflow`.

55
third_party/pi/packages/env/package.json vendored Normal file
View File

@@ -0,0 +1,55 @@
{
"name": "@earendil-works/pi-env",
"version": "1.3.0",
"description": "Remote execution environments for Pi Durable: a small daemon deployed over SSH and its ExecutionEnv client",
"type": "module",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
"exports": {
".": {
"source": "./src/index.ts",
"types": "./dist/index.d.ts",
"import": "./dist/index.js",
"default": "./dist/index.js"
}
},
"sideEffects": false,
"files": [
"dist",
"bin",
"docs",
"README.md",
"CHANGELOG.md"
],
"scripts": {
"clean": "shx rm -rf dist",
"build": "tsc -p tsconfig.build.json",
"build:daemon": "cargo build --manifest-path daemon/Cargo.toml --locked",
"test": "vitest --run",
"prepublishOnly": "npm run clean && npm run build"
},
"keywords": [
"durable-execution",
"agents",
"ssh",
"remote"
],
"author": "Earendil Works",
"license": "MIT",
"repository": {
"type": "git",
"url": "git+https://github.com/earendil-works/pi.git",
"directory": "packages/env"
},
"engines": {
"node": ">=22.19.0"
},
"dependencies": {
"@earendil-works/chord": "1.3.0",
"@earendil-works/pi-durable": "1.3.0"
},
"devDependencies": {
"shx": "0.3.4",
"vitest": "4.1.11"
}
}

397
third_party/pi/packages/env/src/connection.ts vendored Normal file
View File

@@ -0,0 +1,397 @@
import { type ChildProcessWithoutNullStreams, spawn } from "node:child_process";
import { randomBytes } from "node:crypto";
/** Frame types (docs/protocol.md). */
const REQUEST = 1;
const RESULT = 2;
const ERROR = 3;
const EVENT = 4;
const CANCEL = 5;
const PING = 6;
const MAX_FRAME = 16 * 1024 * 1024;
const PING_INTERVAL_MS = 5000;
/** The daemon pings every five seconds; this much silence means the connection is gone. */
const SILENCE_LIMIT_MS = 30_000;
/** How long starting the daemon and its `hello` may take. */
const START_TIMEOUT_MS = 60_000;
export type Json = Record<string, unknown>;
/** A failure reported by the daemon, with its Node-style code. */
export class RemoteError extends Error {
readonly code: string;
readonly path?: string;
readonly fields: Json;
constructor(fields: Json) {
super(typeof fields.message === "string" ? fields.message : "Remote operation failed");
this.name = "RemoteError";
this.code = typeof fields.code === "string" ? fields.code : "unknown";
if (typeof fields.path === "string") this.path = fields.path;
this.fields = fields;
}
}
/** What `hello` reports about the remote machine. */
export interface RemoteInfo {
protocol: number;
version: string;
os: string;
arch: string;
home: string;
tmpdir: string;
/** Path separator: `/`, or `\\` on Windows. */
separator: string;
/** The daemon's working directory, where Node's `path.resolve` would fall back to `process.cwd()`. */
cwd: string;
/** Windows: per-drive working directories (`C:` → `C:\\work`) of the `=C:` variables. */
driveCwds: Record<string, string>;
pid: number;
/** Operations added to protocol 1 that this daemon supports, such as `listFiles`; absent from older daemons. */
features?: string[];
}
/**
* Node's names for the daemon's system and architecture, which `hello` reports in Rust's spelling: `windows` is
* `win32`, `macos` is `darwin`, `x86_64` is `x64`, `aarch64` is `arm64`, so they compare with `process.platform` and
* `process.arch`. Other names are kept.
*/
export function nodePlatform(info: Pick<RemoteInfo, "os" | "arch">): { platform: string; arch: string } {
const platforms: Record<string, string> = { windows: "win32", macos: "darwin" };
const archs: Record<string, string> = { x86_64: "x64", aarch64: "arm64", x86: "ia32" };
return { platform: platforms[info.os] ?? info.os, arch: archs[info.arch] ?? info.arch };
}
export interface ConnectionOptions {
/**
* Command that starts the daemon, before its `serve --token <hex>` arguments, e.g. `["ssh", "-T", "--", "host",
* "~/.pi/mobile/tools/pi-env"]`. A function computes it at each start (detecting and deploying first, for example);
* its failure fails that start, and the next request tries again.
*/
command: readonly string[] | (() => Promise<readonly string[]>);
/** Receives the daemon's and the transport's diagnostic output. */
onLog?: (text: string) => void;
}
export interface Reply {
json: Json;
payload: Uint8Array;
/** The daemon session that answered; handles from it are only valid while it lives. */
session: number;
}
type Pending = {
resolve: (value: Reply) => void;
reject: (error: Error) => void;
onEvent?: (json: Json, payload: Uint8Array) => void;
};
export interface RequestOptions {
payload?: Uint8Array;
/** Aborting sends `cancel`; the request still settles with the daemon's result or error. */
signal?: AbortSignal;
/** Progress events of the request, such as `exec` output. */
onEvent?: (json: Json, payload: Uint8Array) => void;
/** The request's id and session, for `kill`. */
onStart?: (id: number, session: number) => void;
/**
* Only run in this daemon session, for requests on handles it opened. After the connection was lost and started
* again, such requests fail instead of reaching a daemon that never opened the handle.
*/
session?: number;
}
/** Lone surrogates become U+FFFD, as Node encodes strings for system calls. */
function wellFormed(_key: string, value: unknown): unknown {
return typeof value === "string" ? value.toWellFormed() : value;
}
function frame(type: number, id: number, json: Json, payload: Uint8Array = new Uint8Array(0)): Buffer {
const body = Buffer.from(JSON.stringify(json, wellFormed), "utf8");
const header = Buffer.alloc(13);
header.writeUInt32BE(9 + body.length + payload.length, 0);
header.writeUInt8(type, 4);
header.writeUInt32BE(id, 5);
header.writeUInt32BE(body.length, 9);
return Buffer.concat([header, body, payload]);
}
/** The connection failed or ended; a request may succeed once it is started again. */
function lost(message: string): RemoteError {
return new RemoteError({ code: "unknown", message, lost: true });
}
/** Whether an error means the connection was lost, not that the daemon refused the request. */
export function isConnectionLost(error: unknown): boolean {
return error instanceof RemoteError && error.fields.lost === true;
}
function closed(): RemoteError {
return new RemoteError({ code: "unknown", message: "Connection closed" });
}
/** One running daemon. */
class Session {
readonly id: number;
readonly child: ChildProcessWithoutNullStreams;
readonly token: string;
readonly pending = new Map<number, Pending>();
/** Received bytes not yet parsed, without concatenating on every chunk. */
chunks: Buffer[] = [];
buffered = 0;
synced = false;
lastSeen = Date.now();
timer: ReturnType<typeof setInterval> | undefined;
live = true;
constructor(id: number, child: ChildProcessWithoutNullStreams, token: string) {
this.id = id;
this.child = child;
this.token = token;
}
/** The first `length` buffered bytes as one buffer, without consuming them. */
peek(length: number): Buffer {
if (this.chunks[0]!.length < length) {
this.chunks = [Buffer.concat(this.chunks)];
}
return this.chunks[0]!.subarray(0, length);
}
consume(length: number): void {
this.buffered -= length;
while (length > 0) {
const first = this.chunks[0]!;
if (first.length <= length) {
this.chunks.shift();
length -= first.length;
} else {
this.chunks[0] = first.subarray(length);
length = 0;
}
}
}
}
/**
* One connection to a pi-env daemon: started lazily on the first request and started again after it is lost. Requests
* in flight when it is lost fail with code `unknown`; for mutations their outcome is then unknown.
*/
export class Connection {
readonly #options: ConnectionOptions;
#session: Session | undefined;
#ready: Promise<{ info: RemoteInfo; session: Session }> | undefined;
#sessions = 0;
#nextId = 1;
#closed = false;
constructor(options: ConnectionOptions) {
this.#options = options;
}
/** Connect if needed and return what the daemon reported about its machine. */
async info(): Promise<RemoteInfo> {
return (await this.#connect()).info;
}
/** Connect if needed and return the live session's id. */
async session(): Promise<number> {
return (await this.#connect()).session.id;
}
async request(op: string, json: Json, options: RequestOptions = {}): Promise<Reply> {
if (options.session !== undefined) {
const current = this.#session;
if (current === undefined || current.id !== options.session || !current.live) {
throw lost("pi-env connection lost");
}
return this.#send(current, op, json, options);
}
const { session } = await this.#connect();
return this.#send(session, op, json, options);
}
/** Stop the daemon; it kills everything it started. */
close(): void {
this.#closed = true;
if (this.#session) this.#teardown(this.#session, closed());
}
/** Kill one `exec` without aborting it. */
kill(id: number, session: number): void {
const current = this.#session;
if (current?.id === session && current.live) this.#write(current, frame(CANCEL, id, { mode: "kill" }));
}
#connect(): Promise<{ info: RemoteInfo; session: Session }> {
if (this.#closed) return Promise.reject(closed());
if (this.#ready === undefined) {
const ready = this.#start();
this.#ready = ready;
// A failed start is not remembered: the next request tries again.
ready.catch(() => {
if (this.#ready === ready) this.#ready = undefined;
});
}
return this.#ready;
}
#send(session: Session, op: string, json: Json, options: RequestOptions): Promise<Reply> {
if (!session.live) return Promise.reject(lost("pi-env connection lost"));
const id = this.#nextId++;
return new Promise((resolve, reject) => {
const onAbort = () => this.#write(session, frame(CANCEL, id, {}));
const done = () => options.signal?.removeEventListener("abort", onAbort);
session.pending.set(id, {
resolve: (value) => {
done();
resolve(value);
},
reject: (error) => {
done();
reject(error);
},
...(options.onEvent ? { onEvent: options.onEvent } : {}),
});
options.signal?.addEventListener("abort", onAbort, { once: true });
options.onStart?.(id, session.id);
this.#write(session, frame(REQUEST, id, { ...json, op }, options.payload));
if (options.signal?.aborted) onAbort();
});
}
#write(session: Session, data: Buffer): void {
if (session.live) session.child.stdin.write(data);
}
async #start(): Promise<{ info: RemoteInfo; session: Session }> {
let command: readonly string[];
try {
const option = this.#options.command;
command = typeof option === "function" ? await option() : option;
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
throw new RemoteError({ code: "spawn_error", message, lost: true });
}
if (this.#closed) throw closed();
const [program, ...args] = command;
if (program === undefined) throw new RemoteError({ code: "spawn_error", message: "No daemon command" });
const token = randomBytes(16).toString("hex");
const child = spawn(program, [...args, "serve", "--token", token], { stdio: ["pipe", "pipe", "pipe"] });
const session = new Session(++this.#sessions, child, token);
this.#session = session;
let startTimer: ReturnType<typeof setTimeout> | undefined;
const failed = new Promise<never>((_resolve, reject) => {
startTimer = setTimeout(
() => reject(lost(`pi-env did not answer within ${START_TIMEOUT_MS / 1000} s`)),
START_TIMEOUT_MS,
);
child.once("error", (error) =>
reject(new RemoteError({ code: "spawn_error", message: error.message, lost: true })),
);
child.once("exit", (code) => reject(lost(`pi-env exited with code ${code} before it was ready`)));
});
failed.catch(() => {});
child.stdout.on("data", (chunk: Buffer) => this.#onData(session, chunk));
child.stderr.on("data", (chunk: Buffer) => this.#options.onLog?.(chunk.toString("utf8")));
child.on("error", () => this.#teardown(session, lost("pi-env connection lost")));
child.on("exit", () => this.#teardown(session, lost("pi-env connection lost")));
child.stdin.on("error", () => {});
session.timer = setInterval(() => {
this.#write(session, frame(PING, 0, {}));
if (Date.now() - session.lastSeen > SILENCE_LIMIT_MS) {
this.#teardown(session, lost("pi-env connection timed out"));
}
}, PING_INTERVAL_MS);
session.timer.unref();
try {
const { json } = await Promise.race([this.#send(session, "hello", { protocol: 1 }, {}), failed]);
if (json.protocol !== 1) throw lost(`Unsupported protocol ${json.protocol}`);
return { info: json as unknown as RemoteInfo, session };
} catch (error) {
this.#teardown(session, error instanceof Error ? error : lost(String(error)));
throw error;
} finally {
clearTimeout(startTimer);
}
}
#teardown(session: Session, error: Error): void {
if (!session.live) return;
session.live = false;
clearInterval(session.timer);
if (this.#session === session) {
this.#session = undefined;
this.#ready = undefined;
}
session.child.stdin.end();
session.child.kill();
const pending = [...session.pending.values()];
session.pending.clear();
for (const request of pending) request.reject(error);
}
#onData(session: Session, chunk: Buffer): void {
if (!session.live) return;
session.lastSeen = Date.now();
session.chunks.push(chunk);
session.buffered += chunk.length;
if (!session.synced) {
// Shell startup files may print before the daemon runs; skip everything before its sync line.
const marker = Buffer.from(`PI-ENV ${session.token}\n`, "utf8");
const all = session.peek(session.buffered);
const index = all.indexOf(marker);
if (index === -1) {
if (session.buffered > 1024 * 1024) {
session.consume(session.buffered - marker.length);
}
return;
}
const noise = all.subarray(0, index).toString("utf8").trim();
if (noise !== "") this.#options.onLog?.(noise);
session.consume(index + marker.length);
session.synced = true;
}
while (session.live && session.buffered >= 4) {
const length = session.peek(4).readUInt32BE(0);
if (length < 9 || length > MAX_FRAME) {
this.#teardown(session, lost("Corrupt frame from pi-env"));
return;
}
if (session.buffered < 4 + length) return;
const body = session.peek(4 + length).subarray(4);
const type = body.readUInt8(0);
const id = body.readUInt32BE(1);
const jsonLength = body.readUInt32BE(5);
let json: Json;
try {
if (9 + jsonLength > length) throw new Error("JSON length out of range");
const parsed: unknown = JSON.parse(body.subarray(9, 9 + jsonLength).toString("utf8"));
if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed))
throw new Error("not an object");
json = parsed as Json;
} catch {
this.#teardown(session, lost("Corrupt frame from pi-env"));
return;
}
// A plain Uint8Array copy, like `NodeExecutionEnv`'s reader results, not a view of the receive buffer.
const payload = new Uint8Array(body.subarray(9 + jsonLength));
session.consume(4 + length);
this.#dispatch(session, type, id, json, payload);
}
}
#dispatch(session: Session, type: number, id: number, json: Json, payload: Uint8Array): void {
const pending = session.pending.get(id);
if (type === EVENT) {
pending?.onEvent?.(json, payload);
return;
}
if (type !== RESULT && type !== ERROR) return;
if (pending === undefined) return;
session.pending.delete(id);
if (type === RESULT) pending.resolve({ json, payload, session: session.id });
else pending.reject(new RemoteError(json));
}
}

34
third_party/pi/packages/env/src/errors.ts vendored Normal file
View File

@@ -0,0 +1,34 @@
import { err, FileError, type Result } from "@earendil-works/pi-durable/env";
import { RemoteError } from "./connection.ts";
export function abortResult<T>(signal: AbortSignal | undefined, path?: string): Result<T, FileError> | undefined {
return signal?.aborted ? err(new FileError("aborted", "aborted", path)) : undefined;
}
/** `NodeExecutionEnv`'s mapping of Node error codes to `FileError` codes. */
export function toFileError(error: unknown, fallbackPath?: string): FileError {
if (error instanceof FileError) return error;
if (!(error instanceof RemoteError)) {
const cause = error instanceof Error ? error : new Error(String(error));
return new FileError("unknown", cause.message, fallbackPath, cause);
}
const path = error.path ?? fallbackPath;
switch (error.code) {
case "aborted":
return new FileError("aborted", error.message, path, error);
case "ENOENT":
return new FileError("not_found", error.message, path, error);
case "EACCES":
case "EPERM":
return new FileError("permission_denied", error.message, path, error);
case "ENOTDIR":
return new FileError("not_directory", error.message, path, error);
case "EISDIR":
return new FileError("is_directory", error.message, path, error);
case "EINVAL":
case "SYMLINK":
case "NOT_REGULAR":
return new FileError("invalid", error.message, path, error);
}
return new FileError("unknown", error.message, path, error);
}

32
third_party/pi/packages/env/src/index.ts vendored Normal file
View File

@@ -0,0 +1,32 @@
export {
Connection,
type ConnectionOptions,
isConnectionLost,
nodePlatform,
RemoteError,
type RemoteInfo,
type Reply,
type RequestOptions,
} from "./connection.ts";
export { LocalExecutionEnv, type LocalExecutionEnvOptions } from "./list-files.ts";
export { RemoteExecutionEnv, type RemoteExecutionEnvOptions } from "./remote-env.ts";
export {
acceptHostKey,
connectSsh,
deployDaemon,
detectPlatform,
forgetHostKey,
HostKeyChangedError,
HostKeyUnknownError,
localDaemon,
packagedDaemon,
type RemotePlatform,
type SshConnectOptions,
SshError,
type SshTarget,
scanHostKey,
sshArguments,
sshConnection,
} from "./ssh.ts";
export { openTerminal, RemoteTerminal, type TerminalForeground, type TerminalOptions } from "./terminal.ts";
export { RemoteWatcher, type RemoteWatchOptions } from "./watch.ts";

102
third_party/pi/packages/env/src/list-files.ts vendored Normal file
View File

@@ -0,0 +1,102 @@
import type { Context } from "@earendil-works/chord";
import {
type ExecutionEnv,
err,
FileError,
type FileKind,
type FileListEntry,
type FileListing,
type ListFilesOptions,
ok,
type Result,
} from "@earendil-works/pi-durable/env";
import { NodeExecutionEnv } from "@earendil-works/pi-durable/env/node";
import type { Connection } from "./connection.ts";
import { abortResult, toFileError } from "./errors.ts";
/** Kind bytes of `listFiles` records: `f`, `d`, `l`. */
const KINDS: Record<number, FileKind> = { 102: "file", 100: "directory", 108: "symlink" };
/**
* `listFiles` of an absolute path through the daemon: fails with `not_supported` if the daemon does not list files
* (a release before `listFiles`), so callers fall back to their own walk.
*/
export async function listFilesOver(
connection: Connection,
path: string,
options: ListFilesOptions | undefined,
context: Context,
): Promise<Result<FileListing, FileError>> {
for (const key of ["maxEntries", "maxBytes", "timeoutMs"] as const) {
const value = options?.[key];
if (value !== undefined && (!Number.isSafeInteger(value) || value < 0)) {
return err(new FileError("invalid", `${key} must be a non-negative safe integer`, path));
}
}
const aborted = abortResult<FileListing>(context.abortSignal, path);
if (aborted) return aborted;
try {
if (!(await connection.info()).features?.includes("listFiles")) {
return err(new FileError("not_supported", "The pi-env daemon does not list files; update it", path));
}
const { json, payload } = await connection.request(
"listFiles",
// Fields left undefined are not sent; the daemon applies the defaults.
{
path,
hidden: options?.hidden,
gitignore: options?.gitignore,
exclude: options?.exclude,
maxEntries: options?.maxEntries,
maxBytes: options?.maxBytes,
timeoutMs: options?.timeoutMs,
},
context.abortSignal ? { signal: context.abortSignal } : {},
);
const afterAbort = abortResult<FileListing>(context.abortSignal, path);
if (afterAbort) return afterAbort;
// Records of a kind byte, the UTF-8 path, and a NUL.
const entries: FileListEntry[] = [];
const bytes = Buffer.from(payload.buffer, payload.byteOffset, payload.length);
for (let start = 0; start < bytes.length; ) {
const end = bytes.indexOf(0, start);
if (end === -1) break;
const kind = KINDS[bytes[start]!];
if (kind !== undefined) entries.push({ path: bytes.toString("utf8", start + 1, end), kind });
start = end + 1;
}
return ok({ entries, truncated: json.truncated === true });
} catch (error) {
return err(toFileError(error, path));
}
}
export type LocalExecutionEnvOptions = ConstructorParameters<typeof NodeExecutionEnv>[0] & {
/** A connection to this machine's daemon, `new Connection({ command: [localDaemon()] })`, for `listFiles`. */
connection: Connection;
};
/**
* `NodeExecutionEnv` with `listFiles` through a pi-env daemon on the same machine, so local and remote environments
* list files with the same code. Everything else is `NodeExecutionEnv`'s. The connection belongs to the caller, who
* closes it; several environments may share it.
*/
export class LocalExecutionEnv extends NodeExecutionEnv implements ExecutionEnv {
readonly #connection: Connection;
constructor(options: LocalExecutionEnvOptions) {
const { connection, ...node } = options;
super(node);
this.#connection = connection;
}
async listFiles(
path: string,
options: ListFilesOptions | undefined,
context: Context,
): Promise<Result<FileListing, FileError>> {
const resolved = await this.absolutePath(path, context);
if (!resolved.ok) return resolved;
return listFilesOver(this.#connection, resolved.value, options, context);
}
}

957
third_party/pi/packages/env/src/remote-env.ts vendored Normal file
View File

@@ -0,0 +1,957 @@
import { randomUUID } from "node:crypto";
import { type PlatformPath, posix, win32 } from "node:path";
import { StringDecoder } from "node:string_decoder";
import { fileURLToPath } from "node:url";
import type { Context } from "@earendil-works/chord";
import {
type BinaryReader,
type DirReader,
type ExecutionEnv,
ExecutionError,
err,
FileError,
type FileInfo,
type FileKind,
type FileListing,
type FileWatcher,
type LineScan,
type ListFilesOptions,
ok,
type Result,
type ShellExecOptions,
type ShellExecResult,
type ShellOutputInfo,
type ShellOutputSkip,
StreamDecoder,
type TextLine,
type TextLineReader,
type WatchChange,
type WatchTarget,
} from "@earendil-works/pi-durable/env";
import { type Connection, type Json, nodePlatform, RemoteError, type RemoteInfo } from "./connection.ts";
import { abortResult, toFileError } from "./errors.ts";
import { listFilesOver } from "./list-files.ts";
import { RemoteWatcher, type RemoteWatchOptions } from "./watch.ts";
const MAX_TIMEOUT_MS = 2_147_483_647;
const MAX_TIMEOUT_SECONDS = MAX_TIMEOUT_MS / 1000;
/** Bytes per `pread` request of a binary reader; several are in flight at once. */
const READ_CHUNK = 256 * 1024;
const READ_DEPTH = 8;
/** Node's `writeFile` writes at most this much per call and checks for an abort before each. */
const WRITE_CHUNK = 512 * 1024;
const WRITE_DEPTH = 8;
/** Node's `readFile`: chunk of known-size reads, chunk of unknown-size reads, and the largest file it reads. */
const READ_FILE_CHUNK = 512 * 1024;
const READ_FILE_UNKNOWN_CHUNK = 64 * 1024;
const READ_FILE_MAX = 2 ** 31 - 1;
/** `NodeTextLineReader`'s read size. */
const LINE_CHUNK = 64 * 1024;
export interface RemoteExecutionEnvOptions {
connection: Connection;
/** The file namespace: equal ids see the same files (`FileSystem.id`), e.g. `pi-env:<env name>`. */
id: string;
cwd: string;
shellPath?: string;
/** Added to the remote environment of every command that inherits it, like `NodeExecutionEnv`'s `shellEnv`. */
shellEnv?: Record<string, string>;
/** How the daemon watches, like `NodeExecutionEnv`'s `watch` option. */
watch?: RemoteWatchOptions;
}
/** An info record from the daemon. */
type RemoteFileInfo = { name: string; kind: FileKind | "other"; size: number; mtimeSec: number; mtimeNsec: number };
type RemoteDirEntry = { name: string; raw?: number[]; info?: RemoteFileInfo; error?: Json };
/** Node's path rules of the remote system. */
export async function remotePath(connection: Connection): Promise<PlatformPath> {
return nodePlatform(await connection.info()).platform === "win32" ? win32 : posix;
}
function toInfo(path: string, remote: RemoteFileInfo, paths: PlatformPath): Result<FileInfo, FileError> {
if (remote.kind === "other") return err(new FileError("invalid", "Unsupported file type", path));
return ok({
name: paths.basename(path),
path,
kind: remote.kind,
size: remote.size,
mtimeMs: remote.mtimeSec * 1000 + remote.mtimeNsec / 1e6,
});
}
function concat(chunks: readonly Uint8Array[], total: number): Uint8Array {
if (chunks.length === 1) return chunks[0]!;
const bytes = new Uint8Array(total);
let position = 0;
for (const chunk of chunks) {
bytes.set(chunk, position);
position += chunk.length;
}
return bytes;
}
/**
* Read `length` bytes from `offset` as consecutive reads of at most `chunk` bytes, several in flight. Ends early at
* the end of the file. A short read continues from where it ended, as sequential reads would. `aborted` is checked
* after each read; `undefined` means aborted.
*/
async function readRange(
read: (offset: number, length: number) => Promise<Uint8Array>,
offset: number,
length: number,
chunk: number,
aborted: () => boolean,
): Promise<{ chunks: Uint8Array[]; total: number } | undefined> {
const chunks: Uint8Array[] = [];
let total = 0;
let next = offset;
const inFlight: { length: number; bytes: Promise<Uint8Array> }[] = [];
while (total < length) {
while (inFlight.length < READ_DEPTH && next < offset + length) {
const size = Math.min(offset + length - next, chunk);
const bytes = read(next, size);
bytes.catch(() => {});
inFlight.push({ length: size, bytes });
next += size;
}
const head = inFlight.shift()!;
const bytes = await head.bytes;
if (aborted()) return undefined;
if (bytes.length === 0) break;
chunks.push(bytes);
total += bytes.length;
if (bytes.length < head.length) {
// Reads already in flight assumed a full read; continue right after this one.
inFlight.length = 0;
next = offset + total;
}
}
return { chunks, total };
}
/** A daemon handle: valid only in the session that opened it. */
class Handle {
readonly env: RemoteExecutionEnv;
readonly id: number;
readonly session: number;
readonly path: string;
constructor(env: RemoteExecutionEnv, id: number, session: number, path: string) {
this.env = env;
this.id = id;
this.session = session;
this.path = path;
}
request(op: string, json: Json = {}, options: { signal?: AbortSignal; payload?: Uint8Array } = {}) {
return this.env.connection.request(op, { ...json, handle: this.id }, { ...options, session: this.session });
}
async pread(offset: number | undefined, length: number): Promise<Uint8Array> {
return (await this.request("pread", { ...(offset === undefined ? {} : { offset }), length })).payload;
}
async close(): Promise<void> {
await this.request("close").catch(() => undefined);
}
}
class RemoteBinaryReader implements BinaryReader {
readonly #handle: Handle;
#closed = false;
constructor(handle: Handle) {
this.#handle = handle;
}
get #path(): string {
return this.#handle.path;
}
#closedResult<T>(): Result<T, FileError> | undefined {
return this.#closed ? err(new FileError("invalid", "Binary reader is closed", this.#path)) : undefined;
}
async info(context: Context): Promise<Result<FileInfo, FileError>> {
const early = abortResult<FileInfo>(context.abortSignal, this.#path) ?? this.#closedResult<FileInfo>();
if (early) return early;
try {
const { json } = await this.#handle.request("fstat");
return toInfo(this.#path, json as unknown as RemoteFileInfo, await remotePath(this.#handle.env.connection));
} catch (error) {
return err(toFileError(error, this.#path));
}
}
async read(offset: number, length: number, context: Context): Promise<Result<Uint8Array, FileError>> {
const early = abortResult<Uint8Array>(context.abortSignal, this.#path) ?? this.#closedResult<Uint8Array>();
if (early) return early;
if (!Number.isSafeInteger(offset) || offset < 0 || !Number.isSafeInteger(length) || length < 0) {
return err(new FileError("invalid", "Offset and length must be non-negative safe integers", this.#path));
}
try {
const read = await readRange(
(at, size) => this.#handle.pread(at, size),
offset,
length,
READ_CHUNK,
() => context.abortSignal?.aborted === true,
);
if (read === undefined) return err(new FileError("aborted", "aborted", this.#path));
return ok(concat(read.chunks, read.total));
} catch (error) {
return err(toFileError(error, this.#path));
}
}
async scanLines(
options: { startLine: number; endLine?: number },
context: Context,
): Promise<Result<LineScan, FileError>> {
const early = abortResult<LineScan>(context.abortSignal, this.#path) ?? this.#closedResult<LineScan>();
if (early) return early;
const { startLine, endLine } = options;
if (
!Number.isSafeInteger(startLine) ||
startLine < 0 ||
(endLine !== undefined && (!Number.isSafeInteger(endLine) || endLine <= startLine))
) {
return err(new FileError("invalid", "Invalid line range", this.#path));
}
try {
const { json } = await this.#handle.request(
"scanLines",
{ startLine, ...(endLine === undefined ? {} : { endLine }) },
context.abortSignal ? { signal: context.abortSignal } : {},
);
return ok(json as unknown as LineScan);
} catch (error) {
return err(toFileError(error, this.#path));
}
}
async close(_context: Context): Promise<void> {
if (this.#closed) return;
this.#closed = true;
await this.#handle.close();
}
}
class RemoteDirReader implements DirReader {
readonly #handle: Handle;
#done = false;
#closed = false;
constructor(handle: Handle) {
this.#handle = handle;
}
async next(
maxEntries: number,
context: Context,
): Promise<Result<{ entries: FileInfo[]; done: boolean }, FileError>> {
const path = this.#handle.path;
const aborted = abortResult<{ entries: FileInfo[]; done: boolean }>(context.abortSignal, path);
if (aborted) return aborted;
if (this.#closed) return err(new FileError("invalid", "Directory reader is closed", path));
if (!Number.isSafeInteger(maxEntries) || maxEntries <= 0) {
return err(new FileError("invalid", "maxEntries must be a positive safe integer", path));
}
const entries: FileInfo[] = [];
try {
const paths = await remotePath(this.#handle.env.connection);
// Like `NodeDirReader`, count only entries that are part of the listing.
while (!this.#done && entries.length < maxEntries) {
const { json } = await this.#handle.request("readdir", { max: maxEntries - entries.length });
const loopAbort = abortResult<{ entries: FileInfo[]; done: boolean }>(context.abortSignal, path);
if (loopAbort) return loopAbort;
for (const entry of json.entries as RemoteDirEntry[]) {
const entryPath = paths.resolve(path, entry.name);
if (entry.error !== undefined) {
// Removed between enumeration and lstat: not part of the listing any more.
if (entry.error.code === "ENOENT") continue;
return err(toFileError(new RemoteError(entry.error), entryPath));
}
const info = toInfo(entryPath, entry.info!, paths);
if (info.ok) entries.push(info.value);
}
this.#done = json.done === true;
}
return ok({ entries, done: this.#done });
} catch (error) {
return err(toFileError(error, path));
}
}
async close(_context: Context): Promise<void> {
if (this.#closed) return;
this.#closed = true;
await this.#handle.close();
}
}
/** `NodeTextLineReader`: positional reads of a file opened like `open(path, "r")`. */
class RemoteTextLineReader implements TextLineReader {
readonly #handle: Handle;
readonly #decoder = new StreamDecoder();
#offset = 0;
#buffered = "";
#ended = false;
#closed = false;
constructor(handle: Handle) {
this.#handle = handle;
}
async readLine(context: Context): Promise<Result<TextLine | undefined, FileError>> {
const path = this.#handle.path;
const aborted = abortResult<TextLine | undefined>(context.abortSignal, path);
if (aborted) return aborted;
if (this.#closed) return err(new FileError("invalid", "Text line reader is closed", path));
try {
while (true) {
const newline = this.#buffered.indexOf("\n");
if (newline !== -1) {
const text = this.#buffered.slice(0, newline);
this.#buffered = this.#buffered.slice(newline + 1);
return ok({ text, terminated: true });
}
if (this.#ended) {
if (this.#buffered.length === 0) return ok(undefined);
const text = this.#buffered;
this.#buffered = "";
return ok({ text, terminated: false });
}
const bytes = await this.#handle.pread(this.#offset, LINE_CHUNK);
const afterReadAbort = abortResult<TextLine | undefined>(context.abortSignal, path);
if (afterReadAbort) return afterReadAbort;
this.#offset += bytes.length;
if (bytes.length === 0) {
this.#buffered += this.#decoder.decode();
this.#ended = true;
} else {
this.#buffered += this.#decoder.decode(bytes);
}
}
} catch (error) {
return err(toFileError(error, path));
}
}
async close(_context: Context): Promise<void> {
if (this.#closed) return;
this.#closed = true;
this.#buffered = "";
await this.#handle.close();
}
}
/**
* An `ExecutionEnv` on another machine, reached through a pi-env daemon. Results match `NodeExecutionEnv` running on
* that machine: the daemon performs the system calls, and this class applies Node's path, error, and result rules.
*/
export class RemoteExecutionEnv implements ExecutionEnv {
readonly id: string;
cwd: string;
readonly connection: Connection;
readonly #shellPath: string | undefined;
readonly #shellEnv: Record<string, string> | undefined;
readonly #watchOptions: RemoteWatchOptions;
/** Running commands this environment started, for `cleanup()`. */
readonly #running = new Map<number, number>();
constructor(options: RemoteExecutionEnvOptions) {
this.id = options.id;
this.cwd = options.cwd;
this.connection = options.connection;
this.#shellPath = options.shellPath;
this.#shellEnv = options.shellEnv;
this.#watchOptions = options.watch ?? {};
}
/** Node's `resolvePath` on the remote system, with its home directory and working directories. */
async #resolve(path: string): Promise<string> {
const info = await this.connection.info();
const windows = nodePlatform(info).platform === "win32";
const paths = windows ? win32 : posix;
let normalized = path;
if (normalized === "~") {
normalized = info.home;
} else if (normalized.startsWith("~/") || (windows && normalized.startsWith("~\\"))) {
normalized = paths.join(info.home, normalized.slice(2));
} else if (normalized.startsWith("file://")) {
try {
normalized = fileURLToPath(normalized, { windows });
} catch {
// Keep malformed URLs as ordinary paths, as Node does.
}
}
if (paths.isAbsolute(normalized)) return paths.resolve(normalized);
if (windows) {
// A drive-relative path on another drive: Node on Windows resolves it against that drive's working directory
// (`=D:`), else its own working directory if on that drive, else the drive's root.
const drive = /^([a-zA-Z]:)(?![\\/])/.exec(normalized)?.[1];
if (drive !== undefined && drive.toLowerCase() !== this.cwd.slice(0, 2).toLowerCase()) {
let base = info.driveCwds[drive.toUpperCase()] ?? info.cwd;
if (base.slice(0, 2).toLowerCase() !== drive.toLowerCase() && base[2] === "\\") base = `${drive}\\`;
return win32.resolve(base, normalized);
}
}
return paths.resolve(this.cwd, normalized);
}
async #fileOp<T>(
path: string,
context: Context,
run: (resolved: string) => Promise<T>,
check: "before" | "both" = "before",
): Promise<Result<T, FileError>> {
let resolved: string;
try {
resolved = await this.#resolve(path);
} catch (error) {
return err(toFileError(error, path));
}
const aborted = abortResult<T>(context.abortSignal, resolved);
if (aborted) return aborted;
try {
const value = await run(resolved);
if (check === "both") {
const after = abortResult<T>(context.abortSignal, resolved);
if (after) return after;
}
return ok(value);
} catch (error) {
return err(toFileError(error, resolved));
}
}
async #open(resolved: string, json: Json): Promise<{ handle: Handle; json: Json }> {
const reply = await this.connection.request("open", { path: resolved, ...json });
return { handle: new Handle(this, reply.json.handle as number, reply.session, resolved), json: reply.json };
}
async absolutePath(path: string, _context: Context): Promise<Result<string, FileError>> {
try {
return ok(await this.#resolve(path));
} catch (error) {
return err(toFileError(error, path));
}
}
async joinPath(parts: string[], _context: Context): Promise<Result<string, FileError>> {
try {
return ok((await remotePath(this.connection)).join(...parts));
} catch (error) {
return err(toFileError(error));
}
}
/** Node's `readFile`, with or without UTF-8 decoding, over a file opened like `open(path, "r")`. */
async #readFile(path: string, encoding: boolean, context: Context): Promise<Result<Uint8Array | string, FileError>> {
const signal = context.abortSignal;
return this.#fileOp(path, context, async (resolved) => {
const { handle, json } = await this.#open(resolved, { mode: "read" });
const aborted = () => signal?.aborted === true;
const abort = () => new FileError("aborted", "The operation was aborted", resolved);
try {
if (json.statError !== undefined) throw new RemoteError(json.statError as Json);
if (aborted()) throw abort();
const stat = json.stat as RemoteFileInfo;
const size = stat.kind === "file" ? stat.size : 0;
if (size > READ_FILE_MAX) {
throw new FileError("unknown", `File size (${size}) is greater than 2 GiB`, resolved);
}
if (size === 0) {
// Unknown size: sequential reads until the end, as for FIFOs and devices.
const decoder = new StringDecoder("utf8");
const chunks: Uint8Array[] = [];
let total = 0;
let text = "";
while (true) {
if (aborted()) throw abort();
const bytes = await handle.pread(undefined, READ_FILE_UNKNOWN_CHUNK);
if (bytes.length === 0) break;
total += bytes.length;
chunks.push(bytes);
if (encoding) text += decoder.write(bytes);
}
if (!encoding) return Buffer.concat(chunks, total);
return total === 0 ? "" : text + decoder.end();
}
if (!encoding) {
// A single read of the whole size, or reads of 512 KiB until the size or the end of the file.
if (size <= READ_FILE_CHUNK) return Buffer.from(await handle.pread(0, size));
const read = await readRange(
(at, length) => handle.pread(at, length),
0,
size,
READ_FILE_CHUNK,
aborted,
);
if (read === undefined) throw abort();
return Buffer.concat(read.chunks, read.total);
}
// Decoding: reads of min(size, 512 KiB) until the size, a short read, or the end of the file.
const length = Math.min(size, READ_FILE_CHUNK);
const decoder = new StringDecoder("utf8");
let text = "";
let total = 0;
let first = true;
const inFlight: Promise<Uint8Array>[] = [];
let next = 0;
while (true) {
while (inFlight.length < READ_DEPTH) {
const bytes = handle.pread(next, length);
bytes.catch(() => {});
inFlight.push(bytes);
next += length;
}
const bytes = await inFlight.shift()!;
if (aborted()) throw abort();
total += bytes.length;
if (bytes.length === 0 || total === size || bytes.length !== length) {
return first ? Buffer.from(bytes).toString("utf8") : text + decoder.end(Buffer.from(bytes));
}
text += decoder.write(Buffer.from(bytes));
first = false;
}
} finally {
await handle.close();
}
});
}
async readTextFile(path: string, context: Context): Promise<Result<string, FileError>> {
return (await this.#readFile(path, true, context)) as Result<string, FileError>;
}
async readBinaryFile(path: string, context: Context): Promise<Result<Uint8Array, FileError>> {
return (await this.#readFile(path, false, context)) as Result<Uint8Array, FileError>;
}
async openBinaryReader(
path: string,
options: { noFollow?: boolean } | undefined,
context: Context,
): Promise<Result<BinaryReader, FileError>> {
const opened = await this.#fileOp(path, context, async (resolved) => {
return (await this.#open(resolved, { noFollow: options?.noFollow === true })).handle;
});
if (!opened.ok) return opened;
const aborted = abortResult<BinaryReader>(context.abortSignal, opened.value.path);
if (aborted) {
await opened.value.close();
return aborted;
}
return ok(new RemoteBinaryReader(opened.value));
}
async openTextLineReader(path: string, context: Context): Promise<Result<TextLineReader, FileError>> {
const opened = await this.#fileOp(path, context, async (resolved) => {
return (await this.#open(resolved, { mode: "read" })).handle;
});
if (!opened.ok) return opened;
const aborted = abortResult<TextLineReader>(context.abortSignal, opened.value.path);
if (aborted) {
await opened.value.close();
return aborted;
}
return ok(new RemoteTextLineReader(opened.value));
}
async readTextLines(
path: string,
options: { maxLines?: number } | undefined,
context: Context,
): Promise<Result<string[], FileError>> {
if (options?.maxLines !== undefined && options.maxLines <= 0) return ok([]);
const opened = await this.openTextLineReader(path, context);
if (!opened.ok) return opened;
const lines: string[] = [];
try {
while (options?.maxLines === undefined || lines.length < options.maxLines) {
const line = await opened.value.readLine(context);
if (!line.ok) return line;
if (line.value === undefined) break;
lines.push(line.value.text);
}
return ok(lines);
} finally {
await opened.value.close(context);
}
}
/**
* Node's `writeFile` (abort checked before each 512 KiB write) or `appendFile` (no checks, one after): the first
* request creates parents, opens, and writes the first chunk; later chunks go to the same open file.
*/
async #write(path: string, content: string | Uint8Array, append: boolean, context: Context) {
const bytes = typeof content === "string" ? Buffer.from(content, "utf8") : content;
const signal = append ? undefined : context.abortSignal;
return this.#fileOp(
path,
context,
async (resolved) => {
const first = bytes.subarray(0, WRITE_CHUNK);
const keep = bytes.length > WRITE_CHUNK;
const reply = await this.connection.request("write", { path: resolved, append, keep }, { payload: first });
if (!keep) return;
const handle = new Handle(this, reply.json.handle as number, reply.session, resolved);
const inFlight: Promise<unknown>[] = [];
try {
for (let offset = WRITE_CHUNK; offset < bytes.length; offset += WRITE_CHUNK) {
if (signal?.aborted) {
// A chunk that failed before this check fails the write, as it would have in sequence.
const failed = (await Promise.allSettled(inFlight)).find((result) => result.status === "rejected");
if (failed !== undefined) throw failed.reason;
throw new FileError("aborted", "The operation was aborted", resolved);
}
const written = handle.request(
"writeChunk",
{},
{ payload: bytes.subarray(offset, offset + WRITE_CHUNK) },
);
written.catch(() => {});
inFlight.push(written);
if (inFlight.length >= WRITE_DEPTH) await inFlight.shift();
}
// The first failure; later chunks are refused by the daemon, so the file has no gaps.
for (const written of inFlight) await written;
} finally {
await Promise.allSettled(inFlight);
await handle.close();
}
},
append ? "both" : "before",
);
}
writeFile(path: string, content: string | Uint8Array, context: Context): Promise<Result<void, FileError>> {
return this.#write(path, content, false, context);
}
appendFile(path: string, content: string | Uint8Array, context: Context): Promise<Result<void, FileError>> {
return this.#write(path, content, true, context);
}
async truncateFile(path: string, size: number, context: Context): Promise<Result<void, FileError>> {
return this.#fileOp(
path,
context,
async (resolved) => {
if (!Number.isSafeInteger(size) || size < 0) {
throw new FileError("invalid", "File size must be a non-negative safe integer", resolved);
}
await this.connection.request("truncate", { path: resolved, size });
},
"both",
);
}
flushFile(path: string, context: Context): Promise<Result<void, FileError>> {
return this.#fileOp(
path,
context,
async (resolved) => {
await this.connection.request("fsync", { path: resolved });
},
"both",
);
}
async renameFile(sourcePath: string, destinationPath: string, context: Context): Promise<Result<void, FileError>> {
let source: string;
let destination: string;
try {
source = await this.#resolve(sourcePath);
destination = await this.#resolve(destinationPath);
} catch (error) {
return err(toFileError(error, sourcePath));
}
const aborted = abortResult<void>(context.abortSignal, destination);
if (aborted) return aborted;
try {
await this.connection.request("rename", { path: source, to: destination });
return ok(undefined);
} catch (error) {
return err(toFileError(error, source));
}
}
async fileInfo(path: string, context: Context): Promise<Result<FileInfo, FileError>> {
const result = await this.#fileOp(path, context, async (resolved) => {
const { json } = await this.connection.request("lstat", { path: resolved });
return toInfo(resolved, json as unknown as RemoteFileInfo, await remotePath(this.connection));
});
return result.ok ? result.value : result;
}
listDir(path: string, context: Context): Promise<Result<FileInfo[], FileError>> {
return this.#fileOp(path, context, async (resolved) => {
const { handle } = await this.#openDir(resolved);
const entries: RemoteDirEntry[] = [];
try {
for (let done = false; !done; ) {
const { json } = await handle.request("readdir", { max: 1000 });
entries.push(...(json.entries as RemoteDirEntry[]));
done = json.done === true;
}
} finally {
await handle.close();
}
// Node's `readdir` fails on any entry it cannot lstat. libuv sorts names by their bytes on POSIX; on Windows it
// keeps the file system's order, which the daemon reports.
const paths = await remotePath(this.connection);
if (paths === posix) {
const key = (entry: RemoteDirEntry) => Buffer.from(entry.raw ?? Buffer.from(entry.name, "utf8"));
entries.sort((a, b) => Buffer.compare(key(a), key(b)));
}
const infos: FileInfo[] = [];
for (const entry of entries) {
if (context.abortSignal?.aborted) throw new FileError("aborted", "aborted", resolved);
const entryPath = paths.resolve(resolved, entry.name);
if (entry.error !== undefined) throw toFileError(new RemoteError(entry.error), entryPath);
const info = toInfo(entryPath, entry.info!, paths);
if (info.ok) infos.push(info.value);
}
return infos;
});
}
/** Through the daemon; `not_supported` with a daemon released before `listFiles`. */
async listFiles(
path: string,
options: ListFilesOptions | undefined,
context: Context,
): Promise<Result<FileListing, FileError>> {
let resolved: string;
try {
resolved = await this.#resolve(path);
} catch (error) {
return err(toFileError(error, path));
}
return listFilesOver(this.connection, resolved, options, context);
}
async #openDir(resolved: string): Promise<{ handle: Handle }> {
const reply = await this.connection.request("opendir", { path: resolved });
return { handle: new Handle(this, reply.json.handle as number, reply.session, resolved) };
}
async openDirReader(path: string, context: Context): Promise<Result<DirReader, FileError>> {
const opened = await this.#fileOp(path, context, async (resolved) => (await this.#openDir(resolved)).handle);
if (!opened.ok) return opened;
const aborted = abortResult<DirReader>(context.abortSignal, opened.value.path);
if (aborted) {
await opened.value.close();
return aborted;
}
return ok(new RemoteDirReader(opened.value));
}
canonicalPath(path: string, context: Context): Promise<Result<string, FileError>> {
return this.#fileOp(path, context, async (resolved) => {
const { json } = await this.connection.request("realpath", { path: resolved });
return json.path as string;
});
}
async exists(path: string, context: Context): Promise<Result<boolean, FileError>> {
const result = await this.fileInfo(path, context);
if (result.ok) return ok(true);
if (result.error.code === "not_found") return ok(false);
return err(result.error);
}
createDir(
path: string,
options: { recursive?: boolean } | undefined,
context: Context,
): Promise<Result<void, FileError>> {
return this.#fileOp(path, context, async (resolved) => {
await this.connection.request("mkdir", { path: resolved, recursive: options?.recursive ?? true });
});
}
remove(
path: string,
options: { recursive?: boolean; force?: boolean } | undefined,
context: Context,
): Promise<Result<void, FileError>> {
return this.#fileOp(path, context, async (resolved) => {
await this.connection.request("rm", {
path: resolved,
recursive: options?.recursive ?? false,
force: options?.force ?? false,
});
});
}
async createTempDir(prefix: string | undefined, context: Context): Promise<Result<string, FileError>> {
const aborted = abortResult<string>(context.abortSignal);
if (aborted) return aborted;
try {
const { tmpdir } = await this.connection.info();
const paths = await remotePath(this.connection);
const { json } = await this.connection.request("mkdtemp", { path: paths.join(tmpdir, prefix ?? "tmp-") });
return ok(json.path as string);
} catch (error) {
return err(toFileError(error));
}
}
async createTempFile(
options: { prefix?: string; suffix?: string } | undefined,
context: Context,
): Promise<Result<string, FileError>> {
const dir = await this.createTempDir("tmp-", context);
if (!dir.ok) return dir;
let filePath = "";
try {
const paths = await remotePath(this.connection);
filePath = paths.join(dir.value, `${options?.prefix ?? ""}${randomUUID()}${options?.suffix ?? ""}`);
// Node's `writeFile(filePath, "")`, which does not create parents.
await this.connection.request(
"write",
{ path: filePath, append: false, parents: false },
{ payload: new Uint8Array(0) },
);
return ok(filePath);
} catch (error) {
return err(toFileError(error, filePath));
}
}
async watch(
targets: readonly WatchTarget[],
onChange: (change: WatchChange) => void,
context: Context,
): Promise<Result<FileWatcher, FileError>> {
const aborted = abortResult<FileWatcher>(context.abortSignal);
if (aborted) return aborted;
try {
const resolved = await Promise.all(
targets.map(async (target) => ({ ...target, path: await this.#resolve(target.path) })),
);
const afterResolve = abortResult<FileWatcher>(context.abortSignal);
if (afterResolve) return afterResolve;
const watcher = await RemoteWatcher.open(this.connection, resolved, onChange, this.#watchOptions);
const afterOpen = abortResult<FileWatcher>(context.abortSignal);
if (afterOpen) {
await watcher.close(context);
return afterOpen;
}
return ok(watcher);
} catch (error) {
return err(toFileError(error));
}
}
async exec(
command: string | readonly string[],
options: ShellExecOptions | undefined,
context: Context,
): Promise<Result<ShellExecResult, ExecutionError>> {
const signal = context.abortSignal;
if (signal?.aborted) return err(new ExecutionError("aborted", "aborted"));
const timeout = options?.timeout;
if (timeout !== undefined) {
if (!Number.isFinite(timeout) || timeout <= 0) {
return err(new ExecutionError("timeout", "Invalid timeout: must be a finite number of seconds"));
}
if (timeout * 1000 > MAX_TIMEOUT_MS) {
return err(new ExecutionError("timeout", `Invalid timeout: maximum is ${MAX_TIMEOUT_SECONDS} seconds`));
}
}
const inheritEnv = options?.inheritEnv ?? true;
let cwd: string;
try {
cwd = options?.cwd ? await this.#resolve(options.cwd) : this.cwd;
} catch (error) {
return err(new ExecutionError("unknown", error instanceof Error ? error.message : String(error)));
}
if (signal?.aborted) return err(new ExecutionError("aborted", "aborted"));
const env: Record<string, string> = {};
for (const [key, value] of Object.entries(
inheritEnv ? { ...this.#shellEnv, ...options?.env } : { ...options?.env },
)) {
if (value !== undefined) env[key.toWellFormed()] = value;
}
let callbackError: ExecutionError | undefined;
let settled = false;
let running: number | undefined;
const controller = new AbortController();
const onAbort = () => controller.abort();
signal?.addEventListener("abort", onAbort, { once: true });
try {
const { json } = await this.connection.request(
"exec",
{
...(typeof command === "string" ? { command } : { argv: [...command] }),
cwd,
env,
inheritEnv,
...(this.#shellPath === undefined ? {} : { shellPath: this.#shellPath }),
...(timeout === undefined ? {} : { timeoutMs: timeout * 1000 }),
...(options?.spill === undefined ? {} : { spill: options.spill }),
...(options?.window === undefined ? {} : { window: options.window }),
},
{
signal: controller.signal,
onStart: (requestId, session) => {
running = requestId;
this.#running.set(requestId, session);
},
onEvent: (event, payload) => {
if (settled || callbackError !== undefined || event.kind !== "output") return;
const text = Buffer.from(payload.buffer, payload.byteOffset, payload.length).toString("utf8");
if (text === "" || options?.onOutput === undefined) return;
const info: ShellOutputInfo = { stream: event.stream === "stderr" ? "stderr" : "stdout" };
if (event.skipped !== undefined) info.skipped = event.skipped as ShellOutputSkip;
try {
options.onOutput(text, context, info);
} catch (error) {
const cause = error instanceof Error ? error : new Error(String(error));
callbackError = new ExecutionError("callback_error", cause.message, cause);
controller.abort();
}
},
},
);
if (callbackError) return err(callbackError);
return ok({
exitCode: json.exitCode as number,
...(typeof json.spillPath === "string" ? { spillPath: json.spillPath } : {}),
});
} catch (error) {
if (callbackError) return err(callbackError);
if (!(error instanceof RemoteError)) {
return err(new ExecutionError("unknown", error instanceof Error ? error.message : String(error)));
}
const failure =
error.code === "timeout"
? new ExecutionError("timeout", `timeout:${timeout}`)
: error.code === "aborted"
? new ExecutionError("aborted", "aborted")
: error.code === "shell_unavailable" || error.code === "spawn_error"
? new ExecutionError(error.code, error.message)
: new ExecutionError("unknown", error.message);
if (typeof error.fields.spillPath === "string") failure.spillPath = error.fields.spillPath;
return err(failure);
} finally {
settled = true;
signal?.removeEventListener("abort", onAbort);
if (running !== undefined) this.#running.delete(running);
}
}
async cleanup(_context: Context): Promise<void> {
// Kill without aborting, so the commands settle with their killed status, as `NodeExecutionEnv` does.
for (const [id, session] of this.#running) this.connection.kill(id, session);
this.#running.clear();
}
}
export type { RemoteInfo };

539
third_party/pi/packages/env/src/ssh.ts vendored Normal file
View File

@@ -0,0 +1,539 @@
import { spawn } from "node:child_process";
import { createHash, randomBytes } from "node:crypto";
import { existsSync } from "node:fs";
import { mkdir, mkdtemp, readFile, rename, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
import { Connection, type ConnectionOptions } from "./connection.ts";
/** A remote system the package ships a daemon for. */
export type RemotePlatform = {
platform: "linux" | "android" | "darwin" | "windows";
arch: "x64" | "arm64";
/** The remote home directory, in the remote system's own spelling. */
home: string;
/** Windows: the shell `sshd` runs commands with (`DefaultShell`). */
shell?: "cmd" | "powershell";
/** Problems worth telling the owner about, such as Termux without `termux-exec`. */
warnings: string[];
};
/** How to reach a machine with the system `ssh`. */
export interface SshTarget {
/** Host name or `~/.ssh/config` alias. */
host: string;
user?: string;
port?: number;
identityFile?: string;
/** Host keys this application trusts; `acceptHostKey` adds to it. */
knownHostsFile: string;
/** The name keys are stored under, independent of aliases, ports and jump hosts, e.g. `pi-env-<env name>`. */
hostKeyAlias: string;
/** The `ssh` program; default `ssh`. */
ssh?: string;
/** An `ssh` configuration file instead of `~/.ssh/config` (`-F`). */
configFile?: string;
}
/** The remote host's key is not in `knownHostsFile`; `scanHostKey` shows it so the owner can accept it. */
export class HostKeyUnknownError extends Error {}
/** The remote host's key differs from the one in `knownHostsFile`; it is never accepted automatically. */
export class HostKeyChangedError extends Error {}
/** An `ssh` invocation that failed for another reason, with its exit code and diagnostics. */
export class SshError extends Error {
readonly exitCode: number | null;
readonly stderr: string;
constructor(message: string, exitCode: number | null, stderr: string) {
super(message);
this.exitCode = exitCode;
this.stderr = stderr;
}
}
const packageRoot = fileURLToPath(new URL("..", import.meta.url));
/** No leading `-` (it would be read as an `ssh` option) and no whitespace or control characters. */
function checkField(name: string, value: string): void {
if (value === "" || value.startsWith("-") || /[\s\x00-\x1f\x7f]/.test(value)) {
throw new Error(`Invalid ${name}: ${JSON.stringify(value)}`);
}
}
/** A path for an `ssh` option that expands `%` tokens: quoted, with `%` literal. */
function configPath(name: string, path: string): string {
if (path === "" || /["\x00-\x1f\x7f]/.test(path)) throw new Error(`Invalid ${name}: ${JSON.stringify(path)}`);
return `"${path.replaceAll("%", "%%")}"`;
}
/**
* Arguments for `ssh` up to the host: no prompts, no forwarding of any kind, no shared connections, no commands from
* the configuration, no locale forwarding (the remote uses its own), and host keys checked strictly against the
* application's own file under a fixed alias.
*/
export function sshArguments(
target: SshTarget,
strictHostKeys = true,
knownHostsFile = target.knownHostsFile,
): string[] {
checkField("host", target.host);
checkField("host key alias", target.hostKeyAlias);
if (target.user !== undefined) checkField("user", target.user);
if (target.port !== undefined && (!Number.isInteger(target.port) || target.port < 1 || target.port > 65535)) {
throw new Error(`Invalid port: ${target.port}`);
}
const option = (setting: string) => ["-o", setting];
return [
...(target.configFile === undefined ? [] : ["-F", target.configFile]),
"-T",
"-a",
"-x",
...option("BatchMode=yes"),
...option("ClearAllForwardings=yes"),
...option("ForwardAgent=no"),
...option("ForwardX11=no"),
...option("ControlMaster=no"),
...option("ControlPath=none"),
...option("RemoteCommand=none"),
...option("PermitLocalCommand=no"),
...option("SendEnv=-*"),
...option("ServerAliveInterval=15"),
...option(`StrictHostKeyChecking=${strictHostKeys ? "yes" : "accept-new"}`),
...option(`UserKnownHostsFile=${configPath("known hosts file", knownHostsFile)}`),
...option("GlobalKnownHostsFile=none"),
...option("HashKnownHosts=no"),
...option(`HostKeyAlias=${target.hostKeyAlias}`),
...(target.user === undefined ? [] : ["-l", target.user]),
...(target.port === undefined ? [] : ["-p", String(target.port)]),
...(target.identityFile === undefined
? []
: [
...option(`IdentityFile=${configPath("identity file", target.identityFile)}`),
...option("IdentitiesOnly=yes"),
]),
"--",
target.host,
];
}
/** How long a detection, check or upload over `ssh` may take before it is abandoned. */
const SSH_TIMEOUT_MS = 60_000;
const UPLOAD_TIMEOUT_MS = 300_000;
/** Run one remote command over `ssh`, optionally feeding stdin; resolves with stdout, rejects on failure. */
function runSsh(
target: SshTarget,
command: string,
options: { stdin?: Uint8Array; args?: string[]; timeoutMs?: number } = {},
): Promise<string> {
return new Promise((resolve, reject) => {
const args = options.args ?? sshArguments(target);
const child = spawn(target.ssh ?? "ssh", [...args, command], { stdio: ["pipe", "pipe", "pipe"] });
const timeoutMs = options.timeoutMs ?? SSH_TIMEOUT_MS;
const timer = setTimeout(() => {
child.kill();
const diagnostics = Buffer.concat(stderr).toString("utf8");
reject(
new SshError(
`ssh ${target.host} did not finish within ${timeoutMs / 1000} s: ${diagnostics.trim()}`,
null,
diagnostics,
),
);
}, timeoutMs);
const stdout: Buffer[] = [];
const stderr: Buffer[] = [];
child.stdout.on("data", (chunk: Buffer) => stdout.push(chunk));
child.stderr.on("data", (chunk: Buffer) => stderr.push(chunk));
child.stdin.on("error", () => {});
child.on("error", (error) => {
clearTimeout(timer);
reject(new SshError(error.message, null, ""));
});
child.on("close", (code) => {
clearTimeout(timer);
const output = Buffer.concat(stdout).toString("utf8");
const diagnostics = Buffer.concat(stderr).toString("utf8");
if (code === 0) return resolve(output);
if (/REMOTE HOST IDENTIFICATION HAS CHANGED/.test(diagnostics)) {
return reject(
new HostKeyChangedError(`The host key of ${target.host} changed; remove the old key to continue`),
);
}
if (/Host key verification failed|No .* host key is known/.test(diagnostics)) {
return reject(new HostKeyUnknownError(`The host key of ${target.host} is not trusted yet`));
}
reject(
new SshError(`ssh ${target.host} failed with exit code ${code}: ${diagnostics.trim()}`, code, diagnostics),
);
});
child.stdin.end(options.stdin ?? new Uint8Array(0));
});
}
/**
* Connect once with `accept-new` against a temporary known-hosts file to capture the key the host presents, through
* the same route (`~/.ssh/config`, jump hosts) as real connections. Returns its known-hosts lines and fingerprints.
* Showing a fingerprint is not authentication: compare it with one obtained out of band before accepting it.
*/
export async function scanHostKey(target: SshTarget): Promise<{ lines: string[]; fingerprints: string[] }> {
const directory = await mkdtemp(join(tmpdir(), "pi-env-hostkey-"));
try {
const scanned = join(directory, "known_hosts");
// Authentication may fail without the key; the key is recorded before authentication.
await runSsh(target, "exit 0", { args: sshArguments(target, false, scanned) }).catch((error: unknown) => {
if (!(error instanceof SshError) && !(error instanceof HostKeyUnknownError)) throw error;
});
const lines = existsSync(scanned)
? (await readFile(scanned, "utf8")).split("\n").filter((line) => line.trim() !== "")
: [];
if (lines.length === 0) throw new Error(`No host key received from ${target.host}`);
const fingerprints = await new Promise<string[]>((resolve, reject) => {
const child = spawn("ssh-keygen", ["-lf", scanned], { stdio: ["ignore", "pipe", "pipe"] });
let output = "";
child.stdout.on("data", (chunk: Buffer) => {
output += chunk.toString("utf8");
});
child.on("error", reject);
child.on("close", () => resolve(output.split("\n").filter((line) => line.trim() !== "")));
});
return { lines, fingerprints };
} finally {
await rm(directory, { recursive: true, force: true });
}
}
/** A known-hosts line for `alias`: `alias type key [comment]`, no markers, patterns or hashed names. */
function parseHostKeyLine(alias: string, line: string): { type: string; key: string } {
const [host, type, key] = line.trim().split(/\s+/);
if (
host !== alias ||
type === undefined ||
key === undefined ||
!/^(ssh-[a-z0-9-]+|ecdsa-sha2-[a-z0-9-]+|sk-[a-z0-9@.-]+)$/.test(type) ||
!/^[A-Za-z0-9+/]+={0,2}$/.test(key)
) {
throw new Error(`Not a host key line for ${alias}: ${JSON.stringify(line)}`);
}
return { type, key };
}
/** Known-hosts files being changed, so changes to one file happen one at a time. */
const knownHostsChanges = new Map<string, Promise<unknown>>();
/** Rewrite the known-hosts file one change at a time, atomically. */
async function changeKnownHosts(file: string, change: (lines: string[]) => string[]): Promise<void> {
const previous = knownHostsChanges.get(file) ?? Promise.resolve();
const next = previous
.catch(() => {})
.then(async () => {
await mkdir(dirname(file), { recursive: true, mode: 0o700 });
const existing = existsSync(file)
? (await readFile(file, "utf8")).split("\n").filter((line) => line !== "")
: [];
const lines = change(existing);
const temporary = `${file}.${randomBytes(6).toString("hex")}.tmp`;
await writeFile(temporary, lines.length === 0 ? "" : `${lines.join("\n")}\n`, { mode: 0o600, flag: "wx" });
await rename(temporary, file).catch(async (error: unknown) => {
await rm(temporary, { force: true });
throw error;
});
});
knownHostsChanges.set(file, next);
try {
await next;
} finally {
if (knownHostsChanges.get(file) === next) knownHostsChanges.delete(file);
}
}
/**
* Trust host-key lines from `scanHostKey` by adding them to the target's known-hosts file. Only plain lines for the
* target's alias are accepted. A key that differs from a trusted key of the same type is refused with
* `HostKeyChangedError`; `forgetHostKey` must remove the old keys first.
*/
export async function acceptHostKey(target: SshTarget, lines: readonly string[]): Promise<void> {
const alias = target.hostKeyAlias;
const accepted = lines.map((line) => ({ line: line.trim(), ...parseHostKeyLine(alias, line) }));
await changeKnownHosts(target.knownHostsFile, (existing) => {
const trusted = new Map<string, string>();
for (const line of existing) {
const [host, type, key] = line.trim().split(/\s+/);
if (host === alias && type !== undefined && key !== undefined) trusted.set(type, key);
}
const added: string[] = [];
for (const { line, type, key } of accepted) {
const known = trusted.get(type);
if (known === key) continue;
if (known !== undefined) {
throw new HostKeyChangedError(`The ${type} host key of ${target.host} changed; forget the old key first`);
}
trusted.set(type, key);
added.push(line);
}
return [...existing, ...added];
});
}
/** Stop trusting every key stored for the target's alias, e.g. after the owner confirmed a changed host key. */
export async function forgetHostKey(target: SshTarget): Promise<void> {
await changeKnownHosts(target.knownHostsFile, (existing) =>
existing.filter((line) => line.trim().split(/\s+/)[0] !== target.hostKeyAlias),
);
}
/** POSIX detection, run by the remote login shell; a Windows host answers through PowerShell instead. */
const POSIX_PROBE = `sh -c 'echo PI-ENV-PROBE; uname -s; uname -m; uname -o 2>/dev/null || echo -; printf "%s\\n" "$HOME" "\${TMPDIR:--}" "\${LD_PRELOAD:--}"'`;
function powershell(script: string): string {
return `powershell -NoProfile -NonInteractive -EncodedCommand ${Buffer.from(script, "utf16le").toString("base64")}`;
}
const WINDOWS_PROBE = powershell(
"'PI-ENV-PROBE'; 'Windows'; [System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture.ToString(); '-'; $HOME; '-'; '-'",
);
function normalizeArch(machine: string): RemotePlatform["arch"] {
const lower = machine.toLowerCase();
if (lower === "x86_64" || lower === "amd64" || lower === "x64") return "x64";
if (lower === "aarch64" || lower === "arm64") return "arm64";
throw new Error(`Unsupported remote architecture: ${machine}`);
}
type Probe = { system: string; machine: string; os: string; home: string; tmpdir: string; preload: string };
function parseProbe(target: SshTarget, output: string): Probe {
// Login shells may print a banner first.
const lines = output.split(/\r?\n/);
const start = lines.indexOf("PI-ENV-PROBE");
if (start === -1) throw new Error(`Unexpected answer from ${target.host}: ${output.trim()}`);
const [system = "", machine = "", os = "", home = "", tmpdir = "-", preload = "-"] = lines.slice(start + 1);
return { system, machine, os, home, tmpdir, preload };
}
/** Termux works only with its own `TMPDIR` and with `termux-exec`, which makes `#!/usr/bin/env` shebangs work. */
function termuxWarnings(probe: Probe): string[] {
const warnings: string[] = [];
if (probe.tmpdir === "-") warnings.push("TMPDIR is not set; Termux's sshd normally sets it to $PREFIX/tmp.");
if (!probe.preload.includes("termux-exec")) {
warnings.push("termux-exec is not loaded (LD_PRELOAD); scripts with #!/usr/bin/env shebangs will fail.");
}
warnings.push("Android may suspend Termux; run termux-wake-lock on the device to keep the connection alive.");
return warnings;
}
/** Which system the target runs: `uname` through the login shell, or PowerShell on Windows. */
export async function detectPlatform(target: SshTarget): Promise<RemotePlatform> {
let probe: Probe;
try {
probe = parseProbe(target, await runSsh(target, POSIX_PROBE));
} catch (error) {
// cmd.exe or PowerShell as the remote shell: no `sh`, or one that gets the probe's quotes wrong.
if (error instanceof HostKeyUnknownError || error instanceof HostKeyChangedError) throw error;
probe = parseProbe(target, await runSsh(target, WINDOWS_PROBE));
}
// Git Bash as Windows' default SSH shell: ask PowerShell for Windows' own architecture and home spelling.
if (/^(MINGW|MSYS|CYGWIN)/.test(probe.system)) probe = parseProbe(target, await runSsh(target, WINDOWS_PROBE));
const arch = normalizeArch(probe.machine);
const { system, os, home } = probe;
if (system === "Windows") {
// cmd.exe expands %OS%; PowerShell prints it as is.
const shell = (await runSsh(target, "echo %OS%")).includes("Windows_NT") ? "cmd" : "powershell";
return { platform: "windows", arch, home, shell, warnings: [] };
}
if (system === "Darwin") return { platform: "darwin", arch, home, warnings: [] };
if (system === "Linux") {
const android = os === "Android";
return { platform: android ? "android" : "linux", arch, home, warnings: android ? termuxWarnings(probe) : [] };
}
throw new Error(`Unsupported remote system: ${system}`);
}
/** The daemon binary this package ships for a remote system. */
export function packagedDaemon(remote: Pick<RemotePlatform, "platform" | "arch">): string {
const name = `pi-env-${remote.platform}-${remote.arch}`;
return join(packageRoot, "bin", name, remote.platform === "windows" ? "pi-env.exe" : "pi-env");
}
/** The daemon this package ships for the machine it runs on, e.g. for terminals of local sessions. */
export function localDaemon(): string {
const platform = process.platform === "win32" ? "windows" : process.platform;
if (platform !== "linux" && platform !== "android" && platform !== "darwin" && platform !== "windows") {
throw new Error(`No pi-env daemon for ${process.platform}`);
}
return packagedDaemon({ platform, arch: normalizeArch(process.arch) });
}
function quotePosix(text: string): string {
return `'${text.replaceAll("'", "'\\''")}'`;
}
function quotePowerShell(text: string): string {
return `'${text.replaceAll("'", "''")}'`;
}
/** Where a daemon with this content lives on the remote machine: named by its SHA-256, so versions never collide. */
function daemonPath(remote: RemotePlatform, sha256: string): string {
const name = `pi-env-${sha256.slice(0, 32)}`;
return remote.platform === "windows"
? `${remote.home}\\.pi\\mobile\\tools\\${name}.exe`
: `${remote.home}/.pi/mobile/tools/${name}`;
}
const POSIX_HASH = `hash() { if command -v sha256sum >/dev/null 2>&1; then sha256sum "$1" | cut -d' ' -f1; elif command -v shasum >/dev/null 2>&1; then shasum -a 256 "$1" | cut -d' ' -f1; elif command -v openssl >/dev/null 2>&1; then openssl dgst -sha256 "$1" | sed 's/.*= //'; else echo none; fi; }`;
/** Whether the remote file has this content: `present`, `missing`, or (POSIX without a hash tool) `nohash`. */
async function checkDaemon(target: SshTarget, remote: RemotePlatform, file: string, sha256: string): Promise<string> {
if (remote.platform === "windows") {
const check = powershell(
`$f = ${quotePowerShell(file)}; if ((Test-Path -LiteralPath $f) -and ((Get-FileHash -Algorithm SHA256 -LiteralPath $f).Hash.ToLower() -eq '${sha256}')) { 'present' } else { 'missing' }`,
);
return (await runSsh(target, check)).trim().endsWith("present") ? "present" : "missing";
}
const check = `${POSIX_HASH}; f=${quotePosix(file)}; if [ -f "$f" ] && [ "$(hash "$f")" = ${sha256} ]; then echo present; elif [ "$(hash /dev/null)" = none ]; then echo nohash; else echo missing; fi`;
return (await runSsh(target, `sh -c ${quotePosix(check)}`)).trim().split("\n").at(-1) ?? "missing";
}
/**
* Make sure the daemon is on the remote machine, verified by its SHA-256 before it ever runs, and remove daemons of
* other contents. The upload goes to a new temporary file and is renamed into place only once its hash matches.
* Returns the remote path of the binary.
*/
export async function deployDaemon(
target: SshTarget,
remote: RemotePlatform,
binary = packagedDaemon(remote),
): Promise<string> {
if (!existsSync(binary)) throw new Error(`No pi-env daemon for ${remote.platform}-${remote.arch} at ${binary}`);
const bytes = await readFile(binary);
const sha256 = createHash("sha256").update(bytes).digest("hex");
const file = daemonPath(remote, sha256);
const state = await checkDaemon(target, remote, file, sha256);
if (state === "present") return file;
if (state === "nohash") throw new Error(`${target.host} has no sha256sum, shasum or openssl to verify pi-env`);
if (remote.platform === "windows") {
const upload = powershell(
[
"$ErrorActionPreference = 'Stop'",
`$f = ${quotePowerShell(file)}`,
"$d = Split-Path -Parent $f",
"New-Item -ItemType Directory -Force -Path $d | Out-Null",
"$t = Join-Path $d ('.pi-env-' + [guid]::NewGuid().ToString() + '.tmp')",
// PowerShell reads redirected stdin itself, as text lines for `$input`, so the binary comes as base64 lines
// up to an end marker: Windows' sshd may never pass on the end of stdin.
"$text = New-Object System.Text.StringBuilder",
"foreach ($line in $input) { if ($line -eq 'PI-ENV-END') { break }; [void]$text.Append($line) }",
"$bytes = [Convert]::FromBase64String($text.ToString())",
"$out = [IO.File]::Open($t, 'CreateNew', 'Write', 'None'); $out.Write($bytes, 0, $bytes.Length); $out.Close()",
`if ((Get-FileHash -Algorithm SHA256 -LiteralPath $t).Hash.ToLower() -ne '${sha256}') { Remove-Item -LiteralPath $t; throw 'pi-env upload is corrupt' }`,
// A running daemon or a virus scanner can hold the old file for a moment.
"for ($i = 0; ; $i++) { try { Move-Item -Force -LiteralPath $t -Destination $f; break } catch { if ($i -ge 20) { throw }; Start-Sleep -Milliseconds 250 } }",
// Older daemons; one that is running stays until it exits.
"Get-ChildItem -LiteralPath $d -Filter 'pi-env-*.exe' | Where-Object { $_.FullName -ne $f } | ForEach-Object { Remove-Item -LiteralPath $_.FullName -ErrorAction SilentlyContinue }",
"'deployed'",
].join("; "),
);
const lines = bytes.toString("base64").replace(/.{1,76}/g, "$&\n");
await runSsh(target, upload, { stdin: Buffer.from(`${lines}PI-ENV-END\n`), timeoutMs: UPLOAD_TIMEOUT_MS });
return file;
}
const upload = [
"set -e",
POSIX_HASH,
`f=${quotePosix(file)}`,
'd=$(dirname "$f")',
'mkdir -p "$d"',
'chmod 700 "$d"',
't=$(mktemp "$d/.pi-env.XXXXXX")',
'cat > "$t"',
`if [ "$(hash "$t")" != ${sha256} ]; then rm -f "$t"; echo "pi-env upload is corrupt" >&2; exit 1; fi`,
'chmod 700 "$t"',
'mv -f "$t" "$f"',
// Older daemons; running ones keep their file open until they exit.
'for old in "$d"/pi-env-*; do [ "$old" = "$f" ] || rm -f "$old"; done',
"echo deployed",
].join("\n");
await runSsh(target, `sh -c ${quotePosix(upload)}`, { stdin: bytes, timeoutMs: UPLOAD_TIMEOUT_MS });
return file;
}
/** Options for `sshConnection` and `connectSsh`: the target, plus the binary to deploy (default: the one this package ships). */
export interface SshConnectOptions extends SshTarget {
binary?: string;
/**
* POSIX: start the daemon through the user's login shell (`$SHELL -l`), so commands see the environment of
* `~/.profile` and similar files. Off by default: `ssh` runs commands without a login shell.
*/
loginShell?: boolean;
onLog?: ConnectionOptions["onLog"];
}
/** The remote command that starts the daemon at `file`, before the `serve` arguments. */
function launchCommand(remote: RemotePlatform, file: string, loginShell: boolean): string {
if (remote.platform === "windows") {
// PowerShell runs a quoted path only with the call operator; cmd.exe keeps one pair of quotes around a program.
if (remote.shell === "powershell") return `& ${quotePowerShell(file)}`;
return file.includes(" ") ? `"${file}"` : file;
}
// `$0` is the daemon, `"$@"` its arguments; `$SHELL` is the login shell's own name for itself.
if (loginShell) return `exec "$SHELL" -lc 'exec "$0" "$@"' ${quotePosix(file)}`;
return quotePosix(file);
}
/** What an SSH connection learned about its remote system, and the daemon it verified last. */
type SshState = { remote?: RemotePlatform; verified?: string };
/** A `Connection` whose every start detects the remote system (once), verifies or deploys the daemon, then starts it. */
function sshConnectionFor(options: SshConnectOptions, state: SshState): Connection {
return new Connection({
command: async () => {
if (state.remote === undefined) {
const remote = await detectPlatform(options);
for (const warning of remote.warnings) options.onLog?.(`${warning}\n`);
state.remote = remote;
}
// A deployment that just verified the binary counts for the first start.
const file =
state.verified ??
(await deployDaemon(options, state.remote, options.binary ?? packagedDaemon(state.remote)));
state.verified = undefined;
return [
options.ssh ?? "ssh",
...sshArguments(options),
launchCommand(state.remote, file, options.loginShell === true),
];
},
...(options.onLog === undefined ? {} : { onLog: options.onLog }),
});
}
/**
* A `Connection` to the target that does nothing until its first request. Each start detects the remote system (the
* first time), verifies the daemon and deploys it if it is missing or changed, then starts it over `ssh`. A failure
* (no network, an untrusted or changed host key) fails the requests waiting for that start with code `spawn_error`
* and the `ssh` diagnostics as message; the next request tries again. `remote()` is the detected system, once known.
*/
export function sshConnection(options: SshConnectOptions): {
connection: Connection;
remote(): RemotePlatform | undefined;
} {
const state: SshState = {};
return { connection: sshConnectionFor(options, state), remote: () => state.remote };
}
/**
* Detect the remote system and deploy the daemon now, then return a `Connection` that starts it over `ssh`, like
* `sshConnection`. Failures reject here: host keys must already be trusted (`scanHostKey`, `acceptHostKey`), otherwise
* this rejects with `HostKeyUnknownError`.
*/
export async function connectSsh(
options: SshConnectOptions,
): Promise<{ connection: Connection; remote: RemotePlatform }> {
const remote = await detectPlatform(options);
const verified = await deployDaemon(options, remote, options.binary ?? packagedDaemon(remote));
for (const warning of remote.warnings) options.onLog?.(`${warning}\n`);
return { connection: sshConnectionFor(options, { remote, verified }), remote };
}

127
third_party/pi/packages/env/src/terminal.ts vendored Normal file
View File

@@ -0,0 +1,127 @@
import type { Connection, Json } from "./connection.ts";
/** Input larger than this goes to the daemon in several writes; frames are at most 16 MiB. */
const WRITE_CHUNK = 1024 * 1024;
export interface TerminalOptions {
/** Working directory of the program; it must exist. */
cwd: string;
/** Program and arguments. Default: the user's shell (`SHELL` in `env` or the daemon's environment) as a login shell. */
argv?: readonly string[];
/** Added to the daemon's environment, or the whole environment with `inheritEnv: false`. Set `TERM` here. */
env?: Record<string, string>;
inheritEnv?: boolean;
cols: number;
rows: number;
/** The terminal's raw output, in order. */
onData: (bytes: Uint8Array) => void;
}
/** The foreground job of a terminal: its process group id, and its leader's name if the system tells. */
export interface TerminalForeground {
pid: number;
name?: string;
}
/**
* A program running in a pseudo-terminal of a pi-env daemon. It ends when the program exits, `close()` kills it, or
* the connection is lost (the daemon then kills it).
*/
export class RemoteTerminal {
readonly pid: number;
/** The exit code; rejects with the daemon's error, with code `aborted` after `close()`. */
readonly exited: Promise<number>;
readonly #connection: Connection;
readonly #id: number;
readonly #session: number;
readonly #abort: AbortController;
constructor(
connection: Connection,
id: number,
session: number,
pid: number,
exited: Promise<number>,
abort: AbortController,
) {
this.#connection = connection;
this.#id = id;
this.#session = session;
this.pid = pid;
this.exited = exited;
this.#abort = abort;
}
/** Send input; once the terminal has ended, input is dropped. */
write(data: Uint8Array | string): void {
const bytes = typeof data === "string" ? new TextEncoder().encode(data) : data;
for (let offset = 0; offset < bytes.length; offset += WRITE_CHUNK) {
this.#request("ptyWrite", {}, bytes.subarray(offset, offset + WRITE_CHUNK));
}
}
/** Change the size; the program receives `SIGWINCH`. */
resize(cols: number, rows: number): void {
this.#request("ptyResize", { cols, rows });
}
/** Kill the program and everything in its session. */
close(): void {
this.#abort.abort();
}
/**
* The job in the terminal's foreground when it is not the program itself: what a shell runs, such as an editor.
* Undefined while the program is in front (a shell at its prompt). Rejects once the terminal has ended.
*/
async foreground(): Promise<TerminalForeground | undefined> {
const reply = await this.#connection.request("ptyForeground", { target: this.#id }, { session: this.#session });
const foreground = reply.json.foreground as { pid: number; name: string | null } | null;
if (!foreground) return undefined;
return { pid: foreground.pid, ...(foreground.name ? { name: foreground.name } : {}) };
}
#request(op: string, json: Json, payload?: Uint8Array): void {
this.#connection
.request(op, { ...json, target: this.#id }, { session: this.#session, ...(payload ? { payload } : {}) })
// A terminal that ended or a lost connection: `exited` reports it.
.catch(() => {});
}
}
/** Start a program in a new pseudo-terminal of the daemon behind `connection`. */
export function openTerminal(connection: Connection, options: TerminalOptions): Promise<RemoteTerminal> {
const abort = new AbortController();
const started = Promise.withResolvers<RemoteTerminal>();
let request: { id: number; session: number } | undefined;
const exited = connection
.request(
"pty",
{
cwd: options.cwd,
...(options.argv === undefined ? {} : { argv: options.argv }),
...(options.env === undefined ? {} : { env: options.env }),
...(options.inheritEnv === undefined ? {} : { inheritEnv: options.inheritEnv }),
cols: options.cols,
rows: options.rows,
},
{
signal: abort.signal,
onStart: (id, session) => {
request = { id, session };
},
onEvent: (json, payload) => {
if (json.kind === "output") options.onData(payload);
else if (json.kind === "started" && request) {
started.resolve(
new RemoteTerminal(connection, request.id, request.session, Number(json.pid), exited, abort),
);
}
},
},
)
.then((reply) => Number(reply.json.exitCode));
// Failing before it started rejects the open; afterwards, `exited` carries it.
exited.catch(started.reject);
return started.promise;
}

158
third_party/pi/packages/env/src/watch.ts vendored Normal file
View File

@@ -0,0 +1,158 @@
import type { Context } from "@earendil-works/chord";
import { FileError, type FileWatcher, type WatchChange, type WatchTarget } from "@earendil-works/pi-durable/env";
import { type Connection, isConnectionLost } from "./connection.ts";
import { toFileError } from "./errors.ts";
/** How the daemon watches, like `NodeExecutionEnv`'s `NodeWatchOptions`. */
export interface RemoteWatchOptions {
/** Force a mode; by default Windows and file systems that do not report remote changes poll. */
mode?: "native" | "polling";
/** Interval between snapshots in `polling` mode; default 2000 ms. */
pollIntervalMs?: number;
/** Most directories one watcher covers; default 10,000. */
maxDirectories?: number;
}
const RECONNECT_FIRST_MS = 1000;
const RECONNECT_MAX_MS = 30_000;
/**
* A watcher running in the daemon (Durable's `NodeFileWatcher` next to the files). When the connection is lost, it is
* opened again once the connection can be started again, and reports `overflow`: changes in between were not seen.
*/
export class RemoteWatcher implements FileWatcher {
readonly #connection: Connection;
readonly #targets: readonly WatchTarget[];
readonly #onChange: (change: WatchChange) => void;
readonly #options: RemoteWatchOptions;
#mode: "native" | "polling" = "native";
#closed = false;
#controller: AbortController | undefined;
#request: Promise<void> | undefined;
#wake: (() => void) | undefined;
private constructor(
connection: Connection,
targets: readonly WatchTarget[],
onChange: (change: WatchChange) => void,
options: RemoteWatchOptions,
) {
this.#connection = connection;
this.#targets = targets;
this.#onChange = onChange;
this.#options = options;
}
/** Watch `targets` (resolved paths); fails like `NodeFileWatcher.open` if coverage cannot be established. */
static async open(
connection: Connection,
targets: readonly WatchTarget[],
onChange: (change: WatchChange) => void,
options: RemoteWatchOptions = {},
): Promise<RemoteWatcher> {
const watcher = new RemoteWatcher(connection, targets, onChange, options);
await watcher.#start();
return watcher;
}
get mode(): "native" | "polling" {
return this.#mode;
}
async close(_context: Context): Promise<void> {
if (this.#closed) return;
this.#closed = true;
this.#wake?.();
this.#controller?.abort();
await this.#request;
}
#deliver(change: WatchChange): void {
if (this.#closed) return;
try {
this.#onChange(change);
} catch {
// A throwing callback must not stop watching.
}
}
/** Open the daemon's watcher; resolves once it reports coverage. */
#start(): Promise<void> {
return new Promise((resolve, reject) => {
let ready = false;
const controller = new AbortController();
this.#controller = controller;
const request = this.#connection.request(
"watch",
{
targets: this.#targets.map((target) => ({
path: target.path,
recursive: target.recursive === true,
exclude: { hidden: target.exclude?.hidden === true, names: [...(target.exclude?.names ?? [])] },
})),
...this.#options,
},
{
signal: controller.signal,
onEvent: (event) => {
if (event.kind === "ready") {
ready = true;
this.#mode = event.mode === "polling" ? "polling" : "native";
resolve();
} else if (event.kind === "change") {
if (event.mode === "polling") this.#mode = "polling";
if (event.overflow === true) this.#deliver({ overflow: true });
else this.#deliver({ paths: event.paths as string[] });
} else if (event.kind === "error") {
const code = event.code === "permission_denied" ? "permission_denied" : "invalid";
this.#deliver({ error: new FileError(code, String(event.message)) });
this.#closed = true;
}
},
},
);
this.#request = request.then(
() => {},
(error: unknown) => {
if (!ready) {
reject(error);
return;
}
if (this.#closed) return;
if (isConnectionLost(error)) {
void this.#reconnect();
return;
}
this.#deliver({ error: toFileError(error) });
this.#closed = true;
},
);
});
}
async #reconnect(): Promise<void> {
for (let delay = RECONNECT_FIRST_MS; !this.#closed; delay = Math.min(delay * 2, RECONNECT_MAX_MS)) {
await new Promise<void>((resolve) => {
const timer = setTimeout(resolve, delay);
timer.unref();
this.#wake = () => {
clearTimeout(timer);
resolve();
};
});
this.#wake = undefined;
if (this.#closed) return;
try {
await this.#start();
// Changes while disconnected were not seen.
this.#deliver({ overflow: true });
return;
} catch (error) {
if (isConnectionLost(error)) continue;
this.#deliver({ error: toFileError(error) });
this.#closed = true;
return;
}
}
}
}

45
third_party/pi/packages/env/test/conformance.test.ts vendored Normal file
View File

@@ -0,0 +1,45 @@
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { registerEnvConformance } from "@earendil-works/pi-durable/testing";
import { afterAll, describe, expect, it } from "vitest";
import { Connection } from "../src/connection.ts";
import { LocalExecutionEnv } from "../src/list-files.ts";
import { RemoteExecutionEnv } from "../src/remote-env.ts";
import { daemon } from "./daemon.ts";
const connection = new Connection({ command: [daemon] });
afterAll(() => connection.close());
const windows = process.platform === "win32";
const gitBash = join(process.env.ProgramFiles ?? "C:\\Program Files", "Git", "bin", "bash.exe");
registerEnvConformance(
{ describe, expect, it },
"RemoteExecutionEnv over a pipe",
async (use) => {
const cwd = mkdtempSync(join(tmpdir(), "pi-env-conformance-"));
try {
await use(new RemoteExecutionEnv({ connection, id: "pi-env:test", cwd, watch: { pollIntervalMs: 100 } }));
} finally {
// On Windows a killed command's processes can hold the directory for a moment.
rmSync(cwd, { recursive: true, force: true, maxRetries: 50, retryDelay: 100 });
}
},
// Git Bash's `ln -s` copies instead of linking unless native symlinks are enabled.
windows ? { shell: [gitBash, "-c"], symlinks: false, listFiles: true } : { listFiles: true },
);
registerEnvConformance(
{ describe, expect, it },
"LocalExecutionEnv",
async (use) => {
const cwd = mkdtempSync(join(tmpdir(), "pi-env-local-conformance-"));
try {
await use(new LocalExecutionEnv({ connection, cwd, watch: { pollIntervalMs: 100 } }));
} finally {
rmSync(cwd, { recursive: true, force: true, maxRetries: 50, retryDelay: 100 });
}
},
windows ? { shell: [gitBash, "-c"], symlinks: false, listFiles: true } : { listFiles: true },
);

13
third_party/pi/packages/env/test/daemon.ts vendored Normal file
View File

@@ -0,0 +1,13 @@
import { existsSync } from "node:fs";
import { resolve } from "node:path";
/**
* The daemon the tests run: `PI_ENV_DAEMON` (CI points it at release builds), else the one `cargo build` built in
* ../daemon.
*/
export const daemon =
process.env.PI_ENV_DAEMON ??
resolve(import.meta.dirname, `../daemon/target/debug/pi-env${process.platform === "win32" ? ".exe" : ""}`);
if (!existsSync(daemon)) {
throw new Error(`Build the daemon first: npm run build:daemon in packages/env (missing ${daemon})`);
}

361
third_party/pi/packages/env/test/differential.test.ts vendored Normal file
View File

@@ -0,0 +1,361 @@
import { spawn } from "node:child_process";
import { mkdtempSync, realpathSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { Context } from "@earendil-works/chord";
import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context";
import type { ToolExecutionApi } from "@earendil-works/pi-durable";
import type { ExecutionEnv, FileError, Result } from "@earendil-works/pi-durable/env";
import { NodeExecutionEnv } from "@earendil-works/pi-durable/env/node";
import { createBashTool, createReadTool } from "@earendil-works/pi-durable/tools";
import { afterAll, describe, expect, it } from "vitest";
import { Connection } from "../src/connection.ts";
import { RemoteExecutionEnv } from "../src/remote-env.ts";
import { daemon } from "./daemon.ts";
const connection = new Connection({ command: [daemon] });
const context: Context = BACKGROUND_CONTEXT;
const dirs: string[] = [];
afterAll(() => {
connection.close();
for (const dir of dirs) rmSync(dir, { recursive: true, force: true, maxRetries: 50, retryDelay: 100 });
});
function pair(): { local: ExecutionEnv; remote: ExecutionEnv; roots: [string, string] } {
const localRoot = mkdtempSync(join(tmpdir(), "pi-env-local-"));
const remoteRoot = mkdtempSync(join(tmpdir(), "pi-env-remote-"));
dirs.push(localRoot, remoteRoot);
return {
local: new NodeExecutionEnv({ cwd: localRoot }),
remote: new RemoteExecutionEnv({ connection, id: "pi-env:test", cwd: remoteRoot }),
roots: [localRoot, remoteRoot],
};
}
function random(seed: number): () => number {
let state = seed >>> 0;
return () => {
state = (state + 0x6d2b79f5) >>> 0;
let t = state;
t = Math.imul(t ^ (t >>> 15), t | 1);
t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};
}
/** A result with the environment's root replaced and only the error code of failures, for comparison. */
function normalize(value: unknown, root: string): unknown {
const json = JSON.stringify(value, (key, field: unknown) => {
if (field instanceof Uint8Array) return { bytes: Buffer.from(field).toString("hex") };
if (key === "mtimeMs") return "mtime";
if (field && typeof field === "object" && "ok" in field && (field as Result<unknown, FileError>).ok === false) {
return { ok: false, code: (field as unknown as { error: FileError }).error.code };
}
return field;
});
if (json === undefined) return undefined;
// Roots appear JSON-escaped (backslashes doubled on Windows), and resolved paths spell them without Windows' 8.3
// short names (RUNNER~1).
const escaped = (path: string) => JSON.stringify(path).slice(1, -1);
return JSON.parse(json.replaceAll(escaped(realpathSync.native(root)), "<root>").replaceAll(escaped(root), "<root>"));
}
const NAMES = ["a.txt", "b", "dir", "dir/c.txt", "dir/sub", "dir/sub/d.bin", "missing", "a.txt/x"];
const CONTENTS: (string | Uint8Array)[] = [
"",
"hello\n",
"\ufeffbom\r\nline",
Uint8Array.from([0xe2, 0x82, 0x0a, 0xff, 0xef, 0xbb, 0xbf]),
"x".repeat(70_000),
// Crosses write chunks, read chunks and multi-byte characters split between them.
"é".repeat(300_001),
];
type Operation = (env: ExecutionEnv) => Promise<unknown>;
/** A random operation with its arguments fixed, and a label for failure messages. */
function randomOperation(next: () => number): { label: string; run: Operation } {
const args: unknown[] = [];
const pick = <T>(value: T): T => {
args.push(
value instanceof Uint8Array
? "<bytes>"
: typeof value === "string" && value.length > 20
? `<${value.length} chars>`
: value,
);
return value;
};
const nameValue = () => pick(NAMES[Math.floor(next() * NAMES.length)]!);
const contentValue = () => pick(CONTENTS[Math.floor(next() * CONTENTS.length)]!);
const flagValue = () => pick(next() < 0.5);
const small = (limit: number) => pick(Math.floor(next() * limit));
const choices: [string, () => Operation][] = [
[
"writeFile",
() => {
const n = nameValue();
const c = contentValue();
return (env) => env.writeFile(n, c, context);
},
],
[
"appendFile",
() => {
const n = nameValue();
const c = contentValue();
return (env) => env.appendFile(n, c, context);
},
],
[
"readTextFile",
() => {
const n = nameValue();
return (env) => env.readTextFile(n, context);
},
],
[
"readBinaryFile",
() => {
const n = nameValue();
return (env) => env.readBinaryFile(n, context);
},
],
[
"readTextLines",
() => {
const n = nameValue();
const m = small(4);
return (env) => env.readTextLines(n, { maxLines: m }, context);
},
],
[
"fileInfo",
() => {
const n = nameValue();
return (env) => env.fileInfo(n, context);
},
],
[
"exists",
() => {
const n = nameValue();
return (env) => env.exists(n, context);
},
],
[
"listDir",
() => {
const n = nameValue();
return (env) => env.listDir(n, context);
},
],
[
"createDir",
() => {
const n = nameValue();
const r = flagValue();
return (env) => env.createDir(n, { recursive: r }, context);
},
],
[
"remove",
() => {
const n = nameValue();
const r = flagValue();
const f = flagValue();
return (env) => env.remove(n, { recursive: r, force: f }, context);
},
],
[
"renameFile",
() => {
const a = nameValue();
const b = nameValue();
return (env) => env.renameFile(a, b, context);
},
],
[
"truncateFile",
() => {
const n = nameValue();
const size = small(10);
return (env) => env.truncateFile(n, size, context);
},
],
[
"flushFile",
() => {
const n = nameValue();
return (env) => env.flushFile(n, context);
},
],
[
"canonicalPath",
() => {
const n = nameValue();
return (env) => env.canonicalPath(n, context);
},
],
[
"openBinaryReader",
() => {
const n = nameValue();
const noFollow = flagValue();
const offset = small(8);
const length = small(8);
return async (env) => {
const reader = await env.openBinaryReader(n, { noFollow }, context);
if (!reader.ok) return reader;
try {
return [
await reader.value.read(offset, length, context),
await reader.value.scanLines({ startLine: 0 }, context),
];
} finally {
await reader.value.close(context);
}
};
},
],
];
const [label, make] = choices[Math.floor(next() * choices.length)]!;
const run = make();
return { label: `${label}(${args.map((arg) => JSON.stringify(arg)).join(", ")})`, run };
}
describe("RemoteExecutionEnv against NodeExecutionEnv", () => {
it("gives the same results for random file operation sequences", async () => {
for (let seed = 1; seed <= 60; seed++) {
const next = random(seed);
const { local, remote, roots } = pair();
for (let step = 0; step < 25; step++) {
const operation = randomOperation(next);
const expected = normalize(await operation.run(local), roots[0]);
const actual = normalize(await operation.run(remote), roots[1]);
expect(actual, `seed ${seed} step ${step} ${operation.label}`).toEqual(expected);
}
}
}, 120_000);
it.skipIf(process.platform === "win32")(
"reads devices, FIFOs and directories like Node",
async () => {
const { local, remote, roots } = pair();
const results = async (env: ExecutionEnv, root: string) => {
const fifo = join(root, "fifo");
const made = await env.exec(["mkfifo", fifo], undefined, context);
if (!made.ok || made.value.exitCode !== 0) throw new Error("mkfifo failed");
await env.createDir("dir", undefined, context);
// A writer for each blocking open of the FIFO.
const write = (text: string) => spawn("sh", ["-c", `printf '%s' "$1" > "$2"`, "sh", text, fifo]);
write("fifo text\n");
const fifoText = await env.readTextFile("fifo", context);
write("fifo bytes");
const fifoBytes = await env.readBinaryFile("fifo", context);
write("x");
const fifoLines = await env.readTextLines("fifo", undefined, context);
const dirReader = await env.openTextLineReader("dir", context);
const dirLine = dirReader.ok ? await dirReader.value.readLine(context) : undefined;
if (dirReader.ok) await dirReader.value.close(context);
return normalize(
{
nullText: await env.readTextFile("/dev/null", context),
nullBytes: await env.readBinaryFile("/dev/null", context),
nullLines: await env.readTextLines("/dev/null", undefined, context),
fifoText,
fifoBytes,
fifoLines,
dirReaderOpened: dirReader.ok,
dirLine,
dirText: await env.readTextFile("dir", context),
dirBytes: await env.readBinaryFile("dir", context),
},
root,
);
};
expect(await results(remote, roots[1])).toEqual(await results(local, roots[0]));
},
30_000,
);
// macOS file systems refuse names that are not UTF-8.
it.skipIf(process.platform !== "linux" && process.platform !== "android")(
"lists names that are not UTF-8 like Node",
async () => {
const { local, remote, roots } = pair();
for (const root of roots) {
for (const name of [[0x62], [0x61, 0xff], [0xc3, 0xa9], [0x61, 0xfe, 0x7a], [0x7a]]) {
writeFileSync(Buffer.concat([Buffer.from(`${root}/`), Buffer.from(name)]), "x");
}
}
const results = async (env: ExecutionEnv, root: string) => {
const reader = await env.openDirReader(".", context);
const pages: unknown[] = [];
if (reader.ok) {
for (let done = false; !done; ) {
const page = await reader.value.next(2, context);
pages.push(page);
done = !page.ok || page.value.done;
}
await reader.value.close(context);
}
return normalize({ list: await env.listDir(".", context), pages }, root);
};
expect(await results(remote, roots[1])).toEqual(await results(local, roots[0]));
},
);
it("gives the same read and bash tool results", async () => {
const { local, remote, roots } = pair();
const files: [string, string | Uint8Array][] = [
["text.txt", "one\ntwo\n\nthree"],
["big.txt", "line\n".repeat(3000)],
["long.txt", `${"é".repeat(40_000)}\nend`],
["bom.txt", "\ufeffhello"],
["bad.txt", Uint8Array.from([0x61, 0xe2, 0x82, 0x0a, 0x62])],
];
for (const [path, content] of files) {
await local.writeFile(path, content, context);
await remote.writeFile(path, content, context);
}
const api = (env: ExecutionEnv, output: string[]) =>
({
env,
outputWindow: undefined,
output: (chunk: string | Uint8Array) => output.push(String(chunk)),
diagnostic: () => {},
details: async () => {},
}) as unknown as ToolExecutionApi;
const run = async (
env: ExecutionEnv,
tool: ReturnType<typeof createReadTool> | ReturnType<typeof createBashTool>,
args: object,
) => {
const output: string[] = [];
try {
const result = await tool.execute(args as never, api(env, output), context);
return { result, output: output.join("") };
} catch (error) {
return { error: (error as Error).message, output: output.join("") };
}
};
const cases: [ReturnType<typeof createReadTool> | ReturnType<typeof createBashTool>, object][] = [
[createReadTool(), { path: "text.txt" }],
[createReadTool(), { path: "text.txt", offset: 2, limit: 1 }],
[createReadTool(), { path: "big.txt" }],
[createReadTool(), { path: "long.txt" }],
[createReadTool(), { path: "bom.txt" }],
[createReadTool(), { path: "bad.txt" }],
[createReadTool(), { path: "missing.txt" }],
[createBashTool(), { command: "cat text.txt; echo err >&2; exit 0" }],
[createBashTool(), { command: "exit 3" }],
[createBashTool(), { command: "printf 'a\\nb'; ls" }],
];
for (const [tool, args] of cases) {
const expected = normalize(await run(local, tool, args), roots[0]);
const actual = normalize(await run(remote, tool, args), roots[1]);
expect(actual, JSON.stringify(args)).toEqual(expected);
}
}, 60_000);
});

48
third_party/pi/packages/env/test/old-daemon.ts vendored Normal file
View File

@@ -0,0 +1,48 @@
/**
* A stand-in for a daemon released before `hello.features`: it answers `hello` without `features` and refuses every
* other operation as an unknown one, as such a daemon refuses `listFiles`. Started as `node old-daemon.ts serve --token
* <hex>`.
*/
const token = process.argv[process.argv.indexOf("--token") + 1];
process.stdout.write(`PI-ENV ${token}\n`);
function send(type: number, id: number, json: Record<string, unknown>): void {
const body = Buffer.from(JSON.stringify(json), "utf8");
const header = Buffer.alloc(13);
header.writeUInt32BE(9 + body.length, 0);
header.writeUInt8(type, 4);
header.writeUInt32BE(id, 5);
header.writeUInt32BE(body.length, 9);
process.stdout.write(Buffer.concat([header, body]));
}
let buffered = Buffer.alloc(0);
process.stdin.on("data", (chunk: Buffer) => {
buffered = Buffer.concat([buffered, chunk]);
while (buffered.length >= 4 && buffered.length >= 4 + buffered.readUInt32BE(0)) {
const length = buffered.readUInt32BE(0);
const type = buffered.readUInt8(4);
const id = buffered.readUInt32BE(5);
const jsonLength = buffered.readUInt32BE(9);
const json = JSON.parse(buffered.subarray(13, 13 + jsonLength).toString("utf8")) as { op?: string };
buffered = buffered.subarray(4 + length);
if (type !== 1) continue;
if (json.op === "hello") {
send(2, id, {
protocol: 1,
version: "1.1.0",
os: "linux",
arch: "x86_64",
home: "/",
separator: "/",
tmpdir: "/tmp",
cwd: "/",
driveCwds: {},
pid: process.pid,
});
} else {
send(3, id, { code: "EINVAL", message: `unknown operation ${json.op}` });
}
}
});
process.stdin.on("end", () => process.exit(0));

176
third_party/pi/packages/env/test/remote.test.ts vendored Normal file
View File

@@ -0,0 +1,176 @@
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { BACKGROUND_CONTEXT, withAbortSignal } from "@earendil-works/chord/context";
import { getOrThrow, type ShellOutputInfo } from "@earendil-works/pi-durable/env";
import { afterAll, describe, expect, it } from "vitest";
import { Connection, nodePlatform } from "../src/connection.ts";
import { RemoteExecutionEnv } from "../src/remote-env.ts";
import { daemon } from "./daemon.ts";
const context = BACKGROUND_CONTEXT;
const connections: Connection[] = [];
const dirs: string[] = [];
afterAll(() => {
for (const connection of connections) connection.close();
for (const dir of dirs) rmSync(dir, { recursive: true, force: true, maxRetries: 50, retryDelay: 100 });
});
function environment(): { env: RemoteExecutionEnv; connection: Connection } {
const cwd = mkdtempSync(join(tmpdir(), "pi-env-remote-"));
dirs.push(cwd);
const connection = new Connection({ command: [daemon] });
connections.push(connection);
return { env: new RemoteExecutionEnv({ connection, id: "pi-env:test", cwd }), connection };
}
describe("RemoteExecutionEnv", () => {
it("transfers only about a window of output a caller keeps, with exact counts", async () => {
const { env } = environment();
const total = 5_000_000;
let delivered = 0;
let skipped = 0;
let tail = "";
const result = await env.exec(
["sh", "-c", `yes 0123456789abcdef | head -c ${total}`],
{
window: { maxBytes: 1000, maxLines: 20, minIntervalMs: 20, bytesPerSecond: 1_000_000 },
onOutput: (text: string, _context, info: ShellOutputInfo) => {
delivered += Buffer.byteLength(text);
if (info.skipped) {
skipped += info.skipped.bytes;
tail = "";
}
tail += text;
},
},
context,
);
expect(getOrThrow(result).exitCode).toBe(0);
expect(delivered + skipped).toBe(total);
expect(skipped).toBeGreaterThan(0);
expect(delivered).toBeLessThan(total / 10);
// After the last skip comes everything to the end, more than the window.
const full = "0123456789abcdef\n".repeat(Math.ceil(total / 17)).slice(0, total);
expect(full.endsWith(tail)).toBe(true);
expect(tail.length > 1000 || tail.split("\n").length - 1 > 20).toBe(true);
}, 60_000);
it("aborts commands cancelled right after they are sent", async () => {
const { env } = environment();
const started = Date.now();
for (let attempt = 0; attempt < 20; attempt++) {
const controller = new AbortController();
const running = env.exec(["sleep", "5"], undefined, withAbortSignal(controller.signal, context));
if (attempt % 2 === 0) controller.abort();
else setTimeout(() => controller.abort(), attempt % 5);
const result = await running;
expect(result.ok ? "ok" : result.error.code).toBe("aborted");
}
expect(Date.now() - started).toBeLessThan(10_000);
}, 30_000);
it("reports the remote system in Node's names", async () => {
const { connection } = environment();
const { platform, arch } = nodePlatform(await connection.info());
expect(platform).toBe(process.platform);
// CI runs the darwin-x64 release daemon on arm64 macOS through Rosetta, so it reports x64 to arm64 Node.
const rosetta = process.platform === "darwin" && process.arch === "arm64" && arch === "x64";
if (!rosetta) expect(arch).toBe(process.arch);
expect(nodePlatform({ os: "windows", arch: "aarch64" })).toEqual({ platform: "win32", arch: "arm64" });
expect(nodePlatform({ os: "macos", arch: "x86_64" })).toEqual({ platform: "darwin", arch: "x64" });
});
it("reports listFiles as not supported by daemons released before it", async () => {
const { env } = environment();
expect((await env.connection.info()).features).toContain("listFiles");
const connection = new Connection({ command: [process.execPath, join(import.meta.dirname, "old-daemon.ts")] });
connections.push(connection);
const old = new RemoteExecutionEnv({ connection, id: "pi-env:old", cwd: env.cwd });
const listed = await old.listFiles(".", undefined, context);
expect(listed.ok ? "ok" : listed.error.code).toBe("not_supported");
});
it("cancels a listing in the daemon", async () => {
const { env, connection } = environment();
for (let dir = 0; dir < 100; dir++) {
mkdirSync(join(env.cwd, `d${dir}`));
for (let file = 0; file < 100; file++) writeFileSync(join(env.cwd, `d${dir}`, `f${file}`), "");
}
// The cancel follows the request at once, bypassing the client's own abort check, so the daemon stops the walk.
const controller = new AbortController();
controller.abort();
const cancelled = connection.request("listFiles", { path: env.cwd }, { signal: controller.signal });
await expect(cancelled).rejects.toMatchObject({ code: "aborted" });
const listed = getOrThrow(await env.listFiles(".", undefined, context));
expect([listed.entries.length, listed.truncated]).toEqual([10_100, false]);
const bounded = getOrThrow(await env.listFiles(".", { maxEntries: 5000 }, context));
expect([bounded.entries.length, bounded.truncated]).toEqual([5000, true]);
// Creating 10,000 files takes several seconds on Windows CI.
}, 30_000);
it("times out a command that produces no output", async () => {
const { env } = environment();
const started = Date.now();
const result = await env.exec(["sleep", "5"], { timeout: 0.3 }, context);
expect(result.ok ? "ok" : result.error.code).toBe("timeout");
expect(Date.now() - started).toBeLessThan(3000);
});
it("fails requests on handles of a lost connection instead of reaching the new daemon", async () => {
const { env, connection } = environment();
getOrThrow(await env.writeFile("a.txt", "AAA", context));
getOrThrow(await env.writeFile("b.txt", "BBB", context));
const a = getOrThrow(await env.openBinaryReader("a.txt", undefined, context));
const { pid } = await connection.info();
process.kill(pid, "SIGKILL");
await new Promise((done) => setTimeout(done, 300));
// The new daemon numbers its handles from 1 again.
const b = getOrThrow(await env.openBinaryReader("b.txt", undefined, context));
expect((await connection.info()).pid).not.toBe(pid);
const stale = await a.read(0, 3, context);
expect(stale.ok ? new TextDecoder().decode(stale.value) : stale.error.code).toBe("unknown");
expect(new TextDecoder().decode(getOrThrow(await b.read(0, 3, context)))).toBe("BBB");
await a.close(context);
await b.close(context);
});
it("computes the command at each start and retries a start that failed", async () => {
const cwd = mkdtempSync(join(tmpdir(), "pi-env-remote-"));
dirs.push(cwd);
let starts = 0;
const connection = new Connection({
command: async () => {
starts++;
if (starts === 1) throw new Error("ssh gpu-box failed: Network is unreachable");
return [daemon];
},
});
connections.push(connection);
const env = new RemoteExecutionEnv({ connection, id: "pi-env:test", cwd });
const offline = await env.readTextFile("missing.txt", context);
expect(offline.ok ? "ok" : [offline.error.code, offline.error.message]).toEqual([
"unknown",
"ssh gpu-box failed: Network is unreachable",
]);
const command = await env.exec(["sh", "-c", "exit 0"], undefined, context);
expect(command.ok ? command.value.exitCode : command.error.code).toBe(0);
expect(starts).toBe(2);
});
it("writes and reads files larger than one transfer chunk in order", async () => {
const { env } = environment();
const content = new Uint8Array(3_500_017);
for (let index = 0; index < content.length; index++) content[index] = (index * 7919) % 251;
getOrThrow(await env.writeFile("big.bin", content, context));
getOrThrow(await env.appendFile("big.bin", content, context));
const read = getOrThrow(await env.readBinaryFile("big.bin", context));
expect(read.length).toBe(content.length * 2);
expect(Buffer.compare(Buffer.from(read.subarray(0, content.length)), Buffer.from(content))).toBe(0);
expect(Buffer.compare(Buffer.from(read.subarray(content.length)), Buffer.from(content))).toBe(0);
const reader = getOrThrow(await env.openBinaryReader("big.bin", undefined, context));
const range = getOrThrow(await reader.read(1_000_003, 2_000_000, context));
await reader.close(context);
expect(Buffer.compare(Buffer.from(range), Buffer.from(read.subarray(1_000_003, 3_000_003)))).toBe(0);
}, 60_000);
});

92
third_party/pi/packages/env/test/ssh-external.test.ts vendored Normal file
View File

@@ -0,0 +1,92 @@
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context";
import { getOrThrow } from "@earendil-works/pi-durable/env";
import { registerEnvConformance } from "@earendil-works/pi-durable/testing";
import { afterAll, describe, expect, it } from "vitest";
import { type Connection, nodePlatform } from "../src/connection.ts";
import { RemoteExecutionEnv } from "../src/remote-env.ts";
import {
acceptHostKey,
connectSsh,
deployDaemon,
detectPlatform,
type RemotePlatform,
type SshTarget,
scanHostKey,
} from "../src/ssh.ts";
import { daemon } from "./daemon.ts";
/**
* The conformance suite over a real SSH server on this machine, configured by CI: `PI_ENV_SSH_HOST`, `PI_ENV_SSH_USER`
* `PI_ENV_SSH_KEY` and `PI_ENV_SSH_PROGRAM`. On Windows this covers deployment and the daemon's launch through the server's default shell.
*/
const host = process.env.PI_ENV_SSH_HOST;
const root = mkdtempSync(join(tmpdir(), "pi-env-ssh-external-"));
let connected: Promise<{ connection: Connection; remote: RemotePlatform }> | undefined;
async function connect(): Promise<{ connection: Connection; remote: RemotePlatform }> {
connected ??= (async () => {
const target: SshTarget = {
host: host!,
...(process.env.PI_ENV_SSH_USER === undefined ? {} : { user: process.env.PI_ENV_SSH_USER }),
...(process.env.PI_ENV_SSH_PORT === undefined ? {} : { port: Number(process.env.PI_ENV_SSH_PORT) }),
...(process.env.PI_ENV_SSH_KEY === undefined ? {} : { identityFile: process.env.PI_ENV_SSH_KEY }),
...(process.env.PI_ENV_SSH_PROGRAM === undefined ? {} : { ssh: process.env.PI_ENV_SSH_PROGRAM }),
knownHostsFile: join(root, "known_hosts"),
hostKeyAlias: "pi-env-external",
};
// Each stage is logged, so a hang in CI shows where it is.
const log = (text: string) => process.stderr.write(`[ssh-external ${new Date().toISOString()}] ${text}\n`);
log("scanning the host key");
await acceptHostKey(target, (await scanHostKey(target)).lines);
log("detecting the platform");
const remote = await detectPlatform(target);
log(`detected ${JSON.stringify(remote)}; deploying`);
log(`deployed ${await deployDaemon(target, remote, daemon)}; connecting`);
const result = await connectSsh({ ...target, binary: daemon, onLog: (text) => log(`daemon: ${text.trim()}`) });
log("connected; saying hello");
log(`hello ${JSON.stringify(await result.connection.info())}`);
return result;
})();
return connected;
}
afterAll(async () => {
if (connected) (await connected).connection.close();
rmSync(root, { recursive: true, force: true, maxRetries: 50, retryDelay: 100 });
});
const windows = process.platform === "win32";
const gitBash = join(process.env.ProgramFiles ?? "C:\\Program Files", "Git", "bin", "bash.exe");
describe.skipIf(host === undefined)("SSH to this machine's server", () => {
it("reaches the remote system through its default shell", async () => {
const { connection, remote } = await connect();
const info = await connection.info();
expect(nodePlatform(info).platform).toBe(process.platform);
if (windows) expect(remote.shell).toBe(process.env.PI_ENV_SSH_SHELL ?? "cmd");
}, 120_000);
});
if (host !== undefined) {
registerEnvConformance(
{ describe, expect, it },
"RemoteExecutionEnv over SSH",
async (use) => {
const { connection } = await connect();
const info = await connection.info();
const home = new RemoteExecutionEnv({ connection, id: "pi-env:external", cwd: info.home });
const cwd = getOrThrow(await home.createTempDir("pi-env-conformance-", BACKGROUND_CONTEXT));
try {
await use(
new RemoteExecutionEnv({ connection, id: "pi-env:external", cwd, watch: { pollIntervalMs: 100 } }),
);
} finally {
await home.remove(cwd, { recursive: true, force: true }, BACKGROUND_CONTEXT);
}
},
windows ? { shell: [gitBash, "-c"], symlinks: false, listFiles: true } : { listFiles: true },
);
}

299
third_party/pi/packages/env/test/ssh.test.ts vendored Normal file
View File

@@ -0,0 +1,299 @@
import { type ChildProcess, execFileSync, spawn } from "node:child_process";
import { createHash } from "node:crypto";
import {
existsSync,
mkdirSync,
mkdtempSync,
readdirSync,
readFileSync,
rmSync,
statSync,
writeFileSync,
} from "node:fs";
import { createServer, Socket } from "node:net";
import { tmpdir, userInfo } from "node:os";
import { join } from "node:path";
import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context";
import { getOrThrow } from "@earendil-works/pi-durable/env";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { Connection } from "../src/connection.ts";
import { LocalExecutionEnv } from "../src/list-files.ts";
import { RemoteExecutionEnv } from "../src/remote-env.ts";
import {
acceptHostKey,
connectSsh,
forgetHostKey,
HostKeyChangedError,
HostKeyUnknownError,
type SshTarget,
scanHostKey,
sshConnection,
} from "../src/ssh.ts";
import { daemon } from "./daemon.ts";
/** The deployed daemon is named by its content. */
const deployedName = `pi-env-${createHash("sha256").update(readFileSync(daemon)).digest("hex").slice(0, 32)}`;
const context = BACKGROUND_CONTEXT;
function which(program: string): string | undefined {
try {
return execFileSync("sh", ["-c", `command -v ${program}`], { encoding: "utf8" }).trim() || undefined;
} catch {
return undefined;
}
}
const sshd =
process.platform === "win32"
? undefined
: (which("sshd") ?? (existsSync("/usr/sbin/sshd") ? "/usr/sbin/sshd" : undefined));
async function freePort(): Promise<number> {
return new Promise((resolve) => {
const server = createServer().listen(0, "127.0.0.1", () => {
const { port } = server.address() as { port: number };
server.close(() => resolve(port));
});
});
}
async function waitForPort(port: number): Promise<void> {
for (let attempt = 0; attempt < 100; attempt++) {
const open = await new Promise<boolean>((resolve) => {
const socket = new Socket();
socket.once("connect", () => resolve(true)).once("error", () => resolve(false));
socket.connect(port, "127.0.0.1");
}).finally(() => undefined);
if (open) return;
await new Promise((resolve) => setTimeout(resolve, 50));
}
throw new Error("sshd did not start");
}
// A disposable sshd on localhost with its own host key, client key and home directory.
describe.skipIf(sshd === undefined)("SSH bootstrap", () => {
const root = mkdtempSync(join(tmpdir(), "pi-env-ssh-"));
const home = join(root, "home");
let server: ChildProcess | undefined;
let target: SshTarget;
beforeAll(async () => {
execFileSync("mkdir", ["-p", home]);
execFileSync("ssh-keygen", ["-q", "-t", "ed25519", "-N", "", "-f", join(root, "host_key")]);
execFileSync("ssh-keygen", ["-q", "-t", "ed25519", "-N", "", "-f", join(root, "client_key")]);
writeFileSync(join(root, "authorized_keys"), readFileSync(join(root, "client_key.pub")));
const port = await freePort();
writeFileSync(
join(root, "sshd_config"),
[
`Port ${port}`,
"ListenAddress 127.0.0.1",
`HostKey ${join(root, "host_key")}`,
`AuthorizedKeysFile ${join(root, "authorized_keys")}`,
`PidFile ${join(root, "sshd.pid")}`,
"PasswordAuthentication no",
"KbdInteractiveAuthentication no",
"StrictModes no",
`SetEnv HOME=${home}`,
"",
].join("\n"),
);
writeFileSync(join(root, "ssh_config"), "");
server = spawn(sshd!, ["-D", "-e", "-f", join(root, "sshd_config")], { stdio: "ignore" });
await waitForPort(port);
target = {
host: "127.0.0.1",
port,
user: userInfo().username,
identityFile: join(root, "client_key"),
knownHostsFile: join(root, "known_hosts"),
hostKeyAlias: "pi-env-test",
configFile: join(root, "ssh_config"),
};
});
afterAll(() => {
server?.kill();
rmSync(root, { recursive: true, force: true });
});
it("refuses an untrusted host, then deploys and runs the daemon once its key is accepted", async () => {
await expect(connectSsh({ ...target, binary: daemon })).rejects.toBeInstanceOf(HostKeyUnknownError);
const scanned = await scanHostKey(target);
const expected = execFileSync("ssh-keygen", ["-lf", join(root, "host_key.pub")], { encoding: "utf8" }).split(
" ",
)[1];
expect(scanned.fingerprints.join("\n")).toContain(expected);
await acceptHostKey(target, scanned.lines);
mkdirSync(join(home, ".pi/mobile/tools"), { recursive: true });
writeFileSync(join(home, ".pi/mobile/tools/pi-env-0123456789abcdef0123456789abcdef"), "old");
const { connection, remote } = await connectSsh({ ...target, binary: daemon });
try {
expect(remote.home).toBe(home);
const deployed = readFileSync(join(home, ".pi/mobile/tools", deployedName));
expect(deployed.equals(readFileSync(daemon))).toBe(true);
// Daemons of other contents are removed.
expect(readdirSync(join(home, ".pi/mobile/tools")).filter((name) => name.startsWith("pi-env-"))).toEqual([
deployedName,
]);
const env = new RemoteExecutionEnv({ connection, id: "pi-env:test", cwd: home });
getOrThrow(await env.writeFile("over-ssh.txt", "hello", context));
expect(getOrThrow(await env.readTextFile("over-ssh.txt", context))).toBe("hello");
const output: string[] = [];
const result = getOrThrow(
await env.exec(["sh", "-c", 'printf "%s" "$HOME"'], { onOutput: (text) => output.push(text) }, context),
);
expect(result.exitCode).toBe(0);
expect(output.join("")).toBe(home);
} finally {
connection.close();
}
}, 60_000);
it("reuses a verified daemon and replaces a tampered one", async () => {
const file = join(home, ".pi/mobile/tools", deployedName);
const before = statSync(file).mtimeMs;
(await connectSsh({ ...target, binary: daemon })).connection.close();
expect(statSync(file).mtimeMs).toBe(before);
writeFileSync(file, "tampered");
const { connection } = await connectSsh({ ...target, binary: daemon });
try {
expect(readFileSync(file).equals(readFileSync(daemon))).toBe(true);
expect((await connection.info()).home).toBe(home);
} finally {
connection.close();
}
}, 60_000);
it("verifies the daemon again before starting it after a lost connection", async () => {
const { connection } = await connectSsh({ ...target, binary: daemon });
const file = join(home, ".pi/mobile/tools", deployedName);
try {
const { pid } = await connection.info();
process.kill(pid, "SIGKILL");
await new Promise((done) => setTimeout(done, 300));
// Linux refuses to write a running program's file.
writeFileSync(file, "tampered");
// The next start redeploys the verified binary instead of running whatever is there.
expect((await connection.info()).pid).not.toBe(pid);
expect(readFileSync(file).equals(readFileSync(daemon))).toBe(true);
} finally {
connection.close();
}
}, 60_000);
it("lists files over SSH as the local environment lists them", async () => {
const tree = join(home, "tree");
for (const file of ["a/b.txt", "a/c/d.txt", ".hidden", "out/x.js", "skip/y.txt", ".git/HEAD"]) {
mkdirSync(join(tree, file, ".."), { recursive: true });
writeFileSync(join(tree, file), "");
}
mkdirSync(join(tree, "empty"));
writeFileSync(join(tree, ".gitignore"), "out/\n");
const { connection } = await connectSsh({ ...target, binary: daemon });
const local = new Connection({ command: [daemon] });
try {
const remote = new RemoteExecutionEnv({ connection, id: "pi-env:test", cwd: home });
const listing = getOrThrow(await remote.listFiles("tree", { exclude: ["skip"] }, context));
expect(listing).toEqual({
entries: [
{ path: ".gitignore", kind: "file" },
{ path: ".hidden", kind: "file" },
{ path: "a", kind: "directory" },
{ path: "a/b.txt", kind: "file" },
{ path: "a/c", kind: "directory" },
{ path: "a/c/d.txt", kind: "file" },
{ path: "empty", kind: "directory" },
],
truncated: false,
});
const env = new LocalExecutionEnv({ connection: local, cwd: home });
expect(getOrThrow(await env.listFiles("tree", { exclude: ["skip"] }, context))).toEqual(listing);
} finally {
connection.close();
local.close();
}
}, 60_000);
it("starts the daemon through the login shell only when asked", async () => {
// The login shell is the account's shell: sh and bash read .profile, zsh (the macOS default) reads .zprofile.
writeFileSync(join(home, ".profile"), "export PI_ENV_LOGIN=yes\n");
writeFileSync(join(home, ".zprofile"), "export PI_ENV_LOGIN=yes\n");
const login = async (loginShell: boolean) => {
const { connection } = await connectSsh({ ...target, binary: daemon, loginShell });
try {
const env = new RemoteExecutionEnv({ connection, id: "pi-env:test", cwd: home });
const output: string[] = [];
getOrThrow(
await env.exec(
["sh", "-c", 'printf "%s" "$PI_ENV_LOGIN"'],
{ onOutput: (text) => output.push(text) },
context,
),
);
return output.join("");
} finally {
connection.close();
}
};
expect(await login(false)).toBe("");
expect(await login(true)).toBe("yes");
}, 60_000);
it("connects lazily and reports a failed start as the error of the operation", async () => {
const lazyTarget = { ...target, knownHostsFile: join(root, "lazy_known_hosts") };
const { connection, remote } = sshConnection({ ...lazyTarget, binary: daemon });
try {
expect(remote()).toBeUndefined();
const env = new RemoteExecutionEnv({ connection, id: "pi-env:test", cwd: home });
// Nothing is trusted yet: each operation fails with the reason and tries again.
const read = await env.readTextFile("missing.txt", context);
expect(read.ok ? "ok" : read.error.message).toMatch(/not trusted/);
const command = await env.exec(["sh", "-c", "exit 0"], undefined, context);
expect(command.ok ? "ok" : command.error.code).toBe("spawn_error");
await acceptHostKey(lazyTarget, (await scanHostKey(lazyTarget)).lines);
const accepted = await env.exec(["sh", "-c", "exit 0"], undefined, context);
expect(accepted.ok ? accepted.value.exitCode : accepted.error.message).toBe(0);
expect(remote()?.home).toBe(home);
} finally {
connection.close();
}
}, 60_000);
it("accepts only host keys for the alias and never replaces a trusted key silently", async () => {
const knownHosts = join(root, "accept_known_hosts");
const scoped = { ...target, knownHostsFile: knownHosts };
const key = (name: string) => {
const file = join(root, name);
execFileSync("ssh-keygen", ["-q", "-t", "ed25519", "-N", "", "-f", file]);
return readFileSync(`${file}.pub`, "utf8").split(" ").slice(0, 2).join(" ");
};
const first = key("accept_first");
const second = key("accept_second");
await expect(acceptHostKey(scoped, [`other-alias ${first}`])).rejects.toThrow();
await expect(acceptHostKey(scoped, [`@cert-authority pi-env-test ${first}`])).rejects.toThrow();
await Promise.all([
acceptHostKey(scoped, [`pi-env-test ${first}`]),
acceptHostKey(scoped, [`pi-env-test ${first}`]),
]);
expect(readFileSync(knownHosts, "utf8")).toBe(`pi-env-test ${first}\n`);
await expect(acceptHostKey(scoped, [`pi-env-test ${second}`])).rejects.toBeInstanceOf(HostKeyChangedError);
await forgetHostKey(scoped);
await acceptHostKey(scoped, [`pi-env-test ${second}`]);
expect(readFileSync(knownHosts, "utf8")).toBe(`pi-env-test ${second}\n`);
});
it("refuses a changed host key", async () => {
const other = join(root, "other_key");
execFileSync("ssh-keygen", ["-q", "-t", "ed25519", "-N", "", "-f", other]);
const key = readFileSync(`${other}.pub`, "utf8").split(" ").slice(0, 2).join(" ");
const knownHosts = join(root, "changed_known_hosts");
writeFileSync(knownHosts, `pi-env-test ${key}\n`);
await expect(connectSsh({ ...target, knownHostsFile: knownHosts, binary: daemon })).rejects.toBeInstanceOf(
HostKeyChangedError,
);
}, 60_000);
});

162
third_party/pi/packages/env/test/terminal.test.ts vendored Normal file
View File

@@ -0,0 +1,162 @@
import { mkdtempSync, readFileSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterAll, describe, expect, it } from "vitest";
import { Connection, RemoteError } from "../src/connection.ts";
import { openTerminal } from "../src/terminal.ts";
import { daemon } from "./daemon.ts";
const connections: Connection[] = [];
const dirs: string[] = [];
afterAll(() => {
for (const connection of connections) connection.close();
for (const dir of dirs) rmSync(dir, { recursive: true, force: true });
});
function connect(): { connection: Connection; cwd: string } {
const cwd = mkdtempSync(join(tmpdir(), "pi-env-terminal-"));
dirs.push(cwd);
const connection = new Connection({ command: [daemon] });
connections.push(connection);
return { connection, cwd };
}
/** Collects output and waits for text to appear in it. */
function collector(): {
onData: (bytes: Uint8Array) => void;
bytes: () => Buffer;
waitFor: (text: string) => Promise<void>;
} {
const chunks: Buffer[] = [];
const waiting: { text: string; resolve: () => void }[] = [];
const bytes = () => Buffer.concat(chunks);
return {
bytes,
onData: (data) => {
chunks.push(Buffer.from(data));
const all = bytes().toString("latin1");
for (const wait of waiting.splice(0)) {
if (all.includes(wait.text)) wait.resolve();
else waiting.push(wait);
}
},
waitFor: (text) =>
bytes().toString("latin1").includes(text)
? Promise.resolve()
: new Promise((resolve) => waiting.push({ text, resolve })),
};
}
describe.skipIf(process.platform === "win32")("openTerminal", () => {
it("runs a program in a terminal with input, resizing and its exit code", async () => {
const { connection, cwd } = connect();
const output = collector();
const terminal = await openTerminal(connection, {
cwd,
argv: ["/bin/sh"],
env: { PS1: "$ ", TERM: "xterm-256color" },
cols: 80,
rows: 24,
onData: output.onData,
});
expect(terminal.pid).toBeGreaterThan(0);
terminal.write("test -t 0 && echo IS-A-TTY; stty size; echo $TERM\n");
await output.waitFor("IS-A-TTY");
await output.waitFor("24 80");
await output.waitFor("xterm-256color");
terminal.resize(100, 30);
terminal.write("stty size\n");
await output.waitFor("30 100");
terminal.write("exit 3\n");
expect(await terminal.exited).toBe(3);
}, 20_000);
it("reports the job in the foreground, not the shell at its prompt", async () => {
const { connection, cwd } = connect();
const output = collector();
const terminal = await openTerminal(connection, {
cwd,
// Interactive, for job control: the shell puts each job in front of the terminal.
argv: ["/bin/sh", "-i"],
env: { PS1: "$ ", TERM: "xterm-256color" },
cols: 80,
rows: 24,
onData: output.onData,
});
terminal.write("echo READY\n");
await output.waitFor("READY");
expect(await terminal.foreground()).toBeUndefined();
terminal.write("sleep 100\n");
// The shell forks before it execs: wait until the foreground job is the command itself, not the forked shell.
const named = process.platform === "darwin" || process.platform === "linux";
let foreground = await terminal.foreground();
for (let tries = 0; (!foreground || (named && foreground.name !== "sleep")) && tries < 100; tries++) {
await new Promise((resolve) => setTimeout(resolve, 20));
foreground = await terminal.foreground();
}
expect(foreground?.pid).toBeGreaterThan(0);
expect(foreground?.pid).not.toBe(terminal.pid);
if (named) expect(foreground?.name).toBe("sleep");
terminal.close();
await expect(terminal.exited).rejects.toMatchObject({ code: "aborted" });
await expect(terminal.foreground()).rejects.toMatchObject({ code: "EBADF" });
}, 20_000);
it("delivers raw bytes, not decoded text", async () => {
const { connection, cwd } = connect();
const output = collector();
const terminal = await openTerminal(connection, {
cwd,
argv: ["sh", "-c", "printf 'a\\377b'"],
cols: 80,
rows: 24,
onData: output.onData,
});
expect(await terminal.exited).toBe(0);
expect([...output.bytes()]).toEqual([0x61, 0xff, 0x62]);
});
it("keeps input in order", async () => {
const { connection, cwd } = connect();
const terminal = await openTerminal(connection, {
cwd,
argv: ["sh", "-c", "head -c 40000 > input.txt"],
cols: 80,
rows: 24,
onData: () => {},
});
let text = "";
for (let index = 0; index < 4000; index++) {
const piece = `${String(index).padStart(9, "0")}\n`;
text += piece;
terminal.write(piece);
}
expect(await terminal.exited).toBe(0);
expect(readFileSync(join(cwd, "input.txt"), "utf8")).toBe(text);
}, 20_000);
it("kills the program on close", async () => {
const { connection, cwd } = connect();
const terminal = await openTerminal(connection, {
cwd,
argv: ["sleep", "100"],
cols: 80,
rows: 24,
onData: () => {},
});
terminal.close();
await expect(terminal.exited).rejects.toMatchObject({ code: "aborted" });
});
it("fails to open in a missing directory", async () => {
const { connection, cwd } = connect();
await expect(
openTerminal(connection, { cwd: join(cwd, "missing"), cols: 80, rows: 24, onData: () => {} }),
).rejects.toBeInstanceOf(RemoteError);
});
it("rejects writes to a terminal that does not exist", async () => {
const { connection } = connect();
await expect(connection.request("ptyWrite", { target: 999 })).rejects.toMatchObject({ code: "EBADF" });
});
});

14
third_party/pi/packages/env/tsconfig.build.json vendored Normal file
View File

@@ -0,0 +1,14 @@
{
"extends": "../../tsconfig.base.json",
"compilerOptions": {
"outDir": "./dist",
"paths": {
"@earendil-works/chord": ["../chord/dist/index.d.ts"],
"@earendil-works/chord/context": ["../chord/dist/context/index.d.ts"],
"@earendil-works/pi-durable/env": ["../durable/dist/env/index.d.ts"]
},
"rootDir": "./src"
},
"include": ["src/**/*.ts"],
"exclude": ["node_modules", "dist", "**/*.d.ts", "src/**/*.d.ts"]
}

11
third_party/pi/packages/env/vitest.config.ts vendored Normal file
View File

@@ -0,0 +1,11 @@
import { defineConfig } from "vitest/config";
export default defineConfig({
test: {
environment: "node",
},
resolve: {
conditions: ["source"],
},
ssr: { resolve: { conditions: ["source"] } },
});