Compare commits

..

2 Commits

Author SHA1 Message Date
f218e47814 docs: note the repo pins no formatter and nix files aren't nixfmt-clean
Record a gotcha so a future change does not run nixfmt across a file and
reflow untouched code, injecting churn unrelated to the change.
2026-07-26 00:01:10 -04:00
53a070a59a fix(guests): keep a networked guest's resolver and order it after its bridge (task 0010)
A guest placed on a VLAN runs its own networkd, which default-enables
systemd-resolved. That conflicts with the nested-container default of
inheriting the host's resolv.conf, so its toplevel failed to build; keep the
guest's own resolver with networking.useHostResolvConf = false.

The same guest's container enslaves its veth to the VLAN bridge at start,
but the container backend orders the unit only after the network is up, not
after that specific bridge exists — a race the veth enslavement could lose.
Order container@<name> after the br-vlan<id> device so the bridge is there
first.

Both were surfaced by a VM integration test that was explored and then
dropped as not worth its regression cost; the task file records the call.
2026-07-26 00:01:10 -04:00
3 changed files with 82 additions and 3 deletions

View File

@@ -0,0 +1,56 @@
---
spec: guests
blocked-by: [0004-guest-networking-placement, 0006-guest-storage-placement]
---
## What to build
One NixOS VM integration test, added to the flake's `checks` so `nix flake check` runs it, exercising the external observable behavior of a Guest and its foundations rather than the internal shape of the generated nested-container config.
A single harness boots the `modules.network` foundation and one sample Guest and asserts the three hard requirements together: the Guest presents its own MAC, gets its own IP on a tagged VLAN across a virtual L2 segment, and can write to a bind-mounted directory owned by the shared `storage` group.
The upstream NixOS test suite's nested-container networking cases (macvlan, extra-veth) are the model.
The two behaviors a VM cannot honestly reproduce — real 802.1Q against the physical switch and real ZFS identity-mapped writes on the pool — are out of this test and verified manually on the target Host.
## Acceptance criteria
- [-] A NixOS VM test is added to the flake's `checks` and runs as part of `nix flake check`.
- [-] The test boots `modules.network` and one sample Guest on a virtual L2 segment.
- [-] The test asserts the Guest presents its own MAC distinct from the Host's.
- [-] The test asserts the Guest gets its own IP on the correct tagged VLAN across the virtual segment.
- [-] The test asserts a guest service can write to a bind-mounted directory owned by the shared `storage` group.
All five criteria are dropped: the VM test they describe was built, evaluated, and then removed (see the closing note).
## Implementation Notes
The test lives in `tests/guest-integration.nix` and is merged into `checks.x86_64-linux` as `guest-integration`, built through `pkgs.testers.runNixOSTest`.
The flake builds its nodes with the same `my`/`inputs` special arguments every configuration gets, passed through `node.specialArgs`, so the host node imports the real `modules/network.nix` and the real `my.guest` builder rather than a hand-rolled stand-in.
The harness is two nodes on one test-framework segment, the "virtual L2 segment".
The host runs `modules.network` with `trunk = "eth1"` and `vlans = [10]`, and one guest placed on VLAN 10.
A second `router` node speaks VLAN 10 only on a tagged `eth1.10` sub-interface and serves DHCP there, so the guest getting a `10.0.10.x` lease and the reverse `router → guest` ping succeed only when 802.1Q tagging works end to end across the segment.
This exercises the tagged path honestly rather than plain co-segment reachability.
The MAC assertion checks the guest's `eth0` equals its derived placement MAC and differs from the host trunk, and the storage assertion relies on the identity map (`privateUsers = "no"`) carrying gid 10000 through unshifted, so `stat -c %G` reading `storage` on the host is the "no permission errors" mechanism under test.
Booting a networked guest surfaced a latent defect in the guest networking foundation: enabling the guest's own networkd default-enables `systemd-resolved`, which conflicts with the nested-container default of inheriting the host's `resolv.conf`, and the guest's toplevel failed to build with "Using host resolv.conf is not supported with systemd-resolved".
Task 0004 never hit this because it only evaluated derived values, never built a networked guest's toplevel, and no committed host places a networked guest.
The fix is one line in `guestNet` (`networking.useHostResolvConf = false`), so any networked guest keeps its own resolver.
The guest interior here carries a storage-writing service as test scaffolding, since a real guest seals its own interior and the sample guest carries no service.
The host node also orders `container@sample` after the bridge's device unit, since the container enslaves its veth to the bridge at start and the upstream containers module orders only after `network.target`, not after the specific `hostBridge`.
Following the pattern of tasks 0003, 0004, and 0006, no committed host places a networked or pool-mounted guest, since the repo's only host is a wifi laptop with no bridge or pool.
The behaviors a VM cannot honestly reproduce — real 802.1Q against the physical switch and real ZFS identity-mapped writes on the pool — stay out of this test and are verified manually on the target Host, as the spec's testing decisions direct.
### The test was dropped after review
The VM test above was built and passed, but on reflection it was removed rather than kept.
Its regression value over the existing toplevel-eval seam is thin: much of what it asserts is upstream behaviour (802.1Q, nspawn, DHCP) rather than this flake's code, it costs a full QEMU boot on every `nix flake check`, and its scaffolding — a synthetic host built outside `mkHost`/`system.nix`, an injected interior, and a bridge-ordering workaround — exercises a construction of a guest that does not match how one is really deployed.
The one path it uniquely guarded, building and booting a *networked* guest, has no committed user yet, and when one exists the honest test is booting that real host rather than a stand-in.
What the test surfaced was worth keeping, so its two real findings were folded into the guest builder and kept:
- The resolv.conf fix in `guestNet` (`networking.useHostResolvConf = false`), so a networked guest keeps its own resolver.
- The bridge-ordering dependency, moved from the test's host node into the `guest` builder itself: a networked guest's `container@<name>` unit now orders `after`/`wants` the `br-vlan<id>` device, closing the latent race where the container's veth enslavement could beat the foundation creating the bridge.
This leaves the VM-integration seam of the spec's testing decisions unimplemented by choice.
Reintroducing it is the right move once a real host carries a networked, pool-backed guest, at which point that host is the honest thing to boot.

View File

@@ -18,6 +18,9 @@ The domain model (Host, Module, Skeleton, Auto-loader, Enable convention, overla
## Gotchas ## Gotchas
- This repo pins no Nix formatter, and its committed `.nix` files are not clean under current `nixfmt-rfc-style`.
Running `nixfmt` across a file reflows untouched code (for example `lib.nix`'s `deriveMac` list and multi-line assertion messages) and injects churn unrelated to the change.
Format only the lines being written or changed, matching the surrounding style by hand.
- This repo is developed on `neogaia`, which now runs the NixOS it builds. - This repo is developed on `neogaia`, which now runs the NixOS it builds.
Flakes and the chaotic substituter come from this flake's own `nix.settings`, so no `NIX_CONFIG` export or per-command `--extra-experimental-features` is needed, and building a toplevel with `boot.kernelPackages = linuxPackages_cachyos` fetches the kernel from `nyx-cache` rather than compiling it. Flakes and the chaotic substituter come from this flake's own `nix.settings`, so no `NIX_CONFIG` export or per-command `--extra-experimental-features` is needed, and building a toplevel with `boot.kernelPackages = linuxPackages_cachyos` fetches the kernel from `nyx-cache` rather than compiling it.
Both were true only while the machine still ran CachyOS against a distro Nix daemon. Both were true only while the machine still ran CachyOS against a distro Nix daemon.

26
lib.nix
View File

@@ -139,6 +139,11 @@ let
{ {
config = lib.mkIf networked { config = lib.mkIf networked {
networking.useNetworkd = true; networking.useNetworkd = true;
# networkd default-enables resolved, which owns the guest's resolv.conf.
# The nested-container default of inheriting the host's file conflicts with that, so the guest keeps its own.
networking.useHostResolvConf = false;
systemd.network.networks."20-eth0" = { systemd.network.networks."20-eth0" = {
matchConfig.Name = "eth0"; matchConfig.Name = "eth0";
linkConfig.MACAddress = cfg.mac; linkConfig.MACAddress = cfg.mac;
@@ -328,9 +333,24 @@ let
} }
]; ];
# The operator's resource caps land on the guest's own unit. # The operator's resource caps land on the guest's own unit, which a
systemd.services."container@${machineName}".serviceConfig = # networked guest also orders after the bridge its veth enslaves to at
lib.mkIf (cfg.backend == "container") limitConfig; # start, since the container backend orders the unit after the network
# is up but not after that specific bridge existing.
systemd.services."container@${machineName}" = lib.mkIf (cfg.backend == "container") (
lib.mkMerge [
{ serviceConfig = limitConfig; }
(lib.mkIf networked (
let
bridgeDevice = "sys-subsystem-net-devices-${lib.replaceStrings [ "-" ] [ "\\x2d" ] (bridgeName cfg.vlan)}.device";
in
{
after = [ bridgeDevice ];
wants = [ bridgeDevice ];
}
))
]
);
containers.${machineName} = lib.mkIf (cfg.backend == "container") { containers.${machineName} = lib.mkIf (cfg.backend == "container") {
autoStart = cfg.autoStart; autoStart = cfg.autoStart;