|
|
|
@@ -1,140 +0,0 @@
|
|
|
|
# Guests
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Problem Statement
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The user runs a homelab on Proxmox with LXC system containers and Podman, replacing it with a fully declarative NixOS configuration where both the machines and the services they host are NixOS, built from this one flake.
|
|
|
|
|
|
|
|
Each service today is an isolated guest that is a first-class citizen on a VLAN-tagged LAN: it has its own MAC address and its own IP on a specific tagged VLAN, and it writes to shared ZFS pools without permission errors.
|
|
|
|
|
|
|
|
The flake currently has only two concepts, the Host and the Module, and a Module can only be turned on at host level.
|
|
|
|
|
|
|
|
There is no way to express "run this service isolated in its own guest, on this VLAN, with these pool mounts" declaratively, and no way to reuse such a definition across machines.
|
|
|
|
|
|
|
|
The Proxmox setup the user is leaving also imposes an imperative container lifecycle and mutable state that drifts outside version control, which is the thing the migration exists to eliminate.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Solution
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Introduce a third concept alongside Host and Module: the Guest.
|
|
|
|
|
|
|
|
A Guest is a reusable, machine-independent definition that bundles the Modules it runs inside an isolated NixOS instance, realized on a Host as a nested container.
|
|
|
|
|
|
|
|
A Host composes both Modules and Guests and reads as one flat checklist, where a Module turned on directly is a host-level install and a Guest is a service running in its own isolated instance.
|
|
|
|
|
|
|
|
Each Guest becomes a first-class L2 citizen on a chosen tagged VLAN with its own MAC and its own IP, writes to shared pools through a common storage group with no permission juggling, and carries its own resource caps and secrets.
|
|
|
|
|
|
|
|
Every Guest ships with the user's baseline interactive toolset and SSH access, so any guest the user shells into is immediately a workable environment.
|
|
|
|
|
|
|
|
A Guest can also run Podman or other OCI containers in its interior, which is the declarative home for image-only software that is not worth reimplementing in Nix.
|
|
|
|
|
|
|
|
The foundations a Guest stands on — declarative VLAN and bridge networking, ZFS pool import, and a shared write convention — are themselves host-level Modules, so a Host declares its trunk, its VLANs, and its pools once and every Guest attaches to them.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## User Stories
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1. As the operator, I want to define a service as a Guest in its own file, so that the service and everything it runs are one reusable, version-controlled fact.
|
|
|
|
|
|
|
|
2. As the operator, I want a Guest to follow the same file, folder, and namespace conventions as a Module, so that I never learn a second layout and a Guest's location is its namespace.
|
|
|
|
|
|
|
|
3. As the operator, I want a Host to turn a Guest on with `enable` exactly as it turns a Module on, so that a Host stays a single flat checklist of everything it carries.
|
|
|
|
|
|
|
|
4. As the operator, I want a Module placed directly on a Host to mean a host-level install and a Guest to mean an isolated instance, so that the same checklist expresses both placements without ambiguity.
|
|
|
|
|
|
|
|
5. As the operator, I want a Guest to give its service its own MAC and its own IP on a specific tagged VLAN, so that my VLAN-segmented network treats each service as a distinct L2 citizen, exactly as Proxmox did.
|
|
|
|
|
|
|
|
6. As the operator, I want to reuse the MAC addresses my existing containers already use, so that my router's DHCP reservations keep working and the migration needs no network reconfiguration.
|
|
|
|
|
|
|
|
7. As the operator, I want a Guest whose MAC I did not set to still get a stable, readable MAC, so that I can add a reservation for a new guest without hand-assigning addresses.
|
|
|
|
|
|
|
|
8. As the operator, I want a Guest to take its address by DHCP by default and optionally a static address, so that IP management stays centralized at my router where it already lives.
|
|
|
|
|
|
|
|
9. As the operator, I want a build-time error when a Guest names a VLAN its Host has not declared, so that a misplacement fails at evaluation rather than as a broken bridge at runtime.
|
|
|
|
|
|
|
|
10. As the operator, I want a Guest to mount shared pool paths at any granularity, a single folder or a whole pool, read-only or read-write, so that each service sees exactly the data it should.
|
|
|
|
|
|
|
|
11. As the operator, I want every Guest's service to write to shared pools without permission errors, so that I never repeat the Proxmox idmap dance.
|
|
|
|
|
|
|
|
12. As the operator, I want a Guest to receive only the decrypted secrets it names, so that services get their credentials while no guest ever holds a decryption key.
|
|
|
|
|
|
|
|
13. As the operator, I want to cap a Guest's memory, CPU, and process count, so that one misbehaving service cannot starve its Host.
|
|
|
|
|
|
|
|
14. As the operator, I want every Guest to come with fish, tmux, nvim, git, and direnv, so that any guest I shell into feels like my own machine.
|
|
|
|
|
|
|
|
15. As the operator, I want to reach a Guest both directly over SSH with my keys and from its Host with `machinectl`, so that I always have a way in whether or not the network path is open.
|
|
|
|
|
|
|
|
16. As the operator, I want a Guest to run Podman and OCI containers in its interior by turning on a nesting capability, so that image-only software has a declarative home without a separate mechanism.
|
|
|
|
|
|
|
|
17. As the operator, I want the same Guest definition to be deployable on more than one Host by declaring it there with that Host's placement, so that an appliance is portable between machines.
|
|
|
|
|
|
|
|
18. As the operator, I want the networking, storage, and write conventions to be host-level Modules I declare once per Host, so that Guests attach to shared foundations instead of each re-specifying the machine.
|
|
|
|
|
|
|
|
19. As the operator, I want a Guest's innards fixed in the Guest file and only its placement supplied by the Host, so that a Guest behaves identically wherever it runs and reads honestly in isolation.
|
|
|
|
|
|
|
|
20. As the operator, I want to build a Host and know its Guests evaluate and its placements are consistent before I deploy, so that a rebuild is trustworthy.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Implementation Decisions
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### The Guest concept
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- A **Guest** is a new, auto-loaded kind of definition, resolved in `CONTEXT.md` and formalized in ADR 0006.
|
|
|
|
|
|
|
|
The Auto-loader discovers every Guest under `guests/` as a third kind alongside Modules and Hosts.
|
|
|
|
|
|
|
|
- A Guest follows the **Namespace convention** and the file-or-folder rule of a Module verbatim: a plain Guest is one file whose location is its namespace, and a Guest that needs auxiliary files becomes a folder, with subfolders as namespace segments and the same index-node rule (per ADR 0004).
|
|
|
|
|
|
|
|
- A Guest is **Module-shaped**: it declares its own `guests.<path>` option namespace carrying an `enable` plus its placement fields, and guards its body on that `enable` following the **Enable convention**.
|
|
|
|
|
|
|
|
The one difference from a Module is the payload: a Module's body merges settings into the Host, whereas a Guest's body **realizes a nested container** running the Guest's interior.
|
|
|
|
|
|
|
|
- A Guest is **backend-agnostic** through a `backend` field, but only the `container` backend (systemd-nspawn, via the platform's native nested-container mechanism) is built.
|
|
|
|
|
|
|
|
`microvm` is a reserved backend value that is not implemented in this work.
|
|
|
|
|
|
|
|
- A Guest is **sealed and singleton**: the Modules a Guest runs are fixed in the Guest file and are not overridable by a Host, and a Guest is instantiated at most once per Host, keyed by its namespace path.
|
|
|
|
|
|
|
|
Running more than one instance of a service on a Host is out of scope for this work.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Placement interface (Host-side)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
A Host instantiates a Guest by setting fields under `guests.<path>`.
|
|
|
|
|
|
|
|
The Guest owns its interior Modules; the Host owns only this placement.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Field | Meaning | Default |
|
|
|
|
|
|
|
|
| --- | --- | --- |
|
|
|
|
|
|
|
|
| `enable` | Turn the Guest on for this Host | off |
|
|
|
|
|
|
|
|
| `backend` | Realization backend | `container` |
|
|
|
|
|
|
|
|
| `vlan` | Tagged VLAN the Guest lives on; maps to the Host's `br-vlan<id>` bridge by convention | required when networked |
|
|
|
|
|
|
|
|
| `mac` | The Guest's MAC address; set it to reuse an existing address | derived deterministically and surfaced when unset |
|
|
|
|
|
|
|
|
| `address` | Static address on the VLAN | unset, meaning DHCP |
|
|
|
|
|
|
|
|
| `mounts` | Map of guest path to host path, each with a per-mount `readOnly` | read-write per mount |
|
|
|
|
|
|
|
|
| `secrets` | Names of secrets the Guest needs | none |
|
|
|
|
|
|
|
|
| `limits` | `memory`, `cpu`, `tasksMax` caps applied to the guest's unit | uncapped |
|
|
|
|
|
|
|
|
| `nesting` | Grant the nested-container prerequisites so the interior can run Podman/OCI | off |
|
|
|
|
|
|
|
|
| `autoStart` | Start the Guest at boot | on |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- The `vlan` field maps to a bridge by the `br-vlan<id>` **naming convention**, so a Guest states only which VLAN it lives on.
|
|
|
|
|
|
|
|
- A **build-time assertion** ties a Guest's `vlan` to the set of VLANs its Host's network foundation declares, so a Guest on an undeclared VLAN fails the Host build with a clear message.
|
|
|
|
|
|
|
|
- The `mac` field is pinned inside the guest through the guest's own systemd-networkd, which is the only way a nested-container MAC is stable; an unset `mac` derives a stable address from the Guest's namespace path in a locally-administered range, readable via evaluation so the operator can add a reservation.
|
|
|
|
|
|
|
|
- The `mounts` field is realized as the nested container's bind mounts and defaults to read-write, matching the migration reality that services must write to pools.
|
|
|
|
|
|
|
|
- The `nesting` field is what makes the OCI fallback a plain Guest: the Skeleton emits the nested-container cgroup-delegation and capability prerequisites once, so the operator flips one boolean and the interior's `oci-containers` runtime works, with Podman as the default runtime.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Bases and the shared environment
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- The shared base config **splits in two** (recorded under the Skeleton term): a **host base** that carries host-only machinery (bootloader, hardware profile, host identity, boot and garbage-collection timers), and a slim **guest-base** that every nested Guest stands on.
|
|
|
|
|
|
|
|
- Every Guest **imports the full `modules/` tree**, so any Module is available to enable inside a Guest; only Modules whose needs the base meets are enabled in practice.
|
|
|
|
|
|
|
|
- A Guest gets **home-manager and the same primary user** as a Host, which makes every Module placement-agnostic and removes any need for a host-versus-guest Module taxonomy.
|
|
|
|
|
|
|
|
- The **guest-base auto-enables** the `toolkit` bundle and `modules.ssh`, so every Guest has the baseline toolset and SSH access without per-Guest wiring.
|
|
|
|
|
|
|
|
Reaching a Guest by `machinectl` from its Host needs no Guest configuration and is the always-available fallback.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### New and modified Modules
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- **`modules.toolkit`** (new): a deliberate bundle Module turning on fish, tmux, nvim, git, and direnv as one unit.
|
|
|
|
|
|
|
|
The guest-base auto-enables it; a Host enables it explicitly, keeping the Host a full checklist.
|
|
|
|
|
|
|
|
This is a bundle wanted as a unit, distinct from the grouping-directory enables ADR 0004 rejected.
|
|
|
|
|
|
|
|
- **`modules.network`** (new): the host-level networking foundation.
|
|
|
|
|
|
|
|
A Host declares its trunk interface and the set of VLANs to materialize, and the Module emits one bridge per tagged VLAN with systemd-networkd and manages the Host's own management address.
|
|
|
|
|
|
|
|
- **`modules.storage.zfs`** (new): the host-level pool import.
|
|
|
|
|
|
|
|
A Host declares its host id, the pools to import, and their dataset mountpoints; the pools are durable state that is imported, never rebuilt.
|
|
|
|
|
|
|
|
- **A shared `storage` group** with a fixed gid in the shared base gives 1:1 ownership between Host and Guest.
|
|
|
|
|
|
|
|
Because the container backend uses identity mapping, a guest service that writes as the `storage` group lands on the pool as that same group, which is the entire "no permission errors" mechanism.
|
|
|
|
|
|
|
|
- **`modules.ssh`** (reused): the guest-base enables it in a guest flavor, with the operator's authorized keys and a self-generated host key, since a Guest does not carry a per-Guest host identity the way a Host does.
|
|
|
|
|
|
|
|
- **The Skeleton** grows the Guest realization: the Auto-loader's discovery of `guests/`, the compilation of a Guest's interior into a nested container, and the wiring of placement fields (VLAN attachment, MAC pinning, bind mounts, secret mounts, unit caps, nesting prerequisites).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Secrets
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- The **Host is the sole decryptor**, consistent with the host identity of ADR 0002 and the existing sops-nix setup.
|
|
|
|
|
|
|
|
The Host decrypts, and a Guest receives only the specific secret files it names, read-only bind-mounted in, with ownership aligned by the identity mapping.
|
|
|
|
|
|
|
|
A Guest holds no age key.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Testing Decisions
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
A good test here exercises **external, observable behavior of a Guest and its foundations**, not the internal shape of the generated nested-container config.
|
|
|
|
|
|
|
|
The load-bearing behaviors are: a Guest presents its own MAC and its own IP on the correct tagged VLAN, a guest service writes to a shared-group mount without permission error, the baseline toolset and SSH access are present, and a misplaced Guest fails the build.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Two seams, the fewest that cover the work:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1. **The Host toplevel build via `nix flake check`** — the existing, primary seam every Host already has.
|
|
|
|
|
|
|
|
This is where a Guest's evaluation, its placement fields, the base split, the Auto-loader wiring, and the `vlan`-against-declared-VLANs assertion are all verified.
|
|
|
|
|
|
|
|
Cheap targeted evaluations of derived values (a Guest's resolved bridge attachment, its derived MAC, the shared `storage` gid) ride on this same seam.
|
|
|
|
|
|
|
|
2. **One NixOS VM integration test**, added to the flake's `checks` so `nix flake check` runs it — a new seam, and the highest behavioral one available.
|
|
|
|
|
|
|
|
A single harness boots the `modules.network` foundation and one sample Guest and asserts the three hard requirements together: the guest has its own MAC, gets its own IP on a tagged VLAN across a virtual L2 segment, and can write to a bind-mounted directory owned by the shared `storage` group.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Prior art: the flake's `checks.<host>` toplevel builds are the established evaluation seam; the upstream NixOS test suite exercises nested-container networking, including macvlan and extra-veth cases, and is the model for the VM integration test.
|
|
|
|
|
|
|
|
The two behaviors a VM cannot honestly reproduce — real 802.1Q against the physical switch and real ZFS identity-mapped writes on the pool — are verified manually on the target Host rather than in the test.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Out of Scope
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- The `microvm` backend and any hard-isolation guest; the concept reserves the backend value but this work does not build it.
|
|
|
|
|
|
|
|
- Running more than one instance of a service on a Host; the Guest stays singleton and the operator will address multiplicity separately.
|
|
|
|
|
|
|
|
- Exotic OCI images that need their own nested init or unusual storage drivers, which are the future reason to reach for the microvm backend.
|
|
|
|
|
|
|
|
- The concrete homelab Host itself and its real values — its trunk interface name, VLAN ids, pool names, and per-Guest MAC and mount assignments — which are host-specific data supplied when a Host is added.
|
|
|
|
|
|
|
|
- Migrating specific services (the *arr stack, download clients, media servers) into Guests; this work delivers the concept and its foundations, not the service catalogue.
|
|
|
|
|
|
|
|
- Real-switch VLAN behavior and real ZFS identity-mapped writes, which are verified manually on the target Host.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Further Notes
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- The decision to introduce Guest as a third concept, backend-agnostic but container-only for now, sealed and singleton, is recorded in ADR 0006.
|
|
|
|
|
|
|
|
- The "no permission errors" result rests on the container backend's identity mapping being the privileged-container equivalent the operator already trusts from Proxmox; it is deliberately not the unprivileged idmap model, which is what made pool writes painful before.
|
|
|
|
|
|
|
|
- Pools are durable state imported by the Host, never recreated by a rebuild, so a service's data survives any rebuild or reimage.
|
|
|
|
|
|
|
|
- Podman is the default and recommended runtime inside a nesting Guest, matching what the operator already runs; a container-backend Guest is soft-isolated (shared kernel), the same isolation class as the Proxmox LXCs being replaced, so nothing is lost on that axis in the move.
|
|
|
|
|