From 8aba9b65c579ac78f3950fe984ca4657e67c3827 Mon Sep 17 00:00:00 2001 From: alexion Date: Sun, 30 Aug 2026 19:23:39 -0400 Subject: [PATCH] docs(nixos): migrate host provisioning guide --- configs/nixos/README.md | 459 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 459 insertions(+) create mode 100644 configs/nixos/README.md diff --git a/configs/nixos/README.md b/configs/nixos/README.md new file mode 100644 index 0000000..2098e44 --- /dev/null +++ b/configs/nixos/README.md @@ -0,0 +1,459 @@ +# NixOS configuration + +This document covers the procedures that put a machine into the fleet and keep its secrets readable. + +- [Installing a host from the live ISO](#installing-a-host-from-the-live-iso), the destructive one-shot that turns a host in this flake into a running, encrypted machine. +- [Provisioning an already-running host](#provisioning-an-already-running-host), done live on the machine with no reimage. +- [Editing secrets](#editing-secrets), the day-to-day workflow. +- [Recovering a wrongly-provisioned machine](#recovering-a-wrongly-provisioned-machine) from the live ISO. + +The install is destructive: it formats the target disk in full. +Read it end to end before starting. + +## What arrives by hand + +Only the **LUKS passphrase** is entered by hand. +That passphrase unlocks the encrypted disk at boot. +The disk-formatting step asks for the same passphrase. + +Everything else arrives through declared configuration. +The login password is a `sops`-encrypted secret consumed through `hashedPasswordFile`. +The SSH host keys are restored from secrets instead of generated. +No password is set interactively at any point. +`users.mutableUsers = false` means one set by hand would be ignored. + +## Identity before first boot + +A machine reads its secrets with an **age identity** at `/var/lib/sops-nix/key.txt` on its encrypted root. +Register its public half as a recipient of every secrets file the machine needs. +Keep the re-keyed files in the flake's git tree before building the system. +Nix bakes the ciphertext into the store. + +**Generate and register a host's identity before its first boot.** +The login password arrives only from a decrypted secret. +There is no fallback credential: no interactive password, no unlocked root account, and no authorized SSH key. +A first boot without a registered identity cannot decrypt the password hash. +The account has no usable password, and recovery requires the [recovery procedure](#recovering-a-wrongly-provisioned-machine). + +Identities come in two tiers. +The **admin identity** lives in Proton Pass. +It is a recipient of every file and authorizes registering a new host. +Each **host identity** is generated on its own machine and never transmitted. +It reads only that machine's file plus the shared one. +A host identity is not derived from the machine's SSH host key. +That separation lets the SSH host keys be secrets in their own right. + +## Tooling + +This configuration does not install `sops` or `age`. +Run them from nixpkgs as needed: + +```console +$ nix run nixpkgs#sops -- +$ nix shell nixpkgs#age -c age-keygen +``` + +On the live ISO these commands need `--extra-experimental-features 'nix-command flakes'`. +The ISO's daemon does not enable those features. + +## Installing a host from the live ISO + +### 0. Push the monorepo to Gitea + +From your working checkout, make sure `main` is committed and pushed to the Gitea remote: + +```console +$ git push origin main +``` + +The clone in step 2 sees only what has been pushed. +Anything left behind in your working checkout will not reach the machine. +Changes made inside that clone afterwards are separate. +Step 4 deliberately makes one. + +### 1. Boot the live ISO and join wifi + +Boot from a NixOS live ISO (the minimal installer is enough). +The installer logs in as the `nixos` user, who has passwordless `sudo`. + +On the minimal ISO, bring up wifi with `wpa_supplicant`: + +```console +$ sudo systemctl start wpa_supplicant +$ wpa_cli +> add_network +0 +> set_network 0 ssid "YOUR_SSID" +> set_network 0 psk "YOUR_WIFI_PASSWORD" +> enable_network 0 +> quit +``` + +On the graphical ISO, which ships NetworkManager, use `nmcli` instead: + +```console +$ nmcli device wifi connect "YOUR_SSID" password "YOUR_WIFI_PASSWORD" +``` + +Confirm you have connectivity (`ping -c1 github.com`) before continuing. + +### 2. Clone the monorepo locally + +Clone the monorepo onto the live ISO and work from that local checkout: + +```console +$ git clone ssh://gitea@git.alexion.dev:2022/alexion/dev.git +$ cd dev +``` + +Cloning over SSH needs your Gitea SSH key present in the live session, since the ISO starts with none. +If getting the key onto the ISO is inconvenient, clone over HTTPS instead and tell git to skip the self-signed certificate: + +```console +$ git -c http.sslVerify=false clone https://git.alexion.dev/alexion/dev.git +$ cd dev +``` + +Do **not** point `disko-install` straight at the Gitea flake URL. +Gitea serves HTTPS with a self-signed certificate and expects authentication. +Nix's flake fetcher has no easy way to skip certificate verification or supply those credentials mid-install. +A plain `git clone` sidesteps that entirely. +Over SSH there is no TLS. +Over HTTPS, git takes the `sslVerify=false` option above. +The flake fetcher does not. +`disko-install` then consumes the flake from a local path, so it does not fetch the monorepo during the build. +Every other flake input is public and still fetched from GitHub over ordinary, valid TLS. +Only the monorepo is the problem that the local clone solves. + +### 3. Generate the host identity + +Generate the identity in the live session and keep it there until step 6 writes it onto the installed root: + +```console +$ nix shell nixpkgs#age -c age-keygen -o /tmp/key.txt +Public key: age1... +``` + +`age-keygen` prints the public recipient on generation. +Recover it later from the identity itself if the line scrolls away: + +```console +$ nix shell nixpkgs#age -c age-keygen -y /tmp/key.txt +``` + +The private half never leaves this session except onto the target disk. +Do not copy it into the monorepo, and do not carry it to another machine. + +### 4. Register the recipient and re-key + +Add the public recipient to `configs/nixos/.sops.yaml` as a named anchor. +Then list it under every file the host must read: + +```yaml +keys: + - &admin age1m0pk94ysjlw3lmf6pyuv5l5pepvdjss8w0vxjv90dq6ndp02tdgsdwdvue + - &neogaia age14a04vphzjq74epfrz9a09wjw8lzchtru84awzuq2n45d8f42ychqjs89qe + - &newhost age1... + +creation_rules: + - path_regex: configs/nixos/secrets/shared\.yaml$ + key_groups: + - age: + - *admin + - *neogaia + - *newhost +``` + +If the host gets a secrets file of its own, give it a rule too. +`sops` matches a file against these rules to decide who to encrypt it to. +It refuses a file no rule matches with `no matching creation rules found`: + +```yaml + - path_regex: configs/nixos/secrets/newhost\.yaml$ + key_groups: + - age: + - *admin + - *newhost +``` + +Then re-key each file you changed. +This rewrites its data key for the new recipient list without touching any value: + +```console +$ SOPS_AGE_KEY_FILE=/path/to/admin-identity \ + nix run nixpkgs#sops -- updatekeys configs/nixos/secrets/shared.yaml +``` + +Re-keying requires an identity that can already decrypt the file. +The live ISO holds no host identity of its own. +Paste the admin identity out of Proton Pass into a file in the live session for this step. + +Then populate that file. +This needs no existing identity because encrypting only reads recipients: + +```console +$ nix run nixpkgs#sops -- configs/nixos/secrets/newhost.yaml +``` + +A host with `modules.network.ssh.enable` expects one entry per key type. +Name each entry `ssh-host--key`. +Generate each private key with `ssh-keygen -t -N "" -f /tmp/`. +The build fails during evaluation if a declared secret is absent from the file. +A host that enables the daemon without these keys will not install. +Commit the matching public halves beside the host's configuration in plaintext. +Publishing them is their purpose. + +Stage everything you changed. +A flake sees only git-tracked files, so an unstaged `configs/nixos/secrets/newhost.yaml` is invisible to evaluation even though it exists on disk: + +```console +$ git add configs/nixos/.sops.yaml configs/nixos/secrets/ +``` + +Staging is enough for the build. +The commit comes in step 8. +No push is needed here because the install builds from this local clone. + +### 5. Run `disko-install` + +Run the install as root from inside the clone: + +```console +$ sudo nix --extra-experimental-features 'nix-command flakes' run \ + github:nix-community/disko/latest#disko-install -- \ + --flake .#neogaia \ + --disk main /dev/nvme0n1 \ + --write-efi-boot-entries \ + --option extra-substituters https://nyx-cache.chaotic.cx/ \ + --option extra-trusted-public-keys nyx-cache.chaotic.cx:dJxTrgMC3V3cFfyIiBQDQorG6k1LsqurH/srpMSq7qk= +``` + +What each part does: + +- `--flake .#neogaia` installs the `neogaia` host from the local clone. +- `--disk main /dev/nvme0n1` maps disko's `main` disk to the NVMe device. + It matches the device declared in `configs/nixos/hosts/neogaia/disk.nix`. + The explicit device removes doubt about the target. +- `--write-efi-boot-entries` writes the systemd-boot entry into this machine's NVRAM. + The disk stays in the machine it was installed from. +- The two `--option` lines hand the **chaotic binary cache** to the install-time Nix daemon on the live ISO. + +Pass the chaotic substituter explicitly here. +The `nix.settings` in the NixOS configuration configure the installed system's substituters. +They do not configure the live ISO daemon that runs this build. +The ISO daemon has no `substituters` beyond `cache.nixos.org`. +Without these two `--option` flags, the build cannot fetch the prebuilt CachyOS kernel. +It compiles `linuxPackages_cachyos` and its toolchain from source on the USB stick. +The cache avoids that detour. +The install runs as root, and root is a trusted Nix user. +The daemon honours these client-supplied substituter settings. + +Partway through, disko formats the LUKS container and **prompts for a disk-encryption passphrase**. +This is the passphrase you will type at every boot to unlock the disk. +Choose it deliberately. + +When it finishes, it prints `disko-install succeeded`. +`disko-install` unmounts the target filesystem on exit. +Nothing is mounted at this point. +Step 6 remounts it. + +### 6. Write the identity onto the installed root + +Remount the just-installed system with disko. +This reopens the LUKS container and prompts for the passphrase from step 5. +It mounts the subvolumes under `/mnt`: + +```console +$ sudo nix --extra-experimental-features 'nix-command flakes' run \ + github:nix-community/disko/latest#disko -- \ + --mode mount --flake .#neogaia +``` + +Then place the identity generated in step 3. +Make root own it, and make nobody else able to read it: + +```console +$ sudo install -d -m 0755 /mnt/var/lib/sops-nix +$ sudo install -m 0400 -o root -g root /tmp/key.txt /mnt/var/lib/sops-nix/key.txt +``` + +It goes on the root subvolume rather than anywhere mounted later. +The password secret is decrypted before user accounts are created. +That happens earlier than any other mount. + +Confirm the identity matches the recipient you registered before rebooting. +This is the last cheap moment to catch a mismatch: + +```console +$ sudo nix shell nixpkgs#age -c age-keygen -y /mnt/var/lib/sops-nix/key.txt +``` + +### 7. Reboot + +Unmount and reboot into the installed system: + +```console +$ sudo umount -R /mnt +$ sudo reboot +``` + +Remove the USB stick. +At boot you are prompted for the LUKS passphrase from step 5. +After unlocking, log in at the console as `alexion`. +Use the password from the shared secrets file. +The system now has fish, tmux, nvim, and Claude Code. + +If the login is rejected, the identity and the registered recipient disagree. +See [recovery](#recovering-a-wrongly-provisioned-machine). + +### 8. Commit the recipient change + +The re-key from step 4 exists only in the live session's clone, which is gone. +From a machine that is already a recipient of the affected files, repeat the `configs/nixos/.sops.yaml` edit. +Re-key the files, then commit and push: + +```console +$ sudo SOPS_AGE_KEY_FILE=/var/lib/sops-nix/key.txt \ + nix run nixpkgs#sops -- updatekeys configs/nixos/secrets/shared.yaml +$ git add configs/nixos/.sops.yaml configs/nixos/secrets/ +$ git commit -m "feat(secrets): register newhost as a recipient" +$ git push origin main +``` + +Until this lands, the monorepo's copy of each file has one recipient fewer than the copy the new machine was built from. +The next rebuild from the monorepo would lock it out. + +## Provisioning an already-running host + +A machine that is up and running gets its identity live. +There is no reimage and no live ISO. +The running generation is the fallback. +If activation fails, the rebuild fails and the machine keeps working as it is. + +Generate the identity on the machine itself, straight into place: + +```console +$ sudo install -d -m 0755 /var/lib/sops-nix +$ sudo nix shell nixpkgs#age -c age-keygen -o /var/lib/sops-nix/key.txt +$ sudo chmod 0400 /var/lib/sops-nix/key.txt +``` + +Register the printed public recipient in `configs/nixos/.sops.yaml`. +Re-key each file the host must read, exactly as in [step 4](#4-register-the-recipient-and-re-key). +Use the admin identity. + +Then rebuild: + +```console +$ sudo nixos-rebuild switch --flake .#neogaia +``` + +Activation decrypts the secrets with the new identity. +Confirm they materialized before trusting the change: + +```console +$ sudo ls -l /run/secrets/ /run/secrets-for-users/ +``` + +Both directories matter. +Ordinary secrets land in `/run/secrets/`. +A secret marked as needed for user creation decrypts in an earlier stage. +It lands in `/run/secrets-for-users/`, where the login password hash goes. +Check that directory before rebooting. + +Commit and push the recipient change once the rebuild succeeds. + +## Editing secrets + +Opening a file decrypts it into an editor and re-encrypts on save: + +```console +$ sudo SOPS_AGE_KEY_FILE=/var/lib/sops-nix/key.txt \ + nix run nixpkgs#sops -- configs/nixos/secrets/shared.yaml +``` + +`sudo` is needed because the identity is mode `0400` and owned by root. + +**What the workstation can do alone** is anything to a file it is already a recipient of. +For `neogaia`, that means `configs/nixos/secrets/shared.yaml` and `configs/nixos/secrets/neogaia.yaml`. +Changing a value, adding a key, and adding another recipient all work from the host identity. +Each operation only requires decrypting a file the machine can already decrypt. + +**What needs the admin identity** is any file the workstation is not a recipient of. +For example, another machine's `configs/nixos/secrets/.yaml` needs it. +Unlock the admin identity out of Proton Pass for that session and point `SOPS_AGE_KEY_FILE` at it: + +```console +$ SOPS_AGE_KEY_FILE=/path/to/admin-identity \ + nix run nixpkgs#sops -- configs/nixos/secrets/zeus.yaml +``` + +That friction is the point. +A workstation that could decrypt every machine's material would make the admin identity ceremonial. +A compromised laptop would carry the whole fleet with it. +The admin identity stays a break-glass credential rather than something sitting unlocked on a machine. + +Two changes need more than a save. +Rotating the login password means generating a fresh hash with `mkpasswd`. +`users.mutableUsers = false` makes `passwd` inert. +Rebuild after changing the hash. +Re-keying the SSH host keys restarts `sshd`. +The restart is declared and automatic. + +## Recovering a wrongly-provisioned machine + +A machine whose identity and registered recipient disagree boots but cannot be logged into. +The password hash never decrypts, and there is no fallback credential. +Recovery is from the live ISO. + +Boot the ISO, join wifi, and clone the monorepo as in steps 1 and 2. +Then reopen and mount the encrypted root: + +```console +$ sudo nix --extra-experimental-features 'nix-command flakes' run \ + github:nix-community/disko/latest#disko -- \ + --mode mount --flake .#neogaia +``` + +Read the identity actually on the disk. +Compare it against the recipient the monorepo registered: + +```console +$ sudo nix shell nixpkgs#age -c age-keygen -y /mnt/var/lib/sops-nix/key.txt +``` + +**If the monorepo's recipient is right and the disk's identity is wrong**, replace the identity with the one that matches. +Then reboot. +Nothing was built against the wrong key, so no rebuild is needed: + +```console +$ sudo install -m 0400 -o root -g root /path/to/correct-key.txt /mnt/var/lib/sops-nix/key.txt +$ sudo umount -R /mnt && sudo reboot +``` + +**If the disk's identity is right and the monorepo's recipient is wrong**, re-key against the identity on the disk. +Use the admin identity to decrypt. +Then rebuild the target from the ISO: + +```console +$ SOPS_AGE_KEY_FILE=/path/to/admin-identity \ + nix run nixpkgs#sops -- updatekeys configs/nixos/secrets/shared.yaml +$ git add configs/nixos/.sops.yaml configs/nixos/secrets/ +$ sudo NIX_CONFIG="experimental-features = nix-command flakes" \ + nixos-install --root /mnt --flake .#neogaia --no-root-password \ + --option extra-substituters https://nyx-cache.chaotic.cx/ \ + --option extra-trusted-public-keys nyx-cache.chaotic.cx:dJxTrgMC3V3cFfyIiBQDQorG6k1LsqurH/srpMSq7qk= +$ sudo umount -R /mnt && sudo reboot +``` + +The rebuild is required here and not in the first case. +The secrets file is baked into the system closure at build time. +`nixos-install` reuses the already-formatted disk instead of touching the partition table. +The LUKS container and its passphrase are untouched. +It is idempotent if it fails partway. +The substituter flags matter for the same reason they do during the install. +Without them, the CachyOS kernel compiles from source on the USB stick. + +If neither identity is recoverable, generate a new one as in [step 3](#3-generate-the-host-identity). +Register it, re-key, and rebuild. +The machine's own secrets are lost, but everything encrypted to the admin identity survives.