alexion/dev
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(nixos): migrate host provisioning guide

Browse Source
This commit is contained in:
alexion
2026-08-30 19:23:39 -04:00
parent 99aaf3cf58
commit 8aba9b65c5
1 changed files with 459 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

459
configs/nixos/README.md Normal file
View File

@@ -0,0 +1,459 @@
# NixOS configuration
This document covers the procedures that put a machine into the fleet and keep its secrets readable.
- [Installing a host from the live ISO](#installing-a-host-from-the-live-iso), the destructive one-shot that turns a host in this flake into a running, encrypted machine.
- [Provisioning an already-running host](#provisioning-an-already-running-host), done live on the machine with no reimage.
- [Editing secrets](#editing-secrets), the day-to-day workflow.
- [Recovering a wrongly-provisioned machine](#recovering-a-wrongly-provisioned-machine) from the live ISO.
The install is destructive: it formats the target disk in full.
Read it end to end before starting.
## What arrives by hand
Only the **LUKS passphrase** is entered by hand.
That passphrase unlocks the encrypted disk at boot.
The disk-formatting step asks for the same passphrase.
Everything else arrives through declared configuration.
The login password is a `sops`-encrypted secret consumed through `hashedPasswordFile`.
The SSH host keys are restored from secrets instead of generated.
No password is set interactively at any point.
`users.mutableUsers = false` means one set by hand would be ignored.
## Identity before first boot
A machine reads its secrets with an **age identity** at `/var/lib/sops-nix/key.txt` on its encrypted root.
Register its public half as a recipient of every secrets file the machine needs.
Keep the re-keyed files in the flake's git tree before building the system.
Nix bakes the ciphertext into the store.
**Generate and register a host's identity before its first boot.**
The login password arrives only from a decrypted secret.
There is no fallback credential: no interactive password, no unlocked root account, and no authorized SSH key.
A first boot without a registered identity cannot decrypt the password hash.
The account has no usable password, and recovery requires the [recovery procedure](#recovering-a-wrongly-provisioned-machine).
Identities come in two tiers.
The **admin identity** lives in Proton Pass.
It is a recipient of every file and authorizes registering a new host.
Each **host identity** is generated on its own machine and never transmitted.
It reads only that machine's file plus the shared one.
A host identity is not derived from the machine's SSH host key.
That separation lets the SSH host keys be secrets in their own right.
## Tooling
This configuration does not install `sops` or `age`.
Run them from nixpkgs as needed:
```console
$ nix run nixpkgs#sops -- <args>
$ nix shell nixpkgs#age -c age-keygen <args>
```
On the live ISO these commands need `--extra-experimental-features 'nix-command flakes'`.
The ISO's daemon does not enable those features.
## Installing a host from the live ISO
### 0. Push the monorepo to Gitea
From your working checkout, make sure `main` is committed and pushed to the Gitea remote:
```console
$ git push origin main
```
The clone in step 2 sees only what has been pushed.
Anything left behind in your working checkout will not reach the machine.
Changes made inside that clone afterwards are separate.
Step 4 deliberately makes one.
### 1. Boot the live ISO and join wifi
Boot from a NixOS live ISO (the minimal installer is enough).
The installer logs in as the `nixos` user, who has passwordless `sudo`.
On the minimal ISO, bring up wifi with `wpa_supplicant`:
```console
$ sudo systemctl start wpa_supplicant
$ wpa_cli
> add_network
0
> set_network 0 ssid "YOUR_SSID"
> set_network 0 psk "YOUR_WIFI_PASSWORD"
> enable_network 0
> quit
```
On the graphical ISO, which ships NetworkManager, use `nmcli` instead:
```console
$ nmcli device wifi connect "YOUR_SSID" password "YOUR_WIFI_PASSWORD"
```
Confirm you have connectivity (`ping -c1 github.com`) before continuing.
### 2. Clone the monorepo locally
Clone the monorepo onto the live ISO and work from that local checkout:
```console
$ git clone ssh://gitea@git.alexion.dev:2022/alexion/dev.git
$ cd dev
```
Cloning over SSH needs your Gitea SSH key present in the live session, since the ISO starts with none.
If getting the key onto the ISO is inconvenient, clone over HTTPS instead and tell git to skip the self-signed certificate:
```console
$ git -c http.sslVerify=false clone https://git.alexion.dev/alexion/dev.git
$ cd dev
```
Do **not** point `disko-install` straight at the Gitea flake URL.
Gitea serves HTTPS with a self-signed certificate and expects authentication.
Nix's flake fetcher has no easy way to skip certificate verification or supply those credentials mid-install.
A plain `git clone` sidesteps that entirely.
Over SSH there is no TLS.
Over HTTPS, git takes the `sslVerify=false` option above.
The flake fetcher does not.
`disko-install` then consumes the flake from a local path, so it does not fetch the monorepo during the build.
Every other flake input is public and still fetched from GitHub over ordinary, valid TLS.
Only the monorepo is the problem that the local clone solves.
### 3. Generate the host identity
Generate the identity in the live session and keep it there until step 6 writes it onto the installed root:
```console
$ nix shell nixpkgs#age -c age-keygen -o /tmp/key.txt
Public key: age1...
```
`age-keygen` prints the public recipient on generation.
Recover it later from the identity itself if the line scrolls away:
```console
$ nix shell nixpkgs#age -c age-keygen -y /tmp/key.txt
```
The private half never leaves this session except onto the target disk.
Do not copy it into the monorepo, and do not carry it to another machine.
### 4. Register the recipient and re-key
Add the public recipient to `configs/nixos/.sops.yaml` as a named anchor.
Then list it under every file the host must read:
```yaml
keys:
- &admin age1m0pk94ysjlw3lmf6pyuv5l5pepvdjss8w0vxjv90dq6ndp02tdgsdwdvue
- &neogaia age14a04vphzjq74epfrz9a09wjw8lzchtru84awzuq2n45d8f42ychqjs89qe
- &newhost age1...
creation_rules:
- path_regex: configs/nixos/secrets/shared\.yaml$
key_groups:
- age:
- *admin
- *neogaia
- *newhost
```
If the host gets a secrets file of its own, give it a rule too.
`sops` matches a file against these rules to decide who to encrypt it to.
It refuses a file no rule matches with `no matching creation rules found`:
```yaml
- path_regex: configs/nixos/secrets/newhost\.yaml$
key_groups:
- age:
- *admin
- *newhost
```
Then re-key each file you changed.
This rewrites its data key for the new recipient list without touching any value:
```console
$ SOPS_AGE_KEY_FILE=/path/to/admin-identity \
nix run nixpkgs#sops -- updatekeys configs/nixos/secrets/shared.yaml
```
Re-keying requires an identity that can already decrypt the file.
The live ISO holds no host identity of its own.
Paste the admin identity out of Proton Pass into a file in the live session for this step.
Then populate that file.
This needs no existing identity because encrypting only reads recipients:
```console
$ nix run nixpkgs#sops -- configs/nixos/secrets/newhost.yaml
```
A host with `modules.network.ssh.enable` expects one entry per key type.
Name each entry `ssh-host-<type>-key`.
Generate each private key with `ssh-keygen -t <type> -N "" -f /tmp/<type>`.
The build fails during evaluation if a declared secret is absent from the file.
A host that enables the daemon without these keys will not install.
Commit the matching public halves beside the host's configuration in plaintext.
Publishing them is their purpose.
Stage everything you changed.
A flake sees only git-tracked files, so an unstaged `configs/nixos/secrets/newhost.yaml` is invisible to evaluation even though it exists on disk:
```console
$ git add configs/nixos/.sops.yaml configs/nixos/secrets/
```
Staging is enough for the build.
The commit comes in step 8.
No push is needed here because the install builds from this local clone.
### 5. Run `disko-install`
Run the install as root from inside the clone:
```console
$ sudo nix --extra-experimental-features 'nix-command flakes' run \
github:nix-community/disko/latest#disko-install -- \
--flake .#neogaia \
--disk main /dev/nvme0n1 \
--write-efi-boot-entries \
--option extra-substituters https://nyx-cache.chaotic.cx/ \
--option extra-trusted-public-keys nyx-cache.chaotic.cx:dJxTrgMC3V3cFfyIiBQDQorG6k1LsqurH/srpMSq7qk=
```
What each part does:
- `--flake .#neogaia` installs the `neogaia` host from the local clone.
- `--disk main /dev/nvme0n1` maps disko's `main` disk to the NVMe device.
It matches the device declared in `configs/nixos/hosts/neogaia/disk.nix`.
The explicit device removes doubt about the target.
- `--write-efi-boot-entries` writes the systemd-boot entry into this machine's NVRAM.
The disk stays in the machine it was installed from.
- The two `--option` lines hand the **chaotic binary cache** to the install-time Nix daemon on the live ISO.
Pass the chaotic substituter explicitly here.
The `nix.settings` in the NixOS configuration configure the installed system's substituters.
They do not configure the live ISO daemon that runs this build.
The ISO daemon has no `substituters` beyond `cache.nixos.org`.
Without these two `--option` flags, the build cannot fetch the prebuilt CachyOS kernel.
It compiles `linuxPackages_cachyos` and its toolchain from source on the USB stick.
The cache avoids that detour.
The install runs as root, and root is a trusted Nix user.
The daemon honours these client-supplied substituter settings.
Partway through, disko formats the LUKS container and **prompts for a disk-encryption passphrase**.
This is the passphrase you will type at every boot to unlock the disk.
Choose it deliberately.
When it finishes, it prints `disko-install succeeded`.
`disko-install` unmounts the target filesystem on exit.
Nothing is mounted at this point.
Step 6 remounts it.
### 6. Write the identity onto the installed root
Remount the just-installed system with disko.
This reopens the LUKS container and prompts for the passphrase from step 5.
It mounts the subvolumes under `/mnt`:
```console
$ sudo nix --extra-experimental-features 'nix-command flakes' run \
github:nix-community/disko/latest#disko -- \
--mode mount --flake .#neogaia
```
Then place the identity generated in step 3.
Make root own it, and make nobody else able to read it:
```console
$ sudo install -d -m 0755 /mnt/var/lib/sops-nix
$ sudo install -m 0400 -o root -g root /tmp/key.txt /mnt/var/lib/sops-nix/key.txt
```
It goes on the root subvolume rather than anywhere mounted later.
The password secret is decrypted before user accounts are created.
That happens earlier than any other mount.
Confirm the identity matches the recipient you registered before rebooting.
This is the last cheap moment to catch a mismatch:
```console
$ sudo nix shell nixpkgs#age -c age-keygen -y /mnt/var/lib/sops-nix/key.txt
```
### 7. Reboot
Unmount and reboot into the installed system:
```console
$ sudo umount -R /mnt
$ sudo reboot
```
Remove the USB stick.
At boot you are prompted for the LUKS passphrase from step 5.
After unlocking, log in at the console as `alexion`.
Use the password from the shared secrets file.
The system now has fish, tmux, nvim, and Claude Code.
If the login is rejected, the identity and the registered recipient disagree.
See [recovery](#recovering-a-wrongly-provisioned-machine).
### 8. Commit the recipient change
The re-key from step 4 exists only in the live session's clone, which is gone.
From a machine that is already a recipient of the affected files, repeat the `configs/nixos/.sops.yaml` edit.
Re-key the files, then commit and push:
```console
$ sudo SOPS_AGE_KEY_FILE=/var/lib/sops-nix/key.txt \
nix run nixpkgs#sops -- updatekeys configs/nixos/secrets/shared.yaml
$ git add configs/nixos/.sops.yaml configs/nixos/secrets/
$ git commit -m "feat(secrets): register newhost as a recipient"
$ git push origin main
```
Until this lands, the monorepo's copy of each file has one recipient fewer than the copy the new machine was built from.
The next rebuild from the monorepo would lock it out.
## Provisioning an already-running host
A machine that is up and running gets its identity live.
There is no reimage and no live ISO.
The running generation is the fallback.
If activation fails, the rebuild fails and the machine keeps working as it is.
Generate the identity on the machine itself, straight into place:
```console
$ sudo install -d -m 0755 /var/lib/sops-nix
$ sudo nix shell nixpkgs#age -c age-keygen -o /var/lib/sops-nix/key.txt
$ sudo chmod 0400 /var/lib/sops-nix/key.txt
```
Register the printed public recipient in `configs/nixos/.sops.yaml`.
Re-key each file the host must read, exactly as in [step 4](#4-register-the-recipient-and-re-key).
Use the admin identity.
Then rebuild:
```console
$ sudo nixos-rebuild switch --flake .#neogaia
```
Activation decrypts the secrets with the new identity.
Confirm they materialized before trusting the change:
```console
$ sudo ls -l /run/secrets/ /run/secrets-for-users/
```
Both directories matter.
Ordinary secrets land in `/run/secrets/`.
A secret marked as needed for user creation decrypts in an earlier stage.
It lands in `/run/secrets-for-users/`, where the login password hash goes.
Check that directory before rebooting.
Commit and push the recipient change once the rebuild succeeds.
## Editing secrets
Opening a file decrypts it into an editor and re-encrypts on save:
```console
$ sudo SOPS_AGE_KEY_FILE=/var/lib/sops-nix/key.txt \
nix run nixpkgs#sops -- configs/nixos/secrets/shared.yaml
```
`sudo` is needed because the identity is mode `0400` and owned by root.
**What the workstation can do alone** is anything to a file it is already a recipient of.
For `neogaia`, that means `configs/nixos/secrets/shared.yaml` and `configs/nixos/secrets/neogaia.yaml`.
Changing a value, adding a key, and adding another recipient all work from the host identity.
Each operation only requires decrypting a file the machine can already decrypt.
**What needs the admin identity** is any file the workstation is not a recipient of.
For example, another machine's `configs/nixos/secrets/<host>.yaml` needs it.
Unlock the admin identity out of Proton Pass for that session and point `SOPS_AGE_KEY_FILE` at it:
```console
$ SOPS_AGE_KEY_FILE=/path/to/admin-identity \
nix run nixpkgs#sops -- configs/nixos/secrets/zeus.yaml
```
That friction is the point.
A workstation that could decrypt every machine's material would make the admin identity ceremonial.
A compromised laptop would carry the whole fleet with it.
The admin identity stays a break-glass credential rather than something sitting unlocked on a machine.
Two changes need more than a save.
Rotating the login password means generating a fresh hash with `mkpasswd`.
`users.mutableUsers = false` makes `passwd` inert.
Rebuild after changing the hash.
Re-keying the SSH host keys restarts `sshd`.
The restart is declared and automatic.
## Recovering a wrongly-provisioned machine
A machine whose identity and registered recipient disagree boots but cannot be logged into.
The password hash never decrypts, and there is no fallback credential.
Recovery is from the live ISO.
Boot the ISO, join wifi, and clone the monorepo as in steps 1 and 2.
Then reopen and mount the encrypted root:
```console
$ sudo nix --extra-experimental-features 'nix-command flakes' run \
github:nix-community/disko/latest#disko -- \
--mode mount --flake .#neogaia
```
Read the identity actually on the disk.
Compare it against the recipient the monorepo registered:
```console
$ sudo nix shell nixpkgs#age -c age-keygen -y /mnt/var/lib/sops-nix/key.txt
```
**If the monorepo's recipient is right and the disk's identity is wrong**, replace the identity with the one that matches.
Then reboot.
Nothing was built against the wrong key, so no rebuild is needed:
```console
$ sudo install -m 0400 -o root -g root /path/to/correct-key.txt /mnt/var/lib/sops-nix/key.txt
$ sudo umount -R /mnt && sudo reboot
```
**If the disk's identity is right and the monorepo's recipient is wrong**, re-key against the identity on the disk.
Use the admin identity to decrypt.
Then rebuild the target from the ISO:
```console
$ SOPS_AGE_KEY_FILE=/path/to/admin-identity \
nix run nixpkgs#sops -- updatekeys configs/nixos/secrets/shared.yaml
$ git add configs/nixos/.sops.yaml configs/nixos/secrets/
$ sudo NIX_CONFIG="experimental-features = nix-command flakes" \
nixos-install --root /mnt --flake .#neogaia --no-root-password \
--option extra-substituters https://nyx-cache.chaotic.cx/ \
--option extra-trusted-public-keys nyx-cache.chaotic.cx:dJxTrgMC3V3cFfyIiBQDQorG6k1LsqurH/srpMSq7qk=
$ sudo umount -R /mnt && sudo reboot
```
The rebuild is required here and not in the first case.
The secrets file is baked into the system closure at build time.
`nixos-install` reuses the already-formatted disk instead of touching the partition table.
The LUKS container and its passphrase are untouched.
It is idempotent if it fails partway.
The substituter flags matter for the same reason they do during the install.
Without them, the CachyOS kernel compiles from source on the USB stick.
If neither identity is recoverable, generate a new one as in [step 3](#3-generate-the-host-identity).
Register it, re-key, and rebuild.
The machine's own secrets are lost, but everything encrypted to the admin identity survives.