dev/configs/nixos

NixOS configuration

This document covers the procedures that put a machine into the fleet and keep its secrets readable.

• Installing a host from the live ISO, the destructive one-shot that turns a host in this flake into a running, encrypted machine.
• Provisioning an already-running host, done live on the machine with no reimage.
• Editing secrets, the day-to-day workflow.
• Recovering a wrongly-provisioned machine from the live ISO.

The install is destructive: it formats the target disk in full. Read it end to end before starting.

What arrives by hand

Only the LUKS passphrase is entered by hand. That passphrase unlocks the encrypted disk at boot. The disk-formatting step asks for the same passphrase.

Everything else arrives through declared configuration. The login password is a sops-encrypted secret consumed through hashedPasswordFile. The SSH host keys are restored from secrets instead of generated. No password is set interactively at any point. users.mutableUsers = false means one set by hand would be ignored.

Identity before first boot

A machine reads its secrets with an age identity at /var/lib/sops-nix/key.txt on its encrypted root. Register its public half as a recipient of every secrets file the machine needs. Keep the re-keyed files in the flake's git tree before building the system. Nix bakes the ciphertext into the store.

Generate and register a host's identity before its first boot. The login password arrives only from a decrypted secret. There is no fallback credential: no interactive password, no unlocked root account, and no authorized SSH key. A first boot without a registered identity cannot decrypt the password hash. The account has no usable password, and recovery requires the recovery procedure.

Identities come in two tiers. The admin identity lives in Proton Pass. It is a recipient of every file and authorizes registering a new host. Each host identity is generated on its own machine and never transmitted. It reads only that machine's file plus the shared one. A host identity is not derived from the machine's SSH host key. That separation lets the SSH host keys be secrets in their own right.

Tooling

This configuration does not install sops or age. Run them from nixpkgs as needed:

$ nix run nixpkgs#sops -- <args>
$ nix shell nixpkgs#age -c age-keygen <args>

On the live ISO these commands need --extra-experimental-features 'nix-command flakes'. The ISO's daemon does not enable those features.

Installing a host from the live ISO

0. Push the monorepo to Gitea

From your working checkout, make sure main is committed and pushed to the Gitea remote:

$ git push origin main

The clone in step 2 sees only what has been pushed. Anything left behind in your working checkout will not reach the machine. Changes made inside that clone afterwards are separate. Step 4 deliberately makes one.

1. Boot the live ISO and join wifi

Boot from a NixOS live ISO (the minimal installer is enough). The installer logs in as the nixos user, who has passwordless sudo.

On the minimal ISO, bring up wifi with wpa_supplicant:

$ sudo systemctl start wpa_supplicant
$ wpa_cli
> add_network
0
> set_network 0 ssid "YOUR_SSID"
> set_network 0 psk "YOUR_WIFI_PASSWORD"
> enable_network 0
> quit

On the graphical ISO, which ships NetworkManager, use nmcli instead:

$ nmcli device wifi connect "YOUR_SSID" password "YOUR_WIFI_PASSWORD"

Confirm you have connectivity (ping -c1 github.com) before continuing.

2. Clone the monorepo locally

Clone the monorepo onto the live ISO and work from that local checkout:

$ git clone ssh://gitea@git.alexion.dev:2022/alexion/dev.git
$ cd dev

Cloning over SSH needs your Gitea SSH key present in the live session, since the ISO starts with none. If getting the key onto the ISO is inconvenient, clone over HTTPS instead and tell git to skip the self-signed certificate:

$ git -c http.sslVerify=false clone https://git.alexion.dev/alexion/dev.git
$ cd dev

Do not point disko-install straight at the Gitea flake URL. Gitea serves HTTPS with a self-signed certificate and expects authentication. Nix's flake fetcher has no easy way to skip certificate verification or supply those credentials mid-install. A plain git clone sidesteps that entirely. Over SSH there is no TLS. Over HTTPS, git takes the sslVerify=false option above. The flake fetcher does not. disko-install then consumes the flake from a local path, so it does not fetch the monorepo during the build. Every other flake input is public and still fetched from GitHub over ordinary, valid TLS. Only the monorepo is the problem that the local clone solves.

3. Generate the host identity

Generate the identity in the live session and keep it there until step 6 writes it onto the installed root:

$ nix shell nixpkgs#age -c age-keygen -o /tmp/key.txt
Public key: age1...

age-keygen prints the public recipient on generation. Recover it later from the identity itself if the line scrolls away:

$ nix shell nixpkgs#age -c age-keygen -y /tmp/key.txt

The private half never leaves this session except onto the target disk. Do not copy it into the monorepo, and do not carry it to another machine.

4. Register the recipient and re-key

Add the public recipient to configs/nixos/.sops.yaml as a named anchor. Then list it under every file the host must read:

keys:
  - &admin age1m0pk94ysjlw3lmf6pyuv5l5pepvdjss8w0vxjv90dq6ndp02tdgsdwdvue
  - &neogaia age14a04vphzjq74epfrz9a09wjw8lzchtru84awzuq2n45d8f42ychqjs89qe
  - &newhost age1...

creation_rules:
  - path_regex: configs/nixos/secrets/shared\.yaml$
    key_groups:
      - age:
          - *admin
          - *neogaia
          - *newhost

If the host gets a secrets file of its own, give it a rule too. sops matches a file against these rules to decide who to encrypt it to. It refuses a file no rule matches with no matching creation rules found:

  - path_regex: configs/nixos/secrets/newhost\.yaml$
    key_groups:
      - age:
          - *admin
          - *newhost

Then re-key each file you changed. This rewrites its data key for the new recipient list without touching any value:

$ SOPS_AGE_KEY_FILE=/path/to/admin-identity \
    nix run nixpkgs#sops -- updatekeys configs/nixos/secrets/shared.yaml

Re-keying requires an identity that can already decrypt the file. The live ISO holds no host identity of its own. Paste the admin identity out of Proton Pass into a file in the live session for this step.

Then populate that file. This needs no existing identity because encrypting only reads recipients:

$ nix run nixpkgs#sops -- configs/nixos/secrets/newhost.yaml

A host with modules.network.ssh.enable expects one entry per key type. Name each entry ssh-host-<type>-key. Generate each private key with ssh-keygen -t <type> -N "" -f /tmp/<type>. The build fails during evaluation if a declared secret is absent from the file. A host that enables the daemon without these keys will not install. Commit the matching public halves beside the host's configuration in plaintext. Publishing them is their purpose.

Stage everything you changed. A flake sees only git-tracked files, so an unstaged configs/nixos/secrets/newhost.yaml is invisible to evaluation even though it exists on disk:

$ git add configs/nixos/.sops.yaml configs/nixos/secrets/

Staging is enough for the build. The commit comes in step 8. No push is needed here because the install builds from this local clone.

5. Run disko-install

Run the install as root from inside the clone:

$ sudo nix --extra-experimental-features 'nix-command flakes' run \
    github:nix-community/disko/latest#disko-install -- \
    --flake .#neogaia \
    --disk main /dev/nvme0n1 \
    --write-efi-boot-entries \
    --option extra-substituters https://nyx-cache.chaotic.cx/ \
    --option extra-trusted-public-keys nyx-cache.chaotic.cx:dJxTrgMC3V3cFfyIiBQDQorG6k1LsqurH/srpMSq7qk=

What each part does:

• --flake .#neogaia installs the neogaia host from the local clone.
• --disk main /dev/nvme0n1 maps disko's main disk to the NVMe device. It matches the device declared in configs/nixos/hosts/neogaia/disk.nix. The explicit device removes doubt about the target.
• --write-efi-boot-entries writes the systemd-boot entry into this machine's NVRAM. The disk stays in the machine it was installed from.
• The two --option lines hand the chaotic binary cache to the install-time Nix daemon on the live ISO.

Pass the chaotic substituter explicitly here. The nix.settings in the NixOS configuration configure the installed system's substituters. They do not configure the live ISO daemon that runs this build. The ISO daemon has no substituters beyond cache.nixos.org. Without these two --option flags, the build cannot fetch the prebuilt CachyOS kernel. It compiles linuxPackages_cachyos and its toolchain from source on the USB stick. The cache avoids that detour. The install runs as root, and root is a trusted Nix user. The daemon honours these client-supplied substituter settings.

Partway through, disko formats the LUKS container and prompts for a disk-encryption passphrase. This is the passphrase you will type at every boot to unlock the disk. Choose it deliberately.

When it finishes, it prints disko-install succeeded. disko-install unmounts the target filesystem on exit. Nothing is mounted at this point. Step 6 remounts it.

6. Write the identity onto the installed root

Remount the just-installed system with disko. This reopens the LUKS container and prompts for the passphrase from step 5. It mounts the subvolumes under /mnt:

$ sudo nix --extra-experimental-features 'nix-command flakes' run \
    github:nix-community/disko/latest#disko -- \
    --mode mount --flake .#neogaia

Then place the identity generated in step 3. Make root own it, and make nobody else able to read it:

$ sudo install -d -m 0755 /mnt/var/lib/sops-nix
$ sudo install -m 0400 -o root -g root /tmp/key.txt /mnt/var/lib/sops-nix/key.txt

It goes on the root subvolume rather than anywhere mounted later. The password secret is decrypted before user accounts are created. That happens earlier than any other mount.

Confirm the identity matches the recipient you registered before rebooting. This is the last cheap moment to catch a mismatch:

$ sudo nix shell nixpkgs#age -c age-keygen -y /mnt/var/lib/sops-nix/key.txt

7. Reboot

Unmount and reboot into the installed system:

$ sudo umount -R /mnt
$ sudo reboot

Remove the USB stick. At boot you are prompted for the LUKS passphrase from step 5. After unlocking, log in at the console as alexion. Use the password from the shared secrets file. The system now has fish, tmux, nvim, and Claude Code.

If the login is rejected, the identity and the registered recipient disagree. See recovery.

8. Commit the recipient change

The re-key from step 4 exists only in the live session's clone, which is gone. From a machine that is already a recipient of the affected files, repeat the configs/nixos/.sops.yaml edit. Re-key the files, then commit and push:

$ sudo SOPS_AGE_KEY_FILE=/var/lib/sops-nix/key.txt \
    nix run nixpkgs#sops -- updatekeys configs/nixos/secrets/shared.yaml
$ git add configs/nixos/.sops.yaml configs/nixos/secrets/
$ git commit -m "feat(secrets): register newhost as a recipient"
$ git push origin main

Until this lands, the monorepo's copy of each file has one recipient fewer than the copy the new machine was built from. The next rebuild from the monorepo would lock it out.

Provisioning an already-running host

A machine that is up and running gets its identity live. There is no reimage and no live ISO. The running generation is the fallback. If activation fails, the rebuild fails and the machine keeps working as it is.

Generate the identity on the machine itself, straight into place:

$ sudo install -d -m 0755 /var/lib/sops-nix
$ sudo nix shell nixpkgs#age -c age-keygen -o /var/lib/sops-nix/key.txt
$ sudo chmod 0400 /var/lib/sops-nix/key.txt

Register the printed public recipient in configs/nixos/.sops.yaml. Re-key each file the host must read, exactly as in step 4. Use the admin identity.

Then rebuild:

$ sudo nixos-rebuild switch --flake .#neogaia

Activation decrypts the secrets with the new identity. Confirm they materialized before trusting the change:

$ sudo ls -l /run/secrets/ /run/secrets-for-users/

Both directories matter. Ordinary secrets land in /run/secrets/. A secret marked as needed for user creation decrypts in an earlier stage. It lands in /run/secrets-for-users/, where the login password hash goes. Check that directory before rebooting.

Commit and push the recipient change once the rebuild succeeds.

Editing secrets

Opening a file decrypts it into an editor and re-encrypts on save:

$ sudo SOPS_AGE_KEY_FILE=/var/lib/sops-nix/key.txt \
    nix run nixpkgs#sops -- configs/nixos/secrets/shared.yaml

sudo is needed because the identity is mode 0400 and owned by root.

What the workstation can do alone is anything to a file it is already a recipient of. For neogaia, that means configs/nixos/secrets/shared.yaml and configs/nixos/secrets/neogaia.yaml. Changing a value, adding a key, and adding another recipient all work from the host identity. Each operation only requires decrypting a file the machine can already decrypt.

What needs the admin identity is any file the workstation is not a recipient of. For example, another machine's configs/nixos/secrets/<host>.yaml needs it. Unlock the admin identity out of Proton Pass for that session and point SOPS_AGE_KEY_FILE at it:

$ SOPS_AGE_KEY_FILE=/path/to/admin-identity \
    nix run nixpkgs#sops -- configs/nixos/secrets/zeus.yaml

That friction is the point. A workstation that could decrypt every machine's material would make the admin identity ceremonial. A compromised laptop would carry the whole fleet with it. The admin identity stays a break-glass credential rather than something sitting unlocked on a machine.

Two changes need more than a save. Rotating the login password means generating a fresh hash with mkpasswd. users.mutableUsers = false makes passwd inert. Rebuild after changing the hash. Re-keying the SSH host keys restarts sshd. The restart is declared and automatic.

Recovering a wrongly-provisioned machine

A machine whose identity and registered recipient disagree boots but cannot be logged into. The password hash never decrypts, and there is no fallback credential. Recovery is from the live ISO.

Boot the ISO, join wifi, and clone the monorepo as in steps 1 and 2. Then reopen and mount the encrypted root:

$ sudo nix --extra-experimental-features 'nix-command flakes' run \
    github:nix-community/disko/latest#disko -- \
    --mode mount --flake .#neogaia

Read the identity actually on the disk. Compare it against the recipient the monorepo registered:

$ sudo nix shell nixpkgs#age -c age-keygen -y /mnt/var/lib/sops-nix/key.txt

If the monorepo's recipient is right and the disk's identity is wrong, replace the identity with the one that matches. Then reboot. Nothing was built against the wrong key, so no rebuild is needed:

$ sudo install -m 0400 -o root -g root /path/to/correct-key.txt /mnt/var/lib/sops-nix/key.txt
$ sudo umount -R /mnt && sudo reboot

If the disk's identity is right and the monorepo's recipient is wrong, re-key against the identity on the disk. Use the admin identity to decrypt. Then rebuild the target from the ISO:

$ SOPS_AGE_KEY_FILE=/path/to/admin-identity \
    nix run nixpkgs#sops -- updatekeys configs/nixos/secrets/shared.yaml
$ git add configs/nixos/.sops.yaml configs/nixos/secrets/
$ sudo NIX_CONFIG="experimental-features = nix-command flakes" \
    nixos-install --root /mnt --flake .#neogaia --no-root-password \
    --option extra-substituters https://nyx-cache.chaotic.cx/ \
    --option extra-trusted-public-keys nyx-cache.chaotic.cx:dJxTrgMC3V3cFfyIiBQDQorG6k1LsqurH/srpMSq7qk=
$ sudo umount -R /mnt && sudo reboot

The rebuild is required here and not in the first case. The secrets file is baked into the system closure at build time. nixos-install reuses the already-formatted disk instead of touching the partition table. The LUKS container and its passphrase are untouched. It is idempotent if it fails partway. The substituter flags matter for the same reason they do during the install. Without them, the CachyOS kernel compiles from source on the USB stick.

If neither identity is recoverable, generate a new one as in step 3. Register it, re-key, and rebuild. The machine's own secrets are lost, but everything encrypted to the admin identity survives.