I’m having my first unexpected downtime with NixOS!
I decided to move /nix
to its own BTRFS subvolume (which I’ve done on other installations without issue). Of course that didn’t work for some reason; I created the subvolume, rsync -a
the store, change the mountpoints in fileSystems
to reflect the new subvolume, and after I nixos-rebuild switch
I’m greeted with a PAM error: sudo: unable to initialize PAM: Critical error - immediate abort
Unfortunately I was remote at the time, and attempting to log in via SSH just gives me an immediate “connection closed,” so I had to wait to get home to get any further. On arrival home, I can’t even log in via console – typing in my username immediate is greeted with another username prompt.
So I rebooted into an old generation, reviewed everything (which looked OK), and a nixos-rebuild switch
and I’m back at the PAM error. Unfortunately, after a round or two of this, I mv /nix{,.bak}; mkdir -p /nix
, and now I can’t boot into an old generation (I assume because I’ve broken everything that was in /nix
).
So I then boot from a rescue disk, thinking it will be NBD to mount my new @nix
subvolume to /nix
and rebuild the system…
(using $
instead of #
in the prompts below for better markdown syntax highlighting)
[root@rescue:~] $ mount -t btrfs -o compress-force=zstd,subvol=@ /dev/sdb2 /mnt
[root@rescue:~] $ mount -t btrfs -o compress-force=zstd,subvol=@nix /dev/sdb2 /mnt/nix
[root@rescue:~] $ nixos-enter --system /nix/var/nix/profiles/system-159-link
[agenix] creating new generation in /run/agenix.d/1
[agenix] decrypting secrets...
decrypting '/nix/store/brp8s4gqjs63xg9r82cwqnc86z9zj8pw-blacklight_hashed_password.age' to '/run/agenix.d/1/blacklight_hashed_password'...
decrypting '/nix/store/pmak42xzdcyaq3q3cjlvgqh84y8k341v-builder_id_ed25519.age' to '/run/agenix.d/1/builder_id_ed25519'...
decrypting '/nix/store/p2p5n9jn075zq9mnpm8k65mf3l2n7iki-wireless_env.age' to '/run/agenix.d/1/wireless_env'...
[agenix] symlinking new secrets to /run/agenix (generation 1)...
[agenix] chowning...
setting up /etc...
I notice stuff isn’t all mounted, so I mount -a
which seems to work
[root@rescue:~] $ mount -a
[root@rescue:~] $ cd /home/n8henrie/git/nixos
[root@blacklight:/home/n8henrie/git/nixos] $ ls -l
total 40
-rwxr-xr-x 1 n8henrie n8henrie 2276 Feb 1 16:02 build.sh
drwxr-xr-x 1 n8henrie n8henrie 146 Feb 24 15:14 dotfiles
-rw-r--r-- 1 n8henrie n8henrie 7308 Mar 20 02:01 flake.lock
-rw-r--r-- 1 n8henrie n8henrie 4137 Mar 17 08:13 flake.nix
-rw-r--r-- 1 n8henrie n8henrie 3221 Mar 17 08:13 home.nix
-rw-r--r-- 1 n8henrie n8henrie 475 Jan 22 06:19 inputrc.nix
-rw-r--r-- 1 n8henrie n8henrie 1070 Dec 30 12:32 LICENSE
drwxr-xr-x 1 n8henrie n8henrie 106 Mar 20 09:28 modules
-rw-r--r-- 1 n8henrie n8henrie 24 Feb 7 19:48 README.md
drwxr-xr-x 1 n8henrie n8henrie 314 Feb 27 00:51 secrets
drwxr-xr-x 1 n8henrie n8henrie 1408 Mar 17 08:13 services
drwxr-xr-x 1 n8henrie n8henrie 70 Feb 4 13:57 system-configs
drwxr-xr-x 1 n8henrie n8henrie 20 Mar 7 20:53 utils
drwxr-xr-x 1 n8henrie n8henrie 196 Feb 1 16:02 vim
-rw-r--r-- 1 n8henrie n8henrie 3768 Feb 7 19:47 vim.nix
And now for the rebuild
[root@blacklight:/home/n8henrie/git/nixos]$ nixos-rebuild --flake . switch --show-trace
error:
while fetching the input 'git+file:///home/n8henrie/git/nixos'
error: getting the HEAD of the Git tree '/home/n8henrie/git/nixos' failed with exit code 128:
fatal: cannot change to '/home/n8henrie/git/nixos': No such file or directory
Huh?
[root@blacklight:/home/n8henrie/git/nixos]# ls -l
total 40
-rwxr-xr-x 1 n8henrie n8henrie 2276 Feb 1 16:02 build.sh
drwxr-xr-x 1 n8henrie n8henrie 146 Feb 24 15:14 dotfiles
-rw-r--r-- 1 n8henrie n8henrie 7308 Mar 20 02:01 flake.lock
-rw-r--r-- 1 n8henrie n8henrie 4137 Mar 17 08:13 flake.nix
-rw-r--r-- 1 n8henrie n8henrie 3221 Mar 17 08:13 home.nix
-rw-r--r-- 1 n8henrie n8henrie 475 Jan 22 06:19 inputrc.nix
-rw-r--r-- 1 n8henrie n8henrie 1070 Dec 30 12:32 LICENSE
drwxr-xr-x 1 n8henrie n8henrie 106 Mar 20 09:28 modules
-rw-r--r-- 1 n8henrie n8henrie 24 Feb 7 19:48 README.md
drwxr-xr-x 1 n8henrie n8henrie 314 Feb 27 00:51 secrets
drwxr-xr-x 1 n8henrie n8henrie 1408 Mar 17 08:13 services
drwxr-xr-x 1 n8henrie n8henrie 70 Feb 4 13:57 system-configs
drwxr-xr-x 1 n8henrie n8henrie 20 Mar 7 20:53 utils
drwxr-xr-x 1 n8henrie n8henrie 196 Feb 1 16:02 vim
-rw-r--r-- 1 n8henrie n8henrie 3768 Feb 7 19:47 vim.nix
[root@blacklight:/home/n8henrie/git/nixos]$ mount | grep @
/dev/sdb2 on / type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=256,subvol=/@)
/dev/sdb2 on /nix type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=45645,subvol=/@nix)
/dev/sdb2 on /.snapshots type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=261,subvol=/@snapshots)
/dev/sdb2 on /boot type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=258,subvol=/@boot)
/dev/sdb2 on /gnu type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=45646,subvol=/@gnu)
/dev/sdb2 on /home type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=260,subvol=/@home)
/dev/sdc1 on /mnt/n8storage type btrfs (rw,relatime,compress-force=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/@n8storage)
/dev/sdb2 on /var type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=259,subvol=/@var)
Currently running a few checks for nix store and hardware issues…
nix-store --verify --check-contents
btrfs check /dev/sdb2
btrfs scrub start -B /
Any other ideas?