Corrupt partition mount in fstab - cannot start NixOS

sukkeli · May 17, 2020, 8:27pm

Hi!

I have a dual boot setup on my XPS 13, using Ubuntu (for Steam!) and then NixOS. I recently updated my Ubuntu with a fresh install to get 20.04, wiping out the whole partition during the installation process. So, this also means that the uuid of the refreshed partition seems to have changed.

When I go to NixOS side, it does not boot up. It times out on the step where it trying to mount /mnt/ubuntu with uuid nannannaa. And because the uuid has changed, it does not work out. The fstab file contains the uuid from before the Ubuntu update.

I can’t touch fstab, since that is in /nix/store. So, I try updating the line in /etc/nixos/hardware-configuration.nix to have the new uuid of the partition that holds Ubuntu. I cannot do nixos-rebuild switch because it complains about some file it wants trough internet. “systemctl start network-manager” command does not seem to work. It just keeps the cursor on the next line in the terminal and does not do anything at all. systemctl status said that network-manager was “inactive (dead)”. So, because I cannot get to internet, I cannot turn my NixOS to the next state, that I already have tried to update in configuration files.

So, I think my options are:

Try to get an alternative way to access internet, so nixos-rebuild switch might work.
Try to mutate state somehow, so I could cheat NixOS that the missing mount does not matter, or hide it from NixOS somehow. And then do the real fix with configuration.nix and hardware-configuration.nix when I can boot.
Mount a USB, save important files, and then do a fresh install for NixOS.

Any tricks how I could manage to revive my system with the least effort? Thank you in advance!

Austreelis · May 18, 2020, 12:11am

Have you tried configuring your wifi manually like explained in the section Networking in the installer ? I also had to deal with a partition not mounting at boot, and, even though I’m not aware of exactly why, I always had to configure wifi manually using wpa_supplicant -B -i interface -c <(wpa_passphrase 'SSID' 'key') when network was needed to rebuild.

sukkeli · May 18, 2020, 5:52am

Thanks for the idea! I had remembered wpa_supplicant to be severely more involved than that.

Tried it. Unfortunately did not work out of the box. The response

Succesfully initialised wpa_supplicant
Could not read interface  interface flags: No such device 
nl80211: Driver does not support authentication/association or connect commands
nl80211: deinit ifname=interface disabled_11b_rates=0
Could not read interface  interface flags: No such device 
interface: Failed to initialize driver interface.

One idea occurred. Does NixOS allow chrooting? Maybe then I could access with a working network-manager.

Austreelis · May 18, 2020, 9:43am

There is a wiki post about that

sukkeli · May 18, 2020, 3:44pm

Plot thickens.

So, I chrooted to NixOS from Ubuntu 20.04. Networking did not work, but after copying /etc/resolv.com from host to chroot env, I could interface with the outside world. After that I tried to run nixos-rebuild-switch . Got an error in build phase. Apparently nixos-rebuild somehow uses also chroot-type of thing itself and would not allow doing it twice. Ran nixos-rebuild switch --option sandbox false and it worked. However, when I grepped /etc/fstab, I could still see the old uuid of /mnt/ubuntu .

Tried to boot to NixOS. No luck. Complained about same /mnt/ubuntu with old uuid. Chrooted again. Ran nixos-rebuild switch - without the sandboxing option! It still worked. And actually removed the old uuid from fstab. So now in my opinion I have no references to the old uuid in configuration.nix hardware-configuration.nix or fstab. However, when I try to boot to NixOS, it still somehow complains about the old uuid. Cannot boot. From where does it dig out this old reference? Diving in to find out.

ajs124 · May 18, 2020, 4:49pm

Do you maybe have a separate boot partition, you didn’t mount “into” the chroot?
It sounds like you might be booting an old generation of your system, which would be the case if you didn’t update the bootloader entry.

Also, you missed one option, you can change the the uuid back with tune2fs

sukkeli · May 19, 2020, 6:35pm

Yes, good guess! Thank you for the tip! Here is my lsblk:

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0         7:0    0    55M  1 loop /snap/core18/1705
loop1         7:1    0 154,3M  1 loop /snap/chromium/1143
loop2         7:2    0 255,6M  1 loop /snap/gnome-3-34-1804/33
loop3         7:3    0    55M  1 loop /snap/core18/1754
loop4         7:4    0 240,8M  1 loop /snap/gnome-3-34-1804/24
loop5         7:5    0  27,1M  1 loop /snap/snapd/7264
loop6         7:6    0  49,8M  1 loop /snap/snap-store/433
loop7         7:7    0  62,1M  1 loop /snap/gtk-common-themes/1506
nvme0n1     259:0    0   477G  0 disk 
├─nvme0n1p1 259:1    0   600M  0 part /boot/efi
├─nvme0n1p2 259:2    0     3G  0 part 
├─nvme0n1p3 259:3    0    68G  0 part /
├─nvme0n1p4 259:4    0  31,8G  0 part 
├─nvme0n1p5 259:5    0 207,6G  0 part /mnt
├─nvme0n1p6 259:6    0 117,2G  0 part 
└─nvme0n1p7 259:7    0  48,8G  0 part

So, doing the chrooting again. In addition to the mount /dev/disk/by-label/nixos /mnt I ran mount /dev/disk/by-uuid/nvme0n1p1 /mnt/boot . That however did not work so well.

This uuid in the error refers to my nixos partition.

Here is the view of the situation when I’m chrooted to nixos:

lsblk:

loop0         7:0    0    55M  1 loop 
loop1         7:1    0    55M  1 loop 
loop2         7:2    0 240.8M  1 loop 
loop3         7:3    0 154.3M  1 loop 
loop4         7:4    0  62.1M  1 loop 
loop5         7:5    0  27.1M  1 loop 
loop6         7:6    0 255.6M  1 loop 
loop7         7:7    0  49.8M  1 loop 
nvme0n1     259:0    0   477G  0 disk 
├─nvme0n1p1 259:1    0   600M  0 part /boot
├─nvme0n1p2 259:2    0     3G  0 part 
├─nvme0n1p3 259:3    0    68G  0 part 
├─nvme0n1p4 259:4    0  31.8G  0 part 
├─nvme0n1p5 259:5    0 207.6G  0 part /
├─nvme0n1p6 259:6    0 117.2G  0 part 
└─nvme0n1p7 259:7    0  48.8G  0 part

Mounts include the boot one

[root@nixos:~]# mount
/dev/nvme0n1p5 on / type ext4 (rw,relatime)
/dev/nvme0n1p1 on /boot type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
udev on /dev type devtmpfs (rw,nosuid,size=805924k,nr_inodes=2007771,mode=755)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=3,mode=620,ptmxmode=666)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,size=8059216k)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=4029608k,mode=755)
none on /run/keys type ramfs (rw,nosuid,nodev,relatime,mode=750)
tmpfs on /run/wrappers type tmpfs (rw,nodev,relatime)

And this command gives a warning which is probably unrelated:

[root@nixos:~]# nixos-rebuild switch
Running in chroot, ignoring request: show
building Nix...
building the system configuration...
org.freedesktop.DBus.Error.FileNotFound: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
warning: error(s) occurred while switching to the new configuration

[root@nixos:~]#

Early in the boot, when you get menu, I checked the list of generations and they were not the newest ones I think.

ajs124 · May 21, 2020, 4:25pm

The kernel panics because it can’t find the init it’s being to told execute on your root partition, after mounting it.

Which means there is definitely some mismatch between your bootloader configuration on /boot and your /.

Assuming you’re doing these things from a NixOS iso and your config is in /etc/nixos/configuration.nix you can simply run nixos-install after mounting /mnt and /mnt/boot, without chroot/nixos-enter.