Nixos-rebuild doesn't work as expected in a chroot environment

Help
1

I rent a dedicated server from Hetzner running NixOS, and occasionally I bork the system in which I’m unable to boot. This is easy to solve thankfully due to booting into a previous generation, but in order to get access to the boot process I need to order a KVM which is a lengthy manual process.

Alternatively, I can activate a Debian based rescue system on-demand and reboot into that. From there, I just follow the instruction as per “manual chroot” in the wiki.

Then I just edit my configuration as per a line from the page linked above:

You should now be in your NixOS system, and should be able to adjust it by e.g. editing /etc/nixos/configuration.nix and running nixos-rebuild switch as usual.

But rebuilding fails, in several stages:

# nixos-rebuild boot
building Nix...
building the system configuration...
these 2 derivations will be built:
  /nix/store/s45y1lwdhxl31i4j4bvh2ymfj88c80lc-users-groups.json.drv
  /nix/store/2ayjwlal61q2mpxqln67ncxvfi9mg0qa-nixos-system-bliksem-24.05.4469.6e99f2a27d60.drv
error:
       … while setting up the build environment

       error: cannot pivot old root directory onto '/nix/store/s45y1lwdhxl31i4j4bvh2ymfj88c80lc-users-groups.json.drv.chroot/root/real-root': Invalid argument

I’ve found this Issue which mentions a possible solution. But getting this error, which is also documented in the wiki:

# nixos-rebuild boot --option sandbox false
building Nix...
building the system configuration...
these 2 derivations will be built:
  /nix/store/s45y1lwdhxl31i4j4bvh2ymfj88c80lc-users-groups.json.drv
  /nix/store/2ayjwlal61q2mpxqln67ncxvfi9mg0qa-nixos-system-bliksem-24.05.4469.6e99f2a27d60.drv
building '/nix/store/s45y1lwdhxl31i4j4bvh2ymfj88c80lc-users-groups.json.drv'...
building '/nix/store/2ayjwlal61q2mpxqln67ncxvfi9mg0qa-nixos-system-bliksem-24.05.4469.6e99f2a27d60.drv'...
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
efiSysMountPoint = '/boot' is not a mounted partition. Is the path configured correctly?
Traceback (most recent call last):
  File "/nix/store/lpr9a49mfalfir0za242x57377n2yfzq-17nyliyy0k59a6ky037y6fj902f1s2w5-systemd-boot", line 397, in <module>
    main()
  File "/nix/store/lpr9a49mfalfir0za242x57377n2yfzq-17nyliyy0k59a6ky037y6fj902f1s2w5-systemd-boot", line 377, in main
    subprocess.check_call(CHECK_MOUNTPOINTS)
  File "/nix/store/3wb0055984n2whn449hywsl4ag9gcjir-python3-3.11.9/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '/nix/store/za0y24r2l67222kyz8hi2nzi9qpavmav-check-mountpoints' returned non-zero exit status 1.
warning: error(s) occurred while switching to the new configuration

Then there’s the issue of /boot not being mounted. I’m using systemd-boot. But when I try to mount it:

# mount -o umask=077 /dev/disk/by-label/BOOT /boot
mount: /boot: unknown filesystem type 'vfat'.
       dmesg(1) may have more information after failed mount system call.

As a final remedy, nixos-install:

# nixos-install -I /etc/nixos/configuration.nix --root /
building the configuration in //etc/nixos/configuration.nix...
/nix/store/5idf6lnw9zh2d1zdv7q620iw9ggr2m8f-nixos-system-bliksem-24.05.4469.6e99f2a27d60
installing the boot loader...
removing group ‘proc’
setting up /etc...
efiSysMountPoint = '/boot' is not a mounted partition. Is the path configured correctly?
Traceback (most recent call last):
  File "/nix/store/lpr9a49mfalfir0za242x57377n2yfzq-17nyliyy0k59a6ky037y6fj902f1s2w5-systemd-boot", line 397, in <module>
    main()
  File "/nix/store/lpr9a49mfalfir0za242x57377n2yfzq-17nyliyy0k59a6ky037y6fj902f1s2w5-systemd-boot", line 377, in main
    subprocess.check_call(CHECK_MOUNTPOINTS)
  File "/nix/store/3wb0055984n2whn449hywsl4ag9gcjir-python3-3.11.9/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '/nix/store/za0y24r2l67222kyz8hi2nzi9qpavmav-check-mountpoints' returned non-zero exit status 1.
umount: ???: umount failed: No such file or directory.

With the absence of /boot, not much is possible. Unsure what to do next!