Nixos-rebuild in nixos-enter giving a spurious "no such file or directory"

I’m having my first unexpected downtime with NixOS!

I decided to move /nix to its own BTRFS subvolume (which I’ve done on other installations without issue). Of course that didn’t work for some reason; I created the subvolume, rsync -a the store, change the mountpoints in fileSystems to reflect the new subvolume, and after I nixos-rebuild switch I’m greeted with a PAM error: sudo: unable to initialize PAM: Critical error - immediate abort

Unfortunately I was remote at the time, and attempting to log in via SSH just gives me an immediate “connection closed,” so I had to wait to get home to get any further. On arrival home, I can’t even log in via console – typing in my username immediate is greeted with another username prompt.

So I rebooted into an old generation, reviewed everything (which looked OK), and a nixos-rebuild switch and I’m back at the PAM error. Unfortunately, after a round or two of this, I mv /nix{,.bak}; mkdir -p /nix, and now I can’t boot into an old generation (I assume because I’ve broken everything that was in /nix).

So I then boot from a rescue disk, thinking it will be NBD to mount my new @nix subvolume to /nix and rebuild the system…

(using $ instead of # in the prompts below for better markdown syntax highlighting)

[root@rescue:~] $ mount -t btrfs -o compress-force=zstd,subvol=@ /dev/sdb2 /mnt
[root@rescue:~] $ mount -t btrfs -o compress-force=zstd,subvol=@nix /dev/sdb2 /mnt/nix
[root@rescue:~] $ nixos-enter --system /nix/var/nix/profiles/system-159-link 
[agenix] creating new generation in /run/agenix.d/1
[agenix] decrypting secrets...
decrypting '/nix/store/brp8s4gqjs63xg9r82cwqnc86z9zj8pw-blacklight_hashed_password.age' to '/run/agenix.d/1/blacklight_hashed_password'...
decrypting '/nix/store/pmak42xzdcyaq3q3cjlvgqh84y8k341v-builder_id_ed25519.age' to '/run/agenix.d/1/builder_id_ed25519'...
decrypting '/nix/store/p2p5n9jn075zq9mnpm8k65mf3l2n7iki-wireless_env.age' to '/run/agenix.d/1/wireless_env'...
[agenix] symlinking new secrets to /run/agenix (generation 1)...
[agenix] chowning...
setting up /etc...

I notice stuff isn’t all mounted, so I mount -a which seems to work

[root@rescue:~] $ mount -a
[root@rescue:~] $ cd /home/n8henrie/git/nixos
[root@blacklight:/home/n8henrie/git/nixos] $ ls -l
total 40
-rwxr-xr-x 1 n8henrie n8henrie 2276 Feb  1 16:02 build.sh
drwxr-xr-x 1 n8henrie n8henrie  146 Feb 24 15:14 dotfiles
-rw-r--r-- 1 n8henrie n8henrie 7308 Mar 20 02:01 flake.lock
-rw-r--r-- 1 n8henrie n8henrie 4137 Mar 17 08:13 flake.nix
-rw-r--r-- 1 n8henrie n8henrie 3221 Mar 17 08:13 home.nix
-rw-r--r-- 1 n8henrie n8henrie  475 Jan 22 06:19 inputrc.nix
-rw-r--r-- 1 n8henrie n8henrie 1070 Dec 30 12:32 LICENSE
drwxr-xr-x 1 n8henrie n8henrie  106 Mar 20 09:28 modules
-rw-r--r-- 1 n8henrie n8henrie   24 Feb  7 19:48 README.md
drwxr-xr-x 1 n8henrie n8henrie  314 Feb 27 00:51 secrets
drwxr-xr-x 1 n8henrie n8henrie 1408 Mar 17 08:13 services
drwxr-xr-x 1 n8henrie n8henrie   70 Feb  4 13:57 system-configs
drwxr-xr-x 1 n8henrie n8henrie   20 Mar  7 20:53 utils
drwxr-xr-x 1 n8henrie n8henrie  196 Feb  1 16:02 vim
-rw-r--r-- 1 n8henrie n8henrie 3768 Feb  7 19:47 vim.nix

And now for the rebuild

[root@blacklight:/home/n8henrie/git/nixos]$ nixos-rebuild --flake . switch --show-trace
error:
       while fetching the input 'git+file:///home/n8henrie/git/nixos'
       error: getting the HEAD of the Git tree '/home/n8henrie/git/nixos' failed with exit code 128:
       fatal: cannot change to '/home/n8henrie/git/nixos': No such file or directory

Huh?

[root@blacklight:/home/n8henrie/git/nixos]# ls -l
total 40
-rwxr-xr-x 1 n8henrie n8henrie 2276 Feb  1 16:02 build.sh
drwxr-xr-x 1 n8henrie n8henrie  146 Feb 24 15:14 dotfiles
-rw-r--r-- 1 n8henrie n8henrie 7308 Mar 20 02:01 flake.lock
-rw-r--r-- 1 n8henrie n8henrie 4137 Mar 17 08:13 flake.nix
-rw-r--r-- 1 n8henrie n8henrie 3221 Mar 17 08:13 home.nix
-rw-r--r-- 1 n8henrie n8henrie  475 Jan 22 06:19 inputrc.nix
-rw-r--r-- 1 n8henrie n8henrie 1070 Dec 30 12:32 LICENSE
drwxr-xr-x 1 n8henrie n8henrie  106 Mar 20 09:28 modules
-rw-r--r-- 1 n8henrie n8henrie   24 Feb  7 19:48 README.md
drwxr-xr-x 1 n8henrie n8henrie  314 Feb 27 00:51 secrets
drwxr-xr-x 1 n8henrie n8henrie 1408 Mar 17 08:13 services
drwxr-xr-x 1 n8henrie n8henrie   70 Feb  4 13:57 system-configs
drwxr-xr-x 1 n8henrie n8henrie   20 Mar  7 20:53 utils
drwxr-xr-x 1 n8henrie n8henrie  196 Feb  1 16:02 vim
-rw-r--r-- 1 n8henrie n8henrie 3768 Feb  7 19:47 vim.nix
[root@blacklight:/home/n8henrie/git/nixos]$ mount | grep @
/dev/sdb2 on / type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=256,subvol=/@)
/dev/sdb2 on /nix type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=45645,subvol=/@nix)
/dev/sdb2 on /.snapshots type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=261,subvol=/@snapshots)
/dev/sdb2 on /boot type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=258,subvol=/@boot)
/dev/sdb2 on /gnu type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=45646,subvol=/@gnu)
/dev/sdb2 on /home type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=260,subvol=/@home)
/dev/sdc1 on /mnt/n8storage type btrfs (rw,relatime,compress-force=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/@n8storage)
/dev/sdb2 on /var type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=259,subvol=/@var)

Currently running a few checks for nix store and hardware issues…

  • nix-store --verify --check-contents
  • btrfs check /dev/sdb2
  • btrfs scrub start -B /

Any other ideas?

BTRFS scrub and check both pass without errors.

Are you certain that directory is a git repo (in the context of nixos-enter? Does it contain a .git directory? What is the output of git status?

Just documenting for my future reference, I tried mounting everything from outside the chroot (probably an easier way to follow all those symlinks):

$ mount --target-prefix /mnt --fstab "/mnt/$(readlink "/mnt/$(readlink /mnt"$(readlink /mnt/nix/var/nix/profiles/system-159-link)"/etc)/fstab")"

But once I nixos-enter they are unmounted and I have to mount -a regardless. ¯\_(ツ)_/¯

[root@blacklight:/home/n8henrie/git/nixos]$ sudo -u n8henrie bash -c 'git status'
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   flake.lock
        modified:   system-configs/blacklight/hardware-configuration.nix

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        modules/rescue_boot.nix

no changes added to commit (use "git add" and/or "git commit -a")

[root@blacklight:/home/n8henrie/git/nixos]$ sudo -u n8henrie bash -c 'sudo nixos-rebuild --flake /home/n8henrie/git/nixos switch'
error: getting the HEAD of the Git tree '/home/n8henrie/git/nixos' failed with exit code 128:
       fatal: cannot change to '/home/n8henrie/git/nixos': No such file or directory
       
(use '--show-trace' to show detailed location information)

Does show-trace offer any more clues?

If I recall correctly, nix is going to create a store entry for the flake being built, and then build from that. So maybe it is unable to create that store entry. Hopefully show-trace can provide more context about how far it got.

No, I showed the output with --show-trace above; it is the same terse output.

Perhaps of interest, it happens the same if I copy everything to a new directory:

$ cd $(mktemp -d)
$ cp -a ~/git/nixos/. .
$ ls -l
total 40
-rw-r--r-- 1 n8henrie n8henrie 1070 Dec 30 12:32 LICENSE
-rw-r--r-- 1 n8henrie n8henrie   24 Feb  7 19:48 README.md
-rwxr-xr-x 1 n8henrie n8henrie 2276 Feb  1 16:02 build.sh
drwxr-xr-x 1 n8henrie n8henrie  146 Feb 24 15:14 dotfiles
-rw-r--r-- 1 n8henrie n8henrie 7308 Mar 20 02:01 flake.lock
-rw-r--r-- 1 n8henrie n8henrie 4137 Mar 17 08:13 flake.nix
-rw-r--r-- 1 n8henrie n8henrie 3221 Mar 17 08:13 home.nix
-rw-r--r-- 1 n8henrie n8henrie  475 Jan 22 06:19 inputrc.nix
drwxr-xr-x 1 n8henrie n8henrie  106 Mar 20 09:28 modules
drwxr-xr-x 1 n8henrie n8henrie  314 Feb 27 00:51 secrets
drwxr-xr-x 1 n8henrie n8henrie 1408 Mar 17 08:13 services
drwxr-xr-x 1 n8henrie n8henrie   70 Feb  4 13:57 system-configs
drwxr-xr-x 1 n8henrie n8henrie   20 Mar  7 20:53 utils
drwxr-xr-x 1 n8henrie n8henrie  196 Feb  1 16:02 vim
-rw-r--r-- 1 n8henrie n8henrie 3768 Feb  7 19:47 vim.nix
$ sudo nixos-rebuild --flake . switch --show-trace
error:
       … while fetching the input 'git+file:///tmp/tmp.SJjWzsqiyF'

       error: getting the HEAD of the Git tree '/tmp/tmp.SJjWzsqiyF' failed with exit code 128:
       fatal: cannot change to '/tmp/tmp.SJjWzsqiyF': No such file or directory

Could this be from booting with copytoram?

$ cat /proc/cmdline
BOOT_IMAGE=/boot/bzImage findiso= init=/nix/store/fp4a1l2708r7fn0dm0inaq9ri3y4ipaz-nixos-system-rescue-22.11.20230206.13fdd39/init root=LABEL=nixos-22.11-x86_64 boot.shell_on_fail copytoram console=ttyS0,115200 console=tty1 boot.shell_on_fail nohibernate loglevel=4

Well, the problem went away somehow.

I never got nixos-rebuild switch to work from the chroot.

I was able to use the chroot to nix-store --verify --check-contents and make sure the new subvol was mounting properly. Afterwards I rebooted and everything was working again.

It’s great that you no longer are blocked. I’d love to know what had been wrong, but probably it’s not worth more time.

1 Like

Running into this again today, completely different system, also after nixos-enter to try to rescue my system.

Unfortunately my own unresolved thread is my top duck-duck-go result :frowning:

The patch at Nix cannot run other executables in chroot (No such file or directory) · Issue #7602 · NixOS/nix · GitHub allowed nixos-rebuild to run without the “no such file or directory” error, though I got an error about busctl that seems to have prevented it from completing (in spite of a known good config that I can build outside the chroot).

Since I can nix build .#nixosConfigurations.mymachine.config.system.build.toplevel from outside the chroot, I’m sure there’s a way for me to ./result/bin/switch-to-configuration switch and have it take effect on /mnt/nix/... instead of /nix/..., but I’ll have to figure that out another day.

1 Like

This bit me as well today also in the middle of a recovery. It is very confusing since this seems the only post on the internet which discusses the issue.

I simply wanted to re-install my bootloader (since I set my configurationLimit too low during experimentation and bricked my install).

But I am completely unable to do that simple task, 1000 little papercuts and a few hours and I can’t get it recovered. Between private flake inputs, nixos-enter / rebuild issues, lack of space on my recovery iso, I give up completely… I’ve been using flakes for 3 years now and I am completely stuck on how do even recover this properly (and ive done a few recoveries before).

If I had known this would be such a pain I would have saved myself the stress and time and just instantly blown away the previous install… The worst part is that the the working generation is still sitting there in /nix/var/nix/profiles, but I simply cant figure out how to rebuild the bootloader simply (without rebuild/install).

Sorry you’re having trouble.

As this commit is merged into nix master and 2.17.0, I think (?) you should be able to use nix shell to enter an environment with a newer nix as a workaround, something like:

$ nix run github:nixos/nix -- --version
nix (Nix) 2.18.0pre20230814_5542c1f
$ nix shell github:nixos/nix
$ sudo nixos-enter

Or you can manually apply the patch from the issue I linked; if memory serves, I very sloppily did something like cp "$(readlink -f $(type -p nixos-enter))" . && chmod +w ./nixos-enter and then manually edited the file.

2 Likes

I’ll give this go the next time I brick my bootloader, thanks.

This post came up when I was trying to restore the bootloader from a live ISO.

I had thought that from nixos-enter, I could treat the shell as if I was in that NixOS system, & so re-run nixos-rebuild; but, I ran into the error message that comes up in this thread. (The nixos system inside the chroot had an older version of nix).

Coming across another post here helped, since it pointed to much more straightforward instructions in the NixOS wiki for what I was trying to do: Bootloader - NixOS Wiki

NIXOS_INSTALL_BOOTLOADER=1 /nix/var/nix/profiles/system/bin/switch-to-configuration boot