Nixos-rebuild in nixos-enter giving a spurious "no such file or directory"

Help
1

I’m having my first unexpected downtime with NixOS!

I decided to move /nix to its own BTRFS subvolume (which I’ve done on other installations without issue). Of course that didn’t work for some reason; I created the subvolume, rsync -a the store, change the mountpoints in fileSystems to reflect the new subvolume, and after I nixos-rebuild switch I’m greeted with a PAM error: sudo: unable to initialize PAM: Critical error - immediate abort

Unfortunately I was remote at the time, and attempting to log in via SSH just gives me an immediate “connection closed,” so I had to wait to get home to get any further. On arrival home, I can’t even log in via console – typing in my username immediate is greeted with another username prompt.

So I rebooted into an old generation, reviewed everything (which looked OK), and a nixos-rebuild switch and I’m back at the PAM error. Unfortunately, after a round or two of this, I mv /nix{,.bak}; mkdir -p /nix, and now I can’t boot into an old generation (I assume because I’ve broken everything that was in /nix).

So I then boot from a rescue disk, thinking it will be NBD to mount my new @nix subvolume to /nix and rebuild the system…

(using $ instead of # in the prompts below for better markdown syntax highlighting)

[root@rescue:~] $ mount -t btrfs -o compress-force=zstd,subvol=@ /dev/sdb2 /mnt
[root@rescue:~] $ mount -t btrfs -o compress-force=zstd,subvol=@nix /dev/sdb2 /mnt/nix
[root@rescue:~] $ nixos-enter --system /nix/var/nix/profiles/system-159-link 
[agenix] creating new generation in /run/agenix.d/1
[agenix] decrypting secrets...
decrypting '/nix/store/brp8s4gqjs63xg9r82cwqnc86z9zj8pw-blacklight_hashed_password.age' to '/run/agenix.d/1/blacklight_hashed_password'...
decrypting '/nix/store/pmak42xzdcyaq3q3cjlvgqh84y8k341v-builder_id_ed25519.age' to '/run/agenix.d/1/builder_id_ed25519'...
decrypting '/nix/store/p2p5n9jn075zq9mnpm8k65mf3l2n7iki-wireless_env.age' to '/run/agenix.d/1/wireless_env'...
[agenix] symlinking new secrets to /run/agenix (generation 1)...
[agenix] chowning...
setting up /etc...

I notice stuff isn’t all mounted, so I mount -a which seems to work

[root@rescue:~] $ mount -a
[root@rescue:~] $ cd /home/n8henrie/git/nixos
[root@blacklight:/home/n8henrie/git/nixos] $ ls -l
total 40
-rwxr-xr-x 1 n8henrie n8henrie 2276 Feb  1 16:02 build.sh
drwxr-xr-x 1 n8henrie n8henrie  146 Feb 24 15:14 dotfiles
-rw-r--r-- 1 n8henrie n8henrie 7308 Mar 20 02:01 flake.lock
-rw-r--r-- 1 n8henrie n8henrie 4137 Mar 17 08:13 flake.nix
-rw-r--r-- 1 n8henrie n8henrie 3221 Mar 17 08:13 home.nix
-rw-r--r-- 1 n8henrie n8henrie  475 Jan 22 06:19 inputrc.nix
-rw-r--r-- 1 n8henrie n8henrie 1070 Dec 30 12:32 LICENSE
drwxr-xr-x 1 n8henrie n8henrie  106 Mar 20 09:28 modules
-rw-r--r-- 1 n8henrie n8henrie   24 Feb  7 19:48 README.md
drwxr-xr-x 1 n8henrie n8henrie  314 Feb 27 00:51 secrets
drwxr-xr-x 1 n8henrie n8henrie 1408 Mar 17 08:13 services
drwxr-xr-x 1 n8henrie n8henrie   70 Feb  4 13:57 system-configs
drwxr-xr-x 1 n8henrie n8henrie   20 Mar  7 20:53 utils
drwxr-xr-x 1 n8henrie n8henrie  196 Feb  1 16:02 vim
-rw-r--r-- 1 n8henrie n8henrie 3768 Feb  7 19:47 vim.nix

And now for the rebuild

[root@blacklight:/home/n8henrie/git/nixos]$ nixos-rebuild --flake . switch --show-trace
error:
       while fetching the input 'git+file:///home/n8henrie/git/nixos'
       error: getting the HEAD of the Git tree '/home/n8henrie/git/nixos' failed with exit code 128:
       fatal: cannot change to '/home/n8henrie/git/nixos': No such file or directory

Huh?

[root@blacklight:/home/n8henrie/git/nixos]# ls -l
total 40
-rwxr-xr-x 1 n8henrie n8henrie 2276 Feb  1 16:02 build.sh
drwxr-xr-x 1 n8henrie n8henrie  146 Feb 24 15:14 dotfiles
-rw-r--r-- 1 n8henrie n8henrie 7308 Mar 20 02:01 flake.lock
-rw-r--r-- 1 n8henrie n8henrie 4137 Mar 17 08:13 flake.nix
-rw-r--r-- 1 n8henrie n8henrie 3221 Mar 17 08:13 home.nix
-rw-r--r-- 1 n8henrie n8henrie  475 Jan 22 06:19 inputrc.nix
-rw-r--r-- 1 n8henrie n8henrie 1070 Dec 30 12:32 LICENSE
drwxr-xr-x 1 n8henrie n8henrie  106 Mar 20 09:28 modules
-rw-r--r-- 1 n8henrie n8henrie   24 Feb  7 19:48 README.md
drwxr-xr-x 1 n8henrie n8henrie  314 Feb 27 00:51 secrets
drwxr-xr-x 1 n8henrie n8henrie 1408 Mar 17 08:13 services
drwxr-xr-x 1 n8henrie n8henrie   70 Feb  4 13:57 system-configs
drwxr-xr-x 1 n8henrie n8henrie   20 Mar  7 20:53 utils
drwxr-xr-x 1 n8henrie n8henrie  196 Feb  1 16:02 vim
-rw-r--r-- 1 n8henrie n8henrie 3768 Feb  7 19:47 vim.nix
[root@blacklight:/home/n8henrie/git/nixos]$ mount | grep @
/dev/sdb2 on / type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=256,subvol=/@)
/dev/sdb2 on /nix type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=45645,subvol=/@nix)
/dev/sdb2 on /.snapshots type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=261,subvol=/@snapshots)
/dev/sdb2 on /boot type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=258,subvol=/@boot)
/dev/sdb2 on /gnu type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=45646,subvol=/@gnu)
/dev/sdb2 on /home type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=260,subvol=/@home)
/dev/sdc1 on /mnt/n8storage type btrfs (rw,relatime,compress-force=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/@n8storage)
/dev/sdb2 on /var type btrfs (rw,relatime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=259,subvol=/@var)

Currently running a few checks for nix store and hardware issues…

  • nix-store --verify --check-contents
  • btrfs check /dev/sdb2
  • btrfs scrub start -B /

Any other ideas?

2

BTRFS scrub and check both pass without errors.

3

Are you certain that directory is a git repo (in the context of nixos-enter? Does it contain a .git directory? What is the output of git status?

4

Just documenting for my future reference, I tried mounting everything from outside the chroot (probably an easier way to follow all those symlinks):

$ mount --target-prefix /mnt --fstab "/mnt/$(readlink "/mnt/$(readlink /mnt"$(readlink /mnt/nix/var/nix/profiles/system-159-link)"/etc)/fstab")"

But once I nixos-enter they are unmounted and I have to mount -a regardless. ¯\_(ツ)_/¯

[root@blacklight:/home/n8henrie/git/nixos]$ sudo -u n8henrie bash -c 'git status'
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   flake.lock
        modified:   system-configs/blacklight/hardware-configuration.nix

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        modules/rescue_boot.nix

no changes added to commit (use "git add" and/or "git commit -a")

[root@blacklight:/home/n8henrie/git/nixos]$ sudo -u n8henrie bash -c 'sudo nixos-rebuild --flake /home/n8henrie/git/nixos switch'
error: getting the HEAD of the Git tree '/home/n8henrie/git/nixos' failed with exit code 128:
       fatal: cannot change to '/home/n8henrie/git/nixos': No such file or directory
       
(use '--show-trace' to show detailed location information)
5

Does show-trace offer any more clues?

If I recall correctly, nix is going to create a store entry for the flake being built, and then build from that. So maybe it is unable to create that store entry. Hopefully show-trace can provide more context about how far it got.

6

No, I showed the output with --show-trace above; it is the same terse output.

7

Perhaps of interest, it happens the same if I copy everything to a new directory:

$ cd $(mktemp -d)
$ cp -a ~/git/nixos/. .
$ ls -l
total 40
-rw-r--r-- 1 n8henrie n8henrie 1070 Dec 30 12:32 LICENSE
-rw-r--r-- 1 n8henrie n8henrie   24 Feb  7 19:48 README.md
-rwxr-xr-x 1 n8henrie n8henrie 2276 Feb  1 16:02 build.sh
drwxr-xr-x 1 n8henrie n8henrie  146 Feb 24 15:14 dotfiles
-rw-r--r-- 1 n8henrie n8henrie 7308 Mar 20 02:01 flake.lock
-rw-r--r-- 1 n8henrie n8henrie 4137 Mar 17 08:13 flake.nix
-rw-r--r-- 1 n8henrie n8henrie 3221 Mar 17 08:13 home.nix
-rw-r--r-- 1 n8henrie n8henrie  475 Jan 22 06:19 inputrc.nix
drwxr-xr-x 1 n8henrie n8henrie  106 Mar 20 09:28 modules
drwxr-xr-x 1 n8henrie n8henrie  314 Feb 27 00:51 secrets
drwxr-xr-x 1 n8henrie n8henrie 1408 Mar 17 08:13 services
drwxr-xr-x 1 n8henrie n8henrie   70 Feb  4 13:57 system-configs
drwxr-xr-x 1 n8henrie n8henrie   20 Mar  7 20:53 utils
drwxr-xr-x 1 n8henrie n8henrie  196 Feb  1 16:02 vim
-rw-r--r-- 1 n8henrie n8henrie 3768 Feb  7 19:47 vim.nix
$ sudo nixos-rebuild --flake . switch --show-trace
error:
       … while fetching the input 'git+file:///tmp/tmp.SJjWzsqiyF'

       error: getting the HEAD of the Git tree '/tmp/tmp.SJjWzsqiyF' failed with exit code 128:
       fatal: cannot change to '/tmp/tmp.SJjWzsqiyF': No such file or directory

Could this be from booting with copytoram?

$ cat /proc/cmdline
BOOT_IMAGE=/boot/bzImage findiso= init=/nix/store/fp4a1l2708r7fn0dm0inaq9ri3y4ipaz-nixos-system-rescue-22.11.20230206.13fdd39/init root=LABEL=nixos-22.11-x86_64 boot.shell_on_fail copytoram console=ttyS0,115200 console=tty1 boot.shell_on_fail nohibernate loglevel=4
8

Well, the problem went away somehow.

I never got nixos-rebuild switch to work from the chroot.

I was able to use the chroot to nix-store --verify --check-contents and make sure the new subvol was mounting properly. Afterwards I rebooted and everything was working again.

9

It’s great that you no longer are blocked. I’d love to know what had been wrong, but probably it’s not worth more time.

1 Like
10

Running into this again today, completely different system, also after nixos-enter to try to rescue my system.

Unfortunately my own unresolved thread is my top duck-duck-go result :frowning:

11

The patch at Nix cannot run other executables in chroot (No such file or directory) · Issue #7602 · NixOS/nix · GitHub allowed nixos-rebuild to run without the “no such file or directory” error, though I got an error about busctl that seems to have prevented it from completing (in spite of a known good config that I can build outside the chroot).

Since I can nix build .#nixosConfigurations.mymachine.config.system.build.toplevel from outside the chroot, I’m sure there’s a way for me to ./result/bin/switch-to-configuration switch and have it take effect on /mnt/nix/... instead of /nix/..., but I’ll have to figure that out another day.

1 Like
12

This bit me as well today also in the middle of a recovery. It is very confusing since this seems the only post on the internet which discusses the issue.

I simply wanted to re-install my bootloader (since I set my configurationLimit too low during experimentation and bricked my install).

But I am completely unable to do that simple task, 1000 little papercuts and a few hours and I can’t get it recovered. Between private flake inputs, nixos-enter / rebuild issues, lack of space on my recovery iso, I give up completely… I’ve been using flakes for 3 years now and I am completely stuck on how do even recover this properly (and ive done a few recoveries before).

If I had known this would be such a pain I would have saved myself the stress and time and just instantly blown away the previous install… The worst part is that the the working generation is still sitting there in /nix/var/nix/profiles, but I simply cant figure out how to rebuild the bootloader simply (without rebuild/install).

13

Sorry you’re having trouble.

As this commit is merged into nix master and 2.17.0, I think (?) you should be able to use nix shell to enter an environment with a newer nix as a workaround, something like:

$ nix run github:nixos/nix -- --version
nix (Nix) 2.18.0pre20230814_5542c1f
$ nix shell github:nixos/nix
$ sudo nixos-enter

Or you can manually apply the patch from the issue I linked; if memory serves, I very sloppily did something like cp "$(readlink -f $(type -p nixos-enter))" . && chmod +w ./nixos-enter and then manually edited the file.

2 Likes
14

I’ll give this go the next time I brick my bootloader, thanks.