I’m using nixos-rebuild to manage a remote machine, like this:
NIX_SSHOPTS='-o ForwardAgent=yes' nixos-rebuild switch --use-remote-sudo --flake /etc/nixos#edamame --target-host edamame --build-host edamame
This worked up until as a week ago. Trying to rebuild the same configuration. Today it fails with the error:
error: executing '/nix/store/2icvzb9pj2ryvsy1m9jf4kj2mbkgbczj-bash-5.2p26/bin/bash': No such file or directory
The error doesn’t seem to be related to nixos-rebuild:
nix build --no-link --print-build-logs --print-out-paths /etc/nixos#nixosConfigurations.edamame.config.system.build.toplevel --show-trace
NetworkManager.conf> error: executing '/nix/store/2icvzb9pj2ryvsy1m9jf4kj2mbkgbczj-bash-5.2p26/bin/bash': No such file or directory
extlinux-conf-builder.sh> error: executing '/nix/store/2icvzb9pj2ryvsy1m9jf4kj2mbkgbczj-bash-5.2p26/bin/bash': No such file or directory
mounts.sh> error: executing '/nix/store/2icvzb9pj2ryvsy1m9jf4kj2mbkgbczj-bash-5.2p26/bin/bash': No such file or directory
users-groups.json> error: executing '/nix/store/2icvzb9pj2ryvsy1m9jf4kj2mbkgbczj-bash-5.2p26/bin/bash': No such file or directory
builder.pl> error: executing '/nix/store/2icvzb9pj2ryvsy1m9jf4kj2mbkgbczj-bash-5.2p26/bin/bash': No such file or directory
chfn.pam> error: executing '/nix/store/2icvzb9pj2ryvsy1m9jf4kj2mbkgbczj-bash-5.2p26/bin/bash': No such file or directory
chpasswd.pam> error: executing '/nix/store/2icvzb9pj2ryvsy1m9jf4kj2mbkgbczj-bash-5.2p26/bin/bash': No such file or directory
chsh.pam> error: executing '/nix/store/2icvzb9pj2ryvsy1m9jf4kj2mbkgbczj-bash-5.2p26/bin/bash': No such file or directory
error: builder for '/nix/store/s1zks8wwh2lr8vvsxsqdv3qz3s0y8aan-NetworkManager.conf.drv' failed with exit code 1;
last 1 log lines:
> error: executing '/nix/store/2icvzb9pj2ryvsy1m9jf4kj2mbkgbczj-bash-5.2p26/bin/bash': No such file or directory
For full logs, run 'nix log /nix/store/s1zks8wwh2lr8vvsxsqdv3qz3s0y8aan-NetworkManager.conf.drv'.
error: builder for '/nix/store/vnkyxm4prjdrvmy3wwhwxr8ghv4dj2s2-builder.pl.drv' failed with exit code 1;
last 1 log lines:
> error: executing '/nix/store/2icvzb9pj2ryvsy1m9jf4kj2mbkgbczj-bash-5.2p26/bin/bash': No such file or directory
For full logs, run 'nix log /nix/store/vnkyxm4prjdrvmy3wwhwxr8ghv4dj2s2-builder.pl.drv'.
error: builder for '/nix/store/rsrmiyjlsrhbrhnw0mqwj1bd32iykf7h-extlinux-conf-builder.sh.drv' failed with exit code 1;
last 1 log lines:
> error: executing '/nix/store/2icvzb9pj2ryvsy1m9jf4kj2mbkgbczj-bash-5.2p26/bin/bash': No such file or directory
For full logs, run 'nix log /nix/store/rsrmiyjlsrhbrhnw0mqwj1bd32iykf7h-extlinux-conf-builder.sh.drv'.
error: builder for '/nix/store/d0y66n7f7n28yh89i021apr06dgr3qap-mounts.sh.drv' failed with exit code 1;
last 1 log lines:
> error: executing '/nix/store/2icvzb9pj2ryvsy1m9jf4kj2mbkgbczj-bash-5.2p26/bin/bash': No such file or directory
For full logs, run 'nix log /nix/store/d0y66n7f7n28yh89i021apr06dgr3qap-mounts.sh.drv'.
error: 1 dependencies of derivation '/nix/store/x6gj02i723iql11p67mms1jxy4fcf3fa-nixos-system-edamame-24.05.20240306.9df3e30.drv' failed to build
The nix log command doesn’t show much, just a one line log with the same message that the bash binary does not exist. The path exists on the host system, I don’t know why it can’t be found visible during nix build.
I follow nixos-unstable and nixpkgs is pinned to github:NixOS/nixpkgs/612f97239e2cc474c13c9dafa0df378058c5ad8d. Updating to github:NixOS/nixpkgs/9df3e30ce24fd28c7b3e2de0d986769db5d6225 doesn’t change the build error. I can rebuild the host machine from the same commit.
What’s more, copying the flake to the target machine and running nixos-rebuild there works as well.
I’m not sure where to look next, or how to figure out what exactly is failing or why. Any ideas? I’m happy to provide more information.