Substituters on a remote builder for a flake causes build to fail with "no such file"

Help
1

Hi all!
I’m trying to build a flake w/ a remote builder, while having the remote builder pull from a binary cache.
When i run my nix build command, i get the following error.

Any thoughts ? Someone mentioned that Yep, the issue only happens if you attempt to copy a fixed output derivation from a substituter into a chroot store. It's kind of rare but very unpleasant for people who try to do this thing that should work during installation. here nixos-install fails with getting attributes of path /nix/store/...: No such file or directory · Issue #126141 · NixOS/nixpkgs · GitHub - is that what’s happening here?

To be specific:
I run nix run nixpkgs#darwin.linux-builder and leave it open.
I configure nix.conf on the host system (an M1 Mac, darwin-aarch64) to use the remote builder i just started, and to consider my user a trusted substituter, etc.

Then, I attempt to build a linux-aarch64 derivation like this:
nix build 'github:arilotter/dotfiles#nixosConfigurations.kronos.config.system.build.sdImage'
This derivation is supposed to build a linux-aarch64 SD card image that I can boot on a linux-aarch64 system. It uses a remote substituter so I don’t have to build the entire kernel.
I get no problems or warnings about substituters at all after the initial allowance.

This is the error I hit:

error: build of '/nix/store/an92k3fp3g3spzmq6k26fwfywpp09sjd-X-Restart-Triggers.drv' on 'ssh://builder@linux-builder' failed: error: getting status of '/nix/store/7yb9ll5s5byybq4mrgwss07yslvbqpgj-openssh-9.4p1': No such file or directory
error: builder for '/nix/store/an92k3fp3g3spzmq6k26fwfywpp09sjd-X-Restart-Triggers.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/3n6mn6m0kli3df57qqqwdggq72h6qz6j-unit-sshd.service.drv' failed to build
error: 1 dependencies of derivation '/nix/store/qd7nlrdyhp4w9gz8895220m4gpy1a2m8-system-units.drv' failed to build
error: 1 dependencies of derivation '/nix/store/s4mw8xsy6p6nm5sji2scha7b8hm7fm83-etc.drv' failed to build
error: 1 dependencies of derivation '/nix/store/9lfp5xcfdy57yd6da0zh5m05y13z8a9y-nixos-system-kronos-23.11.20230819.d680ded.drv' failed to build
error: 1 dependencies of derivation '/nix/store/85rgnj02i5153pjcfsyfasaimc0cwds7-ext4-fs.img.zst.drv' failed to build
error: 1 dependencies of derivation '/nix/store/0dpxhwl3y1gi4vcbq48khcbfpyfnd333-nixos-sd-image-23.11.20230819.d680ded-aarch64-linux.img.drv' failed to build

Any thoughts?

1 Like
2

I’ve been having a similar problem, also on an aarch64 Mac and trying to use nixpkgs#darwin.linux-builder.

In my case, the problem seems to have been that I was doing nix run nixpkgs#darwin.linux-builder from a dir with a flake.lock, and it used the wrong version of nixpkgs. At least, when I ran rm nixos.qcow2; cd /tmp; nix run nixpkgs#darwin.linux-builder, two things happened:

  • It asked me to run sudo again because the value of the /etc/nix/builder_ed25519 key valid for the VM changed.
  • The buid started to work.

For reference, the error I got when I was trying to build with that remote builder was:

error: build of '/nix/store/674rp75b3w61ki5jfdx7rmddv4cda5cj-users-groups.json.drv' on 'ssh-ng://builder@linux-builder' failed: error:
              … while setting up the build environment

              error: getting attributes of path '/nix/store/0ga1cm2ild3sv9vg64ldizrdpfr72pvv-xgcc-14.3.0-libgcc': No such file or directory
could not copy /nix/store/amxczw58sdg69cgr6wvj45p1x7qrl3cq-etc-fstab: error: Nix daemon disconnected unexpectedly (maybe it crashed?)
could not copy /nix/store/gc7wb4llaj0da6a5b30wivm6bc8xs6pc-etc-nix-registry.json: error: Nix daemon disconnected unexpectedly (maybe it crashed?)
could not copy /nix/store/drz9kmqwnn8n0dwjlc2jhx0gfq1vaddc-etc-os-release: error: Nix daemon disconnected unexpectedly (maybe it crashed?)
could not copy /nix/store/nfs8dqhb0b5dg9l12ip958hgmdblqqks-etc-pam-environment: error: Nix daemon disconnected unexpectedly (maybe it crashed?)
could not copy /nix/store/cqwbm9swvws2f2yjlrn77sqsyjcf95bx-etc-sysctl.d-60-nixos.conf: error: Nix daemon disconnected unexpectedly (maybe it crashed?)
could not copy /nix/store/3mjb46r9qnm40hn1kbkky5qv9f3n2ihf-issue: error: Nix daemon disconnected unexpectedly (maybe it crashed?)
could not copy /nix/store/nzifpb1z88d1rjwk7lf74i2lj80vs78l-nixos-version: error: Nix daemon disconnected unexpectedly (maybe it crashed?)
error: some outputs are unexpectedly invalid
3

@winter (if I got the handle correct, there seems to be more than one winter) told me on a Discord (the Tangled discord) that my diagnosis (nix run using the flake.lock in the dir) is likely wrong. Still, the workaround or running nix run in another dir did work for me. Any explanation for what actually could be going on would be welcome.