Building a flake-based NixOS system remotely

Help
1

I’ve been using nixos-rebuild to deploy a remote flake-based NixOS system, and loving the simplicity.

The only thing I am not able to do is to build the system remotely. This is the invocation:

$ nixos-rebuild switch --target-host monitor --build-host monitor --flake .#monitor
don't know how to build these paths:
  /nix/store/1km1lnysmjmbmad94l6bhm8v6x9xr9bm-nixos-system-monitor-20.09.20210123.a058d00
error: path '/nix/store/1km1lnysmjmbmad94l6bhm8v6x9xr9bm-nixos-system-monitor-20.09.20210123.a058d00' does not exist and cannot be created

If I use --build-host localhost instead, everything works, but then I have to push big packages around.

IIUC, this is not a use-case covered by nix.distributedBuilds: I’m not trying to delegate building local packages to more powertful machines.

Can anyone elucidate me on the error above?

2 Likes
2

I’ve run into this as well. It appears to be a known issue; the nix wiki has the following note:

Warning: Remote building seems to be broken at the moment, which is why the build host is set to “localhost”.

2 Likes
3

I did some investigating. The reason for the failure is that the derivation is not copied to the remote host before the build is started. That’s what don't know how to build these paths means. So, how do we fix this? This is the best flow I could find so far:

$ drv=$(nix eval .#nixosConfigurations.$hostname.config.system.build.toplevel.drvPath | tr -d '"')
$ nix copy --derivation -s --to "ssh://$buildHost" "$drv"
$ ssh $buildHost nix build --no-link "$drv"
$ nixos-rebuild --flake .#$hostname --target-host $targetHost switch

I’ll try to make a nixpkgs PR in the next few days to make sure nixos-rebuild does this (or at least get the ball rolling on a fix in nixos-rebuild).

Note that the snippet above assumes that the build host and the target host are the same, if this is not the same an extra nix copy will have to be added after the third command.

5 Likes
4

Thanks! If you do end up creating a PR, please reference this issue! :heart:

5

I created a pull request here: nixos-rebuild: Allow remote building when using flakes by chvp · Pull Request #119540 · NixOS/nixpkgs · GitHub

3 Likes
6

Out of curiosity, will the PR work for cross compiling as well? I’m trying to figure out a workflow for an upcoming personal project like this: from my amd64 laptop, trigger a build of a full NixOS (flake) system for a raspberry pi on the local network, where the actual compilation happens on another amd64 server and the raspberry pi switches to the new configuration. Not sure if I’m going to need something like nixops for that last part.

1 Like
7

I’m not entirely sure, to be honest. I haven’t yet done any cross-compilation with nix. I would assume that if you followed the steps here, it should work.

1 Like
8

From my experience trying this today it appears that running --target-host from an x86 machine against aarch64 will build the x86 variant on the device then throw the
error: a 'x86_64-linux' with features {} is required to build ... but I am a 'aarch64-linux'.
Presumably because it picks up system from where nixos-rebuild was invokved rather than the target host.

Not sure if you’ve figured out a workaround or if I might have found a bug here.

Edit: Raised nixos-rebuild --target-host will inherit system from invocation and not target host (flake) · Issue #134952 · NixOS/nixpkgs · GitHub

2 Likes