Finally got around to doing this somewhat properly. This was exhaustingly complicated, because there are so many moving parts in nixos-rebuild. Here is a run-down.
The basic idea is to evaluate the target configuration on the local Darwin machine, and then push the derivations to be built on the target machine. For this we pass --argstr system x86_64-linux to all the local runs of nix-instantiate. The rest was figuring out details, such as how to inject that argument into nixos-rebuild in the first place, how not let it go into commands which don’t understand it, how to manage which version of nix is used where, since there are multiple possibilities.
As a first step, the actual package is just a reproduction of what we find in <nixpkgs/nixos>/modules/installer/tools/tools.nix, but obviously with the nix package set to pkgs.nix instead of config.system.nix.package.
The fun starts when nixos-rebuild re-executes the target system’s version of itself. Originally this amounts to whatever is defined by the configuration, but that will be run on Darwin now and we need to tell it to evaluate expressions for Linux explicitly.
nixos-rebuild.nix:
{ pkgs, lib, ... }:
let
path = pkgs.path + /nixos/modules/installer/tools;
fallback = import (path + /nix-fallback-paths.nix);
in
(pkgs.substituteAll {
dir = "bin";
isExecutable = true;
name = "nixos-rebuild";
src = path + /nixos-rebuild.sh;
inherit (pkgs) runtimeShell;
# here is the only difference to the original definition
nix = pkgs.nix.out;
nix_x86_64_linux = fallback.x86_64-linux;
nix_i686_linux = fallback.i686-linux;
path = lib.makeBinPath [ pkgs.jq ];
}).overrideAttrs (old: {
postInstall = ''
# use a patched version of the target configuration's `nixos-rebuild`
# that will work from Darwin
substituteInPlace $out/bin/nixos-rebuild \
--replace "--expr 'with import <nixpkgs/nixos> {}; config.system.build.nixos-rebuild'" "${./nixos-rebuild-patch.nix}"
'';
})
Most importantly we just inject the required arguments to nix right into the extraBuildFlags variable, as we’d need to patch both the initial as well as the target version if we wanted them to support command line parameters. Unfortunately this has consequences:
- We can’t continue using the Linux
nix built for building the target configuration on Darwin, so we have to omit it being added to $PATH.
- We have to filter out the newly added arguments again, because for remote building they are passed verbatim to
nix-store, which does not take --argstr. While there is already a filtering step happening, it lets everything through, as opposed to the top-level command line parsing, which throws an error on unknown parameters.
nixos-rebuild-patch.nix:
with import <nixpkgs/nixos> {};
config.system.build.nixos-rebuild.overrideAttrs (old: {
postInstall = ''
# 1. evaluate for Linux to avoid being stopped by assertions.
# see: <https://github.com/NixOS/nixops/issues/1033>
# 2. do not put the newly built Linux `nix` into `$PATH`, since we're on
# Darwin and are already using the target configuration's version.
substituteInPlace $out/bin/nixos-rebuild \
--replace "extraBuildFlags=()" "extraBuildFlags=(--argstr system x86_64-linux)" \
--replace 'PATH="$tmpDir/nix/bin:$PATH"' ""
# filter `--argstr` parameters back out before passing to remote `nix-store`
pattern="# We don't want this in buildArgs"
# WTF `sed`?! line break is mandatory for `i` command to work
sed -i "0,/$pattern/{/$pattern/i --argstr) shift 2;;
}" $out/bin/nixos-rebuild
'';
}
If you add the package to your environment, you can now run from Darwin:
$ nixos-rebuild --target-host <target> switch
Don’t forget to specify <nixpkgs> and <nixos-config> with environment variables or -I to have predictable results!
Note that for this to Just Work™ on targets with NixOS >=18.09 you need root access to your target host over ssh. There is also an option to run as sudo, which was introduced in 20.03.
Edit: It works. But that won’t work either, because remote $PATH is set to local $PATH, such that essentially only nix will be available, but definitely not sudo. This may be an actual bug, and I submitted a pull request to change the offending line to env PATH="$remoteNix:"'$PATH'.
Hope this helps!