Cross-platform deployments?

Help
1

I have three machines that I’d like to keep in sync:

  • macOS laptop (main machine, managed via nix-darwin)
  • Beefy NixOS machine (x86_64)
  • Small NixOS machine (x86_64)

I have one flake managing all three machines. In an ideal world any change to my config would:

  • Check that the config works on my laptop first
  • Build and deploy the config to my NixOS machines

If I have to run the deployment from my laptop in order to verify that the nix-darwin config works, that’s fine (and probably more practical).

Does anything like that exist, or do I need to cobble something together?

1 Like
2

My setup is very similar.

I have an aarch64 Mac from which I remotely build several NixOS machines using nix run nixpkgs#nixos-rebuild -- --fast --target-host <user>@<ssh_host> --build-host <user>@<ssh_host> --flake .#<nixosConfiguration> --use-remote-sudo switch

If I understand correctly, when your host machine is a different architecture than your target machine you must include --fast. As per man nixos-rebuild:

       --no-build-nix
               Normally, nixos-rebuild first builds the ‘nixUnstable’ attribute in Nixpkgs, and uses the resulting instance of the Nix package manager to build the new system configuration. This is necessary  if
               the NixOS modules use features not provided by the currently installed version of Nix. This option disables building a new Nix.

       --fast  Equivalent to --no-build-nix. This option is useful if you call nixos-rebuild frequently (e.g. if you’re hacking on a NixOS module).

       --build-host host
               Instead  of  building  the  new  configuration locally, use the specified host to perform the build. The host needs to be accessible with ssh, and must be able to perform Nix builds. If the option
               --target-host is not set, the build will be copied back to the local machine when done.

               Note that, if --no-build-nix is not specified, Nix will be built both locally and remotely. This is because the configuration will always be evaluated locally even though  the  building  might  be
               performed remotely.

If you run without --fast it will attempt to build an incompatible CPU architecture, which will error:

error: a 'x86_64-linux' with features {} is required to build '/nix/store/fvb1k8w9xiil240y5p02x5hzpjqvzyay-nixos-rebuild.drv', but I am a 'aarch64-darwin' with features {apple-virt, benchmark, big-parallel, nixos-test}

Although it is possible to setup cross-compilation, I just haven’t done it yet.

--use-remote-sudo will prompt the sudo password of your user on the remote target. It may close and create new SSH connections, and thus prompt you multiple times.

I’m not sure how you’d check that the config works on your laptop first. When rebuilding fails, the behavior is the same as when you build a local machine; it exits with an error without applying the rebuild.

Although I haven’t pushed the config of my remote machines to my flake repo yet, here is a simple example.

There’s also several community projects like Colmena and NixOps. I haven’t used them yet, but I will give Colmena a try because it supports parallel builds which I’m gonna need to manage a fleet of servers.

3

I forgot about the Linux builder that you can enable with nix-darwin! I even already had it partially enabled lol

This article from @tfc explains the whole deal:

Essentially I would put a script together that:

  • Attempts a build for my laptop
  • Attempts builds for the NixOS machines using a parallel command to do the nixos-rebuild for each machine.
  • If those all succeed, make the switch.

I’ve looked at a handful of pre-built options like the ones you mentioned, but the need to first run a build for nix-darwin complicates things just enough that they lose some of their value.

4

@zmitchell you could also create a nix flake app that is the script, and the benefit of that is that you can pass in nix package deps from your flake and deal with things on a system level too.

You can create a script like run-restore.in.sh and set env vars in that like

PSQL15_BINDIR=@PSQL15_BINDIR@
CURRENT_SYSTEM=@CURRENT_SYSTEM@

and then in your flake app

          pg-restore =
            pkgs.runCommand "run-pg-restore" { } ''
              mkdir -p $out/bin
              substitute ${./nix/tools/run-restore.sh.in} $out/bin/pg-restore \
                --subst-var-by PSQL15_BINDIR '${basePackages.psql_15.bin}' \
                --subst-var-by 'CURRENT_SYSTEM' '${system}'
              chmod +x $out/bin/pg-restore
            '';

and then when you run nix run .#pg-restore nix will dynamically replace those env vars and run that script. You can even pass in the CURRENT_SYSTEM as seen above (aarch64-darwin etc ) and detect that in the script because often you will need to adjust how bash commands run on macos vs nixos etc

example if [[ "$CURRENT_SYSTEM" = "aarch64-darwin" ]]; then etc

5

Yeah that’s more or less what I was planning to do.

1 Like
6

Cool! The darwin.linux-builder works very well too. I deal with a situation where the primary dev platform is macos, but the deploy target is aarch64-linux and so have worked to a support a lot of these things

7

I have a basic writeShellApplication working but I’ll probably switch that to a small Rust application so I can run jobs for all hosts in parallel.

1 Like