Nixpkgs support for Linux builders running on macOS

I recently upstreamed into Nixpkgs a convenient darwin.builder script for launching a Linux builder on macOS without requiring access to an existing Linux builder. For more details, see the following blog post, which explains the background and motivation for this change:

12 Likes

Cool! I just integrated this in our internal nix-darwin configuration and so far it seems to work smoothely! No manual intervention required: The VM is run using a nix-darwin managed launchd daemon.

We need SSH enabled on the host machines though, so running the VM on port 22 isn’t possible.

My workaround was to use a custom SSH config where the port can be changed. Here is my nix-darwin module:

{ config, pkgs, lib, ... }:
let
  dataDir = "/var/lib/nixos-builder";
  linuxSystem = builtins.replaceStrings [ "darwin" ] [ "linux" ]
    pkgs.stdenv.hostPlatform.system;
  outerPkgs = pkgs;
  port = 31022;

  # Using `nixpkgs-unstable` here as `darwin.builder` is a relatively new feature. We want the latest updates.
  linuxBuilder = (import "${pkgs.nixpkgs-unstable.path}/nixos" {
    system = linuxSystem;
    configuration = ({ modulesPath, lib, ... }: {
      imports = [ "${modulesPath}/profiles/macos-builder.nix" ];
      virtualisation.host.pkgs = outerPkgs;
      virtualisation.forwardPorts = lib.mkForce [{
        from = "host";
        host.address = "127.0.0.1";
        host.port = port;
        guest.port = 22;
      }];
    });
  }).config.system.build.macos-builder-installer;

  runLinuxBuilder = pkgs.writeShellScriptBin "run-linux-builder" ''
    set -uo pipefail
    trap 's=$?; echo "$0: Error on line "$LINENO": $BASH_COMMAND"; exit $s' ERR
    IFS=$'\n\t'
    mkdir -p "${dataDir}"
    cd "${dataDir}"
    ${linuxBuilder}/bin/create-builder
  '';
in {
  config = {
    #
    environment.systemPackages = [ runLinuxBuilder ];

    # Enable remote builds
    nix.distributedBuilds = true;

    nix.buildMachines = [{
      hostName = "ssh://linux-builder";
      system = linuxSystem;
      maxJobs = 4;
      # This is cheating: KVM isn't actually available (?) but QEMU falls back to "slow mode" in this case
      supportedFeatures = [ "kvm" ];
    }];

    environment.etc."nix/ssh_known_hosts".text = ''
      [127.0.0.1]:31022 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJBWcxb/Blaqt1auOtE+F8QUWrUotiC5qBJ+UuEWdVCb
    '';

    # We can't/want to edit /var/root/.ssh/config so instead we create the config at another location and tell ssh to use that instead by modifying NIX_SSHOPTS
    environment.etc."nix/ssh_config".text = ''
      Host linux-builder
        User builder
        HostName 127.0.0.1
        Port ${toString port}
        IdentityFile ${dataDir}/keys/builder_ed25519
        UserKnownHostsFile /etc/nix/ssh_known_hosts
    '';

    # Tell nix-daemon to use our custom SSH config
    nix.envVars = { NIX_SSHOPTS = "-F /etc/nix/ssh_config"; };

    launchd.daemons.linux-builder = {
      command = "${runLinuxBuilder}/bin/run-linux-builder";
      path = with pkgs; [ "/usr/bin" coreutils nix ];
      serviceConfig = {
        KeepAlive = true;
        RunAtLoad = true;
        StandardOutPath = "/var/log/linux-builder.log";
        StandardErrorPath = "/var/log/linux-builder.log";
      };
    };
  };
}
1 Like

Another thing: Would it make sense if Hydra also built a disk image with boot.binfmt.emulatedSystems = [ "x86_64-linux"];? That would be cool for the new Macs. My current workaround was scripting manually: First start normal VM from cache.nixos.org, then with initial VM build a modified image with the emulatedSystems modified.

Hmm, maybe a more general approach would be to make the VM sort of “self-updating”. Maybe allow using sharedDirectories for mounting a Flake which would contain a NixOS configuration, which would be activated on the first boot of the VM? This way it would be easy for users of darwin.builder to customize the VM slightly without lots of scripting.

1 Like

This is a game changer for me - no more fiddling with nix docker images and getting disappointed when things break!
I agree with Felix that some additional configuration options, such as specifying the port, the image used, etc., would make it even better. I’d also like to see some docs on how to enable it easily (and extensibly - e.g. appending the builder to the buildMachines if the list is already defined elsewhere in another module, etc.) in a nix os configuration (both with flakes and classically).

2 Likes
Hosted by Flying Circus.