I recently upstreamed into Nixpkgs a convenient darwin.builder script for launching a Linux builder on macOS without requiring access to an existing Linux builder. For more details, see the following blog post, which explains the background and motivation for this change:
16 Likes
Cool! I just integrated this in our internal nix-darwin configuration and so far it seems to work smoothely! No manual intervention required: The VM is run using a nix-darwin managed launchd daemon.
We need SSH enabled on the host machines though, so running the VM on port 22 isn’t possible.
My workaround was to use a custom SSH config where the port can be changed. Here is my nix-darwin module:
{ config, pkgs, lib, ... }:
let
dataDir = "/var/lib/nixos-builder";
linuxSystem = builtins.replaceStrings [ "darwin" ] [ "linux" ]
pkgs.stdenv.hostPlatform.system;
outerPkgs = pkgs;
port = 31022;
# Using `nixpkgs-unstable` here as `darwin.builder` is a relatively new feature. We want the latest updates.
linuxBuilder = (import "${pkgs.nixpkgs-unstable.path}/nixos" {
system = linuxSystem;
configuration = ({ modulesPath, lib, ... }: {
imports = [ "${modulesPath}/profiles/macos-builder.nix" ];
virtualisation.host.pkgs = outerPkgs;
virtualisation.forwardPorts = lib.mkForce [{
from = "host";
host.address = "127.0.0.1";
host.port = port;
guest.port = 22;
}];
});
}).config.system.build.macos-builder-installer;
runLinuxBuilder = pkgs.writeShellScriptBin "run-linux-builder" ''
set -uo pipefail
trap 's=$?; echo "$0: Error on line "$LINENO": $BASH_COMMAND"; exit $s' ERR
IFS=$'\n\t'
mkdir -p "${dataDir}"
cd "${dataDir}"
${linuxBuilder}/bin/create-builder
'';
in {
config = {
#
environment.systemPackages = [ runLinuxBuilder ];
# Enable remote builds
nix.distributedBuilds = true;
nix.buildMachines = [{
hostName = "ssh://linux-builder";
system = linuxSystem;
maxJobs = 4;
# This is cheating: KVM isn't actually available (?) but QEMU falls back to "slow mode" in this case
supportedFeatures = [ "kvm" ];
}];
environment.etc."nix/ssh_known_hosts".text = ''
[127.0.0.1]:31022 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJBWcxb/Blaqt1auOtE+F8QUWrUotiC5qBJ+UuEWdVCb
'';
# We can't/want to edit /var/root/.ssh/config so instead we create the config at another location and tell ssh to use that instead by modifying NIX_SSHOPTS
environment.etc."nix/ssh_config".text = ''
Host linux-builder
User builder
HostName 127.0.0.1
Port ${toString port}
IdentityFile ${dataDir}/keys/builder_ed25519
UserKnownHostsFile /etc/nix/ssh_known_hosts
'';
# Tell nix-daemon to use our custom SSH config
nix.envVars = { NIX_SSHOPTS = "-F /etc/nix/ssh_config"; };
launchd.daemons.linux-builder = {
command = "${runLinuxBuilder}/bin/run-linux-builder";
path = with pkgs; [ "/usr/bin" coreutils nix ];
serviceConfig = {
KeepAlive = true;
RunAtLoad = true;
StandardOutPath = "/var/log/linux-builder.log";
StandardErrorPath = "/var/log/linux-builder.log";
};
};
};
}
1 Like
Another thing: Would it make sense if Hydra also built a disk image with boot.binfmt.emulatedSystems = [ "x86_64-linux"];? That would be cool for the new Macs. My current workaround was scripting manually: First start normal VM from cache.nixos.org, then with initial VM build a modified image with the emulatedSystems modified.
Hmm, maybe a more general approach would be to make the VM sort of “self-updating”. Maybe allow using sharedDirectories for mounting a Flake which would contain a NixOS configuration, which would be activated on the first boot of the VM? This way it would be easy for users of darwin.builder to customize the VM slightly without lots of scripting.
2 Likes
This is a game changer for me - no more fiddling with nix docker images and getting disappointed when things break!
I agree with Felix that some additional configuration options, such as specifying the port, the image used, etc., would make it even better. I’d also like to see some docs on how to enable it easily (and extensibly - e.g. appending the builder to the buildMachines if the list is already defined elsewhere in another module, etc.) in a nix os configuration (both with flakes and classically).
2 Likes
This was extremely helpful. So helpful that I couldn’t help but turn it into a nix-darwin module:
2 Likes
Nice, but why not add options to the original module? I think it would be helpful to have options to increase disk size especially, as I’ve found that I sometimes get build errors which I have recently realised are due to a lack of disk space on the VM.
I’ve been playing around with the VM a bit more and have also found that the ability to set authorized keys would also be quite useful. For example, then I can use the VM as a docker environment and set the DOCKER_HOST variable on the mac to connect via SSH to the machine.
It would also be cool if the VM could be integrated with sops-nix 
This is my personal Nix configs that I’m sharing here in case others want to copy/paste and modify as desired