Cross-Compile on Remote Builder

Alright, I have read some other forms regarding this but the most recent one was a year again so I wanted to see if anything has changed.

I have an aarch64-linux Asahi-NixOS laptop and a x86_64-linux NixOS desktop. I have set up the x86_64-linux desktop as a remote builder for the aarch64-linux. This however does not work and I get the error when I sudo nixos-rebuild switch --flake. on the aarch64 system:

brad@nixi ~/../../nix-config > sudo nixos-rebuild switch --flake . --impure
warning: Git tree '/home/brad/.config/nix-config' is dirty
building the system configuration...
warning: Git tree '/home/brad/.config/nix-config' is dirty
error: build of '/nix/store/zhny3wzghdmw9d7brz332pdys808zm42-xwayland-24.1.4.drv' on 'ssh://brad@10.0.0.11' failed: error: a 'aarch64-linux' with features {} is required to build '/nix/store/zhny3wzghdmw9d7brz332pdys808zm42-xwayland-24.1.4.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}
error: builder for '/nix/store/zhny3wzghdmw9d7brz332pdys808zm42-xwayland-24.1.4.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/n8mrv8y31a6r5s36wswbr564fpkg05dg-system-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/2zj5r3n2zjpw28q6nhfjc4mzqjrgnx5k-nixos-system-nixi-25.05.20241205.929116e.drv' failed to build

So the only way I saw around this was to add boot.binfmt.emulatedSystems = [ "aarch64-linux" ] to the configuration.nix of the remote builder. This does work but as others have stated, emulation is slow.

Also even though it does work with emulation, when I rebuild, it is actually building on both machines (remote builder on top and local machine on bottom):

So my main questions are:

  1. Is there a way to cross-compile in this way without using emulation and if so, how? How does Nix’s Hydra do this?

  2. Why is it building on both machines and how can I get it to just build on the remote builder?

It seems like you’ve just set up cross compilation wrong in your configuration. We’d have to see your nix configuration

1 Like

Here is my flake.nix. Both systems are in it. Remote builder is brixos and local machine is nixi:

{
  description = "Your new nix config";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    # stable.url = "nixpkgs/nixos-24.05";
    # master.url = "nixpkgs/master";

    apple-silicon = {
      url = "github:zzywysm/nixos-asahi";
      inputs.nixpkgs.follows = "nixpkgs";
    };

    home-manager = {
      url = "github:nix-community/home-manager/master";
      inputs.nixpkgs.follows = "nixpkgs";
    };

    nix-flatpak = {
      url = "github:gmodena/nix-flatpak";
    };

    nix-darwin = {
      url = "github:LnL7/nix-darwin";
      inputs.nixpkgs.follows = "nixpkgs";
    };

    nix-homebrew = {
      url = "github:zhaofengli-wip/nix-homebrew";
      inputs.nixpkgs.follows = "nixpkgs";
    };

    firefox = {
      url = "github:nix-community/flake-firefox-nightly";
      inputs.nixpkgs.follows = "nixpkgs";
    };

    zen-browser = {
      url = "github:ch4og/zen-browser-flake";
      inputs.nixpkgs.follows = "nixpkgs";
    };

    # Optional: Homebrew taps for crapple
    homebrew-core = {
      url = "github:homebrew/homebrew-core";
      flake = false;
    };
    homebrew-cask = {
      url = "github:homebrew/homebrew-cask";
      flake = false;
    };
    homebrew-bundle = {
      url = "github:homebrew/homebrew-bundle";
      flake = false;
    };
    zen-browser-darwin = {
      url = "github:darbster145/zen-browser-darwin";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs =
    { self
    , nixpkgs
    , home-manager
    , nix-flatpak
    , apple-silicon
    , nix-darwin
    , nix-homebrew
    , ...
    } @ inputs:
    let
      inherit (self) outputs;
      # Supported systems for your flake packages, shell, etc.
      systems = [
        "aarch64-linux"
        "i686-linux"
        "x86_64-linux"
        "aarch64-darwin"
        "x86_64-darwin"
      ];
      # This is a function that generates an attribute by calling a function you
      # pass to it, with each system as an argument
      forAllSystems = nixpkgs.lib.genAttrs systems;
    in
    {
      # Your custom packages
      # Accessible through 'nix build', 'nix shell', etc
      packages = forAllSystems (system: import ./pkgs nixpkgs.legacyPackages.${system});
      # Formatter for your nix files, available through 'nix fmt'
      # Other options beside 'alejandra' include 'nixpkgs-fmt'
      formatter = forAllSystems (system: nixpkgs.legacyPackages.${system}.alejandra);

      # Your custom packages and modifications, exported as overlays
      overlays = import ./overlays { inherit inputs; };
      # Reusable nixos modules you might want to export
      # These are usually stuff you would upstream into nixpkgs
      nixosModules = import ./modules/nixos;
      # Reusable home-manager modules you might want to export
      # These are usually stuff you would upstream into home-manager
      homeManagerModules = import ./modules/home-manager;

      # NixOS configuration entrypoint
      # Available through 'nixos-rebuild --flake .#your-hostname'
      nixosConfigurations = {
        brixos = nixpkgs.lib.nixosSystem {
          specialArgs = { inherit inputs outputs; };
          modules = [
            ./nixos/hosts/brixos/configuration.nix
            nix-flatpak.nixosModules.nix-flatpak
          ];
        };
        nixi = nixpkgs.lib.nixosSystem {
          specialArgs = { inherit inputs outputs; };
          system = "aarch64-linux";
          modules = [
            ./nixos/hosts/nixi/configuration.nix
            nix-flatpak.nixosModules.nix-flatpak
            apple-silicon.nixosModules.apple-silicon-support
          ];
        };
      };

      darwinConfigurations = {
        "JRSYS-MBA-01" = nix-darwin.lib.darwinSystem {
          specialArgs = { inherit inputs; };
          system = "aarch64-darwin";
          modules = [
            ./darwin/hosts/jrsys-mba-01/configuration.nix
            nix-homebrew.darwinModules.nix-homebrew
          ];
        };

        crapple = nix-darwin.lib.darwinSystem {
          specialArgs = { inherit inputs; };
          system = "aarch64-darwin";
          modules = [
            ./darwin/hosts/crapple/configuration.nix
            nix-homebrew.darwinModules.nix-homebrew
          ];
        };
      };
    };

  # Standalone home-manager configuration entrypoint
  # Available through 'home-manager --flake .#your-username@your-hostname'
  #homeConfigurations = {
  #  # FIXME replace with your username@hostname
  #  "your-username@your-hostname" = home-manager.lib.homeManagerConfiguration {
  #    pkgs = nixpkgs.legacyPackages.x86_64-linux; # Home-manager requires 'pkgs' instance
  #    extraSpecialArgs = { inherit inputs outputs; };
  #    modules = [
  #      # > Our main home-manager configuration file <
  #      ./home-manager/home.nix
  #    ];
  #  };
  #};
  #};
}

configuration.nix for remote build machine minus some unnecessary parts:

{ config, pkgs, inputs, ... }:

{
  imports = [
    ./hardware-configuration.nix
    ./gnome.nix
    ./iscsi.nix
  ];

  boot.kernelParams = [ "acpi_enforce_resources=lax" ];
  boot.kernelPackages = pkgs.linuxPackages_latest;
  boot.kernelModules = [ "iscsi_tcp" "it87" "coretemp" ];
  boot.extraModprobeConfig = ''
    options it87 force_id=0x8689
  '';
  boot.extraModulePackages = with config.boot.kernelPackages; [
    it87
  ];

  # Bootloader
  boot.loader = {
    efi = {
      canTouchEfiVariables = true;
    };
    grub = {
      enable = true;
      efiSupport = true;
      device = "nodev";
      useOSProber = true;
    };
  };

  nix.settings.trusted-users = [ "root" "brad" ]; 

  nix.settings.sandbox = true;

  boot.supportedFilesystems = [ "ntfs" ];

  virtualisation.virtualbox.host.enable = true;
  users.extraGroups.vboxusers.members = [ "brad" ];

  zramSwap.enable = true;

  powerManagement.enable = true;

  networking.hostName = "brixos"; # Define your hostname.
  # networking.wireless.enable = true;  # Enables wireless support via wpa_supplicant.

  # Configure network proxy if necessary
  # networking.proxy.default = "http://user:password@proxy:port/";
  # networking.proxy.noProxy = "127.0.0.1,localhost,internal.domain";

  # Enable networking
  networking.networkmanager.enable = true;

  hardware.graphics = {
    enable = true;
    enable32Bit = true;
    extraPackages = with pkgs; [
      rocmPackages.clr
      libva
      libvdpau-va-gl
    ];
  };

  # Set your time zone.
  time.timeZone = "America/Denver";

  # Select internationalisation properties.
  i18n.defaultLocale = "en_US.UTF-8";

  i18n.extraLocaleSettings = {
    LC_ADDRESS = "en_US.UTF-8";
    LC_IDENTIFICATION = "en_US.UTF-8";
    LC_MEASUREMENT = "en_US.UTF-8";
    LC_MONETARY = "en_US.UTF-8";
    LC_NAME = "en_US.UTF-8";
    LC_NUMERIC = "en_US.UTF-8";
    LC_PAPER = "en_US.UTF-8";
    LC_TELEPHONE = "en_US.UTF-8";
    LC_TIME = "en_US.UTF-8";
  };

  # Hyprland
  programs.hyprland.enable = true;

  # KDE Connect
  programs.kdeconnect = {
    enable = true;
  };

  # Configure keymap in X11
  services.xserver = {
    enable = true;
    videoDrivers = [ "amdgpu" ];
    xkb.layout = "us";
    xkb.variant = "";
  };

  # Enable CUPS to print documents.
  services.printing.enable = true;

  # Enable Tailscale
  services.tailscale.enable = true;

  # Enable Sunshine
  services.sunshine = {
    enable = true;
    autoStart = true;
    capSysAdmin = true;
    openFirewall = true;
  };

  # Enable sound with pipewire.
  hardware.pulseaudio.enable = false;
  security.rtkit.enable = true;
  services.pipewire = {
    enable = true;
    alsa.enable = true;
    alsa.support32Bit = true;
    pulse.enable = true;
    # If you want to use JACK applications, uncomment this
    #jack.enable = true;

    # use the example session manager (no others are packaged yet so this is enabled by default,
    # no need to redefine it in your config for now)
    #media-session.enable = true;
  };

  # Enable touchpad support (enabled default in most desktopManager).
  # services.xserver.libinput.enable = true;

  hardware.bluetooth.enable = true; # enables support for Bluetooth
  hardware.bluetooth.powerOnBoot = true; # powers up the default Bluetooth controller on boot

  # Enable QMK and VIA
  hardware.keyboard.qmk.enable = true;

  services.udev.packages = [ pkgs.via ];

  services.openiscsi = {
    enable = true; # Enable openiscsi daemon
    name = "iqn.2024-09.com.nixos:my-nixos-initiator"; # Set your iSCSI initiator name
    discoverPortal = "10.0.0.3";
  };

  # LACT systemd service
  systemd.services.lact = {
    description = "AMDGPU Control Daemon";
    after = [ "multi-user.target" ];
    wantedBy = [ "multi-user.target" ];
    serviceConfig = {
      ExecStart = "${pkgs.lact}/bin/lact daemon";
    };
    enable = true;
  };


  boot.binfmt.emulatedSystems = [ "aarch64-linux" ];

  # Allow unfree packages
  nixpkgs.config = {
    allowUnfree = true;
    crossSystem = {
      config = "aarch64-linux";
    };
  };

...

  # This value determines the NixOS release from which the default
  # settings for stateful data, like file locations and database versions
  # on your system were taken. It‘s perfectly fine and recommended to leave
  # this value at the release version of the first install of this system.
  # Before changing this value read the documentation for this option
  # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
  system.stateVersion = "24.05"; # Did you read the comment?

  programs.coolercontrol.enable = true;

  services.avahi.publish.enable = true;
  services.avahi.publish.userServices = true;


  services.xrdp.enable = true;
  services.xrdp.defaultWindowManager = "${pkgs.gnome-session}/bin/gnome-session";
  services.xrdp.openFirewall = true;

  nix.settings.experimental-features = [ "nix-command" "flakes" ];

And the configuration.nix for the local machine:

{ config, lib, pkgs, apple-silicon, inputs, ... }:

{
  imports =
    [
      # Include the results of the hardware scan.
      ./hardware-configuration.nix
      ../../features/kanata.nix
    ];

  boot.kernelParams = [ "apple_dcp.show_notch=1" ];

  services.udev.extraRules = ''
    SUBSYSTEM=="power_supply", KERNEL=="macsmc-battery", ATTR{charge_control_end_threshold}="80"
  '';

  # Use the grub EFI boot loader.
  boot.loader.grub = {
    enable = true;
    useOSProber = false;
    device = "nodev";
  };
  boot.loader.efi.canTouchEfiVariables = false;

  # Enable zram swap
  zramSwap.enable = true;

  services.logind.extraConfig = ''
    # don’t shutdown when power button is short-pressed
    HandlePowerKey=ignore
  '';
  
  # Auto optimize the nix store
  nix.optimise.automatic = true;

  # Auto Garbage Collect
  nix.gc = {
    automatic = true;
    dates = "weekly";
    options = "--delete-older-than 15d";
  };

  networking.hostName = "nixi"; # Define your hostname.
  # Pick only one of the below networking options.
  # networking.wireless.enable = true;  # Enables wireless support via wpa_supplicant.
  networking.networkmanager.enable = true; # Easiest to use and most distros use this by default.

  # Set your time zone.
  time.timeZone = "America/Denver";

  # Select internationalisation properties.
  # i18n.defaultLocale = "en_US.UTF-8";
  # console = {
  #   font = "Lat2-Terminus16";
  #   keyMap = "us";
  #   useXkbConfig = true; # use xkb.options in tty.
  # };

  # enable GPU support and audio
  hardware.asahi = {
    useExperimentalGPUDriver = true;
    experimentalGPUInstallMode = "replace";
    setupAsahiSound = true;
  };

  # Enable the X11 windowing system.
  services.xserver.enable = true;

  # Enable KDE6
  services.displayManager.sddm.enable = true;
  services.desktopManager.plasma6.enable = true;

...

  security.sudo.wheelNeedsPassword = false;

  nixpkgs.config.allowUnfree = true;

...

  # Some programs need SUID wrappers, can be configured further or are
  # started in user sessions.
  # programs.mtr.enable = true;
  # programs.gnupg.agent = {
  #   enable = true;
  #   enableSSHSupport = true;
  # };

  # List services that you want to enable:

  # SSH 
  services.openssh.enable = true;

  # Open ports in the firewall.
  # networking.firewall.allowedTCPPorts = [ ... ];
  # networking.firewall.allowedUDPPorts = [ ... ];
  # Or disable the firewall altogether.
  networking.firewall.enable = true;

  # DO NOT CHANGE
  system.stateVersion = "24.11";

  # Enable nix experimental features
  nix.settings.experimental-features = [ "nix-command" "flakes" ];
}

hope that helps and thank you