I am trying to build a AMI for EC2.
My flake.nix looks like this (removed anything that wasn’t failing)
{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.11";
flake-utils.url = "github:numtide/flake-utils";
nixos-generators = {
url = "github:nix-community/nixos-generators";
inputs.nixpkgs.follows = "nixpkgs";
};
};
outputs = { self, nixpkgs, flake-utils, nixos-generators, ... }:
flake-utils.lib.eachDefaultSystem
(system : let pkgs = nixpkgs.legacyPackages.${system}; in {
# Generate an AWS AMI for EC2
packages.homelab-ami = nixos-generators.nixosGenerate {
system = "x86_64-linux";
modules = [ ./homelab.nix ];
format = "amazon";
};
});
}
The error I get is
error: a 'x86_64-linux' with features {} is required to build '/nix/store/120lngrg2dlz8z4iyiyhjj4ilangrxbi-configuration.nix.drv', but I am a 'aarch64-darwin' with features {benchmark, big-parallel, nixos-test}
I want to be able to run nix build .#homelab-ami from any nix supported operating system and architecture. Is this currently possible without using a remote runner?
I have attempted to understand what is going on myself. For example I see that nixGenerate calls into nixosSystem https://github.com/nix-community/nixos-generators/blob/896f6589db5b25023b812bbb6c1f5d3a499b1132/flake.nix#L103
and seems to return a derivation based on image.config.system.build.${image.config.formatAttr};. However I haven’t been able to understand the code well enough to find where image.config.system.build is defined in the nixpkgs source.
I am happy to contribute a patch if someone can point me in the correct direction.
not sure if this works here
add this code to your system configuration.nix file
boot.binfmt.emulatedSystems = [ "x86_64-linux" ];
EDIT
sorry this is for NixOS i don’t know how you should do it in non-Nix OS
Looks like this is most likely not supported
https://nixos.org/manual/nix/unstable/contributing/hacking.html?highlight=emulated#platforms
However I am still looking for help understanding why or waiting to see if anyone else has a solution.
It works; you just have to install qemu binfmt emulation on your distro, and then you have to add the necessary paths to extra-sandbox-paths in nix.conf.
Beware that qemu binfmt emulation is extremely slow. The alternative is cross-compilation, but it doesn’t look like nixos-generators is well setup for that. Plus nixpkgs is barely cached for cross-compilation so you’d end up compiling basically all of nixos yourself, and it’s not uncommon for things to simply fail to cross compile.
Point being, the options for building for another platform aren’t great. Frankly, I think it’s better with Nix than it is with anything else, but it’s still rough. My advice is to enable qemu binfmt emulation via your distro, and possibly configure specific packages for cross compilation if you find you need it (e.g. the kernel would be a great candidate if you need to patch it)
Are you able to point me in the correct direction of where the code for generating the images lives?
Ideally I would be able to do this without any emulation layer. While I hope I will be able to improve the state of things, more realistically I will settle for just knowing why I can’t currently do this without emulation.
Notes for later
https://5pi.de/2015/03/13/building-aws-amis-from-scratch/
Okay I looked into this a bit more.
https://github.com/NixOS/nixpkgs/blob/a6fefb39e18b6ff828c04d59ea26d4988135bb88/nixos/lib/make-disk-image.nix is where the disk image is actually created. If I understand correctly it builds the packages and copies the resulting nix store to the disk image. The emulation layer is needed as building packages on aarch64-darwin for target x86_64-linux is difficult (for example GCC requires you recompile it for cross arch builds).
I don’t have a straightforward solution as any solution requires that all packages be able to build cross arch which in our current world requires emulation.