Hey there,
I am fairly inexperienced with the virtualization and cross compilation infrastructure available in nixpkgs.
I know how to use nixos-rebuild --flake build-vm .#<hostname> to generate and run a vm image with the same arch as my own machine.
Question: Is it possible to use nixos-rebuild build-vm from a x86_64-linux system to build and emulate a aarch64-linux system? I have tried a few variants of
{
outputs = {self, nixpkgs }: {
nixosConfigurations.aarch64_base = nixpkgs.lib.nixosSystem {
modules = [
(
{pkgs, ...}: {
system.stateVersion = "23.05";
boot.loader.systemd-boot.enable = true;
boot.loader.efi.canTouchEfiVariables = true;
nixpkgs.localSystem.system = "x86_64-linux";
nixpkgs.crossSystem.system = "aarch64-linux";
users.users.test = {
isNormalUser = true;
extraGroups = [ "wheel" ];
initialPassword = "test";
};
}
)
];
};
};
}
but the resulting run-nixos-vm scripts reference aarch64-linux binaries, which don’t run on x86_64. I am aware of a few independent projects which provide such functionality, so i know its possible, but I would prefer to avoid dependencies other than nixpkgs if possible.
I also would like to avoid setting boot.binfmt.emulatedSystems = ["aarch64-linux"];, to make sure I don’t end up with a qemu-inside-qemu situation.
OK, following AArch64 VM · GitHub, I was able to write something which works:
{
inputs = {
nixos-generators = {
url = "github:nix-community/nixos-generators";
inputs.nixpkgs.follows = "nixpkgs";
};
};
outputs = {self, nixpkgs, nixos-generators}: {
packages."x86_64-linux" = {
aarch64-linux-vm =
let pkgs-x86_64 = import nixpkgs { system = "x86_64-linux"; };
pkgs-aarch64 = import nixpkgs { system = "aarch64-linux"; };
drive-flags = "format=raw,readonly=on";
in pkgs-x86_64.writeScriptBin "run-nixos-vm-aarch64" ''
#!${pkgs-x86_64.runtimeShell} \
${pkgs-x86_64.qemu_full}/bin/qemu-system-aarch64 \
-machine virt \
-cpu cortex-a57 \
-m 2G \
-nographic \
-drive if=pflash,file=${pkgs-aarch64.OVMF.fd}/AAVMF/QEMU_EFI-pflash.raw,${drive-flags} \
-drive file=${self.packages."x86_64-linux".aarch64-linux-iso}/iso/nixos.iso,${drive-flags}
'';
aarch64-linux-iso =
let aarch64-iso-module = {pkgs, lib, ...}: {
system.stateVersion = "23.05";
nixpkgs.buildPlatform.system = "x86_64-linux";
nixpkgs.hostPlatform.system = "aarch64-linux";
boot.loader = {
systemd-boot.enable = true;
efi.canTouchEfiVariables = true;
timeout = lib.mkForce 0;
};
users.users.test = {
isNormalUser = true;
extraGroups = [ "wheel" ];
initialPassword = "test";
};
services.getty.autologinUser = "test";
};
in nixos-generators.nixosGenerate {
system = "x86_64-linux";
format = "iso";
modules = [ aarch64-iso-module ];
};
};
};
}
If anyone has any better ideas, would be interested!
1 Like
You could probably just use nixos-rebuild build-vm if you set virtualisation.vmVariant.virtualisation.host.pkgs to an instantiation of nixpkgs for your local system, which would cause qemu-vm.nix to use your local system’s native qemu package. Then it wouldn’t matter if the main config is cross compiled for aarch64 or “natively” compiled with emulatedSystems.
I am a bit out of my depth here, haven’t done much virtualization. But would you have an example (similar to the one above) that uses this configuration?
Hey Daniel, nice to see you around again. I am getting this error when trying to build the vm script. Any ideas?
 nix build .#aarch64-linux-vm
error: a 'aarch64-linux' with features {} is required to build '/nix/store/a35k6bnpg46njvszmj3fwwqajy8n2ix2-bootstrap-tools.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}
Yo @jonathanlorimer good to see you again! just to make sure we are talking about the same thing, can you try nix build .#aarch64-linux-vm from https://github.com/danielbarter/nixos-config/blob/8a638086af2d2545ed95adfb32d6861bb65cbb26/flake.nix?
I just tested it on a x86_64-linux machine and it worked.
hmmm it seems to be working when I invoke from your config. Must be something to do with how I am using flake-parts cause I copy and pasted your code directly here https://github.com/informalsystems/cosmos.nix/blob/jonathan/arm64-ci/modules/virtualization.nix
Yeah, if flake-parts is messing with systems, then that would explain it
@danielbarter Got it working with flake-parts, turns out I needed to use nixpkgs.legacyPackages."${system}" instead of importing.
Do you know how to change the disk size? I found qemu-resize, but not sure how to thread that into this script (especially since the iso is read-only in the nix store). Maybe there’s an option that can be passed into the nixos-generate function?
@jonathanlorimer : good job!
You could bump up the ram if you are happy with a ramdisk. If you want something else, you will need to figure out the correct arguments to qemu to create a disk file and mount it