I’m trying to build a system architecture with a very lightweight, bare-metal KVM host (for which I have elected to use NixOS, thanks to its immutability), with the ‘actual’ OS running on top.
I want the KVM host to boot directly from its initramfs, without ever having to mount a root disk, which would mean that a read-only Nix store would have to be built into the initramfs. Is there any way to build a system image to boot in such a way?
I want the KVM host to boot directly from its initramfs, without ever having to mount a root disk, which would mean that a read-only Nix store would have to be built into the initramfs. Is there any way to build a system image to boot in such a way?
What exactly you want from the host? Note that using standard stuff from Nixpkgs without careful picking what files to include will probably mean wasting a couple hundreds of MiBs of RAM. Of course any approach of the host completely in RAM has to waste some RAM. It is easy to build an initramfs with /nix/store including precisely the paths you want and put a symlink to whatever init script you want in /; so even without NixOS support for that you can either hack NixOS into initramfs with a custom stage-1, or hack together a very minimal bootscript launching SSH and/or KVM VMs under whatever service supervisor you choose (in the case you try to go for smaller size, systemd reliance on specifically glibc might be or not be annoying)
The host would boil down to: libvirt+qemu+kvm, managing network and VFIO, while expecting the “primary” host to be handling the GPU directly. I’ll figure out on my own how to shrink the memory footprint of such a system (such as using musl libc), but I just want to know how a nixos-install (or nix build) config could produce a stand-alone NixOS initramfs with a single system generation baked in.
The host would boil down to: libvirt+qemu+kvm, managing network and VFIO, while expecting the “primary” host to be handling the GPU directly. I’ll figure out on my own how to shrink the memory footprint of such a system (such as using musl libc), but I just want to know how a nixos-install (or nix build) config could produce a stand-alone NixOS initramfs with a single system generation baked in.
You cannot have NixOS with musl because of systemd. For Nix-based but slightly differently composed stuff in initramfs, I use code like that: https://github.com/7c6f434c/lang-os/blob/7a0e90d7bd78e1900374ad75d663f2bc1f2ee86d/fat-initramfs.nix but I guess there are other examples floating around
After further consideration, I have decided against trying to jam NixOS into the initramfs, as it would be simpler to build the Nix store into a squashfs image and use that as the system root. Thanks for the answers
1 Like