Hi! I posted this question on the discord, but they said it was probably a better question for here. I have Proxmox running on a relatively storage constrained server at home, and I have a few NixOS VMs on it. All my VMs share the same flake.lock, and are deployed at the same time. As a result, I figure that the store has a good bit of duplication across my VMs, and I was wondering if there was any good way to potentially share a /nix/store across these VMs I’m hosting in order to minimize storage usage. I’ve considered putting /nix/store on a network file share, but I’m not sure how robust it is or if it’s even possible for a NixOS machine (The wiki dosen’t seem to mention anything about it on NixOS). At the end of the day it’s not really the end of the world if I can’t share the store, but it’d certainly help me put off buying more storage for a while.
By “NixOS VMs”, do you mean VMs created with
nixos-rebuild build-VM , or just multiple NixOS images?
The script that
nixos-rebuild build-vm creates by default transparently makes the host’s
/nix/store available, which should take care of any duplication. If you’re not already using that, it may serve as inspiration
It’s just multiple unrelated NixOS images (aside from sharing the flake.lock). They aren’t running on top of NixOS, but on top of Proxmox (think VMWare or something). These VMs are running on top of a host that isn’t nix aware in any way. As for the build-vm script, I see the area with the build-vm here, but admittedly I don’t know how to read bash scripts very well. It looks like it just calls
nix-build -A vm with whatever was passed to the script. Nix-build takes me to the
nix binary, and from there on things get a lot more complex.
Using LVM with deduplication was also mentioned on the discord, and I’m starting to think that’s realistically the best option if I do want to bother trying to do the above.
I meant the script that
build-vm writes The source for NixOS VM-building infra lives here: nixpkgs/default.nix at 634141959076a8ab69ca2cca0f266852256d79ee · NixOS/nixpkgs · GitHub
This actually helps a lot, I think I might be able to do the same trick with passing the store path as a VirtFS. This is the first time I’m hearing about them, but I imagine if I make one holding /nix/store or something I’ll be able to pass it to all the guests? Either way, at least I have something to try now!
If you are comfortable with fiddling with mountpoints, you can probably designate one VM for a “master nix store” and have the nix stores of all others reside on overlayfs-mounts.
You may need to use
fileSystems.<name>.depends to ensure
lower are available before the overlay itself is mounted.