Is there a way to share a /nix/store across multiple machines?

Hi! I posted this question on the discord, but they said it was probably a better question for here. I have Proxmox running on a relatively storage constrained server at home, and I have a few NixOS VMs on it. All my VMs share the same flake.lock, and are deployed at the same time. As a result, I figure that the store has a good bit of duplication across my VMs, and I was wondering if there was any good way to potentially share a /nix/store across these VMs I’m hosting in order to minimize storage usage. I’ve considered putting /nix/store on a network file share, but I’m not sure how robust it is or if it’s even possible for a NixOS machine (The wiki dosen’t seem to mention anything about it on NixOS). At the end of the day it’s not really the end of the world if I can’t share the store, but it’d certainly help me put off buying more storage for a while. :slight_smile:

1 Like

By “NixOS VMs”, do you mean VMs created with nixos-rebuild build-VM , or just multiple NixOS images?

The script that nixos-rebuild build-vm creates by default transparently makes the host’s /nix/store available, which should take care of any duplication. If you’re not already using that, it may serve as inspiration :slight_smile:

It’s just multiple unrelated NixOS images (aside from sharing the flake.lock). They aren’t running on top of NixOS, but on top of Proxmox (think VMWare or something). These VMs are running on top of a host that isn’t nix aware in any way. As for the build-vm script, I see the area with the build-vm here, but admittedly I don’t know how to read bash scripts very well. It looks like it just calls nix-build -A vm with whatever was passed to the script. Nix-build takes me to the nix binary, and from there on things get a lot more complex.

Using LVM with deduplication was also mentioned on the discord, and I’m starting to think that’s realistically the best option if I do want to bother trying to do the above.

I meant the script that build-vm writes :slight_smile: The source for NixOS VM-building infra lives here: nixpkgs/default.nix at 634141959076a8ab69ca2cca0f266852256d79ee · NixOS/nixpkgs · GitHub

This actually helps a lot, I think I might be able to do the same trick with passing the store path as a VirtFS. This is the first time I’m hearing about them, but I imagine if I make one holding /nix/store or something I’ll be able to pass it to all the guests? Either way, at least I have something to try now!

1 Like

If you are comfortable with fiddling with mountpoints, you can probably designate one VM for a “master nix store” and have the nix stores of all others reside on overlayfs-mounts.
You may need to use fileSystems.<name>.depends to ensure upper and lower are available before the overlay itself is mounted.