Install some Nix packages to other location than /nix/store

Hello Nixers,

I’ve recently installed NixOS on the following disks setup:

| Disk type | Device      | storage |
|-----------|-------------|---------|
| SSD       | /dev/nvme0n | 128G    |
| HDD       | /dev/sda    | 2T      |

My system as well as a nix store is located inside the SSD disk partition but it is really low on storage capabilities (only 128 GB). Therefore I wonder if it is possible to install some Nix packages on the other drive while still having other packages on the main SSD drive?

I personally thought about creating a script that runs on rebuilding the system and moves chosen installed packages to the other location and then symlinks it to the nix store.

Moreover, Could it be possible to achieve the similar behaviour when building software projects, where all the dependencies would be stored in a project directory instead of /nix/store ?

I would love to hear some advice from you as I am feeling completely hopeless in this matter.

1 Like

This isn’t possible as far as I know; the nix store paths are all by design in one location.

You can, however, move your store root to a different disk/partition: Storage optimization - NixOS Wiki

More advanced file systems (llvm, zfs, btrfs & co) can also transparently create partitions across multiple disks.

Thanks for the answer, llvm could be a good option but still not fulfilling :frowning:

Wouldn’t lvmcache achieve most of the desired benefits? Or do you need to optimise the cold launch time of something rarely used?

I guess in principle you could have two stores, one on the SSD and one on the HDD, and use namespaces with either one bind-mounted on /nix/store/ as needed. Of course they should trust each other as a source of substitution.

It is “possible” but the UX might not be so great. You can pass the --store flag to point Nix to a different file-system location. However, you would have to pass it around quite often and it might get tedious, I think you can set it in the nix.conf as well, but then you’d have to remember to set it back and forth. Probably better to just move the /nix/store onto the bigger drive.

You could just move stuff and symlink to it. Probably need to repeat that once in a while.

Alternatively, you can reinstall with ZFS and use the SSD as a special device. You could tune that to a fast system, but there is a learning curve.

Or just wait a bit until bcachefs becomes stable :slight_smile:

Thanks a lot for the answers. The fact that I have some much freedom is really great. I am certain at one thing, I would definitely tinker with it.

@7c6f434c, LVM seems really good but I have no experience with setting this up. I just want to have a freedom to choose which packages I want to install on SSD and which on HDD. I don’t know if lvmcache offers such a flexibility.

@nrdxp Yeah I thought about it, Sadly, as you’ve said, it’s awful UX-wise,

!bitvic bcachefs is also new to me, so I would dig more into it :smiley:

After all I think I would just try my best to write a Nix package that automatically symlinks packages (If it’s even possible) and use inside my projects and configurations. Sorry if what I’m saying is a complete nonsense but I am totally new to the NixOS whole system philosophy :sweat_smile:

It’s not complete nonsense, this is a frequent-ish request that’s sadly just a bit of a limitation of nix.

Being able to put binaries on specific disks for performance reasons would indeed be cool, and I even feel like a nix store style package storage mechanism is a better abstraction for this than the traditional fhs.

I sadly also think the performance improvement is unnoticeable enough vs just dumping the store on a slower, bigger disk (or spending the 100$ on a larger SSD) that nobody will put in the work to get a useful UI for this. It also mostly applies to personal computers, and is less useful in big build farms and whatnot (since you probably just spin up a new build server with better hardware instead), so I wouldn’t hold my breath for commercial investment.

I think solving this on the filesystem level is your best bet, though it won’t give you package-level resolution, and instead just give you performance approaching the perfect package → disk association.

Nice, I am running NixOS on my desktop, that’s why I asked this question :smile:

I thought that I would witness much bigger performance drop when combining partitions with LVM, if that’s not the case, I will definitely give it a try.

I wouldn’t expect LVM to have noticeable overhead… it doesn’t need to do that much on top of executing the I/O operation.

1 Like

Replit is doing this at scale: Super Colliding Nix Stores

2 Likes

cc storePath as a settable property of derivation · Issue #5067 · NixOS/nix · GitHub