I know that this might be a polarizing topic. This question is meant to be just for exchanging viewpoints.
The default location of the nix store is under /nix.
This is all well and good. However, for NixOS I see some reasons for moving the nix store to a location below /usr (like /usr/nix).
most Linux distributions aside from NixOS have gone through the usrmerge
systemd is starting to depend on usr being merged
a lot of cool immutability features will just work ™ when shipping everything that is immutable (as in not user state and configuration) in /usr
This last point is the most interesting in my opinion.
Systemd is pushing hard for immutable linux distributions and measured boot.
One path to achieve this is by saying that the root partition consists of only a /usr folder (aka usronly) and everything else is ephemeral (will be gone after a reboot).
Happy to hear what pros / cons exist for moving or not moving the nix store.
I will start this with saying that I am doing work on systemd and went through testing the whole usrmerge stuff and I have been called by mkosi developers to bring /usr/store support so that mkosi can pick it up blablablabla. So I have experience with that ecosystem and those features.
Is there any reason valid for NixOS in that link? As you know, NixOS does not really care about FHS and break it already.
Yes, and this has no effect on us, for previous reasons.
Can you cite a cool feature?
While moving the store in /usr/store may slightly help with some systemd stuff, the con is that we lose the cache for all previous nixpkgs version except if someone devises a modulo mechanism and a RUNPATH rewrite mechanism.
If such a person wants that /usr/store at all costs and drive the RFC, the infra work, etc. They may get a /usr/store while not busting all the valuable cache.
Otherwise, the same person can worst case just bind-mount the Nix store and have it live in /usr/store if really needed. I do not see what extra problems would this create.
Overall, there’s nothing that we cannot do to have all the nice features, the question is the effort. I believe that the effort to pull off a /usr/store (without talking about the fact that you need /usr/nix/store if you want to co-exist with /usr/gnu/store) is much higher than just not dealing with /usr.
We certainly lose some automation and freebies, but those are meaningless IMHO because they cannot perfectly work on NixOS and systemd upstream is not building them with NixOS in mind out there. So we do not gain a lot by adopting them.
@RaitoBezarius thanks for providing this context and for working on systemd integration in nix!
I’m not seriously proposing this as I’m sure it would be very unpopular but just for completeness here are some of the cool features:
systemd-sysext would work (and you could create extensions containing derivations that you can mount on any other Linux distro)
systemd-volatile-root / the kernel cmdline systemd.volatile=yes could work out of the box
When I’m on Fedora and want to use a sysext containing a nix store, I can’t do that with a nix store under /nix, because systemd-sysext will only extend the /usr and /opt folders.
Correct.
I wasn’t aware of that. Do you happen to have any pointers to examples on how to use dm-verity with nix? I’m especially interested in having a ephemeral system with only a readonly nix store being on disk. I want that nix store to be mounted with dm-verity and I want to supply the expected verity roothash via a measured component of the boot process (like a measured kernel cmdline parameter in a UKI).
That makes sense, but Nix + Fedora + sysext is probably not a supported usecase and I would find it hard to get effort to support for the reasons mentioned above.
so in this example I’m using systemd’s usrhash functionality and bind-mounting /usr/store to /nix/store afterwards.
But in theory it’s all not needed as we could just ship an /etc/veritytab file in our initrd to directly set up a /nix/store partition with verity support without requiring a /usr partition. it just was a bit easier to get running.
Bind-mounting /usr/store to /nix/store just made it easier to iterate.
to simulate something similar to volatileroot I came up with:
(Which I stole from our ISO builder I think)
This mounts an overlayfs over /nix/store that allows you to use all the nix commands without modifying the underlying verity partition. Of course things don’t persist across
(Edit: this may be moot; I got interrupted and there’ve been a lot of comments since I started writing this… :))
Are there material reasons you suggest /usr instead of /opt?
I’m pretty sure /usr itself is a nonstarter on macOS. IIRC everything but /usr/local is protected by SIP.
(There is some interest/merit in moving /nix from a macOS perspective. The introduction of the read-only root made macOS installs relatively more complex, and that complexity led to dropping support for single-user installs on macOS.)
Thanks for pointing out the SIP situation on macOS. I wasn’t aware of that.
From a Linux-centric perspective, /usr would be better just because of the developments in systemd and their view of how a Linux filesystem would ideally be organized .
Similar to macOS, the direction is that /usr is for the system and this is where more of the authenticated boot is happening and anything else can be reset safely (either on every boot or if the user wants to perform a reset).
This being said, this is all less important for nix since it is already fully declarative and reproducible.
I agree that for NixOS the location should not matter.
For nix though FHS-compliant Linux it would be nice to be more compliant with host OS policies. Otherwise it’s a higher barrier to package nix as a normal package.
I agree /opt or even /var/lib is slightly better than /usr (with caveats below). I personally would normally expect /usr to be handled by the host system’s package manager, not by nix. And for /opt it might depend.
"Applications must never create or require special files or subdirectories in the root directory. Other locations in the FHS hierarchy provide more than enough flexibility for any package.". Both /nix and /guix directories violate that. It’s the main concert of many FHS-based distributions that blocks nix adoption.
"Large software packages must not use a direct subdirectory under the /usr hierarchy.". This one rules out /usr/nix/store as well.
"package to be installed in /opt must locate its static files in a separate /opt/<package> or /opt/<provider> directory tree, where <package> is a name that describes the software package and <provider> is the provider's LANANA registered name.". Looks like to use that we ideally register nix in LSB. /nix/store will still not conform to FHS in spirit: "Programs to be invoked by users must be located in the directory /opt/<package>/bin or under the /opt/<provider> hierarchy. If the package includes UNIX manual pages, they must be located in /opt/<package>/share/man or under the /opt/<provider> hierarchy, and the same substructure as /usr/share/man must be used.". This all hints at the expectation of a reasonable single-package structure and not an indirection.
I think it makes FHS an imperfect model to plug nix into.
But maybe /var/lib is the best one that follows the letter of FHS. And if we follow examples like /var/lib/containers in the world where packages are installed outside the package manager of host OS it might sound like close enough fit.
/var mildly implies the state not shareable with other systems which might be not be quite true. But maybe I think it’s fine.
Everything inside /usr is frozen in Fedora Silverblue. Just like with /opt. But Nix isn’t frozen. Otherwise, I’d have to reboot to install a new Nix package, which doesn’t make any sense. It is variable data, so its place (in case it were to change) should be under /var IMHO.
/var/lib/docker and /var/lib/flatpak exist, so /var/lib/nix sounds like a reasonable option. The file-hierarchy man page suggests that it should generally be possible to wipe /var and still end up with a bootable system. With the Nix store in /var/lib/nix, this wouldn’t be true on NixOS anymore. I’d say the Nix store is closer in principle to a system’s OSTree repo, which is stored in /ostree, which surely isn’t FHS-compliant either. The FHS doesn’t seem to have a notion of “content-addressed storage directory for system components”.
If docker, postgresql, mysql, httpd, … goes there, I would rather wipe /home, /usr, /opt, /etc, reinstall the system than loose production data.
And because of that, admins allocate more disk space to /var than to /usr. Nix isn’t disk efficient. We need more space than usually people sets to /usr, even for an update followed by GC.
I mean if it changes to /var I could install nix in most of the company VMs without asking to resize /, /usr, mount a new /nix or fight with nix config about linking /nix.