/nix => /var/nix, /opt/nix, /usr/local/nix?

1

Summary, see original post below

Proposal: move /nix to:

  • /opt/nix: store directory
  • /var/lib/nix/{profiles/, gc.lock, ...}: everything that now lives under /nix/var/nix
  • /var/log/nix: Nix log files
Benefit
works with macOS Unlocks large user base
matches FHS Allows easier installation on existing systems
Easier to get buy-in from administrators
Shorter store path 6 less characters per referenced path probably adds up to a savings of a few KB in the store :clown_face:
Problem Solutions
Extra work, no benefit on current installs Making installation easier on other systems grows the ecosystem, which benefits Nix
Requires rebuilding all This regularly happens anyway
Historical packages only available via /nix They can be recompiled or patched, and /nix still works for a parallel or shared install
Hardcoded /nix in places Fix the broken code
Needs 2 Hydra builds Bind-mount the store, register both groups of packages. Since the prefix is part of the input hash, there should be no clashes, and invariant packages will be shared, invariant files hardlinked
How to migrate Write tool to do the migration, everything else remains the same
clang resolves symlinks, causing impurity Make sure the store is on a non-symlinked path
/var is a symlink on macOS Use /opt instead
/opt on Fedora Silverblue is immutable, and /usr/local is a symlink The userbase is currently much smaller and there are likely a number of mitigations possible
/nix is used all over the documentation Letā€™s update the documentation, use e.g. $NIX_ROOT, or, since this proposal removes the single root, $NIX_STORE

Original post

Up till now, having the Nix store and metadata live at /nix was a mild inconvenience, and maybe a cute quirk.

However, macOS has been clamping down on writing to /, and I have to wonder if all the trouble is worth it. Why not move /nix to a Filesystem Hierarchy Standard location, that will be more likely to be available?

I can think of these locations:

  • /var/nix: /var is for Variable data, and the Nix store is surely variable, any user with access to nix-shell can cause additions in there.
  • /var/lib/nix: /var/lib is for database storage, and the Nix store is database if you squint
  • /opt/nix: /opt is for Optional installs, and except on NixOS, thatā€™s the case
  • /usr/local/nix: /usr/local is for Local User installs, pretty apt for single-user nix installs

My favorite among these is /var/nix, about all systems have a /var, and itā€™s totally ok to have an extra directory in there. Itā€™s nicely out of the way and doesnā€™t disturb anyone. /var/lib/nix is truly conformant to FHS but a little more hidden.
/opt and /usr/local are not as likely to be existing on all systems.

While weā€™re at it, it would be nice to flatten the /nix/var/nix tree. Just have its children directly under nix/, wherever it is, and preferably symlink the log to /var/log/nix if possible.

To migrate to the new location, we can treat it as a new Nix installation, and either install the same packages, or rewrite store references in the existing ones (cutting some characters from their store path) so they run from the new location. Where possible, you can even bind-mount a single dir on both locations during a transition period, so unchanged files can be hardlinked together.

17 Likes
2

Iā€™d be all in favor of this change, even if it means some
synchronization between hydra changes and a nix update.

The thing thatā€™d scare me most is the likely number of places where we
have hardcoded /nix/store.

While weā€™d be at it, itā€™d probably be possible to also do things that
couldnā€™t be done otherwise because itā€™d require store changes, like
making the store hierarchical to reduce the hurt it inflicts on quite a
few scripts that donā€™t expect hundreds of thousands of entries in a
single directory.

(Not that I expect this to actually happen, but it does sound like a
relatively good idea, even if it means that hydra will be twice as slow
for the duration of the migration period)

Wout Mertens via NixOS Discourse nixos1@discoursemail.com writes:

1 Like
3

As nice as this might be, it would literally destroy pretty much every existing binary cache. Weā€™d lose the ability to use old pinned nixpkgs basically entirely without rebuilding the world, breaking countless non-updated projects. A hydra migration for cache.nixos.org would beā€¦ really, really expensive, and would break the cache for anyone with an older version of nix. Logistically this idea is just unrealistically difficult to pull off.

8 Likes
4

The thing thatā€™d scare me most is the likely number of places where we
have hardcoded /nix/store .

Iā€™ve been running nix in a non-standard store location for several years without much issue, so at least on this front I think weā€™d be fine. The bigger issue is that weā€™d have to go through a painful migration period and weā€™d lose all of our binary cache history. Iā€™d personally be up for a move to /opt/nix or /var/nix, but I can understand why many wouldnā€™t, and itā€™s certainly not a trivial task operationally.

5 Likes
5

I think your personal experience only accounts for the packages you need at the moment. Itā€™d require nixos.org to rebuild vastly, ridiculously more packages for the entirety of countless versions of nixpkgs and probably provision a second cache for the new store path.

2 Likes
7

It really is more efficient to put it all in one dir. If you split it up by the first few characters, then youā€™ll have a bunch of dirs with one or two entries, making the lookups slower. If you use less characters, then youā€™re just moving the problem to a different dir. ls handles my 11k entries just fine.

8

The old cache would still work for the old location, and there could be a rewrite pass to move the packages to the new location (losing some filename characters).

9

You couldnā€™t expect a rewrite pass to be effective.

  1. There are packages that donā€™t encode paths as strings in a way that could be detecting by a find and replace on all the files.
  2. There are tons of file formats that encode the length of strings, and changing to a longer string would make the file invalid.
1 Like
10

I think your personal experience only accounts for the packages you need at the moment. Itā€™d require nixos.org to rebuild vastly, ridiculously more packages for the entirety of countless versions of nixpkgs and probably provision a second cache for the new store path.

The experience was explicitely limited to be about problems with hardcoding, and the cache problem was acknowledged.

The experience does serve as evidence that nothing in the core breaks on the store path change. As for leaf packagesā€¦ most packages are fine, and as for temporarily breaking a few leaf packages with a clear path to fixing them ā€” this happens all the time anyway.

I donā€™t think rebuilding old versions at once is discussed. And after all one can have two Nix stores if desired.

(whether the migration pain of the core use base on NixOS is worth avoiding the volume creation on macOS ā€” and well, installing NixOS also requires to set up partitions ā€” is another question)

4 Likes
11

Random thoughts:

/nix/store/ to 
/var/nix/s/

or another single char. Would make rewriting easier.

Rest of /nix/var/nix moves into /var/nix also makes sense.

Doing this on its own is a high cost, but can enable other high risk action. For example, the new location can also serve as a way to accommodate migration to an intensional store (RFCs are WIP).

1 Like
12

Iā€™ll update the top post to summarize this topic. Iā€™d love to hear some more opposing viewpoints.

13

Since macOS is used as a strong motivator here ā€“ I think it was an Apple engineer who pointed at synthetic.conf in the Catalina GitHub issue. I would rather wait if Apple extends this mechanism in 10.16. Currently, it only allows creating a top-level directory or symlinking another directory at the top-level. It is possible that Apple extends this to firmlinks in the future. E.g. /Users is ā€˜firmlinkedā€™ to the corresponing directory on the data volume (there is a full list of firmlinks in /usr/share/firmlinks).

Or another approach could be to try to approach engineers within Apple to raise our concerns. There are clearly people in Apple who care, or synthetic.conf would not exist.

Another thing on the horizon are ARM Macs. If Apple is indeed switching to ARM, which seems likely at that point, the ARM switch would be a good point to switch to some other directory for Darwin/ARM. It would be a waste to start a new binary cache now and Darwin/x86_64 getting slowly replaced in a year or so. Of course, this is assuming that macOS on ARM would open enough to allow Nix. But no one can know yet.

2 Likes
14

Those are big ifs. Apple is very tight-lipped, and they change their mind on a whim.

Starting a new binary cache is not such an ordeal; every time gcc or glibc gets updated, thatā€™s pretty much what happens.

15

It is. As others have pointed out, you will lose historical caches, which are very useful for reproducibility (especially on macOS where builds are relatively expensive, since most Mac users do not have manycore Macs lying around). And patching older binaries is not trivial. Some have already mentioned the changed path lengths [1]. Some programs have probably stored paths in UTF-16 or perhaps even compressed data.

[1] /var/nix/s is quite cryptic. Moreover, the FHS states that software should be installed into /usr:

/var is specified here in order to make it possible to mount /usr read-only. Everything that once went into /usr that is written to during system operation (as opposed to installation and software maintenance) must be in /var .

To me Nix definitely falls under installation and software maintenance. Moreover:

Applications must generally not add directories to the top level of /var . Such directories should only be added if they have some system-wide implication, and in consultation with the FHS mailing list.

So, it seems that FHS actively discourages adding directories to /var.

https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05.html#purpose31

Also, /var/lib would also be an odd place for the Nix store. It is for application state:

State information is data that programs modify while they run, and that pertains to one specific host.

The Nix store is not really state of a single program. Moreover, it does not pertain to a specific host.

It does say:

/var/lib/<name> is the location that must be used for all distribution packaging support. Different distributions may use different names, of course.

Itā€™s a bit underspecified, but I guess that refers to local package manager state, which would probably in the case of Nix be the SQLite database and gc roots.

https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s08.html

I donā€™t care too much for FHS, but if more FHS compliance is the goal, /var probably not a good location.

Edit: I also looked a bit at the competition, Silverblue/OSTree uses the top-level /sysroot directory.

2 Likes
16

Some programs have probably stored paths in UTF-16 or perhaps even compressed data.

We definitely know there are paths in compressed data. We gzip manpages, and they do contain full paths to files. The question is how to make sure there is nothing important that we donā€™t know about (so cannot special-case) and cannot detect.

2 Likes
17

Doesnā€™t this also apply to the hypothesis that /var/nix is better than a volume?

2 Likes
18

Hmm - they made root read-only, but I donā€™t know what they did with /var, and I no longer have a Mac to check.

19

Wouldnā€™t important things that break cause the automated tests to fail?

20

/var is a symlink:

āÆ ls -ld /var
lrwxr-xr-x 1 root admin 11 Mar  3 17:02 /var -> private/var

I guess that makes /var unfit for the same reasons that symlinking /nix is bad (builds that use realpath somewhere).

Apparently in Appleā€™s view only /Applications, /Library, and /usr/local should be used by third-party apps/installers (outside the userā€™s home directory, obviously):

1 Like
21

For completeness, here are the current system firmlinks in Catalina (10.15.4), which are paths that are on the writable data volume:

āÆ cat /usr/share/firmlinks
/AppleInternal	AppleInternal
/Applications	Applications
/Library	Library
/System/Library/Caches	System/Library/Caches
/System/Library/Assets	System/Library/Assets
/System/Library/PreinstalledAssets	System/Library/PreinstalledAssets
/System/Library/AssetsV2	System/Library/AssetsV2
/System/Library/PreinstalledAssetsV2	System/Library/PreinstalledAssetsV2
/System/Library/CoreServices/CoreTypes.bundle/Contents/Library	System/Library/CoreServices/CoreTypes.bundle/Contents/Library
/System/Library/Speech	System/Library/Speech
/Users	Users
/Volumes	Volumes
/cores	cores
/opt	opt
/private	private
/usr/local	usr/local
/usr/libexec/cups	usr/libexec/cups
/usr/share/snmp	usr/share/snmp