# /nix =\> /var/nix, /opt/nix, /usr/local/nix?
**URL:** https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101
**Category:** Uncategorized
**Created:** [May 8, 2020, 4:06pm UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101 "2020-05-08T16:06:01Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@wmertens](https://discourse.nixos.org/u/wmertens)
#### Post date: [May 8, 2020, 4:06pm UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/1 "2020-05-08T16:06:02Z")
Summary, see original post below
Proposal: move `/nix` to:
- `/opt/nix`: store directory
- `/var/lib/nix/{profiles/, gc.lock, ...}`: everything that now lives under `/nix/var/nix`
- `/var/log/nix`: Nix log files
| Benefit | |
| --- | --- |
| works with macOS | Unlocks large user base |
| matches FHS | Allows easier installation on existing systems |
| | Easier to get buy-in from administrators |
| Shorter store path | 6 less characters per referenced path probably adds up to a savings of a few KB in the store š¤” |
| Problem | Solutions |
| --- | --- |
| Extra work, no benefit on current installs | Making installation easier on other systems grows the ecosystem, which benefits Nix |
| Requires rebuilding all | This regularly happens anyway |
| Historical packages only available via `/nix` | They can be recompiled or patched, and `/nix` still works for a parallel or shared install |
| Hardcoded /nix in places | Fix the broken code |
| Needs 2 Hydra builds | Bind-mount the store, register both groups of packages. Since the prefix is part of the input hash, there should be no clashes, and invariant packages will be shared, invariant files hardlinked |
| How to migrate | Write tool to do the migration, everything else remains the same |
| `clang` resolves symlinks, [causing impurity](https://github.com/NixOS/nix/issues/2925#issuecomment-522482497) | Make sure the store is on a non-symlinked path |
| `/var` is a symlink on macOS | Use /opt instead |
| `/opt` on Fedora Silverblue is immutable, and `/usr/local` is a symlink | The userbase is currently much smaller and there are likely a number of mitigations possible |
| `/nix` is used all over the documentation | Letās update the documentation, use e.g. `$NIX_ROOT`, or, since this proposal removes the single root, `$NIX_STORE` |
## Original post
Up till now, having the Nix store and metadata live at `/nix` was a mild inconvenience, and maybe a cute quirk.
However, [macOS has been clamping down on writing to /](https://github.com/NixOS/nix/issues/2925), and I have to wonder if all the trouble is worth it. Why not move `/nix` to a [Filesystem Hierarchy Standard](https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard) location, that will be more likely to be available?
I can think of these locations:
- `/var/nix`: /var is for Variable data, and the Nix store is surely variable, any user with access to nix-shell can cause additions in there.
- `/var/lib/nix`: /var/lib is for database storage, and the Nix store is database if you squint
- `/opt/nix`: /opt is for Optional installs, and except on NixOS, thatās the case
- `/usr/local/nix`: /usr/local is for Local User installs, pretty apt for single-user nix installs
My favorite among these is `/var/nix`, about all systems have a `/var`, and itās totally ok to have an extra directory in there. Itās nicely out of the way and doesnāt disturb anyone. `/var/lib/nix` is truly conformant to FHS but a little more hidden.
/opt and /usr/local are not as likely to be existing on all systems.
While weāre at it, it would be nice to flatten the `/nix/var/nix` tree. Just have its children directly under `nix/`, wherever it is, and preferably symlink the log to `/var/log/nix` if possible.
To migrate to the new location, we can treat it as a new Nix installation, and either install the same packages, or rewrite store references in the existing ones (cutting some characters from their store path) so they run from the new location. Where possible, you can even bind-mount a single dir on both locations during a transition period, so unchanged files can be hardlinked together.
---
### Author:  [@Ekleog](https://discourse.nixos.org/u/Ekleog)
#### Post date: [May 8, 2020, 8:20pm UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/2 "2020-05-08T20:20:54Z")
Iād be all in favor of this change, even if it means some
synchronization between hydra changes and a nix update.
The thing thatād scare me most is the likely number of places where we
have hardcoded `/nix/store`.
While weād be at it, itād probably be possible to also do things that
couldnāt be done otherwise because itād require store changes, like
making the store hierarchical to reduce the hurt it inflicts on quite a
few scripts that donāt expect hundreds of thousands of entries in a
single directory.
(Not that I expect this to actually happen, but it does sound like a
relatively good idea, even if it means that hydra will be twice as slow
for the duration of the migration period)
Wout Mertens via NixOS Discourse [nixos1@discoursemail.com](mailto:nixos1@discoursemail.com) writes:
---
### Author:  [@ElvishJerricco](https://discourse.nixos.org/u/ElvishJerricco)
#### Post date: [May 8, 2020, 11:53pm UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/3 "2020-05-08T23:53:08Z")
As nice as this might be, it would literally destroy pretty much every existing binary cache. Weād lose the ability to use old pinned nixpkgs basically entirely without rebuilding the world, breaking countless non-updated projects. A hydra migration for [cache.nixos.org](http://cache.nixos.org) would beā¦ _really, really_ expensive, and would break the cache for anyone with an older version of nix. Logistically this idea is just unrealistically difficult to pull off.
---
### Author:  [@bhipple](https://discourse.nixos.org/u/bhipple)
#### Post date: [May 9, 2020, 1:34am UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/4 "2020-05-09T01:34:42Z")
> The thing thatād scare me most is the likely number of places where we
> have hardcoded `/nix/store` .
Iāve been running nix in a non-standard store location for several years without much issue, so at least on this front I think weād be fine. The bigger issue is that weād have to go through a painful migration period and weād lose all of our binary cache history. Iād personally be up for a move to `/opt/nix` or `/var/nix`, but I can understand why many wouldnāt, and itās certainly not a trivial task operationally.
---
### Author:  [@ElvishJerricco](https://discourse.nixos.org/u/ElvishJerricco)
#### Post date: [May 9, 2020, 3:11am UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/5 "2020-05-09T03:11:32Z")
I think your personal experience only accounts for the packages you need at the moment. Itād require [nixos.org](http://nixos.org) to rebuild vastly, _ridiculously_ more packages for the entirety of countless versions of nixpkgs and probably provision a second cache for the new store path.
---
### Author:  [@wmertens](https://discourse.nixos.org/u/wmertens)
#### Post date: [May 9, 2020, 6:53am UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/7 "2020-05-09T06:53:44Z")
> [@Ekleog](#):
>
> While weād be at it, itād probably be possible to also do things that
> couldnāt be done otherwise because itād require store changes, like
> making the store hierarchical to reduce the hurt it inflicts on quite a
> few scripts that donāt expect hundreds of thousands of entries in a
> single directory.
It really is more efficient to put it all in one dir. If you split it up by the first few characters, then youāll have a bunch of dirs with one or two entries, making the lookups slower. If you use less characters, then youāre just moving the problem to a different dir. `ls` handles my 11k entries just fine.
---
### Author:  [@wmertens](https://discourse.nixos.org/u/wmertens)
#### Post date: [May 9, 2020, 6:55am UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/8 "2020-05-09T06:55:57Z")
The old cache would still work for the old location, and there could be a rewrite pass to move the packages to the new location (losing some filename characters).
---
### Author:  [@ElvishJerricco](https://discourse.nixos.org/u/ElvishJerricco)
#### Post date: [May 9, 2020, 7:56am UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/9 "2020-05-09T07:56:22Z")
You couldnāt expect a rewrite pass to be effective.
1. There are packages that donāt encode paths as strings in a way that could be detecting by a find and replace on all the files.
2. There are tons of file formats that encode the length of strings, and changing to a longer string would make the file invalid.
---
### Author:  [@7c6f434c](https://discourse.nixos.org/u/7c6f434c)
#### Post date: [May 9, 2020, 8:09am UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/10 "2020-05-09T08:09:55Z")
> I think your personal experience only accounts for the packages you need at the moment. Itād require [nixos.org](http://nixos.org) to rebuild vastly, _ridiculously_ more packages for the entirety of countless versions of nixpkgs and probably provision a second cache for the new store path.
The experience was explicitely limited to be about problems with hardcoding, and the cache problem was acknowledged.
The experience does serve as evidence that nothing in the core breaks on the store path change. As for leaf packagesā¦ most packages are fine, and as for temporarily breaking a few leaf packages with a clear path to fixing them ā this happens all the time anyway.
I donāt think rebuilding _old_ versions at once is discussed. And after all one _can_ have two Nix stores if desired.
(whether the migration pain of the core use base on NixOS is worth avoiding the volume creation on macOS ā and well, installing NixOS also requires to set up partitions ā is another question)
---
### Author:  [@tomberek](https://discourse.nixos.org/u/tomberek)
#### Post date: [May 9, 2020, 9:58am UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/11 "2020-05-09T09:58:21Z")
Random thoughts:
```auto
/nix/store/ to
/var/nix/s/
```
or another single char. Would make rewriting easier.
Rest of /nix/var/nix moves into /var/nix also makes sense.
Doing this on its own is a high cost, but can enable other high risk action. For example, the new location can also serve as a way to accommodate migration to an intensional store (RFCs are WIP).
---
### Author:  [@wmertens](https://discourse.nixos.org/u/wmertens)
#### Post date: [May 9, 2020, 11:58am UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/12 "2020-05-09T11:58:23Z")
Iāll update the top post to summarize this topic. Iād love to hear some more opposing viewpoints.
---
### Author:  [@danieldk](https://discourse.nixos.org/u/danieldk)
#### Post date: [May 9, 2020, 1:02pm UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/13 "2020-05-09T13:02:23Z")
Since macOS is used as a strong motivator here ā I think it was an Apple engineer who pointed at `synthetic.conf` in the Catalina GitHub issue. I would rather wait if Apple extends this mechanism in 10.16. Currently, it only allows creating a top-level directory or symlinking another directory at the top-level. It is possible that Apple extends this to firmlinks in the future. E.g. `/Users` is āfirmlinkedā to the corresponing directory on the data volume (there is a full list of firmlinks in ` /usr/share/firmlinks`).
Or another approach could be to try to approach engineers within Apple to raise our concerns. There are clearly people in Apple who care, or `synthetic.conf` would not exist.
Another thing on the horizon are ARM Macs. If Apple is indeed switching to ARM, which seems likely at that point, the ARM switch would be a good point to switch to some other directory for Darwin/ARM. It would be a waste to start a new binary cache now and Darwin/x86\_64 getting slowly replaced in a year or so. Of course, this is assuming that macOS on ARM would open enough to allow Nix. But no one can know yet.
---
### Author:  [@wmertens](https://discourse.nixos.org/u/wmertens)
#### Post date: [May 9, 2020, 1:38pm UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/14 "2020-05-09T13:38:37Z")
Those are big ifs. Apple is very tight-lipped, and they change their mind on a whim.
Starting a new binary cache is not such an ordeal; every time gcc or glibc gets updated, thatās pretty much what happens.
---
### Author:  [@danieldk](https://discourse.nixos.org/u/danieldk)
#### Post date: [May 9, 2020, 6:21pm UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/15 "2020-05-09T18:21:57Z")
> [@wmertens](#):
>
> Starting a new binary cache is not such an ordeal;
It is. As others have pointed out, you will lose historical caches, which are very useful for reproducibility (especially on macOS where builds are relatively expensive, since most Mac users do not have manycore Macs lying around). And patching older binaries is not trivial. Some have already mentioned the changed path lengths [1]. Some programs have probably stored paths in UTF-16 or perhaps even compressed data.
[1] `/var/nix/s` is quite cryptic. Moreover, the FHS states that software should be installed into `/usr`:
> `/var` is specified here in order to make it possible to mount `/usr` read-only. Everything that once went into `/usr` that is written to during system operation (as opposed to installation and software maintenance) must be in `/var` .
To me Nix definitely falls under installation and software maintenance. Moreover:
> Applications must generally not add directories to the top level of `/var` . Such directories should only be added if they have some system-wide implication, and in consultation with the FHS mailing list.
So, it seems that FHS actively discourages adding directories to `/var`.
[https://refspecs.linuxfoundation.org/FHS\_3.0/fhs/ch05.html#purpose31](https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05.html#purpose31)
Also, `/var/lib` would also be an odd place for the Nix store. It is for application state:
> State information is data that programs modify while they run, and that pertains to one specific host.
The Nix store is not really state of a single program. Moreover, it does not pertain to a specific host.
It does say:
> `/var/lib/` is the location that must be used for all distribution packaging support. Different distributions may use different names, of course.
Itās a bit underspecified, but I guess that refers to local package manager state, which would probably in the case of Nix be the SQLite database and gc roots.
[https://refspecs.linuxfoundation.org/FHS\_3.0/fhs/ch05s08.html](https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s08.html)
I donāt care too much for FHS, but if more FHS compliance is the goal, `/var` probably not a good location.
Edit: I also looked a bit at the competition, Silverblue/OSTree uses the top-level `/sysroot` directory.
---
### Author:  [@7c6f434c](https://discourse.nixos.org/u/7c6f434c)
#### Post date: [May 9, 2020, 6:57pm UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/16 "2020-05-09T18:57:37Z")
> Some programs have probably stored paths in UTF-16 or perhaps even compressed data.
We definitely know there are paths in compressed data. We gzip manpages, and they do contain full paths to files. The question is how to make sure there is nothing important that we donāt know about (so cannot special-case) and cannot detect.
---
### Author:  [@7c6f434c](https://discourse.nixos.org/u/7c6f434c)
#### Post date: [May 9, 2020, 7:09pm UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/17 "2020-05-09T19:09:26Z")
> [@wmertens](#):
>
> Those are big ifs. Apple is very tight-lipped, and they change their mind on a whim.
Doesnāt this also apply to the hypothesis that `/var/nix` is better than a volume?
---
### Author:  [@wmertens](https://discourse.nixos.org/u/wmertens)
#### Post date: [May 10, 2020, 6:51am UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/18 "2020-05-10T06:51:53Z")
> [@7c6f434c](#):
>
> Doesnāt this also apply to the hypothesis that `/var/nix` is better than a volume?
Hmm - they made root read-only, but I donāt know what they did with /var, and I no longer have a Mac to check.
---
### Author:  [@wmertens](https://discourse.nixos.org/u/wmertens)
#### Post date: [May 10, 2020, 6:53am UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/19 "2020-05-10T06:53:13Z")
> [@7c6f434c](#):
>
> The question is how to make sure there is nothing important that we donāt know about (so cannot special-case) and cannot detect.
Wouldnāt important things that break cause the automated tests to fail?
---
### Author:  [@danieldk](https://discourse.nixos.org/u/danieldk)
#### Post date: [May 10, 2020, 8:38am UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/20 "2020-05-10T08:38:43Z")
> [@wmertens](#):
>
> Hmm - they made root read-only, but I donāt know what they did with /var, and I no longer have a Mac to check.
`/var` is a symlink:
```auto
āÆ ls -ld /var
lrwxr-xr-x 1 root admin 11 Mar 3 17:02 /var -> private/var
```
I guess that makes `/var` unfit for the same reasons that symlinking `/nix` is bad (builds that use `realpath` somewhere).
Apparently in Appleās view only `/Applications`, `/Library`, and `/usr/local` should be used by third-party apps/installers (outside the userās home directory, obviously):
> **[About System Integrity Protection on your Mac - Apple Support](https://support.apple.com/en-us/102149)**
>
> System Integrity Protection is a security technology that helps protect your Mac from malicious software.
---
### Author:  [@danieldk](https://discourse.nixos.org/u/danieldk)
#### Post date: [May 10, 2020, 8:47am UTC](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/21 "2020-05-10T08:47:19Z")
For completeness, here are the current system firmlinks in Catalina (10.15.4), which are paths that are on the writable data volume:
```auto
āÆ cat /usr/share/firmlinks
/AppleInternal AppleInternal
/Applications Applications
/Library Library
/System/Library/Caches System/Library/Caches
/System/Library/Assets System/Library/Assets
/System/Library/PreinstalledAssets System/Library/PreinstalledAssets
/System/Library/AssetsV2 System/Library/AssetsV2
/System/Library/PreinstalledAssetsV2 System/Library/PreinstalledAssetsV2
/System/Library/CoreServices/CoreTypes.bundle/Contents/Library System/Library/CoreServices/CoreTypes.bundle/Contents/Library
/System/Library/Speech System/Library/Speech
/Users Users
/Volumes Volumes
/cores cores
/opt opt
/private private
/usr/local usr/local
/usr/libexec/cups usr/libexec/cups
/usr/share/snmp usr/share/snmp
```
[Next page](https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101.md?page=2)