/nix => /var/nix, /opt/nix, /usr/local/nix?

18

Hmm - they made root read-only, but I don’t know what they did with /var, and I no longer have a Mac to check.

19

Wouldn’t important things that break cause the automated tests to fail?

20

/var is a symlink:

❯ ls -ld /var
lrwxr-xr-x 1 root admin 11 Mar  3 17:02 /var -> private/var

I guess that makes /var unfit for the same reasons that symlinking /nix is bad (builds that use realpath somewhere).

Apparently in Apple’s view only /Applications, /Library, and /usr/local should be used by third-party apps/installers (outside the user’s home directory, obviously):

1 Like
21

For completeness, here are the current system firmlinks in Catalina (10.15.4), which are paths that are on the writable data volume:

❯ cat /usr/share/firmlinks
/AppleInternal	AppleInternal
/Applications	Applications
/Library	Library
/System/Library/Caches	System/Library/Caches
/System/Library/Assets	System/Library/Assets
/System/Library/PreinstalledAssets	System/Library/PreinstalledAssets
/System/Library/AssetsV2	System/Library/AssetsV2
/System/Library/PreinstalledAssetsV2	System/Library/PreinstalledAssetsV2
/System/Library/CoreServices/CoreTypes.bundle/Contents/Library	System/Library/CoreServices/CoreTypes.bundle/Contents/Library
/System/Library/Speech	System/Library/Speech
/Users	Users
/Volumes	Volumes
/cores	cores
/opt	opt
/private	private
/usr/local	usr/local
/usr/libexec/cups	usr/libexec/cups
/usr/share/snmp	usr/share/snmp
22

Thanks! So it seems that it should be either /opt/nix or /usr/local/nix.

An option is to put the store under /opt/nix and the profiles etc under /var/lib/nix. Store path rewrites could then use /opt/nix/./ which is the same length as /nix/store/.

If the store moves to /usr/local/nix, that means stealing 3 characters from the end of store paths when rewriting. This isn’t a problem. Moving it to /usr/local/nix/store will steal 10 characters, which is getting a bit much. It’s also not nice to have 20 extra characters per store path in environments and wrappers.

99.9% of applications is totally fine with /nix being a symlink. So for these historical applications, not patching but having the symlink is definitely an option.

1 Like
23

Obviously no. We do not have functional tests for each package, and even if we had, we would not catch, I dunno, whether LibreOffice loads some resources referenced inside a JAR specifically when drawing a chart.

24

Just adding data points: Fedora Silverblue is supposed to replace traditional Fedora at some point. IIRC /opt used to be a symbolic link in Silverblue, now /opt is synthesized during boot:

https://github.com/coreos/rpm-ostree/pull/1795

/usr/local is also a symlink into /var :confused: .

Note that Nix currently does not work on Fedora Silverblue either unless you hack your way around it or use a mutable Fedora container, because / is immutable. (Or if you run Nix in a user namespace.)

IIRC in Silverblue /var is actually the only thing that is not a symlink and that is not immutable. So, sort-of the opposite of macOS.

Before making a fleshed-out proposal it may be good to talk to the Fedora/Silverblue folks as well. (If the Nix community cares about Fedora.)

2 Likes
25

What if we decided to change the default nix store location only for macOS builds ?

The darwin package tree is already completely distinct from the i386 and x86_64 trees. Built packages have nothing in common.

Advantages are

  1. It only impacts MacOS users for a change that is required by that OS
  2. It makes nixpkgs even more flexible because we would regularly build and test everything with two default stores.
  3. It does not incur such a high penalty for NixOS where configurations are even more likely to depend on hard-coded /nix/store store.

Of course it does not account for other distros like fedora going the same way. But the point about nixos remains valid.

6 Likes
26

It still has the same issue of having to create a separate cache just for macos and rebuilding everything from countless versions of nixpkgs with no reasonable way to hack around it. Plus it’s not totally unheard of for people to want linux derivations in a macos store; uncommon for sure, but occasionally useful.

1 Like
27

Ok, but those people can also set up a symlink for /nix and make it work that way. I know that some packages won’t work that way, but those are rare.

28

As a data point: rg '/nix/store' | wc -l returns 471 lines. A random
one picked in that list is in [1]. As for
rg '[^.a-zA-Z]/nix[^a-zA-Z]' | wc -l, it returns 770 results,
including things like [2]

[1] nixos/modules/system/boot/loader/grub/install-grub.pl
[2] pkgs/development/tools/ocaml/opam/default.nix

Also, quite a few of these are comments, or just unrelated to the actual
search. So it’s not great, but it’s totally fixable – at least if
assuming replacement by a same-length path, as there may be path length
assumptions hardcoded elsewhere that’d be much harder to grep for.

29

If we move /nix in an FHS folder like /opt… Don’t you think we may get some problems when creating sandboxes using buildFHSUserEnv ? I mean, what will contain /opt inside this sandbox ?
Just asking…

30

FWIW, Flatpak (which Fedora is incorporating heavily, especially for its root-is-immutable variant, Silverblue) puts its equivalent of a store in /var/lib/flatpak, so /var/nix or /var/lib/nix does not seem to stray so far from what the de facto standard is.

/var/lib/flatpak
❯ find . -name 'telegram-desktop'
./app/org.telegram.desktop/x86_64/stable/5f2124178d2040bea40a58d326481dbc81895c61e4a299ccd6772b2a13693a8d/files/bin/telegram-desktop
./.removed/org.telegram.desktop-cd353d12c1ea661554a55485e34996c85a67277bd6c5e5047d8ee9240242c8fd/files/bin/telegram-desktop
1 Like
31

This runs the risk of having to continually chase down minor incompatibilities where some package hardcodes /nix/store and is only properly tested on non-macOS platforms.

2 Likes
32

Are there any analytics or telemetry associated with how often a particular platform uses Nix? I say this because there are often cases of sampling bias (somewhat related but actual numbers https://twitter.com/bgolus/status/1080213166116597760)

It might be important to figure out where best to devote resources. E.g. anyone using Fedora silverblue would not be using Nix with a high probability. Same might go for machinetosh.

Just an idea (cheaper than I think making cache changes)- It might help if everytime a package was downloaded the nix client sent a copy of /etc/os-release and uname -a to an open pastebin. A package download is a good measure of actual usage I think, as opposed to a one-time message.

1 Like
33

Why do we need to have telemetry to make this decision?

We used to support MacOS; we have binary caches for them; why drop them?

1 Like
34

I’m not a part of “we”, I’m just some guy making a suggestion as to finding a way to best decide whom to support and whom not to. I don’t think this discussion is about dropping support for apple os (which I’m not suggesting), it’s only about how to go forward with it.

35

We used to support MacOS; we have binary caches for them; why drop them?

There are build machines running Catalina. So it is not about dropping support for macOS, it is about how much annoyance to create to the more numerous NixOS users (all of whom did some partition management during installation) so that the macOS users are not required to do any partition management for Nix use.

36

Anecdotally, at the NixCon in London, there was someone working at CERN who wanted to get everyone there using Nix, but had the problem that getting /nix on the network computers was definitely not an option. How many other organisations faced the same problem and ended up not using Nix?

I think it will not be an issue, we just have to be careful.

Exactly. From an evangelist standpoint, macOS users are a significant part of the possible NixOS user market (for web server etc) and Nix is the gateway drug. More users means more helping hands and a better Nix(OS) for everyone.

OTOH, having a non-trivial part of the user base be on a different store path ensures that nixpkgs is always ready to run from any location, that seems useful as well.

Since Silverblue is Linux, using bind mounts is an option. Also, it’s open source, so we can ask them to make a writeable /opt/nix a possibility.

2 Likes
37

how much annoyance to create to the more numerous NixOS users (all of whom did some partition management during installation) so that the macOS users are not required to do any partition management for Nix use.

I might be projecting, but this at least sounds a little pejorative and dismissive. To be a little more fair–this subset of the question is about installing a package manager, not an OS. Some things to weigh:

  • If you installed macOS from scratch, there’s some chance you’ll have to do a little partition management.
  • It’s a bit weird for user-installed software to automatically create a partition, but we’re preparing to do just that for UX reasons.
    • If there are orgs already saying No to Nix because of /nix, creating a partition will probably only increase that likelihood.
    • In mixed-OS orgs, unless the decision-makers have a nuanced understanding of this issue, they might say No to Nix or NixOS organization-wide, rather than just to Nix on macOS.
  • Automatically creating a partition and bootstrapping it for use is a substantive change to Nix being so simple you can roll down your window and throw it out with rm -rf /nix. People who go run rm -rf /nix and still see a Nix Store in finder are going to be (justifiably) confused (and perhaps excusably suspicious).
1 Like