When nix-env -i is run I know it somehow adds stuff in $out/bin to the path, which I would say is one side effect. I’d like to know any other effects (the difference between nix-build and nix-install)
Edit: thanks to the responses, I’m improving the framing of this question.
Background Info: How side effects currently work
During the one-time installation of nix, the installer edits /etc/profile (and friends)
Those edits pretty much mean, when you open a shell, the bin of your nix profile get added to your PATH.
There are a few other vars. If you want a comprehensive list, find the “nix” in your /etc/profile, ~/.bashrc, etc and follow the file path every time it sources something. Its not very many lines of code.
Your active nix profile is just a folder, often located at ~/.nix-profile
if you want a more exact answer, see the sub-bullet above, or the second comment in this thread
By default nix-env --install will only change what is in the active profile. It doesn’t modify env vars or anything else
This is “profile management” and nix-env is effectively just one of many. Home Manager, Nix-Darwin, NixOS, System Manager, and nix profile are other profile managers.
Revised Question
Given nix-env --install <derivation>, which outputs of that derivation (ex: $out/bin) will get placed in the new nix profile?
Worded differently: which outputs ($out/blahblahblah) will have no effect on the profile?
I know some of the special ones: (Edit: there are no special ones, that was a misconception)
bin
sbin
include
lib
libexec
share
etc
man
But I can’t find any official list or documentation. If there isn’t any, I’d like to try and make one based on the source code.
We do this with Qt plugins being picked up from PATH. Guix has deliberate impurities with PYTHONPATH, IIRC, so if you install Python alongside some Python alongside some packages, it will see those Python packages.
GPU acceleration also requires installation of things to a profile; apps don’t directly access GPU drivers from the Nix store. That’s also why nixGL is needed on non-NixOS.
Some things read config from /etc that NixOS symlinks in there, rather than building it in each time, to avoid lots of package rebuilds that depend on your NixOS configuration. Parts of Fish initialization work this way.
I’m not aware of any systematic list of ‘deliberate impurity’, ‘dynamism’, ‘intentional side effects’, or whatever. But it would be good to articulate something about where such things are used in Nix(OS) and why.
Exploring the symlink structure of your profiles is probably a good place to start. Some of those things are there for your convenience, but many of them are there so programs will work right.
Happy to be corrected, but AFAIK nix-env -i itself has no side-effects outside of /nix.
I am splitting a hair, here, but I suspect you’re in a place where a more detailed mental model will help. Nix-env doesn’t “add” anything to the PATH (i.e., it is not changing the environment variable).
Speaking very generally, a shell profile script on a system using Nix will set up a PATH that includes one or more Nixy locations such as ~/.nix-profile and (on multi-user installs) the “default” profile at /nix/var/nix/profiles/default. Systems using home-manager, nix-darwin, and/or NixOS usually have some additional locations. These locations are in turn just symlinks to the active/current version of some profile.
Caution: I oversimplified the above. Last year the shell profiles that the installer sets up were updated to support using XDG locations in addition to the traditional ~/.nix-profile location. You can read more about this in Follow XDG Base Directory standard by balsoft · Pull Request #5588 · NixOS/nix · GitHub if you want (but I think the details are a distraction from the fundamentals here).
Again speaking very generally, the shell profiles dictate the extent to which this can be thought of to have any side effects. Without a shell profile that sets up a PATH like this, you could invoke nix-env by absolute path and see no effect. If you have a ~normal install, you can refer to the shell profiles (the source for these is in nix/scripts at master · NixOS/nix · GitHub), to see what else they set up.
I might be able to answer the question myself now if I read the related docs/code!
IDK that I would say that. I think it’s fair to say nix-env only modifies files in subdirectories of ~/.nix_profile and /nix, but the side effects of that can be wide-reaching. For example, I believe I can install a git-extension with nix, it change the behavior of my /usr/bin/git. I think a non-expert would call that an intenional side effect that exists beyond /nix.
I think it is good to point that out.
My understanding now is: nix-env installs package files to /nix/store and then takes some extra steps to modify the profile. There’s some kind of patterns (ex: $src/bin) that nix-env scans for in order to make additional modifications to the profile.
Even though it looks like there’s a lot of those cases, I want to try and find all the patterns that will trigger nix-env to do additional modifications to the profile.
Context
I’d love to have an explict installer that gives a checkbox for every binary, manpage entry, etc that comes with a package. For example, only including the git binary from the git package, or excluding the cat binary when installing coreutils.
~/.nix_profile is a symlink to a directory in /nix, so no mutation happens there either.
This is arguably true from a higher level user perspective, but the side effects here only exist because /nix has changed - everything you use that nix-env can affect lives in the nix store.
Whether the difference is meaningful from a user perspective is a different question, but @abathur is splitting hairs here specifically to point out the small flaw in your mental model of what nix-env does.
I think on NixOS this is controlled by environment.pathsToLink. Those are of course just the defaults, which may or may not change depending on what other modules you import.
Almost everything you will notice as a “side effect” gets injected into the environment via share.
I effectively never use nix-env, so I’m not sure where it gets this list from
Nix won’t easily let you get quite that granular, but in theory has multiple outputs. The feature is a bit underused in nixpkgs, sadly.
I was thinking of just using nixpkgs.linkFarm to get the granularity. It wouldn’t save any disk space, but it would at least change the effects on the profile.
That is really helpful to know. I thought the rules would be more-simple for Non-NixOS since there’s no daemons or kernel modules to worry about AFAIK, but maybe not.
If ~/.nix-profile is a symlink to a directory in the store (which is read-only) then its contents can never change – but the destination of the symlink does change when you use nix-env, to point to a different read-only directory, which is how the “mutation” happens. (I realise you understand this, I want to make the point clear to others finding this thread.)
I would characterize any time there’s a difference between what happens when you nix-build and run from the store and when you nix-env and run from your profile as ‘impurity’, at least. (Maybe you wouldn’t call it a ‘side effect’ because it’s intentional and important for intended functioning of the installed programs.)
Imo behavior that depends only on the Nix store is meaningfully different from behavior that also depends on the state of the profiles or environment variables (including PATH).
Maybe what I’m talking about is weaker and more general than side effects, though, because it’s generally (always?) not how Nix affects the environment outside of /nix, but how the outside environment affects the behavior of things that live in /nix.
At the very least, Nix installation/configuration itself does typically change some things on the underlying system— on macOS for example, /etc/bashrc and similar, plus some launchd services, a new APFS container, etc. But that’s not something subsequently affected by new installations or package removals; it’s the same for the whole install.
(In the NixOS case, installation (through nixos-rebuild) can also put stuff under /run. Plus the management of the bootloader and EFI variables isn’t limited to the Nix store.)
Yeah. Generally anything you see in a generation under /nix/var/nix/profiles is there for a purpose like that.
Those are more features of profile managers than core Nix itself. And they vary per profile manager— profile managers include Home Manager, Nix-Darwin, NixOS, System Manager, nix-env, and nix profile.
In some rare cases, packages in Nixpkgs will look in areas managed by profile managers, like /etc, or Qt packages working backwards from PATH to find plugins (which is a Nixpkgs patch). But for the most part they just interact with the symlinks that make up a profile through standard env vars like PATH and MANPATH.
But some profile managers also interface with external init systems, external package managers, the bootloader, files under /run, etc. It just depends on the implementation and its goals, constraints, needs, etc. The ones built into Nix don’t generally do any of that.
This is very enlightening. If I’m understanding correctly nix-env --install is just the basic/default profile manager and effectively its whole job (as a profile manager) is to pick out bits and pieces from derivations and merge them into a profile.
I’ve edited my original question to capture all this, and clarify the question.
Refined Question
Whether its home-manager, nix-env, or nix profile install I can’t find much official documentation for questions like:
what directories (ex: $out/bins_not_for_path/) will have no effect on the profile
of the special directories ($out/libexec) how do I know the appropriate/intended internal structure?
For example, I found a directory called “home” inside my profile’s bin directory. While writing this, I realized its from one of my own derivations. My derivation assumed only executables in $out/bin would be placed in the profile, but it turns out, no, looks like things are linked regardless.
As a starting point, I’d just like to know those answers for nix-env or nix profile install.
I saw comment in the nixpkgs source talking about bin, man, and one other special output. But I haven’t been able to find anything exhaustive or official-looking.
I guess you can look at it like this but for conceptual clarity, I think it is a good idea to consider building an environment and profile management separately.
To be more precise, a profile is just a sequence of generations, one of which is designated as the current one. Each generation corresponds to a derivation (sometimes called environment in this context), generally built from a Nix expression.
You can also build a profile-like derivation using pkgs.buildEnv. This is also what NixOS uses to build the aforementioned system path tree.
The current generation will be symlinked to a well-known path, subdirectories of which are usually advertised through environment variables. The environment variables can be set up beforehand (e.g. added to shell profile file by Nix installer) or as a part of the profile (possibly through profile activation like /etc/set-environment on NixOS).
Both home-manager and NixOS invoke nix-env --set to update the current generation of a profile to a new derivation they built.
nix-env can create and even modify a profile with imperative package management operations like --install but I am not sure how that works as I find imperative management off-putting.
This is so incredibly close to a full answer! If nix-env is using buildEnv under the hood, my only real question is: what is the default value of pathsToLink? (Assuming the user has an empty config). The literal default, in the nixpkgs source is "/", which I assume means “everything”. But I’m not sure if that is exactly what nix-env is using. There’s also some cases of [ "/" "/bin" ] in the nixpkgs source, which kind of go against my theory that “/” means everything.
Side Note: I really appreciate the re-hashing of information; for example, reading that pathsToLink code just now gave me a very different understanding compared to when I read the same code two days ago as part of TLATER’s comment. Context and mental model actually makes all the difference.
Well, Nix is independent of Nixpkgs so it would be weird if it used buildEnv from Nixpkgs (though, coincidentally, nix-shell does have dependency on Nixpkg through NIX_PATH, unfortunately).
I think that provides the last peice I needed! Please correct me if something doesn’t look right.
TLDR: Summary
(Edit: I was wrong don’t treat this as a summary)
The file-system differences between nix-env --install <derivation> and nix-build <derivation> is that, after building, nix-env links all the files from the output of the derivation into a profile directory (skipping only manifest.nix and manifest.json). See technical nuances below if you want the irrelevent details.
Beyond file-system changes, I believe the only official downstream effect (for non-NixOS) is that, because that the nix installer adds the default profile’s profile/bin to the PATH , those commands show up in the normal user-shell. But thatThat’s it, the installer doesn’t add profile-paths to MANPATH or any other vars.
NOTE: not even profile/sbin is added to PATH.
(even though some derivations put binaries in $out/sbin)
The git extensions “side effect” I saw, is actually just a futher-downstream effect of the PATH. Apparently any command that starts with “git-” that is on your PATH will get treated as a git extension.
Some man pages seem to also work this way (PATH based), although I’m still unsure of the exact mechanism
In other words: auto-complete files might be exist in profile/share/bash-completions but unless you manually tell bash to look there, you won’t actually get those completions. Effects outside of PATH (like applications on MacOS or systemd daemons on linux) are only possible if something else (like HomeManager) is making an extra effort to look inside the default nix profile.
Here’s the full technically-correct details for file system changes from nix-env --install
nix-env looks at the manifest.nix in the existing profile, then creates a new generation under /nix/store
Everything from output of the derivation is linked into the new profile directory, except for manifest.nix and manifest.json which are silently ignored
if two installed derivations both have $out/something (and those something’s are not the same thing) then nix-env overwrites it with the latest $out/something (nix profile install reports conflicts)
then nix-env creates a new manifest.nix (note: nix profile install creates a manifest.json instead, and its not allowed for both manifest.json and manifest.nix to exist at the same time)
that profile folder is now done/immutable and stored under /nix/store/-user-environment
however, in the same local directory as the profile (ex: $HOME/.local/state/nix/profiles/) it creates a new directory with an incremented number (ex: $HOME/.local/state/nix/profiles/profile-3-link) that points to the immutable profile one in /nix
then, it sets the local profile symlink (ex: $HOME/.local/state/nix/profiles/profile) to point to the local link (./profile-3-link), which in turn points to the immutable profile in /nix (ex: /nix/store/jrwhm72vs3zcdcmsmds6ilx0jcv2hs7i-user-environment/).
That’s it; no ENV var changes and no other file system changes
Side note:
Interestingly enough, buildEnv doesn’t create a manifest, and nix-env --set on a buildEnv-output creates a profile without a manifest.json or manifest.nix. Even when using the very-not-documented manifest argument of buildEnv, it then only generates a file named manifest rather than manifest.nix or manifest.json. I’m guessing the behavior is left over from a really old version of nix, and that nobody actually uses the manifest argument (which gets passed into mkDerivation, and maybe into the derivation function itself).
I do not think /sbin as described in FHS makes sense on modern Linux system, where root user is typically not accessible other than through sudo. Nixpkgs-built packages should have stuff from sbinmoved to bin, keeping sbin path only for the case something hardcodes reference to it.
I really want to make sure the final answer is exhaustive. So, looking at the latest installer scripts, it seems to me PATH, XDG_DATA_DIRS, and MANPATH are the only ones that contain paths to the profile. Are there others I’m missing?
Interesting, my nix-daemon.sh (nix 2.18) doesn’t have the XDG_DATA_DIRS or MANPATH. Same across MacOS, and my two debian machines (nix 2.12, and 2.17). Which is extra weird looking at the github source, because it looks like MANPATH was added way back in nix 2.0 (although XDG_DATA_DIRS wasn’t added till 2.19). I guess when nix upgrades on non-NixOS it doesn’t upgrade the nix-daemon.sh so I’m still using a super old one.
Thanks for pointing that out since thats the kinda the main point of the post.
There are also NIX_SSL_CERT_FILE and NIX_PROFILES. The former is at read by openssl from Nixpkgs (we patch it and someotherpackages). The latter is used for discovery of variousextensionpoints not covered by XDG_DATA_DIRS much as you described.
Oh, I guess I was only thinking of side effects outside of nix-installed executables (e.g. non-NIX env vars), but those are definitely worth discussing.
For NIX_SSL_CERT_FILE, it doesn’t include the nix profile on NixOS, MacOS, Ubuntu, Debian, Gentoo, Arch, openSUSE Tumbleweed, Old NixOS, Fedora, CentOS, Android, and OpenBSD … so … I think that just leaves Temple OS? Probably not going to cause an issue, but also probably good to have a footnote about it.
For NIX_PROFILES though, that’s major. It probably deserves a whole dedicated bullet point since its less of a side-effect and more-like and entire class of dynamic side effects on specific nix-installed packages. Now that I think about it, it would be nice if packages had a meta attribute that specified what parts of the profile(s) they looked at. I’ll have to edit that into the answer when I get a chance
Alright here’s the revised final answer to the original question, if you want to skim over it @jtojnar
Summary/Conclusion
The misleading answer is that the file-system difference between nix-env --install <derivation> and nix-build <derivation> is, after building, nix-env effectively links all the files from the derivation into your profile directory (e.g. ~/.nix-profile/blahblah will get linked to $out/blahblah of the derivaiton). That’s it, no other files are changed, no runtime-env vars are changed: see bottom of this comment if you need to know every edge case, like $out/manifest.json and $out/manifest.nix being the only skipped paths.
BUT, the complete answer is: those file changes have lots of downstream effects from different practical perspectives:
If you’re making a package: what effects can you intentionally cause? (bash completions, icons, etc)
When installing a package, how can you install just an executable (without installing all the other side effects)?
AKA: receiving global side effects, but avoid causing global side effects
When installing a package, how can you make it behave “pure”, as if nix wasn’t even installed?
AKA: avoid receiving global side effects, but still causing global side effects
When installing, how can I effectively isolate the binary, but still have it as a command?
AKA: avoid receiving and avoid causing global side effects
TLDR: Answers
If you’re writing a derivation and you want side effects, here is the most comprehensive list I know of (March 2024):
bin/ for executables
share/man/ for man pages
share/ for anything handled by XDG_DATA_DIRS. This includes bash completions, icons, themes, localization files, plugins, etc. Ask Google or ask ChatGPT about the XDG_DATA_DIRS directory structure, because it is a standard that is outside of Nix and far too big to post here.
Those^ are definite nix side effects you can cause with a derivation, but we can cause more side effects!
If you want side effects like daemons, or MacOS desktop apps, then look into profile managers such as Home Manager or Nix Darwin. They will need to be installed, but once installed they can cause extra side effects. Each will have their own method of causing those side effects, so read their docs to figure out what to put in $out of your derivation.
If you want an interaction with a specific nix package, search the nixpkgs source code on github for the package, and look for $NIX_PROFILES. If the package uses that, then you’re in luck. And if, for example, it uses the share/blah part of the nix profile, then the $out/share/blah of your derivation is where you should put the file to cause the side effect.
If you want a command from a package but don’t want any other side effects (e.g. no desktop icons, man pages, completions, etc)
(e.g. receive effects but not cause them)
Use nix-build, and find the /nix/store path
Now you’ve basically got two options
Option 1: Some side effects, but probably the effects you want: Add $that_nix_build_path/bin/ to your PATH. This avoids unknown-side effects like desktop icons, and autocompletautocomplete but be warned, other programs/commands will behave differently. FOR EXAMPLE if, the dervivation has an ls command that, for some reason, behaves like catand then put you put that on your PATH, you’re gonna notice other commands behaving differently real quick™. And, even without overriding a standard command, side effects like git extensions and man pages will still happen/work because often man finds the pages based on PATH and git extensions are simply any command on the PATH that start with git-.
Option 2: If you want a lot more isolation than that, for example, not breaking your whole system while still having an ls that behaves like cat; make a shell alias or shell function (in your bashrc, zshrc, etc) that calls the binary directly (e.g. /nix/store/<hash>-<name/bin/ls). The command will effectively be invisible to everything other than shell scripts. This is the most amount of isolation you can get while still sometimes having a practical usecase.
If you want the binary of a derivation to cause global side effects but not *receive effects (e.g. the binary will practically behave as if nix and nixpkgs were not even installed)
Just FYI, you shouldn’t want to do this basically ever (maybe sandbox testing? maybe?) but for education/understanding sake I’ll give the answer.
You’ll need to make a wrapper derivation around the original derivation that has the executable you’re trying to install.
In that wrapper derivation, you’ll need to copy the $out of the original, and overwrite each executable in $out/bin with its own wrapper executable
That wrapper will need to:
unset the NIX_PROFILES var
remove nix-profile path part from PATH, XDG_DATA_DIRS, and MANPATH
then call the original executable
Using nix-env, you can install that wrapper derivation and it will give you executables that behave as if none of your nix-commands or autocompletions or man pages are installed, INCLUDING not even knowing about itself. However, other nix-commands will still behave differently (they’ll receive autocompletions and man-pages of the original derivation). Again, I can’t think of a usecase.
How to completly isolate a command?
Just do both of the answers above at the same time.
Why
All the reliable side effects are because the one-time nix installer adds your default nix profile to some ENV vars.
nix 1.x added the profile to NIX_PROFILES and $profile/bin to PATH …
but as of 2.0 it also adds $profile/share/man/ to MANPATH
and as of 2.19 it also adds $profile/share/ to XDG_DATA_DIRS
Remove those, and there are effectively no intentional side effects.
Footnotes:
not even profile/sbin is added to PATH
(even though some derivations put binaries in $out/sbin)
The nix installer can sometimes add the profile to one other env var; NIX_SSL_CERT_FILE. However, this only happens if the OS is NOT NixOS, MacOS, Ubuntu, Debian, Gentoo, Arch, openSUSE Tumbleweed, Fedora, CentOS, Android, or OpenBSD. So its extremely rare, and is really just a last-ditch effort to find any cert file
Here’s the full technically-correct details for file system changes from nix-env --install
nix-env looks at the manifest.nix in the existing profile, then creates a new generation under /nix/store
Everything from output of your being-installed-derivation is linked into the new profile directory, except for manifest.nix and manifest.json which are silently ignored
if your new derivation has $out/something but it already existed from an old derivation (and the new $out/something is not the same as the old one) then nix-env overwrites it with the latest $out/something
In contrast to nix-env --install the newer nix profile install will reports conflicts instead of overwriting with the new one.
then nix-env creates a new manifest.nix (note: nix profile install creates a manifest.json instead, and its not allowed for both manifest.json and manifest.nix to exist at the same time)
that profile folder is now done/immutable and stored under /nix/store/-user-environment
however, in the same local directory as the profile (ex: $HOME/.local/state/nix/profiles/) it creates a new directory with an incremented number (ex: $HOME/.local/state/nix/profiles/profile-3-link) that points to the immutable profile one in /nix
then, it sets the local profile symlink (ex: $HOME/.local/state/nix/profiles/profile) to point to the local link (./profile-3-link), which in turn points to the immutable profile in /nix (ex: /nix/store/jrwhm72vs3zcdcmsmds6ilx0jcv2hs7i-user-environment/).
That’s it; AFAIK there are no ENV var changes and no other file system changes
