Enabling default LD library search path

Currently, NixOS has default LD library path disabled: nixpkgs/pkgs/development/tools/misc/binutils/default.nix at 3df3c47c19dc90fec35359e89ffb52b34d2b0e94 · NixOS/nixpkgs · GitHub

This prevents nix-ld from being able to “fix” unpatched shared library loading in the same way it fixes unpatched binary execution, as seen from users complaining here and here. The only recourse with LD being built as it is is doing export LD_LIBRARY_PATH=$NIX_LD_LIBRARY_PATH, which is not great because LD_LIBRARY_PATH takes precedence over RUNPATHs provided by nix in patched executables. Re-enabling /usr/lib in LD as a fallback would allow handling this properly.

While I realize this is not the “nix way”, as a user, I want things to “just work”. Other than having to rebuild the world, this seems like a tiny change with great usability benefits. NixOS provides /usr/bin/env to improve usability. Providing /usr/lib with user selected libraries does not seem that different.

This should obviously not affect the building of nix packages - i.e. nix packages should continue to never reference anything out of /usr/lib. But I believe sandboxing already takes care of that.

Everything ‘just works’ in a Distrobox.

Yes, Distrobox is great. However that’s basically saying “NixOS is not usable for most development work, so let’s run a container on top”. I think there is value in making NixOS more usable directly.

Nah, I do plenty of development work in NixOS directly. But it’s an environment that places an emphasis on purity. If you value ‘work right now’ over ‘ensure everything does the pure thing, no exceptions’, you should be working in an environment that is aligned with your values. You can do that from within NixOS, without compromising its purity for everyone else, with a Distrobox.

But NixOS already allows exceptions to purity in favor of usability - /usr/bin/env is an exception, nix-ld is an exception, etc. I am proposing that /usr/lib is a valuable exception, especially if it needs to be explicitly enabled.

What is “valuable” is obviously subjective.

/usr/bin/env is harmless. The odds of getting in hot water because a script invokes the wrong version of that binary are infinitesimal. nix-ld is a thing like Distrobox (though not as comprehensive)—use it if you want, it doesn’t change how the system works for anyone else. Adding an escape hatch to the default loader that everyone uses is a horse of a different color from either of those things.

2 Likes

Why do you think /usr/lib default search path is so different? It would remain empty / non-existent unless populated by nix-ld or similar module, and would have no impact to users that did not choose to use it.

And what makes you so sure of that? I guarantee you that some newish NixOS user will run sudo cool_app_installer.sh per the instructions on CoolApp’s website, and that installer will, with its root privileges, put libraries in /usr/lib. Then three years later they’ll show up in this forum with some impossible behavior caused by a CoolApp library being loaded from that path.

1 Like

That installer can already do that today by setting - or asking user to set - LD_LIBRARY_PATH. If accidental population is a concern, it’s better to have /usr/lib pointing to an empty dir in nix store so it has all the same protections as the rest of nix packages.

The whole point of nix is correct deployment, not global search paths. NixOS is therefore the “correctly-deployed” distro. /usr/lib is literally against the entire purpose of nix and NixOS.
I lump nix-ld into this as well.

I consider getting rid of /usr/lib a massive usability benefit as I don’t have to deal with the nonsense I had to deal with on Arch, for example.

As for software that uses /bin/sh or /usr/bin/env, I wrote a simple shim to log such cases to the system journal so that I know which programs to fix. So my system doesn’t even need those.

2 Likes

Perhaps the people who use NixOS for supposed convenience instead of the abilities it actually provides should simply not use NixOS.

1 Like

The problem is that it won’t “just work” really. Noone will be managing compatibility of these the impure libs with the rest.

I think that’s silly. While a great benefit, people should be able to use NixOS for reasons besides its packaging guarantees.

However, this suggestion is fundamentally changing the way packages are built. Packages in nixpkgs should work on other distros too, without libraries in those distros interfering. Adding state to NixOS packages that isn’t immediately obvious from the configuration is wildly beyond reasonable anyway. If you share your configuration I should be reasonably certain I know how your system works, this breaks that on a fundamental level.

Plus, there are plenty alternatives through fhs envs, distrobox, plain bubblewrap, nix-ld, … None of these fundamentally break nixpkgs guarantees, and they’re all quite easy to use, with nix-ld providing exactly the same ease of use and benefits as this suggestion. Just use them instead.

What reasons are those? If they just want IaC there’s existing tooling for that that doesn’t require NixOS (and I don’t think even the existing NixOS deployment tools achieve IaC in a particularly stellar manner :grimacing: — though I haven’t tried all 20+ tools.)

Setting up the expectation that NixOS is really designed for these nix-ld-esque hacks is setting users up for failure IMO.

2 Likes

IMO NixOS is among the best of the bunch partially on account of precisely the lack of fhs (which is important, but complete purity isn’t necessary for the use case - the culture of it is sufficient). Most other IaC tools can’t reliably do the boot-on-tmpfs thing NixOS does so well, or inherently depend on containers to do anything, resulting in infrastructure you can’t actually deploy without first deploying infrastructure to deploy your infrastructure to.

Plus, among them, it’s basically the only one that is even remotely suited to desktop use. I find it’s also very good at low-maintenance small bare-metal server hosting. These use cases are largely overlooked by all the other IaC tools (with the notable exception of ansible, and maybe yocto for a specific subset of these use cases, but see above paragraph) - they tend to target big businesses and cloud deployments exclusively.

I’ve certainly not used all of them either, and I’m sure the number is orders of magnitude larger than 20, but NIxOS is the only one that has really met my needs.

Ok, long-form answer here because this is the second time this has come up, and so far nobody has voiced why nix-ld is such a problem to me.

Personally, I think nix-ld strikes a decent balance. It’s no worse than what NixOS does today - NixOS already deploys a shim dynamic linker to tell you why your binaries aren’t working. nix-ld does exactly the same, unless you specify the NIX_LD/NIX_LD_LIBRARY_PATH variables, which require explicit setting because the NIX_LD prefix doesn’t make them overlap with anything.

It beats buildFhsUserEnv & co. for some use cases, because it doesn’t require writing a full nix package and doesn’t deploy into a container. This means:

  • I can run software that is allergic to containers with it.

  • I can get stuff to work without the time investment of writing a package by hand. Just put whatever libraries my binary complains about in a shell.nix, run nix-shell, and it works. This can be significantly faster than untangling an override of a package which internally uses buildFhsUserEnv, but e.g. tries to run software downloaded at runtime which uses libraries not part of its normal closure.

  • There’s no separate build step that wraps binaries or anything, so I can just put a self-updating application (curse you, Von Neumann) in one of these shells and it’ll work as intended by its designer.

  • Finally, and most importantly to my mind, it helps users who are new to the ecosystem actually adopt NixOS without having to fully understand nix (the language) right off the bat. Putting a few libraries in shell.nix is much easier to grok than using buildFhsUserEnv (especially when you add nix-alien).

    This may just be a weakness of the buildFhsUserEnv API, to be fair, a simplified version of it could achieve similar things (and nix-alien basically provides that) - but then you run into issues with things that don’t like being containerized.

And what’s the trade-off? Well, a subtly different dynamic linker shim which users could abuse to break NixOS guarantees by setting a variable globally. It can’t interfere with builds, even if for some reason you put it somewhere that lacks sandboxing, since variables don’t propagate there, so there’s no risk of the contamination spreading even if abused this way. I don’t see how this is setting up users for failure, the caveat to its use is tiny.

On that note, perhaps your problem with nix-ld isn’t actually nix-ld, but the programs.nix-ld NixOS module. For some reason we’re setting global variables with it, which does in fact completely break NixOS guarantees, and programs.nix-ld.enable is so easy to just do without understanding anything that I’d call it a footgun. We really oughta stop providing footguns.

Because setting a magic envvar to create a global library search path is exactly not how to package things with nix. Sending people into NixOS without knowing how to package is unfair to them and will only lead to them being frustrated. And then you get said frustrated users reaching out for what appears to be the easiest solution, like OP did here, even though it causes subtler problems in the long run that they don’t appreciate until some program they need breaks and they ragequit NixOS.

If they were going to ragequit anyway, I say save everyone the trouble upfront.

Assuming you build things with nix within the nix sandbox. Most new users do not. If they were, they wouldn’t need nix-ld for 90% of what they claim to need it for. They just want to run some random appimage and expect it to work on all their systems instantly, because they heard about NixOS being the next big thing from [insert youtuber].

I don’t expect users to understand the implications of obscure things like builtins.scopedImport on day 1, but are we really advocating for a flood of StackOverflow nixos.wiki copy-paste programmers who don’t know what the code they write is doing and are unable to debug it even with multiple people helping?

Or to put it another way, why should users who don’t care about what nix has to offer use nix?
And same for NixOS?

I see so many projects pushing for “more adoption” without really taking care to whether said adopters actually want or need what the project has to offer…