I’m providing some software via a nix flake. This requires users to install nix.
A couple of users tried it on a machine running Ubuntu 22.04. I was not present to see this, but the user reports that the nix (multi-user) installer completed without issuing any warnings or errors. Thereafter, any attempt to use any nix command gave the error:
nix: symbol lookup error: /nix/store/....... undefined symbol: sqlite3_error_offset
At this stage, IIUC, they installed sqlite via apt-get. Nix complained about the version being too old. So they installed a more recent version, by hand, into /usr/local/bin (or some such) and pointed LD_LIBRARY_PATH at it.
Problems continued.
Thereafter, I watched a complete removal of nix, following these instructions. The multi-user Nix installer ran without any complaint.
Thereafter nix profile install or nix shell appear to work without any problem, but trying to run just about anything installed with either of these, crashes, although in slightly different ways:
runs successfully
hello
segfault
direnv
cachix
/nix/store/ir0j7zqlw9dc49grmwplppc7gh0s40yf-bash-5.2-p15/bin/bash: /usr/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.36' not found (required by /nix/store/ir0j7zqlw9dc49grmwplppc7gh0s40yf-bash-5.2-p15/bin/bash)
cowsay
Questions
Does the nix installer really rely on the target system having a pre-installed and specific version of sqlite3?
Why on earth does /nix/store/.../bin/bash appear to be looking for an out-of-store libc?
What can be done to get these users out of this hole?
The installer claims to have installed Nix successfully, but running
nix --help
once again gives
undefimed symbol: sqlite3_error_offset
running it through strace gives a bit more information
symbol lookup error: /nix/store/3wqasl97rjiza3vd7fxjnvli2w9l30mk-nix-2.17.0/lib/libnixstore.so: undefined symbol: sqlite3_error_offset
At this stage I’m starting to suspect an error in the installer or in Nix itself, but then I’d expect the internet to be replete with complaints about it, and I don’t find very much beyond
Reports the same problem, also on Ubuntu 22.04. (But I struggle to see how this could depend on a specific host OS version.)
Suggests (thanks @abathur) that going back to nix-2.12.0 avoids the problem. (But this was reported when Nix 2.13.0 was “very recent” (6 months ago) and we are now on 2.17, so it seems unlikely that this problem would have gone unfixed for so many versions.)
Anyway, we’ll try this when the user is available again.
Given that the Determinate Nix Installer provides a simple command to revert everything it did, I would prefer to continue using it, rather than the official installer. But there doesn’t seem to be a way to tell it to use an older version of Nix, other than downgrading DNI itself.
I’m not as versed in troubleshooting Linux install problems, so I wasn’t really sure why the reddit user was seeing the issue.
In retrospect, I think I was ~tricked by the fact that the code using this symbol was very new, but that detail was probably a red herring. The symbol error is probably an issue with the runtime environment causing Nix to run against the system’s sqlite instead of the one the installer adds as part of the store bootstrap:
$ curl -L https://releases.nixos.org/nix/nix-2.17.0/nix-2.17.0-x86_64-linux.tar.xz -o tmp.tar.xz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 21.1M 100 21.1M 0 0 2342k 0 0:00:09 0:00:09 --:--:-- 2531k
$ mkdir flarf
$ tar -xJf tmp.tar.xz -C flarf
$ find flarf -name "*sqlite*"
flarf/nix-2.17.0-x86_64-linux/store/jpj9lx0p2h1vs3gkzj8jh350113bsm84-sqlite-3.39.4
flarf/nix-2.17.0-x86_64-linux/store/jpj9lx0p2h1vs3gkzj8jh350113bsm84-sqlite-3.39.4/lib/libsqlite3.so.0
flarf/nix-2.17.0-x86_64-linux/store/jpj9lx0p2h1vs3gkzj8jh350113bsm84-sqlite-3.39.4/lib/libsqlite3.la
flarf/nix-2.17.0-x86_64-linux/store/jpj9lx0p2h1vs3gkzj8jh350113bsm84-sqlite-3.39.4/lib/libsqlite3.so
flarf/nix-2.17.0-x86_64-linux/store/jpj9lx0p2h1vs3gkzj8jh350113bsm84-sqlite-3.39.4/lib/libsqlite3.so.0.8.6
This misconfiguration could (as well as adding /usr/local/bin to LD_LIBRARY_PATH) probably cause it to pick up other incorrect library versions. Setting aside any change you might have made to try and fix the sqlite error, were LD_LIBRARY_PATH or LD_PRELOAD already set (probably via a shell init file, assuming you’ve rebooted or started a new shell during this testing)?
These users’ .bashrcs were polluting the environment with all sorts of toxic waste. Remove all this hazmat, and Nix starts working properly.
Clearly, I was being naive about the extent to which Nix can be immune to this.
There remains one problem that I’m still aware of: as much as I dislike using nix profile install, it does seem to be the most expedient way of giving these users a sensible version of direnv. Unfortunately, this doesn’t work, complaining that /nix/var/nix/profiles/per-user/<user> does not exist.
I suspect that this is the consequence of the repeated attempts to install and uninstall Nix, which messed up something along the way?
How can this be fixed?
But perhaps this should be asked in a separate topic.
