Python applications set `PYTHONPATH`. How is that not a nix anti-pattern?

Help
1

Hi,

I ran into my variation of NixOS/nixpkgs#284753: “Clash between awscli2 and ansible”:

$ nix-shell -p borgmatic awscli2 --run 'aws --version'
Traceback (most recent call last):
bla bla bla
ImportError: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (/nix/store/0hh6mhg6giww0h1hccdhv1r07ks25x0k-python3.11-urllib3-2.0.7/lib/python3.11/site-packages/urllib3/util/ssl_.py)

The short version is that both borgmatic and awscli2 need the python lib urllib3. And so with this particular incantation, aws gets the new version that borgmatic needs, not the old version aws needs. Reverse the order (nix-shell -p awscli2 borgmatic) and aws starts working, but presumably borgmatic gets broken in subtle ways.

This sounds like it is much bigger than a bug against awscli2.

I would have hoped that nix-shell would somehow wrap the two applications so that they each get exactly the dependencies in the correct versions that they each need.

But no.

Instead, nix-shell sets PYTHONPATH to include an entry for all the dependencies for both borgmatic and awscli2. And so depending on which gets mentioned first, aws (from awscli2) either gets the correct or wrong version of urllib3.

I tried a couple of python applications: ansible, borgmatic, thefuck, and for all of them, PYTHONPATH gets set with paths to their dependencies when run with nix-shell.

There is no way that is going to work reliably unless you only can run a single application with nix-shell and can’t nest them.

Are python applications somehow fundamentally broken on NixOS?

EDIT: A counterargument to this is that this does work:

$ nix-shell -p borgmatic awscli2 --run 'PYTHONPATH= aws --version'
aws-cli/2.13.33 Python/3.11.8 Linux/6.1.87 source/x86_64.nixos.23 prompt/off

So for some reason nix-shell sets PYTHONPATH even though neither borgmatic nor aws needs it. Why?

I thought this form of dependency-hell was exactly the problem NixOS set out to solve. What am I missing?

1 Like
2

This is the description of the nix-shell command from its manpage:

The command nix-shell will build the dependencies of the specified derivation, but not the derivation itself. It will then start an inter‐
active shell in which all environment variables defined by the derivation path have been set to their corresponding values, and the script
$stdenv/setup has been sourced. This is useful for reproducing the environment of a derivation for development.

So you see, nix-shell it’s really tailored towards derivation debugging more than a “generic volatile package install” as it is usually sold. The command nix-shell -p borgmatic awscli2 is more or less equivalent to a:

nix-shell --expr 'with import <nixpkgs> { }; runCommand "dummy" { buildInputs = [ borgmatic awscli2 ]; }'

where both the packages are installed as dependencies of a fake generic derivation and the resulting environment is the env for introspecting that package ( I took it from the same man page, in the examples section).

Now, I don’t know why/how the installing a python build inputs results in a “tainted” environment, but I suppose may be useful for debugging how buildPythonPackage and friends work.

Looking at the definition of the derivation for awscli2 it does something peculiar with the python packages collection that redefines the the ruamel-yaml and urllib3 packages. This makes having both borgmatic and awscli2 as build inputs of another derivation a problem, because they don’t share the same (conflicting) dependencies and Python usually has “flat” dependencies (i.e. all the packages in the environment see the same package collection).

That said, this kind of issues with nix-shell is probably one of the reasons why in the new nix commandline api the work done by it is split among nix develop and nix shell, where the former has the role of utility for derivation debugging and nix shell is the real volatile package installer Executing a:

nix shell nixpkgs#borgmatic nixpkgs#awscli2

brings me into a shell where both the applications work flawlessly (I just tried few commands though).
To use this command you may need to enable the “experimental” commands either temporarily or permanently.

I hope this will help clarify the things a bit … If not, please ask for more!

1 Like
3

Gotcha. Makes sense.

A bit of a shame that:

nix shell nixpkgs#borgmatic nixpkgs#awscli2

is 43% longer to type than:

nix-shell -p borgmatic awscli2

(43 vs 30 chars)

Is there a shorter way of writing: nix shell nixpkgs#borgmatic nixpkgs#awscli2, especially if there are multiple packages required? Or should I just get used to the longer typing?

Also, another difference is that nix-shell installs from 23.11, but nix shell installs from unstable:

$ nix-shell -p awscli2 --run 'aws --version'
aws-cli/2.13.33 Python/3.11.8 Linux/6.1.87 source/x86_64.nixos.23 prompt/off

$ nix shell nixpkgs#awscli2 -c aws --version
aws-cli/2.15.43 Python/3.11.9 Linux/6.1.87 source/x86_64.nixos.23 prompt/off

(Yes I know, I can customize nix registry and nix-channel respectively:

$ nix-channel --list | grep nixos.org
nixos https://nixos.org/channels/nixos-23.11
unstable https://nixos.org/channels/nixpkgs-unstable

$ nix registry list | grep nixpkg
global flake:nixpkgs github:NixOS/nixpkgs/nixpkgs-unstable

)

4

I think that there isn’t any shortcut, you may define a shorter alias and add it to the registry, though.

Yes, this will be fixed in the next release if you switch to use flakes for the installation of your system, alternatively you can set it manually in configuration.nix file using the nix.registry option. But I don’t have a good example for its compilation and may be difficult to get the git revision of nixpkgs out of the copy installed by nix-channel although it should be there, because it’s used to define the version, I think. Maybe someone with more expertise may fill inn this info.

I have switched to use a flake-based system config for years now because I find it simpler to manage.

5

fyi for OP if you don’t want flakes, i have written up how to fix this without flakes Pinning NixOS with npins, or how to kill channels forever without flakes - jade's www site

1 Like
6

Assuming bash, nix shell nixpkgs#{borgmatic,awscli2} should work. You can also make your own script or alias to shorten it further if needed.

2 Likes
7

I can’t find evidence of this anymore but I could have sworn that one of the Nix forks I was looking at recently had ‘nixpkgs as the default flake’ as one of its technical improvements. Anyone else know what I’m remembering?

2 Likes
8

I personally alias nixpkgs to n in my registry, so I would type:

nix shell n#borgmatic n#awscli2

usually.

9

Yes, there is:

$ nix shell -f '<nixpkgs>' borgmatic awscli2

If if you do want to use the flake version:

$ nix shell -f flake:nixpkgs borgmatic awscli2

Only the flake output attrpath syntax requires every output to be prefixed with nixpkgs#; the others can take multiple attrpaths.

6 Likes
10

Put

function n() { nix shell $(printf "nixpkgs#%s " "$@"); }

in a .bashrc somewhere. Feel free to replace n with a character of your choice.

Then just do n borgmatic awscli2 for a 19 character solution :rofl:

1 Like
11

Cool - I didn’t know that about bash printf:

The format is re-used as necessary to consume all of the arguments. If
there are fewer arguments than the format requires, extra format
specifications behave as if a zero value or null string, as appropriate,
had been supplied.

12

I think it’s this:

5 Likes
13

This is fairly short without custom shell function:

nix shell nixpkgs#{borgmatic,awscli2}

EDIT: Flake refs at its best challenging bash syntax highlighters.