Python + Qt woes

[jacg]

All my attempts at using a Python module that requires Qt, fail with the dreaded

qt.qpa.plugin: Could not find the Qt platform plugin "xcb" in ""

(and the, hilarious-in-Nix ‘Reinstalling the application may fix this problem.’) error message.

For example:

nix-shell \
 --pure \
 -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/4d0ee90c6e253d40920f8dae5edb717a7d6f151d.tar.gz \
 -p 'python38.withPackages(ps: with ps; [ pyqtgraph ])' \
 --run "python -c 'import pyqtgraph as pg; pg.mkQApp()'"

crashes with the error

qt.qpa.plugin: Could not find the Qt platform plugin "xcb" in ""
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

/tmp/nix-shell-17747-0/rc: line 1: 20626 Aborted                 (core dumped) python -c 'import pyqtgraph as pg; pg.mkQApp()'

How can I get around this problem?

With the use of --pure and the nailing down a specific version of nixpkgs I would like to think that this is a hermetically sealed and fully reproducible instruction. Is this naive?

[ Aside: is there a flakes-enabled nix shell equivalent of the old nix-shells ability to specify complex ‘packages’: -p 'python38.withPackages(ps: with ps; [ pyqtgraph ])'? ]

[jtojnar]

Qt requires specific environment variables (e.g. QT_PLUGIN_PATH). If you are building a program, wrapQtAppsHook will fix that for you but for shells, I am not aware of a ready-made solution.

I do not think nix shell supports that (other than using an explicit expression like nix shell --impure --expr 'with import ./. {}; python38.withPackages(ps: with ps; [ pyqtgraph ])'). I am not even sure if the support was intentional in nix-shell or just a by-product of it stuffing -p values verbatim to buildInputs of an ad-hoc derivation.

As I understand it, the new Nix CLI aims to be simpler, and one of the approaches is separating the builder scripts from the build products:

• In nix-shell, you were basically transplanted into the environment of mkDerivation builder, with buildInputs on PATH, environment variables set by various setup hooks, and even bash functions from stdenv available.

• nix shell chooses cleaner method of just adding the binaries of requested packages to PATH, not touching much else of the environment. That is conceptually easier but useless for programs that need modified environment.

• nix develop is closer to the nix-shell of old but if you want anything more complex, you need to create the shell derivation yourself: nix develop --impure --expr 'with import ./. {}; mkShell { buildInputs = [ (python38.withPackages (ps: with ps; [ pyqtgraph ])) qt5.wrapQtAppsHook makeWrapper bashInteractive ]; shellHook = \'\'bashdir=$(mktemp -d); makeWrapper "$(type -p bash)" "$bashdir/bash" "\'\'${qtWrapperArgs[@]}"; exec "$bashdir/bash"\'\'; }'

The craziness above actually takes the environment that wrapQtAppsHook would wrap binaries with and runs a new shell with that environment. But at that point, you probably want to save it into shell.nix.

let
   pkgs = with import ./. {};
in
  pkgs.mkShell {
    buildInputs = [
      (python38.withPackages (ps: with ps; [
        pyqtgraph
      ]))
      qt5.wrapQtAppsHook
      makeWrapper
      bashInteractive
    ];

    shellHook = ''
      bashdir=$(mktemp -d)
      makeWrapper "$(type -p bash)" "$bashdir/bash" "''${qtWrapperArgs[@]}"
      exec "$bashdir/bash"
    '';
  }

[jacg]

@jtojnar

nix shell chooses cleaner method of just adding the binaries of requested packages to PATH , not touching much else of the environment. That is conceptually easier but useless for programs that need modified environment.

… which I’m discovering, slowly and painfully, over here.

What is the future of the old nix-shell (and friends) once the new CLI is here? I’m sort of trying (slowly) to transition to using flakes and the new CLI for everything I do, and moving away from the old stuff. Is that an exercise in masochism that’s just asking for trouble? Can your shell.nix example somehow be shoehorned into a flake? (Or should I not even try?)

Incidentally, I’m trying to use your shell.nix sample with direnv and zsh. I think I can manage the zsh part (though it would be nice not to have it bound to a specific choice of shell, and I don’t know how to do that), but direnv seems not to be picking up the python version at all; whereas an explicit nix-shell does. (I’ve triple-checked that my .envrc contains use nix rather than use flake.)

I trust that there’s nothing deep about

   pkgs = with import ./. {};

and that I can just replace that with whatever version of nixpkgs I happen to be pinning.

[Aside, the shell.nix from which I extracted the problem I posted, Just Worked, until I switched from Nixos 20.03 to 20.09.]

[jtojnar]

@jacg

What is the future of the old nix-shell (and friends) once the new CLI is here?

I think nix develop is the replacement for nix-shell. But nothing is final.

@jacg

I’m sort of trying (slowly) to transition to using flakes and the new CLI for everything I do, and moving away from the old stuff. Is that an exercise in masochism that’s just asking for trouble? Can your shell.nix example somehow be shoehorned into a flake? (Or should I not even try?)

You can just put the expression above to devShell attribute of the flake, taking pkgs from legacyPackages of nixpkgs input. (Do not forget that it needs to be per-platform.)

@jacg

Incidentally, I’m trying to use your shell.nix sample with direnv and zsh. I think I can manage the zsh part (though it would be nice not to have it bound to a specific choice of shell, and I don’t know how to do that),

Maybe use $SHELL instead of bash.

@jacg

I trust that there’s nothing deep about

   pkgs = with import ./. {};

and that I can just replace that with whatever version of nixpkgs I happen to be pinning.

Yeah, I just tested it in my Nixpkgs directory.

@jacg

direnv seems not to be picking up the python version at all; whereas an explicit nix-shell does. (I’ve triple-checked that my .envrc contains use nix rather than use flake .)

Do you mean that the python on path does not come from the shell? Thinking about it, execing a shell in the shellHook might be problematic if the shell sources ~/.profile or something. It would probably be better to export the environment variables directly but this was the simplest method that came to mind.

[jacg]

@jtojnar

I think nix develop is the replacement for nix-shell . But nothing is final.

So trying to wean off nix-shell is not such a crazy idea, then?

But nix-shell served two purposes:

1. Give me an environment for hacking away at some package.
2. Execute some programs provided by some package.

It seems that support for the latter via nix shell and nix run is going to be weaker than it was in nix-shell because, as you mentioned above, there’s no way of influencing the environment in which they run. So maybe something is lost?

@jtojnar

Do you mean that the python on path does not come from the shell?

Yes.

@jtojnar

It would probably be better to export the environment variables directly

And what’s the recommended way of doing that?

[jtojnar]

@jacg

So trying to wean off nix-shell is not such a crazy idea, then?

Personally, I use flake’s devShell through flake-compat through use nix in direnv. But going full on Flakes is fine, as long as you accept that the commands might move under your feet (e.g. the somewhat recent run → shell & shell → develop & app → run shuffle).

@jacg

But nix-shell served two purposes:

1. Give me an environment for hacking away at some package.
2. Execute some programs provided by some package.

It seems that support for the latter via nix shell and nix run is going to be weaker than it was in nix-shell because, as you mentioned above, there’s no way of influencing the environment in which they run. So maybe something is lost?

Well, programs provided by package should ideally be self-contained (through the use of wrappers) so we might consider that a benefit that it brings insufficient hermeticity to light.

I would argue that what you are doing when playing with Python REPL is hacking away at some package so you really want nix develop.

@jacg

@jtojnar

It would probably be better to export the environment variables directly

And what’s the recommended way of doing that?

You would need to create a shell function that accepts the same arguments as makeWrapper but modifies the environment instead. Then you could pass it "${qtWrapperArgs[@]}" in a shellHook and avoid execing bash.

But you could also try passing --noprofile or --norc flags to the execed bash to potentially avoid it from overriding your PATH. Not sure which, or even if that is causing the python not to be picked up. You could echo $PATH in the shell hook and in the executed shell to see if it differs, to determine if it is the issue.

[jacg]

@jtojnar

Well, programs provided by package should ideally be self-contained (through the use of wrappers) so we might consider that a benefit that it brings insufficient hermeticity to light.

Actually, I totally agree with that. I guess it’s the brick wall that I’ve hit here that is making me have heretical thoughts.

@jtojnar

I would argue that what you are doing when playing with Python REPL is hacking away at some package so you really want nix develop .

Oh, I think it’s pretty clear that nix develop is the right tool for this sort of thing.

[Aside: I’m getting the impression that when I use direnv with use flake that the build phase is being executed, and also when I manually execute nix develop … unless it’s cached (this has arisen while a bazillion other questions were being raised, so I haven’t kept track of it very well). At first blush I would hope that neither nix develop nor direnv should build the package automatically.]

@jtojnar

You would need to create a shell function that accepts the same arguments as makeWrapper but modifies the environment instead. Then you could pass it "${qtWrapperArgs[@]}" in a shellHook and avoid exec ing bash.

Hrrmmmph … can you help me decipher "${qtWrapperArgs[@]}"? Maybe starting with the [@]: I have no idea what that is (bash? Nix? Druid Runes? APL? Brainfuck?), or where to start looking it up.

[jtojnar]

@jacg

[Aside: I’m getting the impression that when I use direnv with use flake that the build phase is being executed, and also when I manually execute nix develop … unless it’s cached (this has arisen while a bazillion other questions were being raised, so I haven’t kept track of it very well). At first blush I would hope that neither nix develop nor direnv should build the package automatically.]

I have not really used use flake or nix develop since I use everything through flake-compat so I am not sure but I would expect it to just run enough of the builder for setup hooks to run but not run any phases so not building the package itself.

@jacg

Hrrmmmph … can you help me decipher "${qtWrapperArgs[@]}" ? Maybe starting with the [@] : I have no idea what that is (bash? Nix? Druid Runes? APL? Brainfuck?), or where to start looking it up.

That is (including the outer quotes) bash for “take all items in this array and pass them as individual items”. For example:

foo=(
  "alpha beta"
  "gamma"
  "delt\"a"
)
echo "${foo[@]}"

Would do the same as

echo "alpha beta" "gamma" "delt\"a"

[jacg]

My enthusiasm for getting to the bottom of this fizzled out as I had a flake-based solution that did 70% of what I needed, and allowed me to get on with my life … while there were plenty of other things which weren’t letting me get on with my life at all.

By 70%, I mean, when I cd into a directory containing a flake with this solution (with zsh replacing bash) and have .envrc containing use flake:

• It get stuck in this sort of infinite loop

  direnv: loading /tmp/aaaaargh/.envrc
  direnv: using flake
  direnv: using cached dev shell
  direnv: loading /tmp/aaaaargh/.envrc
  direnv: using flake
  direnv: using cached dev shell
  direnv: loading /tmp/aaaaargh/.envrc 
  direnv: using flake
  direnv: using cached dev shell
  direnv: loading /tmp/aaaaargh/.envrc
  

• When I interrupt it, $SHLVL is at about 36 (or some other silly number) …

• but … the python+Qt thing I need, works as it’s supposed to.

This was tolerable (because I need to use it very rarely) until I found myself needing to use a python+qt thing in an environment where experimental Nix is unavailable.

with use nix and shell.nix it seems to work OK-ish with bash, but with zsh, the packages specified in buildInputs are unavailable, so nothing works at all.

Can you suggest how to make some sort of progress?

Fundamentally, I find it hard to believe that I’m the only person on the planet trying to use python+qt+nix-shell(+zsh) and that this isn’t a solved problem.

What’s more, I’m pretty sure that 6 to 8 months ago, it did work.

[jtojnar]

I recently needed Qt environment but could not exec bash so I figured out a way to modify the solution from https://discourse.nixos.org/t/python-qt-woes/11808/2?u=jtojnar to avoid exec.

If we delete the line that contains the exec call (to be sure, I marked it with a random string to be passed as -a flag to exec), we can just source the produced file.

let
   pkgs = with import ./. {};
in
  pkgs.mkShell {
    buildInputs = [
      (pkgs.python38.withPackages (ps: with ps; [
        pyqtgraph
      ]))
    ];
    
    nativeBuildInputs = [
      pkgs.qt5.wrapQtAppsHook
      pkgs.makeWrapper
      pkgs.openssl
    ];

    shellHook = ''
      setQtEnvironment=$(mktemp)
      random=$(openssl rand -base64 20 | sed "s/[^a-zA-Z0-9]//g")
      makeWrapper "$(type -p sh)" "$setQtEnvironment" "''${qtWrapperArgs[@]}" --argv0 "$random"
      sed "/$random/d" -i "$setQtEnvironment"
      source "$setQtEnvironment"
    '';
  }

Maybe avoiding nested bash will solve some of your issues.

[jacg]

That seems to eliminate my infinite-tower-of-shells problem.

Thank you.

[milahu]

a simpler version

# shell.nix
{ pkgs ? import <nixpkgs> {} }:
pkgs.mkShell rec {
  nativeBuildInputs = with pkgs; [
    libsForQt5.wrapQtAppsHook
    makeWrapper
  ];
  buildInputs = [
    (pkgs.python3.withPackages (ps: with ps; [
      pyqtgraph
    ]))
  ];
  # https://discourse.nixos.org/t/python-qt-woes/11808/10
  shellHook = ''
    setQtEnvironment=$(mktemp --suffix .setQtEnvironment.sh)
    echo "shellHook: setQtEnvironment = $setQtEnvironment"
    makeWrapper "/bin/sh" "$setQtEnvironment" "''${qtWrapperArgs[@]}"
    sed "/^exec/d" -i "$setQtEnvironment"
    source "$setQtEnvironment"
  '';
}