Unable to get python3 package working in latest nix

Help
1

So I am currently stumbling on this problem Python virtualenv not working because of sys.prefix mismatch · Issue #66366 · NixOS/nixpkgs · GitHub and I was wondering if anyone could provide a workaround. In context on my work laptop (which is currently running NixOS unstable) I need to install a pip3 python package called stups which also provides python executables.

Previously I used virtualenv method mentioned here https://nixos.wiki/wiki/Python, i.e.

let
  my-python-packages = python-packages: with python-packages; [
    virtualenvwrapper
    # other python packages you want
  ]; 
  python-with-my-packages = python37.withPackages my-python-packages;
in

Unfortunately this fails because of the previously linked issue. I also tried using GitHub - nix-community/pypi2nix: Abandoned! Generate Nix expressions for Python packages however running the generated package with nix-shell requirements.nix -A interpreter results in

building '/nix/store/8g3f5q65g28wa1wpadpkdwcs3lv7mr3b-stups-1.1.21.tar.gz.drv'...
Download error on https://pypi.org/simple/flake8/: [Errno -2] Name or service not known -- Some packages may not be found!
Couldn't find index page for 'flake8' (maybe misspelled?)
Download error on https://pypi.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found!
No local packages or working download links found for flake8
Traceback (most recent call last):
  File "nix_run_setup", line 8, in <module>
    exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
  File "setup.py", line 156, in <module>
    setup_package()
  File "setup.py", line 151, in setup_package
    entry_points={'console_scripts': CONSOLE_SCRIPTS},
  File "/nix/store/dm5p0lmpj0k9p9xb1z62vzmcsmg12pd6-python3.7-setuptools-41.4.0/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
    _install_setup_requires(attrs)
  File "/nix/store/dm5p0lmpj0k9p9xb1z62vzmcsmg12pd6-python3.7-setuptools-41.4.0/lib/python3.7/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/nix/store/dm5p0lmpj0k9p9xb1z62vzmcsmg12pd6-python3.7-setuptools-41.4.0/lib/python3.7/site-packages/setuptools/dist.py", line 720, in fetch_build_eggs
    replace_conflicting=True,
  File "/nix/store/dm5p0lmpj0k9p9xb1z62vzmcsmg12pd6-python3.7-setuptools-41.4.0/lib/python3.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
    replace_conflicting=replace_conflicting
  File "/nix/store/dm5p0lmpj0k9p9xb1z62vzmcsmg12pd6-python3.7-setuptools-41.4.0/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match
    return self.obtain(req, installer)
  File "/nix/store/dm5p0lmpj0k9p9xb1z62vzmcsmg12pd6-python3.7-setuptools-41.4.0/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain
    return installer(requirement)
  File "/nix/store/dm5p0lmpj0k9p9xb1z62vzmcsmg12pd6-python3.7-setuptools-41.4.0/lib/python3.7/site-packages/setuptools/dist.py", line 787, in fetch_build_egg
    return cmd.easy_install(req)
  File "/nix/store/dm5p0lmpj0k9p9xb1z62vzmcsmg12pd6-python3.7-setuptools-41.4.0/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 673, in easy_install
    raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('flake8')
unpacking sources
unpacking source archive /nix/store/0n2i9lazp89i9k7f85bxqg2shs1227vy-raven-6.10.0.tar.gz
source root is raven-6.10.0
builder for '/nix/store/3qmnjz6jfiiyhdn0d8arj5d53cyv8lld-python3.7-clickclick-1.2.2.drv' failed with exit code 1

Does anyone have any ideas where to go on from here? At this point I would just like to get something working, even if its hacky.

2

python on nixos is a bit of a pain, compared to the ‘vanilla way’

have you tried the approach documented in the manual ?
Nixpkgs 23.11 manual | Nix & NixOS → ’ 15.17.3.6. How to consume python modules using pip in a virtualenv like I am used to on other Operating Systems ?’

3

Thanks for the response!

I tried a similar approach outlined at https://nixos.wiki/wiki/Python at

To install pip add virtualenvwrapper to your list of Python packages above.

(this approach is exactly the same, the only difference is putting the installation into a default.nix file so it runs in a seperate shell). Unfortunately the core issue is that virtualenvwrapper fails to install and work properly for the same reasons outlined before, i.e.

Basically there seems to be some regression where virtualenvwrapper is even failing to initialize properly because of removal of PYTHONPATH, i.e. see Python: get rid of `--prefix PYTHONPATH` · Issue #22688 · NixOS/nixpkgs · GitHub.

I can try to replicate what is done at

precisely when I get home (albeit I have to use Python 3.x) however I suspect I will get the same problem. Judging by what is said in the earlier issue, maybe all that needs to be done is to update the NixOS wiki with changes that are with changes that required due to the removal of the PYTHONPATH wrapper (I however have no idea what change is required hence why I am asking here).

4

this works for me,

(nix-shell) 2d [azul:/tmp/lixo123] [venv] $ 
 >>> stups 

  ___ _____ _   _ ___  ___
 / __|_   _| | | | _ \/ __| by ZALANDO
 \__ \ | | | |_| |  _/\__ \
 |___/ |_|  \___/|_|  |___/

 STUPS - To Unleash Penguin Swarms

 Homepage:      https://stups.io
 Documentation: https://docs.stups.io
 GitHub Repos:  https://github.com/zalando-stups

Do you want to configure the STUPS CLI tools now? [Y/n]

where

 >>> cat requirements.txt 
flake8
stups

and

>>> cat default.nix 
with import <nixpkgs> {};
with pkgs.python37Packages;

stdenv.mkDerivation {
  name = "impurePythonEnv1d";
  buildInputs = [
    taglib
    openssl
    git
    libxml2
    libzip
    python37Full
    python37Packages.virtualenv
    stdenv
    libffi
    zlib ];
  src = null;
  shellHook = ''
    unset http_proxy
    export GIT_SSL_CAINFO=/etc/ssl/certs/ca-bundle.crt
    SOURCE_DATE_EPOCH=$(date +%s)
    virtualenv --no-setuptools venv
    source venv/bin/activate
    pip install jedi
    pip install ipython==5.2.0
    pip install pylint
    pip install flake8
    pip install -r requirements.txt
  '';
}
5

Thanks for the help! So I did precisely what you said and ended up getting this error

EDIT: Got it to work, changed virtualenv --no-setuptools venv to virtualenv venv, many thanks!

Although I am curious as to why we are getting different results. Also would you have any idea what is the cause behind Python virtualenv not working because of sys.prefix mismatch · Issue #66366 · NixOS/nixpkgs · GitHub ? Using nix-shell is fine for now, but ideally I would prefer the python executables from stups to be visible globally.

6

'Although I am curious as to why we are getting different results. ’ <— different nixpkgs commit I reckon, can’t recall if I’m running unstable or some old 19.09 commit. I may have not run nix channel update for a wee bit.

anyway,

nix-build 
/nix/store/xna08nigxnhndhhcqzrmydwcgzp806nx-stups-1.1.21

 >>> ./result/bin/stups 

  ___ _____ _   _ ___  ___
 / __|_   _| | | | _ \/ __| by ZALANDO
 \__ \ | | | |_| |  _/\__ \
 |___/ |_|  \___/|_|  |___/

 STUPS - To Unleash Penguin Swarms

 Homepage:      https://stups.io
 Documentation: https://docs.stups.io
 GitHub Repos:  https://github.com/zalando-stups

Do you want to configure the STUPS CLI tools now? [Y/n]:

where default.nix looks like:

#{ lib, buildPythonPackage, fetchPypi }:

with import <nixpkgs> {};
with python3.pkgs;

let
  zalando-aws-cli =
    buildPythonApplication rec {
      pname = "zalando-aws-cli";
      version = "1.2.5.25";

      propagatedBuildInputs =
      with python3Packages;[
        pyjwt
        flake8
        requests
        boto3
        pyaml
        clickclick
        stups-zign
        humanize
        natsort
      ];

      src = fetchPypi {
        inherit pname version;
        sha256 = "sha256:0yv7s9i5ydiaq60m12qq9dk2v24ilpkpad44drhz35ggsrldhqhf";
      };

      doCheck = false;
    };

  zalando-kubectl =
    buildPythonApplication rec {
      pname = "zalando-kubectl";
      version = "1.13.5.74";

      propagatedBuildInputs =
        with python3Packages;[
          flake8
          six
          requests
          stups-zign
          pyperclip
          boto3
          pytest
          arrow
          pystache
          typing
          raven
          zalando-aws-cli
        ];

      src = fetchPypi {
        inherit pname version;
        sha256 = "sha256:0ks1rkfly7928xj66mszvkdq25qpyj8z86bivw8csq2i8nvlflhd";
      };

      doCheck = false;
    };

  stups-fullstop =
    buildPythonApplication rec {
      pname = "stups-fullstop";
      version = "1.1.31";

      propagatedBuildInputs =
        with python3Packages;[
          flake8
          six
          requests
          stups-zign
          pyperclip
          boto3
          pytest
          arrow
          pystache
          typing
          raven
        ];

      src = fetchPypi {
        inherit pname version;
        sha256 = "sha256:0cpsijwmciczn85kd3k8wmhaizh599wpa3swqkvlsrpa1a662212";
      };

      doCheck = false;
    };

  stups-pierone =
    buildPythonApplication rec {
      pname = "stups-pierone";
      version = "1.1.42";

      propagatedBuildInputs =
        with python3Packages;[
          flake8
          six
          requests
          stups-zign
          pyperclip
          boto3
          pytest
          arrow
          pystache
          typing
          raven
        ];

      src = fetchPypi {
        inherit pname version;
        sha256 = "sha256:1bfwl31bmvhvfri52x1zjad8mlkhrnn5h24c69qgf0ahwasy3cn1";
      };

      doCheck = false;
    };

  stups-senza =
    buildPythonApplication rec {
      pname = "stups-senza";
      version = "2.1.138";

      propagatedBuildInputs =
        with python3Packages;[
          flake8
          six
          requests
          stups-zign
          pyperclip
          boto3
          pytest
          arrow
          pystache
          typing
          raven
          stups-pierone
        ];


      src = fetchPypi {
        inherit pname version;
        sha256 = "sha256:0f1qlvrkkmwbhi578bcz9fv64kramag37jcbnp4d90kwfr2cca2v";
      };

      doCheck = false;
    };

  stups-piu =
    buildPythonApplication rec {
      pname = "stups-piu";
      version = "1.1.16";

      propagatedBuildInputs =
        with python3Packages;[
          flake8 six requests stups-zign pyperclip boto3
        ];

      src = fetchPypi {
        inherit pname version;
        sha256 = "sha256:0lxvcmczzqakvp5m571ayxr0vw92ffp4gmv1wjkmpsm64f72xyp2";
      };

      doCheck = false;
    };

  stups-tokens =
    buildPythonApplication rec {
      pname = "stups-tokens";
      version = "1.1.19";

      propagatedBuildInputs =
        with python3Packages;[
          flake8 six requests
        ];

      src = fetchPypi {
        inherit pname version;
        sha256 = "sha256:0q1ir951lcmx9wdcj328x5y165x9rkpzr3v06jbjmmdzrj1ssc3q";
      };

      doCheck = false;
    };


  clickclick =
    buildPythonApplication rec {
      pname = "clickclick";
      version = "1.2.2";

      propagatedBuildInputs =
        with python3Packages;[
          flake8 six click pyaml dnspython
        ];

      src = fetchPypi {
        inherit pname version;
        sha256 = "sha256:08vd1whzr1gz3cfb9iqs200rgj1x9srlwaa6sjmwz41rkjm0m2aa";
      };

      doCheck = false;
    };


  stups-cli-support =
    buildPythonApplication rec {
      pname = "stups-cli-support";
      version = "1.1.20";

      propagatedBuildInputs =
        with python3Packages;[
          flake8 six requests clickclick stups-tokens
        ];

      src = fetchPypi {
        inherit pname version;
        sha256 = "sha256:1nc5va3j9gc9w2vqmg9rbp91kw51xnmshkhifyhfvi7fcdd5r8rw";
      };

      doCheck = false;
    };


  stups-zign =
    buildPythonApplication rec {
      pname = "stups-zign";
      version = "1.2";

      propagatedBuildInputs =
        with python3Packages;[ stups-cli-support ];

      src = fetchPypi {
        inherit pname version;
        sha256 = "sha256:06dga2144xp715lbjgmvq04a3i9hka0p232skpnx2f72pjgb6ral";
      };

      doCheck = false;
    };

  stups-kio =
    buildPythonApplication rec {
      pname = "stups-kio";
      version = "0.1.24";

      propagatedBuildInputs =
        with python3Packages;[ flake8 six requests stups-zign ];

      src = fetchPypi {
        inherit pname version;
        sha256 = "sha256:0n7rywwld9r6xbf2jyy13fwlrziq8h387sd4z4qrzk5s22v7lzjh";
      };

      doCheck = false;
    };
in
  buildPythonApplication rec {

  pname = "stups";
  version = "1.1.21";

  propagatedBuildInputs =
    with python3Packages;[
      flake8
      stups-kio
      stups-piu
      stups-senza
      stups-fullstop
      zalando-kubectl
    ];

  src = fetchPypi {
    inherit pname version;
    sha256 = "sha256:1irdyy5jrv3d3ym6mhr848zanxkbazvw9b7qhlan5bg4vz7ns00c";
  };
  doCheck = false;
}
7

I’ll be honest, with anything that is vanilla I tend to just use docker. Most of the projects I work, being them python based or something else, tend to have lock files pipenv.lock/requirements.freeze/Gemfile.lock and so on which makes iterating on those projects with a team (not using nix) quite troublesome, I simply just choose not to use nix and just do it the same way as the rest of my team.
If pypi2nix or other approaches just worked, then I could consider going full nix, until that happens, well I have bills to pay and nixfying just for the sake of nixfying means I would be too slow to iterate on a project where everyone else just pip installs -r …

I ended up just baking a clinically obese docker image that I can run anywhere,

Hope to not need it one day, until them…

8

Wow thanks, this looks a bit of a nightmare to maintain. Aren’t you basically doing what pypi2nix is doing but manually?

Hmm, sounds like it makes sense to try and track these python issues in NixOS properly. From the looks of things, there are a lot of QoL improvements that need to be made to get python packages to just “work”. tbh my understanding of python is very slim, this is just some CLI tool that we have to use at work (I don’t actually develop with python at all).

Thanks for the help anyways, I will try and merge your changes with the results of pypi2nix when I find some time. If you are resorting to some monolithic docker image to do the same job that nixos is meant to do, then we have some sought of a problem.