How to add a python dependency not (anymore) in the official nixpkgs?

[573]

With the hack shown in

Python: flask app can't find dependencies

which means, my Nixpkgs is quite old… But now I can reproduce your issue (with your nixpkgs revision), and it works with my revision (and argv hack): $ nix-shell -E ' with (import (builtins.fetchTarball { url = "https://github.com/NixOS/nixpkgs/archive/bea1a232c61.tar.gz"; }) { }); (python37.withPackages(p: [ p.flask p.flask-restful ])).env' --run 'FLASK_APP=/tmp/app.py FLASK_ENV=development flask run' Maybe you can bisect git revisions from bea1a232c61 to 8746c77a383f5c and find out which o…

EDIT: Found a commit for asteval at last, so my working hack approach this time became:

nix-shell -E '
with (import (builtins.fetchTarball {
  url = "https://github.com/NixOS/nixpkgs/archive/53676223cc557de4e8b1c5683a147f4db4d02b69.tar.gz";
}) { }); (python36.withPackages(p: [ p.asteval ])).env'

JFTR Non-working approach:

I tried a way to look for python-asteval in the packages collection with the result that there is none:

nix-shell -E '
with (import (builtins.fetchTarball {
  url = "https://github.com/NixOS/nixpkgs/archive/master.tar.gz";
}) { }); (python36.withPackages(p: [ p.asteval ])).env'

error: attribute ‘asteval’ missing, at (string):4:38

So my question becomes “How to add a python dependency not (anymore) in the official nixpkgs ?” and I will change the thread title to reflect this.

Further questions that could have answers already in the manual (JFTR again):
Is there something I can do with using overlays here and also, is there an overlay some of you know currently providing asteval packaged ?
In case not, what approbriate steps could I undertake ? Adding the package to the nixpkgs collection ?
Also where can I find out what fetchPypi is supposed to do ? I had some misunderstandings as interpreted one of it’s purposes being to load python packages also those not on nixpkgs in a derivation. Is that the case ?
(thx @FRidh )

One approach I’ll try is to wrap the asteval package sources in a fetchPypi closure and use the resulting derivation in my jrnl derivation then.

Wrapping asteval etc.

# program/jrnl/default.nix
{ config, lib, pkgs, python3, ... }:
with pkgs;
let
  jrnl = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "jrnl";
  version = "2.3";

  src = fetchFromGitHub {
  owner = "jrnl-org";
  repo = "jrnl";
  rev = "f06bc9ea9be5b1bf2e9d74b6da2bae46ecc192fb";
  sha256 = "1dijla07k1qpl0qx21s8j8yl326g5xyy6lka7sdmp0qp62033hrx";
  };

  propagatedBuildInputs = with pkgs.python36.pkgs; [
python
pyxdg
cryptography
passlib
parsedatetime
keyring
pytz
tzlocal
asteval
colorama
python-dateutil
pyyaml
ansiwrap
  ];

  doCheck = false;
};
asteval = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "asteval";
  version = "0.9.18";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "5d64e18b8a72c2c7ae8f9b70d1f80b68bbcaa98c1c0d7047c35489d03209bc86";
  };
  doCheck = false;
};
# see https://github.com/NixOS/nixpkgs/search?q=textwrap3&unscoped_q=textwrap3
ansiwrap = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "ansiwrap";
  version = "0.8.4";
  extension = "zip";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version extension;
    sha256 = "ca0c740734cde59bf919f8ff2c386f74f9a369818cdc60efe94893d01ea8d9b7";
  };

  propagatedBuildInputs = with pkgs.python36.pkgs; [
textwrap3
  ];

  doCheck = false;
};
textwrap3 = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "textwrap3";
  version = "0.9.2";
  extension = "zip";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version extension;
    sha256 = "5008eeebdb236f6303dcd68f18b856d355f6197511d952ba74bc75e40e0c3414";
  };

  doCheck = false;
};
in
  {
    home.packages = with pkgs; [
      (import jrnl {pname = "jrnl";})
    ];
  }

Still not quite there:

Executing setuptoolsBuildPhase
Traceback (most recent call last):
File “nix_run_setup”, line 8, in
exec(compile(getattr(tokenize, ‘open’, open)(file).read().replace(‘\r\n’, ‘\n’), file, ‘exec’))
File “/nix/store/c8qshsnmy5d9xk3nxj2xy9xc4isnc7ni-python3-3.6.9/lib/python3.6/tokenize.py”, line 452, in open
buffer = _builtin_open(filename, ‘rb’)
FileNotFoundError: [Errno 2] No such file or directory: ‘setup.py’
killing process 31602: No such process
builder for ‘/nix/store/863kkav84dhpiy5pmi2zxnrlkq34lsq6-jrnl-2.3.drv’ failed with exit code 1
error: build of ‘/nix/store/863kkav84dhpiy5pmi2zxnrlkq34lsq6-jrnl-2.3.drv’ failed

Almost there ?

With the following

# program/jrnl/default.nix
{ config, lib, pkgs 
, fetchPypi
, fetchpatch
, cryptography_vectors
, pytest
, pretend
, iso8601
, hypothesis
, ...
 }:
with pkgs;
let
  jrnl = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "jrnl";
  version = "2.3";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "ede6b3783562b03eaab64679d1913d335079535dbc8cca83bffbd65dacb4bcb6";
  };

  propagatedBuildInputs = with pkgs.python36.pkgs; [
#python
pyxdg
cryptography
passlib
parsedatetime
keyring
pytz
tzlocal
asteval
colorama
python-dateutil
pyyaml
ansiwrap
setuptools_scm
  ];

  doCheck = false;
};
asteval = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "asteval";
  version = "0.9.18";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "5d64e18b8a72c2c7ae8f9b70d1f80b68bbcaa98c1c0d7047c35489d03209bc86";
  };
  doCheck = false;
};
# https://github.com/NixOS/nixpkgs/search?q=textwrap3&unscoped_q=textwrap3
ansiwrap = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "ansiwrap";
  version = "0.8.4";
  extension = "zip";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version extension;
    sha256 = "ca0c740734cde59bf919f8ff2c386f74f9a369818cdc60efe94893d01ea8d9b7";
  };

  propagatedBuildInputs = with pkgs.python36.pkgs; [
textwrap3
  ];

  doCheck = false;
};
textwrap3 = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "textwrap3";
  version = "0.9.2";
  extension = "zip";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version extension;
    sha256 = "5008eeebdb236f6303dcd68f18b856d355f6197511d952ba74bc75e40e0c3414";
  };

  doCheck = false;
};
keyring = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "keyring";
  version = "19.3.0";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "ee3d35b7f1ac3cb69e9a1e4323534649d3ab2fea402738a77e4250c152970fed";
  };

  propagatedBuildInputs = with pkgs.python36.pkgs; [
    SecretStorage
    setuptools_scm
    importlib_metadata
    toml
    zipp
  ];

  doCheck = false;
};
SecretStorage = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "SecretStorage";
  version = "3.1.2";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "15da8a989b65498e29be338b3b279965f1b8f09b9668bd8010da183024c8bff6";
  };

  propagatedBuildInputs = with pkgs.python36.pkgs; [
    cryptography
    jeepney
  ];

  doCheck = false;
};
# https://github.com/search?q=.nix+repo%3ANixOS%2Fnixpkgs+path%3A%2Fpkgs%2Fdevelopment%2Fpython-modules%2Fcryptography&type=Code&ref=advsearch&l=&l=
cryptography = pkgs.python36.pkgs.buildPythonPackage rec {
  pname = "cryptography";
  version = "2.8";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "3cda1f0ed8747339bbdf71b9f38ca74c7b592f24f65cdb3ab3765e4b02871651";
  };

  outputs = [ "out" "dev" ];

  buildInputs = [ openssl ];
  propagatedBuildInputs =  with pkgs.python36.pkgs; [
    packaging six pkgs.gcc
  ] ++ stdenv.lib.optional (!isPyPy) cffi;

  doCheck = false;
};
jeepney = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "jeepney";
  version = "0.4.3";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "3479b861cc2b6407de5188695fa1a8d57e5072d7059322469b62628869b8e36e";
  };

  doCheck = false;
};
setuptools_scm = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "setuptools_scm";
  version = "3.3.3";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "bd25e1fb5e4d603dcf490f1fde40fb4c595b357795674c3e5cb7f6217ab39ea5";
  };

  doCheck = false;
};
importlib_metadata = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "importlib_metadata";
  version = "1.5.2";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "dfc83688553a91a786c6c91eeb5f3b1d31f24d71877bbd94ecbf5484e57690a2";
  };
  
  propagatedBuildInputs = with pkgs.python36.pkgs; [
    setuptools_scm
    zipp
  ];

  doCheck = false;
};
zipp = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "zipp";
  version = "0.5.2";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "4970c3758f4e89a7857a973b1e2a5d75bcdc47794442f2e2dd4fe8e0466e809a";
  };
  
  propagatedBuildInputs = with pkgs.python36.pkgs; [
    setuptools_scm
    toml
  ];

  doCheck = false;
};
toml = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "toml";
  version = "0.10.0";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "229f81c57791a41d65e399fc06bf0848bab550a9dfd5ed66df18ce5f05e73d5c";
  };

  doCheck = false;
};
in
  {
    home.packages = with pkgs; [
      (import jrnl {pname = "jrnl";})
    ];
  }

jrnl seems to be built though stuck at installing…

wrapping `/nix/store/z7kmxqigshc8hwgfp43p7h672hadyz3g-jrnl-2.3/bin/jrnl’…
pythonCatchConflictsPhase
Found duplicated packages in closure for dependency ‘setuptools-scm’:
setuptools-scm 3.3.3 (/nix/store/k03sym20l725wghmx73lagq70i78wi84-setuptools_scm-3.3.3/lib/python3.6/site-packages)
setuptools-scm 3.3.3 (/nix/store/43gzhvj2gksqzyxpyqp84b2w3nd80r81-python3.6-setuptools_scm-3.3.3/lib/python3.6/site-packages)

Package duplicates found in closure, see above. Usually this happens if two packages depend on different version of the same dependency.
killing process 4961: No such process
builder for ‘/nix/store/13f5xfkx66iyd0qc1h1z8sr83pv4lpq6-jrnl-2.3.drv’ failed with exit code 1
error: build of ‘/nix/store/13f5xfkx66iyd0qc1h1z8sr83pv4lpq6-jrnl-2.3.drv’ failed

Still missing something…

jrnl is built now but installation fails:

Just removing the setuptools_scm “closure” or should I say derivation from my expression above resulted in a lucky build, only thing there is one error still, but that is something I could live with as I may use jrnl running /nix/store/7m0r7qhy8c9sm34qj596mizf5hlkdhk5-jrnl-2.3/bin/jrnl directly. Ugly but works!

# program/jrnl/default.nix
{ config, lib, pkgs 
, fetchPypi
, fetchpatch
, cryptography_vectors
, pytest
, pretend
, iso8601
, hypothesis
, ...
 }:
with pkgs;
let
  jrnl = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "jrnl";
  version = "2.3";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "ede6b3783562b03eaab64679d1913d335079535dbc8cca83bffbd65dacb4bcb6";
  };

  propagatedBuildInputs = with pkgs.python36.pkgs; [
pyxdg
cryptography
passlib
parsedatetime
keyring
pytz
tzlocal
asteval
colorama
python-dateutil
pyyaml
ansiwrap
setuptools_scm
  ];

  doCheck = false;
};
asteval = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "asteval";
  version = "0.9.18";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "5d64e18b8a72c2c7ae8f9b70d1f80b68bbcaa98c1c0d7047c35489d03209bc86";
  };
  doCheck = false;
};
# https://github.com/NixOS/nixpkgs/search?q=textwrap3&unscoped_q=textwrap3
ansiwrap = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "ansiwrap";
  version = "0.8.4";
  extension = "zip";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version extension;
    sha256 = "ca0c740734cde59bf919f8ff2c386f74f9a369818cdc60efe94893d01ea8d9b7";
  };

  propagatedBuildInputs = with pkgs.python36.pkgs; [
textwrap3
  ];

  doCheck = false;
};
textwrap3 = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "textwrap3";
  version = "0.9.2";
  extension = "zip";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version extension;
    sha256 = "5008eeebdb236f6303dcd68f18b856d355f6197511d952ba74bc75e40e0c3414";
  };

  doCheck = false;
};
keyring = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "keyring";
  version = "19.3.0";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "ee3d35b7f1ac3cb69e9a1e4323534649d3ab2fea402738a77e4250c152970fed";
  };

  propagatedBuildInputs = with pkgs.python36.pkgs; [
    SecretStorage
    setuptools_scm
    importlib_metadata
    toml
    zipp
  ];

  doCheck = false;
};
SecretStorage = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "SecretStorage";
  version = "3.1.2";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "15da8a989b65498e29be338b3b279965f1b8f09b9668bd8010da183024c8bff6";
  };

  propagatedBuildInputs = with pkgs.python36.pkgs; [
    cryptography
    jeepney
  ];

  doCheck = false;
};
# https://github.com/search?q=.nix+repo%3ANixOS%2Fnixpkgs+path%3A%2Fpkgs%2Fdevelopment%2Fpython-modules%2Fcryptography&type=Code&ref=advsearch&l=&l=
cryptography = pkgs.python36.pkgs.buildPythonPackage rec {
  pname = "cryptography";
  version = "2.8";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "3cda1f0ed8747339bbdf71b9f38ca74c7b592f24f65cdb3ab3765e4b02871651";
  };

  outputs = [ "out" "dev" ];

  buildInputs = [ openssl ];
  propagatedBuildInputs =  with pkgs.python36.pkgs; [
    packaging six pkgs.gcc
  ] ++ stdenv.lib.optional (!isPyPy) cffi;

  doCheck = false;
};
jeepney = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "jeepney";
  version = "0.4.3";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "3479b861cc2b6407de5188695fa1a8d57e5072d7059322469b62628869b8e36e";
  };

  doCheck = false;
};
importlib_metadata = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "importlib_metadata";
  version = "1.5.2";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "dfc83688553a91a786c6c91eeb5f3b1d31f24d71877bbd94ecbf5484e57690a2";
  };
  
  propagatedBuildInputs = with pkgs.python36.pkgs; [
    setuptools_scm
    zipp
  ];

  doCheck = false;
};
zipp = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "zipp";
  version = "0.5.2";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "4970c3758f4e89a7857a973b1e2a5d75bcdc47794442f2e2dd4fe8e0466e809a";
  };
  
  propagatedBuildInputs = with pkgs.python36.pkgs; [
    setuptools_scm
    toml
  ];

  doCheck = false;
};
toml = pkgs.python36.pkgs.buildPythonApplication rec {
  pname = "toml";
  version = "0.10.0";

  src = pkgs.python36.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "229f81c57791a41d65e399fc06bf0848bab550a9dfd5ed66df18ce5f05e73d5c";
  };

  doCheck = false;
};
in
  {
    home.packages = with pkgs; [
      (import jrnl {pname = "jrnl";})
    ];
  }

Missing is the pythonImportsCheckPhase:

Successfully installed jrnl-2.3
/tmp/nix-build-jrnl-2.3.drv-0/jrnl-2.3
Finished executing pipInstallPhase
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/7m0r7qhy8c9sm34qj596mizf5hlkdhk5-jrnl-2.3
strip is /nix/store/ajrrkivdfvp8dp4vdg5hp1h5hblmanc9-binutils-2.31.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/7m0r7qhy8c9sm34qj596mizf5hlkdhk5-jrnl-2.3/lib /nix/store/7m0r7qhy8c9sm34qj596mizf5hlkdhk5-jrnl-2.3/bin
patching script interpreter paths in /nix/store/7m0r7qhy8c9sm34qj596mizf5hlkdhk5-jrnl-2.3
checking for references to /tmp/nix-build-jrnl-2.3.drv-0/ in /nix/store/7m0r7qhy8c9sm34qj596mizf5hlkdhk5-jrnl-2.3…
Rewriting #!/nix/store/8a73adngr6j974nv3scsnbsi47gn4i81-python3-3.6.9/bin/python3.6 to #!/nix/store/8a73adngr6j974nv3scsnbsi47gn4i81-python3-3.6.9
wrapping `/nix/store/7m0r7qhy8c9sm34qj596mizf5hlkdhk5-jrnl-2.3/bin/jrnl’…
pythonCatchConflictsPhase
pythonRemoveBinBytecodePhase
pythonImportsCheckPhase
Executing pythonImportsCheckPhase
killing process 6002: No such process
error: opening file ‘/nix/store/7m0r7qhy8c9sm34qj596mizf5hlkdhk5-jrnl-2.3/default.nix’: No such file or directory

Anybody who could help from here ?

@danbst, @ivanbrennan maybe ?