How I can successfully override a python lib with overlays?

I m trying to create a dev environment for a python/flask project but some libs do not exists in nixpkgs. So I created a overlay:

{ pkgs, fetchurl }:

with pkgs.python38Packages;

self: super: {
  "python-docx" = super.buildPythonPackage rec {
    pname = "python-docx";
    version = "0.8.11";
    src = fetchPypi {
      inherit pname version;
      sha256 = "bc76ecac6b2d00ce6442a69d03a6f35c71cd72293cd8405a7472dfe317920024";
    };
    doCheck = false;
    checkPhase = "";
    propagatedBuildInputs = [ lxml ];
  };
  "docxtpl" = super.buildPythonPackage rec {
    pname = "docxtpl";
    version = "0.11.1";
    src = fetchPypi {
      inherit pname version;
      sha256 = "1w7ah501mw7zz2rvkmnj32b6pwcna0yijgd0ax1z5g1irs9fm62w";
    };
    format = "wheel";
    doCheck = false;
    propagatedBuildInputs = [
      super."python-docx"
      jinja2
      lxml
      six
    ];
  };
  "flask-debugtoolbar" = super.buildPythonPackage rec {
    pname = "flask-debugtoolbar";
    version = "0.11.0";
    src = fetchurl {
      url = "https://files.pythonhosted.org/packages/0e/60/c3e8de2191ecf78d750d1038ab45195b97a9ee569351865d60d01941bcc6/Flask_DebugToolbar-0.11.0-py2.py3-none-any.whl";
      sha256 = "1cy5kavcap4k317v5ks8aanba032jycwr0bf6xl3q8wrqpa816hf";
    };
    format = "wheel";
    doCheck = false;
    propagatedBuildInputs = [
      blinker
      flask
      itsdangerous
      werkzeug
    ];
  };
  "flask-moment" = super.buildPythonPackage rec {
    pname = "flask-moment";
    version = "1.0.1";
    src = fetchurl {
      url = "https://files.pythonhosted.org/packages/3d/de/1389a8c7e5eaf96a4b8ea5abccc262f7eee47ce76aca3a0c1b3527930fad/Flask_Moment-1.0.1-py3-none-any.whl";
      sha256 = "15y4xmp7vzjin7vdx6l7gqhw17wyg02bmfnmwyx5ra1msmk56lgj";
    };
    format = "wheel";
    doCheck = false;
    propagatedBuildInputs = [
      flask
    ];
  };
  "flask-shell-ipython" = super.buildPythonPackage rec {
    pname = "flask-shell-ipython";
    version = "0.4.1";
    src = fetchurl {
      url = "https://files.pythonhosted.org/packages/10/6b/a45278cbff711cc7b8904bd38679a8cc249c4db825a76ae332302f59d398/flask_shell_ipython-0.4.1-py2.py3-none-any.whl";
      sha256 = "1lwhk080kvzr3ys6wl55zbnic1zx0p6ikiwr4xjxy7l3svxb84pj";
    };
    format = "wheel";
    doCheck = false;
    propagatedBuildInputs = [
      flask
      ipython
      click
    ];
  };
  "flask-weasyprint" = super.buildPythonPackage rec {
    pname = "flask-weasyprint";
    version = "0.6";
    src = fetchurl {
      url = "https://files.pythonhosted.org/packages/0f/b5/7292f35a199ea50ac82b06d2a9000a51d16282469f674686fcadc2ae00fb/Flask_WeasyPrint-0.6-py2.py3-none-any.whl";
      sha256 = "1brccbjnnanmkq1c6xjnhqi74rnz0d55qagbzqankc718z6j9138";
    };
    format = "wheel";
    doCheck = false;
    propagatedBuildInputs = [
      flask
      weasyprint
    ];
  };
  "jinja2-webpack" = super.buildPythonPackage rec {
    pname = "jinja2-webpack";
    version = "0.2.0";
    src = fetchurl {
      url = "https://files.pythonhosted.org/packages/bd/e3/adae7bc99a68f5b0cc9c89a1681e073bc1b29d78ac98b964691720f0989a/jinja2_webpack-0.2.0-py2.py3-none-any.whl";
      sha256 = "0dvz4lwk7nlxghbivas9fjbbggiav3mw83lxr0j6cfnsxvln3kzr";
    };
    format = "wheel";
    doCheck = false;
    propagatedBuildInputs = [
      jinja2
    ];
  };
  "sqlalchemy-mixins" = super.buildPythonPackage rec {
    pname = "sqlalchemy-mixins";
    version = "1.4";
    src = fetchurl {
      url = "https://files.pythonhosted.org/packages/c8/28/2d018a09097c30746461f50da4a2ff3346a1ed6a5bac7bc12583124ddbda/sqlalchemy_mixins-1.4-py3-none-any.whl";
      sha256 = "0rbzfr0m2b28miyc3ii70wma999wh949sxdv0m90r6hgrk36rxfj";
    };
    format = "wheel";
    doCheck = false;
    propagatedBuildInputs = [
      sqlalchemy
      six
    ];
  };
  "validate-docbr" = super.buildPythonPackage rec {
    pname = "validate-docbr";
    version = "1.8.2";
    src = fetchurl {
      url = "https://files.pythonhosted.org/packages/a6/ff/f68743b87156a375d33e278f081d6ab95793dda9bfae5f0fb4e376d426e5/validate_docbr-1.8.2-py3-none-any.whl";
      sha256 = "1ym5m2fxjsfw5qkcrq28dyv50bg8yib6q4mqdz0xpqwpxjw8g0i0";
    };
    format = "wheel";
    doCheck = false;
  };
}

However the lib python-docx is failing in two tests:

patching script interpreter paths in /nix/store/8qcr8i0wmadjvfisi2x04ay8kbjcllfi-python3.8-python-docx-0.8.10
checking for references to /build/ in /nix/store/8qcr8i0wmadjvfisi2x04ay8kbjcllfi-python3.8-python-docx-0.8.10...
Executing pythonRemoveTestsDir
Finished executing pythonRemoveTestsDir
running install tests
============================= test session starts ==============================
platform linux -- Python 3.8.9, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
rootdir: /build/python-docx-0.8.10, configfile: tox.ini
collected 1460 items

tests/test_api.py ...                                                    [  0%]
tests/test_blkcntnr.py ....F...........                                  [  1%]
tests/test_document.py .............................                     [  3%]
tests/test_enum.py .........                                             [  3%]
tests/test_package.py ........                                           [  4%]
tests/test_section.py .................................................. [  7%]
..........................................                               [ 10%]
tests/test_settings.py .........                                         [ 11%]
tests/test_shape.py .............                                        [ 12%]
tests/test_shared.py ....................                                [ 13%]
tests/test_table.py .................................................... [ 17%]
........................................................................ [ 22%]
.......                                                                  [ 22%]
tests/dml/test_color.py ...............................                  [ 24%]
tests/image/test_bmp.py ...                                              [ 24%]
tests/image/test_gif.py ...                                              [ 25%]
tests/image/test_helpers.py ....                                         [ 25%]
tests/image/test_image.py ......................................         [ 28%]
tests/image/test_jpeg.py ......................................          [ 30%]
tests/image/test_png.py .........................                        [ 32%]
tests/image/test_tiff.py ...........................                     [ 34%]
tests/opc/test_coreprops.py ..................................           [ 36%]
tests/opc/test_oxml.py ..............                                    [ 37%]
tests/opc/test_package.py ...................                            [ 38%]
tests/opc/test_packuri.py .........                                      [ 39%]
tests/opc/test_part.py ........................                          [ 41%]
tests/opc/test_phys_pkg.py ...................                           [ 42%]
tests/opc/test_pkgreader.py ......................                       [ 43%]
tests/opc/test_pkgwriter.py ...........                                  [ 44%]
tests/opc/test_rel.py ..............                                     [ 45%]
tests/opc/parts/test_coreprops.py ..                                     [ 45%]
tests/oxml/test__init__.py ........                                      [ 46%]
tests/oxml/test_ns.py .......                                            [ 46%]
tests/oxml/test_styles.py ..                                             [ 46%]
tests/oxml/test_table.py ............................................... [ 50%]
..................                                                       [ 51%]
tests/oxml/test_xmlchemy.py ............................................ [ 54%]
.................................                                        [ 56%]
tests/oxml/parts/test_document.py ......                                 [ 56%]
tests/oxml/text/test_run.py ...                                          [ 57%]
tests/parts/test_document.py ..................                          [ 58%]
tests/parts/test_hdrftr.py ......                                        [ 58%]
tests/parts/test_image.py .......                                        [ 59%]
tests/parts/test_numbering.py .....                                      [ 59%]
tests/parts/test_settings.py ...                                         [ 59%]
tests/parts/test_story.py ..............                                 [ 60%]
tests/parts/test_styles.py ..                                            [ 60%]
tests/styles/test_latent.py ............................................ [ 63%]
..................                                                       [ 65%]
tests/styles/test_style.py ............................................. [ 68%]
..................................F...                                   [ 70%]
tests/styles/test_styles.py .......................................      [ 73%]
tests/text/test_font.py ................................................ [ 76%]
.....................................................................    [ 81%]
tests/text/test_paragraph.py .....................................       [ 84%]
tests/text/test_parfmt.py .............................................. [ 87%]
........................................................................ [ 92%]
..                                                                       [ 92%]
tests/text/test_run.py ................................................. [ 95%]
.....................                                                    [ 97%]
tests/text/test_tabstops.py ..........................................   [100%]

=================================== FAILURES ===================================
________________ DescribeBlockItemContainer.it_can_add_a_table _________________

self = <tests.test_blkcntnr.DescribeBlockItemContainer object at 0x7ffff4ef5a90>
add_table_fixture = (<docx.blkcntnr.BlockItemContainer object at 0x7ffff5ac40a0>, 2, 2, 1828800, '<w:tbl xmlns:w="http://schemas.openxmlfo...  <w:tcPr>\n        <w:tcW w:type="dxa" w:w="1440"/>\n      </w:tcPr>\n      <w:p/>\n    </w:tc>\n  </w:tr>\n</w:tbl>')

    def it_can_add_a_table(self, add_table_fixture):
        blkcntnr, rows, cols, width, expected_xml = add_table_fixture
        table = blkcntnr.add_table(rows, cols, width)
        assert isinstance(table, Table)
>       assert table._element.xml == expected_xml
E       AssertionError: assert '<w:tbl xmlns...w:body>\n\n\n' == '<w:tbl xmlns...tr>\n</w:tbl>'
E         Skipping 814 identical leading characters in diff, use -v to show
E           r>
E         - </w:tbl>
E         + </w:tbl>
E         ?         +
E         + </w:body>
E         + ...
E
E         ...Full output truncated (2 lines hidden), use '-vv' to show

tests/test_blkcntnr.py:37: AssertionError
_ Describe_ParagraphStyle.it_can_change_its_next_paragraph_style[next_set_fixture0] _

self = <tests.styles.test_style.Describe_ParagraphStyle object at 0x7ffff4b8cb50>
next_set_fixture = (_ParagraphStyle('None') id: 140737315902560, _ParagraphStyle('None') id: 140737302483728, '<w:style xmlns:w="http://s...openxmlformats.org/wordprocessingml/2006/main" w:type="paragraph" w:styleId="H">\n  <w:next w:val="B"/>\n</w:style>\n')

    def it_can_change_its_next_paragraph_style(self, next_set_fixture):
        style, next_style, expected_xml = next_set_fixture
        style.next_paragraph_style = next_style
>       assert style.element.xml == expected_xml
E       assert '<w:style xml...styles>\n\n\n' == '<w:style xml...n</w:style>\n'
E         Skipping 136 identical leading characters in diff, use -v to show
E           </w:style>
E         + <w:style w:type="paragraph" w:styleId="B"/>
E         + </w:styles>
E         +
E         +

tests/styles/test_style.py:474: AssertionError
=============================== warnings summary ===============================
docx/section.py:7
  /build/python-docx-0.8.10/docx/section.py:7: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
    from collections import Sequence

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
FAILED tests/test_blkcntnr.py::DescribeBlockItemContainer::it_can_add_a_table
FAILED tests/styles/test_style.py::Describe_ParagraphStyle::it_can_change_its_next_paragraph_style[next_set_fixture0]
================== 2 failed, 1458 passed, 1 warning in 7.90s ===================
builder for '/nix/store/4sgn9ry5xhf1b8s5i7fcqr0bi6n6l662-python3.8-python-docx-0.8.10.drv' failed with exit code 1
cannot build derivation '/nix/store/57wqqi4d1flyc5k83x95wv2xkp42gk1n-python3-3.8.9-env.drv': 1 dependencies couldn't be built
error: build of '/nix/store/57wqqi4d1flyc5k83x95wv2xkp42gk1n-python3-3.8.9-env.drv' failed

My problem is: in my overlay, I build the python-docx v0.8.11, disabling the checkPhase, however in the derivation’s build, the version which is built is 0.8.10. What I’m doinf wrong?

I’m using the overlay as:

packageOverrides = callPackage ./extra-legacy-packages.nix {};

  python38 = python38Full.override { inherit packageOverrides; };

  pythonWithPackages = python38.withPackages(ps: with ps; [
    flask-debugtoolbar
    flask-moment
    flask-shell-ipython
    flask-weasyprint
    jinja2-webpack
    sqlalchemy-mixins
    validate-docbr
    docxtpl
  ] ++ pypkgs);

Maybe this is exactly what you are asking, but python-docx does exist in nixpkgs already and it looks like it’s at 0.8.10 so it seems python-docx isn’t getting pulled in from your overlay but from nixpkgs.

Taking a guess, maybe change super."python-docx" to self."python-docx"?

1 Like

is exactly what is happening! yeah, which does not exists is docxtpl that needs python-docx to be built.

i already tried to change from super. to self. but this invalidate the overlay philosophy and i also receive this error:

these derivations will be built:
  /nix/store/y6zidf2k001ckqzihv1s46psr92plll6-python3.8-docxtpl-0.11.1.drv
  /nix/store/ahqy18nvk0pnxdp99k5b1s6zv60ng9ci-python3-3.8.9-env.drv
building '/nix/store/y6zidf2k001ckqzihv1s46psr92plll6-python3.8-docxtpl-0.11.1.drv'...
Sourcing python-remove-tests-dir-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing wheel setup hook
Using wheelUnpackPhase
Sourcing pip-install-hook
Using pipInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
Sourcing python-namespaces-hook
unpacking sources
Executing wheelUnpackPhase
Finished executing wheelUnpackPhase
patching sources
configuring
no configure script, doing nothing
building
no Makefile, doing nothing
installing
Executing pipInstallPhase
/build/dist /build
ERROR: You must give at least one requirement to install (see "pip help install")
builder for '/nix/store/y6zidf2k001ckqzihv1s46psr92plll6-python3.8-docxtpl-0.11.1.drv' failed with exit code 1
cannot build derivation '/nix/store/ahqy18nvk0pnxdp99k5b1s6zv60ng9ci-python3-3.8.9-env.drv': 1 dependencies couldn't be built
error: build of '/nix/store/ahqy18nvk0pnxdp99k5b1s6zv60ng9ci-python3-3.8.9-env.drv' failed

reference: Self and super in nix overlays - Stack Overflow

or i missed something in the definition of overlays?

It’s possible self is correct and just now you are seeing issues with the package(s) in the overlay. It’s also still possible I’m wrong :-p . Have you verified your python-docx works as a standalone package?

1 Like

I tried to use python-docx from pkgs.python38Packages, but I got this:

these 2 derivations will be built:
  /nix/store/wjrvbb1rxy9jkjn9f97n1cijg65rs58m-python3.8-docxtpl-0.11.1.drv
  /nix/store/xxhp0v9ldd4gvk0vqlw6k8ivw886c2bf-python3-3.8.11-env.drv
this path will be fetched (0.21 MiB download, 1.07 MiB unpacked):
  /nix/store/g81q67h7wbdwz7m9lbggx6lwli62564n-python3.8-python-docx-0.8.11
copying path '/nix/store/g81q67h7wbdwz7m9lbggx6lwli62564n-python3.8-python-docx-0.8.11' from 'https://cache.nixos.org'...
building '/nix/store/wjrvbb1rxy9jkjn9f97n1cijg65rs58m-python3.8-docxtpl-0.11.1.drv'...
Sourcing python-remove-tests-dir-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing wheel setup hook
Using wheelUnpackPhase
Sourcing pip-install-hook
Using pipInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
Sourcing python-namespaces-hook
unpacking sources
Executing wheelUnpackPhase
Finished executing wheelUnpackPhase
patching sources
configuring
no configure script, doing nothing
building
no Makefile, doing nothing
installing
Executing pipInstallPhase
/build/dist /build
ERROR: You must give at least one requirement to install (see "pip help install")
error: builder for '/nix/store/wjrvbb1rxy9jkjn9f97n1cijg65rs58m-python3.8-docxtpl-0.11.1.drv' failed with exit code 1;
       last 10 log lines:
       > Finished executing wheelUnpackPhase
       > patching sources
       > configuring
       > no configure script, doing nothing
       > building
       > no Makefile, doing nothing
       > installing
       > Executing pipInstallPhase
       > /build/dist /build
       > ERROR: You must give at least one requirement to install (see "pip help install")
       For full logs, run 'nix log /nix/store/wjrvbb1rxy9jkjn9f97n1cijg65rs58m-python3.8-docxtpl-0.11.1.drv'.
error: 1 dependencies of derivation '/nix/store/xxhp0v9ldd4gvk0vqlw6k8ivw886c2bf-python3-3.8.11-env.drv' failed to build

Ok, i maybe found the solution: use the hardcoded url from pypi to the wheel file! Also I used the python-docx from pkgs