(python) Package duplicates in closure when overriding

Help
1

I’m trying to override Python to use a weakened version of OpenSSL. That overridden interpreter works with a good number of packages, but it does not at all with an equal number. Mostly what I end up with is that the failing packages have duplicate dependencies in their closure?

Here is a pretty minimal example to reproduce:

let
  pkgs = import (builtins.fetchTarball {
      url = "https://github.com/NixOS/nixpkgs/archive/refs/tags/23.05.tar.gz";
      sha256 = "10wn0l08j9lgqcw8177nh2ljrnxdrpri7bp0g7nvrsn9rkawvlbf";
    }) {
      config = {};
      overlays = [ opensslOverlay pythonOverlay ];
    };

  opensslOverlay = final: prev: {
    weakOpenssl = prev.openssl.overrideAttrs (old: {
      configureFlags = [ "enable-ssl3" "enable-ssl3-method" "enable-weak-ssl-ciphers" ] ++ (old.configureFlags or []);
    });
  };

  pythonOverlay = final: prev: {
    weakPython = final.python311.override {
      openssl = true;
      openssl_legacy = final.weakOpenssl;
      self = final.weakPython;
    };
  };
in
pkgs.mkShell {
  name = "weakpy";
  buildInputs = with pkgs.weakPython.pkgs; [ python paramiko ];   # cryptography pyopenssl azure-mgmt-network boto3 ,...
  shellHook = ''
    python -c 'import paramiko; print(f"imported {paramiko}")'
  '';
}

Why is that? Is it a nixpkgs bug? How do I track these down? How do I eventually fix that for all the Python packages I want to use with the weakend OpenSSL?

Found duplicated packages in closure for dependency 'six': 
  six 1.16.0 (/nix/store/41cwsgbqz23lr8pz73y91nabclgd31gb-python3.11-six-1.16.0/lib/python3.11/site-packages)
  six 1.16.0 (/nix/store/skpaa2kjggn9ycxi842rpj4izydvbfb8-python3.11-six-1.16.0/lib/python3.11/site-packages)

Errors just linked for brevity.

2

I also noticed, that for some packages, I can get them working by overriding their OpenSSL:

  pythonOverlay = final: prev: {
    weakPython = final.python311.override {
      openssl = true;
      openssl_legacy = final.weakOpenssl;
      self = final.weakPython;

      packageOverrides = pyfinal: pyprev: {
        pyopenssl = pyprev.pyopenssl.override {
          openssl = final.weakOpenssl;
        };
        cryptography = pyprev.cryptography.override {
          openssl = final.weakOpenssl;
        };
      };
    };
3

python can’t handle having two different versions of the same packages in its dependencies. You can try nix why-depends or diffoscope to figure out from where those different six are coming, maybe somewhere are impurely pythons mixed together or you need to overwrite python3Minimal, too. I am just guessing at this point.

I am not sure if you can create python3.pkgs.openssl to circumvent this.

1 Like
4

I’m not really sure, what you mean by that. openssl is a function argument to many python package definitions. They expect you to pass in a version of OpenSSL (as their openssl argument) that the package than should use and build against. They don’t refer to python3.pkgs.openssl at all, in fact, I don’t think that normally exists.

5

My idea was, if you could trick pythonPackages by defining this variable to use that openssl over pkgs.openssl. I don’t know if that works but it would be interesting to try out.

6

Ah! Gotcha you meant to override pkgs.openssl (i.e. the top level openssl symbol) not as I interpreted pkgs.python3.pkgs.openssl (which does not exist, hence my confusion)! But wouldn’t that affect a lot of packages, even non-python ones? That is the whole reason, why I gave the weakened Python a new symbol instead of overriding pkgs.pythonInterpreters.python3 or something, I didn’t want to use the weak openssl in everyting.

7

I meant this :upside_down_face: You can create new packages in overlays, they don’t need to exist beforehand.