Weird overlay behavior with python packages

Help
1

Recently I’ve encountered a problem with my personal overlay containing custom versions of Python packages from nixpkgs. When using two overlays for a package (replacing the existing package with my version; overriding some options of that package), it seems like the package entry “resets” to the nixpkgs version. Curiosly, this does not happen if a package is in pkgs root.

Here is a simplified example.

Customized packages:

custom_hello.nix:

{ stdenv, optionA ? false, optionB ? false }:

stdenv.mkDerivation {
  pname = "hello";

  passthru = {
    inherit optionA optionB;
  };
}

custom_numpy.nix:

{ buildPythonPackage, optionA ? false, optionB ? false }:

buildPythonPackage {
  pname = "numpy";

  passthru = {
    inherit optionA optionB;
  };
}

Overlays:

overlays_hello.nix:

{
  overlay1 = self: super: with super; {
    hello = callPackage ./custom_hello.nix {};
  };

  overlay2 = self: super: {
    hello = super.hello.override { optionA = true; };
  };
}

overlays_numpy.nix:

{
  overlay1 = self: super: {
    python3 = super.python3.override {
      packageOverrides = pself: psuper: with super.python3.pkgs; {
        numpy = callPackage ./custom_hello.nix {};
      };
    };
  };

  overlay2 = self: super: {
    python3 = super.python3.override {
      packageOverrides = pself: psuper: {
        numpy = psuper.numpy.override { optionA = true; };
      };
    };
  };
}

In nix-repl

nix-repl> o_h = import ./overlays_hello.nix 
nix-repl> o_n = import ./overlays_numpy.nix
# "Root" package
nix-repl> pkgs = import <nixpkgs> { overlays = [ o_h.overlay1 ]; } 
nix-repl> pkgs.hello.optionA                                       
false
nix-repl> pkgs = import <nixpkgs> { overlays = [ o_h.overlay1 o_h.overlay2 ]; }
nix-repl> pkgs.hello.optionA                                                    
true
# All good!
# Python package
nix-repl> pkgs = import <nixpkgs> { overlays = [ o_n.overlay1 ]; }              
nix-repl> pkgs.python3Packages.numpy.optionA
false
nix-repl> pkgs = import <nixpkgs> { overlays = [ o_n.overlay1 o_n.overlay2 ]; }
nix-repl> pkgs.python3Packages.numpy.optionA                                    
error: anonymous function at /nix/store/ifg0cf324ylhvicadnh3pq0rzn8p8n9w-nixos-21.05.2132.733682c3292/nixos/pkgs/development/python-modules/numpy/default.nix:1:1 called with unexpected argument 'optionA', at /nix/store/ifg0cf324ylhvicadnh3pq0rzn8p8n9w-nixos-21.05.2132.733682c3292/nixos/lib/customisation.nix:69:16
# :(

Am I missing something here? Is it a bug?

2

Common mistake, you need to pass in self.

3

Thanks for the answer. Unfortunately, it didn’t work in my case. I used the following overlays_numpy.nix:

{
  overlay1 = self: super: rec {
    python3 = super.python3.override {
      self = python3;
      packageOverrides = pself: psuper: with super.python3.pkgs; {
        numpy = callPackage ./custom_numpy.nix {};
      };
    };
  };

  overlay2 = self: super: rec {
    python3 = super.python3.override {
      self = python3;
      packageOverrides = pself: psuper: {
        numpy = psuper.numpy.override { optionA = true; };
      };
    };
  };
}

The same error:

nix-repl> o_n = import ./overlays_numpy.nix                                     

nix-repl> pkgs = import <nixpkgs> { overlays = [ o_n.overlay1 o_n.overlay2 ]; }

nix-repl> pkgs.python3Packages.numpy.optionA                                    
error: anonymous function at /nix/store/ifg0cf324ylhvicadnh3pq0rzn8p8n9w-nixos-21.05.2132.733682c3292/nixos/pkgs/development/python-modules/numpy/default.nix:1:1 called with unexpected argument 'optionA', at /nix/store/ifg0cf324ylhvicadnh3pq0rzn8p8n9w-nixos-21.05.2132.733682c3292/nixos/lib/customisation.nix:69:16

In this thread it is mentioned that python.override breaks if it is used in more than one overlay. Is it true? If it is, how to work around this situation?

4

After some more experimentation it seems to me that when overriding python package set the original Python package set is always used as “super”. E.g., newly added packages via python.override disappear even if python = python.override { self = python; packageOverrides self: super: {}; };. Here is an example:

let
  pkgs = import <nixpkgs> {};

  testPkg = ({ buildPythonPackage, someOption ? false }:
    buildPythonPackage {
      pname = "my-test";
      version = "0.1";
      passthru = { inherit someOption; };
  });

  # First override
  python3_1 = pkgs.python3.override {
    self = python3_1;
    packageOverrides = self: super: {
      my-test = super.pkgs.callPackage testPkg { inherit (super) buildPythonPackage; };
    };
  };

  # Second override
  python3_2 = python3_1.override {
    self = python3_2;
    packageOverrides = self: super: {
      my-test = super.my-test.override { someOption = true; };
    };
  };

in builtins.trace python3_2.pkgs.my-test.someOption derivation { }

While calling nix-build --no-out-link multi_python_overrides.nix it returns not this:

trace: true
error: derivation name missing

but this

error: attribute 'my-test' missing, at multi_python_overrides.nix:23:17

If package override function returns an empty attrset, the error is about missing my-part attribute in python3_2.pkgs. Trying to modify only the package set as per manual (myPythonPackages = pkgs.pythonPackages.override { overrides = self: super: {...}; };), the same problem arises.

I’ve searched the Nixpkgs docs and haven’t found any clue about what I’m missing. I haven’t seen any examples of multiple overrides though.

5

Essentially in the current state of things, Python overrides do not compose. I don’t have my browser history at hand, but if you search for issues and PRs on GitHub you will find multiple unfinished attempts to rework this.

1 Like
6

Thanks a lot for the info. Well, it’s quite unfortunate, and I don’t feel knowledgeable enough of Nix/Nixpkgs to do a proper rework myself. I’ll just try to hack a solution that would at least work in my specific case.

7

Indeed, that’s Python's packageOverrides isn't composable · Issue #44426 · NixOS/nixpkgs · GitHub.

1 Like