Another pythonPackages overlay not working thread

My code, fauxmo: GitHub - n8henrie/fauxmo at nix-module

Having read and tried multiple variations from the reading:

I am still really struggling to figure how to provide an overlay via my flake that can be used to successfully provide python3Packages.fauxmo.

It seemed like the pythonPackagesExtensions was the newest and most promising approach, so I left it commented out as I tried other approaches, but nothing has worked so far – when I import this overlay into my system, I can often get a successful build, but the resulting module has python with uvloop but no fauxmo.

I created a devShell in the flake that used the overlay, and it correctly builds and has fauxmo available (e.g. python -m fauxmo.cli --help). However, when I include this overlay in a system, the resulting systemd service can’t find the module – and for good reason, it’s not there:

$ systemctl cat fauxmo | awk -F= '/ExecStart/ { print $2 }' | xargs cat
#!/nix/store/xmwznc4kdln0rg4jwpmdfkpgyff01jmz-bash-5.2-p15/bin/bash
set -e
/nix/store/1bzhmyvlidmmpwh5j070bg9ga8im58cc-python3-3.11.6-env/bin/python -m fauxmo.cli  -c "/home/n8henrie/git/fauxmo/config.json"
$ ls /nix/store/1bzhmyvlidmmpwh5j070bg9ga8im58cc-python3-3.11.6-env/lib/python3.11/site-packages/
__pycache__  README.txt  sitecustomize.py  _sysconfigdata__linux_aarch64-linux-gnu.py  uvloop  uvloop-0.19.0.dist-info

This makes me very confused. It seems to find the module at build time / doesn’t fail to build, but for some reason it’s not there.

To save a click, the crux of the module is:

 systemd.services.${service_name} = let
        pythonWithFauxmo =
          pkgs.python3.withPackages
          (ps:
            with ps; [
              fauxmo
              uvloop
            ]);
        after = ["network-online.target"];
      in {
        description = service_name;
        inherit after;
        requires = after;
        script = let
          verbosity =
            if cfg.verbosity == 0
            then ""
            else "-" + lib.concatStrings (builtins.genList (_: "v") cfg.verbosity);
        in ''${pythonWithFauxmo}/bin/python -m fauxmo.cli ${verbosity} -c "${cfg.configFile}"'';

my current version of the overlay:

 overlays.default = _: prev: rec {
        python3 = let
          packageOverrides = _: _: {
            inherit (self.outputs.packages.${prev.system}) fauxmo;
          };
        in
          prev.python3.override {
            inherit packageOverrides;
            self = python3;
          };
      };

Have also tried (without success):

pythonPackagesExtensions =
 prev.pythonPackagesExtensions
 ++ [
   (
     _: _: {
       inherit (self.outputs.packages.${prev.system}) fauxmo;
     }
   )
 ];
};

When I try to build the package directly from the machine that includes the overlay, it find the package but fails:

$ nix build .#nixosConfigurations.homeslice.pkgs.python3.pkgs.fauxmo
...
error:
       … in the condition of the assert statement

         at /nix/store/52yawfmb2rz0sm07px5zcrgv3y78v27v-source/lib/customisation.nix:249:17:

          248|     in commonAttrs // {
          249|       drvPath = assert condition; drv.drvPath;
             |                 ^
          250|       outPath = assert condition; drv.outPath;

       … in the right operand of the OR (||) operator

         at /nix/store/52yawfmb2rz0sm07px5zcrgv3y78v27v-source/pkgs/development/interpreters/python/passthrufun.nix:28:45:

           27|         if lib.isDerivation value then
           28|           lib.extendDerivation (valid value || throw "${name} should use `buildPythonPackage` or `toPythonModule` if it is to be part of the Python packages set.") {} value
             |                                             ^
           29|         else

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: fauxmo should use `buildPythonPackage` or `toPythonModule` if it is to be part of the Python packages set.

Any ideas on why this is building successfully when I nixos-rebuild switch when the model is obviously missing at runtime?

I assume this message (when I try to build directly) is probably because I’m mixing nixpkgs somehow – a different set used for the overlay vs the system – or something like that (based on the link above regarding the rich python package).

Thanks for any help / suggestions!

EDIT: Should probably also include the python module in question; I changed it to use the callPackage pattern as I thought that could sometimes help with these issues (of mixing pkgs sets), but it made no difference:

{
  lib,
  python3Packages,
}: let
  inherit (python3Packages) buildPythonPackage setuptools-scm;
in
  buildPythonPackage {
    pname = "fauxmo";
    version =
      builtins.head
      (lib.findFirst (v: v != null) null
        (builtins.map
          (builtins.match "^__version__ = \"(.*)\"")
          (lib.splitString "\n" (builtins.readFile ./src/fauxmo/__init__.py))));
    src = lib.cleanSource ./.;
    format = "pyproject";
    propagatedBuildInputs = [setuptools-scm];
  }

Oh, hey, look! It’s me! :joy_cat: This is how I got the overlays to work; it’s a little confusing, so if you need any help, feel free to ask!

{
  overlays.default = final: prev: {
    python3 = python3.override (super: {
      packageOverrides =
        composeExtensions (super.packageOverrides or (_: _: { })) (new: old: {
          fauxmo = new.callPackage ({ buildPythonPackage, setuptools-scm, ... }:
            buildPythonPackage {
              pname = "fauxmo";
              version = builtins.head (lib.findFirst (v: v != null) null
                (builtins.map (builtins.match ''^__version__ = "(.*)"'')
                  (lib.splitString "\n"
                    (builtins.readFile ./src/fauxmo/__init__.py))));
              src = lib.cleanSource ./.;
              format = "pyproject";
              propagatedBuildInputs = [ setuptools-scm ];
            }) { };
        });
    });
  };
}

If that doesn’t work, try adding any combination of these to the python3 override block:

stdenv = final.gccStdenv;
self = final.python3;
pythonAttr = "python3";
1 Like

I see you do use buildPythonPackage, so it must be the drv.pythonModule == python check (pkgs/development/interpreters/python/python-packages-base.nix) that’s failing.

TLDR I’m not 100% sure what’s going on, but the way you’d usually write this is you’d use your pythonPackages’ callPackage instead of pkgs.callPackage:

pythonPackagesExtensions = ... + [ (python-final: python-prev: {
  fauxmo = python-final.callPackage ./fauxmo.nix { };
}) ];

# fauxmo.nix
{ buildPythonPackage, ... }:
buildPythonPackage ...
1 Like

You two are both amazing, I should have asked sooner. Thank you!

Sure enough, the simple change to use the python-final.callPackage worked. I figured it would have something to do with the wrong pkgs set and callPackage!

Something as simple as this seems to work (also changed fauxmo.nix to default.nix).

overlays.default = _: prev: { 
     python3 = prev.python3.override { 
          packageOverrides = py-final: _: { 
            fauxmo = py-final.callPackage ./. {}; 
          }; 
     }; 
};

EDIT: I suspect using the pythonPackagesExtensions as suggested by @SergeK may have some other advantages as well (maybe this automatically keeps python3Packages and python3.pkgs in sync?), which looks like:

overlays.default = _: prev: {
  pythonPackagesExtensions =
    prev.pythonPackagesExtensions
    ++ [(py-final: _: {fauxmo = py-final.callPackage ./. {};})];
};
1 Like

pythonPackageExtensions is mentioned in the nixpkgs manual (Nixpkgs 24.05 manual | Nix & NixOS) and makes it so that all versions of python have the package as part of it.

If you use it and your package only works with a range of python versions, you can add in a disabled flag in your package nix file (something like as disabled = !isPy3 or disabled = !pythonAtLeast "3.8") so that nix won’t try and build it if an end user tries to use a python version that it wont work with.

1 Like

Yes, as I mentioned above pythonPackageExtensions seems to be working well.

Thanks for adding the bit about disabled – I certainly find these highly builder-specific options / arguments really difficult to discover / guess (relevant: How to search nixpkgs by e.g. `buildInputs`? - #6 by n8henrie). I’ve been brainstorming how best to go through all of nixpkgs and find all examples of a specific term (e.g. pythonPackagesExtensions) and then present the user with the attrset keys and values that are used most frequently, not deeply resolved / evaluated, sorted by frequency, with links to source. I think this would be really helpful / instructional for new users.