How to mix and match Python libraries from stable/unstable

Help
1

Mixing versions of packages between the latest stable NixOS release and nixos-unstable is very easy. However, I don’t understand how to do the same thing for Python libraries. This is my failed attempt to mix xgboost from nixpkgs-unstable with scikit-learn from nixos-24.11 using an overlay.

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  inputs.nixpkgs-stable.url = "github:NixOS/nixpkgs/nixos-24.11";

  outputs =
    { nixpkgs, nixpkgs-stable, ... }:
    let
      system = "x86_64-linux";
      pkgs-stable = import nixpkgs-stable { system = system; };
      pkgs = import nixpkgs {
        system = system;
        overlays = [
          (final: prev: {
            pythonPackagesExtensions = prev.pythonPackagesExtensions ++ [
              (pythonFinal: pythonPrev: {
                scikit-learn = pkgs-stable.python312Packages.scikit-learn;
              })
            ];
          })
        ];
      };
      pythonPkgs = p: with p; [
        xgboost
        scikit-learn
        # this didn't work either
        # pkgs-stable.python312.scikit-learn
      ];
      python = pkgs.python312.withPackages pythonPkgs;
    in
      {
      devShells.${system}.default = pkgs.mkShellNoCC { packages = [ python ]; };
    };
}

Does anyone know how to make this work?

2

Binary/app packages are easy to mix because they are totally isolated and can coexist in the same environment. However, python libraries are another story.

You cannot have various versions of a python library within a python environment. Python resolves the importations with whatever version it encounters on the installed library’s location (a.k.a. “site-packages”). Because of this, the python packages on nixpkgs have to be synchronized, in the sense that there are no packages with impossible requirements (at the python lib level).

If the error is some kind of library collision, you will have to override the dependencies of scikit-learn to match the ones defined by the unstable nixpkgs.

Fortunately, when testing the build, no error was raised. Try to update the flakes inputs again.

The ones I got success were

unstable 3a05eebede89661660945da1f151959900903b6a
stable 5d7db4668d7a0c6cc5fc8cf6ef33b008b2b1ed8b

3

So it is possible to make it work for packages that don’t share any dependencies then?

I forgot to mention that the flake I shared works in the sense that it won’t raise any errors when entering the shell environment with nix develop; however, when you enter the python shell and try to import sklearn you get a ModuleNotFoundError (import xgboost works fine). I don’t know whether my flake.nix is wrong or this “silent fail” is expected when mixing Python libraries from different releases.

4

You can ignore collisions if you use pkgs.python312.buildEnv.override to build the python environment, however it seems like withPackages has the same limitation that buildEnv has…

The derivations you pass on pythonPkgs are filtered to ensure they are python module derivations. If you pass some non python module derivation, it gets ignored. However, even if a derivation is a python module, it also gets ignored if it was generated for another interpreter different from the base one.

This detail present on the src code of nixpkgs

i.e. ./nixpkgs/pkgs/development/interpreters/python/python-packages-base.nix on the definitions of hasPythonModule and requiredPythonModules

Therefore, if the two nixpkgs versions you are using differ from its base python version, depending on which version you call python312.withPackages or python312.buildEnv the other nixpkgs modules gets ignored. That is why pkgs-stable.python312.scikit-learn does not work.

From what I know, there are two options to make this work, but using a single nixpkgs version as package source.

  1. Overlays: you can use overlays as you were trying but taking in account some other details. In this approach, you could use the nixpkgs-stable to get the nix src file to be able to call callPackage.
  2. Python patch: with-out overlays, you can use scikit-learn.overridePythonAttrs to force a kind of rollback over the latest version on nixpkgs. Commonly, only an override over the pkg version. Here you can also use the nixpkgs-stable src file if the override requires so many adjustments that it is better to copy the original file.
1 Like