Re-using nixpkgs derivations for different package versions

Help
1

Say I’m writing a shell.nix and want to use a package that is available in (already imported) <nixpkgs> but of different version. For instance, say I want to use pytorch-lightning = 0.9.0 whereas <nixpkgs> contains a derivation python3Packages.pytorch-lightning with version="0.8.5.". Can I reuse the expression to get 0.9.0?

Let’s take a look at the derivation for pytorch-lightning:

...
buildPythonPackage rec {
  pname = "pytorch-lightning";
  version = "0.8.5";

  disabled = isPy27;

  src = fetchFromGitHub {
    owner = "PyTorchLightning";
    repo = pname;
    rev = version;
    sha256 = "12zhq4pnfcwbgcx7cs99c751gp3w0ysaf5ykv2lv8f4i360w3r5a";
  };
...

So, to get the desired behaviour I’d need to set a new version and I’d need src.rev to get recomputed.

  1. As far as I understand this attribute set is evaluated eagerly and there’s no reliable way to re-evaluate it with new version
  2. Even that I know I need to reset version and src.rev, I can’t seem to understand how to get this derivation rebuilt from new sources: (ps."pytorch-lightning".overrideAttrs (o: o // { version="0.9.0"; name="pytorch-lightning-0.9.0"; src = o.src.overrideAttrs (o2: o2 // {rev = "0.9.0"; });})) triggers building a new derivation but it still downloads ./pytorch_lightning-0.8.5-py3-none-any.whl

Thanks

2

Related: There isn't a clear canonical way to refer to a specific package version. · Issue #93327 · NixOS/nixpkgs · GitHub

What I couldn’t understand is why isn’t it a convention in Nixpkgs to put version/revision/source in parameters list with a default: {..., version ? "0.8.5", ...}:, instead of hard-coding it in lets and recs. This would allow at least a chance of reusability.

As far as I understand, all those packages are going to be rewritten as flakes, so at least Nixpkgs is going to take the form of inputs -> outputs with substitutable inputs. Would I be able to patch inputs for specific packages (like versions of python/npm/haskell packages)?

A typical python-packaging derivation just calls fetchPypi and while Nixpkgs doesn’t have to guarantee different values of version to work, it would still be nice to have an option to re-use the existing derivation without much boilerplate - at one’s own risk.

I also had a moment when I wanted to try and build colmap (the existing nixpkgs derivation) with custom gcc and there was no obvious way to do that. I remember trying to overlay the pkgs with gcc=customGcc but that, predictably, triggered rebuilding of entire closure. This is probably valid behaviour, but I guess the point of nix is that it makes composing things easier and I in particular wanted to compose the prebuilt libraries and custom gcc to rebuild colmap only. Even if there’s no compatibility guarantee.

@vcunat’s response in the issue feels valid, but the point that the author makes is very important for managing dev environments; my hope for nix is to eventually replace the entirety of pips, poetrys and condas in my workflow.

P.S. I’m aware of mach-nix and poetry2nix (and I’ve seen similar solutions for wrapping stack/cabal) and as far as I understand they aren’t trying to manage python with nix, but rather wrap and hide setuptools runs, which would resolve transitive dependencies and fetch the wheels from the network without nix’s knowledge. In particular, if there are two python packages that both depend on pytorch=1.6.0 (let’s assume hey are also locked to exact same revisions), I am not sure if, with poetry2nix or mach-nix, that huge pytorch binary distribution would be re-used between them. My expectation instead is that poetry2nix would produce two derivations, each with its own copy of pytorch. I’m still yet to check it. Felt like mentioning this because it’s not stated in READMEs or elsewhere and might not even be currently considered as a possible usecase

Thanks