I was wondering why lib.mkShell { inputsFrom = [ package ]; } needs to evaluate package.src? When package uses a large source, this results in all of this source being copied into the Nix store. This seems unnecessary. I’m using package.overrideAttrs (old: src = null) to avoid this, but it feels like a hack.
A full example that demonstrates this is here: nix-large-repo/default.nix at d902cfda32c095bc67eb583cc51b43471034d7cf · blitz/nix-large-repo · GitHub
That is because, when extracting which attributes of package derivations to merge, only the inputsFrom attribute is excluded (src isn’t excluded):
Mmh. As I understand it, the problem is that lib.subtractLists actually needs to compare the derivations and in the process of doing so evaluates them.
this leaves actual dependencies of the derivations in
inputsFrom, but never the derivations themselves
I wonder why this would be relevant? The PR that introduced this behavior is: mkShell: exclude inputsFrom from merged inputs by greedy · Pull Request #137005 · NixOS/nixpkgs · GitHub
And indeed if I remove the lib.subtractLists step above, the evaluation of the normal shell environment is as fast as the crude hack of removing src from it.
So is it sensible to revert to the behavior before the above PR? The special usecase that prompted the above PR could be an opt-in feature.
The commit message to the PR you pointed out explains it:
This commit filters out any dependencies from one package in inputsFrom to another when computing the shell environment’s inputs. This supports the use case where several closely related packages (perhaps even built from the same source tree) are being mutually developed.
I made a mistake initially. I’m still not 100% sure what is the actual cause of slowdown. But, I believe that subtractLists is the real reason behind the slowdown (not src’s size). The subtractLists function has the complexity of O(nm). So yes, while a bigger in size source will cause slowdown, that contributes to the slowdown because of subtractLists’s complexity.
The slowdown definitely happens due to src being copied into the Nix store. It’s slow enough to just see it going on with -vvv. And src is evaluated because subtractLists needs to compare elements (if I understand it correctly).
Regarding:
This commit filters out any dependencies from one package in inputsFrom to another when computing the shell environment’s inputs. This supports the use case where several closely related packages (perhaps even built from the same source tree) are being mutually developed.
This seems to be optimizing a specific usecase that is not common and i also wouldn’t expect mkShell to optimize this. If I have inputsFrom = [ a b ]; and b is an input of a, it’s weird if mkShell just removes b. Hiding this behind an opt-in option to streamline the default case still seems like a good option here.
I assume then the slowdown is only a one time hit. Once src has been copied to the Nix store, that variable shouldn’t matter anymore.
When src is ‘fetchFromGitHub’ yes, but if its ‘./.’ then the path will be copied every time.
I see. Nix will copy source to the store. Not much to do here. You can prevent copying when src is set to ./. only in a git repository with no changes (modified files). I’m pretty sure that this is the behaviour with flakes, I’m not sure about non-flakes (haven’t used much).
Because that’s the point of the inputsFrom attribute: you don’t want to build the actual derivation, since it may not even build (yet), but get the dependencies in order to work on fixing the build.
Maybe. The whole idea behind mkShell and most of the Nixpkgs tooling is to work with remote sources that are immutable from the point of view of the packager. Therefore indeed as @thefossguy notes, copying them to the store should happen exactly once. So in a sense inputsFrom = [ (callPackage ./package {}) ] is holding it wrong, but it’s of course a reasonable use case.
I’d presume it’s not all that special, but maybe it can be separated more cleanly. The question is how often subtractLists makes a difference, or whether it can be simply be removed without breaking a lot of stuff or people’s workflows.
It depends. For people that use Nix to get developer shells for their own projects, this pops up quite a lot. We have this in pretty much all our internal projects: You have your sources and the Nix expressions to build everything in one repo. And given that you already have the derivation to build your software why not do mkShell with inputsFrom on it. The alternative is to duplicate your dependencies.
Yes, I’m using it exclusively that way myself. I meant to say the tool may simply be targeting a different use case, and maybe it’s better to work around it instead of against it, because otherwise it will just break other people’s ways of doing things.