It calls source files via pkgs.callPackage and a globally scoped pkgs only for the reason that fetchers (the only attributes supposedly ever used in source files) do not have a separate handle / scope.
This is confusing since it suggests the necessity of this global scope when calling file and hides away the reality that only the hypothetical fetch* interface is used.
If I had to coin it, I would probably call it a deceptive scope mismatch ?
Has the community already found a coining for the necessity of this convoluted arrangement?
I’m looking for a somewhat generally accepted name to be able to mentally isolate it as a “historic glitch” when reasoning about folder hierarchies and scopes.
This is somewhat relevant when looking at nix code as trying to be self-documenting, because an inattentive (inexperienced) reader would tend to imply that nixpkgs version is important and could break things down within this sub folder tree. But the reality is, only changes to the quite stable hypothetical fetcher* interface could break things, the nixpkgs version can thus be largely arbitrary (within those constraints). Hence, there is no real dependence on pkgs.
Well, I think the confusing part is really the whole setup of the Nix configuration:
This package could simply import the sources directly, like src = callPackage ../../sources/arc-theme-red.nix {}. There is no need for the default.nix / call.nix stuff in the sources directory at all.
But it’s his personal config files, I don’t think it’s worthwhile to pick at them too closely. For comparison I just use git repos everywhere and patch / rebase them as necessary.
While this is true, had it been the case I wouldn’t have got the opportunity to reason and learn ablut the good / bad things of NixOS/nixpkgs.
Encapsulating the caller forces the usage of well defined interfaces. I think that’s a worthwhile pattern.
Already builtins.fetchTar & fetchGit don’t seem to have the same return type. Which is unfortunate, and already cost me a couple of hours of very crypric error messages.
Maybe we could also coin this problem the dilemma of global scope.
Thank you for all the suggestions to work around the question .
It is true, that callPackages although not stated in that article, intrduces an object oriented pattern with implicit attributes of self.
Hence, we can also coin the observation deceptive object mismatch. Fetchers are not factored into their appropriate object which promotes to import them directly (while having or not callPackage’s OO convenience attached).
But it might as well be there is no abstract generally accepted term in the community to get a mindful handle on this situation. Maybe nobody has seen it as a problem that merits attention.
I’m speaking in the learning category, so taking distinct notice of those kinds of (anti-)patterns and discussing them is a means of apprenticeship.
Addendum: once flakes have really landed, we could probably reason about a “fetcher flake”, that nicely wraps up my conclusion in this particularily peculiar case.
I think what most disturbs me about this all is that stable (real fethchers) and unstable (real packages) apis are put together in one single scope. Well both may be unstable, but conceptually with largely different cycle times.
I wonder though if using one nixpkgs for fetchers and another for packages can modify a specific target output derivation? My belly says no, but are there surprises?
While I agree, it doesn’t necessarily promote interfaces as effectively. That aspect should be tackled conceptually by competent parties with the goal of making inner workings more transparent to veterans and newcomers alike. In a monorepo, it requires greater discipline and willpower, though. Separation of concerns and repos would promote proper interfaces implicitly as a hard requirement.
I am not sure if I correctly understand your point. But if the question is why fetchers are not separated from the rest of nixpkgs: many fetchers in nixpkgs depend on other packages in nixpkgs. For instance, fetchZip uses the unzip derivation, fetchurl (not the built-in one) uses the curl derivation, fetchgit uses the git and cacert derivations.
So, the fetchers depend on the rest of nixpkgs and nixpkgs depends on the fetchers. Since these are coupled, it makes sense to bundle them together.
If you want to be explicit, you can just use inherit to use only specific attributes as @jonringer suggested.
I did understand that there are interdependencies and sorry if I might lack the nix primitives in my vocabulary to communicate better (in the sense of a conceptual glossary).
I also know how to work around what I see.
What I’m really looking is for a generally accepted glossary entry in the form of this
Scope Mismatch:
For historical reasons (or whatever reasons?), build-support / fetchers are in the global scope although their interdependency with pkgs is special: while they do depend on some.pkgs, their output does not change in function of those dependency. They therefore could be factored out into say pkgs-lib.
This attempt of mine is a) poor wording and b) poor concept reflecting my current state of understanding of things.
This statement stands in opposition to itself.
If A depends on B and B changes, A must be expected to change too. Otherwise A wouldn’t depend on B in the first place.
One of the core principles of nix is, to always assume that any change in any dependency will break reproducibility and therefore the dependant must be reevaluated.
If unzip changes, fetchZip is not the same fetchZip anymore. If it wasn’t like this, Nix wouldn’t be Nix.
Therefore they can not be factored out. They are 100% dependent on nixpkgs.
Factoring out can can only be accomplished by removing the dependency to nixpkgs.
And that has been done already. The factored out part is called builtins.fetchXXX
builtins is basically your pkgs-lib.
I think it doesn’t require any special vocabulary to talk about those phenomena/conditions. Trying to coin problems before having them discussed and identified as such, might cause confusion.
Thanks for this reasoning! It accurately explains part of my observation under the current premise of how nix works. And it furthers my understanding of this matter.
I can understand that this is true while outputs are not content addressable — which seems to be discussed in RFC62. If they where, — and pardon the lack of glossary on my side — then there are two categories of reproducibility concerns, those that depend on the input and those that don’t.
In such a reasoning frame, clearly pkgs/pkgs-lib and good part of pkgs/built-support would fall under the category where a variation in input to the constructor would not change a variation in output.
Coming from my point of view, this might be considered a shortcoming of the current state of nix and would strongly support RFC62, while I’m able to rewrite my hypothetical glossary entry from:
to (in my own poor wording):
Global Namespace & Poor Factoring:
Nix is not (yet) content addressable. This implies that certain categories of expressions — by definition — inter-depend, while measured by their output they actually don’t. Much part (if not all) of pkgs/pkgs-lib and pkgs/built-support fall into that category. For this reason, those categories are currently shipped with NixOS/nixpkgs and not factored out into their own set of libraries that present a consolidated set of predictable interfaces to the nixpkgs project. If RFC62 is accepted and materialized, this might eventually change.
@DavHau Thinking about this a little further. By the standards of the nix language, is a variation in input to the constructor of a function “equivalenced” to a variation in the input to that function?
When using a different unzip in our example, the constructed function is not the same. However the inputs and outputs to that different function exhibit no variatinos. In a content addressable paradigm, this question is still easy to resolve, but in pre-content-adressable world, is this abstraction legit and possible?
The fetchers produce fixed-output derivations by specifying outputHashAlgo and outputHashAlgo. So, they are already content-adressable. This is the reason why a change to unzip does not result in all packages using fetchzip being rebuilt.
If a dependency of a fetcher is changed in such a way that the the fetched changes, then either the fetcher needs to work around this change (e.g. by normalizing the output) , or the hash in every derivation that uses that fetcher needs to be updated.
Of course, it is possible to separate fetchers (as builtins shows). But there is a good reason that there are fetchers in nixpkgs, it means that you can make more complex fetchers that use other derivations in nixpkgs plus that you can write them in Nix (as opposed to C++). You could separate them out of nixpkgs as well, but that would mean that you’d to maintain various derivations (e.g. unzip, fossil, darcs, compilers, stdenv itself, etc.) in two places. The only downside that I see is that you need all of nixpkgs to use these fetchers. But is anyone really using Nix without nixpkgs?
That is really interesting. So it already is content addressable. It also means that I can safely use pinned version of nixpkgs on fetchers while using another pinned nixpkgs version for the evaluation of packages and configuration. This allows for cleaner patterns in a well-structured repo, and I would explicitly not have to pass nixpkgs all around the place, rather only to the (alas single) entry point of my build evaluation.
In the context of the initial example from Silvan’s nix config I could safely write:
{
fetchers = import <nixpkgs>; # or use a flake
}
Then the only thing that would really be nice would be to enable this form through some intra-nixpkgs monorepo factoring (without actually breaking it apart):
{
fetchers = import <nixpkgs/pkgs/pkgs-lib/fetchers>; # or use a flake
}
It is an abstract benefit, but it would encourage to write more concise derivations (exhibiting the intricacies you explained about fixed output derivations) and at least give expressive means to avoid putting everything into the same global namespace (which might be a real problem of it’s own).
Thank you for this proposal, it (and the abstract idea behind it) will help me to write an interface rich more readable code style.
As an aside, I also placed a ganeral rule upon myself that imports from ../ will be only ever allowed in files that start with __ it seems to be of good service so far.
.