I noticed that in one (pretty short) CI job of mine significant time is spent fetching derivations ending in ...-source. Now I assume those are all dependencies of my flake. My question is however, whether those are only fetch when needed in the specific package I am trying to build or if Nix always fetches all flake dependencies? Is this fetching process parallelized?
2 Likes
They could be flake inputs or FODs without an explicit name, in which case source is oftentimes assigned by default.
Yes, if they are flake inputs.
No, if they’re actually packages, only FODs in the closure would be fetched.
I don’t believe so, if those are flake inputs or some builtin fetchers, as eval is single-threaded.
Actual packages can be fetched in parallel.
Ok, to clarify: Yes I am talking about flake inputs. So this means if, for example one package in my flake depends on crane to build a rust crate, the crane flake source will get pulled even if I evaluate a completely unrelated package in the flake, that does not make use of crane in any way?
EDIT: So I checked and there does seem like not all inputs are downloaded all the time: this is a package using crane and this is one does not require crane (you can compare the “Checking binary cache” step). I still wonder what the exact specified/expected behavior is though…
It doesn’t matter what your packages use or not. If your flake has the crane flake as a flake input, the crane flake gets downloaded. Lazy trees by edolstra · Pull Request #6530 · NixOS/nix · GitHub is intended to fix this, but it’s far from done.
1 Like
How do you explain the second CI build then. It is a C build and crane does not seem to be fetched, although it is a flake input…
What command are you running?
Using a flake reference instead of the local flake the equivalent of the command run would be nix eval --raw github:jzbor/nix-overlay#buttermilk
*when locking.
Once the flake is locked, other users and CI only need to fetch a flake input when it is referenced by an expression that is actually evaluated. In this case Nix does something like mapAttrs builtins.fetchTree on the lock file nodes, and since it’s a lazy language, and it doesn’t (afaik) make the mistake of forcing those attribute values ahead of time, this works out nicely.
For instance, if I use an input only for devShells but not for packages, users of that locked flake’s packages attribute won’t have to fetch that input.
The degree to which inputs are actually optional depends on the nature of the flake, and occasionally how it’s written.
Lazy paths by roberth · Pull Request #11367 · NixOS/nix · GitHub is a promising approach that delivers part of what lazy-trees brings to the table, but this is mostly about removing the copying to the store part, and not so much about the actual fetching.
The plan of Reuse input lock files · Issue #7730 · NixOS/nix · GitHub is to actually make locking lazy for transitive inputs, so that might have a more noticeable effect. (Unless you have slow-ish I/O to the store)
The Nix team has had to make time for other things, so progress on these issues hasn’t been as steady as I’d hoped. Help is of course appreciated if you want to contribute to these developments.
3 Likes
Note that since last week, Nix no longer tries to substitute flake sources. So starting in Nix 2.25, you’ll see a lot fewer of those “fetching …-source” messages.
6 Likes
Thanks for the explanation and the references. Your answers do help me alot with some current considerations regarding flakes.