tl;dr flake dependency graphs get turned into trees unless you use .follows and I can’t figure out why, I also don’t know why flake.lock contains transitive dependencies
Current behavior
Diagram meaning
Since this question is only about dependencies between flakes, the diagram represents a flat directory structure of flakes with no outputs, like this:
Represents the following repository:
/a/flake.nix
/b/flake.nix
/c/flake.nix
/a/flake.nix:
{
inputs = {
b.url = "path:../b";
c.url = "path:../c";
};
outputs = { ... }: { };
}
/b/flake.nix, /c/flake.nix:
{
inputs = { };
outputs = { ... }: { };
}
Currently, flake evaluation constructs its dependency graph by recursively traversing inputs from the current flake and creating a tree of dependencies (unless overridden by .follows). When the same flake is encountered multiple times while building the graph, each encounter results in a separate instance of that flake.
Here’s a dependency graph that illustrates this:
A user’s natural expectation would be that the final dependency graph would only contain one instance of d and e. Instead, flake.lock from the perspective of a looks like this:
And if we were to add f and g following the same pattern, each of those would have four instances, and so on. It gets extra fun with cycles, as it’ll work if you only use one .follows, but one of the lockfiles will also contain the entire contents of the other. This isn’t a problem in many real use cases, and the assumption that the dependency graph resembles a tree mostly holds. A notable exception is that almost everything depends on nixpkgs, which is addressed by .follows.
However, this is not the case in a monorepo, where all your flakes use the same version of your internal dependencies. Dependency graphs can get quite tangled and there’s a real possibility that the exact same version of the exact same flake might be instantiated several times. Flake evaluation is (supposed to be) pure, so this is not a correctness problem, but a performance one.
A bigger problem is that the entire dependency graph gets written to flake.lock, including transitive dependencies. Conceptually, the job of the lockfile is to pin exact versions of more broadly specified dependencies. Including transitive dependencies does not fit into that and causes additional problems. It’s not particularly bothersome for external dependencies, where transitive dependencies can simply be thought of as part of that external dependency, but when depending on path: flakes within the same repo it goes from redundant to harmful. That part of the lockfile is automatically generated and should not be committed to source control, but it’s contained within the same file that contains pinned external dependencies. If you update a flake’s inputs, you also have to remember to manually update all the flakes that depend on it.
Workarounds
Don’t have a rat’s nest of internal dependencies between flakes in a monorepo. Instead, just have a single flake containing all your external dependencies. No more nix build .#executable for you, now it’s nix build ../deps#whatwouldvebeenaflake-executable.
An alternative would be to have that centralized /deps/flake.nix, but also have per-project flakes that contain only one input that refers to path:../deps. All flake.locks except /deps/flake.lock would be gitignored, but then you’d still have the problem of nix not automatically picking up changes to dependencies of path: dependencies without manually running nix flake update. At least rm is shorter to type, and it wouldn’t spam your git logs with generated JSON.
But why?
Implementation detail? Because other languages do it this way? Maybe, but I think I’m missing something. I can’t see a good reason why transitive dependencies couldn’t just be picked up from /nix/store, utilizing the same lazy fetching as the inputs themselves. And if we’re hypothetically changing the lockfile to contain only external pins, then we might as well get rid of the tree-ification by deduplicating the flakes before evaluation, since it doesn’t impact correctness.
So my question is, why not do it this way? I highly doubt this is something that was overlooked during the design process, so what feature of flakes necessitates the current design?