After having major difficulties with large lock files#6626, and cluttered APIs I’ve come up with a scheme that allows one to declare private flake inputs in a project, i.e. inputs that do not propagate to the lockfile of any consuming users of your flake.
The work I did in this PR is really a conflation of two concepts, the deferred locking might also be useful for organizations using flakes, but I’d really just like to keep the scope of this thread on the private input segment.
A brief outline of the implementation:
We use the upstream call-flake.nix directly to evaluate all these inputs in Nix code, using this function. It is important this function is kept dependencyless in order to avoid infinite recursion.
Then we define a seperate subflake that is not wired in to the top-level flake using the normal flake input mechanism, instead it is wired in manually in Nix. The only major downside at this point is that this narHash must be updated manually whenever the private subflake is updated, otherwise updates don’t propagate to the top-level.
The reason I am posting this here is to point out the the current flake API may be insufficient for large mono-repo style projects, and adding a proper mechanism for what I’ve done here would be greatly desireable just to avoid the practical heavy cost linked in #6626 at the top of this post, while also allowing for a cleaner API, e.g: using this mechanism I am able to make a distinct API boundary where inputs at the top-level of our projects are expected to defer locking to the consumer (there is no lockfile for the top-level flake), while inputs in the private subflake are expected to be locked by the subflake itself.
I’m very interested in gaining some feedback on whether or not this approach would be useful to others in the community, and if so, build some momentum for making it a proper feature.
I do think a binary distinction between public and private might actually fall short over time.
Just as a monorepo has different horizontal & vertical output scopes (e.g. orizontal: installables, library functions, data, container, nixosModules; vertical: app1, app2), these scopes may only take a subset of inputs.
So depending on one’s output interests, these interests would strictly propagate to a subset of inputs, only.
Now what is a better idiom to express scopes? I don’t know. Last time I was reasoning about this I sort of came to the conclusion that it was impossible to reverse the function logic and infer inputs from outputs, without costly probing.
That would ultimately mean a manual metadata annotation, but which way to express this, I have no idea… Maybe subflakes will do that job, one day, but then also see @Growpotkin 's experience…
It would be really nice if we had builtins.cache key something as stepping stone to be able to at least conscientiously control the eval cache at the well-crafted and appropriate level of granularity.