Pre-RFC: Implement dependency retrieval primitive

[physics-enthusiast]

@RaitoBezarius

If this is unintentional, I cannot understand it by your link, can you elaborate where is the unintentional component of using an unsafeDiscard somewhere?

Not sure if this is a bug that has been fixed since then, but the link seems to indicate that the toFile builtin discards string context (I’m assuming this is because it converts said string into a store path with no corresponding .drv).

@RaitoBezarius

If you want to undo the unsafeDiscard, it is very easy to fix though and is an arbitrary limitation you seem to be putting on yourself I think, you just need to make unsafeDiscardReferences be the identity function. Then the context strings will always be preserved at any cost.

If I’m understanding you correctly, this would require recompiling a custom Nix?

@RaitoBezarius

On the eval-time fetcher situation, yes, IFD will cause this. Not sure if this is a bug?

I don’t think it’s a bug, but it is a blocker for what I’m trying to do.

@RaitoBezarius

Still, what you described about synthesizing .ifd.nix files to put them on the side and re-evaluate them if necessary seems to be quite the tall order.

@RaitoBezarius

Sure, but are you not trying to bring hermeticism in terms of primops and primitives inside the evaluator? This is a Pandora box that requires a lot of background work. Given the state of the Nix interpreter, I am not so sure if it’s a realistic target.

Maybe I missed something, but this seems quite achievable by adding a little bit of context data to generic Nix objects. Just add an extra data field to these types and propagate the union of that field whenever a primop is called. Even if you have to address each one independently, there’s only a handful of fundamental Nix object types so it should be quite doable.

@RaitoBezarius

With things like dynamic derivations being cooked, it also creates interesting questions about whether it’s even worth to go for IFD tracking.

Implementing tracking for dynamic derivations is quite easy. All you need is somewhere to store (transiently in this case) the new type of reference (the derivation that produced it, which we already know). See the first point of the tentative getInstantiationDeps rules.

@RaitoBezarius

In addition, you do not really talk about it, but recursive Nix seems completely ignored in your ideas. Not sure if this is also a good idea to track dependencies across multiple Nix invocations, but… if IFDs are in scope, why not recursive Nix itself ?

I was going to say that nix show derivation, nix-store --dump-db and analogues don’t work in recursive-nix, but I went back and tested it and to my surprise, they do! I had considered it, but turns out the discussions I was reading were outdated and did not reflect the actual implementation, which admittedly caused me to assume it was much more limited than it was. To be fair, I haven’t exhaustively tested more complex dependency chains yet, and since there’s no way for inner nix to pass references to those store paths back to outer nix in pure eval (not even with IFDs) I’d have to implement the recursion in bash, but this is more or less the getRunDeps and getBuildDeps functionality I was looking for. My only concern is that we can only rely on it continuing to work on the condition that:

• recursive-nix ends up getting stablized (very likely)
• nix show derivation continues to work within inner nix (fairly likely)
• --dump-db has a nix-command equivalent by the time we move off of the old CLI (likely)
• inputDrvs and inputSrcs remain accessible via the .drv format or analogues (somewhat likely)
• current or future experimental features do not lead to a restriction on what portions of the input closure inner nix is allowed to access (uncertain)

@RaitoBezarius

Plus… I may be wrong, but the set of usecases for IFDs are quite small, given the fact that Nixpkgs forbids IFDs and that you probably want to dependency track things inside nixpkgs.

@RaitoBezarius

I would also be interested to compare this to an idealized model of things like Flakes which aims to lock down the hermeticism of your ambient set of expressions.

Now that you mention it, Flakes do effectively act like IFDs. They inject arbitrary Nix expressions (when used as the input of another Flake) while themselves being invisible external dependencies that cannot be discovered by their dependents. As they are now, Flakes seem to cause all kinds of problems for offline (re)builds in general, and cause even rebuilds on the same machine to require workarounds if done without Internet access. While I concede that IFDs aren’t exactly common, Flakes are and we can use getInstantiationDeps to address them too (and more or less deal with IFDs for “free”).

@RaitoBezarius

I fail to understand. Why is a native mean better than external tooling that reuses (yes, a somewhat unstable) Nix API? Can you elaborate on what do you gain by having it “natively” and what does native means in the context?

@RaitoBezarius

Well, I, for sure, deployed offline nixpkgs cache, so I must be doing impossible things.

@RaitoBezarius

You would like a “native way” to do things, which seems either in the language or the default Nix implementation CLI somehow

Maybe it’ll be helpful if I outline my specific usecase to help clarify what I mean by terms like “native” and “maximal percentage of packages built from source”. I want:

• A collection of files
• (Ideally) generated at the same time as a nix build or nixos-rebuild of a chosen target (i.e. the information can be contained within a Nix expression)
• Containing only store paths/derivation outputs that require internet access to build
• Which can be portably transmitted to a machine with an effectively empty store
• Such that it is able to build said target package
• Without internet access

Right now the options for offline backups are:

• nix copying the run-time closure
  • Compact (file/folder-size-wise)
  • But package configuration is fixed and it cannot be rebuilt
• nix copying the build-time closure or outright cloning the entire store
  • Rebuilds are possible (unless you use Flakes or copy the whole store and use the workaround mentioned)
  • But every transitive dependency under the sun must be copied and saved into the binary cache. At that point if you’re on NixOS and the target package is system.build.toplevel you may as well just move /home elsewhere and image the partition.

The ability to back up only downloaded sources combined with Nix’s reproducibility give you the best of both worlds. By building the intermediate derivations only when you need them (and then optionally GC-ing them afterwards), you get the rebuild flexibility of a build-time closure at the persistent disk space cost of a run-time closure.

@RaitoBezarius

Nix expressions are supposed to be possible to evaluate for very long timescales, introducing primops have real consequences because if they are the wrong thing to do, it is very hard to get rid of them or modify their observable behavior. Hence, the hurdle to create the faith in new primops, especially complicated ones that needs to integrate with more information tracking than is currently exposed, is quite high.

I guess, in a sense, this (pre-)RFC is as much a request for assurance that a certain capability (hermetic dependency retrieval) will always be supported as it is a request for the implementation of a specific means of achieving it. I still believe that a primop that directly returns references to all immediate dependencies of type <x> (that can be used by other Nix expressions) is still the most ergonomic, composeable and flexible solution to the problem, but since recursive-nix gets us most of the way there for build-time (albeit with suboptimal ease-of-use and uncertain stability like I outlined above) and Tvix’s path-scanning for run-time, I think I can settle for just getInstantiationDeps (or some alternative that makes possible the same ends) if my justification for the other two is considered insufficient.
Out of curiosity, would an optional derivation advanced attribute be less “risky” to introduce than a primop, in your opinion?

@RaitoBezarius

Action items that I would suggest for you: implement your primops as Nix (dynamic) plugins and share them so we can take a look and see how it goes, I guess? I believe that some of them requires quite the machinery inside of Nix.

I’ll give it a try. I suspect getInstantiationDeps might require recompiling a full fork though, since I need to modify the internal behavior of existing builtins like import to yield additional information rather than merely adding new ones (assuming I correctly understood what plugins are able to do. Can plugins hook function calls?).