Pre-RFC: Implement dependency retrieval primitive

[RaitoBezarius]

(Just to be clear as you explain all of this, I work on Tvix and I implemented context strings in Tvix for example.)

@physics-enthusiast

True, and we currently have no way to get the results of that scan (rather than the whole closure) without reimplementing the entire path scanning infrastructure ourselves. Admittedly the only usecase I can think of at the moment is a marginal disk space reduction if you are sure your derivation depends only on inert files and not other executables, but I included it for conceptual symmetry with the other two.

Alternatively, you can call this function: nix/src/libstore/path-references.cc at 74e4bc9b1dedc6a3228b2cde0175f7e1c89bc14d · NixOS/nix · GitHub without reimplementing the logic.

You can also use Tvix code for that: https://cs.tvl.fyi/depot/-/blob/tvix/glue/src/refscan.rs if you prefer Rust.

@physics-enthusiast

Almost, but not quite. inputSrcs are instantiation-time dependencies. inputDrvs rely on string context for dependency resolution, and it is possible to intentionally or unintentionally lose it. I could be wrong on this, but since .drv creation and string interpolation is done at eval-time, if that “invisible” inputDrv itself depends on some kind of eval-time fetcher (which now cannot reach the internet), you can get an evaluation failure. More importantly, there is currently no native means of actually obtaining inputDrvs and inputSrcs information. Your only options are external tooling or fragile hacks that rely on an implementation detail (the .drv format).

If this is intentional, I’m not sure what is the goal of retrieving them again. If this is unintentional, I cannot understand it by your link, can you elaborate where is the unintentional component of using an unsafeDiscard somewhere?

If you want to undo the unsafeDiscard, it is very easy to fix though and is an arbitrary limitation you seem to be putting on yourself I think, you just need to make unsafeDiscardReferences be the identity function. Then the context strings will always be preserved at any cost.

---

On the eval-time fetcher situation, yes, IFD will cause this. Not sure if this is a bug?

Still, what you described about synthesizing .ifd.nix files to put them on the side and re-evaluate them if necessary seems to be quite the tall order.

For what value? Nixpkgs is forbidden to use IFD anyway except for replaceRuntimeDependencies.

@physics-enthusiast

IFDs introduce arbitrary Nix expressions, not just strings that can hold context. For instance, I can import a .nix file produced by a derivation containing a description of another derivation, which I then go on to use. Not being able to build that .nix file is an evaluation failure, but it leaves no trace on the string context. More egregiously, I can straight up write a .nix file containing a single extremely long string literal that is actually just a binary (to get around readFile preserving context). Even if the imported derivation did absolutely nothing at all, I would still need it to instantiate successfully! (hence the term “instantiation-time dependency”)

Sure, but are you not trying to bring hermeticism in terms of primops and primitives inside the evaluator? This is a Pandora box that requires a lot of background work. Given the state of the Nix interpreter, I am not so sure if it’s a realistic target.

Plus… I may be wrong, but the set of usecases for IFDs are quite small, given the fact that Nixpkgs forbids IFDs and that you probably want to dependency track things inside nixpkgs.

With things like dynamic derivations being cooked, it also creates interesting questions about whether it’s even worth to go for IFD tracking.

In addition, you do not really talk about it, but recursive Nix seems completely ignored in your ideas. Not sure if this is also a good idea to track dependencies across multiple Nix invocations, but… if IFDs are in scope, why not recursive Nix itself ?

So, in the end, I am not so sure about what does it bring on the table to track instantiation-time dependencies. I would also be interested to compare this to an idealized model of things like Flakes which aims to lock down the hermeticism of your ambient set of expressions.

---

“There’s no native means of actually obtaining […]” and…

@physics-enthusiast

If I’m understanding what it does, not really? It seems to evaluate an expression in a multithreaded manner and return each individual derivation in an attrset as soon as they are available (as opposed to when the whole thing finishes) but it doesn’t appear to do any dependency tracking at all. It prints the derivation attributes (inputDrvs, etc) but that’s information we could already get.

I fail to understand. Why is a native mean better than external tooling that reuses (yes, a somewhat unstable) Nix API? Can you elaborate on what do you gain by having it “natively” and what does native means in the context?

@physics-enthusiast

As I have demonstrated, it is currently not possible even with external tooling, since we lack certain information available only to the Nix evaluator. Even if it were, I think the usecase is significant enough to justify a Nix Language solution, rather than, say, one involving processing nix eval output with jq. If you search “nixos offline rebuild” and “nixos get all sources” you will see that this is at least a somewhat commonly requested feature. It was either add new behaviour or modify existing ones (breaking compatibility), and I felt that this was the most elegant solution.

Well, I, for sure, deployed offline nixpkgs cache, so I must be doing impossible things.
But more seriously, I am not totally convinced by this conclusion. What I understand that you demonstrated is that:

1. You would like to see through IFDs, that’s not currently feasible indeed
2. You would like a “native way” to do things, which seems either in the language or the default Nix implementation CLI somehow
3. I still do not see a compelling usecase, e.g. you can build offline cache just fine today. Yes, everyone ask how to do it again and again because no one really took the time to make it a “one click button” operation, to be honest, except for marsnix which is quite undocumented though.

nix eval + jq seems totally fine to me, and I do not see why this is a problem. Actually, I would rather see (1) to be done by exposing more of the metadata and seeing how hard it is to propagate this information which is enclosed indirectly in the interpreter without doing (2) which is orthogonal to the whole thing.

It would inform us more precisely about the consequences of that pre-RFC because while you say that this is the most elegant solution, what I do not see are the consequences on the interpreter itself, e.g. would that prevent some features to pop up, etc.

Nix expressions are supposed to be possible to evaluate for very long timescales, introducing primops have real consequences because if they are the wrong thing to do, it is very hard to get rid of them or modify their observable behavior. Hence, the hurdle to create the faith in new primops, especially complicated ones that needs to integrate with more information tracking than is currently exposed, is quite high.

Action items that I would suggest for you: implement your primops as Nix (dynamic) plugins and share them so we can take a look and see how it goes, I guess? I believe that some of them requires quite the machinery inside of Nix.