Pre-RFC: Implement dependency retrieval primitive

[physics-enthusiast]

Summary

Expand the dependency tracking system with 3 new primops getRunDeps, getBuildDeps and getInstantiationDeps.

Motivation

Currently, the task of obtaining a complete list of the dependencies of a given derivation is (IMO) needlessly difficult. Existing tools to achieve this like exportReferencesGraph are both poorly documented and insufficiently granular, leading to persistent difficulties in usefully ‘transmitting’ the results of derivations to resource-constrained machines. In particular, there are several discourse threads (such as this one) and multiple github issues (the most recent of which being this one) detailing the (only partially resolved) problems with getting rebuilds to work without an Internet connection. Another example would be this open 7 year old issue which wants a way to get the run-time closure of only immediate build-time dependencies. Nix should already have this kind of information as it goes about computing closures and doing path rewriting, we just need to make it available to the language user.

Detailed design

Introduce 3 primops, which correspond to what I consider to be the 3 types of dependencies:

• getRunDeps - Immediate dependencies any executable in any output needs to run (your standard ${} runtime deps)
• getBuildDeps - Immediate dependencies required to build the derivation (emphasis on immediate; i.e. not the closure). If I have only the results of getBuildDeps and the .drv of the relevant derivation in the store, running nix build .drv^* should always succeed. This should not include the build dependencies of the rest of the run-time closure (since you can build a package without being able to run it). Additionally, only when retrieved via this function, the resultant derivation has all of its run-time dependencies recursively converted into build-time dependencies (since to be usable in a build, the derivation must have access to its run-time dependencies at build-time). This rewriting occurs in-memory in the Nix evaluator, so there is no impact on the storepath of the actual derivation.
• getInstantiationDeps - Immediate instantiation dependencies. Necessary to address IFDs, builtin fetchers and RFC 92. Tentative rules would be
  • dynamic derivers (A) are instantiation-time dependencies of their outputs (A.out)
  • builtin fetchers and storePath are instantiation-time dependencies of any derivation that references them
  • Import-From-Derivation (IFD) creates some sort of “nix fragment” at an intermediate store path (.ifd.nix maybe? TBC). This store path has the actual derivation as an instantiation-time dependency. Like getBuildDeps, only when retrieved via this function (getInstantiationDeps <expr>.ifd.nix), the derivation has all of its build-time and run-time dependencies recursively converted into instantiation-time dependencies. “Instantiation-time” in this case means the final instantation before nix build successfully exits, not the intermediate instantations performed in the event of recursive IFDs. This fragment would in turn be the instantiation-time dependency of any derivation that requires its contents to be reachable and well-defined. The purpose of this intermediate is to allow caching of the result of an IFD without needing the whole derivation it came from. This is useful if e.g. you only need a single byte from a 10GB download.

“Immediate” is important because you can get dependency closures from a function that returns immediate dependencies (via recursion or genericClosure) but not vice-versa.

Examples and Interactions

• Run-time closure: Equivalent to the result of applying the existing closureInfo to a store path.
• Build-time closure of the run-time closure: Equivalent to the result of applying the existing closureInfo to a .drv, assuming no use of unsafeDiscardReferences.
• Immediate build-time dependencies of run-time closure, minus the closure itself: Minimal set of store paths to rebuild a package.
• Immediate instantiation-time dependencies of build-time closure, minus the closure itself: Static build graph (i.e. after dynamic derivations have been resolved).
• Instantiation-time closure of the build-time closure of the run-time closure (henceforth referred to as the “total closure”): All packages built when you build the derivation in question while forbidding substitution.
• Only the FODs in the total closure: All external dependencies in a build, including those needed to instantiate it. When used as an offline binary cache for a rebuild, maximal percentage of packages built from source (everything except nonfree and bootstrap binaries). Build behaves the same way online and offline as long as all instantiation-time derivations are deterministic.
• Only the FODs and Nix fragments in the total closure: When used as an offline binary cache for a rebuild, maximal percentage of packages built from source that still provides unconditional equivalence of online and offline builds (i.e. they will succeed/fail at the same rate). This is only slightly less source-based than the FOD-only subset of the total closure because it’s theoretically possible to sneak a binary through a Nix fragment via readFile. When used as a cache, for most intents and purposes it’s source-based enough while being significantly faster to rebuild (by skipping IFDs).

The last two are what I’m ultimately after.

Notably, since these are Nix language primops (that return Nix objects) rather than derivation advanced attributes or CLI commands, we have access to a lot more dependency tracking information via the evaluator, and also conveniently sidestep all of the issues around having to encode a canonical set of dependencies into each store path. Under this proposal, the same derivation (in terms of input or content hash) instantiated differently can have different sets of instantiation-time dependencies. Finally, the results of these functions can be made lazy (since we do not need to track and save the dependencies of everything on the off chance they will be needed later), only the relevant dependency or dependency chains (in the case of closures) that we actually end up needing.

Drawbacks

• Since arbitrary Nix expressions can be introduced at instantiation-time, all current and future Nix objects will require some minimal context to track their Nix fragment of origin, increasing the ongoing maintenance burden.
• High potential for cross-interaction with current and future builtins. Resultant conflicts must be resolved to maintain hermeticity of the obtained closure.
• May limit the extent to which evaluation of a Nix expression can be parallelized or distributed, should that become a goal in the future.

Alternatives

Instead of separating them, add a single primop that returns a lazy attrset containing all 3. It may also be possible to avoid implementing the recursive run->build and run+build->instantiation dependency conversions mentioned above at the cost of end-user complexity in constructing closures (getInstantiationClosure is now recursive application of [union of getRunDeps, getBuildDeps and getInstantiationDeps] rather than recursive application of getInstantiationDeps alone), Nixpkgs complexity (getInstantiationClosure implemented in lib), primop complexity (the get*Deps functions return a lazy attrset containing “immediate” and “closure” dependency lists rather than a single list directly) or primop count (2 primops getDeps and getClosure that return the 3 dependency types each or 6 primops, one for each combination of dependency type and choice of “immediate” or “closure”).

Prior art

• exportReferencesGraph and its wrapper closureInfo. Lacks specificity. Either run-time closure or entire build-time closure only, leading to large space inefficiencies. Also fails to capture instantiation-time dependencies which can cause evaluation failures if they rely on being able to download things from the internet.
• Internal Nixpkgs maintainer script find-tarball.nix and its wrapper nixpkgs-mirror-tarballs. Only finds explicitly declared dependencies. Also seems to miss instantiation-time dependencies. Additionally, does not appear to account for IFDs or dynamic derivations.
• marsnix. Couldn’t find enough documentation to figure out how it’s supposed to be used, but from what little I could discern, likely shares the same problems as exportReferencesGraph.

Unresolved questions

The name(s) and number of potential primops are all undecided at the moment.

Do feel free to share any feedback you may have so that I can decide whether to take the time to make this into a proper RFC.

[RaitoBezarius]

I wonder if you tried to use nix-eval-jobs and looked into it for the needs you described.

@physics-enthusiast

• getRunDeps - Immediate dependencies any executable in any output needs to run (your standard ${} runtime deps)

My understanding is that those are the result of reference scanning the .drv modulo certain fields.

@physics-enthusiast

• getBuildDeps - Immediate dependencies required to build the derivation (emphasis on immediate; i.e. not the closure). If I have only the results of getBuildDeps and the .drv of the relevant derivation in the store, running nix build .drv^* should always succeed. This should not include the build dependencies of the rest of the run-time closure (since you can build a package without being able to run it). Additionally, only when retrieved via this function, the resultant derivation has all of its run-time dependencies recursively converted into build-time dependencies (since to be usable in a build, the derivation must have access to its run-time dependencies at build-time). This rewriting occurs in-memory in the Nix evaluator, so there is no impact on the storepath of the actual derivation.

Those are inputDrvs / inputSrcs in the .drv, right?

@physics-enthusiast

• getInstantiationDeps - Immediate instantiation dependencies. Necessary to address IFDs, builtin fetchers and RFC 92. Tentative rules would be
  • dynamic derivers (A) are instantiation-time dependencies of their outputs (A.out)
  • builtin fetchers and storePath are instantiation-time dependencies of any derivation that references them
  • Import-From-Derivation (IFD) creates some sort of “nix fragment” at an intermediate store path (.ifd.nix maybe? TBC). This store path has the actual derivation as an instantiation-time dependency. Like getBuildDeps, only when retrieved via this function (getInstantiationDeps <expr>.ifd.nix), the derivation has all of its build-time and run-time dependencies recursively converted into instantiation-time dependencies. “Instantiation-time” in this case means the final instantation before nix build successfully exits, not the intermediate instantations performed in the event of recursive IFDs. This fragment would in turn be the instantiation-time dependency of any derivation that requires its contents to be reachable and well-defined. The purpose of this intermediate is to allow caching of the result of an IFD without needing the whole derivation it came from. This is useful if e.g. you only need a single byte from a 10GB download.

Not very familiar with RFC92 actual use in production yet, onto the builtins fetchers or IFDs:

nix-repl> :p builtins.getContext "${(pkgs.fetchurl { url = "https://google.com"; sha256 = lib.fakeHash; })}"
{
  "/nix/store/471r12wkczk1p9vy5gai1fj6mf6pmjrg-google.com.drv" = {
    outputs = [ "out" ];
  };
}
nix-repl> :p builtins.getContext "${(builtins.fetchurl { url = "https://google.com"; sha256 = "0hpcpqs10lm7hb5s2z2g30zil26w89k6jvd396p080dkv7f977kn"; })}"
{
  "/nix/store/4ccwh3bx4n3yq7q6cdzwpfy3mnd262c3-google.com" = { path = true; };
}

It seems to me that the .drv will have a dependency on the builtin fetchers or IFDs because of the context strings, so I’m not exactly sure I understand the whole paragraph on the instantiation-time dependencies.

---

Overall, I understand where you are getting at with this proposal but I fail to see the inherent need for new primops vs. nix-evaling and just reconstructing what you need from there.

From there, I have one single question which is not addressed by the proposal: Under the assumptions, you can realize your goals wrt to offline caching / FODs, is there any need to be able to do this in Nix itself?

[physics-enthusiast]

@RaitoBezarius

I wonder if you tried to use nix-eval-jobs and looked into it for the needs you described.

If I’m understanding what it does, not really? It seems to evaluate an expression in a multithreaded manner and return each individual derivation in an attrset as soon as they are available (as opposed to when the whole thing finishes) but it doesn’t appear to do any dependency tracking at all. It prints the derivation attributes (inputDrvs, etc) but that’s information we could already get.

@RaitoBezarius

My understanding is that those are the result of reference scanning the .drv modulo certain fields.

True, and we currently have no way to get the results of that scan (rather than the whole closure) without reimplementing the entire path scanning infrastructure ourselves. Admittedly the only usecase I can think of at the moment is a marginal disk space reduction if you are sure your derivation depends only on inert files and not other executables, but I included it for conceptual symmetry with the other two.

@RaitoBezarius

Those are inputDrvs / inputSrcs in the .drv, right?

Almost, but not quite. inputSrcs are instantiation-time dependencies. inputDrvs rely on string context for dependency resolution, and it is possible to intentionally or unintentionally lose it. I could be wrong on this, but since .drv creation and string interpolation is done at eval-time, if that “invisible” inputDrv itself depends on some kind of eval-time fetcher (which now cannot reach the internet), you can get an evaluation failure. More importantly, there is currently no native means of actually obtaining inputDrvs and inputSrcs information. Your only options are external tooling or fragile hacks that rely on an implementation detail (the .drv format).

@RaitoBezarius

It seems to me that the .drv will have a dependency on the builtin fetchers or IFDs because of the context strings, so I’m not exactly sure I understand the whole paragraph on the instantiation-time dependencies.

IFDs introduce arbitrary Nix expressions, not just strings that can hold context. For instance, I can import a .nix file produced by a derivation containing a description of another derivation, which I then go on to use. Not being able to build that .nix file is an evaluation failure, but it leaves no trace on the string context. More egregiously, I can straight up write a .nix file containing a single extremely long string literal that is actually just a binary (to get around readFile preserving context). Even if the imported derivation did absolutely nothing at all, I would still need it to instantiate successfully! (hence the term “instantiation-time dependency”)

@RaitoBezarius

From there, I have one single question which is not addressed by the proposal: Under the assumptions, you can realize your goals wrt to offline caching / FODs, is there any need to be able to do this in Nix itself?

As I have demonstrated, it is currently not possible even with external tooling, since we lack certain information available only to the Nix evaluator. Even if it were, I think the usecase is significant enough to justify a Nix Language solution, rather than, say, one involving processing nix eval output with jq. If you search “nixos offline rebuild” and “nixos get all sources” you will see that this is at least a somewhat commonly requested feature. It was either add new behaviour or modify existing ones (breaking compatibility), and I felt that this was the most elegant solution.

[RaitoBezarius]

(Just to be clear as you explain all of this, I work on Tvix and I implemented context strings in Tvix for example.)

@physics-enthusiast

True, and we currently have no way to get the results of that scan (rather than the whole closure) without reimplementing the entire path scanning infrastructure ourselves. Admittedly the only usecase I can think of at the moment is a marginal disk space reduction if you are sure your derivation depends only on inert files and not other executables, but I included it for conceptual symmetry with the other two.

Alternatively, you can call this function: nix/src/libstore/path-references.cc at 74e4bc9b1dedc6a3228b2cde0175f7e1c89bc14d · NixOS/nix · GitHub without reimplementing the logic.

You can also use Tvix code for that: refscan.rs - depot - Sourcegraph if you prefer Rust.

@physics-enthusiast

Almost, but not quite. inputSrcs are instantiation-time dependencies. inputDrvs rely on string context for dependency resolution, and it is possible to intentionally or unintentionally lose it. I could be wrong on this, but since .drv creation and string interpolation is done at eval-time, if that “invisible” inputDrv itself depends on some kind of eval-time fetcher (which now cannot reach the internet), you can get an evaluation failure. More importantly, there is currently no native means of actually obtaining inputDrvs and inputSrcs information. Your only options are external tooling or fragile hacks that rely on an implementation detail (the .drv format).

If this is intentional, I’m not sure what is the goal of retrieving them again. If this is unintentional, I cannot understand it by your link, can you elaborate where is the unintentional component of using an unsafeDiscard somewhere?

If you want to undo the unsafeDiscard, it is very easy to fix though and is an arbitrary limitation you seem to be putting on yourself I think, you just need to make unsafeDiscardReferences be the identity function. Then the context strings will always be preserved at any cost.

---

On the eval-time fetcher situation, yes, IFD will cause this. Not sure if this is a bug?

Still, what you described about synthesizing .ifd.nix files to put them on the side and re-evaluate them if necessary seems to be quite the tall order.

For what value? Nixpkgs is forbidden to use IFD anyway except for replaceRuntimeDependencies.

@physics-enthusiast

IFDs introduce arbitrary Nix expressions, not just strings that can hold context. For instance, I can import a .nix file produced by a derivation containing a description of another derivation, which I then go on to use. Not being able to build that .nix file is an evaluation failure, but it leaves no trace on the string context. More egregiously, I can straight up write a .nix file containing a single extremely long string literal that is actually just a binary (to get around readFile preserving context). Even if the imported derivation did absolutely nothing at all, I would still need it to instantiate successfully! (hence the term “instantiation-time dependency”)

Sure, but are you not trying to bring hermeticism in terms of primops and primitives inside the evaluator? This is a Pandora box that requires a lot of background work. Given the state of the Nix interpreter, I am not so sure if it’s a realistic target.

Plus… I may be wrong, but the set of usecases for IFDs are quite small, given the fact that Nixpkgs forbids IFDs and that you probably want to dependency track things inside nixpkgs.

With things like dynamic derivations being cooked, it also creates interesting questions about whether it’s even worth to go for IFD tracking.

In addition, you do not really talk about it, but recursive Nix seems completely ignored in your ideas. Not sure if this is also a good idea to track dependencies across multiple Nix invocations, but… if IFDs are in scope, why not recursive Nix itself ?

So, in the end, I am not so sure about what does it bring on the table to track instantiation-time dependencies. I would also be interested to compare this to an idealized model of things like Flakes which aims to lock down the hermeticism of your ambient set of expressions.

---

“There’s no native means of actually obtaining […]” and…

@physics-enthusiast

If I’m understanding what it does, not really? It seems to evaluate an expression in a multithreaded manner and return each individual derivation in an attrset as soon as they are available (as opposed to when the whole thing finishes) but it doesn’t appear to do any dependency tracking at all. It prints the derivation attributes (inputDrvs, etc) but that’s information we could already get.

I fail to understand. Why is a native mean better than external tooling that reuses (yes, a somewhat unstable) Nix API? Can you elaborate on what do you gain by having it “natively” and what does native means in the context?

@physics-enthusiast

As I have demonstrated, it is currently not possible even with external tooling, since we lack certain information available only to the Nix evaluator. Even if it were, I think the usecase is significant enough to justify a Nix Language solution, rather than, say, one involving processing nix eval output with jq. If you search “nixos offline rebuild” and “nixos get all sources” you will see that this is at least a somewhat commonly requested feature. It was either add new behaviour or modify existing ones (breaking compatibility), and I felt that this was the most elegant solution.

Well, I, for sure, deployed offline nixpkgs cache, so I must be doing impossible things.
But more seriously, I am not totally convinced by this conclusion. What I understand that you demonstrated is that:

1. You would like to see through IFDs, that’s not currently feasible indeed
2. You would like a “native way” to do things, which seems either in the language or the default Nix implementation CLI somehow
3. I still do not see a compelling usecase, e.g. you can build offline cache just fine today. Yes, everyone ask how to do it again and again because no one really took the time to make it a “one click button” operation, to be honest, except for marsnix which is quite undocumented though.

nix eval + jq seems totally fine to me, and I do not see why this is a problem. Actually, I would rather see (1) to be done by exposing more of the metadata and seeing how hard it is to propagate this information which is enclosed indirectly in the interpreter without doing (2) which is orthogonal to the whole thing.

It would inform us more precisely about the consequences of that pre-RFC because while you say that this is the most elegant solution, what I do not see are the consequences on the interpreter itself, e.g. would that prevent some features to pop up, etc.

Nix expressions are supposed to be possible to evaluate for very long timescales, introducing primops have real consequences because if they are the wrong thing to do, it is very hard to get rid of them or modify their observable behavior. Hence, the hurdle to create the faith in new primops, especially complicated ones that needs to integrate with more information tracking than is currently exposed, is quite high.

Action items that I would suggest for you: implement your primops as Nix (dynamic) plugins and share them so we can take a look and see how it goes, I guess? I believe that some of them requires quite the machinery inside of Nix.

[physics-enthusiast]

@RaitoBezarius

If this is unintentional, I cannot understand it by your link, can you elaborate where is the unintentional component of using an unsafeDiscard somewhere?

Not sure if this is a bug that has been fixed since then, but the link seems to indicate that the toFile builtin discards string context (I’m assuming this is because it converts said string into a store path with no corresponding .drv).

@RaitoBezarius

If you want to undo the unsafeDiscard, it is very easy to fix though and is an arbitrary limitation you seem to be putting on yourself I think, you just need to make unsafeDiscardReferences be the identity function. Then the context strings will always be preserved at any cost.

If I’m understanding you correctly, this would require recompiling a custom Nix?

@RaitoBezarius

On the eval-time fetcher situation, yes, IFD will cause this. Not sure if this is a bug?

I don’t think it’s a bug, but it is a blocker for what I’m trying to do.

@RaitoBezarius

Still, what you described about synthesizing .ifd.nix files to put them on the side and re-evaluate them if necessary seems to be quite the tall order.

@RaitoBezarius

Sure, but are you not trying to bring hermeticism in terms of primops and primitives inside the evaluator? This is a Pandora box that requires a lot of background work. Given the state of the Nix interpreter, I am not so sure if it’s a realistic target.

Maybe I missed something, but this seems quite achievable by adding a little bit of context data to generic Nix objects. Just add an extra data field to these types and propagate the union of that field whenever a primop is called. Even if you have to address each one independently, there’s only a handful of fundamental Nix object types so it should be quite doable.

@RaitoBezarius

With things like dynamic derivations being cooked, it also creates interesting questions about whether it’s even worth to go for IFD tracking.

Implementing tracking for dynamic derivations is quite easy. All you need is somewhere to store (transiently in this case) the new type of reference (the derivation that produced it, which we already know). See the first point of the tentative getInstantiationDeps rules.

@RaitoBezarius

In addition, you do not really talk about it, but recursive Nix seems completely ignored in your ideas. Not sure if this is also a good idea to track dependencies across multiple Nix invocations, but… if IFDs are in scope, why not recursive Nix itself ?

I was going to say that nix show derivation, nix-store --dump-db and analogues don’t work in recursive-nix, but I went back and tested it and to my surprise, they do! I had considered it, but turns out the discussions I was reading were outdated and did not reflect the actual implementation, which admittedly caused me to assume it was much more limited than it was. To be fair, I haven’t exhaustively tested more complex dependency chains yet, and since there’s no way for inner nix to pass references to those store paths back to outer nix in pure eval (not even with IFDs) I’d have to implement the recursion in bash, but this is more or less the getRunDeps and getBuildDeps functionality I was looking for. My only concern is that we can only rely on it continuing to work on the condition that:

• recursive-nix ends up getting stablized (very likely)
• nix show derivation continues to work within inner nix (fairly likely)
• --dump-db has a nix-command equivalent by the time we move off of the old CLI (likely)
• inputDrvs and inputSrcs remain accessible via the .drv format or analogues (somewhat likely)
• current or future experimental features do not lead to a restriction on what portions of the input closure inner nix is allowed to access (uncertain)

@RaitoBezarius

Plus… I may be wrong, but the set of usecases for IFDs are quite small, given the fact that Nixpkgs forbids IFDs and that you probably want to dependency track things inside nixpkgs.

@RaitoBezarius

I would also be interested to compare this to an idealized model of things like Flakes which aims to lock down the hermeticism of your ambient set of expressions.

Now that you mention it, Flakes do effectively act like IFDs. They inject arbitrary Nix expressions (when used as the input of another Flake) while themselves being invisible external dependencies that cannot be discovered by their dependents. As they are now, Flakes seem to cause all kinds of problems for offline (re)builds in general, and cause even rebuilds on the same machine to require workarounds if done without Internet access. While I concede that IFDs aren’t exactly common, Flakes are and we can use getInstantiationDeps to address them too (and more or less deal with IFDs for “free”).

@RaitoBezarius

I fail to understand. Why is a native mean better than external tooling that reuses (yes, a somewhat unstable) Nix API? Can you elaborate on what do you gain by having it “natively” and what does native means in the context?

@RaitoBezarius

Well, I, for sure, deployed offline nixpkgs cache, so I must be doing impossible things.

@RaitoBezarius

You would like a “native way” to do things, which seems either in the language or the default Nix implementation CLI somehow

Maybe it’ll be helpful if I outline my specific usecase to help clarify what I mean by terms like “native” and “maximal percentage of packages built from source”. I want:

• A collection of files
• (Ideally) generated at the same time as a nix build or nixos-rebuild of a chosen target (i.e. the information can be contained within a Nix expression)
• Containing only store paths/derivation outputs that require internet access to build
• Which can be portably transmitted to a machine with an effectively empty store
• Such that it is able to build said target package
• Without internet access

Right now the options for offline backups are:

• nix copying the run-time closure
  • Compact (file/folder-size-wise)
  • But package configuration is fixed and it cannot be rebuilt
• nix copying the build-time closure or outright cloning the entire store
  • Rebuilds are possible (unless you use Flakes or copy the whole store and use the workaround mentioned)
  • But every transitive dependency under the sun must be copied and saved into the binary cache. At that point if you’re on NixOS and the target package is system.build.toplevel you may as well just move /home elsewhere and image the partition.

The ability to back up only downloaded sources combined with Nix’s reproducibility give you the best of both worlds. By building the intermediate derivations only when you need them (and then optionally GC-ing them afterwards), you get the rebuild flexibility of a build-time closure at the persistent disk space cost of a run-time closure.

@RaitoBezarius

Nix expressions are supposed to be possible to evaluate for very long timescales, introducing primops have real consequences because if they are the wrong thing to do, it is very hard to get rid of them or modify their observable behavior. Hence, the hurdle to create the faith in new primops, especially complicated ones that needs to integrate with more information tracking than is currently exposed, is quite high.

I guess, in a sense, this (pre-)RFC is as much a request for assurance that a certain capability (hermetic dependency retrieval) will always be supported as it is a request for the implementation of a specific means of achieving it. I still believe that a primop that directly returns references to all immediate dependencies of type <x> (that can be used by other Nix expressions) is still the most ergonomic, composeable and flexible solution to the problem, but since recursive-nix gets us most of the way there for build-time (albeit with suboptimal ease-of-use and uncertain stability like I outlined above) and Tvix’s path-scanning for run-time, I think I can settle for just getInstantiationDeps (or some alternative that makes possible the same ends) if my justification for the other two is considered insufficient.
Out of curiosity, would an optional derivation advanced attribute be less “risky” to introduce than a primop, in your opinion?

@RaitoBezarius

Action items that I would suggest for you: implement your primops as Nix (dynamic) plugins and share them so we can take a look and see how it goes, I guess? I believe that some of them requires quite the machinery inside of Nix.

I’ll give it a try. I suspect getInstantiationDeps might require recompiling a full fork though, since I need to modify the internal behavior of existing builtins like import to yield additional information rather than merely adding new ones (assuming I correctly understood what plugins are able to do. Can plugins hook function calls?).

[RaitoBezarius]

@physics-enthusiast

Not sure if this is a bug that has been fixed since then, but the link seems to indicate that the toFile builtin discards string context (I’m assuming this is because it converts said string into a store path with no corresponding .drv).

nix-repl> :p builtins.getContext "${(builtins.toFile "abc" "def")}"
{
  "/nix/store/043113974zgxifcy4j8s1sx8fa8h9d7f-abc" = { path = true; };
}

I do see a source context here.

nix-repl> :p builtins.getContext "${(builtins.toFile "abc" "${pkgs.systemd}")}"
error:
       … while calling the 'getContext' builtin
         at «string»:1:1:
            1| builtins.getContext "${(builtins.toFile "abc" "${pkgs.systemd}")}"
             | ^

       … while evaluating the argument passed to builtins.getContext

       … while calling the 'toFile' builtin
         at «string»:1:25:
            1| builtins.getContext "${(builtins.toFile "abc" "${pkgs.systemd}")}"
             |                         ^

       error: files created by builtins.toFile may not reference derivations, but abc references !out!mmxfq68bslpjqki0f7svg31cjv2pnr36-systemd-255.2.drv

Though, interestingly… !

nix-repl> :p builtins.getContext "${(builtins.toFile "abc" "${builtins.toFile "def" "ghi"}")}"
{
  "/nix/store/kc46v2cfdvilicx6g97qqg49a2q1hi8j-abc" = { path = true; };
}

or

nix-repl> :p builtins.getContext "${(builtins.toFile "abc" "${/tmp/nvim.raito}")}"             
{
  "/nix/store/wjsm4z6rk51q846h2xdrshkn8pss99wf-abc" = { path = true; };
}

Both of them look like bugs to me.

@physics-enthusiast

If I’m understanding you correctly, this would require recompiling a custom Nix?

Yep, I don’t find this that shocking, theoretically speaking, we should be able to swap implementations of primops more dynamically for such usecases, e.g. Tvix enable (you can disable IO, etc.).

@physics-enthusiast

I don’t think it’s a bug, but it is a blocker for what I’m trying to do.

Right, it is just that I’m trying to distinguish statusquo, intended things and what is the new thing that you want to achieve in this pre-RFC.

@physics-enthusiast

Maybe I missed something, but this seems quite achievable by adding a little bit of context data to generic Nix objects. Just add an extra data field to these types and propagate the union of that field whenever a primop is called. Even if you have to address each one independently, there’s only a handful of fundamental Nix object types so it should be quite doable.

So, to the Value structure, right? This is the most performance critical structure of the whole interpreter and the one you will have the most trouble touching IMHO without decreasing performance.

For me, I think it’s a bit hard to properly add the extra data in the current interpreter because of the work to do before doing anything new, hence the suggestions to work “around” it.

@physics-enthusiast

Implementing tracking for dynamic derivations is quite easy. All you need is somewhere to store (transiently in this case) the new type of reference (the derivation that produced it, which we already know). See the first point of the tentative getInstantiationDeps rules.

I know this is conceptually easy, I’m talking about the implementation in this case.

@physics-enthusiast

I was going to say that nix show derivation, nix-store --dump-db and analogues don’t work in recursive-nix, but I went back and tested it and to my surprise, they do! I had considered it, but turns out the discussions I was reading were outdated and did not reflect the actual implementation, which admittedly caused me to assume it was much more limited than it was. To be fair, I haven’t exhaustively tested more complex dependency chains yet, and since there’s no way for inner nix to pass references to those store paths back to outer nix in pure eval (not even with IFDs) I’d have to implement the recursion in bash, but this is more or less the getRunDeps and getBuildDeps functionality I was looking for. My only concern is that we can only rely on it continuing to work on the condition that:

It would be nice to collect a repository of examples.
(which could become test cases or something)

@physics-enthusiast

• recursive-nix ends up getting stablized (very likely)

I don’t think recursive-nix will be stabilized anytime soon.

@physics-enthusiast

• nix show derivation continues to work within inner nix (fairly likely)

Or that a replacement still work.

@physics-enthusiast

• inputDrvs and inputSrcs remain accessible via the .drv format or analogues (somewhat likely)

I think the contention here is .drv as an implementation detail, but I think you make a great case that a condition to hide the impl detail is to provide those knobs.

@physics-enthusiast

Now that you mention it, Flakes do effectively act like IFDs. They inject arbitrary Nix expressions (when used as the input of another Flake) while themselves being invisible external dependencies that cannot be discovered by their dependents. As they are now, Flakes seem to cause all kinds of problems for offline (re)builds in general, and cause even rebuilds on the same machine to require workarounds if done without Internet access. While I concede that IFDs aren’t exactly common, Flakes are and we can use getInstantiationDeps to address them too (and more or less deal with IFDs for “free”).

Flakes causing a bunch of problems is unrelated to Flakes as a concept. It’s a consequence of their suboptimal implementation, IMHO.

The whole offline thingie is also a consequence of excessive sharp edges too IMHO.

@physics-enthusiast

Maybe it’ll be helpful if I outline my specific usecase to help clarify what I mean by terms like “native” and “maximal percentage of packages built from source”. I want:

• A collection of files
• (Ideally) generated at the same time as a nix build or nixos-rebuild of a chosen target (i.e. the information can be contained within a Nix expression)
• Containing only store paths/derivation outputs that require internet access to build
• Which can be portably transmitted to a machine with an effectively empty store
• Such that it is able to build said target package
• Without internet access

So you basically said that you want to know all FODs to rebuild a certain thing, which is A function `rebuildDependencyClosure` that produces an efficient closure for offline rebuilds · Issue #180529 · NixOS/nixpkgs · GitHub for me.

To produce it at the same time during nix build or nixos-rebuild seems a UX concern, I would put that aside.

Maybe what we want is something like nix offlinize $store_path while being online, etc.

To transmit what is needed to build a closure to another machine seems like that, you want to have the outPath of the rebuild dependency closure and just apply classical nix copy for it.

---

@physics-enthusiast

Right now the options for offline backups are:

• nix copying the run-time closure
  • Compact (file/folder-size-wise)
  • But package configuration is fixed and it cannot be rebuilt
• nix copying the build-time closure or outright cloning the entire store
  • Rebuilds are possible (unless you use Flakes or copy the whole store and use the workaround mentioned)
  • But every transitive dependency under the sun must be copied and saved into the binary cache. At that point if you’re on NixOS and the target package is system.build.toplevel you may as well just move /home elsewhere and image the partition.

Hm, you could also do so many other things? I feel like you are talking about on-the-shelf options, not all the options.

You could use something like marsnix and do a filter of what you want. You could do (a) and download all FODs too.

@physics-enthusiast

The ability to back up only downloaded sources combined with Nix’s reproducibility give you the best of both worlds. By building the intermediate derivations only when you need them (and then optionally GC-ing them afterwards), you get the rebuild flexibility of a build-time closure at the persistent disk space cost of a run-time closure.

Not sure if I agree that it gives the best of both worlds, but yes, it caters to the usecase “I’d like to modify my configuration offline and rebuild it and I have spare capacity”.

@physics-enthusiast

I guess, in a sense, this (pre-)RFC is as much a request for assurance that a certain capability (hermetic dependency retrieval) will always be supported as it is a request for the implementation of a specific means of achieving it. I still believe that a primop that directly returns references to all immediate dependencies of type <x> (that can be used by other Nix expressions) is still the most ergonomic, composeable and flexible solution to the problem, but since recursive-nix gets us most of the way there for build-time (albeit with suboptimal ease-of-use and uncertain stability like I outlined above) and Tvix’s path-scanning for run-time, I think I can settle for just getInstantiationDeps (or some alternative that makes possible the same ends) if my justification for the other two is considered insufficient.

It makes sense to me, I agree that something like getInstantiationDeps would be interesting. I wished this was built-in as a result of doing Flakes but maybe that should be one of the component for fixing the Flakes layering violations.

@physics-enthusiast

Out of curiosity, would an optional derivation advanced attribute be less “risky” to introduce than a primop, in your opinion?

Well, take for example disallowedReferences, it’s currently an advanced derivation attribute which is broken, it adds references back when you disallow them. Nixpkgs has a workaround for them, which effectively split the set of packages in nixpkgs into the disjoint union of bootstrap packages (non-workarounded) and anything that uses stdenv.mkDerivation (workarounded), which is quite problematic if you wanted to implement disallowedReferences the right way because you would bust all the years of cache if so.

What is the solution here? We would need derivation special attributes versioning to be able to move the ecosystem forward.

Thus, it’s the same as primops, just slightly different and less visible. And I can cite many other special derivation attributes which are broken IMHO but unfixable.

@physics-enthusiast

I’ll give it a try. I suspect getInstantiationDeps might require recompiling a full fork though, since I need to modify the internal behavior of existing builtins like import to yield additional information rather than merely adding new ones (assuming I correctly understood what plugins are able to do. Can plugins hook function calls?).

It’s OK to recompile a full fork of the interpreter, take 2.18 as a snapshot, work your feature in there, and showcase it.

“Can plugins do […]?” Well, there’s two answers:

Theoretically, yes, it’s C++, you can make any trampoline you want .
In practice, I’m not so sure you can do that idiomatically.

I would just take the shortest path if you value your time, so even if it’s ugly and bad, as long as you get an interesting result, there’s value to me.

[roberth]

@RaitoBezarius

Though, interestingly… !

nix-repl> :p builtins.getContext "${(builtins.toFile "abc" "${builtins.toFile "def" "ghi"}")}"
{
  "/nix/store/kc46v2cfdvilicx6g97qqg49a2q1hi8j-abc" = { path = true; };
}

The context only needs to track direct references. The closure is fine:

$ nix-store -q -R /nix/store/kc46v2cfdvilicx6g97qqg49a2q1hi8j-abc
/nix/store/j5hzmq2wpf6rz7bq2w27lvxr4m1jl22s-def
/nix/store/kc46v2cfdvilicx6g97qqg49a2q1hi8j-abc

[RaitoBezarius]

It makes sense. So this reveals indeed a gap in the language-level features to introspect the closure without IFDs as this makes it impossible to analyze the dependencies of any toFile as far as I understand it.

[roberth]

Not exclusively in the evaluator without IFD, correct.
Getting references (or their closure) may require IFD, but not when it’s a constant path, or when first taking derivationOf (TBD) on a non-dynamic derivation output, to produce build dependencies instead of output references. Currently all derivations are non-dynamic, as the RFC 92 implementation is not quite working yet. (That’s a lot of dense, somewhat incomplete detail; hope it helps a bit though)

[physics-enthusiast]

@roberth Would something like a generic function inverse be on the table? Some inverse such that inverse (primop arg1 arg2) produces { function = primop; arguments = [ arg1 arg2 ]; } (or equivalent). That way we’d only need to propagate import contexts through the small subset of interactions that could forceValue an imported object without a function call, such as attrset attribute selections and imported lambdas which themselves call other imported lambdas, since we can recover import dependencies from the inputs of any other interaction by recovering the input themselves via inverse. Since most Values are literals or the result of a function call (and hence have empty context), the performance impact of import contexts would be minimal, and we would simultaneously be able to avoid exposing internal lazy/strict behaviour to the user by leaving it to them to figure out which argument to include. For this particular usecase inverse would be very nice, but since it directly runs counter to the functional paradigm of pure functions I suspect the proposal wouldn’t get very far.

[roberth]

@physics-enthusiast

directly runs counter to the functional paradigm of pure functions

Yes, this would break so many rules; it must not be done.
It’s better to have a little helper function that achieves the same goals where it’s needed, but even then, you’re exposing a rather arbitrary and unstructured piece of information whose details nobody should have to depend on. Consider for example that tests and updateScript dependencies are also in there.

[physics-enthusiast]

@roberth

Yes, this would break so many rules; it must not be done.

That’s rather unfortunate, but unsurprising.
Hmm, how about an extension of the flake interface? Right now the currently unsolved problem is something along the lines of:

1. Need to capture a strict superset of all import calls (because even a single one missing is an eval failure).
2. Need to make the superset only as big as is necessary to achieve Point 1 so we don’t end up hitting throws and aborts that were meant to be lazy. Basically, backing up imports should never itself act as a strict operation.
3. The set required by Point 2 is slowly shrinking as more aspects of the language become lazy.
4. Point 3 renders the output of whatever mechanism used for this purpose both unstable and unstabilisable without significantly constraining future increases in laziness.
5. Because of Point 4 and implicit pure-nix return value stability guarantees, I can’t just make this a primop.

Leaving it to the user was an attempt to get around this, but what if I instead shove the perpetually-unstable parts into the output of some flake subcommand (let’s call it nix flake reify) and declare that while using it ensures you can evaluate the flake in question offline, whatever actually ends up in the file is an implementation detail? Then, the stable, well-behaved remainder (inputDrvs, inputSrcs, and dynamic drivers) can be retrieved via primop.

Also,

@physics-enthusiast

Almost, but not quite. inputSrcs are instantiation-time dependencies. inputDrvs rely on string context for dependency resolution, and it is possible to intentionally or unintentionally lose it. I could be wrong on this, but since .drv creation and string interpolation is done at eval-time, if that “invisible” inputDrv itself depends on some kind of eval-time fetcher (which now cannot reach the internet), you can get an evaluation failure.

after some more thinking, I’m retracting this statement. Just came to the realization that even if you could lose inputDrvs, those missing dependencies would not be mounted in the sandbox of their dependent either, so the online build would fail too (if it tried to access said path). So build dependencies are indeed just inputDrvs, and instantiation dependencies are inputSrcs + dynamic derivers + imports.

P.S. To expound on my concern about imports, consider the following toy example:

let
instanDep = <something that does a download>
importedBool = import (instanDep/false.nix);
in
usedElsewhere = if importedBool then derivation {...} else derivation {...}

Now we have an invisible dependency on instanDep (evaluation fails without it present). Notably, this is an issue whether instanDep is produced by build-time IFD of a FOD or eval-time builtin fetcher (which is how flake inputs are obtained, last I checked).

[roberth]

Tracking which values may depend on which IFDs would be costly in terms of performance or implementation complexity.

Current solutions are either to keep track of IFDs “by hand” in the expression language, or to log them, or evaluate with a separate store that will contain all IFDs by the end of it.

By the way it’s not just import that triggers the IFD phenomenon, but also readFile and other functions that get information from the file system.

[physics-enthusiast]

@roberth

Tracking which values may depend on which IFDs would be costly in terms of performance or implementation complexity.

Current solutions are either to keep track of IFDs “by hand” in the expression language, or to log them, or evaluate with a separate store that will contain all IFDs by the end of it.

In case I wasn’t sufficiently clear before, my previous reply was an alternative to the getInstantiationDeps I presented in my initial post. The idea was that to get around the function stability issues with the original proposal, we give up some granular control but retain airgappability at the installable level by just logging all the import calls (and equivalents that result in anything that isn’t a derivation, like readFile) made during a given nix build. In other words, we would no longer track imports per derivation or per Nix Value, but rather per final flake (i.e. the one not used as an input to other flakes). By appending to a global list, we no longer need to set up any complicated hermetic tracking structures. At the end of a build the store paths in the list are copied into an opaque file that allows you to nix eval a single flake/flake subattribute offline. Essentially, what I’m trying to achieve is the flake version of Deterministic evaluation of Nix expressions (and tracking who produced a derivation) · Issue #553 · NixOS/nix · GitHub.

@roberth

By the way it’s not just import that triggers the IFD phenomenon, but also readFile and other functions that get information from the file system.

Not just IFDs, but for airgappability we need any import or import-equivalent that depends on the existence of something outside the store to not throw. This includes, for example, your run of the mill import-from-fetchTarball.