Recursive Nix experiment

Development
1

Today it hit me that Recursive Nix is potentially already possible. So I did a bit of an experiment and it works!

Here is the write-up:

https://zimbatm.com/experiment-recursive-nix/

7 Likes
2

Nice post! I like that you write about all the trial and error steps.

I do wonder, does garbage collection work correctly with this?

1 Like
3

Thanks! If people like it I would be happy to write more about hacking sessions like that. It’s almost raw so it’s easy to write up, I just did a single pass to clarify things a bit.

If the outer nix has a reference to the inner one then garbage-collection should not remove the needed dependencies as expected.

2 Likes
4

I think there’s a security caveat to this: allowing /nix/var is
probably equivalent to disabling the sandbox, against a malicious
derivation (though it would likely still protect against simple
mistakes)

1 Like
5

I really like this kind of write-ups, please continue to share your hacking sessions :-).

In a sense, what you did is close to @edolstra’s experimental recursive nix support: give access to the nix daemon from the sandbox. Eelco implementation is more complex as it restricts queries to avoid impurities.

1 Like
6

Yeah good points, I added that /nix/var is a security and build purity in the known issues section. This setup mainly allows to test the feature out before @edolstra’s patch hits master.

Probably the next thing to do would be to extend on this and build an actual package to see how it would workout.

1 Like
7

Wasn’t the plan going more along the lines of ret-cont nix?

https://github.com/NixOS/rfcs/pull/40

8

What are the issues with Import From Derivation?

9

This thread is more about exploring possibilities than advocating any changes in nix. Obviously the additional data points might influence future design decisions.

10

One issue is that there is no longer a single evaluation phase. Consider:

let
  pkgs = import <nixpkgs> {};
  inner = pkgs.runCommand "inner" {} "echo '{hello = 4;}' > $out";
  # this is the import from the derivation output
  outer = import inner;
in
  outer # returns `{ hello = 4; }`

In order to evaluate “outer”, the “inner” derivation has to be built first.

A more realistic example would to re-use the nix code that is being defined upstream. Eg:

let
  pkgs = import <nixpkgs> {};
  inner = pkgs.fetchFromGitHub {
    owner = "direnv";
    repo = "direnv";
    rev = "v2.20.1";
    sha256 = "...";
  };
  # this is the import from the derivation output
  outer = import inner { inherit pkgs; };
in
  outer

For nixpkgs specifically, it would mean that a lot of the nix code could be outsourced to the upstream project if they adopt nix as a build system. But the issue is that commands like nix-env -qaP would have to first fetch the upstream project in order to evaluate the derivation and return the meta information.

With recursive nix we can build the upstream nix code but also enforce that the metadata are coming from nixpkgs directly.

11

I feel like we’re pretty close to being able to just define a wrapper around IFD that enforces that meta comes from a local definition, something like

importFromDerivation = src: meta: (import src) // { inherit meta; }

Unfortunately it appears as though // isn’t strictly lazy on its LHS, it still at least checks that it’s a set rather than merely assuming that, so we can’t use this construct to get access to meta without doing the import. If there was some way to write { inherit (import src) *; } // { inherit meta; } that would work but there’s no actual syntax for that (or rather, the syntax is //).

Of course, for nix-env we also need the name property, which usually also requires the version, but we could stick those in this importFromDerivation definition too, if only we could make // behave the way we want.

12

Based on that idea, how about:

importFromDerivation = { name, src, meta }:
  let
    upstream = import src;
  in
  {
    inherit name meta;
    # create a fake derivation, maybe add a few other keys
    outPath = upstream.outPath
  }

That being said, turning on IFD on Hydra wouldn’t give us the guarantee that everyone is doing IFD properly.

13

There’s also passthru to consider (or any attributes added directly to the resulting derivation). And of course any derivation created this way won’t be overridable either, which we’d really prefer to support.

As for “doing IFD properly”, could we have a pass that just checks the .name and .meta properties of every derivation with IFD turned off to make sure that it works?

Or, heck, maybe we should just push a change into Nix itself that adds a Builtins.importFromDerivation that explicitly takes the name and meta attributes in addition to the source, and then allow that to work with allow-import-from-derivation = false.

14

I’ve been using Nix recursively for years, and it’s been an important component of some projects. The details are on the recursive Nix github issue

It’s really hacky and impure though, so a more integrated solution would be nice :wink:

3 Likes