Build derivation on first run

Help
1

There are some packages, that I rarely use, but still need occasionally. Some of these packages are kind of large, so I don’t want to permanently include them in my NixOS closure.

nix run nixpkgs#someBinary is mostly fine, but it’s not perfect. My main UX gripes with this approach are

  1. I have muscle memory for some of these commands, so I often forget that I don’t have the package permanently installed and try to use someBinary directly, before correcting it to nix run ...
  2. The command is kind of long to type and doesn’t always play nicely with shell history/autocompletion
  3. The name of the package in nixpkgs doesn’t always match with the actual binary name, so I sometimes have to go to search.nixos.org.
  4. Even with a pinned nixpkgs version in my registry, the nix run nixpkgs#... approach still doesn’t include the overlays/customizations that I declare in my /etc/nixos

So, I had the following idea. In principle, it is possible to instantiate a derivation and get its corresponding derivation (/nix/store/aaaa...-someBinary.drv) without building/downloading the actual full derivation (/nix/store/bbbb...-someBinary).

I have verified that this works in the CLI:

  1. DRV=$(nix eval --raw .#someBinary.drvPath), see that the .drv file now exists
  2. OUT=$(nix eval --raw .#someBinary.outPath), check that the output path doesn’t exist at this point
  3. use nix-build --no-out-link ${DRV} to show that you can build the package straight from the .drv
    (btw, can somebody tell me, how to build straight from the .drv using the new nix3 CLI?)
  4. check that ${OUT} now exists

So, now all we have to do is to automate an approximation of the above with nix. I wrote the following wrapper

mkLazyPackage = package:
  pkgs.writeShellScriptBin
  package.meta.mainProgram
  ''
    # Instead of using the package directly, remember its `.drv` path
    DRV="${package.drvPath}"

    # And build it when the user tries to run the package
    PACKAGE="$(${pkgs.nix}/bin/nix-build --no-out-link "$DRV")"

    # Run the main program by default
    exec "$PACKAGE/bin/${package.meta.mainProgram}" "$@"
  '';

that could be used like this in your NixOS config

environment.systemPackages = with pkgs;
  map mkLazyPackage [
    # Lazy packages (will be built/installed on first run)
    someBigApplication
    anotherBigApplication
    ...
  ];

Unfortunately, this doesn’t work. The supposedly “lazy” packages are built/downloaded during the NixOS build.

Further testing seems to indicate that using package.drvPath in a derivation makes it depend on package itself (even though the actual package shouldn’t be required to build itself).

Does anybody know, how to get the package.drvPath string without getting a dependency on the package itself?

1 Like
2

I’m not sure how similar its underlying approach to the problem is, but I’d look at GitHub - nix-community/comma: Comma runs software without installing it. [maintainers=@Artturin,@burke,@DavHau], which implements a comma (,) command that you can use a bit like nix run ... or nix-shell ... --command ... (with the benefits of brevity and being able to use the executable name instead of the package name).

You might decide that it’s close enough as-is, or its implementation may help clarify the way forward?

1 Like
3

I am aware of comma, but for me it generally has most of the same problems that nix run does.

I’d really like the ability to “prepare” derivations (possibly with custom modifications/overlays/etc applied) ahead of time in my system configuration, without building them. And then to be able to “materialize” them only when I actually need them.

The mkLazyPackage example is actually one of the simplest use cases that I had in mind. I have a lot of more complex things that I would like to do if/when I figure out, how to refer to drvPaths without depending on the actual package.

1 Like
4

After some more trial and error, it seems that

builtins.unsafeDiscardOutputDependency package.drvPath

does what I wanted. At least in my testing, it seems to behave like I expected the original “lazy” derivation to work: /nix/store/aaaa...-someBinary.drv is created and referenced in the lazy script, but /nix/store/bbbb...-someBinary is only created after running the script.

What is weird is that nix-store --query and nix path-info both seem to think that the closures of the original lazy script version (that leaked the package into the context) and the one with unsafeDiscardOutputDependency are the same, which is really weird. It seems that they don’t actually see the /nix/store/bbbb...-someBinary path as a dependency even without unsafeDiscardOutputDependency.

Also, I am not totally 100% sure if after the unsafeDiscardOutputDependency hack, the .drv file would still be included together with the “lazy” package in all cases (such as when the “lazy” package was downloaded from a cache, rather than built locally, or after a gc sweep of the store).

5

I would just make shell aliases for foo="nix run nixpkgs#foo"