Letting upstream use Nix

Development
1

Per @zimbatm:

As a side note, and related to point (1), it would be wonderful if the upstream developer could benefit from using nix as well. Imagine if the author of github.com/foo/bar had a default.nix file already defined, which is it necessary to re-declare the same thing inside of nixpkgs?

All we should have to do in nixpkgs should be:

bar = callPackageTwice (fetchFromGitHub { repo = "github.com/foo/bar"; rev = "..."; sha256 = "..."; ...});

It’s the same argument as letting the upstream developer be responsible to controlling the dependency set (and security updates).

I think there’s a lot of power and utility to be had here, especially with some straightforward tooling along update and review.

4 Likes
2

While I like Nix very much, I think “upstream” should be agnostic to any package manager or distribution as much as possible.

letting the upstream developer be responsible to controlling the dependency set (and security updates).

This is funny :slight_smile: The upstream developers can’t be aware of all existing or possible use-cases and integration scenarios. The best they can do is not to get in the way with their *.nix, ./debian, Dockerfile etc. efforts.

1 Like
3

Let’s imagine that callPackageTwice is introduced for a second.

Hydra could be extended to allow import-from-derivation but only for callPackageTwice. We had a lot of discussions on IFD but I think with that reduced scope we might come to an agreement.

Nixpkgs could get a new pkgs/upstream folder where each sub-folder only contains metadata to fetch from upstream.

The pkgs/upstream/overlay.nix could look like:

self: pkgs:
pkgs.lib.mapAttrs (name: value:
  if value == "directory" then
    pkgs.callPackageTwice ./. + name;
  else null
) (builtins.readDir ./.)

Over time, pkgs/top-level/all-packages.nix would shrink size and only include the complicated stuff and overrides.

It would also be quite easy to add a few maintainer scripts:

$ ./maintainers/scripts/new-package.sh github.com/owner/repo@version
Creating pkgs/upstream/repo ...
  Found rev c58d0fc3677d455024cd21916f4570c5e2d23727
  Found sha256 121fz549vlxdasw0lcx6v394x8g34nadvrfs4svw3b65n0shv1ky

$ ./maintainers/scripts/update-all-upstream.sh
Scanning pkgs/upstream ...
- repo ...
   no new upstream version found
- other-project
   found new version 1.2.3, updating

So gradually move more parts of nixpkgs to become only metadata so they become scriptable and introspectable.

4 Likes
4

By the way, this is not entirely my idea. @Mic92 wanted to create an overlay for something similar at https://github.com/nix-community/NUR . The repo is still empty and I don’t know what happened but that might be another way to test the idea.

5

We’d still want to be able to override or modify upstream derivations. I think it would be great if upstream projects can fully specify their dependencies.

1 Like
6

How hard would it be, ultimately, to prototype callPackageTwice though?

7

Since we have GitHub - shlevy/nix-plugins: Collection of miscellaneous plugins for the nix expression language i assume it’s just a matter of writing some C++, adding it to your nix config, and try it.

That said, I can’t do C++, but I’m hopeful someone who actually knows the language might be able to help out.

8

Does it even need to be that? Intuitively it’s like:

callPackageTwice = attrs: let
  innerPackage = callPackage attrs;
in
  callPackage innerPackage;

I assume there’s something naive about that?

1 Like
9

I don’t have much to add other than I’d love this. I actually ended up trying something similar for the gup package when I realised that IFD was possible (mind blown), but the build failures on hydra made me back it out.

@zimbatm is it possible / safe to whitelist IFD for this specific use? Is it dangerous for hydra to run unreviewed / untrusted code? I’m hoping not, but I don’t know.

Also, be aware that IFD breaks nix-collect-garbage (Figure out how to make IFD work properly · Issue #954 · NixOS/nix · GitHub - derivations get cleaned up when they shouldn’t), which you’d want to fix before leaning too heavily on this.

10

I think the main reason why IFD is disabled on Hydra is because it’s a way to enforce that no derivation will be evaluated during nix-env -qa. Otherwise Nix would have to at least download the remote repository to fetch the metadata.

So callPackageTwice should be designed in such a way that the metadata and name can be queried without having to fetch and evaluate the remote nix code.

11

I now have added actual content to NUR: https://github.com/nix-community/NUR

12

One thing that’s always awkward with IFD is what do you set src to in the imported derivation?

I’ve been using this trick:

	src = if lib.isStorePath ../. then ../. else (fetchFromGitHub { ... });

(opam2nix-packages/default.nix at 54b8f7ebdeef8f845018f2069e7ff2d593c3ab6b · timbertson/opam2nix-packages · GitHub)

That seems kind of awkward, and I also have an open bug which implies it doesn’t always work for some people. Does anyone know of a better way? If you have a real src set which fetches a commit from github, you’re going to be fetching two versions - one for the nix derivation, another for the actual source. Which gets very confusing.

13

I’ve been using fetchTarball with a sha256 to fetch a tarball of the repo. In the Nix code it is sufficient to set src = ./.;.

1 Like