Namespacing/scoping a group of packages

Help
1

I have set up a very basic overlay that adds a bunch of packages on top of nixpkgs— basically along these lines:

However, I’m coming to appreciate that for a variety of reasons, I need to move these into a “namespace” so that instead of nix build .#desktop, instead I would do nix build .#ros.desktop. I’m finding it very confusing trying to understand exactly how to do this in such a way that the callPackage calls are still able to resolve dependencies within this group of packages, but also up a layer in the larger pool of nixpkgs packages.

For example, my packages all have their Python dependencies pointing at pythonPackages.thing, so I’d like to be able to steer pythonPackages = python38Packages within this my namespace and not cause mass rebuilds and chaos elsewhere in the system.

The examples of this that I’ve seen are all really intense, for example nix-ros-overlay (a project in the same space that I am building on) achieves names like rosPackages.melodic.desktop using this setup:

But there’s a lot of other complexity going on there, and I’m having trouble isolating what it is that’s making it work— possibly it’s something going on here with makeExtensible?

Is the explanation easy, or alternatively are there docs that cover this somewhere?

1 Like
2

tldr, but did you try just nesting your packages in an attrs:

{
  packageA = ...;
  packageB = ...;
}
3

Yes I did, but it seems all the callPackage and lazy evaluation machinery breaks down— none of the packages can refer to each other any more once they’re inside that separate mapping.

4

This is called a scope in Nixpkgs. The functions you’re looking for are lib.makeScope and pkgs.newScope, which are used together. The last time I checked, I couldn’t find any documentation for this, but it looks like this is what’s used to create the various namespaces in Nixpkgs. git grep will show you many examples.

I hope that helps. I’ve looked at this stuff briefly but haven’t gone through with actually doing it, because of the apparent complexity. Hopefully someone with experience can chime in.

4 Likes
5

Thanks for the pointer! Yes indeed to the complexity, I’m looking at the various scopes in nixpkgs (emacs, Python, gnome-3, and it seems there’s very little consistency in how things are achieved, and a lot of extra abstraction and trickery going on that seems to obscure the meat of how it works. Hopefully someone else knows of an article or minimal example demonstrating how to make this work— so far my googling is turning up very little. :frowning:

6

A good example of adding a new package set can be found: cinnamon.xapps: init at 1.6.8 · NixOS/nixpkgs@632c4f2 · GitHub

1 Like
7

Okay, thanks! The cinnamon example is helpful. However, one thing I’m still hazy on is understanding how this plays with overlays. It looks like the entry point to it is a callPackage interface (though recurseIntoAttrs is new to me):

So in my situation, I tried doing the makeScope inside my overlay, and this works fine:

final: prev: {
  ros = prev.lib.makeScope prev.newScope (self: with self; {
    my_fancy_package = callPackage path/to/fancy {};
  });
}

But once I want to do things like have packages depend on each other, or be able to do overrides of packages, I feel like I get off into the weeds again. Definitely this is a major conceptual/documentation gap for what I’d expect would be a common need for commercial Nix users! It would be great to have a Pill-style article explaining exactly how this works and why.

8

I’ve have a video demonstrating an overlay here: https://www.youtube.com/watch?v=dGAL3gMXvug

9

Thanks, that is helpful! I think I do have a grasp on overlays on their own— my confusion is when it comes to composing multiple overlays, supplying a new scope in an overlay, and overlaying a scope from another overlay.

10

Okay! So I’ve produced a small example— here’s a trivial, flake-friendly overlay that doesn’t use any scopes, just has a pie package that depends on apple and peach:

And here’s a version of it that’s been adjusted so that the packages are now fruit.pie, etc:

If, as in my case, I’m actually generating hundreds of these scoped packages automatically and then will want to patch some of them, it looks like the two main options are to do a “top level” overlay that patches inside the scope, such as is demonstrated here:

https://nixos.wiki/wiki/Overlays#Overriding_a_package_inside_a_scope

Or, the overlay could be inserted inside the scope, but then you need to manually merge the layers with composeExtensions/composeManyExtensions, as discussed in A function to compose overlays? · Issue #33258 · NixOS/nixpkgs · GitHub.

Hopefully this MWE example is helpful to others who have found this all a bit difficult to get started on.

3 Likes
11

Can I put my inputs in an argument, make it a lambda, and use callPackage?

12

I suppose technically you could with an overlay like final: prev: { inherit inputs; }. Then callPackages would have inputs in scope to pass around.