Not-RFC: Reserve a top-level "alt" attrset for overlaying onto

Development RFCs
1

Cross-post of PR 452282.

Summary

Reserve at the top-level of Nixpkgs an alt attrset for overlaying additional package sets onto.

Motivation

We believe Nixpkgs to be the solution that solves the widest variety of problems in software deployment. This is a logical conclusion given that a Nixpkgs expression has the widest variety of packages and potential configurations available relative to any other solution.

For developers and admins to solve an increasing variety of problems, the variety internal to Nixpkgs must increase accordingly. Increasing the automony of developers allows them to increase the rate at which they solve problems.

The recommended way of autonomously augmenting Nixpkgs is to apply overlays. As internal conflicts intensify and the ratio of reviewers to committers widens, the variety of Nixpkgs maintainers is decreased and more packages will inevitably be pushed onto overlays.

Overlays however suffer from a few problems:

  • no indication of which packages are original and which are overlayed
  • naming clashes
  • evaluation overhead
  • lack of conventions for composition

An empty attrset at the top-level of Nixpkgs is proposed as a namespace for attaching overlayed package sets to address these issues.

Detailed design

Reserve an empty attrset alt at the top-level of Nixpkgs.

Nixpkgs maintainers shall not declare anything within this attrset.
Exceptions may be made in the future for functions with names prefixed by an underscore (_).

Overlays that provide specialised package sets are encouraged to declare package sets within the alt attrset.

Resolving name clashes within the alt attrset is the responsibility of overlay maintainers and end-users.
Nixpkgs maintainers abdicate all authority over the use of alt.

Examples and Interactions

Overlays should declare package in a single set within alt:

let altName = "uniLab321"; in

final: prev:
let  
  callPackage = final.newScope final.alt.${altName};
in {
  prev.alt // {
    ${altName} = {
      foo = callPackage ./foo { };
      bar = callpackage ./bar { };
    };
  };
}

Overlays that conform to the alt namespace convention can be lazily applied to Nixpkgs using an intermediate overlay.

{
  overlayComposition = final: prev: {                                        
    alt = prev.alt // final.lib.mapAttrs (name: f: (f final prev).alt.${name}) {
      uniLab321 = import ../lab/overlay.nix
      bakaGe = import (<games> + /overlay.nix);
      guix = import ./guix-adapter.nix;
      localHackspace = import /local/hackspace/overlay.nix;
      autonomousMurderCorp = import ../work/overlay.nix;
    };
  };
}

Drawbacks

  • Political decentralisation.
  • No reliable mechanism to resolve cross overlay dependencies.
  • Probably worse error messages.

Alternatives

  • Wait for it to get into Nixpkgs
  • Fork Nixpkgs
  • --extra-experimental-features flakes
2

I like the initiative because there is a historic habit of overlaying nixpkgs to add custom attributes, which has all the downsides you mention, however:

I think this is generally no longer true. It’s simply not recommended to augment nixpkgs like this at all - overlays are primarily there to permit tree-wide overrides when necessary (where changing existing attributes is the goal).

To create additional package sets, just create a separate attrset and import that into your context instead. There is no reason to marry it into the pkgs attrset through an overlay. In a NixOS context, use _module.args if you want to propagate that to module args, instead of abusing pkgs as a global variable.

Using overlays this way is an anti-pattern to begin with IMO, we should not encourage it by carving out a special niche for it.

6 Likes
3

If you don’t want a proliferation of nixpkgs instances, and you want to use, say, out-of-tree unfree packages, you have to either callPackage it yourself (which is redundant in cases where that’s already done for you) or use an overlay (ideally provided by the third-party project). This is especially noticeable with flakes, as their structure doesn’t provide any other easy way to inject a nixpkgs instance - but overlays are that way.

If the out-of-tree package is a dependency you want to inject into many other packages that you’re packaging from e.g. a 3rd source, an overlay would also make sense in that case - you could pass around .overrides and whatnot but that gets clunky.

1 Like
4

I think overlays are usually a bit complex to expose a package set. It’s also hard to discover which packages they actually export. Maybe it would be better to export a packageset that just takes a nixpkgs instance as an input?

flake.packageset = pkgs: {
 myPackage = pkgs.call package ./my_package.nix; 
}

Maybe we need to specify that a bit more so it works similar to overlays?

3 Likes
5

if i understand then i really like the idea - i’ve become a fan of outputs which have no dependency on inputs, such as nixosModules and overlays… so adding packageSets sounds great to me

to be clear, this is what you mean with your example, right @lassulus?

{
  outputs = _: {
    packageSets.default = pkgs: {
      myPackage = pkgs.callPackage ./my_package.nix;
      # more packages here
    };

    packageSets.somethingElse = pkgs: {
      # etc...
    };
  };
}
2 Likes
6

Just passing by… I think it would quite easy to do with flake.parts and a custom module. I will give it a try just out of curiosity.

7

I’ve been calling the function that does this using. You give it a scope and a set of recipes/package-functions and it handles the details. A more limited interface than overlays, but less footguns. Most people dont need the full power of overlays.

A library to play with these ideas: GitHub - tomberek/cook

The idea is to make it standard for people to expose the set of recipes, prior to creating a fixed-point and without self-references of overlays. Then the lib converts it into the overlays, adds it to the scope provided - usually a nixpkgs - and provides access to the added packages as well as the modified scope.

This hopefully encourages people to use the recipe concept that requires no system parameter, less chance of misuse, fewer concepts to learn. People can access these directly via recipes.my-package without having an intervening function call.

2 Likes
8

Come to think of it, why do flakes insist on the system for packages when nixpkgs already has meta.platforms?

1 Like
9

It’s a poor approximation and replacement for builtins.currentSystem that leaked into the different interfaces of Flakes.

2 Likes
10

looks neat, thanks for sharing

maybe you’re missing an important part of what made this pattern interesting to me though: the complete and total independence from inputs

maybe the difference is too subtle to care about if you live in a flake only world, etc… but explicitly having to pass in pkgs instead of inputs is a big feature here for me

1 Like
11

Yes that was basically the idea. Maybe we want to have an interface which makes this easier to use with non nixpkgs instances? But overlays have the same problem, that they seem to only work for nixpkgs.
It would be nice if this could also be integrated into nix run so we can run packages from a default set? (or flakes should only export one package set anyway?) It could take nixpkgs either from the flake registry on the system or if nixpkgs exists as an input from there? There is some room for ideas how to make it really ergonomic :slight_smile:

1 Like
12

great ideas here

or flakes should only export one package set anyway?

strong disagree! i think it might be reasonable to make default special with respect to nix run, but that doesn’t need to limit the usefulness of exporting multiple package sets!

It would be nice if this could also be integrated into nix run

this is giving me ideas… :slight_smile: let’s be honest… legacyPackages was always a biased and bad idea as an output (at this point i can’t be the only one to think this, right?) - replacing it with what you’re saying here sounds amazing

as a stretch goal, i would really like to see some unification of “outputs schema” between flakes and non flakes… as i mentioned nixosModules and overlays are great, but packageSets as a replacement to legacyPackages would really round this out nicely! a project which is meant for consumption can have a default.nix which is simply imported into the flake.nix outputs… yeah?

1 Like
13

Agreed. The intention is that these recipes should be fully independent from inputs. It should be possible to ((import ./flake.nix).outputs {input1=null; ... }).recipes and have it be usable or callPackage-able anywhere else.

14

sorry, i must have misunderstood the code
i will give it another look over

thanks for mentioning!

15

I recently swapped to using npins, and needed to make sure all my projects were consumable outside of flakes. This is indeed pretty much the pattern I ended up with.

Take my neovim config as an example. Rather than having any logic within the flake.nix, the packages output just calls a local file:

  packages = forAllSystems (pkgs: {
    default = import ./default.nix { inherit pkgs; inherit (inputs) mnw; };
  });

And then this is the contents of the default.nix:

{ pkgs, mnw }:

mnw.lib.wrap pkgs {
  # package config here
}

We’ve now made the package consumable without flakes. All it needs is some attributes pkgs and mnw, and it’ll return a derivation. In my system config, I don’t go through the flake.nix at all, and instead pass the relevant npins sources to this file directly:

  neovimPackage = import "${sources.meovim}/default.nix" { inherit pkgs; mnw = import sources.mnw; };

And it works the same as going through flakes!

You’ll notice that we have the file take in pkgs, rather than the uninstantiated nixpkgs input. This is by design. Flakes are really the only schema that chooses to include system logic (which I think was a mistake in retrospect). By avoiding flake-y patterns, non-flake consumers have an easier time.

This pattern isn’t original, by the way - far from it. This is pretty much how projects used to be consumed pre-flakes. But as flakes have increasingly been adopted by the community, I find less projects providing non-flake interfaces. Sure, flake-compat is always around, but I honestly consider that a hacky fix. It’s really easy to make your project consumable without flakes - you just have to move the contents outside of the flake.nix.

2 Likes
16

That doesn’t quite work though, right? The point of legacyPackages is that it isn’t nested, not that it doesn’t have a system-specific top level.

As an aside, I find myself often sorely in need of this kind of pkgs-bound output when providing builders - it feels wrong to mix those into any interface that’s going to be used with nix run, though.

What about a separate builders output that works the same?

17

right, which is what @lassulus had briefly mentioned - i suggested that packageSets.default could be a special package set which is treated as legacyPackages is

it would still be nice to have multiple package sets available when you’re providing a flake as more of a library to be consumed by other flakes, for example

18

Nevermind, I caught on - the crux is that any packageSets entry needs to be evaluated anyway - there’s no way to make nix flake show work with these outputs.

I don’t think that makes it a good replacement for legacyPackages though, you’re just removing functionality for the sake of it at that point. Getting a list of top-level packages in nixpkgs without having to eval anything is nice for all kinds of things.