Correct "modern/harmonised" way to refer to NUR, unstable and different nixpkgs

ppenguin · April 1, 2022, 10:00am

Apparently I’m struggling (again) with some nix basics.

I used to have in my HM configs something like

let 
   unstable = import (builtins.fetchTarball "...") { config = { ... }; }; 
in 
  ... 
  home.packages = with pkgs; [ ... ] ++ (with unstable; [ ... ]);

Now the need has arisen to add more overlays, including some branches of my nixpkgs fork for some not-yet merged PRs, and NUR.

To achieve that I thought I’d use the overlay mechanism in one place, so that everywhere else I can just directly refer to pkgs as the combined result without worrying where they come from (I thought that was the main point of overlays?). The main reason for wanting this is to avoid the risk of unnecessarily having multiple derivations in a complete generation (generally if I use an unstable package I want it to be the only instance on my system, at least for the user) and convenient maintenance of home.nix (and similar for configuration.nix)

# overlays.nix
let
   unstable = import ...
   ppenguin-pandoc = import (fetchFromGitHub ...) # ref to own feature branch
in
   nixpkgs.overlays [
      (final: previous: {
         flutter = unstable.flutter;
         pandoc-xnos = ppenguin-pandoc.python3Packages.pandoc-xnos;   
      })
   ];

# home.nix
imports = [ ./overlays.nix ];
home.packages = [ flutter pandoc-xnos ];

now gives the desired result, but has two problems:

to make it compatible with future updates, it should actually be python3Packages.pandoc-xnos, but I can’t seem to define this correctly without overwriting the original pkgs.python3Packages
I’d like to express things like flutter = unstable.flutter; in an expression that directly takes a list with all package names, to avoid verbosity and tedious maintenance.

Things I could find online are confusing, since there are many ways to achieve it, some of which seem to be superseded by “better” methods, though it is quite unclear which way is actually “current” while being also practical (and working) for this use case.

TLATER · April 1, 2022, 10:34am

I think you can do that with inherit (unstable) flutter foo bar;

ppenguin · April 1, 2022, 4:08pm

@TLATER Thanks. But it appears that when I do that it tries to pull in pretty much all packages in existence, not only the ones that are in my selection…

Am I doing something wrong?

My overlay definition now reads

  nixpkgs.overlays = [
    (final: previous: {
      inherit (unstable) flutter dart ninja nautilus-open-any-terminal vscode-extensions;
      inherit (unstable.libsForQt5) kdeconnect-kde qtstyleplugin-kvantum;
     ...
    })
  ];

and then I simply list all packages I want in home.nix under home.packages.

But as soon as I do home-manager switch I see many packages being installed (mostly in alphabetical order) that I definitely didn’t have before, and that I didn’t select…

shadowrylander · April 1, 2022, 6:29pm

Um, should each package not have their own instance of (final: previous: { ... })?

ppenguin · April 1, 2022, 7:49pm

Ah, then it’s logical that it doesn’t work!
So is there any way to achieve what I want (i.e. defining an overlay containing selected packages from an imported nixpkgs tree) in an elegant/compact manner?

TLATER · April 1, 2022, 8:49pm

@shadowrylander Why? You only need to create one overlay, each can set a bunch of attributes. There’s even an example of an overlay that sets multiple on the wiki. You’ll also find that pretty much all flakes set multiple attrs in overlays.

The final arg in fact exists so that you can sequence multiple packages in one overlay correctly.

@ppenguin I think that is expected, depending on just how many packages you see. If you depend on multiple versions of nixpkgs, you’ll pull in multiple versions of all the dependencies too. This is fundamentally how nix works; each nix package has its own static set of packages it depends on, and unstable has newer versions of a lot of dependencies, so all your unstable packages will pull in a whole tree of updates. This might even include a separate glibc.

Within a single nixpkgs version upstream largely tries to only include one version of everything, but as soon as you have multiple all bets are off.

That said, if it pulls in another version of Firefox or such, maybe inherit has odd semantics I’m not aware of? You can also try lib.attrVals ["flutter" "foo" "bar"] pkgs, and that might even look nicer with this many packages.

jtojnar · April 1, 2022, 9:30pm

That is expected.

If you use nixpkgs.overlays, it literally replaces the packages you use in the pkgs used by your system. So if you replace ninja, everything on your system depending on ninja will have to be rebuilt.

For installing programs, I do not see anything wrong in passing them to environment.systemPackages or home.packages like you did initially. But if you really want a simple list, I would do something like:

home.packages =
  let
    flutter = unstable.flutter;
    pandoc-xnos = ppenguin-pandoc.python3Packages.pandoc-xnos;
  in
  with pkgs; [
    flutter
    pandoc-xnos
  ];

Though for developer tools, it is good practice to install them to each project locally using Nix shell.

shadowrylander · April 1, 2022, 11:21pm

@TLATER @ppenguin Ah, okay; sorry about that!

ppenguin · April 2, 2022, 1:07pm

Thanks, but I’m not only talking about mere dependencies, I’m literally talking about nix appearing to alphabetically copy packages I’ve never seen, even not as dependencies. It was hundreds of packages about half-way the alphabet… eh, it appears that this was caused by a too heavy override of the second overlay!

I think I fixed it, is this the correct way to do this?

let
   ppenguin-pandoc = import (pkgs.fetchFromGitHub {...});
in {
  nixpkgs.overlays = [
    (final: previous: {
      ...
      python3Packages = ppenguin-pandoc.python3Packages // previous.python3Packages;
    })
  ];

The point was to avoid having to define each unstable package with package = unstable.package; but at the same time to avoid having to throw around unstable between my different nix files (that are imported) or repeating the import of unstable. And the intention was to additionally achieve that any things like programs.xxxx.enable would seamlessly use the same (e.g. unstable.xxxx) without actually having to set its package attribute because it would be already overlaid at the toplevel.

Yeah, I was actually already doing that for flutter, so about to remove it, but it was convenient for testing because it’s known to (persistently) have a huge version gap between stable and unstable.

jtojnar · April 2, 2022, 2:10pm

The proper way to override nested attribute sets is with overlay like this:

(final: prev: {
  python3 = prev.python3.override {
    packageOverrides = finalPythonPackages: prevPythonPackages: {
      pandoc-xnos = prevPythonPackages.overridePythonAttrs (originalAttrs: {
        …
      };
    };
  };
})

See also the manual.

This way, just like with general overlays, consistency will be ensured – that is, packages depending on pandoc-xnos will use the overridden package.

If you just replace python3Packages by merging the attribute sets like that, only packages referring to python3Packages.pandoc-xnos will see the overridden package, not those using python3.pkgs.pandoc-xnos or those inside python3Packages, since you are not updating the fix point.

Note that we are using overridePythonAttrs instead of just replacing the package with a random package. Python package are sensitive to the Python derivation used so unless ppenguin-pandoc depends on the same Python derivation as the rest of the set (hard to ensure), the package will likely be unusable.

Right, but that is not the use case for overlays – they are meant to consistently replace a package throughout the whole package set. They allow what you want as a side-effect, but the corollary is that anything that depends on the the replaced package (even transitively) will have to be rebuilt from scratch.

ppenguin · April 2, 2022, 7:52pm

@jtojnar @TLATER
Thanks you all for the enlightenment, so now with the (complex, I couldn’t have thought of that easily myself) python override and the inherit thing it appears to work like a charm

(One small change: since I do not want to override an attribute of an existing package but an entire package in the overlay, I suppose the python thingy should look like this?

    (final: prev: {
      python3 = prev.python3.override {
        packageOverrides = finalPythonPackages: prevPythonPackages: {
            inherit (ppenguin-pandoc.python3Packages) pandoc-xnos;
        };
      };
    })

)

Ah, I see what you mean, yes that makes sense; indeed that would negate the “independent parallel dependencies” benefit of nix, though TBH for system- or user-wide installs I actually think that’s a good baseline for either packages that are a “permanent member” of the set and/or have no complex dependents/dependencies; and the upside is that keeps the system lean. So here it matches the use case quite well for me.