Selective use of differents sets of nixpkgs packages

Help
1

I am creating a nixpkgs overlay like so:

inputs: final: prev:

let

  mkNixpkgsBase =
    { nixpkgs, system }:
    import nixpkgs {
      inherit system;
      overlays = [ inputs.self.overlays.default ];
      config = {
        allowUnfree = true;
      };
    };

in
{
  previousPackages = mkNixpkgsBase {
    inherit (prev) system;
    nixpkgs = inputs.nixpkgs-previous;
  };
  stablePackages = mkNixpkgsBase {
    inherit (prev) system;
    nixpkgs = inputs.nixpkgs-stable;
  };
  release2505Packages = mkNixpkgsBase {
    inherit (prev) system;
    nixpkgs = inputs.nixpkgs-2505;
  };

}

I am then using them in NixOS configuration like so:

  environment.systemPackages = with pkgs; [
    pkgs.previousPackages.yubioath-flutter
  ];

However, this results in high memory usage and slower evaluation time. Any ways to optimize this so I can use older and newer packages selectively from nixpkgs? This method is nice and flexible, but long evaluation times (circa 4-5 seconds added if I use a package from one of the nixpkgs sets (e.g. release2311Packages.taskwarrior-tui). Would using legacyPackages be faster? I know it has other disadvantages (https://discourse.nixos.org/t/using-nixpkgs-legacypackages-system-vs-import/17462/19).

2

It will always have a significant overhead. Ultimately nixpkgs must be evaluated, and that is by far the most expensive thing you do. Grabbing multiple nixpkgs versions should be avoided whenever possible.

I’m uncertain what the performance impact would be, but in a flake world I would suggest not using overlays for this. Overlays as a tool exist to change the contents of the pkgs variable, and overlay packages in such a way that they modify dependencies recursively in the tree. This is simply counter the very idea of flakes, which suggests that rather than a huge centralized pkgs instance we use smaller flakes that independently provide packages. Overlays have a purpose still, but they should not be the preferred solution to everything.

There is a performance impact to this, and the way you resolve your nixpkgses breaks the eval cache. I’m uncertain this is 4-6 seconds of eval time, but it will cost something.

What you’re actually doing is just using pkgs as a global variable to hold a reference to something. The overlay mechanism is not at all intended for this, and a better mechanism for introducing new global variables exists through _module.args / specialArgs - there’s no need to rely on overlays here. It just persists in guides everywhere because people have been given a hammer, and this problem sure looks like a nail…

I’d wager that you don’t need the overlay from self either, and should just use the packages exposed through self.packages, making that whole function superfluous. You most likely also don’t need to recursively apply the same overlay to each nixpkgs instance either.

Instead, I’d recommend using specialArgs to pass references to your inputs. To enable unfree packages in your non-pkgs package sets, use GitHub - numtide/nixpkgs-unfree: nixpkgs with the unfree bits enabled.

This is not per-se for performance reasons, it’s just that you’re significantly overcomplicating something quite simple.

1 Like
3

Thanks for the reply. I ended up with the overlay solution after few iterations of trying different ways to expose packages and an overlay from https://github.com/nialov/nix-extra where I am using this solution (https://github.com/nialov/nix-extra/blob/ae4a6abfeb335381239a73d2be40bc0ed896d22e/overlays/default.nix). I only now noticed the increase in evalution time and memory usage. I understand the criticism of using overlays for things it is not supposed to be used for but it felt like the most useful way for accessing older packages with the same overlay applied across all the exposed nixpkgs package sets. E.g. I could access pkgs.taskfzf, a package in nix-extra, to access the nixos-unstable built version or pkgs.release2505Packages.taskfzf to access a version built with nixpkgs of nixos-25.05.

I felt the other solutions more complex when trying to use different versions of (custom-built) packages across repositories.

4

While I appreciate that this is convenient, do you need it? You almost certainly don’t need to install the same version built with different dependencies of one package in a given single configuration.

I.e., why not just use the nixpkgs.overlays NixOS option and access only pkgs.taskfzf? For an end user the effect of changing dependency versions of a given package is almost vanishingly small, so I would not expect to see these kinds of hacks in a NixOS configuration, save for maybe a one-off thing for a specific package (which I then would still not use nested overlays for, since that’d be a thing to remove in the longer term, once the package/dependency is fixed).

This is also ultimately an issue caused by nix-extra being poorly designed - since it doesn’t expose packages, you cannot easily use the built-in feature of flakes for this. In a flake that exposes packages, you could simply change the inputs.nixpkgs.follows to whatever version of nixpkgs you like; multiple times to get different dependency trees. Since you seem to contribute to it, perhaps consider fixing that?

1 Like
5

I.e., why not just use the nixpkgs.overlays NixOS option and access only pkgs.taskfzf? For an end user the effect of changing dependency versions of a given package is almost vanishingly small, so I would not expect to see these kinds of hacks in a NixOS configuration, save for maybe a one-off thing for a specific package (which I then would still not use nested overlays for, since that’d be a thing to remove in the longer term, once the package/dependency is fixed).

The reason is often due to the package build failing with newer nixpkgs due to dependency or other issues. But that does not mean the package is not useful. Fixing a package might take a lot of time, which, when doing this as a hobby, there might not always be. Of course fixing, or upstreaming to nixpkgs, would be ideal.

This is also ultimately an issue caused by nix-extra being poorly designed - since it doesn’t expose packages, you cannot easily use the built-in feature of flakes for this. In a flake that exposes packages, you could simply change the inputs.nixpkgs.follows to whatever version of nixpkgs you like; multiple times to get different dependency trees. Since you seem to contribute to it, perhaps consider fixing that?

nix-extra does expose packages and part of the reason for the overlay approach is also seen there, to succinctly keep older, packages that fail with newer nixpkgs, still exposed through packages (as well as the nested nixpkgs in one single overlay). See https://github.com/nialov/nix-extra/blob/ae4a6abfeb335381239a73d2be40bc0ed896d22e/per-system.nix#L44.

I do not want to claim my solution to be good, just explaining my reasonings! I also do not clearly see how would there be less instances of nixpkgs to import and evaluate if I refactored according to your suggestion? I am not familiar with the specifics of eval caching etc. If I use inputs.nix-extra.packages.x86_64-linux.frackit, which is built with older nixpkgs, I would still need to evaluate that older nixpkgs?

6

Hm, in that case I would simply update the lockfile of nix-extra less frequently, or pin the nixpkgs input of the package whose dependencies break frequently to a version that is not broken and only bump that when I have time to fix the fallout.

As-is, you’re solving an upstream (nix-extra) problem downstream (your NixOS config), which makes splitting the projects almost pointless. It’s a “leaky” abstraction - your configuration should be able to rely on your package working if you use the flake’s package, otherwise the flake isn’t doing its job correctly. By using an overlay you’re fundamentally breaking the contract.

Doesn’t mean the projects should be merged, but you’re definitely not maintaining appropriate layers of abstraction, which results in these headaches.

Right, yeah, see my point above. If you have these issues, you should be pinning different versions of nixpkgs for those packages that need them, and not try to offer versions of those packages for all nixpgkgs under the sun via overlays.

If this results in an undue maintenance burden, I’d suggest splitting nix-extra into more flakes, revolving around the ecosystems that have similar update cadences/dependency issues.

Solving the underlying upstream problem that causes simple bumps to fail would probably also not be a bad idea, though I agree that’s likely actual work cleaning up tech-debt rather than a high-level project design consideration.

And that’s all I wanted to know :slight_smile: I’m trying to be helpful here - my point is that your abstractions are all over the place, which I think stems from misunderstanding of the tools available to you, and perhaps painting yourself into a corner by focusing too much on overlays. This is a high-level API/design issue, not a performance/feasibility/implementation issue.

Of course your solution works, I’m just looking at it and going “hm, this isn’t at all how a flake is supposed to look, and it makes you add a bunch of boilerplate; Why is that so, and can we improve matters by adopting a more idiomatic approach”?

Theoretically only when that older nixpks changes. Nix keeps a local cache of the evaluated state of flakes, allowing it to skip most of the evaluation once it has done so once, thereby reducing the time it takes to evaluate this overall. This isn’t available if you use import to evaluate the default.nix in the root directory of a flake that refers to some version of nixpkgs.

In practice, I’m curious if this would affect performance much in your case. It could, but I’m uncertain.

My primary concern is helping you adopt a more idiomatic approach to reduce the boilerplate in your repository, both to help you simplify your repositories (perhaps leading to better performance, or at least more easily benchmarked evaluation), and help along the negative opinion others have of nix that results from it. I think it can be quite simple, it’s just that (relative) newcomers are very prone to over-engineering and sharing their over-engineered solutions, resulting in this becoming the de-facto way people write nix projects.