It’s time to stop using overlays (in most cases, see below).
This tidbit greatly helps me out! I have been wondering about how overlays and flakes interoperate.
For nixos modules, you can use specialArgs to pass different package sets.
Very interesting.
I’m curious though about the “When to use overlays” case.
Let’s say I’ve got a flake F that depends on a flake D which in turn depends on a package P inside nixpkgs.
I get that with overlays I can modify P in a way that fits my needs. But with the recommended “composition over inheritance” wouldn’t I loose the ability to override D that I would otherwise have with overlays?
Yes, that’s true if you use flakes, or built your own replacement of nixos-rebuild. What I have most often seen is the full inputs being passed to the specialArgs instead of just the packages for the current system though. It would be nice to set a convention for that.
That’s right. This is a limitation of how the flakes work; there is no mechanism to override flakes. For that use-case, each flake would also have to expose an overlay. Probably this should be part of a dedicated Flakes best practices article.
This only solves the case where you want to pull in an additional new package which was pinned with Niv.
There is a big use case where you want to take an existing Nixpkg artifact and muck with it a bit (change the version or some flag).
I don’t see how that can be mitigated without the use of an overlay.
I also don’t see the problem with:
other-dep = import sources.other-dep { pkgs = prev; };
This is propagating the Nixpkgs downstream still which is the right thing to do and doesn’t end up with copies?
As long as you are at the root of the dependency tree, that’s fine. The main point is that overlays get cargo-culted a bit too much and then get used in dependencies as well, ending in a proliferation of instances of nixpkgs.
The post proposes the following flake:
{
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
inputs.other-dep.url = "github:other/dep";
# Use the same version of nixpkgs as us
inputs.other-dep.inputs.nixpkgs.follows = "nixpkgs";
outputs = { self, nixpkgs, other-dep }@inputs: {
packages = nixpkgs.lib.genAttrs [ "x86_64-linux" ] (system:
let
p = {
nixpkgs = inputs.nixpkgs.legacyPackages.${system};
# other-dep would also access `inputs.nixpkgs.legacyPackages.${system}`
# thus only using a single instance of it.
other-dep = inputs.nixpkgs.packages.${system};
};
in
# your code here accessing `p.nixpkgs` and `p.other-dep`
);
};
}
I do not quite understand the line other-dep = inputs.nixpkgs.packages.${system}. The input other-dep actually does not seem to be used. Did he mean it to be inputs.other-dep.packages.${system}?
The problem are not overlays but rather misuse of overlays. A flake should primarily expose an overlay and optionally a nixpkgs instance with that overlay applied. The user of that flake should then import the flakes overlay into his own nixpkgs instance.
No, a flake should definitely not expose an overly as primary means.
It should primarily provide packages and nixosModules.
If it was like this and more modules would provide a package option, then in the long term overlays will become less and less important and will eventually become a tool for conscious root level manipulations of the tree, rather than a tool to add leaf packages and random rebuilds…
My impression was that flakes should generally expose both an overlay and packages (and defaultPackage if applicable). The overlay is used for composition, and the packages set is used when consuming the flake directly. This way you can evaluate nixpkgs once with several overlays applied if you want, and you can still just nix run the flake or pull packages out of its packages set if there’s an applicable binary cache this will pull from.
There’s still a downside where each flake that redeclares nixpkgs will potentially require downloading and locking a separate copy of nixpkgs (e.g. if one is nixpkgs-unstable, another is master, another is nixos-21.11-small, etc). That’s annoying. But that can be mitigated by using inputs.foo.inputs.nixpkgs.follows = "nixpkgs" (though the repetition there is annoying). And flakes that don’t really care about what version they use can just avoid declaring it at all such that it comes from the nix registry.
Using a flake as input and setting it’s follows for nixpkgs to whatever you want, is effectively like an overlay, but without the computational overhead for re-evaluating nixpkgs.
At least from what I understand.
While at the same time, the way how most flake-overlays are written, still require a “follows” to be applied against your nixpkgs. At least in those flakes that expose both.
Since flake inputs are fetched lazily, I don’t think you need to worry about this if you are using a correctly created overlay.
The point of the article is that Flakes should never* import nixpkgs { inherit system; } but use nixpkgs.legacyPackages.${system} instead.
As a secondary option, you could also expose an overlay. But the primary mechanism should be to expose packages. Overlays, by design, cannot be used reliably without reading their implementation. Sometimes they have a dependency on another overlay, and the ordering can only be understood by reading the code. Same with knowing which attributes are going to be extended or overridden.
*: unless your project is not public. Or if nixpkgs needs to be parameterized.
Given legacyPackages is defined as:
Aren’t those expressions equivalent?
They are currently, though nixpkgs might transition to a more efficient version.
Let’s say you have a tree of flakes:
- A inputs:
- nixpkgs
- B
- B inputs:
- nixpkgs
And assuming that you’re using the “follows” feature in A where:
{
inputs.B.inputs.follows.nixpkgs = "nixpkgs";
}
Now if nixpkgs.legacyPackages gets accessed from A or B, they will both be using the same instances of nixpkgs. That’s the main mechanism we can use to avoid re-creating nixpkgs for each flake.
Here is a follow-up discussion about how to use unfree packages with that requirement:
No, overlays are still far more powerful, in a way I don’t think we should be throwing away. By composing overlays, I can change a lower level dependency if, e.g. I need to fix a vulnerability in a common system library, and that gets applied to all packages in base nixpkgs and all overlays, without having to fork nixpkgs. Or if I don’t like the configure flags that you applied to your package, I can change that with override, and then I don’t have to manually go back and thread it through to all my other dependencies; the overlays just make sure it happens automatically.
It doesn’t require reevaluating nixpkgs once per overlay; it’s all just one evaluation of nixpkgs, with minimal overhead per-overlay.
The composition abilities of overlays is unmatched and we should definitely not be throwing it away.
That said, yes, please still use follows. We really don’t want a node_modules situation on our hands, with thousands of copies of the same repos, all for one project. (Sidenote: This is one of the things fixed by the fixed point style of nixpkgs; we don’t have this hierarchy of packages saying they want other packages. Instead, we have a flat set of known definitions of each specific thing. It’s one of my only gripes with the design of flakes)
As I said, in my opinion overlays should be only used for “conscious root level changes of the dependency graph”.
They are not a tool to add leaf packages, they are a tool to fine-tune the full tree.
As you said, security fixes you want to apply system, at the potential cost of recompiling everything.
Or if course if you want unsafe march=native or a safer explicit version of this.
This are legitimate use cases of overlays.