Why it is needed to do `final: prev: { ... }` when creating an overlay

Hello! I have a question about syntax and how overlays work in general.
I was trying to add unstable packages to my system using flakes. After some googling I stumbled upon this page Flakes - NixOS Wiki. At some moment it suggests the following:

overlay-unstable = final: prev: {
  unstable = nixpkgs-unstable.legacyPackages.${prev.system}; # considering nixpkgs-unstable is an input registered before.
};

What does final: prev: { ... } does? My guess from the syntax is that it is a curried function that has two arguments but… why? I know this is a pattern for overlays but I am confused about why it is like that, how can I know where this come from and why the arguments have to be called final and prev.

Thanks

1 Like

Hi!

I am not an overlay expert, I found:

The comments after # explain it a little

final: # package set with all overlays applied, a "fixed" point
prev: # state of the package set before applying this overlay
{
  my-new-package = prev.hello;
  some-other-package = final.my-new-package;
}

from:

https://discourse.nixos.org/t/what-are-overlays/14680/2

also this tries to explain:

https://nixos.wiki/wiki/Overlays

1 Like

Don’t forget Nixpkgs Manual Chapter 3: Overlays which states

Overlays are Nix functions which accept two arguments, conventionally called self and super , and return a set of packages.

The first argument ( self ) corresponds to the final package set. You should use this set for the dependencies of all packages specified in your overlay.

The second argument ( super ) corresponds to the result of the evaluation of the previous stages of Nixpkgs. It does not contain any of the packages added by the current overlay, nor any of the following overlays. This set should be used either to refer to packages you wish to override, or to access functions defined in Nixpkgs.

1 Like

I like the classic PEMDOS example.

If we wrote f( v ) = v * 2; and g( v ) = v + 3 and run an overlay :

let
  env = self: { x = 1; };
  ov = final: prev: {
    f = v: v * 2;
    g = v: v + 3;

    x = final.f prev.x;
    y = final.g final.x

    i = final.g prev.x;
    j = final.f final.i;
  };
in lib.extends ov env

This can show a bit about order of operations, and if you experiment a bit you will discover why you can’t do something like prev.g.

It’s not a curried function.

It’s a fixed point evaluation. Each overlay can be thought of as a link in the chain, and evaluating the series allows you to create a single chain. prev allows you to cheaply reference the package set before your overlay is applied, and final allows you to reference the package set after all overlays are applied.

My guess from the syntax is that it is a curried function that has two arguments but… why?

No currying, the implementation is essentially:

let
  # accumulate all of the overlays, and then leave a "hole" for us to pass the final package set
  # toFix :: Attr Set -> Attr Set
  toFix = lib.foldl' (lib.flip lib.extends) (self: {}) overlays;
in
  lib.fix toFix

In English, the fold and extends will reduce the array of overlays (which are just functions), into a single function which takes the final package set (which we reference with final), and “closes the loop” with lib.fix.

fix :: (a -> a) -> a
fix = f: let x = f x; in x;

fix just applies lazily applies the function to itself. Which is also why the term self appears so often when doing fixed-point operations. It’s also the reason why self: super: { } used to be the norm for overlays, but this really only made sense if you had a background in mathematics. final: prev: { } is more similar to the mental model of packaging.

https://github.com/NixOS/nixpkgs/blob/5097b7e3a0d7756dfcd448311eba084b31be47cf/lib/fixed-points.nix has lots of documentation and examples on what the operations are doing.

EDIT:
NixOS Modules also used fixed-point concepts. Instead of overlays, you have modules; modules only have one argument so prev doesn’t exist, and final is represented by NixOS’s config value.

5 Likes

Also, just to be very explicit, they don’t have to be, it’s just convention. It’s just a function definition.

You could call the arguments jim: bob:, but please don’t do that.

Within a flakes overlays output they actually have to be named final and prev or nix flake check will warn.

2 Likes

Sorry for reviving the topic, but I still don’t understand what you guys are meaning. I know that final and prev are required names (self and super are deprecated) but what they are for, I have no idea and it’s bugging me. Accumulate and then leave a “hole” for us to pass the final package set, does not explain it to me sadly.

They represent 2 “states”.

prev is pkgs without this one overlay (but all the others)
final is pkgs with this (and all other) overlays

1 Like