How is fixed point used for overriding nixpkgs packages?

tim1 · November 15, 2023, 6:12am

17.3. Fixed point

…
nix-repl> fix = f: let result = f result; in result
nix-repl> pkgs = self: { a = 3; b = 4; c = self.a+self.b; }
nix-repl> fix pkgs
{ a = 3; b = 4; c = 7; }
…

17.3.1. Overriding a set with fixed point

Given that self.a and self.b refer to the passed set and not to
the literal set in the function, we’re able to override both a and
b and get a new value for c :
nix-repl> overrides = { a = 1; b = 2; }
nix-repl> let newpkgs = pkgs (newpkgs // overrides); in newpkgs
{ a = 3; b = 4; c = 3; }
nix-repl> let newpkgs = pkgs (newpkgs // overrides); in newpkgs // overrides
{ a = 1; b = 2; c = 3; }
In the first case we computed pkgs with the overrides, in the second
case we also included the overriden attributes in the result.

17.4. Overriding nixpkgs packages

We’ve seen how to override attributes in a set such that they get recursively picked by dependant attributes. This approach can be
used for derivations too, after all nixpkgs is a giant set of
attributes that depend on each other.

To do this, nixpkgs offers config.packageOverrides . So nixpkgs
returns a fixed point of the package set, and packageOverrides is
used to inject the overrides.

Create a config.nix file like this somewhere:
{
  packageOverrides = pkgs: {
    graphviz = pkgs.graphviz.override { withXorg = false; };
  };
}
Now we can build e.g. asciidoc-full and it will automatically use the
overridden graphviz:
nix-repl> pkgs = import <nixpkgs> { config = import ./config.nix; }
nix-repl> :b pkgs.asciidoc-full
Note how we pass the config with packageOverrides when importing
nixpkgs . Then pkgs.asciidoc-full is a derivation that has
graphviz input ( pkgs.asciidoc is the lighter version and doesn’t
use graphviz at all).

Since there’s no version of asciidoc with graphviz without X support
in the binary cache, Nix will recompile the needed stuff for you.

17.6. Conclusion

Nix applications will depend on specific versions of libraries, hence
the reason why we have to recompile asciidoc to use the new graphviz
library.

The newly built asciidoc will depend on the new graphviz, and old asciidoc will keep using the old graphviz undisturbed.

What does it mean by “override attributes in a set such that they get recursively picked by dependant attributes”? Could you explain that in terms of the example in 17.3.1?

What does it mean by “nixpkgs returns a fixed point of the package set”?

Isn’t a fixed point something belonging to a function, and the package set isn’t a function?
Why does nixpkgs return a fixed point? Is nixpkgs something similar to the example let newpkgs = pkgs (newpkgs // overrides); in newpkgs in 17.3.1.?

What does it mean by “packageOverrides is used to inject the overrides”?

Why “Then pkgs.asciidoc-full is a derivation that has graphviz input”?

Why “The newly built asciidoc will depend on the new graphviz”, and “old asciidoc will keep using the old graphviz undisturbed”?

Thanks.

fricklerhandwerk · November 15, 2023, 8:00am

The documentation on how fixed-point computation in Nixpkgs works has never been great. @infinisil @roberth and me recently put quite some effort in clarifying lib.fixedPoints.fix, and I think by now it’s a good starting point to answer your questions. It’s obviously very much incomplete though. There’s also intermittent work going into better documenting and explaining overlays and overrides, but it will take a while to materialise.

tim1 · November 16, 2023, 1:44am

can you or someone try to reply to my questions?

toraritte · February 11, 2024, 6:29am

I’ve been struggling with these same basics, so here’s my take, but take it with a grain (or more like a pound) of salt as this may be a “blind leading the blind” scenario. (Linking the fixed-point combinator wikipedia because it helped somewhat.)

For starters, a reminder to self, here’s fix from Nixpkgs’ lib:

fix = f: let x = f x; in x;

Taking a simpler and shorter example than the one in Nix Pills:

f = self: { a = 27; b = self.a; }

fix f    #=> { a = 27; b = 27; }

because fix f evaluates to

= f(
    f( 
      f(...)))

In this case, evaluating 3 levels is enough

f (f f)

because

     ┏━━━━━━━━━━━━━━━━━━┓
     ┃                  ∨
= (self: { a = 27; b = self.a; }) (
  ---∧----------f-------┃-------- ( 
     ┃                  ┃
     ┗━ (self: { a = 27; b = self.a; }) (
        --------------f-┃-------------- (
                        ┃
              (self: { a = 27; b = self.a; }) (...)))
              ----------┃---f---------------- (...)))
                        ┃
                        ∨

= { a = 27;
    b = (
             ┏━━━━━━━━━━━━━━━━━━┓
             ┃                  ∨
          (self: { a = 27; b = self.a; }) (
          ---∧----------f-------┃-------- ( 
             ┃                  ┃
             ┗━ (self: { a = 27; b = self.a; }) (
                --------------f-┃-------------- (
        ).a                     ┃
  }                             ┃
                                ∨

= { a = 27;
    b = 
        { a = 27;

          # Nix is lazy, so this can happily keep
          # recursing  as `b` is never requested,
          # only `a`, which is now ready.
          b = ( (self: ... (self: ... ( ...))).a
        }.a
  }

1. Question 1

What does it mean by the following? Could you explain that in terms of the example in 17.3.1?

We’ve seen how to override attributes in a set such that they get recursively picked by dependant attributes.

To recap:

fix = f: let x = f x; in x;
f = self: { a = 27; b = self.a; }

fix f    #=> { a = 27; b = 27; }

1.1 Overriding the output

Taking the first newpkgs example in section 17.3.1 and re-writing it using f and a set to update the initial attribute a:

let newpkgs = pkgs (newpkgs // overrides); in newpkgs
let newpkgs =   f  (newpkgs // { a = 9; };  in newpkgs =
                -----------┃------------         ∧
                           ┗━━━━━━━━━━━━━━━━━━━━━┛

     ┏━━━━━━━━━━━━━━━━━━┓
     ┃                  ∨
= (self: { a = 27; b = self.a; }) (newpkgs // { a = 9; }) = 
   --∧----------f---------------   ----------┃----------
     ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

                    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                    ∨                             ┃                  
= { a = 27; b = (newpkgs // { a = 9; }).a; } = newpkgs =

= { a = 27; b = ({ a = 27; b = (newpkgs // { a = 9; }).a; } // { a = 9; }).a; } =
                 ---------------newpkgs--------------------
  ------------------------------newpkgs----------------------------------------                  
= { a = 27; b = ({ a = 27; b = ... } // { a = 9; }).a; } =
                 ----newpkgs--------
= { a = 27; b = ({ a = 9; b = ... }).a; } =
                 ----newpkgs-------
= { a = 27; b = 9; }

1.2 Overriding both input & output

Adding explicit parentheses to the second newpkgs example in section 17.3.1 makes it easier to see that it simply updates the resulting attribute set with the same “overriding” attribute set:

let newpkgs = pkgs (newpkgs // overrides); in newpkgs // overrides
let newpkgs =   f  (newpkgs // { a = 9; };  in newpkgs   // { a = 9; } =
            = ( f  (newpkgs // { a = 9; };  in newpkgs ) // { a = 9; } =
            # ...reduction steps in section 1.1 above...
            = { a = 27; b = 9; } // { a = 9; } =
            = { a = 9; b = 9; }

1.3 “Overridable” `fix`es

newpkgs can be generalized as an “overridable” fix:

overridable_fix  = f: overrides: let x = f (x // overrides); in x
overridable_fix' = f: overrides: overridable_fix f overrides // overrides

override = { a = 9; }
overridable_fix  f override   #=> { a = 27; b = 9; }
overridable_fix' f override   #=> { a = 9;  b = 9; }

2. Remaining questions

TODO: These are mostly notes for myself to figure out the answers, because I’ve been asking the same things for years.

What does it mean by “nixpkgs returns a fixed point of the package set”?

Not sure how accurate this is, but I imagine Nixpkgs’ “fixed points” the same way as in math where the invariant points are fully resolved Nix expressions that only need to be realized.
Isn’t a fixed point something belonging to a function, and the package set isn’t a function?

I remember reading somewhere that package sets are functions, but not sure, so leaving these here:
- NixOS Discourse What are package sets?
- Reddit Fully setting up a custom private Nix repository
- Sander van der Burg Managing private Nix packages outside the Nixpkgs tree
- github gist more package set links
- Why does nixpkgs return a fixed point? Is nixpkgs something similar to the example let newpkgs = pkgs (newpkgs // overrides); in newpkgs in 17.3.1.?
(Yeah, I would love to see high-level architectural diagrams on how Nixpkgs relates to concepts such as fixed points.)
What does it mean by “packageOverrides is used to inject the overrides”?

Again, have no answer, but the Nixpkgs issue #43266 “Deprecate packageOverrides?” has tons of info.
Why “Then pkgs.asciidoc-full is a derivation that has graphviz input”?
Why “The newly built asciidoc will depend on the new graphviz”, and “old asciidoc will keep using the old graphviz undisturbed”?