Understanding how mkDerivation attributes are evaluated when using overrideAttrs

Help
1

I need to override the Nixpkgs PulseAudio version and thought I’d try learning overlays in the same go. I started out with this:

(self: super:
  {
    pulseaudio = super.pulseaudio.overrideAttrs (_: rec {
      version = "13.99";

      src = super.fetchurl {
        url = "http://freedesktop.org/software/pulseaudio/releases/pulseaudio-${version}.1.tar.xz";
        sha256 = "030a7v0khp6w683km81c6vpch1687pvx2gvscnzkjq4f0z6138g6";
      };
    });
  }
)

The relevant parts of the recipe whose derivation I’m overriding is:

{ (...) libOnly ? false (...) }:

stdenv.mkDerivation rec {
  name = "${if libOnly then "lib" else ""}pulseaudio-${version}";
  version = "13.0";

  src = fetchurl {
    url = "http://freedesktop.org/software/pulseaudio/releases/pulseaudio-${version}.tar.xz";
    sha256 = "0mw0ybrqj7hvf8lqs5gjzip464hfnixw453lr0mqzlng3b5266wn";
  };

  (...)

  postInstall = lib.optionalString libOnly ''
    rm -rf $out/{bin,share,etc,lib/{pulse-*,systemd}}
    sed 's|-lltdl|-L${libtool.lib}/lib -lltdl|' -i $out/lib/pulseaudio/libpulsecore-${version}.la
  ''
    + (...)

From what I gather, overrideAttrs produces a recipe with a new attribute set that mkDerivation will evaluate when called. That is, any non-overridden attribute that refers to an overridden attribute - such as version in my example - will use the overridden attribute when evaluated. This is as opposed to overrideDerivation, which results in mkDerivation being evaluated before applying the overridden attributes.

But with my first overlay attempt, postInstall still uses the old version attribute, and therefore building it with libOnly = true fails. So I tried adding this line:

postInstall = super.pulseaudio.postInstall;

This works, which makes me think that super.pulseaudio.postInstall is “inlined” when I declare it this way. Therefore, I would expect the same to be true for name when I add this:

name = super.pulseaudio.name;

But it isn’t - the resulting derivation, although fully functional with the overridden version, has an output path that still uses the old version name. (I realize that pname solves this, but I can only work with the original recipe as it is.)

What am I not understanding here? Why does it work for postInstall (and is that an idiomatic way to do it?) but not name, and how am I reading the Nixpkgs manual wrong when it comes to overrideAttrs?

2 Likes
2

This is not correct.

In the original definition it said

stdenv.mkDerivation rec {
  name = "${if libOnly then "lib" else ""}pulseaudio-${version}";
  version = "13.0";
  …
}

The reference to version in the name attribute isn’t some recursive lookup in the derivation. The rec keyword makes it so all definitions in the attribute set are available to other expressions within the same set. It’s actually equivalent to writing

let value = "13.0"; in
stdenv.mkDerivation {
  name = "${if libOnly then "lib" else ""}pulseaudio-${value}";
  version = value;
  …
}

The use of overrideAttrs can change the version attribute in the attribute set ultimately passed to mkDerivation, but it doesn’t affect anything else that referenced the same attribute via the rec. In order to do that you have to override those attributes too. And in this case it means you need to copy the postInstall as well.

In order to do what you had expected the original derivation would have had to be written like

let drv = stdenv.mkDerivation {
  name = "${if libOnly then "lib" else ""}pulseaudio-${drv.version}";
  version = "13.0";

  …

  postInstall = lib.optionalString libOnly ''
    rm -rf $out/{bin,share,etc,lib/{pulse-*,systemd}}
    sed 's|-lltdl|-L${libtool.lib}/lib -lltdl|' -i $out/lib/pulseaudio/libpulsecore-${drv.version}.la
  ''
    + …;
  …
};
in drv

but nobody writes packages this way.

This doesn’t do anything useful. I can’t tell you why it appears to work for you without being able to play with it myself. Are you sure you tested with libOnly = true after adding that?

3

To elaborate a bit more on rec { … }, it basically acts as though every defined attribute were part of a let binding (as multiple bindings within the same let are mutually recursive). By that I mean

rec {
  a = c + 1;
  b = a + 1;
  c = 3;
}

evaluates the same way as

let
  a = c + 1;
  b = a + 1;
  c = 3;
in {
  inherit a b c;
}

Hopefully this formulation makes it more obvious that overriding c in the attribute set won’t affect a or b.

Also FWIW you can think of it the opposite way too, where a let binding is equivalent to a with rec { … } expression, though I find the other way more helpful to think about.

1 Like
4

This is also why most people override the version and src when doing overrideAttrs, as src will still point to the original url.

5

To elaborate a bit more on rec { … } , it basically acts as though every defined attribute were part of a let binding (as multiple bindings within the same let are mutually recursive).

Thanks, that makes it clear. I see that

rec { a = 1; b = a; } // { a = 2; }

evaluates to { a = 2; b = 1; }, when I was thinking that it would result in { a = 2; b = 2; }.

(I also realize that the distinction between overrideAttrs and overrideDerivation is about overriding attributes before and after mkDerivation is called, respectively - far from what I initially thought.)

As for the postInstall thing, doing

nix-build -E 'let pkgs = import <nixpkgs> {}; in pkgs.callPackage pkgs.pulseaudio.override { libOnly = true; }'

with postInstall = super.pulseaudio.postInstall in my overlay does make it build successfully. I fully see your point, and I can’t explain that.

For completeness, this is the overlay that I ended up with:

(self: super:
  {
    pulseaudio = super.callPackage
      ({ libOnly ? false }:

      super.pulseaudio.overrideAttrs (_: rec {
        name = "${if libOnly then "lib" else ""}pulseaudio-${version}";
        version = "13.99";

        src = super.fetchurl {
          url = "http://freedesktop.org/software/pulseaudio/releases/pulseaudio-${version}.1.tar.xz";
          sha256 = "030a7v0khp6w683km81c6vpch1687pvx2gvscnzkjq4f0z6138g6";
        };

        postInstall = super.lib.optionalString libOnly ''
          rm -rf $out/{bin,share,etc,lib/{pulse-*,systemd}}
          sed 's|-lltdl|-L${super.libtool.lib}/lib -lltdl|' -i $out/lib/pulseaudio/libpulsecore-${version}.la
        ''
          + ''
          moveToOutput lib/cmake "$dev"
          rm -f $out/bin/qpaeq # this is packaged by the "qpaeq" package now, because of missing deps
        '';
      }))
    {};
  }
)
6

Your overlay is wrapping the derivation in another callPackage, which itself is overridable. So when you override it in your nix-build you’re only overriding the libOnly flag that exists in your overlay, not overriding the libOnly flag used to evaluate super.pulseaudio. Perhaps if you try something like

(super.pulseaudio.override { inherit libOnly; }).overrideAttrs (_: rec {
  …
})
7

Sorry, I should have been clear about which overlay I was referring to. With

(self: super:
  {
    pulseaudio = super.pulseaudio.overrideAttrs (_: rec {
      version = "13.99";

      src = super.fetchurl {
        url = "http://freedesktop.org/software/pulseaudio/releases/pulseaudio-${version}.1.tar.xz";
        sha256 = "030a7v0khp6w683km81c6vpch1687pvx2gvscnzkjq4f0z6138g6";
      };  

      postInstall = super.pulseaudio.postInstall;
    }); 
  }
)

(the one from my initial post), the nix-build command succeeds.

So when you override it in your nix-build you’re only overriding the libOnly flag that exists in your overlay, not overriding the libOnly flag used to evaluate super.pulseaudio .

I’m confused - once I declare an overlay for the derivation, is it not the overridden pulseaudio that I’m referring to in pkgs? And when it is passed a libOnly flag, be that from my nix-build command or any from other package in all-packages.nix, is it not evaluated as such?

I tried the super.pulseaudio.override { inherit libOnly; } solution that you suggested, but the libOnly argument isn’t found when referenced from the attributes in the overridden derivation. Generally, in an overlay, how does one access the input arguments to the original recipe?

8

There’s a detailed write up on the difference located here, nixpkgs manual

9

So when you override it in your nix-build you’re only overriding the libOnly flag that exists in your overlay, not overriding the libOnly flag used to evaluate super.pulseaudio .

Okay, I think I understand what you mean now. Like you said (pretty clearly), I’m wrapping my overridden derivation in a new function with a new argument (that might as well be called onlyLib or whatever, provided I use that name in the overlay too), and using callPackage to call that function. The reason it works is that the new libOnly is used by overrideAttrs, and the other libOnly is never used since all the attributes that need it are overridden.

However, since I’m wrapping the original package with a function, pkgs.pulseaudio is now a function that only takes libOnly and I’ve locked myself out of overriding any of the arguments to the original function. E.g.

nix-build -E '(import <nixpkgs> {}).pulseaudio.override { x11Support = true; }'

fails because I try to call my wrapped function with an unexpected argument.

Just needed to say that out loud.

10

Thanks for clearing this up for me! Still struggling with how to get the overlay right, but figured I was getting off topic, so I followed it up in How to use input arguments inside overrideAttrs.