Recent breaking Nix change worthy of greater attention

1

I would like to call attention to this change log item in a recently merged Nix PR from @edolstra:

Selecting derivation outputs using the attribute selection syntax
(e.g. nixpkgs#glibc.dev) no longer works.

That means on master today:

nix --extra-experimental-features 'nix-command flakes' build 'nixpkgs#glibc.dev' --json | jq

[
  {
    "drvPath": "/nix/store/1xq9n2p2hlw7fbxzjc1sp96l9jz8s9gr-glibc-2.34-115.drv",
    "outputs": {
      // bin output, huh?
      "bin": "/nix/store/058drky7qcyd04rzqcmxh86xmifw96dx-glibc-2.34-115-bin"
    }
  }
]

as opposed to

[
  {
    "drvPath": "/nix/store/1xq9n2p2hlw7fbxzjc1sp96l9jz8s9gr-glibc-2.34-115.drv",
    "outputs": {
      // dev, matching CLI
      "dev": "/nix/store/pvn23vycg674bj6nypjcfyhqbr85rqxa-glibc-2.34-115-dev"
    }
  }
]

There is instead a new syntax ^<outputs> and ^* which allows selecting outputs.

Now, I have nothing against the new syntax, which is more powerful (since it allows selecting multiple outputs). But, the lack of .dev or .lib doing what people expect I consider an unforced error needlessly confusing to new users.

These issues were not surprises, but discussed in nix: Respect meta.outputsToInstall, and use all outputs by default by edolstra ¡ Pull Request #6426 ¡ NixOS/nix ¡ GitHub. I was about to call out @edolstra for ignoring those concerns, but rereading the nix: Respect meta.outputsToInstall, and use all outputs by default by edolstra ¡ Pull Request #6426 ¡ NixOS/nix ¡ GitHub I will give him a partial pass in that most of the discussion was over bikeshedding the details of the new feature (^, previously !) rather than debating whether/how to break the existing feature.

In particular, note there there is no technical reason we cannot keep the old feature via alternative means and also have the new feature.

In Nixpkgs today, we have

nix-repl> (stdenv.mkDerivation { name = "asdf"; outputs = [ "out" "dev" ]; }).outputSpecified
error: attribute 'outputSpecified' missing, at (string):1:1

nix-repl> (stdenv.mkDerivation { name = "asdf"; outputs = [ "out" "dev" ]; }).out.outputSpecified
true

This is used so lib.getFoo doesn’t “override” a manual choice of output. We could do outputSpecified in Nix itself and then use that to make foo.<output> work as intended. I pointed this out in the earlier PR thread to no avail.

I do no know how others think, but I wanted to spread word here to figure out.

(There is also an ambiguity/inconsistency in the new syntax relative to that in accepted RFC 92 that also got ignored, but this is a finer technical point and it is legitamate for changes slated to be stabilized sooner to preempt those that aren’t, so I do not wish to belabor that point in this thread.)

10 Likes
2

My two cents on the matter is that while breaking functionality is confusing to new users, keeping legacy compatibility is even more confusing down the line since it usually becomes rather unintuitive later (take for instance nix-shell -p and nix-shell '<nixpkgs>' -A for example). It also brings the burden of maintaining old syntax (which may become an obstacle for further improvement).

I don’t have any opinion on the mentioned syntax change, If anything I find the new syntax more clear. I do however urge nix to provide a rather useful error message when using ‘well-known’ outputs as attributes (a message like The specified attribute (foo.dev) is not a derivation, if you’re refering to a derivation output please use the foo^dev syntax), specially on well-known outputs (such as dev, out, man, …).

4 Likes
3

What’s the problem with the legacy syntax? I don’t see any benefit of the new syntax, except being more confusing.

1 Like
4

Having nixpkgs#glibc.dev output the bin output makes absolutely no sense. If it’s not going to output the dev output then it shouldn’t be allowed on the CLI in the first place. I don’t particular care how it’s handled so long as it isn’t something near-to-but-decidedly-not what the user intended.

8 Likes
5

What’s the problem with the legacy syntax?

It doesn’t differentiate between attribute and output.

1 Like
6

The old approach had a lot of problems:

  • nix-env ignores the user’s output attribute selection (as in nix-env -iA foo.bin) entirely if the derivation has a meta.outputsToInstall attribute.
  • nix-env and nix-build have different behaviour.
  • There was no way to select multiple outputs except by specifying a package multiple times (nix-env -iA foo.man foo.bin), but that results in multiple profile elements that could get out of sync during upgrades.

These issues are now fixed in the new CLI. Also, the .all syntax never worked in the new CLI, but you can now use ^* for that.

Arguably the ^ syntax should be added to the Nix language (so you would write foo^bin instead of foo.bin). That would allow us to get rid of the very expensive duplication of derivation attrsets for each output.

6 Likes
7

Thank you, that’s all good. That doesn’t explain why nixpkgs#glibc.dev should output the bin output. That seems wrong.

1 Like
8

Perhaps glibc’s meta.outputsToInstall should be adjusted for that?

9

Yes, we can’t misdirect by acting like the old and new ways are mutually exclusive!

10

Because the .dev output, just like the default output, contains a meta.outputsToInstall field, so it behaves just the same.

11

But there is no reason we cannot do the outputSpecified selected trick. (Few new readers, make foo and foo.out different by setting this field so we can tell the difference between.) Since Nixpkgs does it already and everyone is using stdenv.mkDerivation there is no practical overhead.

I pointed these out and you ignored it.

12

Yea I’d rather just get an error than this behavior. dev should output dev or the eval should die. Anything else is just going to be maddening. I’m sure that without this thread, I would have found myself scratching my head for a good while when I inevitably encounter this.

5 Likes
13

Maybe it’s worth keeping in mind that there are more than just than the two options “keep the old syntax forever” and “roll out new, breaking syntax immediately”. One good option is:

  1. Warn on usage of old syntax, link to an explanation, introduce new syntax optionally
  2. Deprecate old syntax with error
  3. Once the detection of the old syntax (needed for the specific deprecation message) imposes too much maintenance burden, just remove it.

Each step is one release. This 3-step pattern is well known in software maintenance, and maybe this kind of change would have profited from it.

6 Likes
14

That is true, but there still is no reason we couldn’t have foo.out for easy cases, and foo^outs,dev,man for advanced cases. We should be having separate discussions of

  1. Do we want the new syntax (probably!)
  2. Do we want the old syntax short term (definitely! Probably. Definitely it should either not work at all, or have semantics matching Nix expressions.)
  3. Do want the the old syntax long term (open to debate)

Also, as @ElvishJerricco said, if we do get rid of the old syntax that should mean foo.dev stops working, not that it silently just ignores the .dev part.

5 Likes
15

Why definitely? I understand that some automation could break, but keeping both around will be confusing.

(Also, AFAIU this is still an experimental feature: If there’s a moment to break it, it’s before it becomes stable.)

1 Like
16

OK Fair enough.

Because . will be allowed for records in general, I find it odd removing it from the CLI unless we also get rid of foo.dev in expressions in general, which is large breaking change. But I will grant that making foo.dev not work in the CLI might be odd, but at least avoids the massive-foot gun.

Edited the above.

1 Like