produces 1 as a result. So the way scoping in the nix language works is non local in that sense.
This is actually documented here. Now while for the reasons explained in rfc 110, this does allow some more static analysis on nix code compared to the scoping rules one would expect (innermost bound variables first), I personally feel strongly, that it is very unfortunate design.
As suggested in the rfc thread, I opened this issue to discuss this further.
So in particular this means that it is impossible to use with-bindings in library code that is used by another party as a black box, since the calling code could always bind variables that then silently change the semantics of the library code.
right thanks for pointing that out!
That makes things slightly less bad, then its just an issue of not being able to analyze a piece of code without looking at the whole file…
I agree that this is very counter-intuitive. Putting the problem into other words: with is the only type of expression that’s context-sensitive even without unbound variables. This means that even if you have a fully-working with expression, where all variables resolve and can be evaluated like this:
pkgs: with pkgs; [ foo bar ]
Which evaluates to pkgs: [ pkgs.foo pkgs.bar ]. But once you nest that expression in another, the result can change! E.g.
let foo = "foo";
in pkgs: with pkgs; [ foo bar ];
Which evaluates to pkgs: [ "foo" pkgs.bar ] instead!
This is problematic because:
When you encounter a with, you need to know the context to know how it evaluates
You need to make sure that you aren’t accidentally introducing variables used by the with expression. Or in other words: You can declare new/fresh variables without even using them, but still change the evaluation result!
Or here’s another way to look at it: Variables used in with expressions are neither free variables (because they don’t need to be declared) nor bound variables (because it uses a declared variable if it exists), they are something inbetween, I’d maybe call that “opportunistic variables”. This makes it hard to understand since most languages don’t have such a concept (reasonably so!).
While I agree with the statement that this is confusing and unusual behavior, I actually like it.
It avoids scenarios in which a definition I don’t know about may cause issues, since anything given an explicit name within the file I can see has precedence. This solves a lot of potential mistakes that make these “import everything” statements considered bad practice in other languages, because a change to code elsewhere can’t suddenly override what is immediately visible.
It’s of course still confusing because the user probably doesn’t expect this behavior, and these styles of generic import statements probably just shouldn’t be a thing in any language. But I think nix’ with is one of the nicest.
Specifically, in the with pkgs; example, code can never break because a new attribute is added to pkgs. This is crucial and therefore I think that the current design of with is strictly better than the alternative.
Additionally, being to resolve some of the scope statically is good from an implementor’s perspective, since it allows for a certain amount of optimization even in the face of with (which can only perform so well). You can always infer statically what will be resolved dynamically and what statically.
(In the end: What’s done is done and we need to stay backwards compatible. If you don’t like with, it is very possible to write code that doesn’t require it, at the expense of a few let inherit (…) …; or leading attribute paths written. I am also not really convinced by any of the proposed alternatives.)
The current system is definitely the lesser of the two evils. It’s unintuitive, but what it prevents is much worse.
The potential for the binding structure to change in files you haven’t even modified is limited to multi-with situations and thus much more manageable.
I agree that this is intuitive however I do think it is better than the alternative. If you have code like:
let
mypkg = import ./mypackage;
in with pkgs; [ vim mypkg ]
The alternative version of with would break if someone added mypkg to nixpkgs. I think this is a huge footgun that it makes sense to avoid. Basically it is balancing the weight of this potential breakage with the potential breakage of adding a new local. However since the latter is at least local to the file it seems like a smaller footgun and therefore the better.
But I agree that both modes are bad. I think the best solution is to deprecate and stop using with entirely. It simply has too many footguns. I think the mentioned RFC is a great alternative to this very common use of with so I think that is the way to go. As we remove the use cases with will become less common and maybe can eventually be removed.
Why do we need to deprecate with in order to stop using it in contexts that does more harm than good? Why can’t we just expect people to understand its down sides? Nix is mainly a configuration language, but it’s also often used in one-liner expressions in the command line. I think with is a net positive when used as:
nix-shell -p 'python3.withPackages (p: with p; [ pandas requests x y z ])'
This actually makes me wish we had with-lambdas like with: E which means p: with p; E so that my nix-shell becomes:
nix-shell -p `python3.withPackages (with: [ pandas requests x y z ])
We don’t have to deprecate it. But providing alternatives to features that are both unintuitive and have significant footguns seems like a big benefit to me. If people can use a more clear and more reliably syntax that seems like a win.
The example you gave would actually be a bit shorter with the new syntax anyways:
nix-shell -p 'python3.withPackages (p: p.[ pandas requests x y z ])'
But that is tangential anyways. I don’t think anyone is planning on taking with away. Some people have expressed the desire to eventually ban it in nixpkgs but that won’t stop your own derivations or shell commands from using with.