Best practices with `with`

Is this advice sane:

If you’re unsure which with within an expression with with with nested layers works with which context, it might be time to consider whether using with with any scope is necessary at all—with apologies for the excessive use of with in this sentence, which, much like excessive with in code, leaves you with confusion about which with aligns with which part.

8 Likes

Haha. Yes, with is footgun land. Not just because of nested with expressions, but just scoping in general when a binding is already available - with generally has lower precedence.

I personally stick to avoiding with as much as possible; if I use it, it’s generally only in nixpkgs, and it’s constrained to a single-line expression. let inherit ... in is clearer for me, and more importantly, it’s easier for tooling to reason about.

See Best practices — nix.dev documentation for more.

Actually we even found some footguns in nixpkgs about with - Documentation: style guide for use of `with` · Issue #292468 · NixOS/nixpkgs · GitHub

8 Likes

I wanted to post this in Meta, but it would have been the wrong kind of meta.

3 Likes

By the way, nixd warns about some with related problem.

I like this overall litmus test. I want to gauge people’s opinions on specific cases where with is commonly used within the Nix ecosystem.

1 - Ban with from being used at all in nixpkgs

I’ve seen this opinion shared a few times. Much of this has centered around a previous RFC, which seems to have unfortunately stalled.

2 - meta = with lib; {

This is what I see the most agreement on - no.

3 - maintainers = with lib.maintainers; [

I’ll group other small-level with uses into this, such as with lib.licenses;.

4 - environment.systemPackages = with pkgs; [

Some quick Github statistics - this is used double as much as environment.systemPackages; = [, without the use of with. Here’s the search without with, and the search including with.

5 - multiple, separate with lists merged together at the end

This is something I’ve used a few times within my personal configuration, yet have never seen brought up in the with debates. Here’s an example:

let
  stablePackages = with pkgs; [
    git
    vim
    etc
  ];
  
  unstablePackages = with pkgs-unstable; [
    vscode
    firefox
  ];
in
{
  environment.systemPackages = stablePackages ++ unstablePackages;
}

If you have strong opinions on why/why not any of these should be used, please make a reply explaining why! I have my personal opinions on each of these, but I want to start by hearing what other people think.

3 Likes

Btw @lucasew has been working on introduce npv_169 to detect top level withs by lucasew · Pull Request #142 · NixOS/nixpkgs-vet · GitHub which would disallow introducing new instances of certain uses of with in Nixpkgs, but we still need a bit more consensus as to what the exact check should be. So this is a great thread for this :slight_smile:

4 Likes

To be fair, I understand the problem that people solve using with. It’s pretty awkward to type a long attr path tens of times in one list. I have even grudgingly resorted to with myself there even if I feel the language would be better without.

But now I started to wonder why I haven’t done instead

let p = pkgs.pythonPkgs.some.really.long.attrpath; in [ p.hello p.bye ].

Actually, I suspect, but this is hugely subjective and may be completely wrong even about my own motivations, the reason may be that Nix language constructs in some vague way “feel” heavy, perhaps because I invariably need to produce lines full of closing parentheses, brackets, braces and semicolons in a seemingly random order.

This may be a matter of just getting used to it, but no way am I as enticed to use an if-else expression in Nix than in, say, Haskell or Python (or in C for the ternary operator, although I’ve long since banished that as a choice I allow for myself).

Now if this is so, I think that’s just sad, because I cannot imagine a good solution for this. The language is what it is. I assume all the semicolons are also necessary to resolve ambiguities…

I wonder if it isn’t a bit of a newbie trap? I don’t recall groking inherit until a few years in; a much preferred method, but I do prefer when things are explicit in the more complex Nix code bases I now write.

with shouldn’t be used just as from lib import * shouldn’t be used in Python.

2 Likes

That’s exactly how I think.

with is useful to avoid repetition in, for example, lists of packages and maintainers.

But there are a lot of places in nixpkgs that there is basically a with lib; at the top of the file.

And, in my experience, this is the kind of stuff that easily introduce sneaky bugs which static analysis such as nil will not detect. If there is a open with somewhere nil will just refuse to report anything related to symbol existence deeper in the tree.

BTW I want to introduce nil itself somehow in nixpkgs, there is stuff in there that people just forget because it’s not caught, like version used in meta.changelog, which is not defined because some day that package was rewriten in finalAttrs form and forgot to change the changelog but CI was green so nobody realised. That would be caught in static analysis.

1 Like

nixf-tidy was part of CI already, perhaps you could work with that?

with can be used in certain cases to make things more readable:

type = with types; either int str

Apart from that and package lists, it’s kind of a footgun.

5 Likes

Would a way to define these cases be those where you only access symbols from inside the scope (i.e. either, int and str must all come from types)?

2 Likes

And it shouldn’t be more than one expression IMO.

But yes, nixf-tidy used to check for this exact criterion (escaping-with). Though it was deemed too strict at least in nixpkgs.

I like that. @waffle8946: Maybe it could work like the formatter, where new files need to be linted. FWIW, as my Nix experience has grown, I’ve tried to turn my code style a lot more into declaring what I need upfront, and providing an easy path there for new contributors seems like a good idea.

1 Like

Hmm, I still like putting a with lib.types on top of a whole options block, though. Typing that for every type in a big, but relatively trivial module gets old fast - and for that you still need the cfg or config bindings, and sometimes more.

Completions from nixd or similar can help, of course, but I’m rarely a fan of things that add enough boilerplate that you almost need fancy editor features.

Maybe I just have to cope in the name of progress, but I did want to raise that.

I’d argue that going back to putting a with expr scoped around a fairly large attrset would make it harder to use tooling (because the tooling doesn’t know whether lib.types contains the various bindings you end up using). So tooling okays it, but then you go to build and get an avoidable eval error. And usually defining options is hundreds of lines per module of boilerplate anyway :smile:

I agree that writing out types would get old fast, which is why I personally go with inherit (lib.types) foo bar baz; at the top; for me that’s the simple compromise that doesn’t hinder tooling, since you’re explicitly specifying which bindings to use.

4 Likes

That also has the advantage of looking like languages that have an import feature, if it’s mostly toward the top of the file. Nix may be purely functional, but lots of people understand dependency injection, even if there’s a function wrapping the actual injection.

(This is incidentally how I teach it to coworkers: if you structure your code well using callPackage and such it’s mostly just a fancy dependency injection framework that happens to have the world’s most powerful build system underneath).

1 Like

with is only a problem if it causes ambiguities. If there is no ambiguity then there is no problem.

So far the semantic of with has been a non-issue for evaluation purposes. a NixOS module can make use of with lib; while later making use of with types; or with pkgs;.

The problem of with comes with partial evaluation. With a bit of experience comes the knowledge that there is no attribute named types, mkIf or mkOption within pkgs.

Maybe we could implement some rules remove these ambiguities. One suggestion would be that we should have a single with scope, making the last with hide all names from the previous one.

Thus with lib; with types; should only leave names from lib.types but no longer names from lib. The same goes for with lib; with pkgs; should only leave names from pkgs but none from lib.

Note, this new rules, despite changing the semantic does not change the capabilities of with, as the previous can be rewritten as with lib // types; or with lib // pkgs;, making it explicit that both sets are of interest for unbound name lookup and that names of the later are preferred in case they appear in both.

It also breaks tooling to some degree, unless the tooling itself embeds a nix evaluation step.

This would not be implemented, since that would break eval of older expressions.
The only option would be a new syntax, such as:

Although, in the case of package lists, there’s an alternative option in the works that would make it more ergonomic to use the inherit syntax, since you wouldn’t need to slap builtins.attrValues in front of each definition:

And in any case, it should be clear that “ambiguity” or not, with lib; at the top of a file is a massive antipattern.

2 Likes