Removing most uses of top-level `with`

Across my career, I’ve enjoyed doing tree-wide refactors that move the state of the world further just a little bit. With top-level with statements, there’s an opportunity to do that in nixpkgs , so I’ve assigned myself the following issue.

As nix.dev explains in its “Best Practices” section:

There are a number of problems with using with at the top level of a Nix file:

  • Static analysis can’t reason about the code, because it would have to actually evaluate this file to see which names are in scope.
  • When more than one with used, it’s not clear anymore where the names are coming from.
  • Scoping rules for with are not intuitive, see this Nix issue for details.

Therefore, do not use with at the top of a Nix file. Explicitly assign names in a let expression.

Example:

let
 pkgs = import <nixpkgs> {};
 inherit (pkgs) curl jq;
in

To implement this refactor, we need to do two things:

  1. Determine the set of names that were pulled into scope with the with.
  2. Add those names in a let binding at the same place as the with.

For lib, that list is 424 items long.

So, I made a small Bash script to help with doing this.

Here’s an example PR that I made using this tool.

14 Likes

Awesome, this would be a huge improvement in ease of readability! I vividly remember being thoroughly confused where names come from, and that made Nixpkgs feel like way too much magic.

6 Likes

I’m glad you agree!

My commitment is to produce patches which are pure refactors, with no behavior changes. What will kill this effort is nit-picking to try to get to the smallest possible set of changes in each file, or “one-more-thinging”.

In particular, I’m going to unify lib.something into just something if there’s an inherit (lib) something up top, as this is both more performant (which usually doesn’t matter) and is more clear that this is the same symbol (which usually does matter for readability.)

My call to action for you (and any other Nixpkgs committers) is that I could use reviews and merges on the flurry of PRs to come. :slight_smile:

1 Like

The script is neat, but do make sure that scope checking errors aren’t masked by a different with, which could be anywhere in the file, disabling scope checking in its particular subtree.
Checking that by hand (using find in $EDITOR) might be the most effective way to do that.

1 Like

masked by a different with, which could be anywhere in the file, disabling scope checking in its particular subtree.

Don’t I know it. My workflow relies on having vim open with (at least) two panes, one of which is running refactor-top-level-with-away.sh, and the other one has the file being edited open.

I try to do the following, and any advice you have on gotchas is welcome.

  1. Read the file for other with statements. As you mention, that’s a key way to accidentally introduce a semantic change. Thankfully I find there’s not too much shadowing of the lib names, and lib is far and away the top contributor to top-level with. I also look for rec uses, though I’m less clear on what bad looks like there.

  2. Look through the list of inherit statements for false positives. I tried to make the script susceptible to false positives and immune to false negatives. Since the script thinks that every word in a string is a possible identifier, it’ll bring those into scope. Usually this is something like and, all, any, or, modules, optional or the like. I haven’t found a place yet in nixpkgs where this extra symbol does damage: it’s just … extra. But let’s have it gone if we see it.

  3. Run any appropriate tests on the file.

  4. Hit enter and take the commit.

2 Likes

Maintaining such a list is yet another chore. We should rather optimise the interpreter to be more efficient about this.

5 Likes