2022-11-14 Nixpkgs Architecture Team Meeting #17

Development Nixpkgs Architecture
1

The Nixpkgs Architecture Team will have the next meeting on Monday, 2022-11-14 16:30 CET. The meeting is open for anybody to join.

Tentative Agenda

Auto-called packages RFC issues

In order to be more focused on a single issue, we will try out a new agenda format of going through the issues of the RFC that is currently being worked on: Issues · nixpkgs-architecture/simple-package-paths · GitHub

1 Like
2

Notes

intro

Auto-called packages RFC draft review

proposal: package.nix instead of default.nix, https://github.com/nixpkgs-architecture/auto-called-packages/pull/8

  • Benefit: Can’t accidentally nix-build these paths anymore

    • Leaves room open for a default.nix that can be nix-build’t

  • @Ericson2314:

    • File paths are an implementation detail
    • We can do extra migrations out the kindness of our heart.
    • But we shoudln’t feel “boxed in” by this. If we can’t do as good of a “bonus” thing for downstream coe snooping at unstable file layouts, that’s fine.

  • @roberth: PR’s need to be rebased, lots of them

    • @Ericson2314: PR’s change the implementation, is expected. Tool to automate PR rebases has a lot of flexibilities

  • zooming out, the choice is between

    • package-function/<name>
      • new stuff would be foo/<name>, bar/<name>.
    • <name>/package-function
      • new stuff would be <name>/foo <name>/bar -

  • Concretely:

    • Example: pkgs/auto/hello/package-function.nix
    • Example: auto/hello/package-function.nix ?

  • @Ericson2314: directory should imply structure; avoid exceptions (+1)

    • What do do with bootstrap, build-support, etc?
    • What to do with fetchgit (things that are libraries are not exactly packages)
    • lib?
    • @tomberek: dir structure should be a close match to attr structure
    • during transition will have new/old separation
    • @infinisil: Proposal: Change the RFC to only do the transition for packages, not functions like fetchers and other special values
    • @tomberek: The directory structure / attribute structure mismatch is currently confusing. Might be better to keep it the same

  • might be controversial

  • @growpotkin: Could use default.nix for filling out non-default arguments

    • @roberth: Doesn’t scale to nixpkgs’ scale. Can have default.nix be something like (import ../.. {}).hello instead though
    • @Ericson2314:
      • Agreed we should save default.nix for things that are not footguns that cannot be called. A good general rule.
      • A bit wary about this specific default.nix, because definitions should not know their own name.
      • But perhaps it is less bad because the package directories were already supposed to be 1-1 with attribute names.

File name proposals:

  • @roberth: Something with “fun”/“functions”

  • pkg-fun.nix, same length as default.nix :slight_smile:

  • Other ideas:

    • package-function.nix
    • package-fun.nix
    • package.nix
    • proto-derivation.nix
    • prototype.nix
    • blueprint.nix
    • default.nix
    • pkg.nix

  • @Ericson2314: Naming details shouldn’t block the RFC

Directory root

In nixpkgs root:

  • pkg/<name>/pkg-fun.nix

  • component/<name>/pkg-fun.nix, too long

  • part/<name>/pkg-fun.nix

  • thing/<name>/pkg-fun.nix

  • mod/<name>/pkg-fun.nix, we have modules already

  • pkg/<name>/pkg-fun.nix

  • comp

  • uniform

  • unit

    • +1 @infinisil
    • +1 @growpotkin
    • +1 @roberth
    • @Ericson2314 proposed pkgs/unit/<name>/*.nix
      • @roberth: pkgs not necessary.
      • @roberth: Implementation should include rename tool, shouldn’t do many renames manually
      • @roberth: Doesn’t make sense to move nixos modules into pkgs
      • Not disagreeing with pkgs/unit, for now only for packages
    • unit/<name>/*.nix
    • @infinisil: Just a name
    • @Ericson2314: Good to think ahead, less ossifying, can change things later as well. We’re trying to think ahead, but not set in stone.
    • @growpotkin strictness is flexibility!
      • @Ericson3214: +90000
    • @infinisil: Encouraging more evolution

@growpotkin’s sharding experiment

  • Ran into some performance issues with readDir.
  • Automatically generate single file with all paths, much faster.
    • @infinisil: builtins.listDir could be faster
      • @roberth: This is an implementation detail (and readDir attrValues can be made lazy)
  • 8000 nodejs packages, sharding node packages by scope + base name, plus one-letter sharding for unscoped
  • @Ericson2314: Attribute union should be faster in Nix, currently linear
  • @Ericson2314: Fundamentally, we should do things like sharding because limitations in things outside our control, like the filesystem. When the implementation is in Nix itself we should just fix it.
    • My guess is that sharding is good for the filesystem (and for git’s content addressing), and if Nix doesn’t like it we should fix Nix to get along with it.
  • @growpotkin: Beware of case-insensitive file systems!

outcomes, todos

2 Likes
3

Kool! I really enjoy this!

Out of completness, I’d propose how Standard already calls the idea of a unit today, that’d be cell.

A Cell groups (domain-) related functionality together.

But if unit is chosen, I’m happy to change that naming. It’s the same basic organization principle.

1 Like
4

Personally I’ve always been confused by std’s naming of things :sweat_smile:. You saying that cell is like the unit here clears at least that up.

A Cell groups (domain-) related functionality together.

I think a better description is needed than that, it’s not clear what “related functionality” here is. Assuming we are actually talking about the same concept, I’d probably describe a “unit” for nixpkgs as:

A collection of standardized files related to the same software component

3 Likes
5

A collection of standardized files related to the same software component

Yep, that is pretty good, and in the context of nixpkgs, “software component” is actually the domain.

Overall, maybe the “domain” needs a tid more clarification, as well.

In my understanding, there are two domains:

  • horizontal (e.g. software component - oriented)
  • vertical (e.g. tooling - informed)

What we’re doing here (and also in std) is rotate the angle of aspect about 90° from vertical to horizontal.

In my opinion, this is extremely powerful since it caters the contribution intent of the domain expert. I’d honestly expect a measurable quality improvement coming out of this change.

Sidenote: std renamed some of its most fabolous naming, already :wink: