2022-11-21 Nixpkgs Architecture Team Meeting #18

[Infinisil]

Nixpkgs Architecture Team Meeting #18

• Past meeting notes
• Recording
• Matrix announcement
• Recorded by: @infinisil
• Lead by: @infinisil
• Meeting notes by: @Ericson2314
• Present members: @infinisil @Ericson2314 @growpotkin

Notes

• @infinisil shows Add script for transition by infinisil · Pull Request #15 · nixpkgs-architecture/simple-package-paths · GitHub

  • merge it!

• Explain what `unit` is · Issue #16 · nixpkgs-architecture/simple-package-paths · GitHub

  • agree to close as fixed / no action needed (will be in RFC)

• @Ericson2314: src.nix can be separate from pkg-fun.nix

  • @growpotkin: Yes yes yes

• Implementation: Fix references · Issue #14 · nixpkgs-architecture/simple-package-paths · GitHub

  • relative path …/…/…
    • agree to try to fix in script
  • Various issues with multiple versions
    • @Ericson2314: let’s just skip packages with multiple versions for now
      • wary of having foo and foo123 units
      • wary of unit/foo but foo = foo6
    • @infinisil: OK sounds good

• What about `CODEOWNERS` · Issue #9 · nixpkgs-architecture/simple-package-paths · GitHub

  • @infinisil: let’s just make package sets within nixpkgs out of scope
    • @Ericson2314: agreed
  • @growpotkin: but if individual packages are in CODEOWNERS we should upgrade
    • all: agreed

• Concerns over `readDir` speed · Issue #17 · nixpkgs-architecture/simple-package-paths · GitHub read dir perf

  • Improve `readDir` performance · Issue #7314 · NixOS/nix · GitHub related nix issue

  • getdents is also on FreeBSD, maybe macOS, not just linux.

  • O(…) thinking:

    x is disk access time
    y is memory allocation time
    n is the number of top-level attributes

    Before: O(x * stat + y * n)
    After: O(x * stat * n + y * n)
    After (with OS support): O(x * stat + y * n)
    After (with lazy readDir): O(x * stat + y * n)

• To do sharding or not · Issue #1 · nixpkgs-architecture/simple-package-paths · GitHub

  • let’s shard
  • @growpotkin @Ericson2314: Should be fixed-prefix, makes it much easier

    • 2-prefix, li has 1500 entries

      • 2/2-prefix, libpng → li/bp/libpng/pkg-fun.nix
      • lzma → lz/ma/lzma/pkg-fun.nix, what about packages with the same prefix?
      • lzmalzma → lz/ma/lzmalzma/pkg-fun.nix
      • lz → lz/lz/pkg-fun.nix
      • lz → lz/_unscoped/pkg-fun.nix
      • Single-letter packages: [ “R” “h” “j” “o” “q” “t” ]

    • a and a_b package attributes goes into a_ prefix

    • 1-letter packages x-/--/x/pkg-fun.nix

    • 2-letter packages xy/--/xy/pkg-fun.nix

    • 3-letter packages xy/z-/xyz/pkg-fun.nix

    • 4-letter packages xy/zw/xyzw/pkg-fun.nix

    • 5-letter packages xy/zw/xyzwa/pkg-fun.nix

    • 6-letter packages xy/zw/xyzwab/pkg-fun.nix

    • 7-letter packages xy/zw/xyzwabc/pkg-fun.nix

    • 6+ packges 5pkgs/xy/zw/ab/xyzwab/pkg-fun.nix

    • log_(26^2) 40,0000 ~= 4.6020599913

    • Largest directories by prefix length for top-level attributes

      • 1: 1606
      • 2: 1173
      • 3: 950 (just below GitHub’s 1000 limit, might go over with some more packages)
      • 4: 236 (comfortably below the 1000 limit)
        • haskellPackages: 202
        • pythonPackages: 182
        • perlPackages: 136

    • Nixpkgs has on the order of 10’000 packages per package set and 100’000 packages in total

    • 2/2 or 4 prefix?

      • With 4 prefix we have 536 prefix-directories

        • 1 packages x---/x/pkg-fun.nix

        • 2 packages xy--/xy/pkg-fun.nix

        • 3 packages xyz-/xyz/pkg-fun.nix

        • 4 packages xyzw/xyzw/pkg-fun.nix

        • 5 packages xyzw/xyzwa/pkg-fun.nix

        • 6 packages xyzw/xyzwab/pkg-fun.nix

        • 7-lettt packages xyzw/xyzwabc/pkg-fun.nix

        • Can’t be misinterpreted as a flag, there’s no packages starting with a -

        • _0ve/_0verkill/pkg-fun.nix, already a problem with windows though, not a concern

      • Underscore looks better than _
      • But for - you don’t have to hold shift for most keyboard layouts
      • If we used 2/2, we’d have to use more readDir calls, so 4-prefix is better in that regard

[Growpotkin]

“Dash gang rise up - let the pinky rest!”

For too long we have suffered under the oppressive yolk on the shift modifier - no more!

[roberth]

Why would we need to pad the prefix?

[Growpotkin]

Without padding the distinction between the shard directory and the package directory gets less obvious.

You could do it either way, one of these looks more pleasing than the other in my opinion, but for those who thumb their noses at aesthetics: if you don’t pad you get singleton packages like jq under ./auto/jq/jq/default.nix or /auto/u/u/default.nix and people will PR ./auto/xz/default.nix or ./auto/emacs/emacs/default.nix. Stale bot picks it up a few months after they get “please PR again with the right name: ./auto/xz/xz/default.nix and ./auto/emac/emacs/default.nix” and never get around to it.

It is somewhat clearer to readers and contributors that ./auto/xz--/xz/default.nix and ./auto/u---/u/default.nix have a rule : “4 character prefix, pad with dashes”, where ./auto/foo/foo/default.nix ./auto/u/u/default.nix, ./auto/fizz/fizz/default.nix, ./auto/emac/emacs/default.nix is a mystery.

Globs of constant length are also harder to botch in scripts in my experience.

But you’re right that ./auto/xz--/xz/default.nix is equivalent to ./auto/xz/xz/default.nix in terms of utility ( 16 precious bits are a shame to waste after all ).

[Infinisil]

Let’s continue the discussion in Sharding and other updates by infinisil · Pull Request #20 · nixpkgs-architecture/simple-package-paths · GitHub

[Sandro]

@Infinisil

• But for - you don’t have to hold shift for most keyboard layouts

FYI you need to hold shift on a Germany keyboard for /, too.

@Infinisil

@infinisil shows Add script for transition by infinisil · Pull Request #15 · nixpkgs-architecture/simple-package-paths · GitHub

I don’t see how that would handle name collisions which we certainly have.

[Infinisil]

What name collisions?

[Sandro]

If we would convert paths like

pkgs/development/libraries/abcde
pkgs/applications/misc/abcde
pkgs/development/python-modules/abcde

then they would all collide in pkgs/auto/abcde

[Infinisil]

The attribute name is used for the new directory name, so pkgs.abcde would be in pkgs/unit/abcd/abcde. The file where abcde was defined previously has no effect on the new location.

[Sandro]

So pkgs.python3Packages.abcde and pkgs.abcde are put in different directories?

[Infinisil]

Only toplevel packages, no other package sets. You can read about all the details here: GitHub - nixpkgs-architecture/simple-package-paths: Nix RFC draft for auto-calling packages in nixpkgs

[Infinisil]

And Sharding and other updates by infinisil · Pull Request #20 · nixpkgs-architecture/simple-package-paths · GitHub contains some recently discussed changes