2023-01-16 Nixpkgs Architecture Team Meeting #25

Development Nixpkgs Architecture
1

Notes

Originally we just wanted:

# pkg-fun.nix
{ stdenv }: stdenv.mkDerivation { .. }

But because then adding callPackage overrides would cause people to have to move back to all-packages.nix, something like

# args.nix
pkgs: { stdenv = pkgs.stdenv_noCC; }

would be good. But then @roberth is proposal to just generalize it fully like this:

# pkgs-attr.nix
pkgs: callPackage ./pkg-fun.nix { stdenv = pkgs.stdenv_noCC; }

Or even

# default.nix:
{ pkgs ? import ../.. {} }: pkgs.callPackage ./pkg-fun.nix { stdenv = pkgs.stdenv_noCC; }

  • @infinisil: Not isolated

  • @growpotkin: That’s the use case

  • @roberth: Allows later innovation with how packages are created, e.g. modular packages. Don’t like having to scan 5 files to determine what should happen, hard to contribute

  • @Ericson2314: Do we need any of these files?

  • @roberth: Well the default.nix allows for experimentation without changing the schema.

  • @infinsil: Well what about enforcing the structure?

    • It could in principle handle a lot of things that we want to put things out of scope.

  • @Ericson2314: Not always easy to conform to structure, but when you do there’s predictability. Fine as a preliminary pattern, we want better dependency specifications (e.g. splicing)

  • @tomberek: Invariant to expect from consumers?

    • @roberth: Only use case is through readDir, directory structure isn’t a public interface

  • @infinisil: Ideal end state: Every file in a unit directory is a standard

  • @tomberek: Interactions are hard, just readDir and callPackage is simple, maybe not as generic, minimize the interface

    • @roberth: Simple for who? For contributors it would be easy to not have to look at rules declared by e.g. this team

  • @Ericson2314: String separation between describing inputs in terms of needs, done by the package, and outside world injecting dependencies which satisfy those needs.

  • @tomberek: In case we don’t want to use automatically used callPackage, have to dig into it, original function is “hidden”

    • @roberth: Would have to use .override, shouldn’t dig into unit dirs
    • @infinisil: Could expose unit directories as unstable API point

  • @Ericson2314: Overrides are often very related between packages

    • @infinisil: Like python/python3 pattern in all-packages.nix
      • @Ericson2314: Should use version bounds or something like that

  • @infinsil: Feels risky expanding something near the end of the RFC drafting.

  • @Ericson2314: So the problem we’re trying to solve is someone trying to write a unit file, and then they find actually they are outside what units handle, and then they need to start over / do things completely different.

  • @Ericson2314: what if we separate units we autocall vs units we don’t autocall vs non units?

  • @roberth: We could come back to allowing access to unit directories from all-packages.nix

    • So that we don’t need args.nix / default.nix / whatever
    • It’s more powerful than those solutions

  • @tomberek: Allows paying down technical debts, passing arguments is technical debt, have a firewall against technical debt into unit directory

  • @tomberek: What if you ask directly for the thing you want, like pkg-funs.hello-and-cowsay.

    • @roberth: Doesn’t have all the right arguments, needs some duplication

  • @roberth: Simple solution: Allow all-packages.nix to call into unit dirs

  • pkgs/unit/foo/foo and foo = callPackage ./some/other/dir {}

  • @ericson2314: two dirs

    • manual units (all packages calls)
    • auto units (sharded dir auto-call)
    • all-packages.nix and auto units shoudl be disjoint

  • @infinisil: Could do CI check to make sure it’s a callPackage <unit dir>, not another dir, if there is a unit directory

    • @roberth: config.callPackageDuplicationCheck
    • @infinisil: Generic config.checkNoConflictingAttributes
    • @Ericson2314: Require a callUnit not callPackage, for now callUnit = callPackage or so but presumably would change in the future. Maybe only for CI for now

  • @Ericson2314: Put something in unit directory to indicate whether it’s being called from all-packages.nix

  • @roberth: all-packages.nix is on top of auto called packages in the overlay layering

  • Tool should only move attributes without arguments in the callPackage call

Simple solution to just allow all-packages.nix to refer to unit directories with CI checks consensus

Some code we looked at

  • @growpotkin: pkg-fun.nix understanding check with variants
    Some relatively generic builder similar to linkFarm, in this case it builds a package
    that symlinks hello and cowsay, but a variant of the package will add fzf.
# unit/hell/hello-and-cowsay/pkg-fun.nix
{ stdenv, lndir, paths }:
stdenv.mkDerivation {
  name = "hello-and-cowsay";
  argsList = builtins.concatStringsSep "\n" (
    builtins.attrValues ( builtins.mapAttrs ( t: f:
      "mkdir -p ${t}; lndir ${f} ${t};"
    ) paths )
  );
  passAsFile        = ["argsList"];
  nativeBuildInputs = [lndir];
  buildCommand      = ''. "$argsListPath";'';
}

# unit/hell/hello-and-cowsay/default.nix
{ pkgs   ? import ../../.. {}
, lib    ? import ../../../lib {}
, stdenv ? pkgs.stdenv
, xorg   ? pkgs.xorg
, lndir  ? xorg.lndir
, hello  ? pkgs.hello
, cowsay ? pkgs.cowsay
, paths  ? { hello = hello.outPath; cow = cowsay.outPath; }
}: lib.makeOverridable ( import ./pkg-fun.nix ) {
  inherit stdenv lndir paths;
}

The variant

unit/hell/hello-cowsay-and-fzf/default.nix
{ pkgs             ? import ../../.. {}
, stdenv           ? pkgs.stdenv
, xorg             ? pkgs.xorg
, lndir            ? xorg.lndir
, hello            ? pkgs.hello
, cowsay           ? pkgs.cowsay
, fzf              ? pkgs.fzf
, hello-and-cowsay ? pkgs.hello-and-cowsay
, paths            ? {
    hello     = hello.outPath;
    cow       = cowsay.outPath;
    "sub/fzf" = "${fzf}/bin";
  }
}: ( hello-and-cowsay.override {
  inherit stdenv lndir paths;
} ).overrideDrvAttrs ( _: {
  name = "hello-cowsay-and-fzf";
} )

Action items

1 Like