(WIP)(RFC) A new nixpkgs frontend for language infrastructures

Development
2

How:

Frontend:

A rough sketch:

A function (p: [ ... ]) :: a -> List is called a selector. Note that nothing prevents a Nix selector from including items not part of the original p (though this may be undesirable). You can add things from the external scope.

An AttrSetDrv is a derivation with some extra attributes tacked on. This can be done with e.g. pkgs.hello // { two = 2; }.

type Selector = AttrSet -> [ Drv ] (TODO: It may be reasonable to generalize to some list of plugin type.)
type Overlay = self : AttrSet -> super : AttrSet -> AttrSet
type Root = Drv || AttrSetDrv

The main function of this library mkRoot, returns the following structure:
root

  • .extend :: Overlay -> AttrSetDrv - is basically .extend from makeExtensible
  • .withPackages :: Selector -> AttrSetDrv - returns a new Root with some packages applied (using the implementation in ._api)
  • .nixpkgs :: AttrSet - a reference to the nixpkgs parameter passed to the system. This idea is to prevent cluttering up our own package set’s name scope
  • ._api :: AttrSet - a carrier for things things the system needs to fulfill it’s functionality, such as concrete implementations of interface functions
    • .withPackages :: scope : AttrSet -> root : Root -> selector : Selector -> AttrSetDrv -
      this is the concrete implementation of the function that applies a selector to the set of packages scope, and applies the result of this to the root
    • .root :: Drv - this stores the derivation used as the Root, and which is passed to things like .withPackages

Extra conventions (originating from additional layers):

  • .lib - misc. functions or whatever
  • .config - misc. variables

See packages.nix at the top level of any of the linked package sets for examples on how to call mkRoot, or the examples in nix-rootedoverlay//doc/tests.

A usage example can be found at [RFC] Proposal: Engineering a Common "Plugin" Infrastructure · Issue #59344 · NixOS/nixpkgs · GitHub

To repeat it here, in nix-rer// you can do:

nix-shell -E "(import ./. {}).surrogate.withPackages (p: [ p.cran.purrr ])" -v --run "R --quiet -e 'library(purrr); map(list(1,2,3), ~(..1+1))'"

Library users:

nix-rer is currently the most advanced project, with some things not fully decided yet. The conventions followed there are also used in the other projects.

Directory structure:

  • default.nix - an attrset containing the roots, which call to packages.nix, enabling the usual default.nix workflow. It takes nixpkgs as a parameter allowing one to embed or use it standalone.
  • packages.nix - calls out to nix-rootedoverlay and passes to it the layers of the package set as well as the implementations of the interface functions such as withPackages.
    It uses functions from nix-rootedoverlay/lib.nix, which provides some implementations that in theory should be sufficiently generic to cover a lot of use cases.
  • extern/ - contains a git submodule for nix-rootedoverlay as well as a pinned nixpkgs for standalone usage and testing
  • layers/ - contains the set of overlays that are composed to construct the package set. The overlays are automatically composed in ascending lexical order (as implemented by attrset keys)
    The actual structure here is not enforced by the library but I use the following conventions:
    • 0_base.nix - the “implicit” layer that is the base of the layer set, it’s located in nix-rootedoverlay//0_base.nix
    • 1_util.nix - contains .lib and .config
    • 2_base_packages.nix - contains mainly the derivations used for Roots, and possibly other derivations.
      Note these are accessible to ._api because interface in mkRoot { interface = self: ... ; ...; } takes the fixpoint.
    • 3_autopackages/ - is a special convention, any directories ending with autopackages have their contents automatically imported into a layer.
      The goal with this is to make “one package per file” easier.
      The result is an attrset with the attribute names as the suffix-stripped names of the nix files in the directory, and the values as the import of the file.
      Directories are ignored.
      • json - contains data such as the actual package version information, URLs, hashes, whatever.
      • patches - contains some patch files which are automatically applied to the appropriate package’s source code
    • 4_overrides.nix - applies manual modifications to packages from the autopackages layer
    • ... - whatever other stuff you want to add
  • lib/ - all the rest of the R specific code
  • doc/, deprecated/ - the current structure of these is temporary, doc/ will be cleaned up and deprecated will be removed.
1 Like