(WIP)(RFC) A new nixpkgs frontend for language infrastructures

So this is something I’ve been working on for a while and tried to get feedback, without much luck I feel.
Now I realized I might as well try here as well, and I can reuse this post as the documentation I’m currently lacking.
It’s not a real RFC yet because the implementation is still in flux, and design changes may happen. However, the basic structure feels pretty stable.

You are all cordially invited to shower upon me your desires and criticisms - do your worst! :slight_smile: :stuck_out_tongue:


TL;DR: .withPackages (and some extra like enableDebugging) for All The Things!
Ideally this will result in less duplicated code, more consistency, and a knowable, reliable, and convenient toolset for dealing with the constructs this system wants to deal with.


  • library user: someone trying to write nixpkgs stuff
  • frontend user: someone trying to install plugins in an app or packages for an interpreter, for example

Fundamental deliverables:

  • (primary) to provide a consistent interface for things that fit into the application+plugin/interpreter+library analogy.
    i.e. a consistent user facing frontend for plugins and language infrastructures.
  • (secondary) to provide an abstracting library for implementors of language and plugins backends, which supplies common functionality and glue.
    This has the upside of, ideally, hiding some unnecessary complexity from the library user.

These goals could in theory be considered separately, but here they are currently handled as one demonstrator implementation.

Library goals:

  • should be composable, where appropriate (yes this is vague)
  • escape hatches should be availible so the library user may implement their own “local” overriders of some functionality if desired -
    i.e. it is explicitly a goal, that it should not be necessary to duplicate unrelated code - “local” “patches” should remain “local”.
    I personally find large, unmodifiable let expressions in nixpkgs rather annoying.
  • Provide amenities to make library user life easier, which will in turn hopefully make the experience for frontend users better as well
  • separate code from data


  • demonstrate plugins handling with Ghidra (demonstrated.)
  • refactor the R library infrastructure, to demonstrate the transferrability/general suitability of the approach to other backends (demonstrated,
    but not fully functional, further work may yield some useful feature creep - WIP)
  • write actual RFC
  • get RFC accepted :wink:
  • apply to all the things.

Historical notes:

  • rooted-overlay started in, and was factored out of - the Ghidra plugin handling infrastructure
  • nix-rer is the first POC/test of the initial implementaiton of rootedoverlay, and the results and missing features are being fed back into rootedoverlay

[RFC] Proposal: Engineering a Common "Plugin" Infrastructure

general library for “rooted” package+plugin systems. @84f14ba at time of post.

Ghidra demonstrator @b8c4562 at time of post.

R demonstrator (a mostly full refactoring of r-modules, very WIP) @25f87a4 at time of post.

Misc Notes:

  • I’m looking for suggestions for better terminology
  • The automatically imported, numbered layers approach is applicable to any system of overlays and perhaps could/should be separated out.

I’d be happy to discuss on freenode/#nixos-wg-ai as well (but you can find me on most of the other nix channels too).




A rough sketch:

A function (p: [ ... ]) :: a -> List is called a selector. Note that nothing prevents a Nix selector from including items not part of the original p (though this may be undesirable). You can add things from the external scope.

An AttrSetDrv is a derivation with some extra attributes tacked on. This can be done with e.g. pkgs.hello // { two = 2; }.

type Selector = AttrSet -> [ Drv ] (TODO: It may be reasonable to generalize to some list of plugin type.)
type Overlay = self : AttrSet -> super : AttrSet -> AttrSet
type Root = Drv || AttrSetDrv

The main function of this library mkRoot, returns the following structure:

  • .extend :: Overlay -> AttrSetDrv - is basically .extend from makeExtensible
  • .withPackages :: Selector -> AttrSetDrv - returns a new Root with some packages applied (using the implementation in ._api)
  • .nixpkgs :: AttrSet - a reference to the nixpkgs parameter passed to the system. This idea is to prevent cluttering up our own package set’s name scope
  • ._api :: AttrSet - a carrier for things things the system needs to fulfill it’s functionality, such as concrete implementations of interface functions
    • .withPackages :: scope : AttrSet -> root : Root -> selector : Selector -> AttrSetDrv -
      this is the concrete implementation of the function that applies a selector to the set of packages scope, and applies the result of this to the root
    • .root :: Drv - this stores the derivation used as the Root, and which is passed to things like .withPackages

Extra conventions (originating from additional layers):

  • .lib - misc. functions or whatever
  • .config - misc. variables

See packages.nix at the top level of any of the linked package sets for examples on how to call mkRoot, or the examples in nix-rootedoverlay//doc/tests.

A usage example can be found at https://github.com/NixOS/nixpkgs/issues/59344#issuecomment-508827104

To repeat it here, in nix-rer// you can do:

nix-shell -E "(import ./. {}).surrogate.withPackages (p: [ p.cran.purrr ])" -v --run "R --quiet -e 'library(purrr); map(list(1,2,3), ~(..1+1))'"

Library users:

nix-rer is currently the most advanced project, with some things not fully decided yet. The conventions followed there are also used in the other projects.

Directory structure:

  • default.nix - an attrset containing the roots, which call to packages.nix, enabling the usual default.nix workflow. It takes nixpkgs as a parameter allowing one to embed or use it standalone.
  • packages.nix - calls out to nix-rootedoverlay and passes to it the layers of the package set as well as the implementations of the interface functions such as withPackages.
    It uses functions from nix-rootedoverlay/lib.nix, which provides some implementations that in theory should be sufficiently generic to cover a lot of use cases.
  • extern/ - contains a git submodule for nix-rootedoverlay as well as a pinned nixpkgs for standalone usage and testing
  • layers/ - contains the set of overlays that are composed to construct the package set. The overlays are automatically composed in ascending lexical order (as implemented by attrset keys)
    The actual structure here is not enforced by the library but I use the following conventions:
    • 0_base.nix - the “implicit” layer that is the base of the layer set, it’s located in nix-rootedoverlay//0_base.nix
    • 1_util.nix - contains .lib and .config
    • 2_base_packages.nix - contains mainly the derivations used for Roots, and possibly other derivations.
      Note these are accessible to ._api because interface in mkRoot { interface = self: ... ; ...; } takes the fixpoint.
    • 3_autopackages/ - is a special convention, any directories ending with autopackages have their contents automatically imported into a layer.
      The goal with this is to make “one package per file” easier.
      The result is an attrset with the attribute names as the suffix-stripped names of the nix files in the directory, and the values as the import of the file.
      Directories are ignored.
      • json - contains data such as the actual package version information, URLs, hashes, whatever.
      • patches - contains some patch files which are automatically applied to the appropriate package’s source code
    • 4_overrides.nix - applies manual modifications to packages from the autopackages layer
    • ... - whatever other stuff you want to add
  • lib/ - all the rest of the R specific code
  • doc/, deprecated/ - the current structure of these is temporary, doc/ will be cleaned up and deprecated will be removed.

Library implementation:

In hindsight, maybe this got too long. :joy:

Edit: Something I feel I failed to make clear is that this approach is supposed to provide a context where standard conventions for an interface can be specified. These standards are what ._api is suppose to hold implementations of for the specific implementation.

One more thing: some nontrivial (but typical overlay-y) overlay overriding examples:

It looks like I’d probably need to update them a bit though.