(WIP)(RFC) A new nixpkgs frontend for language infrastructures

Development
1

What:

TL;DR: language interpreters and applications with libraries or plugins respectively, organized as a sub-packageset, using an overlay system, with composable .withPackages:
.withPackages (and co.) for All The Things!

Ideally this will result in less duplicated code, more consistency, and a knowable, reliable, and convenient toolset for dealing with the constructs this system wants to deal with. I feel the current implementations for language package sets and application plugin systems are not like this.

This is something I’ve been working on for a while and tried to get feedback, without much luck I feel.
Now I realized I might as well try here as well, and I can reuse this post as the documentation I’m currently lacking.
It’s not a real RFC yet because the implementation is still in flux, and design changes may happen. However, the basic structure feels pretty stable.

You are all cordially invited to shower upon me your desires and criticisms - do your worst! :slight_smile: :stuck_out_tongue:

Terms:

  • library user: someone trying to write nixpkgs stuff
  • frontend user: someone trying to install plugins in an app or packages for an interpreter, for example

Fundamental deliverables:

  • (primary) to provide a consistent interface for things that fit into the application+plugin/interpreter+library analogy.
    i.e. a consistent user facing frontend for plugins and language infrastructures.
  • (secondary) to provide an abstracting library for implementors of language and plugins backends, which supplies common functionality and glue.
    This has the upside of, ideally, hiding some unnecessary complexity from the library user.

These goals could in theory be considered separately, but here they are currently handled as one demonstrator implementation.

Library goals:

  • should be composable, where appropriate (yes this is vague)
  • escape hatches should be availible so the library user may implement their own “local” overriders of some functionality if desired -
    i.e. it is explicitly a goal, that it should not be necessary to duplicate unrelated code - “local” “patches” should remain “local”.
    I personally find large, unmodifiable let expressions in nixpkgs rather annoying.
  • Provide amenities to make library user life easier, which will in turn hopefully make the experience for frontend users better as well
  • separate code from data

Roadmap:

  • demonstrate plugins handling with Ghidra (demonstrated.)
  • refactor the R library infrastructure, to demonstrate the transferrability/general suitability of the approach to other backends (demonstrated,
    but not fully functional, further work may yield some useful feature creep - WIP)
  • write actual RFC
  • get RFC accepted :wink:
  • apply to all the things.

Historical notes:

  • rooted-overlay started in, and was factored out of - the Ghidra plugin handling infrastructure
  • nix-rer is the first POC/test of the initial implementaiton of rootedoverlay, and the results and missing features are being fed back into rootedoverlay

Links:
[RFC] Proposal: Engineering a Common "Plugin" Infrastructure
[RFC] Proposal: Engineering a Common "Plugin" Infrastructure · Issue #59344 · NixOS/nixpkgs · GitHub

general library for “rooted” package+plugin systems. @84f14ba at time of post.
GitHub - deliciouslytyped/nix-rootedoverlay: A simple Ouroboros is overrated.

Ghidra demonstrator @b8c4562 at time of post.
GitHub - deliciouslytyped/nix-ghidra-wip: WIP repository for prototyping ghidra packaging based on deliciouslytyped/nix-rootedoverlays

R demonstrator (a mostly full refactoring of r-modules, very WIP) @25f87a4 at time of post.
GitHub - deliciouslytyped/nix-rer: redoing the R package set

Misc Notes:

  • I’m looking for suggestions for better terminology
  • The automatically imported, numbered layers approach is applicable to any system of overlays and perhaps could/should be separated out.

I’d be happy to discuss on freenode/#nixos-wg-ai as well (but you can find me on most of the other nix channels too).

7 Likes
2

How:

Frontend:

A rough sketch:

A function (p: [ ... ]) :: a -> List is called a selector. Note that nothing prevents a Nix selector from including items not part of the original p (though this may be undesirable). You can add things from the external scope.

An AttrSetDrv is a derivation with some extra attributes tacked on. This can be done with e.g. pkgs.hello // { two = 2; }.

type Selector = AttrSet -> [ Drv ] (TODO: It may be reasonable to generalize to some list of plugin type.)
type Overlay = self : AttrSet -> super : AttrSet -> AttrSet
type Root = Drv || AttrSetDrv

The main function of this library mkRoot, returns the following structure:
root

  • .extend :: Overlay -> AttrSetDrv - is basically .extend from makeExtensible
  • .withPackages :: Selector -> AttrSetDrv - returns a new Root with some packages applied (using the implementation in ._api)
  • .nixpkgs :: AttrSet - a reference to the nixpkgs parameter passed to the system. This idea is to prevent cluttering up our own package set’s name scope
  • ._api :: AttrSet - a carrier for things things the system needs to fulfill it’s functionality, such as concrete implementations of interface functions
    • .withPackages :: scope : AttrSet -> root : Root -> selector : Selector -> AttrSetDrv -
      this is the concrete implementation of the function that applies a selector to the set of packages scope, and applies the result of this to the root
    • .root :: Drv - this stores the derivation used as the Root, and which is passed to things like .withPackages

Extra conventions (originating from additional layers):

  • .lib - misc. functions or whatever
  • .config - misc. variables

See packages.nix at the top level of any of the linked package sets for examples on how to call mkRoot, or the examples in nix-rootedoverlay//doc/tests.

A usage example can be found at [RFC] Proposal: Engineering a Common "Plugin" Infrastructure · Issue #59344 · NixOS/nixpkgs · GitHub

To repeat it here, in nix-rer// you can do:

nix-shell -E "(import ./. {}).surrogate.withPackages (p: [ p.cran.purrr ])" -v --run "R --quiet -e 'library(purrr); map(list(1,2,3), ~(..1+1))'"

Library users:

nix-rer is currently the most advanced project, with some things not fully decided yet. The conventions followed there are also used in the other projects.

Directory structure:

  • default.nix - an attrset containing the roots, which call to packages.nix, enabling the usual default.nix workflow. It takes nixpkgs as a parameter allowing one to embed or use it standalone.
  • packages.nix - calls out to nix-rootedoverlay and passes to it the layers of the package set as well as the implementations of the interface functions such as withPackages.
    It uses functions from nix-rootedoverlay/lib.nix, which provides some implementations that in theory should be sufficiently generic to cover a lot of use cases.
  • extern/ - contains a git submodule for nix-rootedoverlay as well as a pinned nixpkgs for standalone usage and testing
  • layers/ - contains the set of overlays that are composed to construct the package set. The overlays are automatically composed in ascending lexical order (as implemented by attrset keys)
    The actual structure here is not enforced by the library but I use the following conventions:
    • 0_base.nix - the “implicit” layer that is the base of the layer set, it’s located in nix-rootedoverlay//0_base.nix
    • 1_util.nix - contains .lib and .config
    • 2_base_packages.nix - contains mainly the derivations used for Roots, and possibly other derivations.
      Note these are accessible to ._api because interface in mkRoot { interface = self: ... ; ...; } takes the fixpoint.
    • 3_autopackages/ - is a special convention, any directories ending with autopackages have their contents automatically imported into a layer.
      The goal with this is to make “one package per file” easier.
      The result is an attrset with the attribute names as the suffix-stripped names of the nix files in the directory, and the values as the import of the file.
      Directories are ignored.
      • json - contains data such as the actual package version information, URLs, hashes, whatever.
      • patches - contains some patch files which are automatically applied to the appropriate package’s source code
    • 4_overrides.nix - applies manual modifications to packages from the autopackages layer
    • ... - whatever other stuff you want to add
  • lib/ - all the rest of the R specific code
  • doc/, deprecated/ - the current structure of these is temporary, doc/ will be cleaned up and deprecated will be removed.
1 Like
3

Library implementation:
(…later)

4

In hindsight, maybe this got too long. :joy:

Edit: Something I feel I failed to make clear is that this approach is supposed to provide a context where standard conventions for an interface can be specified. These standards are what ._api is suppose to hold implementations of for the specific implementation.

5

One more thing: some nontrivial (but typical overlay-y) overlay overriding examples:

It looks like I’d probably need to update them a bit though.

6

I am having trouble reading this thread. I wanted to give feedback but also have not enough energy to untangle all the data here, sorry!

If I understand properly, the problem you are trying to solve is unify and improve how sub-package-sets are formed in nixpkgs?

1 Like
7

I’m really tired right now and I’ve been focusing on other projects but I will try to give at least a direct answer (feel free to ask more - worst case I’ll go back and elaborate):

Yes, if im reading that correctly.

I started with wanting a uniform .withPackages for everything that has an interpreter, or is a thing with plugins.

Having wanted that I, tried to write something that allows a reasonable concrete implementation for each instance, while attempting to follow some principles I have about being able to override things and such.

Maybe another way to phrase it is looking for a common abstraction to dependencies with a rooted-tree-of depth-1 topology. Though I actually haven’t thought about it like that yet. I only thought about the concrete case of say, R packages and the R interpreter, Ghidra plugins and Ghidra, Idris packages and Idris.

What I seem to have wanted so far is a uniform, ergonomic interface for getting the root item with some scope of packages, and the ability to turn on debugging without needing to dig around in the guts of the system (this latter thing isn’t implemented yet).

I’m not very good at explanations so if you have any suggestions on how I can do better, please say so.
I hope that helped a bit.

8

This is both about a consistent way to use (as a user) such structures, and a consistent way to implement them (as a nixpkgs developer).

9

Sounds good and useful to do. Pretty much every packagesets in nixpkgs have invented their own way of composing their packages together. Having a unified way of handling those would allow users to learn the extension mechanism once and be done with it.

Have a look at Add pkgs.overrideWithScope by nbp · Pull Request #44196 · NixOS/nixpkgs · GitHub which tries to achieve similar goals.

3 Likes
10

Yeah, that’s basically the intention, and what I mean when I start throwing things around like “coherent” “homogeneous” etc.

Thanks for reminding me about that link. I lost it somewhere. I need to start keeping a better bibliography. :slight_smile:

11

It’s a deep topic. Just the single top-level fix-point is already hard enough to understand. Then add per-package-set fix-points like the one in haskellPackages, and then think about how to compose packages that depend on 2 or more package sets, each of them with their own fix-point and potential overrides. :exploding_head:

Good luck :smiley:

The hard part is to explain all of this with words and not get the reader confused. I believe a picture or even an animation could do wonders to share the understanding of how things are working.

4 Likes