Haumea - Filesystem-based module system for Nix

Announcements
1

This is not related to or a replacement for NixOS modules. It is closer to the module systems of traditional programming languages, with support for file hierarchy and visibility.

12 Likes
2

BTW, “haumea” is also name of the machine running the DB for hydra.nixos.org

2 Likes
3

This is a beautiful thing. Thank you!

1 Like
4

Good to know! Hopefully it won’t be too confusing

5

One of the benefits of using an approach like that is that it encourages the developer to expose the whole tree of dependencies. I noticed that Nix experts sometimes build a ball of highly coupled code that only they can understand, and this creates the sentiment that Nix is complicated. I don’t know if that was your motivation to write this @figsoda?

In that design space we also have:

Both projects also provide a small Nix CLI wrapper that maps the current folder to an attribute, so you can run mg or bld and it builds everything in that sub-tree. This is moving things more in the Bazel-like direction (even though Nix isn’t as great as incremental rebuilds).

The TVL Kit goes one step further and generates Buildkite pipelines directly from the tree metadata.

Haumea supports the private/public split, unlike the two other projects.

2 Likes
6

My initial motivation was for Nix libraries to be able to structure its API into files and directories without boilerplate files of just import statements, though having a tree of packages is also something I had in mind once I started writing haumea.

I did find myself decoupling code and also having smaller files in general, which IMO makes things easier to read. This also has the added benefit of preventing merge conflicts, and not having to enforce things like alphabetical ordering.

With haumea, things should just work with the vanilla Nix CLI, though you wouldn’t be able to build packages in a sub-tree just by cding into the directory, which means it wouldn’t work well with tools like josh (which TVL is using) out of the box.

7

Humea is a really nice and useful abstraction and it feels like years of experiments with digga, std and paisano distilled into just the right library.

For the sake of completeness, it’s worth mentioning std-action which afaik pioneered this approach with GH Actions with some more Nix-specific general improvements.

Appreciating that humea is a generalized super-set of Standard/Paisano’s loader, I wonder if you wrote humea in such a way that Paisano could adopt it as a special implementation of a loader?

If so, please give a hint, as modularization of Standard/Paisano-based projects can only be a good thing leading to new capabilities :cowboy_hat_face:

I also wonder, if humea can be adopted to load a module system submodule tree in such a way that each folder represents a tree node, e.g. of the NixOS module system, leveraging imports = []. The module system semantics are currently not ideally covered in Hive - another Paisano based project - and I’ve been on the lookout for better support for a while.

2 Likes
8

Currently haumea has an interface for overriding the default loader, which makes it possible to import the nix files however you want (or even not import it, like rakeLeaves). I’m not familiar with the specific designs of Paisano, so I’m not sure exactly how helpful that is. I am thinking of having an option to do post-processing on directories, which will make it possible to e.g. treat default.nix specially, maybe that would also help.

Absolutely! You can use haumea to load a properly structured directory into a single NixOS module. I still haven’t tried it, I’m planning to experience this with my NixOS config, though I’m pretty sure it will just work with something like this

inputs: haumea.load {
  src = ./modules;
  inherit inputs;
}
1 Like
9

Excellent!

Expect a contribution soon, if not from me from some other Standard / Hive user :heart:.

… and humea to become a trusted upstream for Hive / Standard. :grinning_face_with_smiling_eyes:

In this comtext, Paisano is pretty much humea.load limited to a disciplining three levels. It does other things, and that means it can be probably factorized further :grinning_face_with_smiling_eyes:

2 Likes
10

Haumea v0.1.1 released with support for module transformers, allowing more customization on the directory level, e.g. treating default.nix differently using transformers.liftDefault

3 Likes
11

Haumea v0.2.0 released with changes to transformer and two new transformers. Thanks @blaggacao for the contributions!

Diff: Comparing v0.1.1...v0.2.0 · nix-community/haumea · GitHub

Breaking Changes

  • Transformers now accept a ccursor as an argument.
    The type signature of transformer have changed
    from { ... } -> a to [ String ] -> { ... } -> a

Features

  • transformers.hoistAttrs and transformers.hoistLists bring a specific attribute name at all levels to the root.
  • load: transformer now also accepts a list or a nested list of functions.
1 Like
12

Not at all confusing – the docs are clear and concise. After using it once to replace a bunch of spaghetti code in one of my repos, I was able to use it again in another spot intuitively. Fortunately I did all this after the introduction of liftDefault, without which I’m not sure it would have got me very far since that treatment of default.nix files is already a common pattern in existing directory structures.

2 Likes
13

Haumea v0.2.1 released with loaders.scoped to load files with scopedImport instead of import

14

Haumea now has a book!

3 Likes
15

I’ve noticed a trend lately with the design of new tools becoming increasingly abstract. It’s an interesting development, and I can see how it could be beneficial in many ways. However, I sometimes find myself struggling to bridge the gap between abstract concepts and their practical applications, particularly when the provided documentation or related literature also adopts this high level of abstraction.

Take, for example, the multitude of “foo bar” examples. While they do serve a purpose in illustrating theoretical concepts, I often find it challenging to translate these into real-world, production-level scenarios. Perhaps it’s just a matter of my learning style requiring more concrete examples to grasp these concepts fully.

Furthermore, when it comes to reading explanations, I can’t help but feel they’re written in a manner that might only be comprehensible to those deeply versed in nix-lang. I’m not suggesting that these resources should be simplified to the point of losing their technical integrity. Still, a balance could be struck to make them more accessible to a broader audience.

8 Likes
16

I do think starting with a practical example is a good introduction to any tool. For haumea, this part from the book https://nix-community.github.io/haumea/api/load.html is good to present to newcomers:

For a directory like this:

<src>
├─ foo/
│  ├─ bar.nix
│  └─ __baz.nix
├─ _bar/
│  └─ baz.nix
└─ baz.nix


The output will look like this:


{
  foo.bar = <...>;
  baz = <...>;
}

This is a pretty good basic usage example that would be nice to have in a more immediate visible place, such as the readme.

1 Like
17

Thank you @luxus for the feedback!

Unfortunately, for a language like Nix, it is very easy for projects to get abstract if it’s not facing end users. It is kind of unavoidable to have abstract Nix code instead of more practical applications as examples for haumea’s documentation, though feel free to make suggestions!

I added the example to the introduction page (thank you @testplayername for the suggestion), and made some other minor improvements. Hopefully this makes it easier to understand.

Being the person that wrote this project, I might look at this project very differently versus someone seeing it for the first time. I’m trying to avoid jargon and advanced Nix topics when possible, but I might still make assumptions without realizing, so feedback like this really helps!

My goal is for the introduction and getting started chapters to give the users a basic idea, and the API reference would go deeper into the details and more technical side of things.

1 Like
18

v0.2.2 has been released with the new matcher interface, thanks @blaggacao for the contribution!

Features

load: loader now also accepts a list of matchers for loading non-Nix files

The following matchers and functions available under matchers:

  • always always matches the file regardless of its file name
  • extension matches the file by its extension
  • json loads all JSON files
  • nix is the default matcher if the loader is a function and not a list of matchers
  • regex matches the file using the given regex
  • toml loads all TOML files
1 Like
19

Having read through the documentation I’m wondering if & how it’s possible to combine haumea and nixosModules in the best way?

2 Likes
20

It is definitely possible to combine haumea and nixosModules! Since the initial announcement, I’ve switched my whole configuration to be based on haumea (into one big module).

There are efforts to improve the integration with NixOS modules with the transformers feature. The built-in hoistAttrs and hoistLists transformers provides a way to specify things like imports and config at nodes of the tree, though at the current state you might run into infinite recursions.

Another option is to have multiple directories loaded by haumea, so you wouldn’t need to use imports or config at the nodes, or you can just have one big module like me and not worry about creating your own NixOS modules, which has been working great for me.

2 Likes