Announcing Nomia, a general resource manager inspired by Nix

Hello Nixers!

I’ve been using Nix since 2010, and since 2012 most of my professional work has been related to Nix in one way or another. Over that time I’ve gained an ever deeper appreciation for the power of its theoretical foundations, but also of the limitations inherent in that framework as we currently understand it. I can reliably build, use, and distribute Nix-unaware software packages, but I can’t take advantage of Nix-aware efficiency improvements such as build caches below the package level or long-running processes amortizing build environment startup costs.

I can use Nix to compile individual modules in my project and combine them together, or use Nix to build whole packages and combine them together, but I can’t do both. I can use Nix to very precisely identify a package, but if I ever want a less precise notion or non-deterministic specifications I have to either work against Nix or outside of it altogether. I can use Nix to manage my packages, but if I want to realize any other resources that depend on packages and each other, such as a collection of interdependent microservices using Nix packages, or a complex computational graph using Nix to build the executable for each stage, I can’t take advantage of any of Nix’s properties for those other resources.

The longer I faced these problems and sought solutions, the more I realized Nix is handling its domain exactly as it should. Within the scope of this domain, which I now think of as providing composable specifications for precise, efficient general package management, Nix is doing all of the right things. What I perceived as Nix’s limitations were more a matter of scope. The question became: Is there some more general system, one which encompasses the Nix store, which could address these use cases?

Today I’m very excited to announce a project to build such a system: Nomia! Nomia is a universal system, designed to provide composable specifications for precise and efficient specification of any resource in any domain. Nix derivations fall out as a special case. For example, a derivation where we run bash from bootstrap-tools on a script that simply does echo > $out might in Nomia look like:


package:build-in-sandbox?args=[ bash, -c, $script ](
  exe: filesystem:by-content?name=bootstrap-tools&sha256=faede8fa5734e721deb694cb1ad335ddca2ea821046ff9b3b893176054b9499b/bin/bash,
  script: filesystem:by-content?sha256=b97662dfb9667dc7b501d37bf8bf5a2826dcde43decc6039a2a44356cfb483d7
)

But we can do so much more:

  • We can implement the “intensional store”, allowing packages to be content-addressed and users to have their own binary caches. When building a name addressed by its build script (like the build-in-sandbox example above), we could store the outputs in a namespace (Nomia’s word for a store) that uses content-addressed storage modulo self-reference, and emit and store a reduction signifying that the initial name in our package namespace can be replaced by a more refined content-addressed name in the content-addressed store. We could even use this to have project-specific namespaces, where you have an automatically trusted binary cache that doesn’t impact any other projects but shares results when the contents are the same or if caches are mutually trusted with other projects.
  • We can implement recursive Nix. Every time we resolve some Nomia resource within a build, we can emit and store a reduction signifying that the name we’ve been building can be replaced by a more refined name that takes the new resource as input. When we’re done with the build, we’d end up with a fully realized build graph that has no recursive calls.
  • We could have any number of namespaces with different rules for naming packages, all without impacting users who don’t rely on them and all realized at build time. We could have git:exact-fetch?repo=https://github.com/NixOS/nixpkgs&rev=785d2c03a0d that has access to a git repo cache and doesn’t require an extra hash; gcc:compile(filesystem:by-content?sha256=deac66ccb79f6d31c0fa7d358de48e083c15c02ff50ec1ebd4b64314b9e6e196) that can compile a single module with gcc without requiring a full sandbox and sharing previous compilations; private-filesystem?user=root:by-content?sha256=cc10ba3e72e09f1e742e4155968aa081fd309c76b6acf28eea1e3046438f005e that holds secret files only accessible to the user who owns the namespace; filter-source:by-gitignore which filters out gitignored files, etc.
  • We can give names to arbitrary resources, not just packages. Imagine NixOS where system services and their dependencies on each other (and on packages!) were managed in their own “service store”, ensuring some dependent service is realized when the top-level is. Or nixops where the dependencies between your nodes gets the sharing and garbage collection that Nix provides packages. All within the same general system!

One use case falling out of this that I’m especially excited about: cached incremental builds for compiled languages! Anyone who has used Nix to manage development for their own compiled projects, especially in a polyrepo scenario, has felt the pain of rebuilding everything when you just want to make one fairly localized change deep in the dependency stack. We could have, say, a GHC namespace for efficient haskell compliation and a Cabal namespace for building whole Haskell packages that delegates compilation to the GHC namespace, and reuse any previously done compilations down to the finest dependency grain!

So where does this leave Nix? Well, Nomia just covers the “Nix store” piece of the puzzle. The Nix expression language, nixpkgs, the Nix command line tools, etc. are not in scope. But if we take those tools and build them on top of Nomia, without compromising nixpkgs compatibility or impeding existing workflows, we can get all of the benefits Nix currently provides as a coherent part of the broader system. There is, of course, a lot of up-front technical work and community buy-in needed to get there, but we believe this can be done without significant disruption to those not involved in the implementation.

To learn more about Nomia, join the community to see what we’re up to, and start contributing, go check out the README and the resources linked from there, including an invitation to our Discord and a detailed technical deep dive. To understand how Nomia fits in with our mission at Scarf, see the blog post we just published. We’d love to hear your thoughts and feedback!

14 Likes

This is an interesting proposition. Personally I haven’t used Nix for long enough for these short-comings to come up for me, so I can’t comment in a very inteligent way, but a few things come to mind:

  1. It’s my impression that some of these restrictions on Nix are there very much on purpose, and removing them also removes guarantees that Nix provides. For instance, if you allow cabal to manage reusing incremental build artifacts, then if there’s some mistake cabal made, all of a sudden builds aren’t reproducable. Is your intent that the Nix/Nixpkgs/NixOS that we re-build on top of Nomia would reproduce those guarantees? And the advantage of Nomia then is that other tools can be built on top of it, or the guarantees can be selectively loosened in well-understood domains?

  2. You say that services and packages can’t depend on each other. I’d agree packages can’t depend on services (my understanding is this is so you can build packages on non-nixos system), but services often depend on packages. Either by putting them in systemPackages or referring to them directly. See services.xserver.windowManager.xmonad.extraPackages, for instance. I imagine I’m just misunderstanding what you’re referring to.

  3. Could your last bullet point be a replacement for systemd? That seems like a thing a lot of people want.

1 Like

Thanks for your interest!

It’s my impression that some of these restrictions on Nix are there very much on purpose, and removing them also removes guarantees that Nix provides.

Yes, absolutely true! Nomia will allow, for example, names for resources that are highly non-deterministic. A few key points on this:

  • This loosening is opt-in on a case by case basis. You can construct names which are guaranteed to have the same level of strictness that Nix does, or you can construct names that are looser (or depend on other names that are looser).
  • Through the reduction mechanism, these looser names can associated with successively more refined and precise names, recovering some of the properties. For example, you might ask for a “the latest version of Firefox”, and then when you realize the resource you might get informed that “the latest version” at this instantiation reduces to “Firefox 87.0”, which in turn reduces to “unpack this exact tarball and build it with this exact version of Make and GCC etc.”.

Is your intent that the Nix/Nixpkgs/NixOS that we re-build on top of Nomia would reproduce those guarantees? And the advantage of Nomia then is that other tools can be built on top of it, or the guarantees can be selectively loosened in well-understood domains?

Precisely, though we’d also want to expose the less strict names through the Nix expression language (though we’d keep that out of nixpkgs!).

services often depend on packages. Either by putting them in systemPackages or referring to them directly. See services.xserver.windowManager.xmonad.extraPackages , for instance.

Yes of course, but this is an ad hoc relationship and we can’t manage services as proper first class resources like packages. I can’t, for example, say “as long as my xmonad service is alive, the xmonad package should be kept around”. Instead we work around that by serializing an entire system configuration into a “package” and using that to manage the services within it. And we lose Nix’s nice properties for the relationship we want to manage.

Could your last bullet point be a replacement for systemd?

For parts of it, yes. I don’t see a space for the actual init process itself, nor many of the other functions systemd provides. But to the extent systemd is a service manager, absolutely.

Question of Understanding:

Does this bridge the gap to bazel builds in terms of efficient caching of component builds so that only necessary parts are rebuilt?

Yes, that aspect of bazel would be covered by the “recursive-Nix” style functionality. We are also investigating whether the bazel language, or something close to it, could directly be a frontend to Nomia much like we’re suggesting the Nix language could be. In that case, there’d be native inter-operation between Nix packages and bazel builds.

3 Likes

Re: broader mission.

Very honorable!

Please don’t miss the opportunity to have a look at numtide/devshell. It’s solutioning in the same problem space as scarf dev env manager and ideas have been reverberating to tackle a service manager, too.

I have been actively working with it and can intimately connect with the broader mission behind scarf. So I’m interested about your opinions of intersection and outlook / migration paths, etc.

2 Likes

Yeah, I’ve had a brief chat with @zimbatm about Numtide’s work and devshell in particular, I’ll take a deeper look! We’re eagerly seeking overlap like this to find opportunities to join forces with other projects, if you think devshell is a good fit then let’s discuss what that might look like!

I barely begin to grasp the value props behind nomia. My hipshot reaction would be: devshell can play the role of a bridge technology. In a way that it generalizes hands on from the acquainted problem domain into the generalized problem domain that nomia tries to solve.

Also, with numtide at the helm, certain organizational agility requirements might be naturally met, already.

I really love how you boldly went ahead with such a daunting idea and were able to recrute and organize the resources to make it happen.

1 Like

This is an insanely general concept, but its key ideas are something I often contemplated without being able (or eager enough) to formalize.

My favorite passages:

If you take the Church side of the Church-Turing thesis, name substitution is what computation is.

Because of determinism, using names forces us to say exactly what we mean. Domain-specificity of resource types and contextuality allow us to say exactly what we mean, and no stricter, especially if the contextual inputs are fine-grained. Together, this gives us an expressive specification that lets us rely on names and know what to expect with the resulting resources, across domains, modulo implementation bugs.

Even if nothing ever becomes of this particular implementation practically, the mental (or formal) framework Nomia provides is in itself incredibly valuable to see things from a different perspective. As in, most software developers do not or cannot use functional programming languages for various reasons, but looking at certain problems from that angle can make them much easier to handle than without.

Another example of managing resources declaratively I was kind of missing from the enumeration, because I think it is conceptually highly advanced and perfectly fits your application domain: Sander van der Burg’s declarative process management framework, which just happens to be implemented in nix.

2 Likes

Thank you! Scarf also deserves a lot of credit here, a seed-stage startup committing to a long-term investment like this requires a lot of courage and confidence!

2 Likes

Ah, I hadn’t seen this recent work from Sander, thanks for the pointer!

Glad you find the model valuable :slight_smile: Let’s see about making this particular implementation practical!

After skimming the whitepaper a bit, this looks really great. Almost like a userspace implementation of RedoxOS and it’s “everything is a URI” concept. Maybe someday, when both projects mature, they can get hitched. :grinning_face_with_smiling_eyes:

Anyway, I hope I can find some excuse to get my hands dirty with this very soon. Thank you!

3 Likes

This is very exciting. I had built a system for a specific “domain” (data transformation pipeline) using the same principles as Nix, but specifically weaker to meet the particular needs I had and I was able to avoid a ton of complexity - for example I had no need to support self-references and had strong control over where references would show up. This made it much easier to implement+understand and make supporting it extremely simple (pure key-value store and an extremely simple API). It also supported “reductions” (per the deep-dive paper) though it was interesting to see exactly what that cost in terms of implementation.

I wonder if specific optimizations/simplifications can be tied to specific relaxations of the naming semantics, and thus allow a slew of mix-match possibilities.

2 Likes

Yes, a lot of the necessary complexity of Nix comes from the fact that it needs to work for every package everywhere, no matter how poorly behaved. In many cases, things like sandboxes, reference scanning, self-references support, etc. can be omitted altogether.

Can someone eli5 what it means to me as a NixOS user?

Currently packages are implemented on top of Nix, though eventually they will be implemented on top of native Nomia package namespaces.

Does that mean that they will not have anything to do with Nix at some point? If so, why should I use it if I want to use nix? (I hope I don’t sound confrontational).

Can someone eli5 what it means to me as a NixOS user?

If you mean the “everything is a URI” idea, it basically means that in addition to store paths and derivations for your Nix packages, you’ll also have similar paths to identify services and entire system configurations. This would give you a single language to talk about all of these resources, and you could do things like “take my running configuration, but add in exactly the service running in that other configuration (even if it’s built against different versions of NixOS/nixpkgs)” or use Nix’s dependency inspection tools to see how you services relate to each other and the packages they depend on.

Does that mean that they will not have anything to do with Nix at some point?

No, it means that they will not be implemented on top of the Nix store. We’re expecting packages to still be ultimately reduced to Nix expressions resolved in the context of nixpkgs, and to hopefully have Nix itself built on top of Nomia. The reason you might want to use it alongside Nix is to get the seamless integration with the other aspects of the Scarf CLI, such as first class environment manipulation and Nix-aware build tools.

2 Likes

I wondered since yesterday: would that include remote config apis, eg. yang models in networking? I’d assume so, but wasn’t able to find an explicit clue.

It’s not a use case I’ve explicitly developed, but yes it should work.

While the real principles will require more experience to elucidate, I expect that any time that it’s significantly cheaper to identify a resource than to instantiate it, it will be worthwhile to represent it in Nomia. This also goes recursively for any resource needed as an input to a resource that’s cheaper to identify than instantiate.

2 Likes

Wow that’s fun.
From the practical standpoint,
Will its tooling be inherently cross-platform? A huge drawback of nix (which prevents me from using it for real outside of my pet projects) is that it wouldn’t run on winnt.
For working with resources from code — are libraries for major programming ecosystems expected to be available at some point?

Yes, at least the core mechanisms will be. We are writing in rust, working to identify clean interfaces to platform-specific capabilities, and keeping an eye toward nostd support so we could even support embedded use cases.

Yes. On our medium term roadmap we’re evaluating either Rust or Haskell in part based on the overlap with the Nix community, but in the long run the goal is that most such systems have Nomia backends.

4 Likes