Thank you! Scarf also deserves a lot of credit here, a seed-stage startup committing to a long-term investment like this requires a lot of courage and confidence!
4 Likes
Ah, I hadn’t seen this recent work from Sander, thanks for the pointer!
Glad you find the model valuable 
Let’s see about making this particular implementation practical!
2 Likes
After skimming the whitepaper a bit, this looks really great. Almost like a userspace implementation of RedoxOS and it’s “everything is a URI” concept. Maybe someday, when both projects mature, they can get hitched. 
Anyway, I hope I can find some excuse to get my hands dirty with this very soon. Thank you!
7 Likes
This is very exciting. I had built a system for a specific “domain” (data transformation pipeline) using the same principles as Nix, but specifically weaker to meet the particular needs I had and I was able to avoid a ton of complexity - for example I had no need to support self-references and had strong control over where references would show up. This made it much easier to implement+understand and make supporting it extremely simple (pure key-value store and an extremely simple API). It also supported “reductions” (per the deep-dive paper) though it was interesting to see exactly what that cost in terms of implementation.
I wonder if specific optimizations/simplifications can be tied to specific relaxations of the naming semantics, and thus allow a slew of mix-match possibilities.
5 Likes
Yes, a lot of the necessary complexity of Nix comes from the fact that it needs to work for every package everywhere, no matter how poorly behaved. In many cases, things like sandboxes, reference scanning, self-references support, etc. can be omitted altogether.
2 Likes
Can someone eli5 what it means to me as a NixOS user?
Currently packages are implemented on top of Nix, though eventually they will be implemented on top of native Nomia package namespaces.
Does that mean that they will not have anything to do with Nix at some point? If so, why should I use it if I want to use nix? (I hope I don’t sound confrontational).
2 Likes
Can someone eli5 what it means to me as a NixOS user?
If you mean the “everything is a URI” idea, it basically means that in addition to store paths and derivations for your Nix packages, you’ll also have similar paths to identify services and entire system configurations. This would give you a single language to talk about all of these resources, and you could do things like “take my running configuration, but add in exactly the service running in that other configuration (even if it’s built against different versions of NixOS/nixpkgs)” or use Nix’s dependency inspection tools to see how you services relate to each other and the packages they depend on.
Does that mean that they will not have anything to do with Nix at some point?
No, it means that they will not be implemented on top of the Nix store. We’re expecting packages to still be ultimately reduced to Nix expressions resolved in the context of nixpkgs, and to hopefully have Nix itself built on top of Nomia. The reason you might want to use it alongside Nix is to get the seamless integration with the other aspects of the Scarf CLI, such as first class environment manipulation and Nix-aware build tools.
4 Likes
I wondered since yesterday: would that include remote config apis, eg. yang models in networking? I’d assume so, but wasn’t able to find an explicit clue.
2 Likes
It’s not a use case I’ve explicitly developed, but yes it should work.
While the real principles will require more experience to elucidate, I expect that any time that it’s significantly cheaper to identify a resource than to instantiate it, it will be worthwhile to represent it in Nomia. This also goes recursively for any resource needed as an input to a resource that’s cheaper to identify than instantiate.
4 Likes
Wow that’s fun.
From the practical standpoint,
Will its tooling be inherently cross-platform? A huge drawback of nix (which prevents me from using it for real outside of my pet projects) is that it wouldn’t run on winnt.
For working with resources from code — are libraries for major programming ecosystems expected to be available at some point?
2 Likes
Yes, at least the core mechanisms will be. We are writing in rust, working to identify clean interfaces to platform-specific capabilities, and keeping an eye toward nostd support so we could even support embedded use cases.
Yes. On our medium term roadmap we’re evaluating either Rust or Haskell in part based on the overlap with the Nix community, but in the long run the goal is that most such systems have Nomia backends.
7 Likes
It seems like some of the excitement is in allowing Nix expressions to refer to things that are very unlike to contents of the existing Nix store. But could this sort of layer also be useful for ‘unifying’ different package managers that actually do keep Nix-like promises, so-to-speak? One sort of use case I have in mind is things like a Nix user using Guix to pull in the latest Guile, or a Guix user relying on Nix to provide dependencies that may not be suitable for inclusion in Guix.
Similarly, since Nomia is interested in supporting other languages, it seems like it could be a really useful bridge for providing scaffolding for package sets
- a Nix user on macOS could use Nomia to write impure Nix expressions that include a few packages or services from impure sources (MacPorts? Homebrew?) before gradually pulling them out one-by-one
- developers of new languages (like Nickel, Guix’s DSL, or Distri’s package recipe format) could use Nomia to allow their new tools to depend on or refer to parts of Nixpkgs for things they haven’t packaged yet, without having to directly interoperate with the Nix CLI or language
- users of other configuration languages could use Nomia to refer to packages and services provided by Nix idiomatically, if their language has a Nomia frontend
Are these kinds of things among the intended uses of Nomia?
4 Likes
Yes, precisely! Or, as I put it the other day in the Nomia discord channel:
One aspect of Nomia hinted at in the docs but not fully fleshed out that I’d like to cover better: Nomia as a general FFI whenever the integration point can be expressed via resource interactions. This is sort of like how Unix pipes act as a general FFI between arbitrary processes, except where named pipes let you even splat in the full source code of the other side of the pipe when you open it.
In particular this will allow any language to integrate with the Nix expression langauge. Not directly able to e.g. create a nixexpr function that is backed by some function in Haskell, but being able to reference and manipulate any packages that any nix expression evaluates to
3 Likes
Cool! I have some other questions that are just to sort of check my understanding of the vision of Nomia.
In a way, these ‘reductions’ remind me a little bit of nix-store --optimise, because of the way they deduplicate. In this case, it seems like what we’re really trying to save is time rather than disk space, but could Nomia likewise emit ‘reductions’ for things already built and dumped in the store if it notices that they’re equivalent in some sense after the fact?
It seems like this kind of elimination process could also handle lockfile generation for any project that is ultimately realized via Nix, no matter how it’s specified at the top level:
- Specify a Nix build using nondeterministic signifiers (e.g., source control URIs, which go out to the network and eventually resolve to the latest committed version)
- Use this kind of transformatino process until every source in the graph is a fixed-output derivation
- Dump the names of the root of every tree in the forest
This is kinda cool because it leaves the Nix community free to modify and even mix and match schemas for specifying Nix resources without explicitly defining or ever changing a lockfile format. Nix lockfiles could effectively just be any collections of Nomia names that have the right properties of determinism
So in binary package managers, an implicit global namespace is how ‘packages find each other’. Packages are treated as equivalent to one another, regardless of where they come from, based on name. In the Nix world, packages in Nixpkgs share a global namespace and refer to each other by name at build time. Nixpkgs has a mechanism for inserting out-of-tree dependencies into that shared namespace so that packages written without that modified dependency in mind can refer to it: overlays.
Nomia could standardize and generalize that sort of functionality, pulling it down to the store layer instead of a library written in the Nix language, in the form of explicitly configured reductions. Instruct Nomia to emit a reduction for a high-level name, one which is not fully-determined, like ‘php’, and now everything else in that namespace which was declared to depend on ‘php’ will depend on our modified version of that resource.
But because Nomia wants to name things that are more general than just files on disk, we could also use user-configured reductions like this to override service definitions, or details of a deployment like which host a service is to run on.
Would this kind of stuff be an abuse of Nomia, or is it aligned with your vision? It seems to me like it could be really nice for rolling out simple collections of Nix-specified packages and services without implementing complicated libraries for managing overrides in the Nix DSL. (With in-language override mechanisms remaining a possibility for folks who want something custom and know the Nix language well but don’t know or care how Nomia is implemented/how to extend it.)
So Nomia is expected to know, and maybe reason, about the guarantees offered by certain names (or maybe whole namespaces). Could this be used to classify package isolation?
To describe what I mean, I want to borrow the notion of ‘exchange directory’ from Distri, a neat little research distro from a former Debian developer inspired in part by Nix. An exchange directory is a shared (impure) resource on the filesystem that an otherwise isolated package is allowed to consult. So packages can’t depend on things outside the Nix store, but applications are usually still expected to read configuration from $HOME if they find it, and often (but not always! 
) /etc as well.
One of the historical goals with Nixpkgs has apparently been to ensure that packages always work when you run them from the Nix store, without depending on being installed to a profile. This means we have to do weird things to make software that inherently relies on impurity by design, like some runtime-loaded plugin systems, work right. In the case of Qt, we’ve opted to make applications unconditionally search for plugins based on the PATH that is set in their environment when they launch. This gives us a certain robustness (applications can be launched directly from the store) at the cost of a certain kind of impurity that can be difficult to manage, and for some use cases we might want to make a different tradeoff. Could judicious use of Nomia namespaces help track/manage these tradeoffs more explicitly without having to teach Nomia or Nix how to reason about different aspects of package isolation?
I’m picturing something like:
- guarantees have names
- those names could mean something to Nomia (e.g., this resource does not depend on any exchange directories; it promises not to read from any files outside the store)
- but guarantee names don’t have to mean anything in that sense
- guarantees can be presumptively satisfiable (satisfied unless a resource declares violating them) or explicitly satisfiable (satisfied only if a resource declares meeting them)
- packages can declare that they meet or violate some guarantees
- namespaces can declare that they meet or violate some guarantees
- if you’re working in a namespace that declares a guarantee, and you depend on a package (perhaps from another namespace) which is incompatible with it, Nomia throws an error and tells you that resource can’t be instantiated
Is this kind of thing (i.e., managing not just impurity but robustness) part of the intended picture for Nomia?
I believe recursive Nix, for example, is mentioned in the original Flakes RFC as it is in your post. Let me know if I correctly understand how Nomia addresses the problems that Nix flakes has been an experiment in solving.
Does (or, more weakly, could) Nomia subsume the purpose of the input side of the flake.nix schema— and only that portion of the flake.nix schema? In other words: half of the work of the flakes schema does is to provide a deliberately impure syntax in the Nix language for a certain kind of resource (that kind of resource also being called ‘flakes’ of course ), but which can always be reduced to a fully determined (fixed-output) resource in the lockfile.
The other half (the outputs schema) is a promise that Nix tools will understand what can be done with resources of that kind once it is handed one:
-
nix runwill know how to ‘run a flake’ by looking at its declaration of a default application -
nixos-rebuildwill know how to instantiate a flake as an operating system by looking for a host declared in the flake -
nix developwill know how to set up a shell environment for the development of a package associated with the flake - … and so on
In a Nomia-based implementation of Nix flakes, I imagine the layering would be slightly different than with the current prototype, because it would be natural for Nomia to handle resolving a name equivalent to flake:nixpkgs#fish all the way down to the package, and not just the flake:nixpkgs part. But essentially, Nomia’s task would just be to hand over resources that nixos-rebuild or nix profile install understands, and it wouldn’t have to care about what those look like. The substantive features like Nix flakes aims to offer in the CLI could remain more or less independent of Nomia, but they would leverage Nomia for input URIs. Is that right?
4 Likes
I think you understand it pretty well!
Yes, for example I imagine we might want “multihashing” in a content-addressed namespace, where the same resource can be identified by multiple different hash algorithms.
Yes, one of the nice things about Nomia’s approach here is that the high level “unlocked” description and the lower-level “locked” description are all on the same level with respect to the system as a whole, so you can mix and match as well. You could for example have a notion of “airplane mode” locking where any indeterminism that requires network access is resolved but any other indeterminism (e.g. reading local files) is kept unresolved until later.
This is aligned with the vision. I expect some users will want to restrict their systems to the fully specified byte-for-byte compatible approach like Nix gives today, but in practice we often want to describe and share resources at different levels of abstraction.
If I understand what you’re saying here, yes. Different resource types have different guarantees, and different contextual names (i.e. those which depend on resources passed in from outside of Nomia) will each have their own semantics, which can be statically analyzed to determine properties of the whole.
Yes, exactly. In fact you could use those kinds of input specifications independent of the rest of the flakes mechanism, just with a normal nix expression.
2 Likes
Right— overrides within Nix can be much more minute than replacing one resource with another, both resources more or less given as wholes, and that’s what’s great about the various overriding mechanisms we have in Nixpkgs today. This can make them really convenient and useful despite the way that they rely on machinery that can be tricky if you have to dig into its guts for some reason. But user-configured reductions in Nomia would work in a very simple way, at the cost of being a bit coarser. So both would be good to have for different situations.
That’s a very cool example. And yeah, it suggests that Nomia would be useful for managing different kinds of guarantees like I was thinking about in the more complicated example I tried to outline in my earlier post. 
I’m really excited about what Nomia might be able to do and be! Seems relevant to a whole range of real problems. It also seems clear that Nomia’s aims are conservative even though they are powerful, in that they seem unlikely to disrupt or hamper the development of features within Nix or products based on Nix that are unrelated to Nomia’s problem domain.
Looking forward to playing with it as soon as it’s ready for that!
2 Likes
This sounds really awesome (and way over my head at the moment…), thank you!
The “content-addressed Nix” annoucement was posted around the time this conversation wrapped up in May - did it affect any of the subsequent work on Nomia?
I assume the answer is no because Nomia seems to occupy a whole another level, and content-addressability in Nix is “just” an extra feature. I barely grasps the basics of either though so I could be totally wrong.
I’ve discussed the relationship of the projects with @edolstra and @thufschmitt , ultimately I hope for Nomia to cover what the CA store and other features currently provide but obviously there is significant work to be done before that’s feasible.
2 Likes
I’ve been wondering whether this project was still active, since signs of life on the GitHub repo have been scarce. I guess Scarf has a lot else on its plate and you’re waiting for a few more things to settle in Nix itself?
I still think Nomia is a very interesting idea. Care to make any adjustments to the timeline on the roadmap given on GH?
I’m no longer working with Scarf but I am continuing with Nomia, albeit with less time to dedicate to it at the moment.