Hello Nixers!
I’ve been using Nix since 2010, and since 2012 most of my professional work has been related to Nix in one way or another. Over that time I’ve gained an ever deeper appreciation for the power of its theoretical foundations, but also of the limitations inherent in that framework as we currently understand it. I can reliably build, use, and distribute Nix-unaware software packages, but I can’t take advantage of Nix-aware efficiency improvements such as build caches below the package level or long-running processes amortizing build environment startup costs.
I can use Nix to compile individual modules in my project and combine them together, or use Nix to build whole packages and combine them together, but I can’t do both. I can use Nix to very precisely identify a package, but if I ever want a less precise notion or non-deterministic specifications I have to either work against Nix or outside of it altogether. I can use Nix to manage my packages, but if I want to realize any other resources that depend on packages and each other, such as a collection of interdependent microservices using Nix packages, or a complex computational graph using Nix to build the executable for each stage, I can’t take advantage of any of Nix’s properties for those other resources.
The longer I faced these problems and sought solutions, the more I realized Nix is handling its domain exactly as it should. Within the scope of this domain, which I now think of as providing composable specifications for precise, efficient general package management, Nix is doing all of the right things. What I perceived as Nix’s limitations were more a matter of scope. The question became: Is there some more general system, one which encompasses the Nix store, which could address these use cases?
Today I’m very excited to announce a project to build such a system: Nomia! Nomia is a universal system, designed to provide composable specifications for precise and efficient specification of any resource in any domain. Nix derivations fall out as a special case. For example, a derivation where we run bash from bootstrap-tools on a script that simply does echo > $out might in Nomia look like:
package:build-in-sandbox?args=[ bash, -c, $script ](
exe: filesystem:by-content?name=bootstrap-tools&sha256=faede8fa5734e721deb694cb1ad335ddca2ea821046ff9b3b893176054b9499b/bin/bash,
script: filesystem:by-content?sha256=b97662dfb9667dc7b501d37bf8bf5a2826dcde43decc6039a2a44356cfb483d7
)
But we can do so much more:
- We can implement the “intensional store”, allowing packages to be content-addressed and users to have their own binary caches. When building a name addressed by its build script (like the
build-in-sandboxexample above), we could store the outputs in a namespace (Nomia’s word for a store) that uses content-addressed storage modulo self-reference, and emit and store a reduction signifying that the initial name in our package namespace can be replaced by a more refined content-addressed name in the content-addressed store. We could even use this to have project-specific namespaces, where you have an automatically trusted binary cache that doesn’t impact any other projects but shares results when the contents are the same or if caches are mutually trusted with other projects. - We can implement recursive Nix. Every time we resolve some Nomia resource within a build, we can emit and store a reduction signifying that the name we’ve been building can be replaced by a more refined name that takes the new resource as input. When we’re done with the build, we’d end up with a fully realized build graph that has no recursive calls.
- We could have any number of namespaces with different rules for naming packages, all without impacting users who don’t rely on them and all realized at build time. We could have
git:exact-fetch?repo=https://github.com/NixOS/nixpkgs&rev=785d2c03a0dthat has access to a git repo cache and doesn’t require an extra hash;gcc:compile(filesystem:by-content?sha256=deac66ccb79f6d31c0fa7d358de48e083c15c02ff50ec1ebd4b64314b9e6e196)that can compile a single module with gcc without requiring a full sandbox and sharing previous compilations;private-filesystem?user=root:by-content?sha256=cc10ba3e72e09f1e742e4155968aa081fd309c76b6acf28eea1e3046438f005ethat holds secret files only accessible to the user who owns the namespace;filter-source:by-gitignorewhich filters out gitignored files, etc. - We can give names to arbitrary resources, not just packages. Imagine NixOS where system services and their dependencies on each other (and on packages!) were managed in their own “service store”, ensuring some dependent service is realized when the top-level is. Or nixops where the dependencies between your nodes gets the sharing and garbage collection that Nix provides packages. All within the same general system!
One use case falling out of this that I’m especially excited about: cached incremental builds for compiled languages! Anyone who has used Nix to manage development for their own compiled projects, especially in a polyrepo scenario, has felt the pain of rebuilding everything when you just want to make one fairly localized change deep in the dependency stack. We could have, say, a GHC namespace for efficient haskell compliation and a Cabal namespace for building whole Haskell packages that delegates compilation to the GHC namespace, and reuse any previously done compilations down to the finest dependency grain!
So where does this leave Nix? Well, Nomia just covers the “Nix store” piece of the puzzle. The Nix expression language, nixpkgs, the Nix command line tools, etc. are not in scope. But if we take those tools and build them on top of Nomia, without compromising nixpkgs compatibility or impeding existing workflows, we can get all of the benefits Nix currently provides as a coherent part of the broader system. There is, of course, a lot of up-front technical work and community buy-in needed to get there, but we believe this can be done without significant disruption to those not involved in the implementation.
To learn more about Nomia, join the community to see what we’re up to, and start contributing, go check out the README and the resources linked from there, including an invitation to our Discord and a detailed technical deep dive. To understand how Nomia fits in with our mission at Scarf, see the blog post we just published. We’d love to hear your thoughts and feedback!
Let’s see about making this particular implementation practical!