I’ve been using Nix with NixOS as my daily driver for 6 months. Lately I’ve been trying to understand “what is the current state of Nix?” and this is what I understand:
There are many ways to do everything (flakes, flake-parts, flake-utils, custom nix, opinionated configurations like snowfall, etc).
Each way requires boilerplate code.
Each solution is essentially a bunch of functions that take care of most of the boilerplate, or adds modularity.
Documentation is very divided about which way is the correct way.
Because of these things, it feels as if there is no correct way to go about doing things like creating packages, building packages, maintaining custom packages, creating development environments, or even supplying inputs to functions. This might suggest that it is not a bad idea, at this point in the nix ecosystem, to just write my own custom solutions to the boilerplate problems. Would you agree with that statement?
This is not meant to detract in any way from the incredible work from the nix community. I think that Nix is the best solution to package management that exists and I’m honored to see it evolve.
I think it’s more that there are many things to do.
Specifically:
flake-parts is useful for writing projects that others depend on by making flake outputs more composable.
Using it for pure NixOS configurations does not strike me as a great idea, it adds nothing to that use case, since it’s not required for consuming flake-parts projects and doesn’t provide any additional utility for the NixOS module system. It only really makes sense if you provide other outputs, in which case your project requirements become more complex.
I’ll admit I don’t like the project in general, so far it’s only gotten in my way when I’ve tried it.
flake-utils is a random mix of utility functions, none of which are useful purely for a NixOS configuration
One of them is there to tape over a design problem in flakes that NixOS configurations are not affected by, and the rest are there to help people migrate from traditional project definitions (that would not typically be NixOS configurations)
opinionated configurations of course exist, but they are effectively splinter distributions - they are subsets of the NixOS distribution that make subtly different choices which suit some types of users or systems better
That isn’t quite everything, but all these do very different things. I don’t think it makes sense to compare them like this.
Yes, there are many things that people do with nix projects. It’s just a very versatile tool. I don’t think this will change.
There is one aspect where there are many ways to the same end goal, and that is the definition of inputs to a nix project. I think by “custom nix” you mean something that falls under this.
For this there are indeed basically three or four options:
nix channels
Probably what you mean by “custom nix”
npins
You can also do the stuff npins does by hand
flakes
Why are there so many options for this? Well, nix is old, and was originally designed when people hadn’t quite figured out that keeping track of project inputs isn’t trivial (because back then the software ecosystem didn’t yet have the mini-library usage explosion that nodejs brought).
This gave rise to the initial design of nix channels. It’s simple, really have a nice command that manages an environment variable and downloads some stuff and that’s how you update your system.
The problem with it is that this fundamentally breaks reproducibility - the moment you copy your project to another system, it will have different inputs, and hence have a good chance to break. The nodejs ecosystem realized this is a problem ca. 2014 or so, long after nix had already made this design decision.
People recognized this is a problem nix also has, and the community went looking for solutions. Turns out there’s a neat trick you can do:
nix allows importing arbitrary nix store paths (and in fact directories), so we can get nix to download a specific version of an input and import it. Great, we can now completely stop using nix-channel, and just leave it in the tool for legacy reasons.
It’s kind of inconvenient to update though, since you have to go and manually bump hashes in the code. So the community wrote niv, which is a little tool that updates hashes of inputs for you, and that was later forked to npins when niv maintenance stalled.
Now, separately, the nix project maintainers wanted to have a native solution in nix for this problem. They also wanted to have something similar to npm’s project.json so that the various things that nix projects are used for could be standardized a bit. They also wanted cached and pure evaluation, which was nontrivial originally given that nix files can come from anywhere and nix projects assumed this.
So they went to the drawing board, and designed this file format called “flake” which solved all of these things in one fell swoop. This was presented in an RFC, and ultimately implemented in nix - it has a handful of things that the community isn’t fully on board with yet, and some limitations that people aren’t happy with, though. It being this monster change, and some social challenges around project leadership, has left its acceptance in limbo for some years.
In short, these aren’t really different options, but incremental designs to the solution of a fundamental part of nix - reproducibility of inputs.
So, in effect:
nix channels
Don’t use them in 2024 unless you are a newbie who will struggle with the concept of “inputs” and just want a more standard Linux distribution feel
npins
Use this by default in 2024
flakes
Use them if you’re ok with experimental features, and the limitations don’t bother you
The ecosystem has also widely started adopting them as the standard they were proposed for, so knowing how to use them can really help you consume other projects more easily
Boilerplate sort of implies that an expression is “useless”, but the projects you mention specifically add stuff to give all the various things the nix ecosystem can do structure.
“Traditional” nix use, especially in the case of NixOS, has very little “boilerplate”, but that clearly is something the ecosystem is moving away from. I’d argue that most of this is not just boilerplate that can be safely ignored, but most of it is actually quite meaningful for expressing what your project actually does - just in the NixOS case it turns out you’re only using a specific subset of the features.
I’d argue this is maybe true for flake-utils, but certainly not for any of the other things you mention.
flake-utils should not be used in modern projects, though, except to avoid one design flaw with flakes (the system debacle). Besides this it only provides utility functions of questionable quality to make old nix projects adopt flakes more easily.
The official documentation is very clearly aligned with one single thing, especially for NixOS configurations.
There are lots of blog posts, discourse posts and other sources that will point you in all kinds of other directions if you have more specific requirements. I can see how a new-ish user might read all of these and not really understand the purpose of the specific projects, or exactly understand where their needs lie.
For example, a simple single-system NixOS configuration does not benefit from flakes much, so almost all the stuff you mention isn’t even useful. Channels are mostly fine if you only have one system and know how they work. Yet it will seem like lots of people are telling you to use flakes if you don’t understand the mismatch between your requirements and whomever they are giving advice to.
If the documentation seems disjoint, it’s because identifying a single “correct way” is very, very hard, and ultimately depends on what you want to achieve.
I think there is one aspect in which the ecosystem’s documentation is lacking today, though. There is no good guide that helps an individual who wants to use NixOS on their desktop to navigate the parts of the ecosystem they should be interested in (nor how NixOS itself works). The NixOS manual is too much of a reference doc for that, and it only mentions official nixpkgs-related projects, leaving out many useful third party things.
I’ve started writing a book that intends to be a more clean guide specifically for people who want to use NixOS on a desktop (so nothing about large cloud deployments, no talk about package sets, flakes are going to be an appendix at best - despite my being very vocal about liking them, …, but definitely mention of disko, nixos-anywhere, etc.).
I think documentaion like this will help our class of users quite a bit. Besides the large body of companies that are using nix as a build or deployment tool, there is a good chunk of users among us who just want to use an awesome Linux distro - but because we’re all volunteers (and most of us are busy maintaining the actual distro) there’s not as much manpower behind us.
I should say that there is the unofficial NixOS flake book, which is a step in the right direction, but it focuses - unsurprisingly - too much on flakes, which makes it not very suited to guiding you through using NixOS, especially for a newbie. Which is a shame; there’s lots of good stuff in there, and I don’t yet know how I’m going to bridge the gap between writing something from scratch and actually combining efforts with documentation with very similar goals.
Not per-se. I’d say that if you’re running into boilerplate problems you’ve perhaps been misled a little, and are trying to shortcut stuff you shouldn’t be doing in the first place. But this is hard to judge without knowing what exactly you consider “boilerplate”.
That said, I definitely prefer ignoring all the various frameworks. Pure NixOS basically just needs a better input management system, anything more than NixOS+niv or NixOS+flakes (using nixpkgs.lib.nixosSystem) just adds unnecessary complexity - until you start maintaining your own package/module set that you want others to depend on.
Thank you for the response. It was very eye opening. The following line in particular stuck out to me:
I think I was missing this concept in the following way.
One thing I am particularly interested in doing with Nix is using it in a company setting. Specifically I want to have an overlay of Nixpkgs where I can put all my custom packages that are useful to the company (but not necessarily useful to the nixpkgs community at large). I asked a question about this and was given an impressive solution from Drupol.
My work involves creating numerical simulations of biomedical devices, and one part of the goal is to have simple one-line terminal commands like python-env or development-env, etc. that I can run to build a flake that (along with direnv) will create an environment in the current directory that has all the specific versions of the huge mess of software that I need for each simulation. I’ve been working on packaging these commands as nixpkgs in my overlay and I ended up writing what I thought was a bunch of boilerplate code. Things like which system the package should work on, or specifying where my nixpkgs overlay is, seemed repetitive and “boilerplatey” to me. I wanted to build functions that just allow me to write something like “include custom-overlay” or “include system configurations” in the flake.nix file, instead of having those lines of code in each project.
After reading your response, I think its wrong to call those lines of code boilerplate, because they are expressing important aspects of what the project does. In a way, flake-parts does allow me to simplify my flakes in the way I mentioned above, but I’m really hesitant to put all my eggs into the flake-parts basket. It seemed to me that I should just write my own solutions to my own problems, and use only flakes and no external libraries.
I’ll keep pondering which direction I want to go. I really appreciate your thorough and extremely helpful response.
Fair! I’d assumed NixOS - having a use case where you actually publish a flake for downstream use changes things a fair bit. I’m far less well-versed with it than Drupol, but IMO flake-parts should fit your use case very well.
… is specifically the design flaw in flakes that flake-utils addresses (and frameworks like flake-parts tend to have functionality that circumvents it too). I personally believe the design intent (make users be explicit about what architectures they support - and test!) is good, but…
many software ecosystems aren’t explicit enough about it that developers would care or interact with it. It’s also awkward to have to use stuff like flake-utils to ergonomically say “all of these architectures”.
It’s one of the big problems left with flakes, basically. I’m curious where this will go; at the very least, I think something that replaces forAllSystems should become a built-in function, or some syntactic sugar for it should be added.
Software architecture is always a fun problem. Do share the broad picture of your solution when you arrive at one, always better to learn with the hindsight of practice
My understanding is that there’s a big ecosystem-defining difference between flakes and other alternatives (pinning manually, niv, npin, gridlock).
There’s a problem of direct dependencies, which usually boils down to “I want to know the hash of nixpkgs I depend on”.
Then, there’s separate problem of transitive dependencies (dependencies of your dependencies). For indirect dependencies, just specifying hashes won’t cut it: if you have two deps that depend on two slightly different nixpkgs, you probably don’t want to have that duplicated, and rather use a single nixpkgs close enough to both. You also want to let the user choose the versions of transitive dependencies.
That is, while direct dependencies require specifying a version, transitive dependencies are about version constraints — specifying a set of version you are hopefully compatible with (notably, some of the versions might not even be written yet at the point of requirement specification).
Solving the problem of indirect dependencies would allegedly allow the ecosystem to evolve in a more decentralized way. And that is a hard problem to solve, because you need everyone to agree on a specific way of managing transitive dependencies. This is in contrast to direct dependency management, where you can use a completely bespoke tool and nobody but you would notice.
And my understanding is that the flakes try to address this second problem, while it is out of scope for niv and the like.
No informed opinion neither on how good is the flakes solution nor on whether it is a problem worth solving at all.
Are you saying that, for indirect dependencies, flakes is the best solution, and therefore flakes are the de facto current state of Nix? And anything written on top of flakes like flake-parts, flake-utils, FlakeHub, etc. etc, are more or less equivalent custom solutions to managing flakes?
It’s generally not considered particularly adequate yet, see the need for flakehub.
Flakes is the direction the NixOS ecosystem is going in the long term (recent developments probably pushed that date out a few more years…). It’s as of yet incomplete, and it’d be hard to call this specific aspect of it “best”. It is another aspect of flakes that will probably see some changes in the future, you mention flakehub yourself.
I wouldn’t call them equivalent, they’re very different things. flake-parts is a framework for writing “library” flakes (package sets, nix function libraries, third party NixOS modules, …) in a way that integrates better with other projects, flake-utils is basically just a way to prevent having to write system strings repeatedly, flakehub is a way to simulate semver versioning for flake inputs by downloading tarballs from a proprietary, centralized repository with a fancy http API. You could quite reasonably use them all in the same project.
Besides input/dependency pinning, I have always found it a bit awkward to define my own derivations for building packages not yet in the nixpkgs repository. I did a bit of a meta search a while ago and the following issues caught my attention:
These issues all mention the same question implicitly: “how do we improve the way we define derivations?” While I am not very involved with the community currently, I would really like to have solutions for the problems mentioned in these issues at some point. (Even more willing to help get it done.)
I am strongly of the opinion that improving the derivation situation in nixpkgs will allow to improve both documentation and decentralized extensions of nixpkgs. I suspect that a clearer, possibly more uniform, derivation interface will make it easier for the community to come up with a more uniform approach to supply precise dependencies to a project.
I’ve been using NixOS about a year now, and dove straight into flakes from the beginning. I knew I wanted to avoid the problems with channels, and assumed flakes were the only other option.
However, since then I’ve felt more and more like flakes add a bunch of boilerplate. I appreciate the package pinning, but am not a fan of everything being experimental/impure, all the old commands that still exist but shouldn’t be used, the emphasis on storing flakes in Git (I don’t), etc.
You aren’t the first person I’ve seen suggest using niv or npins, I’m tempted to see what it would take to drop flakes and revert to those tools. No channels and no flakes, sounds pretty slim.
Is this what you use? You mentioned that you like flakes, what are your reasons for recommending niv?
As far as I can tell, and as you mentioned, flakes is the future NixOS is headed towards. Does that make using niv/npins a temporary solution at best?
Are we all doomed to switch to flakes at some point? Or do you think it will have evolved into something worthy of replacing niv at that point?
You mean the --impure flag? If you’re not using flakes everything is impure even without that flag, nix just doesn’t tell you because pure evaluation isn’t a concept in nix 1.0 land. If you are using flakes but still need that flag that just means you have a mistake in your configuration that makes pure evaluation impossible.
There’s some calls for backporting pure eval to the old CLI interfaces, but afaik that hasn’t gone anywhere because the dev team just is too busy and prefers pushing flakes further with the time they have. Maybe with lix & co that might change.
There’s plenty of things not supported in the 2.0 command line yet, for which the 1.0-style commands aren’t just fine to use, but absolutely necessary.
The 1.0 commands that “shouldn’t be used” are disrecommended because they’re footguns with incredibly confusing behavior, especially for newer users. Not using the 2.0 commands doesn’t help with that, they’re just as problematic when you’re using exclusively 1.0 commands.
The 2.0 CLI is supposed to be separate from flakes anyway, though I appreciate they aren’t as cleanly separated as they should be. The roadmap to flake stabilization the community agreed on includes splitting them apart, but well, that will take a long time, and there’s drama around project leadership that we shouldn’t go into. Again, the forks might help with this in the long-term.
Flakes support other backends too. For other VC tools it seems to be limited to mercurial currently, but I’d imagine more will be added over time.
If we’re talking local paths nix will happily consume them without git, too, it’s just subject to this long standing problem - IMO the biggest blocker to wider flake adoption.
Forcing nix to not use git can be nice if you want to avoid nix badgering you about dirty trees, but I’m very curious what other reasons anyone could have not to put code in a git repo. Do you prefer another VC tool that’s not yet supported?
No, I use flakes. I’m not too bothered by the experimentalness since I use nix primarily for personal purposes and stay very in the loop, so the potential for breaking changes doesn’t phase me much. The input schema is useful when I depend on other projects, and I understand the limitations and how things fit together fairly well (and I’m not shy to read the underlying code). Pure evaluation is great. A few things annoy me, but I’ve found my way around them over the years.
I just wrote some 3 paragraphs responding to weird rough edges around flakes that are problematic for you, and make the process of learning to use an already niche and complex ecosystem even more cumbersome. I believe this stuff will be polished away eventually, but well, it’s been like 4 years now.
niv gives you 90% of the flake features a normal user would care about for a NixOS configuration, and is much easier to adopt while isolating you from a bunch of not entirely polished parts of the ecosystem. Flakes also make the official documentation really hard to use, which is a massive problem for new adopters - niv doesn’t have this problem.
That doesn’t mean flakes don’t have some advantages that make me personally prefer them.
I recommend niv over the others primarily because to my knowledge it’s the one with the biggest community share - there’ll be more guides and polish. My call is to err on the side of that for users avoiding flakes, but I’m happy for that view to be challenged.
~ish. Traditional nix isn’t going anywhere. I’d be (positively) surprised to see flakes stabilize without these glaring issues still in tact before another 5 years have passed. Even if you don’t use flakes, with a modern nix implementation you can still consume flake inputs, so there’s no risk of the ecosystem leaving you behind.
Anything’s temporary on a long enough time scale, but I don’t think you have to worry about niv & co today. Besides, there’s almost no switching cost with niv, it’s pretty simple.
I think the problem isn’t really the concept of flakes itself. It’s that there’s such a massive split today. You’re either 100% nix 2.0 or you don’t use it at all. Lots of things are conflated between the 2.0 cli, pure eval, flake schemas, etc. It’s just hard to get the hang of it all in one go.
If the community roadmap is followed the changes will feel much more granular, and the official documentation will also actually cover the topic (without being completely changed overnight). Especially for NixOS users, a lot of rough edges could really easily be smoothed over so people don’t have to look behind the abstraction to understand what’s going on so much.
With proper, official documentation, fewer moving parts and a cleaner implementation I think this won’t feel like such a significant choice anymore. Eventually there could be a simple niv → flake conversion script that instantly makes stuff work without any changes.
There’s a lot of work to be done before the ecosystem gets there, but I think if properly done flakes will eventually feel like a no-brainer. Of course, this means that we need more people who’re willing to do such work, so hopefully in the mean time niv tides folks like you over
As member of the documentation and Nix maintainer teams, I generally agree with @TLATER’s assessment, with a few corrections:
Use npins instead of niv.
niv hasn’t been maintained in years and has a few small kinks that won’t get fixed unless someone forks it. npins authors merge pull requests, and generally it’s more polished.
The current stable release of Nix is 2.x while the “new CLI” and flakes are supposed to be stable for a hypothetical 3.x.
Not the whole development team is pushing flakes, it’s really only Eelco working on lazy trees. We did do some stabilisation work on the new CLI last year, but that stalled again for multiple reasons (in my view it’s mainly too little available time and too hard architectural problems). I’m personally focusing on helping users leverage what’s there by expanding and restructuring documentation, and improving the contributor experience so more people learn how it works on the inside and hopefully help fix bugs and add more tests.
The old CLI commands that aren’t officially deprecated but really shouldn’t be used because they’re unfixable footguns are nix-env and nix-channel. In my opinion nix profile and nix registry are only slight improvements based on the exact same concepts, and I recommend not going into the imperative realm at all.
In particular I’d like to stress that most of everything that’s old in terms of when it was originally designed is quirky and has more moving parts than needed, which makes the inner workings hard to understand for new users, hard to explain for documentarians, and hard to change for developers without breaking everyone‘s setups or habits. Concretely these are profiles, channels, packages (in the Nixpkgs sense), modules (in NixOS), and flakes only add a bunch more concepts to the pile.
That said, one reason flakes have seen wide adoption is that the new features provide a solution to some important problems, such as evaluation time (with caching) and source tracking (with the inputs schema). But it’s not the only possible solution, and not optimal for a bunch of reasons such as overall not great architecture and maintainability of the underlying code. And I care about that a lot, because we’re extremely resource constrained; getting bogged down by inflexible code on top of that makes any progress very hard.
I recommend to learn Nix bottom-up and using only what you actually understand and can fix yourself. (Concretely: store objects, derivations, and the Nix language are the foundations of it all and will probably never change.) This is the most reliable long-term strategy in the current situation. Don’t hold your breath for others to do things for you.
Thanks for your “peek behind the curtain” from your highly involved perspective and your hard work!
… but that would severely limit my ability to actually get things done using nix(OS) (so I’m pretty much already deeply bought-in to flakes).
But I wouldn’t mind if the future would bring something flake-like that solves many of flake’s problems even if it would be very much backward incompatible.
Most Nix users have already once invested big upfront by learning Nix(OS), I’d expect most be willing to do it again to transition to a 3.0 if it were to bring (many) breaking changes, if the benefits are obvious, e.g. unified handling, solving current (perceived) problems with the status quo (both flakes and otherwise).
Since it seems the major hurdle for progress in the flakes department is backward compatibility, maybe it should be considered to do away with that hurdle?
I’ve been using Nix with NixOS as my daily driver for 6 months.
Welcome!
There are many ways to do everything (flakes, flake-parts, flake-utils, custom nix […]
Yes.
Each way requires boilerplate code.
Yes.
Each solution is essentially a bunch of functions that take care of most of the boilerplate, or adds modularity.
Yes.
Documentation is very divided about which way is the correct way.
Not exactly: the official documentation documents stable, mainstream Nix. Third-parties like Determinate Systems corporation provide unofficial guides that recommend non-standard, experimental technology like Flakes.
Because of these things, it feels as if there is no correct way to go about doing things like creating packages, building packages, maintaining custom packages, creating development environments, or even supplying inputs to functions.
Yes, this is a major failure of project leadership. The community is currently working on new governance that will hopefully improve this situation.
This might suggest that it is not a bad idea, at this point in the nix ecosystem, to just write my own custom solutions to the boilerplate problems. Would you agree with that statement?
It depends on what exactly you have in mind. Writing helper functions, for example, is often a good idea.
Trying to replace pins or other more complex unofficial Nix tools I think is a bad idea with only six months experience. I would strongly recommend only using the standard, official technologies until you hit a particular pain point that requires reaching for something like pins in the first place. Some of the unofficial tools like Home Manager offer such clear advantages that many of us want to see them integrated into the official project. The unfortunate reality is that Nix is not mature enough to have all this figured out for you, and you will have to learn and adopt different approaches as you get more familiar with the ecosystem.
This is not meant to detract in any way from the incredible work from the nix community. I think that Nix is the best solution to package management that exists and I’m honored to see it evolve.
I think you are right to ask the questions above and we as the Nix community need to provide a better onboarding / default experience. It used to be much easier to get started with Nix, and I hope one day it can be again.
