Is this the current state of things?

Help
8

Are you saying that, for indirect dependencies, flakes is the best solution, and therefore flakes are the de facto current state of Nix? And anything written on top of flakes like flake-parts, flake-utils, FlakeHub, etc. etc, are more or less equivalent custom solutions to managing flakes?

9

This is fair, yes.

It’s generally not considered particularly adequate yet, see the need for flakehub.

Flakes is the direction the NixOS ecosystem is going in the long term (recent developments probably pushed that date out a few more years…). It’s as of yet incomplete, and it’d be hard to call this specific aspect of it “best”. It is another aspect of flakes that will probably see some changes in the future, you mention flakehub yourself.

I wouldn’t call them equivalent, they’re very different things. flake-parts is a framework for writing “library” flakes (package sets, nix function libraries, third party NixOS modules, …) in a way that integrates better with other projects, flake-utils is basically just a way to prevent having to write system strings repeatedly, flakehub is a way to simulate semver versioning for flake inputs by downloading tarballs from a proprietary, centralized repository with a fancy http API. You could quite reasonably use them all in the same project.

3 Likes
10

Which I found mentioned here: Recommended upgrade path from 23.11 to 24.05

1 Like
11

I think GitHub - andir/npins: Nix dependency pinning. Very similar to Niv but has a few features that I personally wanted. is also worth consideration.

2 Likes
12

Mmm… useful info - thanks, all.

The fact that I have now been using NixOS for quite some months and had never even heard of ‘niv’ until just now indicates the scale of the problem…

5 Likes
13

Besides input/dependency pinning, I have always found it a bit awkward to define my own derivations for building packages not yet in the nixpkgs repository. I did a bit of a meta search a while ago and the following issues caught my attention:

These issues all mention the same question implicitly: “how do we improve the way we define derivations?” While I am not very involved with the community currently, I would really like to have solutions for the problems mentioned in these issues at some point. (Even more willing to help get it done.)

I am strongly of the opinion that improving the derivation situation in nixpkgs will allow to improve both documentation and decentralized extensions of nixpkgs. I suspect that a clearer, possibly more uniform, derivation interface will make it easier for the community to come up with a more uniform approach to supply precise dependencies to a project.

2 Likes
14

About this, the infamous post where I vented some frustrations on splicing:

15

I’m new here but loving the nix project. This was a great explanation on several doubts I also had, I hope to be able to find your book when is ready.

1 Like
16

First, very informative post, thank you!

I’ve been using NixOS about a year now, and dove straight into flakes from the beginning. I knew I wanted to avoid the problems with channels, and assumed flakes were the only other option.

However, since then I’ve felt more and more like flakes add a bunch of boilerplate. I appreciate the package pinning, but am not a fan of everything being experimental/impure, all the old commands that still exist but shouldn’t be used, the emphasis on storing flakes in Git (I don’t), etc.

You aren’t the first person I’ve seen suggest using niv or npins, I’m tempted to see what it would take to drop flakes and revert to those tools. No channels and no flakes, sounds pretty slim.

  • Is this what you use? You mentioned that you like flakes, what are your reasons for recommending niv?
  • As far as I can tell, and as you mentioned, flakes is the future NixOS is headed towards. Does that make using niv/npins a temporary solution at best?
  • Are we all doomed to switch to flakes at some point? Or do you think it will have evolved into something worthy of replacing niv at that point?
1 Like
17

You mean the --impure flag? If you’re not using flakes everything is impure even without that flag, nix just doesn’t tell you because pure evaluation isn’t a concept in nix 1.0 land. If you are using flakes but still need that flag that just means you have a mistake in your configuration that makes pure evaluation impossible.

There’s some calls for backporting pure eval to the old CLI interfaces, but afaik that hasn’t gone anywhere because the dev team just is too busy and prefers pushing flakes further with the time they have. Maybe with lix & co that might change.

There’s plenty of things not supported in the 2.0 command line yet, for which the 1.0-style commands aren’t just fine to use, but absolutely necessary.

The 1.0 commands that “shouldn’t be used” are disrecommended because they’re footguns with incredibly confusing behavior, especially for newer users. Not using the 2.0 commands doesn’t help with that, they’re just as problematic when you’re using exclusively 1.0 commands.

The 2.0 CLI is supposed to be separate from flakes anyway, though I appreciate they aren’t as cleanly separated as they should be. The roadmap to flake stabilization the community agreed on includes splitting them apart, but well, that will take a long time, and there’s drama around project leadership that we shouldn’t go into. Again, the forks might help with this in the long-term.

Flakes support other backends too. For other VC tools it seems to be limited to mercurial currently, but I’d imagine more will be added over time.

If we’re talking local paths nix will happily consume them without git, too, it’s just subject to this long standing problem - IMO the biggest blocker to wider flake adoption.

Forcing nix to not use git can be nice if you want to avoid nix badgering you about dirty trees, but I’m very curious what other reasons anyone could have not to put code in a git repo. Do you prefer another VC tool that’s not yet supported?

No, I use flakes. I’m not too bothered by the experimentalness since I use nix primarily for personal purposes and stay very in the loop, so the potential for breaking changes doesn’t phase me much. The input schema is useful when I depend on other projects, and I understand the limitations and how things fit together fairly well (and I’m not shy to read the underlying code). Pure evaluation is great. A few things annoy me, but I’ve found my way around them over the years.

I just wrote some 3 paragraphs responding to weird rough edges around flakes that are problematic for you, and make the process of learning to use an already niche and complex ecosystem even more cumbersome. I believe this stuff will be polished away eventually, but well, it’s been like 4 years now.

niv gives you 90% of the flake features a normal user would care about for a NixOS configuration, and is much easier to adopt while isolating you from a bunch of not entirely polished parts of the ecosystem. Flakes also make the official documentation really hard to use, which is a massive problem for new adopters - niv doesn’t have this problem.

That doesn’t mean flakes don’t have some advantages that make me personally prefer them.

I recommend niv over the others primarily because to my knowledge it’s the one with the biggest community share - there’ll be more guides and polish. My call is to err on the side of that for users avoiding flakes, but I’m happy for that view to be challenged.

~ish. Traditional nix isn’t going anywhere. I’d be (positively) surprised to see flakes stabilize without these glaring issues still in tact before another 5 years have passed. Even if you don’t use flakes, with a modern nix implementation you can still consume flake inputs, so there’s no risk of the ecosystem leaving you behind.

Anything’s temporary on a long enough time scale, but I don’t think you have to worry about niv & co today. Besides, there’s almost no switching cost with niv, it’s pretty simple.

I think the problem isn’t really the concept of flakes itself. It’s that there’s such a massive split today. You’re either 100% nix 2.0 or you don’t use it at all. Lots of things are conflated between the 2.0 cli, pure eval, flake schemas, etc. It’s just hard to get the hang of it all in one go.

If the community roadmap is followed the changes will feel much more granular, and the official documentation will also actually cover the topic (without being completely changed overnight). Especially for NixOS users, a lot of rough edges could really easily be smoothed over so people don’t have to look behind the abstraction to understand what’s going on so much.

With proper, official documentation, fewer moving parts and a cleaner implementation I think this won’t feel like such a significant choice anymore. Eventually there could be a simple niv → flake conversion script that instantly makes stuff work without any changes.

There’s a lot of work to be done before the ecosystem gets there, but I think if properly done flakes will eventually feel like a no-brainer. Of course, this means that we need more people who’re willing to do such work, so hopefully in the mean time niv tides folks like you over :wink:

4 Likes
18

As member of the documentation and Nix maintainer teams, I generally agree with @TLATER’s assessment, with a few corrections:

  • Use npins instead of niv.

    niv hasn’t been maintained in years and has a few small kinks that won’t get fixed unless someone forks it. npins authors merge pull requests, and generally it’s more polished.

    I just updated the beginner tutorial to recommend npins: Automatically managing remote sources with niv — nix.dev documentation

  • The current stable release of Nix is 2.x while the “new CLI” and flakes are supposed to be stable for a hypothetical 3.x.

  • Not the whole development team is pushing flakes, it’s really only Eelco working on lazy trees. We did do some stabilisation work on the new CLI last year, but that stalled again for multiple reasons (in my view it’s mainly too little available time and too hard architectural problems). I’m personally focusing on helping users leverage what’s there by expanding and restructuring documentation, and improving the contributor experience so more people learn how it works on the inside and hopefully help fix bugs and add more tests.

  • The old CLI commands that aren’t officially deprecated but really shouldn’t be used because they’re unfixable footguns are nix-env and nix-channel. In my opinion nix profile and nix registry are only slight improvements based on the exact same concepts, and I recommend not going into the imperative realm at all.

In particular I’d like to stress that most of everything that’s old in terms of when it was originally designed is quirky and has more moving parts than needed, which makes the inner workings hard to understand for new users, hard to explain for documentarians, and hard to change for developers without breaking everyone‘s setups or habits. Concretely these are profiles, channels, packages (in the Nixpkgs sense), modules (in NixOS), and flakes only add a bunch more concepts to the pile.

That said, one reason flakes have seen wide adoption is that the new features provide a solution to some important problems, such as evaluation time (with caching) and source tracking (with the inputs schema). But it’s not the only possible solution, and not optimal for a bunch of reasons such as overall not great architecture and maintainability of the underlying code. And I care about that a lot, because we’re extremely resource constrained; getting bogged down by inflexible code on top of that makes any progress very hard.

I recommend to learn Nix bottom-up and using only what you actually understand and can fix yourself. (Concretely: store objects, derivations, and the Nix language are the foundations of it all and will probably never change.) This is the most reliable long-term strategy in the current situation. Don’t hold your breath for others to do things for you. :slight_smile:

16 Likes
19

Thanks for your “peek behind the curtain” from your highly involved perspective and your hard work!

… but that would severely limit my ability to actually get things done using nix(OS) (so I’m pretty much already deeply bought-in to flakes).

But I wouldn’t mind if the future would bring something flake-like that solves many of flake’s problems even if it would be very much backward incompatible.

Most Nix users have already once invested big upfront by learning Nix(OS), I’d expect most be willing to do it again to transition to a 3.0 if it were to bring (many) breaking changes, if the benefits are obvious, e.g. unified handling, solving current (perceived) problems with the status quo (both flakes and otherwise).

Since it seems the major hurdle for progress in the flakes department is backward compatibility, maybe it should be considered to do away with that hurdle?

3 Likes
20

I’ve been using Nix with NixOS as my daily driver for 6 months.

Welcome!

  1. There are many ways to do everything (flakes, flake-parts, flake-utils, custom nix […]

Yes.

  1. Each way requires boilerplate code.

Yes.

  1. Each solution is essentially a bunch of functions that take care of most of the boilerplate, or adds modularity.

Yes.

  1. Documentation is very divided about which way is the correct way.

Not exactly: the official documentation documents stable, mainstream Nix. Third-parties like Determinate Systems corporation provide unofficial guides that recommend non-standard, experimental technology like Flakes.

Because of these things, it feels as if there is no correct way to go about doing things like creating packages, building packages, maintaining custom packages, creating development environments, or even supplying inputs to functions.

Yes, this is a major failure of project leadership. The community is currently working on new governance that will hopefully improve this situation.

This might suggest that it is not a bad idea, at this point in the nix ecosystem, to just write my own custom solutions to the boilerplate problems. Would you agree with that statement?

It depends on what exactly you have in mind. Writing helper functions, for example, is often a good idea.
Trying to replace pins or other more complex unofficial Nix tools I think is a bad idea with only six months experience. I would strongly recommend only using the standard, official technologies until you hit a particular pain point that requires reaching for something like pins in the first place. Some of the unofficial tools like Home Manager offer such clear advantages that many of us want to see them integrated into the official project. The unfortunate reality is that Nix is not mature enough to have all this figured out for you, and you will have to learn and adopt different approaches as you get more familiar with the ecosystem.

This is not meant to detract in any way from the incredible work from the nix community. I think that Nix is the best solution to package management that exists and I’m honored to see it evolve.

I think you are right to ask the questions above and we as the Nix community need to provide a better onboarding / default experience. It used to be much easier to get started with Nix, and I hope one day it can be again.

6 Likes
21

I consider this a gamble. Surely there are people for whom this is true, and surely there’s a huge pool of new or potential users who couldn’t care less for backwards compatibility because they have no code relying on stable features. But we also have on the order of 15 000 users (according to my estimates based on various metrics) and at least 80 companies (according to a list of the NixOS Foundation’s maintains), for some of which Nix is mission-critical. And it’s questionable how important it is to merely have more users, because arguably users who don’t contribute are a “liability” for the ecosystem: they produce more bug reports or questions than they resolve, binding maintainers’ attention – a scarce resource, actually our only scarce resource – that cannot be used for other things.

I would like everyone to be realistic about what one can actually get done with Nix(OS). It’s certainly one of the most powerful software systems in the world, but many things are far from finished, polished, pleasant to use, or economical to maintain long-term. As I see it, the reason that many people, like you, are “bought-in” to flakes is that the feature was literally advertised and “sold”. This is not how technology and science should work in my opinion. I would have preferred to see all that energy invested into stabilising the overall design (or implementing alternatives), improving testing, cleaning up the code, and writing technical documentation so users and contributors can convince themselves what’s worth investing into.

This is not to say that “progress” is impossible or undesirable. We just have to be clear about what our goals and priorities are, what “progress” is even supposed to mean, and weigh that against our individual and collective capacity to make good on any such desires.

Related: Things You Should Never Do, Part I – Joel on Software

9 Likes
22

Thanks for the reply. I’ve learned a lot in this thread and since I made the post, I’ve decided to go all in on flakes, but I’m avoiding all other helping libraries like flake-parts and flake-utils, and instead just writing my own helper functions to achieve what those libraries do. This isn’t supposed to be a critique of flake-parts or flake-utils. I see now what problems they are fixing, and they do a great job. I’m just trying to take @fricklerhandwerk 's suggestion to use “only what you actually understand and can fix yourself”.

I think we’ll get there eventually. Right now Nix can be a difficult tool to learn, and I think that is ok. I found Nix because I was having very nuanced problems with my package manager (apt at the time) that not very many people would actually care about, and I started looking around to find better solutions to dependency handling. When I found Nix I pretty quickly recognized it as a solution to my problems. I never expected it to be a simple drag-and-drop GUI or something like that. Sometimes technical problems require technical solutions. That’s part of growing and moving forward, and the joy of peeling back the layers of the Nix technology can be very rewarding.

I will say that, while I do find many aspects of Nix to be difficult to understand at first, once I understand them, I’m almost always amazed at how elegant and thoughtful the solution is in hindsight.

5 Likes
23

I agree with most, and the points about companies depending on nix and focus of nix-team resources are indeed very important to make. I’d hazard a guess though that many that rely on nix(OS) for more or less mission critical systems might actually be using flakes too.

Just brainstorming here: if one would agree that the concept of flakes is actually the future (and its problems mostly implementation-related), could it be an alternative to at some point simply call the “status quo” stable and only spend efforts on necessary backports, and focus most innovative/dev efforts on something like nix 3.0 / flakes 2.0 or nix-ng / flakes-ng that doesn’t necessarily need to be backwards compatible?

Considering how conservative other stable distros are this would give quite some time to “freeze” status quo and focus efforts elsewhere. And given the nature of nixOS one could even run complete nixOS vs nixOS-NG installations as bootable generations for dev/test/avant-garde scenarios.

Maybe such an approach could be an implementation of:

I would have preferred to see all that energy invested into stabilising the overall design (or implementing alternatives), improving testing, cleaning up the code, and writing technical documentation so users and contributors can convince themselves what’s worth investing into.

24

That is essentially what Eelco was evidently planning for in the past 5 years, and I think this is the Joel rewrite scenario. I don’t see how we’d have the resources to support a legacy system and develop a sufficiently new one at the same time. Look where we are right now: There’s essentially two Nixes baked into one, none of which is clearly superior, none of them clearly moving forward, and most new users are (rightfully) confused. Each of these systems is too large to maintain with what seems to be a total of 20h/week, while 20 issues or pull requests are added to the queue each week. There’s reasons for all of that, and again the question is what we, collectively - maintainers, contributors, users - want from all this and what we’re willing to invest to get it.

I’ve been arguing for a long time, following what @Ericson2314 has been arguing for even longer, that Nix should be a lot smaller, such that people can build their own use cases on top of it instead of piling more responsibilities into a code base that can barely sustain itself. Just making that transition would be enough work on its own. And all that would be to merely make sure that we don’t break people’s code that relies on what’s there, and prevent more people from relying on code that we can’t possibly afford to keep maintaining forever. Before we get that sorted, I can’t even imagine how we can make upstream innovation happen responsibly. For now, innovation should be for the ecosystem to have fun with.

Take Tvix for comparison. @tazjin has been rewriting the Nix language evaluator bug for bug since 2020, and now @flokli is rewriting the Nix store bug for bug. Sure, a greenfield project allows for a better architecture and that enables more agility and leaves room for innovation, but even after all this time, Tvix is still not a drop-in replacement for Nix (maybe that’s not intended though).

7 Likes
25

I realise this is very dangerous (just thinking out loud here), but

support a legacy system

in this case seems to be the thing that prevents nix from “doing the right thing”, as in

stabilising the overall design (or implementing alternatives), improving testing, cleaning up the code, and writing technical documentation

… and maybe also allow for

Nix should be a lot smaller, such that people can build their own use cases on top of it instead of piling more responsibilities into a code base that can barely sustain itself. Just making that transition would be enough work on its own

In other words, for the sake of argument, if one were to really put the former (“legacy system”) on the backburner, in favour of the latter, there might be more capacity for the arguably needed “modernisation drive”?

Very dangerous indeed since this would basically constitute “abandoning” part of the ecosystem in its current form. But, as a thought experiment: say all core contributors of nix and top-level architecture would abandon “nix-legacy” and focus on “nix-ng”, that would leave us with status quo for nix and nixpkgs for, say, a year, or two? Non-core team could still be updating packages in nixpkgs, and (I think) actually many prolific nixpkgs maintainers are not necessarily deeply involved in nix and NixOS modules. So one would “get by” in the meantime with nix as-is, and nixpkgs being partly in “maintenance mode” (stable/backports) and partly community-updated (unstable).

Obviously the glaring risk is the time estimate above, so I’m not necessarily advocating for this but just wanted to make such thoughts part of the discussion and get a feeling for the overall strategy and sentiments in the core-community.

Just as backround: I’m basically already betting full-on on Nix(OS) for my personal and my startup’s infrastructure/dev needs, so on one hand I’m dependent on (some) stability, but on the other hand I’m very much interested in nix(OS) becoming maximally robust and future-proof, and I’d surely be willing to put up with some hardships in transitioning if it would serve the long-term goals.

26

@ppenguin there are, I’d say, 3 things here, not 2:

  1. Technical quality deep within the bowls of Nix
  2. Next gen User-facing “icing on the cake stuff”
  3. Legacy maintenance

My view, and @fricklerhandwerk’s, is that 3 and 1 are the priorities, and 2 can be done by other people. So it’s not like we are abandoning the brave new world, but we’re trying to lay the foundation while doing maintenance, and someone else can erect a building on top.

6 Likes
27

@Ericson2314 Thanks for your clarification. I just got the impression from @fricklerhandwerk that (mostly) regarding (1) there’s still challenges to decide the focus and the direction, as I perceive/understand it mainly between “let’s organically improve” vs. “let’s try to do it right (more radically) with what we have learnt until now”, where the second option might very well mean to (have to) give up (some?) backwards compatibility. And that there’s no resources to do both.

2 Likes