What should stable NixOS prioritize?

here are two cents of mine:

summary: +1 to optimize on people’s time and release May/November

background
I use NixOS in ‘production’ (my own mail server) and as daily driver (laptop)
for production: I want it to be reliable (only security updates or backports of broken packages)
as daily driver: I use EXWM so I personally do not really care about the big DEs, but my environment should just work.

If shifting the release months helps the NixOS developers, then that’s perfect.

5 Likes

I use NixOS as my main Linux OS. I’m not that experienced, but declarative configuration makes my life way easier than imperative.

I don’t mind tinkering and often do, but having a ‘stable default’ (plasma desktop) that you can just go back to, where just everything works, is a big point for me.

Perhaps also aiming GNOME was a wrong decision? It think it’s a lot of work to target two huge DEs. (but perhaps people willing to work on GNOME are wholly different people willing to work on KDE.)

5 Likes

R.e. server vs desktop, I would say that desktop users / daily driver users should be on the unstable channel. There is simply too much desktop software for backporting to be realistic with the current workflow. And considering that Arch Linux and its derivatives are often recommended for desktop users, the rolling release model can work well for the desktop. There is enough quickly-expiring software (browsers, chat clients, etc.) that the stable version is more often broken than unstable, in terms of usability.

I think the real issue though is the release being delayed due to strange software problems. This is similar to unstable being delayed (New Merge Policy for an always-green hydra). In both cases, the only real solution seems to be to ensure someone is actively working on fixing these failures; maybe the NixOS foundation could hire a release manager, or put up bounties on release-critical bugs.

2 Likes

Although I agree in general. I don’t think unstable is at the point that it’s realistic to expect people to put with the occasional breakages.

Yes, but in what I proposed above, it’s easier to just package the latest than having to troubleshoot our “unusual mix” of software versions.

All I really do is just polish docs and help with the process. I think having a reproducible process is more important than hiring a personality. As I mentioned Marketing Team: Can we present Nix/NixOS better? , we create a pretty difficult situation to release on time, and our release date always slips because of it.

2 Likes

Given how NixOS is structured and developed, I am not sure running on unstable is for everyone, maybe stable boot system / DE configuration with high-churn applications (which will not be that stable or that reliable in real world whatever version you use…) installed from unstable, and maybe some high-impact high-quality well-stabilised software installed from unstable.

I usually rebuild my system from master, though. Maybe rolling back if it gets too broken… But I also split the package set into many shards, which can be rolled back or delayed independently.

A post was split to a new topic: Package set shards

You release managers convinced me! Until the upstream projects test more comprehensively, we’re best off aligning ourselves to their ways. There’s ways to use any freed-up contributor time than fighting to make an odd combination of these components work.

Perhaps there will be a time “post wayland” when things settle down, but I get the sense right now the guts of mainstream Desktop Linux are rather unstable (in the original sense of “shifting around”, not necessary “broken” :)). This state of affairs increases the costs of non-alignment.

I absolutely agree with what @Mathnerd314 is saying about us needing to improve CI and what-not so we aren’t constantly shooting ourselves in the foot, but I think this is a little different, because I doubt it will be possible to incrementally update these desktop components while not breaking things — there is no magic “grey code” upgrade path.

  • First of all, even the most ambitious testing plans don’t call for the necessary suite of consumer hardware and CIing physical interactions (does hibernate work? Is there tearing? etc.)

  • Even if we did all that, we might be stuck contributing to the underlying packages much more than we can afford to actually fix discovered issues.

3 Likes

Overall GNOME is and has been easier to upkeep than plasma and has a lot more purity atm

6 Likes

I’ve already mentioned above that having an eternally rolling channel for desktop distribution isn’t going to work really well in nixos because we for 1. can’t move fast enough and 2. for GNOME using desktops the stack changing will make maintaining unstable pretty hard or broken for periods What should stable NixOS prioritize? 3. Having a stable release with a frozen base of packages allows us with our resources to provide a better experience

I’m not sure if the NixOS foundation, being a non profit, can hire people so they can make profit to work on something? idk, I would have to check but if I was being payed to do releases it probably would have been a slightly different story.

2 Likes

I’m not sure if the NixOS foundation, being a non profit, can hire people so they can make profit to work on something?

I’m pretty sure non-profits can have employees (and contractors)…

1 Like

NixOS isn’t really a product you can sell.

You could have have consulting around it, and that’s essentially what tweag is.

However, I’m all for hiring a NixOS/Nixpkgs full-timer. It would solve a lot of issues with community engagement to have someone review PRs, respond to issues, be aware of most things at a high-level.

I spend 20-40 hrs a week on nixpkgs, and most of that is “maintenance”, I’m no longer making many PRs or commits. I would love to go back to be just a “contributor”

5 Likes

Hehe, I felt similar, you cannot make commits like that when you’re a RM. Lol, if the foundation is interested and having two people at least part timing they’re before you :grin:

1 Like

I would like to share my view to this topic.

First I would like to state that I really value the work that everybody is putting into make NixOS a great Desktop distribution. I use it for years now (9years) and I can not imagine using something else.

Also I think we all agree that managing a Desktop distribution is a lot of work. I have witnessed numerous super human efforts into what NixOS is today. This kind of efforts also burned out many contributors in the past (including myself). I don’t think this is a sustainable model going forward or at it is at least very unhealthy.

I do think postponing the release for 2 (or even more) months - as proposed by @jonringer - will offload some of the desktop integration work to other distributions. I think this is the easiest change we can do immediately (eg. next release) and observe if we are next release are going to be easier to do.

I do hover see that in order to improve NixOS we need to increase our contributors. And since the contributors are first users and then contributors we need to first increase our users. I don’t think focusing on Desktop will improve substantially, only because the market for linux Desktop is not as big as - for example - Server market is (this also includes all the cloud stuff). Also the problem space with Server version of NixOS would be much smaller and more achievable for smaller teams like NixOS.

I’m not entirely sure how to get to this Sever version of NixOS but here is what I would propose:

  • NixOS release process stays the same, it is not nice to break expectations

  • create NixOS Server (or NixOS Minimal or NixOS Cloud) version of NixOS which is released 2 months before NixOS, only to be used as a base on which Desktop stabilization is going to happen.

  • we would also need to create a new channel which is between “unstable” and “stable”. We might even call it “server” (or whatever this version of NixOS is going to be called). For certain period “server” and “stable” will actually be the same and we can mirror them, so there is no need to back port to both branches all the time, just in those 2 months when “server” and “stable” channel are different.

I think above solution would solve the problems mentioned in this thread so far (at least to my understanding). Also it wont break the current way of doing things if we give it a try.

I find it a bit funny that the only way to help Desktop version of NixOS is to de-prioritize it and focus on Server version of NixOS first until our user base grows and is able to support the effort. But I do understand that not everybody sees it this way and want to work on Desktop version of NixOS today and we shouldn’t prevent that (or make it harder) in any way.

EDIT: probably off topic but I think there should be a release team behind release manager which would help resolving release related issues. I think that would make the job of release manager much less stressful and manageable.

7 Likes

Also, I would personally like to see a ‘mini-ZHF’ between releases; where we take 2 weeks just cleaning up some the “packaging debt” on master. Not having to deal with two divergent branches would make it a lot easier to iterate.

We could just make it into a small community event, have documentation ready on finding, fixing, or marking broken packages. Then at the end, we could do a post on https://nixos.org/news.html with:

During the NixOS 2021 Spring Cleaning, we were able to fix X, and remove Y broken packages. We want to thank the following members for their contributions:
58 PersonA
32 PersonB

These interim fixes go a long way to preparing us for the stable releases and keeping Nixpkgs the best package repository in the world.

And if github had the ability to give people flairs (similar to member or contributor, but arbitrary). Then it would be kind of cool for people to “collect” badges.

13 Likes

I don’t believe GitHub does, but Discourse has custom badges.

2 Likes

@garbas While yes I do want more contributors / users. I’m not sure that’s the solution for this.

For one, if we have news users -> new contributors, but not process changes, then we have more PRs -> more backlog -> more straining to stay afloat.

More broadly, I really do believe increasing capacity at the tall part of the distribution (via paid employees or productivity improvements for existing core contributors) is qualitatively quite different than increasing the long tail of occasional contributors. And by the standards of other git repos, at least, we already have an excellent long tail.

Also I think that neither server or desktop users is where we are loosing mindshare: I think the “Nix for Ops” story, while it can be improved, is pretty strong at the moment.

Rather, I think it’s “Nix for Developers” that is the weak. There’s a few angles here, but for the purposes of the conversation at hand, I think the issue that Nix doesn’t have enough mindshare among the upstream projects that we’re packaging.

I don’t get the sense that key upstream systemd, gnome, KDE, whatever devs are really aware of us at all, and I think that hampers things. I’m not wishing that they privilage us over other distros—that won’t happen. But I think just having them aware of us could help in a few hard to predict ways. Just taking some guesses:

  • Maybe Wayland dev eager for X to die would help us write some new NixOS modules
  • Maybe we could help upstream projects do CI, development environments, etc. Upstream dogfooding would have so much amazing effects.
  • Maybe things like OSTree and Flatpack could integrate with us better.

All this stuff could help with big issues that a longer tail of contributors are less to be able to chip away at incrementally. Also, if we are lucky and get any “high profile converts”, they are likely likely to both have outside impact and help bring in more users.

One last thing is that despite Red Hat getting it’s money from servers not Desktop, I’d say much of the “Poettering Clique” stuff that has driven most of the change is often quite desktop oriented. So being more in the loop on the desktop side of things is good for being on top of Linux userland issues in general. (Servers may not care about DBus but they do care about init.)

9 Likes

What is the intended purpose of delaying only the desktop release by two months instead of the entire release?
Is it so that server users don’t have to get used to a changed schedule?
My understanding from this thread is that the issue of desktop fixes holding up the release would be mostly solved by shifting it to a more convenient date, so I’m really not sure what a separate version would accomplish.

The majority of my time spent in nixpkgs has been, wow we’re really strained, and a lot of our core devs are incredibly strained. I’ve made many mentions that, as same as you’ve mentioned, without process changes we’re not going to handle such growth because we’re struggling to sustain the status quo.

I’ve been working on this maybe happening for upstream pantheon/elementary. Since they use GitHub it makes it easy because of the cachix actions. Overall, if we want to improve the ease of integration of these projects, it’s going to mean the same means as why, for example, Ubuntu adopting these things are not too difficult. They have them in their CI, they catch stuff and help them resolve things. I completely agree that, not just time is going to make this better on all fronts, it requires communication and being proactive.

4 Likes

For one, if we have news users -> new contributors, but not process changes, then we have more PRs -> more backlog -> more straining to stay afloat.

One thing you might miss in my post which you didn’t include is that the surface is much much smaller. Especially the QA portion of the effort. Services are much easier to test then desktop applications and by easier I mean you can use automate it and for desktop applications a lot of times a manual testing is required.

More broadly, I really do believe increasing capacity at the tall part of the distribution (via paid employees or productivity improvements for existing core contributors) is qualitatively quite different than increasing the long tail of occasional contributors. And by the standards of other git repos, at least, we already have an excellent long tail.

But those payed employees actually come from using NixOS on servers. I know all my paid projects involving NixOS were about running NixOS on the server in one or the other form.

Also I think that neither server or desktop users is where we are loosing mindshare: I think the “Nix for Ops” story, while it can be improved, is pretty strong at the moment.

I wouldn’t agree that NixOS DevOps story is strong. The solution is far from polished. We have many half bakes solutions that all lack that last 10% of the work. Having releases on time is just one item in the that 10% bucket.

Rather, I think it’s “Nix for Developers” that is the weak. There’s a few angles here, but for the purposes of the conversation at hand, I think the issue that Nix doesn’t have enough mindshare among the upstream projects that we’re packaging.

I agree that this is an issue, but that is a separate topic. I think efforts to make Nix great for developers are just around the corner. You can see that messaging on the website changed into that direction already. Eelco is working tirelessly to bring new Nix UI out. Documentation is being added in a form of guides/tutorials targeting mostly Nix audience at the moment. So things are already in motion on this end.

2 Likes

Change in the schedule is not a problem. Problem is that deadlines are important for server users, since you need to plan your activities and postpone others.

I don’t think it was said that this will solve the problem, but it would definitely help with requiring less effort. I think shifting the release schedule for 2 months is the least we can do and this can be already done for next release.

What we could try - maybe in a year - is to roll out a DevOps focused NixOS release (separately branded then the current release) scheduled at the time that suits DevOps world and maybe even helps with the stabilization of the main NixOS release.