The Myth of Stable Unstable

Help
1

Well, yet another “should I give up NixOS” or “should I go stable/unstable” post.

Before start, I’d like to point out that what I am talking about is desktop use, which means you install lots of random softwares that could sometimes be very niche, and you often need to get things done ASAP.

NixOS on server is fantastic, but not the focus in this post. Again, this post is only about desktop experience.

I’ve been using NixOS on desktop for ~3 months, on server for longer. I started my desktop experience with 24.05 stable, and switched to flakes&nixos-unstable, for the reason that IME/fontconfig/Wayland/… support is much better on unstable. As I am building up more and more complex setup with my real-wolrd life getting busier, troubles come.

The Release Cycle of nixos-unstable

Wherever you make a post asking about which channel to go, there will be people telling you “unstable is really stable”. A recent survey says it’s almost half-half in total, but I believe unstable takes a bigger part on desktop.

In a traditional rolling distro, when a single package breaks, what the maintainers do is very simple: fix and test themselves, push to the repo, wait for mirrors to sync, user upgrade. I admit there are bigger problems and more complex models, but what’s happening the most often are small mistakes, like upgrade through CI but the package build need some little edits.

In nixos-unstable things are a little different. Something breaks, people submitting issue - then here comes the fun part: Sometimes it’s too niche, no one can fix it (usually happens with a build system that require networking and FHS); Assume it’s fixed, it takes a few hours to a few days to be reviewed and merged. Things gets worse when the fix is part of a bigger treewide rewrite, which is not merged so quickly. Assume it’s merged. We must wait 4-5 days for the CI and cache until it’s really available on nixos-unstable.

Pessimistic speaking, that could be weeks in total.

What’s worse, while you are waiting, another package breaks. Congratulations, keep waiting until you find a window period that you can finnaly upgrade.

In nixos_zhcn telegram group we have a bot tracking when PRs are merged and pushed to staging/master/small/unstable. Part of the use case is tracking when they can finally upgrade without breaking things. In the recent weeks it seems to happen more frequently, every nixos-unstable upgrade is like patients in bed waiting for potent medicine.

I know, I know, there are workarounds, which I am already using as well.

You can simply copy it to your configuration and callPackage or use an overlay, if the fix is not complex. You can add master or unstable-small to your configs and pick some packages from it, if you have the resources to build. You can add an old rev of nixpkgs in your config.

These don’t always work. Sometimes a module doesn’t give you the option to choose package. Sometimes what’s broken is down below the abstration so that you cannot simply modify.

We build all the abstraction of Nix to upgrade “seemslessly”. But here we are spending more time on the abstraction layers per se.

I gonna say something that doesn’t feel right: Archlinux has a news channel for possible breaking changes, which instructs the users how to fix manually. Some nix guy criticizes this for “not maintaining things well and dump their responsibility to the users”. But look above, isn’t that what we do?

I wish we have a news channel for unstable, seriously.

Containers: Escape hatches

Nix solves the problem that podman/docker tries to solve in a way more elegant way.

Guess you all have heard about this.

Python + CUDA. Do I need to say more?

Or any other thing that is too complicated or poorly maintained to use in Nix.

I admit it is possible to figure out how to get everything working on Nix. But just “waste” some space for a container you will save plenty of time. It’s all about time cost (can’t remember the abbreviation phrase).

How many packages do you have in your flatpak/distrobox?

A very interesting finding is that, while all immutable distro I know are focusing on the concept of “minimal immutable host + application container”, NixOS community is, on the other hand, trying to install everything on host even when it’s super complicated on non-FHS.

Migrating to Stable?

Personally I am planning to migrate my desktop as well as all my vps to stable after the release of 24.11. And put things in flatpak/distrobox as much as possible on desktop. Yep the “immutable host + application container”.

But as I recall how I started with 24.05, I hesitate. Wayland, Nvidia, IME …

Every year is the year of Linux desktop.

While enjoying the freedom and customizability we cherish, we probably already forget what a complete desktop experience feels like. Got used to dealing with the glitches without even realizing it.

Desktop experience is evolving really fast, which means we need to keep moving forward. Point release cycle of half a year is still too long. As well as the gaming, performance cliché. Maybe releasing every month like tumbleweed is a more desirable sweet point.

May be I should simply install Archlinux. Nix as a package manager is available everywhere anyway.

Conclusion

I write this post not to criticize the selfless maintainers of an open source project. I have the highest respect for their work and I am learning to contribute as a newbie as well.

The point is that we should see the limitations.

No one should be blamed for the terrible experience of mentioned “upgrade window”, but this doesn’t mean the problem doesn’t exit.

All these words are from the perspective of a unmature linux user. Any criticize is welcomed. Maybe you can share you “best practice” as well.

2

Here’s the open secret: nixpkgs does not have a channel that’s verifiably stable.
For 90% of the year there’s really only two viable channels to use on NixOS:

  • unstable aka nixos-unstable, or
  • stable aka nixos-YY.MM (replace YY.MM with the most recent May/November release, currently 24.05).

All changes to stable must go to unstable first (via some other branches) and then get backported via label or a manual backport.
If it’s a browser update, it usually gets backported for security reasons.
For other security changes, it should get backported, but there’s no guarantee.
And for really any usability changes/bugfixes that involve breakages, they cannot be backported by policy. Morever even backportable fixes, are often forgotten to get backported in the first place.
And from experience, I would say most contributors to nixpkgs are more likely to be using unstable, as that’s close to master (the branch PRs are made against).
So practically, stable is like an abandoned branch that sometimes gets attention.

However, unstable isn’t stable either, since the philosophy often comes up of “unstable is meant to be unstable, let’s just break things”. So you get wildly breaking changes within days/weeks of each other.

Hence there really isn’t a great channel to sit on if you want to avoid breakage while still having a not-so-buggy system. It’s a matter of tradeoffs, and for me personally, keeping close to the channel that actually gets fixes (unstable) is where I realistically feel is more manageable for me.

We have at least 2:

https://nixos.org/manual/nixos/unstable/release-notes

That’s a meme that misunderstands docker and nix, funnily enough.
Docker and nix do different things (containerisation vs correct deployment), though docker can be abused to pretend to do nix’s job.

2 Likes
3

That kinda exists in the form of the release notes, doesn’t it? You don’t have to wait for releases to read them.

Changes will still slip through without a release note from time to time, of course, but that’s no different from Arch’s news channel.

In fairness, I think they are sometimes a bit more neglected than they should ideally be, and backfilled during release season. Maybe this is where improvements can be made.

One, at this point, inkscape, and not because the nixpkgs package doesn’t work. Over the years my software use has stabilized. Flatpak is very cool, and I’d still love to use it if it worked, but ironically I find that so much software also just doesn’t play nicely with flatpaks, and portals have always been semi broken for me. NixOS integration tends to be better for what I end up using.

nix-ld fills the gap when I need stuff to just work in the short term.

I can definitely see people heavily using ML stuff struggling. That said, just using straight pip+nix-ld and just (partially) skipping the immutability for that ecosystem - while utterly heretical - is probably a good middle ground if you frequently run into issues there.

I find NixOS stable + a custom nvidia package works just fine for me. Sure, evolution is rapid, but wayland and nvidia have finally reached the sweet spot where stuff kind of just works for me.

Gaming performance is a cute thing to optimize for if you find it fun, but even then driver updates won’t massively impact that. Still, as much as nvidia causes problems from time to time, it’s a once-every-few-months issue that I’ve gotten very adept at dealing with, so I probably spend like 10 minutes on it every 6 months.

I limit my use of unstable packages to, like, wlroots+sway at this point, which is quite comfortable: dotfiles/nixos-config/hosts/yui/nvidia/default.nix at a4ba1914e337f50e70b0a5c81dc6334e1ae16273 · TLATER/dotfiles · GitHub

Honestly, 99% of the desire for unstable seems to be just that doing anything with an nvidia GPU kinda sucks. Most other parts of the Linux desktop experience work just fine if you operate on 6 month cycles. The other 1% is waiting for small features here and there, but tempering expectations isn’t so hard when it’s not the difference between barely functional like it is with nvidia.

This is unless you develop, of course, but there you always have the escape hatch of nix-ld.

Bottom line, my opinion is that people run into far fewer problems on stable. It’s unreasonable to expect the same from unstable, I don’t understand why people keep being upset at the thing literally called unstable being unstable. Small mistakes and breaking changes happen, and you won’t always have warning. If you’re not ok with that, don’t subject yourself to it.

You’ll also notice that issues grow fewer as you use the distro for longer - three months is not a lot of time to find your sweet spot.

Folks looking for the year of the Linux desktop experience on the other hand would probably be happier in Fedora land. NixOS - while it has the potential to be the underlying tech for something more hands-off - is a build-your-own-desktop toolkit distro (as is Arch), and it simply takes a few months or years before your config reaches a level of stability where you don’t really need to touch it.

But I get the feeling that you actually want to build your own desktop :wink:

2 Likes
4

I switched from unstable to using releases years ago, mostly because I don’t like dealing with occasional breakage on upgrades when I ought to be doing something different. The rolling release model does not work for me, and I prefer having all breaking changes which require manual intervention bundled up into a twice per year event.

So to me, this is more a communication/advertising issue in which people create expectations about the stability of the unstable channel which do not hold true. And I agree with you that the news mechanisms for unstable breakage could be a lot better though.

7 Likes
5

Well there’s only 64 replies in the first post, which is from 2022. Second one is a release note. Good to know we have these, but that doesn’t touch the problem here because breakages happen way more often than that.

Not very realistic but I was expecting something like “package X is broken, to fix it you need overlay Y until the PR Z is merged”.

Breaking changes are obviously expected in unstable, and they are timely announced. But between the breakage apeearing, and all related packages being fixed and available in cache, there’s a significant time lag, which ruins the experience.

For example, in the last month I’ve compiled xwayland 3 times for god-knows reason. Some other things: tlp, yazi, steam, all telegram forks … breaks. All can be workarounded, but browsing the issues for a weekly upgrade is too tiring.

6

Yes I’ve mostly seen it used for substantial breakages, and a more detailed list of breakages should be mentioned in the release notes.

Would be nice to have I guess. Usually there is a comment on the PR or issue on gh, but a dedicated place to have that info could avoid the inevitable spam on the PR/issue.

Then use stable :slight_smile: Again it’s a matter of tradeoffs, unstable does require more hands-on fixing of issues if you want it immediately fixed.
But also I don’t think it’s the end of the world to wait a week or two for fixes, given that nixpkgs is going to be far ahead of all non-rolling release distros by nature anyway (caveat: for packages where the update script is maintained or the maintainer is quick about updates).

1 Like
7

Well use cases can vary a lot.

Steam has seirous issue with CJK fonts, which is fixed in middle of this year on NixOS. And I am also waiting for Electron apps (tried to get rid of but there are still a few remain) to bump to Electron 33 where text-input-v3 is finally supported, which means you can finally TYPE in wayland if your compositor is not kwin/hyprland. Same way to Xwayland and fractional scaling. That’s very basic needs, to see text clearly and type text.

Some people may already find their experiences comfortable early in 10 years ago. But as I already emphasized, desktop use case has more niche stuffs. People go cutting-edge for practical reasons.

1 Like
8

Fair. But this is the second part of my opinion; once you’ve hit these edge cases that revolve around your use cases (CJK fonts, electron, …), and resolved them, you don’t have to do so again. Picking a few packages from unstable for a year or two, and then slowly going to low-maintenance seems to be how NixOS goes.

1 Like
9

I basically agree with you.

Writing this post because the “wait a week or two for fixes” happens too frequently recently. They kinda overlaps.

I feel it is a pity that people still have to wait so long after the maintainer finds out how to fix it. I wish It could be better.

10

I will add another caveat to this - mixing stable and unstable is pretty much unsupported, so you are even more on your own* than using just one channel exclusively. And of course having multiple nixpkgs instances comes with its own performance and incompatibility problems, especially for GUI apps and “core” dependencies - complaints around glibc or qt version mismatches are common here.

One other option for people that want to use stable would be to use a scheme similar to what I use for unstable - a long-running fork of nixpkgs wherein I can pull in PRs and make my own fixes directly in the code. Updating the branch consists of fetching the corresponding upstream branch (either nixos-unstable or nixos-YY.MM) and then rebasing upon it.

The main benefit is some familiarity with the codebase and being more resourceful than having to simply wait for others. It also could result in less work than using unstable directly since you avoid the frequent breakages of unstable, while allowing you time to essentially backport your own fixes that you care about. The downside of this approach is that less is cached by Hydra and more is built locally, of course, but it prevents the abovementioned version mismatches. Additionally since it’s an untested zone, you would have to figure out any fixes on your own*.

That’s fair, depending on your usecase. I don’t have a good solution other than what I do regarding the long-running fork, which I recognise is even more manual work and puts you on your own* in a different way.

*I mean you can ask here on Discourse if you need help of course, but I mean it’s not something that nixpkgs itself will concern itself with supporting.

(Also, another unfortunate thought I just realised: nixos-unstable is not actually supported by nixpkgs - really the only supported branch is master, which is also one of the worst branches to use for a NixOS system, since nothing that changed recently is cached or tested.)

11

This is true, but as long as the number of such franken-NixOS violations is low, chances are quite low that this actually results in breakage.

Much lower IME than running unstable directly, anyway.

12

I am mostly on unstable because I want to have more influence on the changes on the services I care about and see early when changes break things.

When I want to focus on something and not deal with new bugs, I just don’t update my flake or fork for a few days.

1 Like