NixOS “stable” branch

1

Sorry, bit of a rant.

What is the point of a “stable” branch when it is anything but stable? There is an issue with inetutils that currently blocks me from upgrading my machines: inetutils: build failure · Issue #488689 · NixOS/nixpkgs · GitHub . And this is not the first time in recent memory. There was an issue with the tailscale package not that long ago that forced me to compile a kernel with custom patches in order to update: tailscale: Build failure with portlist tests on NixOS 25.05 - "seek /proc/net/tcp: illegal seek" · Issue #438765 · NixOS/nixpkgs · GitHub . That took weeks to resolve.

I know, beggars can’t be choosers. NixOS and nixpkgs like many open-source projects rely on the efforts of unpaid volunteers for a lot of the work. I do not want to blame anyone here. I definitely could be contributing more myself.

My question is this: Is this even recognized as a problem? If a package breaks why is it not rolled back immediately until it is fixed? What good is the CI system if you integrate anyway even if there is a basic build failure?

There is also no transparency around this. What even is the plan here? When can we expect a fix? Who is taking responsibility? What could I do to make something happen?

This lack of transparency in general around processes and responsibilities is by the way one factor (alongside a family and a full-time job) that keeps me from contributing more. I never know what exactly it takes to get anything merged.

I sometimes seriously contemplate migrating most of my stuff to Debian and my gaming desktop to Fedora but I just cannot live without a declarative configuration anymore. Is there a NixOS derivative that takes stability more serious that I somehow missed? Or is Guix System any better (although I like the Nix language and prefer it to Scheme for this purpose)?

Edit: typo

1 Like
2

Channel blockers only include “critical” packages - ‌the rest are built prior to advancing ‌thebchannel but not considered blockers if ‌they fail. Arguably inetutils failing should be a blocker but you’d have to request it.

Stable branch isn’t guaranteed to be 100% bug-free in any case, backports are generally only done for “non-breaking changes” but it’s possible something was missed - and some packages (like kernels) must be upgraded for security reasons anyway, so that can easily be a vector for breaking changes.

3

Arguably inetutils failing should be a blocker but you’d have to request it.

Where would someone request that inetutils failing should be a blocker?

Stable branch isn’t guaranteed to be 100% bug-free in any case, backports are generally only done for “non-breaking changes” but it’s possible something was missed

In the case of inetutils it was to resolve the recent telnetd CVE. It makes sense to back port that, but at the same time if that breaks multiple systems it feels like there’s not much point.

1 Like
4

I’m also suffering from this issue but I would like to emphasize that I love Nix/Nixpkgs/NixOS. I don’t think there is a fundamental problem with stability in this community.

5

Well, that’s the problem. inetutils in nixpkgs doesn’t even have any maintainer. So “random people” have to keep stepping in e.g. to fix CVEs.

2 Likes
6

And darwin builds in particular are tough. In my experience they keep getting more regressions than linux builds, and we seem to have insufficient “darwin contributors”. So people who do not want to touch darwin (like myself) have to keep stepping in to at least help to fix the worse issues (say, unblock channels and mass failures), as the active darwin people are too few to manage.

4 Likes
7

Well, that’s the problem. inetutils in nixpkgs doesn’t even have any maintainer. So “random people” have to keep stepping in e.g. to fix CVEs.

Ah, if it doesn’t have a maintainer that makes sense. I submitted a pull request since no one else had, but since its my first one its a bit sloppy and is taking some time to get merged :sweat_smile:

1 Like
8

I want to add that I also love Nix, NixOS, nixpkgs. It’s just that I am frustrated because something like this has now happened twice in less than half a year.

I know that “stable” does not mean “bug-free”, but a single package breaking should not prevent system updates. It should not matter who the maintainer of inetutils is. If a commit breaks a package, it should be reverted and the impact on anything else should be minimized. If I break main at my job, I or one of my co-workers will revert that commit. Ideally, CI would catch this before it ever gets merged. Doesn’t matter how beneficial the commit theoretically would be if it would actually work. I don’t know what’s controversial about that. Or am I misunderstanding something? Is the issue that this only breaks on darwin? Even so, that’s a separate release branch (nixpkgs-25.11-darwin) and the commit could get reverted separately for that branch.

After re-reading this, I think my stance can be summarized the following way: every package should be “critical” on the stable channel. Otherwise it is by definition not stable. Breaking packages should only be tolerated on the unstable channel. If any package is still broken before a new NixOS release, it is either important enough to postpone the release or it should be thrown out. I am honestly suprised that anything else has ever been seriously considered. What is the point of CI if you simply ignore it?

Edit: Obviously, allowances should be made for flaky tests. Those should be disabled temporarily and should be fixed in the medium term.

I have actually looked into doing something about this particular package, instead of only complaining. I am thinking about adding myself as a maintainer to the package. But I don’t fully understand the project and package yet. The patches come from a Codeberg repository that is not mentioned on the upstream homepage. Is that even an official repository? In any case, the patches do not seem to be part of any official upstream release yet. I would probably not have included them in the first place. My stance as a maintainer would be something like:

  1. Only official upstream releases go into the unstable channel.
  2. Only official upstream bugfix releases go into the stable channel.
  3. Only if there is a serious bug or vulnerability, I would consider updating to another upstream minor release or backporting a patch from an upstream release.

Would that be an acceptable stance?

1 Like
9

Yeah, I think everyone in the community is aware this is a problem. You’re not really offering any solutions either, though:

The reason they aren’t is because hydra cannot feasibly build all the 120000 packages for every commit.

We also have stuff like this in the repos, which I don’t think anyone seriously thinks should be a release blocker.

This too isn’t feasible; the thousands of python packages nobody uses that are mirrored into nixpkgs would be continuous release blockers, and if they were simply thrown out lots of higher-level packages would need to be removed.

nixpkgs is simply much, much too big. It’s impossible to apply standard CI practices to this repository, it’s a very different beast from other software projects. Even applying other distro’s practices doesn’t work well, since we often cannot reuse existing binaries, which makes the build overhead of each individual commit much greater than on other distros.

I do agree this is a problem, nixpkgs needs at least some metadata to expose the maintenance guarantees of individual packages and modules. Whether something is a release blocker or not - and what the criteria for being a blocker are - should be clearly advertised. It shouldn’t be a game of blame-guessing to tell whether a package has an active maintainer, at least for “important” stuff.

There’s just too much variance for anyone to really keep track of what is and what isn’t reliable. Realistically, I’m in the camp of “nixpkgs needs to be split” - there should at least be a “core” set of packages that actually has guarantees; if the rest of the accretion disk around it sometimes breaks, at least it’s clearly projected that nobody is looking too closely at it.

Having multiple different sets would make a ton of sense too - y’know, a set of packages actively maintained without which the distro is considered defunct, a set of packages which have had someone reasonably reliable assigned to them for a number of years, and then all the cruft that random drive-by contributors added over the years and where you have no idea whether it’s even working or not.

This does kinda exist in practice with the release blockers, but that list is not exactly designed to fit any particular purpose, and since it’s not exposed to users it doesn’t really mean anything.

Alas, words are cheap; this discussion happens regularly. You’ll start shaving yaks about flakes or not flakes and other things long before you get to a real structured reorganization of nixpkgs. For the foreseeable future, nixpkgs will stay as it is.

I doubt anyone would object to this. This is what the word “stable” means. In theory.

In practice, many upstreams play it fast-and-loose with what is “bugfix” and what is more. Sometimes higher-level packages depend on something lower-level, but make that dependency cross non-bugfix releases, so now we have to make a breaking update to something in nixpkgs that isn’t even related to the package being updated. Things get messy in practice.

I appreciate your frustration, but the people working on nixpkgs aren’t idiots. Usually there’s a good reason non-stable upgrades happen for a specific package. You’ll probably find something that causes issues if you dig deeper into this package - that, or it is simply effectively unmaintained and you’re the first to notice its bitrot. nixpkgs has a lot of undermaintained packages.

8 Likes