Aggressive Kernel removal on EOL in NixOS

Development
21

For the record, I agree that EOLing EOL kernels is correct; after all, users rely on nixpkgs being roughly up-to-date on security, so that they can just pull in changes on their own schedule. nixpkgs in this case follows upstream advice.

However, it’s not great that users on NixOS stable end up with their hardware suddenly being unsupported by stable.

If the policy is that out-of-tree modules should have an *Unstable package which will be backported more frequently, but with fewer guarantees than the LTS variant, just like the linux_latest kernel, that largely solves that problem. The nvidia driver doesn’t have that yet, though.

I believe the other good alternative is kicking linux_latest from stable entirely, forcing users with too new hardware to use unstable instead. Either resolves the problem of users using kernels that will disappear in a little while and leave them trapped.

1 Like
22

As far as I remember, we already have seen removal of kernels, that are “not yet EOL” because they would have been EOL’d during the upcoming release and it should be avoided to let that happen…

I sadly can not proof this by evidence right now, though remember a discussion around this topic during summer.

This harms also unstable!

23

Do you mean this? linux_4_9: remove by Ma27 · Pull Request #200218 · NixOS/nixpkgs · GitHub

See the comments in that thread for discussion.

24

No, I do not mean that. I was talking about a 5.x kernel. Sadly I do not remember the exact version nor do I remember a rough point in time. Just that it was this year (2022) and not in the last 4 weeks.

25

No longterm 5.x kernel has ended support yet, so I suppose you didn’t mean those.

EDIT: but yes, generally I’d think that’s what’s done… about the precautionary removal of nixpkgs attributes that will get out of support during stable NixOS.

2 Likes
26

I am not talking about 5.15 or any LTS.

I am talking about a random kernel that would have gone EOL after branch off, but before the next release.

I do not remember anymore if this was before 22.05 or during it. But I am sure that there was such an event.

27

I’m a user that been cut out of 5.19 kernel… can’t upgrade to 6.0.x due to the Nvidia issue.
can’t go back to 5.15 due to issues with hibernation in my laptop… so I’m kind of stuck. I’ve switched to a zen kernel until the issue with Nvidia will be solved. hopefully soon… if there’s a way to tell Nixos which version of Nvidia to install it might solve the issue for me… but this is simply unfortunate situation.

1 Like
28

Can we provide instructions on the Wiki on how to import a pinned nixpkgs tarball boot.kernelPackages? This works for me:

  boot.kernelPackages = let 
        pinnedNixPkgs = import (pkgs.fetchFromGitHub {
            owner = "nixos";
            repo = "nixpkgs";
            rev = "35955e360c6851e9a53fed945a9e98cfea5d67be";
            hash = "sha256-M4ml2ECaO7UBKsMhtw3ZOMFE/4y51W84xoFlmsb2sHw=";
        }) {};
        myKernel = pinnedNixPkgs.linux_5_19;
        in pkgs.recurseIntoAttrs (pinnedNixPkgs.linuxPackagesFor myKernel);
4 Likes
29

You can also just use flakes: Switch to unstable linux kernel to work around bluetooth/nvidia being incompatible by TLATER · Pull Request #214 · TLATER/dotfiles · GitHub

The problem isn’t finding a workaround, never updating will also circumvent it. The problem is that stable may suddenly drop support for hardware.

2 Likes
30

As one of the maintainers of the Linux kernels in nixpkgs (and also taking care of backports and thus also removals) I guess I need to chime in as well:

  • First of all the obvious: continuing to maintain these (as other distros probably do) is not an option. We lack manpower and (most likely) in-depth knowledge to do it in a reliable way. Also, simply scanning for CVEs in changelogs is definetely not sufficient because not every security fix is marked as such.
  • I thought a bit about the suggestion to simply mark EOLed kernels as insecure and I’m not sure I like it: we already use flags to disable “undesired” behavior for the “average” user, namely experimental-features in Nix. But still, most of the people just enabled it and even tutorials advertised that without talking too much about the consequences and now people seem surprised that things change within e.g. the source tree abstraction. To get back to the original problem: I’m afraid that too many people will do that without thinking too much about the consequences and if people have a 0day exploit on their system then, the bad press will probably get back to us.
  • Unless we leave it as-is (and perhaps make it in the manual more explicit) I think the best way is to prohibit non-LTS kernel software (and perhaps provide a linux_latest that is documented to be frequently changed in potentially breaking ways).

Regarding the originally described problem: I think that we can’t do too much about it and shouldn’t advertise a potentially unsafe fix. However, nixpkgs has with overlays a fairly strong mechanism to change things in your package set, so that seems to be the proper solution.

I am talking about a random kernel that would have gone EOL after branch off, but before the next release.

So I just checked

  • 22.05
  • 21.11
  • 21.05

and I couldn’t find such a kernel.

As mentioned, 4.9 was because it was an LTS version. If somebody still uses it, there are usually good reasons for that and in case of LTS software it should be safe to assume that it remains maintained throughout the entire stable cycle, so we wanted to avoid bad surprises here.

10 Likes
31

Maybe we should nuke the allow insecure config.nix option.

32

I came to this thread to understand why I couldn’t update my kernel (and thanks for the explanations). I recently reinstalled, and am now on a lower kernel (5.15 rather than 5.19). The thread has helped me understand why I couldn’t get at least as far as 5.19. While I think some options would be preferable, at least some documentation describing this - including why the unstable branch may result in an older LTS kernel.

2 Likes
33

at least some documentation describing this - including why the unstable branch may result in an older LTS kernel.

Yes I fully agree. Added this to my todo list :slight_smile:

6 Likes
34

Hi there,

I’m a newbie NixOS user and I gladly help with some feedback on this issue. Please take into consideration the fact that I’m not very technical – I do know how to code, but I haven’t been able to fully grasp the nix language, you can consider me a “amateur sysadmin”. I had spent many hours configuring my machines with my bare understanding of flakes and modules and a trial+error approach. I barely understand how my current system, with multiple flakes and copying+pasting of other users’ nix configurations work. I do this on a test machine – no important data is at risk.

Having said that, updating my Framework machine worked fine with Stable 22.05 in late October. At some point, I guess around November 10th, the graphical drivers and wifi drivers stopped working. It might be related to a Framework firmware upgrade. Unfortunately, as I was following this: Erase your darlings: immutable infrastructure for mutable systems - Graham Christensen ZFS install, I am now in the limbo described by the OP.

Happy to help with any input on this issue. For the time being, I am trying zfsUnstable. I do understand that my use case is fairly strange: the recommendation should be that newbie users don’t use ZFS on NixOS.

The concept of NixOS is really inspiring. I look forward to see this ecosystem maturing. When NixOS works, it works so well that I feel trapped – I can’t go back to other distros, it feels like traveling back to 2010…

Thank you,
Lon

1 Like
35

Yep, looks like you’re out of luck on stable, just like me you need at least 5.16 for wifi+bluetooth, which is EOL: https://github.com/NixOS/nixos-hardware/blob/0099253ad0b5283f06ffe31cf010af3f9ad7837d/framework/default.nix#L19

That said, your graphics driver should not be broken to my knowledge. The stable default/LTS kernel on stable is not so old that it doesn’t support 11th gen Intel processors.

You can:

If you would otherwise just never update, the pinning old stable option is no less secure, but prevents other packages being outdated.

It’s trickier with zfs, because the limitation is just that it does not support the latest non-EOL kernel, so you’re kind of just out of luck. I wonder if we should mark the kernels as insecure if everybody is going to just manually import old kernels anyway.

4 Likes
36

Not sure if I’m misunderstanding you, but even on stable we the latest Linux kernels available. They’re just not compatible with latest zfsStable - as on unstable.

37

Yeah, I read that as “The LTS kernel …”

1 Like
38

Even worse, I was trying to use -hardened versions :sweat_smile: – a recipe for unsupported quirks: strange firmware problems with i915 + zfs + hardened kernel.

39

As one of the maintainers of the Linux kernels in nixpkgs (and also taking care of backports and thus also removals)

First of all, thank you for the time you’re investing in maintaining this critical part of NixOS!

First of all the obvious: continuing to maintain these (as other distros probably do) is not an option. We lack manpower and (most likely) in-depth knowledge to do it in a reliable way.

What is the typical amount of maintenance that fixed version kernel requires? I understand that every time a kernel-related change is proposed for Nixpkgs, it needs to be compatible with all current kernel packages. So the workload likely grows with the amount of supported kernels. But I can also imagine that unsupported kernel could account for more issues, compared to LTS kernels. What I wonder is how much work is this in practice? Are talking about something on the scale of a few hours per week, per month or per year? I’m trying to better understand how the Nixpkgs maintainers workload would be impacted by keeping EOL-ed kernels around for a bit longer.

To get back to the original problem: I’m afraid that too many people will do that without thinking too much about the consequences and if people have a 0day exploit on their system then, the bad press will probably get back to us.

One of my NixOS machines is a laptop with AMD CPU and nVidia GPU released this year. IIRC, Linux 5.17 was the first kernel that properly supported this machine (the most essential features, there’s still some minor driver stuff that will likely be fixed with 6.1 / 6.2), so using an 5.15 is simply not an option. On the other hand, I am also using ZFS, so 5.19 is still the only option for me, since 6.0 is not yet supported by ZFS (stable).

I’d really appreciate if you could take a moment to think about people in situations similar to mine, and suggest a proper way forward, as from your post it sounds like you’re implying that one should either:

  • not use NixOS (god forbid one becomes a victim of zero day exploit and dares to publicly blame NixOS for their misfortunes it in the press)
  • not use ZFS (perhaps it should removed from nixpkgs?)
  • not use new hardware (?!)

From my simplistic point of view, the principled (*) solution is to either completely eliminate ZFS and nVidia support from Nixpkgs entirely or: accept the simple rule (I haven’t thought much about it, so it can most likely be improved) that boot.zfs.package.latestCompatibleLinuxPackages should never be downgraded and that it should always point to a version that is also compatible with nvidia’s unfree proprietary driver. If that means keeping one EOL kernel, simply mark its package as insecure and let users take the risk if they want.

(*) I’m saying “principled” as I think that if maintainers don’t want to maintain certain features (with the complex interactions that they may bring) they should simply not keep them in the codebase and advertise them as supported or they should be supported to the extent practically possible.

[…] But still, most of the people just enabled it and even tutorials advertised that without talking too much about the consequences and now people seem surprised that things change within e.g. the source tree abstraction.

I really don’t get why we are even discussing this. Every user is making a choice with how they’re managing their system. If something was marked as “experimental”, “insecure” or “broken”, the users have willingly taken the risk and have no one to blame if something goes wrong but themselves. But the situation is also not black and white - one makes the risk-benefit analysis for their particular situation and decides whether e.g. using flakes is worth the risk.

Please excuse me for sounding overly confrontational, and not appreciative of work that all Nixpkgs maintainers are putting in, but to me it seems illogical to take positions out of fear of the mere potential for “bad press” (e.g. being against keeping EOL-ed kernel as insecure package) but at the same time straight up breaking existing users’ setup (by removing the only kernels that supported their setup from the stable Nixpkgs branch).

40

Hardware enablement usually falls under " Version updates which include new functionality (but no breaking changes)". https://github.com/NixOS/nixpkgs/blob/c0be9e4c71820c7a8cf01a1f6d4b68427b4c7b77/doc/contributing/submitting-changes.chapter.md#acceptable-backport-criteria

2 Likes