Kernel panic - new generations won't work

Help
1

NB: I’m still learning nix and nixos.

I did a

nix-channel --update nixos; nixos-rebuild boot

then rebooted.

I got to stage 1 and entered the LUKS password. It then quickly scrolls and gives me lots of errors and a kernel panic.

The errors look like:

/nix/store/.../activate line 450: chmod: command not found
/nix/store/.../activate line 535: readlink: command not found
[   10.125486] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00

I tried the previous generation and got the same thing. I went back several generations and finally found a generation that boots cleanly (94). That generation is from 4 days ago (Apr 24). I verified that several other previous generations before 94 work, but none after 94 to the current (104) do.

I removed everything I had done from my configuration.nix, rebuilt and rebooted and still get a kernel panic.

I went further and removed almost everything from my configuration.nix, until it was a bare minimum, but still get a kernel panic on reboot.

This doesn’t seem like disk corruption since, as I mentioned above, all generations <= 94 work fine. What else can I trobuleshoot?

2

The nixpkgs versions each if the generations is built against.

3

I’m not sure how to use that to troubleshoot the issue I’ve described. Please elaborate.

4

You said, nothing changed in your configuration itself, so the only thing that could have changed was nixpkgs itself.

Probably because you updated your channels.

So perhaps there is a problem with the upstream nixpkgs or software distributed by it that now breaks your system.

5

what nixpkgs channel are you using?

6

I had similar thoughts.

21.11.

7

You might be overlooking the fact that, when you update your channel, nix will essentially pull a newer version of the nixpkgs source. If you compare the two, and then start bisecting you might find some answers.

Alternatively, you could try using nix store diff-closures to diff the generations.

8

I tried removing the very first things I added to my configuration.nix a few weeks ago when I first installed and finally found the culprit.

  system.activationScripts.symlinks = {
    text = ''
      source ${config.system.build.setEnvironment} # get base packages into this activationScript so e.g. which, curl etc. work
      ...
    '';
  };

When I comment out that source line, the kernel panic stops. I forgot that I added that to make it simpler, but otherwise hadn’t changed that block for a while.
Now I need to determine why that works if I simply do a nixos-rebuild switch but breaks the boot. Or perhaps there’s a better way to do what I’m accomplishing by extending nix functions (I’m still very newb).

9

Thanks for this. I just ran into the exact same problem myself. Did you ever end up finding a way around this?