Root on tmpfs and init script not found

krostar · November 23, 2024, 10:34pm

Hello,

I’m playing with running the root partition on tmpfs for my Raspberry Pi.

Currently, I’m encountering a kernel panic during the boot process, seemingly at the end of stage 1. The error message I’m seeing is:

switch_r[   5.430465] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[   5.438863] fbcon: Taking over console
[   5.442693] CPU: 0 PID: 1 Comm: switch_root Not tainted 6.6.54 #1-NixOS
[   5.449413] Hardware name: Raspberry Pi 4 Model B Rev 1.5 (DT)
[   5.455332] Call trace:
[   5.457813]  dump_backtrace+0xa0/0x128
[   5.461630]  show_stack+0x20/0x38
[   5.464997]  dump_stack_lvl+0x48/0x60
[   5.468722]  dump_stack+0x18/0x28
[   5.472089]  panic+0x330/0x3a0
[   5.475191]  do_exit+0x8b4/0x9d0
[   5.478472]  do_group_exit+0x3c/0xa0
[   5.482103]  __arm64_sys_exit_group+0x20/0x28
[   5.486529]  invoke_syscall+0x50/0x128
[   5.490340]  el0_svc_common.constprop.0+0x48/0xf0
[   5.495120]  do_el0_svc+0x24/0x38
[   5.498489]  el0_svc+0x48/0x138
[   5.501682]  el0t_64_sync_handler+0x120/0x130
[   5.506110]  el0t_64_sync+0x190/0x198
[   5.509829] SMP: stopping secondary CPUs
[   5.513812] Kernel Offset: 0x1815400000 from 0xffffffc080000000
[   5.519819] PHYS_OFFSET: 0x0
[   5.522739] CPU features: 0x0,80000201,3c020000,0000421b
[   5.528130] Memory Limit: none
[   5.531237] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100 ]---
oot: can't execute '/sbin/init': No such file or directory

The error “can’t execute ‘/sbin/init’: No such file or directory” seems to makes sense, as /sbin/init wouldn’t exist in the tmpfs after reboot. I’m trying to make this file available during the boot process.

I’ve attempted to persist /sbin/init using the following NixOS configuration options:

  fileSystems."/persist".neededForBoot = true;
  environment.persistence."/persist".files = ["/sbin/init"];

or also by copying it within boot.initrd.postMountCommands:

  boot.initrd.postMountCommands = ''
    mkdir -p $targetRoot/sbin
    cp $targetRoot/persist/sbin/init $targetRoot/sbin/init
    echo "using persisted init file"
  '';

I’ve tried these options individually and in combination, with no success.

The /persist/sbin/init file is copied from /sbin/init during the initial nixos-install.

Despite these efforts, the kernel panic persists. I guess the /sbin/init comes from the cmdline.txt containing init=/sbin/init, but I’m unsure where this file is being sourced from in this stage of the boot process, because I’m failing to expose it where it is expected to be.

Here’s the relevant nixos configuration:

  disko = {
    devices = {
      disk = {
        sdcard = {
          device = "/dev/disk/by-id/mmc-SH64G_0x9973a25d";
          type = "disk";
          content = {
            type = "gpt";
            partitions = {
              firmware = {
                label = "FIRMWARE";
                type = "EF00";
                size = "512M";
                priority = 1;
                content = {
                  type = "filesystem";
                  format = "vfat";
                };
              };
              nix = {
                label = "NIX";
                size = "100%";
                priority = 10;
                content = {
                  type = "filesystem";
                  format = "ext4";
                  mountpoint = "/nix";
                };
              };
            };
          };
        };

        msata = {
          device = "/dev/disk/by-id/ata-KINGSTON_SKC600MS512G_50026B7785128AB7";
          type = "disk";
          content = {
            type = "gpt";
            partitions = {
              swap = {
                size = "16G";
                content = {
                  type = "swap";
                };
              };
              data = {
                size = "100%";
                content = {
                  type = "btrfs";
                  extraArgs = ["-f"];
                  subvolumes = {
                    "/data" = {
                      mountpoint = "/data";
                      mountOptions = ["compress=zstd" "noexec" "noatime"];
                    };

                    "/persist" = {
                      mountpoint = "/persist";
                    };
                  };
                };
              };
            };
          };
        };
      };

      nodev = {
        "/" = {
          fsType = "tmpfs";
          mountOptions = [
            "defaults"
            "mode=755"
            "size=512M"
          ];
        };
        "/tmp" = {
          fsType = "tmpfs";
          mountOptions = [
            "defaults"
            "mode=755"
            "nodev"
            "noexec"
            "nosuid"
            "size=2G"
          ];
        };
      };
    };
  };

  fileSystems = {
    "/persist".neededForBoot = true;
    "/tmp".neededForBoot = true;
  };

  raspberry-pi-nix = {
    board = "bcm2711";
    firmware-partition-label = config.disko.devices.disk.sdcard.content.partitions.firmware.label;
  };

And the steps I’m following during installation:

nix-copy the top-level derivation on the installer
use disko to format and mount the disks
run nixos-install with --system $derivation (this initially fails with “mv: cannot move ‘/sbin/init.tmp’ to ‘/sbin/init’: Device or resource busy”, but rerunning it seems to work)
copy /mnt/sbin/init to /mnt/persist/sbin/init
mount the FIRMWARE partition to /mnt/boot/firmware and copy config.system.build.kernel to kernel.img, config.system.build.initialRamdisk to initrd, cmdline.txt from config.boot.kernelParams, and config.txt from config.hardware.raspberry-pi.config-output
reboot without the installer.

Full boot output:

<<< NixOS Stage 1 >>>

loading module btrfs...
loading module dm_mod...
running udev...
Starting systemd-udevd version 255.9
kbd_mode: KDSKBMODE: Inappropriate ioctl for device
starting device mapper and LVM...
Scanning for Btrfs filesystems
mount: mounting /dev/disk/by-partlabel/disk-msata-data on /persist-tmp-mnt/persist failed: No such file or directory
umount: can't unmount /persist-tmp-mnt/persist: Invalid argument
mounting none on /...
waiting for device /dev/disk/by-partlabel/disk-msata-data to appear...
mounting /dev/disk/by-partlabel/disk-msata-data on /persist...
checking /dev/disk/by-partlabel/NIX...
fsck (busybox 1.36.1)
[fsck.ext4 (1) -- /mnt-root/nix] fsck.ext4 -a /dev/disk/by-partlabel/NIX
/dev/disk/by-partlabel/NIX: clean, 127846/3866624 files, 1025285/15460352 blocks
mounting /dev/disk/by-partlabel/NIX on /nix...
mounting none on /tmp...
mounting /mnt-root/persist/var/lib/nixos on /var/lib/nixos...
using persisted init file
switch_r[   5.430465] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[   5.438863] fbcon: Taking over console
[   5.442693] CPU: 0 PID: 1 Comm: switch_root Not tainted 6.6.54 #1-NixOS
[   5.449413] Hardware name: Raspberry Pi 4 Model B Rev 1.5 (DT)
[   5.455332] Call trace:
[   5.457813]  dump_backtrace+0xa0/0x128
[   5.461630]  show_stack+0x20/0x38
[   5.464997]  dump_stack_lvl+0x48/0x60
[   5.468722]  dump_stack+0x18/0x28
[   5.472089]  panic+0x330/0x3a0
[   5.475191]  do_exit+0x8b4/0x9d0
[   5.478472]  do_group_exit+0x3c/0xa0
[   5.482103]  __arm64_sys_exit_group+0x20/0x28
[   5.486529]  invoke_syscall+0x50/0x128
[   5.490340]  el0_svc_common.constprop.0+0x48/0xf0
[   5.495120]  do_el0_svc+0x24/0x38
[   5.498489]  el0_svc+0x48/0x138
[   5.501682]  el0t_64_sync_handler+0x120/0x130
[   5.506110]  el0t_64_sync+0x190/0x198
[   5.509829] SMP: stopping secondary CPUs
[   5.513812] Kernel Offset: 0x1815400000 from 0xffffffc080000000
[   5.519819] PHYS_OFFSET: 0x0
[   5.522739] CPU features: 0x0,80000201,3c020000,0000421b
[   5.528130] Memory Limit: none
[   5.531237] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100 ]---
oot: can't execute '/sbin/init': No such file or directory

Am I going the right direction ? Is there something obvious I’m missing here ?

If you have any ideas on how to debug/fix this issue, please share your thoughs.
Thank you by advance,

Alexis

ElvishJerricco · November 24, 2024, 12:59pm

/sbin/init doesn’t exist on nixos at all. You are chasing a red herring trying to persist that. And if you started with the rpi sd-image, I’m pretty sure the cmdline.txt file isn’t supposed to exist at all. The pi boots up by starting u-boot, which then boots the kernel with the per-generation init=/nix/store/.... parameter depending on the generation you booted. If it’s trying to boot /sbin/init, something is very messed up (or maybe u-boot is just passing along the contents of that cmdline.txt that shouldn’t be there or something).

krostar · November 24, 2024, 1:38pm

Hello, thank you for your response.

From what I’m understanding that nix-community/raspberry-pi-nix is doing, it does not enable uboot by default.

It instead sets the kernel parameter boot.kernelParams to init=/sbin/init, which works because of this configuration that enable boot.loader.initScript.enable = true, which populate that /sbin/init file. The content of this file is a shell exec of the right nixos derivation init.

When executing nixos-install, I do have a /sbin/init file in /mnt/sbin/init, that is the file I’m trying to persist in order to expose it during the boot process.

Surely I could try to use uboot, but I’m interested in better understanding the issue here, as it is an opportunity for me to understand things better and learn along the way.

krostar · November 24, 2024, 3:32pm

The more I spend time on this the more I’m sure of the fact that my current configuration does create the file in the place I want it to be when the system boots.

Adding debugs command to boot.initrd.postMountCommands like cat $targetRoot/sbin/init | head -n 5 clearly show that this file exist.

but still get

switch_root: can't execute '/sbin/init': No such file or directory

along with the kernel panicking

Kernel panic - not syncing: Attempted to kill init!

I’m now even more confused

ElvishJerricco · November 24, 2024, 8:37pm

Ah sorry, I didn’t know you were using that raspberry-pi-nix repo. That does indeed explain why you’re not using u-boot and why you need /sbin/init.