NixOS ppc64le VM does not have /dev/vda device

I think I’m close to being able to cross-compile a NixOS VM for powerpc64le. On my repro-nixos-ppc64le-cross-vm branch, I can run nix-build nixos/ -A vm -I nixos-config=./config.nix. The resulting VM fails during stage 1 boot, because it needs to mount /dev/vda which does not exist.

Am I simply missing a kernel module? On x86_64, the /dev/vda device is created by virtio_blk. Are there other options I need to enable in my kernel config?

config.nix:

{ lib, pkgs, ... }:
{
  imports = [
    ./nixos/modules/virtualisation/qemu-vm.nix
  ];

  nixpkgs = {
    crossSystem = lib.systems.examples.powernv;
    #localSystem = { system = "x86_64-linux"; };
  };

  boot.kernelPackages = pkgs.linuxPackages_testing;

  virtualisation.qemu.guestAgent.enable = false;

  virtualisation.qemu.consoles = [ "ttyAMA0,115200n8" ];
  #virtualisation.qemu.diskInterface = "scsi";

  virtualisation.memorySize = 2048;
  virtualisation.graphics = false;
  virtualisation.useBootLoader = false;

  boot.initrd.kernelModules = [ "virtio_pci" "virtio_blk" ];
  boot.kernelModules = [ "virtio_blk" ];

  documentation.enable = lib.mkForce false;
  documentation.nixos.enable = lib.mkForce false;
}

VM stage 1 boot log:

<<< NixOS Stage 1 >>>

loading module virtio_balloon...
loading module virtio_console...
loading module virtio_rng...
loading module virtio_pci...
loading module virtio_blk...
loading module dm_mod...
running udev...
Starting version 247
kbd_mode: KDSKBMODE: Inappropriate ioctl for device
Gstarting device mapper and LVM...
hwclock: can't open '/dev/misc/rtc': No such file or directory
mke2fs 1.45.6 (20-Mar-2020)
The file /dev/vda does not exist and no size was specified.
waiting for device /dev/vda to appear.......................
Timed out waiting for device /dev/vda, trying to mount anyway.
mounting /dev/vda on /...
mount: mounting /dev/vda on /mnt-root/ failed: No such file or directory

An error occurred in stage 1 of the boot process, which must mount the
root filesystem on `/mnt-root' and then start stage 2.  Press one
of the following keys:

  r) to reboot immediately
  *) to ignore the error and continue
1 Like

Maybe see in a rescue shell what partitions it detects at all:

$ cat /proc/partitions

Thanks. It’s interesting, it looks like it isn’t detecting any partitions at all, except the ramdisk:

major minor  #blocks  name

   1        0      65536 ram0
   1        1      65536 ram1
   1        2      65536 ram2
   1        3      65536 ram3
   1        4      65536 ram4
   1        5      65536 ram5
   1        6      65536 ram6
   1        7      65536 ram7
   1        8      65536 ram8
   1        9      65536 ram9
   1       10      65536 ram10
   1       11      65536 ram11
   1       12      65536 ram12
   1       13      65536 ram13
   1       14      65536 ram14
   1       15      65536 ram15

Edit: Seem to be making progress with lsmod. x86_64 VM has all of these modules:

9pnet_virtio 20480 0 - Live 0xffffffffc0149000
virtio_net 53248 0 - Live 0xffffffffc02dc000
9pnet 86016 1 9pnet_virtio, Live 0xffffffffc0133000
net_failover 24576 1 virtio_net, Live 0xffffffffc012c000
virtio_blk 20480 0 - Live 0xffffffffc0220000
failover 16384 1 net_failover, Live 0xffffffffc0202000
hid_generic 16384 0 - Live 0xffffffffc01f0000
usbhid 57344 0 - Live 0xffffffffc029f000
hid 135168 2 hid_generic,usbhid, Live 0xffffffffc027d000
sr_mod 28672 0 - Live 0xffffffffc0275000
cdrom 65536 1 sr_mod, Live 0xffffffffc0264000
ata_piix 36864 0 - Live 0xffffffffc0216000
libata 249856 1 ata_piix, Live 0xffffffffc0226000
uhci_hcd 49152 0 - Live 0xffffffffc0209000
atkbd 32768 0 - Live 0xffffffffc01f9000
libps2 20480 1 atkbd, Live 0xffffffffc01ea000
ehci_hcd 94208 0 - Live 0xffffffffc010a000
usbcore 278528 3 usbhid,uhci_hcd,ehci_hcd, Live 0xffffffffc01a5000
scsi_mod 229376 2 sr_mod,libata, Live 0xffffffffc016c000
crct10dif_pclmul 16384 0 - Live 0xffffffffc00ff000
crct10dif_common 16384 1 crct10dif_pclmul, Live 0xffffffffc00fa000
crc32c_intel 24576 0 - Live 0xffffffffc00f3000
virtio_pci 28672 0 - Live 0xffffffffc0124000
i8042 28672 0 - Live 0xffffffffc00eb000
usb_common 16384 3 uhci_hcd,ehci_hcd,usbcore, Live 0xffffffffc0105000
rtc_cmos 28672 1 - Live 0xffffffffc00e3000
serio 24576 4 atkbd,i8042, Live 0xffffffffc00d8000
dm_mod 139264 0 - Live 0xffffffffc00b5000
virtio_rng 16384 0 - Live 0xffffffffc00b0000
rng_core 16384 1 virtio_rng, Live 0xffffffffc00ab000
virtio_console 40960 0 - Live 0xffffffffc00a0000
virtio_balloon 24576 0 - Live 0xffffffffc0099000
virtio_ring 32768 7 9pnet_virtio,virtio_net,virtio_blk,virtio_pci,virtio_rng,virtio_console,vi0
virtio 16384 7 9pnet_virtio,virtio_net,virtio_blk,virtio_pci,virtio_rng,virtio_console,virtio_0

and my powerpc64le VM only has these:

crc32c_vpmsum 10357 0 - Live 0xc00800000004a000
dm_mod 163837 0 - Live 0xc0080000000c1000
virtio_blk 17004 0 - Live 0xc00800000006f000
virtio_rng 4659 0 - Live 0xc00800000004e000
rng_core 12454 1 virtio_rng, Live 0xc008000000053000
virtio_console 31825 0 - Live 0xc00800000005b000

Going to try to track down and enable each of these modules.

Depending on your abilities or stubbornness you could try to debug what options to use by building linux from source: Kernel Debugging with QEMU - NixOS Wiki
The entry point in the linux kernel is init/main.c, however I would start with adding printk statements to drivers/virtio and block/blk-mq-virtio.c.

You can use this minimal nixos image: https://github.com/Mic92/vmsh/blob/616a0a20754190f987f508d5a171a2f7867ba8a3/nix/minimal-vm.nix

Wow! I haven’t done any kernel hacking before. Looks like fun though, and I’m not afraid to get my hands dirty. May be a while before I can make any headway but I’ll report any findings if I do.

Virtio-io has two main mechanism for registering hardware depending on the architecture: pcie and mmio
I think for pcie uses bus enumeration and for mmio it was device tree files.

I see there is also a ppc64le_guest_defconfig, maybe I will have more luck with that than the powernv_defconfig I am currently using.

Edit: Unfortunately not. Guess it’s printk time :slight_smile:

I have made some progress. The PowerNV QEMU machine does not have a default PCIe layout so it has to be specified manually, according to Create new page · legoater/qemu Wiki · GitHub

By specifying pcie.0 as the bus for the virtio-blk-pci device, /dev/vda appears and is mounted by stage 1. It then fails to mount the store:

checking /dev/vda...
fsck (busybox 1.32.1)
[fsck.ext4 (1) -- /mnt-root/] fsck.ext4 -a /dev/vda
/dev/vda: recovering journal
/dev/vda: clean, 231/32768 files, 10160/131072 blocks
mounting /dev/vda on /...
mounting store on /nix/.ro-store...
[    6.798669][  T196] 9pnet_virtio: no channels available for device store
[    6.798669][  T196] 9pnet_virtio: no channels available for device store
mount: mounting store on /mnt-root/nix/.ro-store failed: No such file or directory

An error occurred in stage 1 of the boot process, which must mount the
root filesystem on `/mnt-root' and then start stage 2.  Press one
of the following keys:

I suspect this is a similar issue, and I just need to attach the 9pnet_virtio device to a PCI slot as well.

Sure enough, someone on the qemu-ppc mailing list was kind enough to point out the correct invocation to manually specify the pcie bus for a 9pnet_virtio device.

For anyone who may come across this in the future, instead of the abbreviated form:

-virtfs local,path=/nix/store,security_model=none,mount_tag=store

we need to use the long form to specify the pcie bus:

-fsdev local,id=fsdev1,path=/nix/store,security_model=none
-device virtio-9p-pci,fsdev=fsdev1,mount_tag=store,bus=pcie.1

My VM is now able to fully boot :slight_smile: