NixOS vm under FreeBSD bhyve failing

Hi all,

I’m trying to get NixOS running as a VM on FreeBSD, using bhyve. I’m following these instructions: 21.7. FreeBSD as a Host with bhyve

My CPU seems to support the required instructions.

It seems to get pretty far, but eventually fails:

kbd_mode: KDSKBMODE: Inappropriate ioctl for device
Gstarting device mapper and LVM...
mounting tmpfs on /...
waiting for device /dev/root to appear.......................
Timed out waiting for device /dev/root, trying to mount anyway.
mounting /dev/root on /iso...
mount: mounting /dev/root on /mnt-root/iso failed: No such file or directory

An error occurred in stage 1 of the boot process, which must mount the
root filesystem on `/mnt-root' and then start stage 2.  Press one
of the following keys:

  r) to reboot immediately
  *) to ignore the error and continue
Full boot log
rdmsr to register 0xcd on vcpu 0
                                [    0.000000] Linux version 5.15.85 (nixbld@localhost) (gcc (GCC) 11.3.0, GNU ld (GNU Binutils) 2.39) #1-NixOS SMP Wed Dec 21 16:36:38 UTC 2022
[    0.000000] Command line: console=ttyS0 BOOT_IMAGE=(cd0)/boot/bzImage
[    0.000000] x86/fpu: x87 FPU will use FXSAVE
[    0.000000] signal: max sigframe size: 1440
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fffe] usable
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000003fffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.6 present.
[    0.000000] DMI:  BHYVE, BIOS 1.00 03/14/2014
[    0.000000] unchecked MSR access error: RDMSR from 0xcd at rIP: 0xffffffffaf0699f4 (native_read_msr+0x4/0x30)
[    0.000000] Call Trace:
[    0.000000]  <TASK>
[    0.000000]  ? cpu_khz_from_msr+0x4d/0x110
[    0.000000]  ? native_calibrate_cpu_early+0x31/0x210
[    0.000000]  ? determine_cpu_tsc_frequencies+0x1f/0xdc
[    0.000000]  ? tsc_early_init+0x15/0x30
[    0.000000]  ? setup_arch+0x4f3/0xc75
[    0.000000]  ? start_kernel+0x6d/0x6c5
[    0.000000]  ? x86_family+0x5/0x30
[    0.000000]  ? secondary_startup_64_no_verify+0xc2/0xcb
[    0.000000]  </TASK>
[    0.000000] tsc: Detected 1333.333 MHz processor
[    0.000058] last_pfn = 0x40000 max_arch_pfn = 0x400000000
[    0.000097] Disabled
[    0.000101] x86/PAT: MTRRs disabled, skipping PAT initialization too.
[    0.000113] CPU MTRRs all blank - virtualized system.
[    0.000149] x86/PAT: Configuration [0-7]: WB  WT  UC- UC  WB  WT  UC- UC  
[    0.000304] found SMP MP-table at [mem 0x000f0000-0x000f000f]
[    0.000528] RAMDISK: [mem 0x0401f000-0x04d4afff]
[    0.000562] ACPI: Early table checksum verification disabled
[    0.000575] ACPI: RSDP 0x00000000000F2400 000024 (v02 BHYVE )
[    0.000587] ACPI: XSDT 0x00000000000F2480 000044 (v01 BHYVE  BVXSDT   00000001 INTL 20200430)
[    0.000618] ACPI: APIC 0x00000000000F2500 000072 (v01 BHYVE  BVMADT   00000001 INTL 20200430)
[    0.000631] ACPI: FACP 0x00000000000F25D2 00010C (v05 BHYVE  BVFACP   00000001 INTL 20200430)
[    0.000647] ACPI: DSDT 0x00000000000F27D2 000926 (v02 BHYVE  BVDSDT   00000001 INTL 20200430)
[    0.000659] ACPI: FACS 0x00000000000F2792 000040
[    0.000669] ACPI: FACS 0x00000000000F2792 000040
[    0.000679] ACPI: HPET 0x00000000000F2712 000038 (v01 BHYVE  BVHPET   00000001 INTL 20200430)
[    0.000691] ACPI: MCFG 0x00000000000F2752 00003C (v01 BHYVE  BVMCFG   00000001 INTL 20200430)
[    0.000700] ACPI: Reserving APIC table memory at [mem 0xf2500-0xf2571]
[    0.000705] ACPI: Reserving FACP table memory at [mem 0xf25d2-0xf26dd]
[    0.000708] ACPI: Reserving DSDT table memory at [mem 0xf27d2-0xf30f7]
[    0.000711] ACPI: Reserving FACS table memory at [mem 0xf2792-0xf27d1]
[    0.000715] ACPI: Reserving FACS table memory at [mem 0xf2792-0xf27d1]
[    0.000718] ACPI: Reserving HPET table memory at [mem 0xf2712-0xf2749]
[    0.000721] ACPI: Reserving MCFG table memory at [mem 0xf2752-0xf278d]
[    0.000787] Setting APIC routing to physical flat.
[    0.001684] No NUMA configuration found
[    0.001690] Faking a node at [mem 0x0000000000000000-0x000000003fffffff]
[    0.001699] NODE_DATA(0) allocated [mem 0x3fffb000-0x3fffffff]
[    0.001915] Zone ranges:
[    0.001921]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.001927]   DMA32    [mem 0x0000000001000000-0x000000003fffffff]
[    0.001932]   Normal   empty
[    0.001936]   Device   empty
[    0.001939] Movable zone start for each node
[    0.001942] Early memory node ranges
[    0.001944]   node   0: [mem 0x0000000000001000-0x000000000009efff]
[    0.001948]   node   0: [mem 0x0000000000100000-0x000000003fffffff]
[    0.001953] Initmem setup node 0 [mem 0x0000000000001000-0x000000003fffffff]
[    0.002004] On node 0, zone DMA: 1 pages in unavailable ranges
[    0.002664] On node 0, zone DMA: 97 pages in unavailable ranges
[    0.050990] ACPI: PM-Timer IO Port: 0x408
[    0.051042] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
[    0.051160] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-31
[    0.051172] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
[    0.051178] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
[    0.051190] ACPI: Using ACPI (MADT) for SMP configuration information
[    0.051195] ACPI: HPET id: 0x80860701 base: 0xfed00000
[    0.051224] smpboot: Allowing 4 CPUs, 0 hotplug CPUs
[    0.051247] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.051253] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]
[    0.051258] [mem 0x40000000-0xffffffff] available for PCI devices
[    0.051262] Booting paravirtualized kernel on bare hardware
[    0.051270] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[    0.058582] setup_percpu: NR_CPUS:384 nr_cpumask_bits:384 nr_cpu_ids:4 nr_node_ids:1
[    0.064797] percpu: Embedded 55 pages/cpu s186904 r8192 d30184 u524288
[    0.064956] Built 1 zonelists, mobility grouping on.  Total pages: 257792
[    0.064964] Policy zone: DMA32
[    0.064969] Kernel command line: console=ttyS0 BOOT_IMAGE=(cd0)/boot/bzImage
[    0.065033] Unknown kernel command line parameters "BOOT_IMAGE=(cd0)/boot/bzImage", will be passed to user space.
[    0.067820] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.069260] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.069528] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.080255] Memory: 982100K/1048184K available (12295K kernel code, 2282K rwdata, 7312K rodata, 1884K init, 2472K bss, 65824K reserved, 0K cma-reserved)
[    0.081678] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.081768] Kernel/User page tables isolation: enabled
[    0.081859] ftrace: allocating 35282 entries in 138 pages
[    0.106707] ftrace: allocated 138 pages with 3 groups
[    0.107448] rcu: Hierarchical RCU implementation.
[    0.107454] rcu: 	RCU event tracing is enabled.
[    0.107456] rcu: 	RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=4.
[    0.107460] 	Trampoline variant of Tasks RCU enabled.
[    0.107463] 	Rude variant of Tasks RCU enabled.
[    0.107464] 	Tracing variant of Tasks RCU enabled.
[    0.107467] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.
[    0.107470] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.119973] NR_IRQS: 24832, nr_irqs: 592, preallocated irqs: 16
[    0.120423] rcu: 	Offload RCU callbacks from CPUs: (none).
[    0.120584] random: crng init done
[    0.120988] Console: colour EGA 80x25
[    0.740636] printk: console [ttyS0] enabled
[    0.745678] ACPI: Core revision 20210730
[    0.751131] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 113919999973 ns
[    0.763305] APIC: Switch to symmetric I/O mode setup
[    0.775333] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x13381e82524, max_idle_ns: 440795248224 ns
[    0.781318] Calibrating delay loop (skipped), value calculated using timer frequency.. 2666.66 BogoMIPS (lpj=1333333)
[    0.782303] pid_max: default: 32768 minimum: 301
[    0.782303] LSM: Security Framework initializing
[    0.782303] landlock: Up and running.
[    0.782303] Yama: becoming mindful.
[    0.782303] SELinux:  Initializing.
[    0.782303] LSM support for eBPF active
[    0.782303] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.782303] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
rdmsr to register 0x3a on vcpu 0
                                rdmsr to register 0x140 on vcpu 0
                                                                 [    0.782303] Last level iTLB entries: 4KB 48, 2MB 0, 4MB 0
[    0.782303] Last level dTLB entries: 4KB 256, 2MB 16, 4MB 16, 1GB 0
[    0.782303] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[    0.782303] Spectre V2 : Mitigation: Retpolines
[    0.782303] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[    0.782303] Spectre V2 : Spectre v2 / SpectreRSB : Filling RSB on VMEXIT
[    0.782303] MDS: Mitigation: Clear CPU buffers
[    0.782303] MMIO Stale Data: Unknown: No mitigations
[    0.782303] Freeing SMP alternatives memory: 32K
[    0.782303] smpboot: CPU0: Intel(R) Celeron(R) CPU  J3060  @ 1.60GHz (family: 0x6, model: 0x4c, stepping: 0x4)
[    0.785816] Performance Events: unsupported p6 CPU model 76 no PMU driver, software events only.
[    0.787526] rcu: Hierarchical SRCU implementation.
[    0.791687] smp: Bringing up secondary CPUs ...
[    0.793495] x86: Booting SMP configuration:
rdmsr to register 0x3a on vcpu 1
                                rdmsr to register 0x140 on vcpu 1
                                                                 [    0.794337] .... node  #0, CPUs:      #1
[    0.661748] smpboot: CPU 1 Converting physical 0 to logical die 1
rdmsr to register 0x3a on vcpu 2
                                rdmsr to register 0x140 on vcpu 2
                                                                 [    0.873817]  #2
[    0.661748] smpboot: CPU 2 Converting physical 0 to logical die 2
rdmsr to register 0x3a on vcpu 3
                                rdmsr to register 0x140 on vcpu 3
                                                                 [    0.966953]  #3
[    0.661748] smpboot: CPU 3 Converting physical 0 to logical die 3
[    1.047443] smp: Brought up 1 node, 4 CPUs
[    1.048315] smpboot: Max logical packages: 4
[    1.049318] smpboot: Total of 4 processors activated (8636.17 BogoMIPS)
[    1.053328] devtmpfs: initialized
[    1.055546] x86/mm: Memory block size: 128MB
[    1.060596] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[    1.061331] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    1.064431] pinctrl core: initialized pinctrl subsystem
[    1.068318] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    1.070485] audit: initializing netlink subsys (disabled)
[    1.074457] audit: type=2000 audit(1674671376.293:1): state=initialized audit_enabled=0 res=1
[    1.078566] thermal_sys: Registered thermal governor 'bang_bang'
[    1.079315] thermal_sys: Registered thermal governor 'step_wise'
[    1.080322] thermal_sys: Registered thermal governor 'user_space'
[    1.082382] cpuidle: using governor menu
[    1.085688] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[    1.086318] ACPI: bus type PCI registered
[    1.087317] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    1.089992] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
[    1.091324] PCI: not using MMCONFIG
[    1.093326] PCI: Using configuration type 1 for base access
[    1.102696] Kprobes globally optimized
[    1.107330] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    1.129599] ACPI: Added _OSI(Module Device)
[    1.130424] ACPI: Added _OSI(Processor Device)
[    1.131358] ACPI: Added _OSI(3.0 _SCP Extensions)
[    1.133509] ACPI: Added _OSI(Processor Aggregator Device)
[    1.134503] ACPI: Added _OSI(Linux-Dell-Video)
[    1.136319] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[    1.137322] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
[    1.140514] ACPI: 1 ACPI AML tables successfully acquired and loaded
[    1.144641] ACPI: Interpreter enabled
[    1.145351] ACPI: PM: (supports S0 S5)
[    1.146317] ACPI: Using IOAPIC for interrupt routing
[    1.147472] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
[    1.148520] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in ACPI motherboard resources
[    1.149354] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    1.157831] ACPI: PCI Root Bridge [PC00] (domain 0000 [bus 00])
[    1.159310] acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    1.161383] PCI host bridge to bus 0000:00
[    1.162328] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    1.163325] pci_bus 0000:00: root bus resource [io  0x0d00-0x1fff window]
[    1.164378] pci_bus 0000:00: root bus resource [io  0x2000-0x211f window]
[    1.165331] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xc01fffff window]
[    1.166328] pci_bus 0000:00: root bus resource [mem 0x800000000-0x8000fffff window]
[    1.167328] pci_bus 0000:00: root bus resource [bus 00]
[    1.170337] pci 0000:00:00.0: [1275:1275] type 00 class 0x060000
[    1.172676] pci 0000:00:00.0: reg 0x30: [mem 0x00000000-0x000007ff pref]
[    1.176645] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
[    1.179931] pci 0000:00:01.0: reg 0x30: [mem 0x00000000-0x000007ff pref]
[    1.182402] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000
[    1.184554] pci 0000:00:02.0: reg 0x10: [io  0x2000-0x201f]
[    1.185533] pci 0000:00:02.0: reg 0x14: [mem 0xc0000000-0xc0001fff]
[    1.189457] pci 0000:00:02.0: reg 0x30: [mem 0x00000000-0x000007ff pref]
[    1.194834] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000
[    1.196315] pci 0000:00:03.0: reg 0x10: [io  0x2080-0x20ff]
[    1.197527] pci 0000:00:03.0: reg 0x14: [mem 0xc0002000-0xc0003fff]
[    1.201345] pci 0000:00:03.0: reg 0x30: [mem 0x00000000-0x000007ff pref]
[    1.206361] ACPI: PCI: Interrupt link LNKA configured for IRQ 5
[    1.208354] ACPI: PCI: Interrupt link LNKB configured for IRQ 6
[    1.209631] ACPI: PCI: Interrupt link LNKC configured for IRQ 0
[    1.210322] ACPI: PCI: Interrupt link LNKC disabled
[    1.211720] ACPI: PCI: Interrupt link LNKD configured for IRQ 0
[    1.212318] ACPI: PCI: Interrupt link LNKD disabled
[    1.215495] ACPI: PCI: Interrupt link LNKE configured for IRQ 0
[    1.216319] ACPI: PCI: Interrupt link LNKE disabled
[    1.219458] ACPI: PCI: Interrupt link LNKF configured for IRQ 0
[    1.220345] ACPI: PCI: Interrupt link LNKF disabled
[    1.221635] ACPI: PCI: Interrupt link LNKG configured for IRQ 0
[    1.222318] ACPI: PCI: Interrupt link LNKG disabled
[    1.223734] ACPI: PCI: Interrupt link LNKH configured for IRQ 0
[    1.224318] ACPI: PCI: Interrupt link LNKH disabled
[    1.230387] iommu: Default domain type: Translated 
[    1.231331] iommu: DMA domain TLB invalidation policy: lazy mode 
[    1.233334] vgaarb: loaded
[    1.236881] NetLabel: Initializing
[    1.237315] NetLabel:  domain hash size = 128
[    1.238317] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    1.239443] NetLabel:  unlabeled traffic allowed by default
[    1.241314] PCI: Using ACPI for IRQ routing
[    1.552481] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
[    1.553316] hpet0: 8 comparators, 32-bit 16.777216 MHz counter
[    1.888157] clocksource: Switched to clocksource tsc-early
[    1.966744] VFS: Disk quotas dquot_6.6.0
[    1.973792] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.981387] pnp: PnP ACPI init
[    1.983769] system 00:02: [io  0x0220-0x0223] has been reserved
[    1.986049] system 00:02: [io  0x0224-0x0227] has been reserved
[    1.988278] system 00:02: [io  0x04d0-0x04d1] has been reserved
[    1.990506] system 00:02: [io  0x0400-0x0407] has been reserved
[    1.992761] system 00:02: [mem 0xe0000000-0xefffffff] has been reserved
[    1.995531] pnp: PnP ACPI: found 6 devices
[    2.039107] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    2.046487] NET: Registered PF_INET protocol family
[    2.052656] IP idents hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    2.098774] tcp_listen_portaddr_hash hash table entries: 512 (order: 1, 8192 bytes, linear)
[    2.105243] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    2.112554] TCP established hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    2.126440] TCP bind hash table entries: 8192 (order: 5, 131072 bytes, linear)
[    2.133773] TCP: Hash tables configured (established 8192 bind 8192)
[    2.143960] MPTCP token hash table entries: 1024 (order: 2, 24576 bytes, linear)
[    2.159356] UDP hash table entries: 512 (order: 2, 16384 bytes, linear)
[    2.164577] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes, linear)
[    2.169875] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    2.175093] NET: Registered PF_XDP protocol family
[    2.179593] pci 0000:00:01.0: BAR 6: assigned [mem 0xc0004000-0xc00047ff pref]
[    2.185566] pci 0000:00:02.0: BAR 6: assigned [mem 0xc0004800-0xc0004fff pref]
[    2.194297] pci 0000:00:03.0: BAR 6: assigned [mem 0xc0005000-0xc00057ff pref]
[    2.197394] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    2.208752] pci_bus 0000:00: resource 5 [io  0x0d00-0x1fff window]
[    2.217686] pci_bus 0000:00: resource 6 [io  0x2000-0x211f window]
[    2.223282] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xc01fffff window]
[    2.231898] pci_bus 0000:00: resource 8 [mem 0x800000000-0x8000fffff window]
[    2.240011] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[    2.246811] PCI: CLS 0 bytes, default 64
rdmsr to register 0x34 on vcpu 1
                                [    2.250863] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x13381e82524, max_idle_ns: 440795248224 ns
[    2.254820] Trying to unpack rootfs image as initramfs...
[    3.535030] clocksource: Switched to clocksource tsc
[    3.569417] Initialise system trusted keyrings
[    3.591920] workingset: timestamp_bits=40 max_order=18 bucket_order=0
[    3.616232] zbud: loaded
[    4.044997] Key type asymmetric registered
[    4.080259] Asymmetric key parser 'x509' registered
[    4.092980] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[    4.148512] Freeing initrd memory: 13488K
[    4.160210] io scheduler mq-deadline registered
[    4.165443] io scheduler kyber registered
[    4.173448] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    4.179871] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[    4.189490] 00:04: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A
[    4.202172] intel_pstate: CPU model not supported
[    4.206963] ashmem: initialized
[    4.209821] drop_monitor: Initializing network drop monitor service
[    4.215688] NET: Registered PF_INET6 protocol family
[    4.230524] Segment Routing with IPv6
[    4.233890] In-situ OAM (IOAM) with IPv6
rdmsr to register 0xc0011029 on vcpu 2
                                      [    4.241037] IPI shorthand broadcast: enabled
[    4.245762] sched_clock: Marking stable (3584582319, 660748920)->(5070804126, -825472887)
[    4.256073] registered taskstats version 1
[    4.260838] Loading compiled-in X.509 certificates
[    4.267028] zswap: loaded using pool lzo/zbud
[    4.275669] Key type .fscrypt registered
[    4.282603] Key type fscrypt-provisioning registered
[    4.309507] Freeing unused kernel image (initmem) memory: 1884K
[    4.320377] Write protecting the kernel read-only data: 22528k
[    4.327832] Freeing unused kernel image (text/rodata gap) memory: 2040K
[    4.335794] Freeing unused kernel image (rodata/data gap) memory: 880K
[    4.342794] Run /init as init process

<<< NixOS Stage 1 >>>

loading module loop...
[    4.565921] loop: module loaded
loading module overlay...
loading module hv_vmbus...
modprobe: ERROR: could not insert 'hv_vmbus': No such device
loading module hv_storvsc...
[    4.837097] SCSI subsystem initialized
modprobe: ERROR: could not insert 'hv_storvsc': No such device
loading module vfat...
loading module nls_cp437...
loading module nls_iso8859-1...
loading module dm_mod...
[    5.082647] device-mapper: ioctl: 4.45.0-ioctl (2021-03-22) initialised: dm-devel@redhat.com
running udev...
Starting version 251.7
[    5.881708] rtc_cmos 00:05: registered as rtc0
[    5.950953] rtc_cmos 00:05: alarms up to one day, y3k, 114 bytes nvram
[    6.833472] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
[    6.838867] i8042: Warning: Keylock active
[    6.905090] serio: i8042 KBD port at 0x60,0x64 irq 1
[    6.907135] serio: i8042 AUX port at 0x60,0x64 irq 12
[    7.166609] virtio-pci 0000:00:02.0: virtio_pci: leaving for legacy driver
[    7.182664] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver
[    7.534467] input: AT Raw Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
[    7.738377] virtio_blk virtio1: [vda] 33554432 512-byte logical blocks (17.2 GB/16.0 GiB)
[    7.794980]  vda: vda1 vda2
[    8.094701] virtio_net virtio0 enp0s2: renamed from eth0
kbd_mode: KDSKBMODE: Inappropriate ioctl for device
Gstarting device mapper and LVM...
mounting tmpfs on /...
waiting for device /dev/root to appear.......................
Timed out waiting for device /dev/root, trying to mount anyway.
mounting /dev/root on /iso...
mount: mounting /dev/root on /mnt-root/iso failed: No such file or directory

An error occurred in stage 1 of the boot process, which must mount the
root filesystem on `/mnt-root' and then start stage 2.  Press one
of the following keys:

  r) to reboot immediately
  *) to ignore the error and continue

I’ve tried several versions of set root=LABEL=NIXOS_ISO (or {nixos-minimal..., /dev/sda1, etc}), recommended in issues with similar errors, with the same error every time.

After setting up linux and initrd in the Grub CLI, this is the command I’m running:

# bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,/dev/zvol/pfSense/nixos0 \
    -s 4:0,ahci-cd,nixos-minimal-22.11.1705.b83e7f5a04a-x86_64-linux.iso -l com1,stdio -c 4 -m 1024M nixos0

It’s amd64 FreeBSD 12, using a zvol as the disk, and the nixos image listed above.

Any ideas?

Relevant reading:

Have you tried creating a UEFI VM instead? This may or may not solve the issue, but could be another setup to attempt.

With UEFI I get a boot menu, choose nixos installer, and then:

error: no suitable video mode found.
Booting in blind mode
rdmsr to register 0xcd on vcpu 0
                                rdmsr to register 0x3a on vcpu 0
                                                                rdmsr to register 0x140 on vcpu 0
                                                                                                 rdmsr to register 0x3a on vcpu 1
                                                                                                                                 rdmsr to register 0x140 on vcpu 1
                                                                                                                                                                  rdmsr to register 0x3a on vcpu 2
                                                                                                                                                                                                  rdmsr to register 0x140 on vcpu 2
                                                                                                                                                                                                                                   rdmsr to register 0x3a on vcpu 3
                     rdmsr to register 0x140 on vcpu 3
                                                      rdmsr to register 0x34 on vcpu 1
                                                                                      rdmsr to register 0xc0011029 on vcpu 1

Keyboard doesn’t seem to work, freezes.

NM, was able to go into the “quirks” menu and set it to use serial, and then got all the way to a login prompt – was just a display issue. Thanks!

I made a more comprehensive writeup of getting a NixOS VM running under bhyve in pfSense, which seems to require a bit more work than what is described for FreeBSD.