How to ssh to qemu emulated aarch64 NixOS?

Help
1

I’m continuing to mess with NixOS-Pi on BTRFS subvolume-based compressed root. It seems like it would be a lot easier if I could get it working with qemu – I’ve burned through several SD cards over the last year+ on this project, and I’m tired of burning, walking across the room to the Pi, unplugging, replugging, etc.

Unfortunately I can’t get keyboard or network / SSH access to work.

The host I’m running from is arch linux, x86_64.

My setup process is basically:

$ wget 'https://hydra.nixos.org/build/182205320/download/1/nixos-sd-image-22.05.1322.915f5a5b3cc-aarch64-linux.img.zst'
$ zstd -d nixos-sd-image-22.05.1322.915f5a5b3cc-aarch64-linux.img.zst
$ qemu-img resize -f raw nixos-sd-image-22.05.1322.915f5a5b3cc-aarch64-linux.img 4G
$ mv nixos-sd-image-22.05.1322.915f5a5b3cc-aarch64-linux.img nixos-22.05.img

Next, I mount the image and copy u-boot-rpi3.bin from partition 1, and bcm2837-rpi-3-b.dtb from /boot/nixos/0s76maj95yj311viyg7yhrrp300ag8s6-linux-5.15.49-dtbs/broadcom/.

Finally, I run this:

#!/usr/bin/env bash
# nixos.sh

# $ qemu-system-aarch64 --version
# QEMU emulator version 7.0.0
# Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers

set -Eeuf -o pipefail
set -x

main() {
  local img=${1:-./nixos-22.05.img}
  qemu-system-aarch64 \
    -M raspi3b \
    -dtb ./bcm2837-rpi-3-b.dtb \
    -m 1G \
    -smp 4 \
    -drive "format=raw,file=${img}" \
    -kernel ./u-boot-rpi3.bin \
    -device usb-net,netdev=net0 \
    -netdev user,id=net0,hostfwd=tcp::2222-:22 \
    -usb -device usb-host,hostbus=1,hostaddr=2 \
    -append "console=ttyAMA0 root=/dev/mmcblk0p2 rw rootwait" \
    -serial stdio
}
main "$@"

I see what looks like a pretty normal boot process which eventually gets to a login prompt.

At the login prompt, my USB keyboard doesn’t seem to work (I’ve confirmed those bus numbers), but it does get stuck in the qemu window; I have to SSH into the host from another machine and run this to get my keyboard back:

$ sudo sh -c 'echo 0000:01:00.0 > /sys/bus/pci/drivers/xhci_hcd/unbind'
$ sudo sh -c 'echo 0000:01:00.0 > /sys/bus/pci/drivers/xhci_hcd/bind'

What I’d really like is to get SSH access, but that doesn’t work either. ssh -vvv -p 2222 root@localhost shows me:

...
debug1: Connecting to localhost [127.0.0.1] port 2222.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug1: Connection established.
...

but then hangs there. In the window running qemu, every time I attempt SSH I see:

qemu: Slirp: Failed to send packet, ret: -1

Lots of googling but no answers that I can find. Disabling the firewall on the host makes no difference.

Also tried the dtb for the 3b-plus, as well as the ones directly from raspberrypi/firmware, no difference with any of these that I can tell.

Any ideas? Anyone have SSH access working with an emulated aarch64 image?

Maybe it’s just a problem with the -M raspi3b or the dtb files or something?

2

to me it looks a lot like a problem with the network interface of the device, especially the Slirp error message. Could be that some kernel module (in the host or client?) is missing or broken…

Maybe it works with a tap interface as described in this blog?
If not I would try different things with the -netdev parameter from the official docs.

3

I agree it looks like an odd error!

I don’t think it’s the host though; the below very similar script works fine for a recent (2022-04) Raspberry Pi OS image; with a copy of the dtb and kernel8 from the /boot partition, I’m able to use the keyboard as well as SSH into the guest without difficulty:

#!/usr/bin/env bash

# $ qemu-system-aarch64 --version
# QEMU emulator version 7.0.0
# Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
 
# Working fine for Raspberry Pi OS 2022-04

set -Eeuf -o pipefail
set -x

main() {
  local img=${1:-./2022-04-04-rpios.img}
  qemu-system-aarch64 \
    -M raspi3b \
    -dtb ./rpios/bcm2710-rpi-3-b.dtb \
    -m 1G \
    -smp 4 \
    -drive "format=raw,file=${img}" \
    -kernel ./rpios/kernel8.img \
    -device usb-net,netdev=net0 \
    -netdev user,id=net0,hostfwd=tcp::2222-:22 \
    -append "console=ttyAMA0 root=/dev/mmcblk0p2 rw rootwait" \
    -nographic
}

main "$@"

I’ve tried virtually all combinations of these options I could find! None seem to work, and as noted above it seems like these work fine with an official RPi image, so I’m not sure where the issue lies.

Perhaps as you suggest I need to enable a few more kernel modules in my image?

Is there anyone that is using qemu to access an aarch64 NixOS guest?

4

I’m not sure user mode networking will work here. Try using a bridge or tun interface.

5

Interesting, what makes you say that? (This is definitely over my head.)

It looks like NixOS at least knows about slirp – is it just not enabled or something? https://github.com/NixOS/nixpkgs/blob/3eec36f837f4dd5fcd6437eda2279cdafd60080f/pkgs/development/libraries/libslirp/default.nix

6

The problem isn’t that networking isn’t emulated (you can reach the internet from the VM just fine), it’s that the host machine doesn’t know how to reach the VM.

You tried to remedy this by forwarding a port and that could work with user-mode network emulation I think but I’m not sure your config is correct.

Having a proper network that the OS knows about to do routing etc. for you as expected would be better.

7

I tried enabling settings found here in my sdimage, specifically:

import [ <nixpkgs/nixos/modules/profiles/qemu-guest.nix> ];
...
services.qemuGuest.enable = true;

which made no apparent difference.

8

Also, the boot process hangs for the usual 90 seconds on a start job is running for /sys/sub....../devices/eth0, so I guess the network device isn’t being detected or set up correctly?

9

How do you build the image?
I haven’t looked into it yet but I’m running Pis as servers and am wiling to help solve this problem because it would be nice for testing.

10

Thanks for your willingness to help!

I’m starting with just the pre-made aarch64 sd image from hydra.

11

Ah right I see it now in the first post.
Skipped it a bit too fast :).

12

I had the same problems as you with your example config.
I gave nixos-rebuild build-vm a try and in that VM they keyboard works fine and I have internet.
I check tomorrow what other options that VM uses.

13

Huh. So is that from an x86 NixOS in both cases?

14

Yes just a quick test with the config of my notebook to see if there maybe is a general problem.

15

The problem for the ethernet might be this one here:

I looked at the issue from 虎游 @hu60.cn about the dwc2 problem, and it looks like the issue there is that he/she is using a modular kernel, but the root disk image does not contain any modules for that kernel. It looks like at least cdc_ether.ko and cdc_subset.ko are needed for the ethernet to work. I checked that by using the 2020-08-20-raspios-buster image that is referenced on the page where they downloaded that kernel, and with that the network comes up.

When I build my Raspi Flake I can see the files in the result.
However I’m not good enough to know if they are in the image provided by NixOS.

As for the keyboard issue I found out that it seems to be working without a problem in the uboot menu but then failes at a later stage.
And with these parameters you don’t have to search for the bus number:
-usb -device usb-mouse -device usb-kbd \

16

I tried this project and it works just fine (SSH and keyboard) and the qemu command looks very similar to yours.

I’m going to test the parameters to see if the help but doubt it.
My guess is more that we need a different NixOS image for this.
Edit: I replaced the images from the repo with the kernel and image from NixOS and get the same behavior.

17

The same working behavior (as with the repo’s original image)? Or the same non-working behavior (as with the NixOS image)?

18

The same non-working behavior.

19

Not too surprising, as I noted here I have no problems with a Raspberry Pi OS image.

20

Right, I seem to be having a hard time reading :sweat_smile:
Well at least we got the same behavior :slight_smile:
I’m currently trying to build my config as an image and see if that boots.

1 Like