# How to ssh to qemu emulated aarch64 NixOS?
**URL:** https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081
**Category:** Help
**Created:** [July 1, 2022, 10:27pm UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081 "2022-07-01T22:27:40Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@n8henrie](https://discourse.nixos.org/u/n8henrie)
#### Post date: [July 1, 2022, 10:27pm UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/1 "2022-07-01T22:27:40Z")
I’m continuing to mess with [NixOS-Pi on BTRFS subvolume-based compressed root](https://discourse.nixos.org/t/btrfs-pi-wont-boot-from-compressed-subvolume/18462). It seems like it would be a lot easier if I could get it working with qemu – I’ve burned through several SD cards over the last year+ on this project, and I’m tired of burning, walking across the room to the Pi, unplugging, replugging, etc.
Unfortunately I can’t get keyboard or network / SSH access to work.
The host I’m running from is arch linux, x86\_64.
My setup process is basically:
```plaintext
$ wget 'https://hydra.nixos.org/build/182205320/download/1/nixos-sd-image-22.05.1322.915f5a5b3cc-aarch64-linux.img.zst'
$ zstd -d nixos-sd-image-22.05.1322.915f5a5b3cc-aarch64-linux.img.zst
$ qemu-img resize -f raw nixos-sd-image-22.05.1322.915f5a5b3cc-aarch64-linux.img 4G
$ mv nixos-sd-image-22.05.1322.915f5a5b3cc-aarch64-linux.img nixos-22.05.img
```
Next, I mount the image and copy `u-boot-rpi3.bin` from partition 1, and `bcm2837-rpi-3-b.dtb` from `/boot/nixos/0s76maj95yj311viyg7yhrrp300ag8s6-linux-5.15.49-dtbs/broadcom/`.
Finally, I run this:
```bash
#!/usr/bin/env bash
# nixos.sh
# $ qemu-system-aarch64 --version
# QEMU emulator version 7.0.0
# Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
set -Eeuf -o pipefail
set -x
main() {
local img=${1:-./nixos-22.05.img}
qemu-system-aarch64 \
-M raspi3b \
-dtb ./bcm2837-rpi-3-b.dtb \
-m 1G \
-smp 4 \
-drive "format=raw,file=${img}" \
-kernel ./u-boot-rpi3.bin \
-device usb-net,netdev=net0 \
-netdev user,id=net0,hostfwd=tcp::2222-:22 \
-usb -device usb-host,hostbus=1,hostaddr=2 \
-append "console=ttyAMA0 root=/dev/mmcblk0p2 rw rootwait" \
-serial stdio
}
main "$@"
```
I see what looks like a pretty normal boot process which eventually gets to a login prompt.
At the login prompt, my USB keyboard doesn’t seem to work (I’ve confirmed those bus numbers), but it does get stuck in the qemu window; I have to SSH into the host from another machine and run this to get my keyboard back:
```plaintext
$ sudo sh -c 'echo 0000:01:00.0 > /sys/bus/pci/drivers/xhci_hcd/unbind'
$ sudo sh -c 'echo 0000:01:00.0 > /sys/bus/pci/drivers/xhci_hcd/bind'
```
What I’d _really_ like is to get SSH access, but that doesn’t work either. `ssh -vvv -p 2222 root@localhost` shows me:
```auto
...
debug1: Connecting to localhost [127.0.0.1] port 2222.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug1: Connection established.
...
```
but then hangs there. In the window running `qemu`, every time I attempt SSH I see:
```auto
qemu: Slirp: Failed to send packet, ret: -1
```
Lots of googling but no answers that I can find. Disabling the firewall on the host makes no difference.
Also tried the dtb for the 3b-plus, as well as the ones directly from raspberrypi/firmware, no difference with any of these that I can tell.
Any ideas? Anyone have SSH access working with an emulated aarch64 image?
Maybe it’s just a problem with the `-M raspi3b` or the dtb files or something?
---
### Author:  [@Melkor333](https://discourse.nixos.org/u/Melkor333)
#### Post date: [July 2, 2022, 6:35am UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/2 "2022-07-02T06:35:17Z")
to me it looks a lot like a problem with the network interface of the device, especially the `Slirp` error message. Could be that some kernel module (in the host or client?) is missing or broken…
Maybe it works with a tap interface as described in [this blog](https://azeria-labs.com/emulate-raspberry-pi-with-qemu/)?
If not I would try different things with the `-netdev` parameter from the [official docs](https://wiki.qemu.org/Documentation/Networking).
---
### Author:  [@n8henrie](https://discourse.nixos.org/u/n8henrie)
#### Post date: [July 2, 2022, 9:50pm UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/3 "2022-07-02T21:50:55Z")
> [@Melkor333](#):
>
> to me it looks a lot like a problem with the network interface of the device
I agree it looks like an odd error!
I don’t think it’s the host though; the below _very_ similar script works fine for a recent (2022-04) Raspberry Pi OS image; with a copy of the dtb and `kernel8` from the `/boot` partition, I’m able to use the keyboard as well as SSH into the guest without difficulty:
```bash
#!/usr/bin/env bash
# $ qemu-system-aarch64 --version
# QEMU emulator version 7.0.0
# Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
# Working fine for Raspberry Pi OS 2022-04
set -Eeuf -o pipefail
set -x
main() {
local img=${1:-./2022-04-04-rpios.img}
qemu-system-aarch64 \
-M raspi3b \
-dtb ./rpios/bcm2710-rpi-3-b.dtb \
-m 1G \
-smp 4 \
-drive "format=raw,file=${img}" \
-kernel ./rpios/kernel8.img \
-device usb-net,netdev=net0 \
-netdev user,id=net0,hostfwd=tcp::2222-:22 \
-append "console=ttyAMA0 root=/dev/mmcblk0p2 rw rootwait" \
-nographic
}
main "$@"
```
> [@Melkor333](#):
>
> If not I would try different things with the `-netdev` parameter from the [official docs](https://wiki.qemu.org/Documentation/Networking).
I’ve tried virtually all combinations of these options I could find! None seem to work, and as noted above it seems like these work fine with an official RPi image, so I’m not sure where the issue lies.
Perhaps as you suggest I need to enable a few more kernel modules in my image?
Is there anyone that is using qemu to access an aarch64 NixOS guest?
---
### Author:  [@Atemu](https://discourse.nixos.org/u/Atemu)
#### Post date: [July 3, 2022, 2:01am UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/4 "2022-07-03T02:01:55Z")
I’m not sure user mode networking will work here. Try using a bridge or tun interface.
---
### Author:  [@n8henrie](https://discourse.nixos.org/u/n8henrie)
#### Post date: [July 3, 2022, 12:15pm UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/5 "2022-07-03T12:15:01Z")
> [@Atemu](#):
>
> not sure user mode networking will work
Interesting, what makes you say that? (This is definitely over my head.)
It looks like NixOS at least knows about slirp – is it just not enabled or something? [nixpkgs/pkgs/development/libraries/libslirp/default.nix at 3eec36f837f4dd5fcd6437eda2279cdafd60080f · NixOS/nixpkgs · GitHub](https://github.com/NixOS/nixpkgs/blob/3eec36f837f4dd5fcd6437eda2279cdafd60080f/pkgs/development/libraries/libslirp/default.nix)
---
### Author:  [@Atemu](https://discourse.nixos.org/u/Atemu)
#### Post date: [July 4, 2022, 10:26am UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/6 "2022-07-04T10:26:19Z")
The problem isn’t that networking isn’t emulated (you can reach the internet from the VM just fine), it’s that the host machine doesn’t know how to reach the VM.
You tried to remedy this by forwarding a port and that could work with user-mode network emulation I think but I’m not sure your config is correct.
Having a proper network that the OS knows about to do routing etc. for you as expected would be better.
---
### Author:  [@n8henrie](https://discourse.nixos.org/u/n8henrie)
#### Post date: [July 4, 2022, 10:53am UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/7 "2022-07-04T10:53:07Z")
I tried enabling settings found [here](https://alpmestan.github.io/notes/nixos-qemu.html) in my sdimage, specifically:
```nix
import [];
...
services.qemuGuest.enable = true;
```
which made no apparent difference.
---
### Author:  [@n8henrie](https://discourse.nixos.org/u/n8henrie)
#### Post date: [July 4, 2022, 11:22am UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/8 "2022-07-04T11:22:00Z")
Also, the boot process hangs for the usual 90 seconds on `a start job is running for /sys/sub....../devices/eth0`, so I guess the network device isn’t being detected or set up correctly?
---
### Author:  [@Nebucatnetzer](https://discourse.nixos.org/u/Nebucatnetzer)
#### Post date: [July 4, 2022, 6:12pm UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/9 "2022-07-04T18:12:31Z")
How do you build the image?
I haven’t looked into it yet but I’m running Pis as servers and am wiling to help solve this problem because it would be nice for testing.
---
### Author:  [@n8henrie](https://discourse.nixos.org/u/n8henrie)
#### Post date: [July 4, 2022, 6:39pm UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/10 "2022-07-04T18:39:23Z")
Thanks for your willingness to help!
I’m starting with just the pre-made aarch64 sd image from hydra.
---
### Author:  [@Nebucatnetzer](https://discourse.nixos.org/u/Nebucatnetzer)
#### Post date: [July 4, 2022, 6:49pm UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/11 "2022-07-04T18:49:21Z")
Ah right I see it now in the first post.
Skipped it a bit too fast :).
---
### Author:  [@Nebucatnetzer](https://discourse.nixos.org/u/Nebucatnetzer)
#### Post date: [July 4, 2022, 9:02pm UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/12 "2022-07-04T21:02:09Z")
I had the same problems as you with your example config.
I gave `nixos-rebuild build-vm` a try and in that VM they keyboard works fine and I have internet.
I check tomorrow what other options that VM uses.
---
### Author:  [@n8henrie](https://discourse.nixos.org/u/n8henrie)
#### Post date: [July 4, 2022, 11:26pm UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/13 "2022-07-04T23:26:52Z")
Huh. So is that from an x86 NixOS in both cases?
---
### Author:  [@Nebucatnetzer](https://discourse.nixos.org/u/Nebucatnetzer)
#### Post date: [July 5, 2022, 4:20am UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/14 "2022-07-05T04:20:57Z")
Yes just a quick test with the config of my notebook to see if there maybe is a general problem.
---
### Author:  [@Nebucatnetzer](https://discourse.nixos.org/u/Nebucatnetzer)
#### Post date: [July 5, 2022, 5:46pm UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/15 "2022-07-05T17:46:12Z")
The problem for the ethernet might be this one here:
> **[USB Ethernet device (RNDIS) does not work on several tested operating systems...](https://gitlab.com/qemu-project/qemu/-/issues/198#note_599825827)**
>
> The USB ethernet device does not work on most versions of operating systems I have tested. For each operating system the command to use this device was: -netdev...
> I looked at the issue from 虎游 [@hu60.cn](https://discourse.nixos.org/hu60.cn) about the dwc2 problem, and it looks like the issue there is that he/she is using a modular kernel, but the root disk image does not contain any modules for that kernel. It looks like at least cdc\_ether.ko and cdc\_subset.ko are needed for the ethernet to work. I checked that by using the 2020-08-20-raspios-buster image that is referenced on the page where they downloaded that kernel, and with that the network comes up.
When I build my Raspi Flake I can see the files in the result.
However I’m not good enough to know if they are in the image provided by NixOS.
As for the keyboard issue I found out that it seems to be working without a problem in the uboot menu but then failes at a later stage.
And with these parameters you don’t have to search for the bus number:
`-usb -device usb-mouse -device usb-kbd \`
---
### Author:  [@Nebucatnetzer](https://discourse.nixos.org/u/Nebucatnetzer)
#### Post date: [July 5, 2022, 7:00pm UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/16 "2022-07-05T19:00:28Z")
I tried this project and it works just fine (SSH and keyboard) and the qemu command looks very similar to yours.
> **[GitHub - farabimahmud/emulate-raspberry-pi3-in-qemu: I have tried to figure...](https://github.com/farabimahmud/emulate-raspberry-pi3-in-qemu)**
>
> I have tried to figure out a way to run raspberry pi using QEMU but could not. So here I am putting all the references I have pieced together that should just work out of the box. - GitHub - farab...
I’m going to test the parameters to see if the help but doubt it.
My guess is more that we need a different NixOS image for this.
Edit: I replaced the images from the repo with the kernel and image from NixOS and get the same behavior.
---
### Author:  [@n8henrie](https://discourse.nixos.org/u/n8henrie)
#### Post date: [July 5, 2022, 7:41pm UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/17 "2022-07-05T19:41:36Z")
> [@Nebucatnetzer](#):
>
> and get the same behavior.
The same working behavior (as with the repo’s original image)? Or the same non-working behavior (as with the NixOS image)?
---
### Author:  [@Nebucatnetzer](https://discourse.nixos.org/u/Nebucatnetzer)
#### Post date: [July 5, 2022, 7:42pm UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/18 "2022-07-05T19:42:37Z")
The same non-working behavior.
---
### Author:  [@n8henrie](https://discourse.nixos.org/u/n8henrie)
#### Post date: [July 5, 2022, 7:43pm UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/19 "2022-07-05T19:43:54Z")
> [@Nebucatnetzer](#):
>
> I tried this project and it works just fine (SSH and keyboard)
Not too surprising, as I noted [here](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/3) I have no problems with a Raspberry Pi OS image.
---
### Author:  [@Nebucatnetzer](https://discourse.nixos.org/u/Nebucatnetzer)
#### Post date: [July 5, 2022, 7:46pm UTC](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081/20 "2022-07-05T19:46:39Z")
Right, I seem to be having a hard time reading 😅
Well at least we got the same behavior 🙂
I’m currently trying to build my config as an image and see if that boots.
[Next page](https://discourse.nixos.org/t/how-to-ssh-to-qemu-emulated-aarch64-nixos/20081.md?page=2)