Nixos + vfio gpu passthrough

My hardware:

  • GTX 1080
  • RX 6900 XT (https://youtu.be/kAgIQGZUIP0?si=0XjxB8curF5tmzYV&t=324)
  • AMD Ryzen 7 5700X 3.4 GHz 8-Core Processor
  • ASRock B550M PG RIPTIDE Micro ATX AM4 Motherboard, bios 2.80 (could upgrade to 3.40, but I’m afraid to lose vfio/passthrough compatibility as changelogs are practically null).

Astrid tech’s guide is one I’ve followed verbatim multiple times (A GPU Passthrough Setup for NixOS (with VR passthrough too!)) with no results. I even purchased the same motherboard + CPU that the guide talks about.

EFI/BIOS (default + changes = ):
|- ADVANCE
|----XMP-II = ENABLED

|----PSS SUPPORT = ENABLED
|----NX MODE = ENABLED
|----SVM MODE = ENABLED***
|----SMT MODE = ENABLED
|----AMD fTPM SWITCH = AMD CPU fTPM
|- ADVANCE\AMD CBS\NBIO COMMON OPTIONS
|----IOMMU = ENABLED***
|----DMA PROTECTION = AUTO
|----DMAr SUPPORT = AUTO
|----PCIe ARI SUPPORT = DISABLE
|----PCIe ARI ENUMERATION = DISABLE
|----ENABLE AER CAP = DISABLED
|----SRIS = AUTO

Pastebins:
|- configuration.nix
|----pre-requisites.nix (pre-requisites.nix - Pastebin.com)
|----vfio-isis.nix (vfio-isis.nix - Pastebin.com)
|- boot log (boot dump - Pastebin.com)
|- sudo dmesg | grep -i -e DMAR -e IOMMU && lspci -nnk (asdfasdf - Pastebin.com)

I don’t understand why the vfio drivers are not attached to my RX 6900 XT.

08:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] (rev c0)
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 3952
	Kernel driver in use: amdgpu
	Kernel modules: amdgpu

Please help me attach the vfio drivers to the RX 6900 XT.

You could be interested into looking glass:

https://looking-glass.io/

They are just preparing their new release:

2 Likes

Im doing the same thing right now and ran into a whole mess of issues lol. But I’ve only been using Nix for a week or so. Its hard to do on any regular distro but it is doubly as hard on Nix ngl. But what I had to do was add:

boot.kernelParams = [
    intel_iommu=on
    iommu=pt
    vfio-pci.ids="10de:2489,10de:228b"
];

note the PCI ID’s “,” separated.
You can get this by running:

lspci -nn | grep -iE '(nvidia|amd)'

and finding your GPU. The Ids are in “[brackets:like_this]” You need both the GPU ID and the GPU Audio ID if you have it.

this will blacklist the GPU from getting attached to drivers from what I understand. Its a pain though. Also I also used Astrid’s blog as reference but it’s important to note that this is not a full guide and does skip some important steps. She just shows some of the adapted steps in NixOS.

I ran into a ton of issues, like virt-manager crashing because it didn’t have a GTK cursor enabled… and virt-manager not being to access any directory besides “/” lol I might also do a write up because all the existing docs are essentially nil

edit: lspci I believe is in pkgs.usbutils or pkgs.toybox I think. You can use “nix search nixpkgs lspci”

3 Likes
  • quick install of all my apps with a simple config file? check.
  • volatile system? check.
  • single use / runtime apps? check.
  • gpu-passthrough / gaming? check.
  • vms? check.

wish I could get it to work though.

sigh.

I’ll pay anyone that would help me set this up. I’ll send you a mobo+cpu. I will donate to your favorite charity. Anything… you name it. Help this geek out. I lack the technical knowledge to continue and after all this time I’m frankly burnt out.

What I have to share is probably of no use, but I was just going down this rabbit hole a few minutes ago and decided to put it off. I found Notes on PCI Passthrough on NixOS using QEMU and VFIO and GitHub - j-brn/nixos-vfio: NixOS modules to automate vfio setups which you might find interesting.

I don’t have specific experience here, but did you try adding "amdgpu" to boot.blacklistedKernelModules?

Thank you all for your help! Without you guys I wouldn’t have made it.

I’m very sorry it took this long for me to give a reply. I moved to a new apartment and my NAS (where I had 80-90% of my data) died on me so I had to make a new account. I’m putting that data recovery journey on hold for a couple of days. That’s why I’m on a new account and I can’t mark the issue as solved.

For anyone searching the forums for this topic I recommend you read: How To Ask Questions The Smart Way by Eric Steven Raymond (http://www.catb.org/esr/faqs/smart-questions.html) first. That will get you in the right mindset and offer some valuable insights for the battles to come.

Then read:

Finally use:

  1. Notes on PCI Passthrough on NixOS using QEMU and VFIO
  2. A GPU Passthrough Setup for NixOS (with VR passthrough too!)

:dragon: Dragons slayed:

  • GPU pt. Fixed: Alexbakker + Astrid’s blog is 80-98% of the way.
  • VFIO driver not loading. Fixed: Blocked it from loading and used Alexbakker’s script.
  • Mouse and Keyboard pt. Fixed: usb.sh script by astrid.
  • Microsoft Basic Display adapter loads, the GPU appears, then it rolls back to the Ms Basic Display Adapter.Fixed: Bios setting disabled resizable BAR.

:dragon: Dragons (next challenges):

  • Motherboard audio pt.
  • Sound blaster AE7 pt.
  • A DIY solution that probably involves PCIe extensions (8-10 inches long) for 2 GPU’s and 1 sound card. That will mount all the hardware and improve cooling on my case (thermaltake v21).
  • Try looking glass.

My setup:

  • GPU(1): AMD Radeon RX 6900 XT for the guest OS.
  • GPU(2): Nvidia GTX 1080 for the host OS.
  • CPU: Ryzen 7 5700X 3.4 GHz 8-Core Processor
  • Mobo: ASRock B550M PG RIPTIDE Micro ATX AM4 Motherboard
  • Host: nixos 23.11 (tapir)
  • Guest: Windows 10
  • AMD Adrenaline driver 21.4.1
  • Monitor(1): Old crappy Asus
  • Monitor(2): Samsung G3 144hz

Bios:

  • XMP 2.0 enabled. Nothing to do with VFIO just remember to take advantage of it :slight_smile:
  • Advance\PCI configuration\SR-IOV Support = disabled. This disables resizable bar which was needed to finally install the graphic card drivers.
  • Advance\AMD CBS\NBIO Common Options >IOMMU = enabled.
  • Advance\AMD CBS\NBIO Common Options >DMA Protection = disabled
  • Advance\AMD CBS\NBIO Common Options >DMAr support = disabled
  • Advance\AMD CBS\NBIO Common Options >PCIe ARI support = disabled
  • Advance\AMD CBS\NBIO Common Options >Enable AER Cap = disabled

/etc/nixos/configuration.nix:

  1. blacklist the kernel module of the GPU you want to passthrough (pt)
  2. make sure the kernel modules for the GPU pt + vfio-pci are available
  3. bind the vfio drivers
  4. kernel parameters for pt and iommu
  5. software: toybox, pciutils, virt-manager, qemu, libvirt, OVMFFull, dnsmasq, usbutttils, dmidecode.

...  
  # Bootloader.
  boot.loader.systemd-boot.enable = true;
  boot.loader.efi.canTouchEfiVariables = true;
  boot.blacklistedKernelModules = [ "amdgpu" ];
  boot.initrd.availableKernelModules = [ "amdgpu" "vfio-pci" ];
  boot.initrd.preDeviceCommands = ''
    DEVS="0000:09:00.0 0000:09:00.1"
    for DEV in $DEVS; do
      echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
    done
    modprobe -i vfio-pci
  '';
  boot.kernelPackages = pkgs.linuxPackages_latest;
  boot.kernelParams = [ "amd_iommu=on" "iommu=pt" "pcie_aspm=off" ];
  boot.kernelModules = [ "kvm-amd" "vfio_pci" "vfio" "vfio_iommu_type1" "vfio_virqfd" "amd_iommu_v2" "iommu_v2" ];
...
  # List packages installed in system profile. To search, run:
  # $ nix search wget
  environment.systemPackages = with pkgs; [
    vim # Do not forget to add an editor to edit configuration.nix! The Nano editor is also installed by default.
    wget
    toybox
    pciutils
    virt-manager
    qemu
    libvirt
    OVMFFull
    dnsmasq
    usbutils
    dmidecode
  ];
...
3 Likes

so does all this blacklisting of the passed through GPU then render it disabled to be used for things like video editing on the nix side?