Nixos + vfio gpu passthrough

Help
1

My hardware:

  • GTX 1080
  • RX 6900 XT (https://youtu.be/kAgIQGZUIP0?si=0XjxB8curF5tmzYV&t=324)
  • AMD Ryzen 7 5700X 3.4 GHz 8-Core Processor
  • ASRock B550M PG RIPTIDE Micro ATX AM4 Motherboard, bios 2.80 (could upgrade to 3.40, but I’m afraid to lose vfio/passthrough compatibility as changelogs are practically null).

Astrid tech’s guide is one I’ve followed verbatim multiple times (A GPU Passthrough Setup for NixOS (with VR passthrough too!)) with no results. I even purchased the same motherboard + CPU that the guide talks about.

EFI/BIOS (default + changes = ):
|- ADVANCE
|----XMP-II = ENABLED
|----PSS SUPPORT = ENABLED
|----NX MODE = ENABLED
|----SVM MODE = ENABLED***
|----SMT MODE = ENABLED
|----AMD fTPM SWITCH = AMD CPU fTPM
|- ADVANCE\AMD CBS\NBIO COMMON OPTIONS
|----IOMMU = ENABLED***
|----DMA PROTECTION = AUTO
|----DMAr SUPPORT = AUTO
|----PCIe ARI SUPPORT = DISABLE
|----PCIe ARI ENUMERATION = DISABLE
|----ENABLE AER CAP = DISABLED
|----SRIS = AUTO

Pastebins:
|- configuration.nix
|----pre-requisites.nix (pre-requisites.nix - Pastebin.com)
|----vfio-isis.nix (vfio-isis.nix - Pastebin.com)
|- boot log (boot dump - Pastebin.com)
|- sudo dmesg | grep -i -e DMAR -e IOMMU && lspci -nnk (asdfasdf - Pastebin.com)

I don’t understand why the vfio drivers are not attached to my RX 6900 XT.

08:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] (rev c0)
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 3952
	Kernel driver in use: amdgpu
	Kernel modules: amdgpu

Please help me attach the vfio drivers to the RX 6900 XT.

2

You could be interested into looking glass:

https://looking-glass.io/

They are just preparing their new release:

2 Likes
3

Im doing the same thing right now and ran into a whole mess of issues lol. But I’ve only been using Nix for a week or so. Its hard to do on any regular distro but it is doubly as hard on Nix ngl. But what I had to do was add:

boot.kernelParams = [
    intel_iommu=on
    iommu=pt
    vfio-pci.ids="10de:2489,10de:228b"
];

note the PCI ID’s “,” separated.
You can get this by running:

lspci -nn | grep -iE '(nvidia|amd)'

and finding your GPU. The Ids are in “[brackets:like_this]” You need both the GPU ID and the GPU Audio ID if you have it.

this will blacklist the GPU from getting attached to drivers from what I understand. Its a pain though. Also I also used Astrid’s blog as reference but it’s important to note that this is not a full guide and does skip some important steps. She just shows some of the adapted steps in NixOS.

I ran into a ton of issues, like virt-manager crashing because it didn’t have a GTK cursor enabled… and virt-manager not being to access any directory besides “/” lol I might also do a write up because all the existing docs are essentially nil

edit: lspci I believe is in pkgs.usbutils or pkgs.toybox I think. You can use “nix search nixpkgs lspci”

3 Likes
5
  • quick install of all my apps with a simple config file? check.
  • volatile system? check.
  • single use / runtime apps? check.
  • gpu-passthrough / gaming? check.
  • vms? check.

wish I could get it to work though.

sigh.

I’ll pay anyone that would help me set this up. I’ll send you a mobo+cpu. I will donate to your favorite charity. Anything… you name it. Help this geek out. I lack the technical knowledge to continue and after all this time I’m frankly burnt out.

6

What I have to share is probably of no use, but I was just going down this rabbit hole a few minutes ago and decided to put it off. I found Notes on PCI Passthrough on NixOS using QEMU and VFIO and GitHub - j-brn/nixos-vfio: NixOS modules to automate vfio setups which you might find interesting.

7

I don’t have specific experience here, but did you try adding "amdgpu" to boot.blacklistedKernelModules?

1 Like
9

so does all this blacklisting of the passed through GPU then render it disabled to be used for things like video editing on the nix side?

10

Here you create a custom.

  options.vfio.enable = with lib;
    mkEnableOption "Configure the machine for VFIO";

There rest of the config only applies if it is enabled.
In configuration.nix

imports = [./vfio-isis.nix]
vfio.enable = true;

Since configuration.nix is not available, make sure you’ve included that line, and then nixos-rebuild.

Additionally, you might prefer to use the nixos modules for libvirtd and qemu.
See here