sKeletoN_SN wrote on 2023-11-27, 11:26:
darry wrote on 2023-11-27, 05:17:
sKeletoN_SN wrote on 2023-11-26, 23:17:
So if I understand it corretly:
- You basically didn't take the GPU ids from its corresponding IOMMU group, adding them to grub and pass them to a VFIO driver.
No,I did not, at least not on my Debian 12 setup so far.
Additionally, in case the question pops up, as that board has arguably braindead device vs IOMMU grouping, if I passthrough any PCIE card, I also need to passthrough the onboard NIC, SATA controllers and chipset based USB controllers, which are all in the same IOMMU group. I use a USB NIC connected to the CPU provided USB controller (which is in its own IOMMU group). Working around the IOMMU issue would imply using a patched kernel which supports ACS override. I do not want to go there, less for the security concerns than because I do not want to have to patch and and compile kernel every time I want to update it I believe some distros offer precompiled kernels with that patch included but, last time I checked, Debian was not among them.
You do not need to compile the kernel in order to use the ACS patch (at least with my experience of using Manjaro). ACS patch gets applied by passing an argument to the GRUB_CMDLINE_LINUX_DEFAULT parameter which is "pcie_acs_override=downstream,multifunction". After this is done you can generate a new grub file with "sudo grub-mkconfig -o /boot/grub/grub.cfg". However, I do not know how it will work in Debian, but in my case with Manjaro linux I had no problems.
AFAICT, stock Debian kernels do not have this patch included (I added pcie_acs_override and rebooted as a test).
On a completely different note, I am able to passthrough a GTX 750 TI to a Windows XP VMrunning on a 440FX machine type, which allows 4GB (3.25GB or so usable).
If I used the same VM definition and either replace the GTX 750 TI passthrough with a GTX 970 OR just add a GTX 970 TI as a secondary GPU, none of the GPUs works in the VM (both get a code 10, as confirmed through a remote desktop session). All theses combination work fine if I use a Q35 machine type, which limits Windows XP to 2GB of RAM . Maybe this has something to do with the fact that the GTX 750TI is a chipset provided PCIE slot (x4 PCIE 2.0 ) running through a PCIE to PCIE bridge or two, but I suspect it is more likely to be something specific to the 970 and I don't care enough to swap slots to verify as I don't care enough about 1.25 extra GB of RAM or have any intention of running anything other than a 970 (and do not want have it bandwidth starved).
[1]
root@pvr2:~# uname -a
Linux pvr2 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64 GNU/Linux
root@pvr2:~# cat /etc/*release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@pvr2:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.1.0-13-amd64 root=UUID=[OBFUSCATED] ro iommu=1 amd_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream,multifunction
root@pvr2:~# ./ig.sh | egrep "IOMMU Group 11"
IOMMU Group 11 16:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 xHCI Compliant Host Controller [1022:43d5] (rev 01)
IOMMU Group 11 16:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
IOMMU Group 11 16:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
IOMMU Group 11 20:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 11 20:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 11 20:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 11 22:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU Group 11 25:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] [10de:1380] (rev a2)
IOMMU Group 11 25:00.1 Audio device [0403]: NVIDIA Corporation GM107 High Definition Audio Controller [GeForce 940MX] [10de:0fbc] (rev a1)