darry wrote on 2023-11-27, 05:17:

sKeletoN_SN wrote on 2023-11-26, 23:17:

darry wrote on 2023-11-26, 18:08:

I have not bothered to do it cleanly, TBH.

I just boot to a non graphical target (set with systemctl) and I run rmmod -f nouveau (most often not even necessary, I only nedded to rmmod when passing through a second GPU, 750TI, as a test).

I did do it more cleanly a while back on another host running Debian 11, AFAICR.

So if I understand it corretly:
- You basically didn't take the GPU ids from its corresponding IOMMU group, adding them to grub and pass them to a VFIO driver.

No,I did not, at least not on my Debian 12 setup so far.

Additionally, in case the question pops up, as that board has arguably braindead device vs IOMMU grouping, if I passthrough any PCIE card, I also need to passthrough the onboard NIC, SATA controllers and chipset based USB controllers, which are all in the same IOMMU group. I use a USB NIC connected to the CPU provided USB controller (which is in its own IOMMU group). Working around the IOMMU issue would imply using a patched kernel which supports ACS override. I do not want to go there, less for the security concerns than because I do not want to have to patch and and compile kernel every time I want to update it I believe some distros offer precompiled kernels with that patch included but, last time I checked, Debian was not among them.

You do not need to compile the kernel in order to use the ACS patch (at least with my experience of using Manjaro). ACS patch gets applied by passing an argument to the GRUB_CMDLINE_LINUX_DEFAULT parameter which is "pcie_acs_override=downstream,multifunction". After this is done you can generate a new grub file with "sudo grub-mkconfig -o /boot/grub/grub.cfg". However, I do not know how it will work in Debian, but in my case with Manjaro linux I had no problems.

sKeletoN_SN wrote on 2023-11-27, 11:26:

darry wrote on 2023-11-27, 05:17:

sKeletoN_SN wrote on 2023-11-26, 23:17:

So if I understand it corretly:
- You basically didn't take the GPU ids from its corresponding IOMMU group, adding them to grub and pass them to a VFIO driver.

No,I did not, at least not on my Debian 12 setup so far.

Additionally, in case the question pops up, as that board has arguably braindead device vs IOMMU grouping, if I passthrough any PCIE card, I also need to passthrough the onboard NIC, SATA controllers and chipset based USB controllers, which are all in the same IOMMU group. I use a USB NIC connected to the CPU provided USB controller (which is in its own IOMMU group). Working around the IOMMU issue would imply using a patched kernel which supports ACS override. I do not want to go there, less for the security concerns than because I do not want to have to patch and and compile kernel every time I want to update it I believe some distros offer precompiled kernels with that patch included but, last time I checked, Debian was not among them.

You do not need to compile the kernel in order to use the ACS patch (at least with my experience of using Manjaro). ACS patch gets applied by passing an argument to the GRUB_CMDLINE_LINUX_DEFAULT parameter which is "pcie_acs_override=downstream,multifunction". After this is done you can generate a new grub file with "sudo grub-mkconfig -o /boot/grub/grub.cfg". However, I do not know how it will work in Debian, but in my case with Manjaro linux I had no problems.

AFAICT, stock Debian kernels do not have this patch included (I added pcie_acs_override and rebooted as a test).

On a completely different note, I am able to passthrough a GTX 750 TI to a Windows XP VMrunning on a 440FX machine type, which allows 4GB (3.25GB or so usable).

If I used the same VM definition and either replace the GTX 750 TI passthrough with a GTX 970 OR just add a GTX 970 TI as a secondary GPU, none of the GPUs works in the VM (both get a code 10, as confirmed through a remote desktop session). All theses combination work fine if I use a Q35 machine type, which limits Windows XP to 2GB of RAM . Maybe this has something to do with the fact that the GTX 750TI is a chipset provided PCIE slot (x4 PCIE 2.0 ) running through a PCIE to PCIE bridge or two, but I suspect it is more likely to be something specific to the 970 and I don't care enough to swap slots to verify as I don't care enough about 1.25 extra GB of RAM or have any intention of running anything other than a 970 (and do not want have it bandwidth starved).

[1]
root@pvr2:~# uname -a
Linux pvr2 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64 GNU/Linux

root@pvr2:~# cat /etc/*release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

root@pvr2:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.1.0-13-amd64 root=UUID=[OBFUSCATED] ro iommu=1 amd_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream,multifunction

root@pvr2:~# ./ig.sh | egrep "IOMMU Group 11"
IOMMU Group 11 16:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 xHCI Compliant Host Controller [1022:43d5] (rev 01)
IOMMU Group 11 16:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
IOMMU Group 11 16:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
IOMMU Group 11 20:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 11 20:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 11 20:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 11 22:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU Group 11 25:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] [10de:1380] (rev a2)
IOMMU Group 11 25:00.1 Audio device [0403]: NVIDIA Corporation GM107 High Definition Audio Controller [GeForce 940MX] [10de:0fbc] (rev a1)

The GTX 970 is a weird card in that some of its upper memory is accessed differently, which can cause lag in some very rare scenarios. I suspect that you're right about the bridge, but a comparison with a 980 would clarify whether it's a GTX 970-specific issue.

Kahenraz wrote on 2023-11-30, 03:03:

The GTX 970 is a weird card in that some of its upper memory is accessed differently, which can cause lag in some very rare scenarios. I suspect that you're right about the bridge, but a comparison with a 980 would clarify whether it's a GTX 970-specific issue.

Curiosity got the best of me .

In a Windows XP VM, on a B450 chipset board :

The GTX 970 works fine with a 440fx machine type off one of the chipset driven PCIE slots .
The GTX 970 does not work with a 440fx machine type in the CPU driven PCIE slot.
The GTX 970 does work in any PCIE slot with a Q35 machine type.

The memory issue with the GTX 970 was actually the cause for a class action lawsuit against NVIDIA. I owned a 970 for years and loved it. The memory issue did exist but it was very overblown, mostly because games generally never used that much memory to ever trigger it. It's a black mark on an otherwise perfect card.

Kahenraz wrote on 2023-12-01, 16:08:

The memory issue with the GTX 970 was actually the cause for a class action lawsuit against NVIDIA. I owned a 970 for years and loved it. The memory issue did exist but it was very overblown, mostly because games generally never used that much memory to ever trigger it. It's a black mark on an otherwise perfect card.

The fact that it is essentially the newest and fastest (with reasonable TDP) card usable in Windows XP makes it a great option . It also does decent job of covering a significant part of the Windows 7 era .

I tried running the GTX 970 through an Asmedia x1 PCIE switch off the CPU PCIE slot, but had no more success than running it directly with a 440FX machine type. I'm about ready to give up for now .

As an update, I am currently finishing up my XP VM setup and will be getting back to Windows 98 SE + Voodoo 3 after that.

Having the option to boot into a PAE patched XP kernel addresses the 2GB RAM limit with the Q35 machine type (if it ever becomes an issue).