[vfio-users] (good) working GPU for passthrough?

[Bronek Kozicki]

On Fri, 18 Jan 2019, at 1:39 PM, Bronek Kozicki wrote:
> On Thu, 17 Jan 2019, at 10:39 PM, Kash Pande wrote:
> > On 2019-01-17 2:03 p.m., Tobias Geiger wrote:
> > > I tried a VEGA 64 - i was able to live with the ACS patch needed here
> > > (Z170 Chipset... not needed with the old HD7800, but whatever...) -
> > > but i couldn't stand the reset/FLR/Bug which forces you at least to
> > > suspend/resume the host when you want to reboot only the guest... 
> > 
> > 
> > AIUI, this does not happen using the Q35 machine type with proper PCIe
> > root complex heirarchy.
> > 
> 
> 
> I do not have great experience with "Q35 machine type with proper PCIe 
> root complex hierarchy". Selection of the chipset type Q35 in the virt-
> manager upon creation does not seem to work with UEFI type of firmware 
> (as recommended for vfio passthrough). Or perhaps I should have 
> downloaded a more recent Tianocore UEFI firmware for my virtual 
> machines?


So I did some more experiments and learned what I suppose everyone else knew already: if switching to SCSI for CDROM, make sure to change the type of the controller to VirtIO from the default LSI.

Also, the host machine must have EFI firmware for guest to boot from. For ArchLinux the appropriate package is ovmf (or ovmf-git from AUR, for adventurous). Setting nvram in /etc/libvirt/qemu.conf must point to the EFI binaries installed by this package. Other distributions also have this package https://pkgs.org/download/ovmf , but that's not news.

Initially, my attempts were failing because OVMF EFI is unable to boot from LSI SCSI controller. This is obviously unrelated to GPU passthrough, except for me changing the incorrect CDROM type from IDE to SCSI (I would not have that problem if I changed to SATA).

Comparing the .xml it is obvious that virt-manager actually constructs the PCI topology with PCIe root for q35, here is appropriate diff excerpt from two nearly identical machines, one created as i440fx and another as q35 (with the above extra steps).

-    <controller type='pci' index='0' model='pci-root'/>
+    <controller type='pci' index='0' model='pcie-root'/>
+    <controller type='pci' index='1' model='pcie-root-port'>
+      <model name='pcie-root-port'/>
+      <target chassis='1' port='0x10'/>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
+    </controller>
+    <controller type='pci' index='2' model='pcie-root-port'>
+      <model name='pcie-root-port'/>
+      <target chassis='2' port='0x11'/>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
+    </controller>
+    <controller type='pci' index='3' model='pcie-to-pci-bridge'>
+      <model name='pcie-pci-bridge'/>
+      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
+    </controller>
+    <controller type='pci' index='4' model='pcie-root-port'>
+      <model name='pcie-root-port'/>
+      <target chassis='4' port='0x12'/>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
+    </controller>
+    <controller type='pci' index='5' model='pcie-root-port'>
+      <model name='pcie-root-port'/>
+      <target chassis='5' port='0x13'/>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
+    </controller>
+    <controller type='pci' index='6' model='pcie-root-port'>
+      <model name='pcie-root-port'/>
+      <target chassis='6' port='0x14'/>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
+     </controller>

This is starting to make sense, going to try passthrough with q35 and UEFI now.


B.