Xen with libvirt and SR-IOV

[Jim Fehlig]

On 3/22/23 04:23, nospam at godawa.de wrote:
> Jim Fehlig schrieb:
>>
>> What is the libvirt version?
> 
> It's the "latest and greatest" I get from this source:
> 
> [root at xen1 ~]# virsh --version
> 6.6.0
> 
> [root at xen1 ~]# libvirtd --version
> libvirtd (libvirt) 6.6.0
> 
> [root at xengfs1f ~]# yum list | grep libvirt
> libvirt.x86_64                           1:6.6.0-6.xen415.el7 @centos-virt-xen-415
> libvirt-bash-completion.x86_64           1:6.6.0-6.xen415.el7 @centos-virt-xen-415
> libvirt-client.x86_64                    1:6.6.0-6.xen415.el7 @centos-virt-xen-415
> libvirt-daemon.x86_64                    1:6.6.0-6.xen415.el7 @centos-virt-xen-415

That's an old libvirt, but there hasn't been a lot of changes to the PCI 
passthrough code. One notable change that came with libvirt 6.8.0

https://gitlab.com/libvirt/libvirt/-/commit/9d15647dcb96831c93ac8c1d67c47265b5ed9072

However, that wouldn't be needed unless you were using the 'permissive' setting 
in your xl config.

>> I know it's not complete, but https://libvirt.org/formatdomain.html makes an 
>> attempt to identify hypervisor support for the various configuration settings.
> 
> I will "recheck" it, but I don't see anything new, that might be missing.
> 
> 
>> I've used both configurations successfully. Has the sriov virtual function 
>> been unbound from the native driver and bound to xen-pciback? 
> 
> Should be, otherwise it would not work with XL?
> 
> 
>> You can check which driver is in use with something like
>>
>> ls /sys/bus/pci/devices/0000\:81\:02.6/driver
> 
> I can't see the exact definition, but again, same output when starting with xl 
> or virsh:
> 
> [root at xen1 ~]# ls /sys/bus/pci/devices/0000\:81\:02.6/driver
> 0000:81:02.0  0000:81:02.7  0000:81:03.6  0000:81:04.5  0000:81:05.4 
> 0000:81:06.5  0000:81:07.4  0000:81:08.3  0000:81:09.2       module  slots
> 0000:81:02.1  0000:81:03.0  0000:81:03.7  0000:81:04.6  0000:81:05.5 
> 0000:81:06.6  0000:81:07.5  0000:81:08.4  0000:81:09.3       new_id  uevent
> 0000:81:02.2  0000:81:03.1  0000:81:04.0  0000:81:04.7  0000:81:06.0 
> 0000:81:06.7  0000:81:07.6  0000:81:08.5  0000:81:09.4       new_slot  unbind
> 0000:81:02.3  0000:81:03.2  0000:81:04.1  0000:81:05.0  0000:81:06.1 
> 0000:81:07.0  0000:81:07.7  0000:81:08.6  0000:81:09.5       permissive
> 0000:81:02.4  0000:81:03.3  0000:81:04.2  0000:81:05.1  0000:81:06.2 
> 0000:81:07.1  0000:81:08.0  0000:81:08.7  bind               quirks
> 0000:81:02.5  0000:81:03.4  0000:81:04.3  0000:81:05.2  0000:81:06.3 
> 0000:81:07.2  0000:81:08.1  0000:81:09.0  irq_handlers       remove_id
> 0000:81:02.6  0000:81:03.5  0000:81:04.4  0000:81:05.3  0000:81:06.4 
> 0000:81:07.3  0000:81:08.2  0000:81:09.1  irq_handler_state  remove_slot
> 
> [root at xen1 ~]# cat /sys/bus/pci/devices/0000\:81\:02.6/device
> 0x154c
> [root at xen1 ~]# cat /sys/bus/pci/devices/0000\:81\:02.6/vendor
> 0x8086
> [root at xen1 ~]# cat /sys/bus/pci/devices/0000\:81\:02.6/subsystem_vendor
> 0x8086
> 
> [root at xen1 ~]# ls /sys/bus/pci/devices/0000\:81\:02.6/driver/module/drivers/
> pci:pciback  xen-backend:xen-pciback
> 
> [root at xen1 ~]# ls /sys/bus/pci/devices/0000\:81\:02.6/driver/0000:81:02.6
> broken_parity_status  consistent_dma_mask_bits  dma_mask_bits    enable        
> local_cpus  numa_node  reset      resource0_wc  subsystem  uevent
> class                 d3cold_allowed            driver           irq        
> modalias    physfn     resource   resource3     subsystem_device  vendor
> config                device                    driver_override local_cpulist  
> msi_bus     power      resource0  resource3_wc subsystem_vendor
> 
> 
>> If bound to xen-pciback, it vf should also appear in the output of 'xl 
>> pci-assignable-list'.
> 
> Before starting the VM, it's there:
> 
> [root at xengfs1f ~]# xl pci-assignable-list | grep "0000:81:02.6"
> 0000:81:02.6

Ah, xen-pciback is already bound to the device, so no need for managed='yes' in 
your device config. The 'managed' attribute tells libvirt whether or not to 
manage attaching/detaching drivers to/from the device. If you've done that 
elsewhere, set managed='no', or simply drop the attribute since 'no' is the 
default. It's explained in the 'pci' attribute of the hostdev element

https://libvirt.org/formatdomain.html#usb-pci-scsi-devices

But I wouldn't expect that minor config optimization to fix your issue. I tried 
starting a VM with managed='yes' and the device already assigned to xen-pciback, 
but it still worked for me. The sriov vf was visible and usable in the VM.

Are there any errors from xen-pciback in dom0, or any hints in the output of 'xl 
dmesg'? Also, it might be worth comparing the relevant nodes in the output of 
xenstore-ls between VM started with xl and libvirt.

Regards,
Jim