<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word"> [AMD Official Use Only] <div class="WordSection1"> Hi,<o:p></o:p> I am using Fedora 33, with the following KVM, qemu and libvirt versions:<o:p></o:p> QEMU 5.1.0<o:p></o:p> libvirt 6.6.0<o:p></o:p> KVM 5.14.18<o:p></o:p> <o:p> </o:p> We have done pass-through of a PCIe NVMe device to the guest running on FC33<o:p></o:p> using either virt-manager or virsh and then we do the hot-unplug of the device<o:p></o:p> while it is attached to the guest.<o:p></o:p> <o:p> </o:p> The device is no longer seen on the guest hardware device list on virt-manager<o:p></o:p> and then we hotplug the device again and we are able to use it on the Host,<o:p></o:p> but when we try to re-attach it to the guest, we get the following error message:<o:p></o:p> <o:p> </o:p> Requested operation is not valid, PCI device 0000:c4::00.0 is in use by driver QEMU,<o:p></o:p> Domain fedora 33. <o:p></o:p> <o:p> </o:p> So somehow libvirt still thinks the hot-unplugged device is attached.<o:p></o:p> <o:p></o:p> Tracing the flow of hot un-plug event from guest to host :<o:p></o:p> <o:p> </o:p> ->Guest pcie hotplug support detected the NVMe driver unplug (from guest kernel logs): <o:p></o:p> <o:p> </o:p> pciehp: Slot (0-6): Attention button pressed<o:p></o:p> <o:p> </o:p> pciehp: Slot (0-6): Powering off due to to button press.<o:p></o:p> <o:p> </o:p> -> Also looks like the guest notified Host/KVM (from host kernel logs):<o:p></o:p> <o:p> </o:p> pcieport: 0000:c4:0000.0: pciehp: Slot (208): Card not present<o:p></o:p> <o:p> </o:p> -> Correspondingly, vfio-pci module notified Qemu :<o:p></o:p> <o:p> </o:p> vfio-pci: 0000:c4:0000.0: Relaying device request to user (#0)<o:p></o:p> <o:p> </o:p> -> Then the un-plugged device reset is done.<o:p></o:p> <o:p> </o:p> vfio-pci: vfio_bar_restore: reset recovery - restoring BARs<o:p></o:p> <o:p> </o:p> pci 0000:c4:00.0: Removing from iommu group 105.<o:p></o:p> <o:p> </o:p> -> Next tried to verify if libvirt detected the DELETED_DRIVE event from qemu.<o:p></o:p> <o:p> </o:p> Running SystemTap script to capture events between qemu and libvirt :<o:p></o:p> <o:p> </o:p> stap examples/systemtap/qemu-monitor.stp<o:p></o:p> <o:p> </o:p> When the NVMe drive is attached to VM the following log output is seen from SystemTap:<o:p></o:p> <o:p> </o:p> execute "device-add", driver: "vfio-pci", host: "0000:c4:00.0", id: "hostdev0", bus: "pci.7", addr: "0".<o:p></o:p> <o:p> </o:p> When we hot-unplug the NVMe drive, the following log output is seen from SystemTap:<o:p></o:p> <o:p> </o:p> event: DEVICE_DELETED, device: "hostdev0", path: "/machine/peripheral/hostdev0".<o:p></o:p> <o:p> </o:p> So it looks like that qemu sent the "DEVICE_DELTED" event to libvirt, but libvirt has still not removed the attached<o:p></o:p> device from its bookeeping list.<o:p></o:p> <o:p> </o:p> I understand there is already a thread from 20202, discussing a similar issue :<o:p></o:p> <a href="https://www.spinics.net/linux/fedora/libvirt-users/msg12590.html">https://www.spinics.net/linux/fedora/libvirt-users/msg12590.html</a><o:p></o:p> <o:p> </o:p> But I am not sure if there is any fix/support added for this recently. <o:p></o:p> <o:p> </o:p> Looking for any feedback related to above and PCI device passthrough and hotplug support. <o:p></o:p> <o:p> </o:p> Thanks,<o:p></o:p> Ashish<o:p></o:p> </div> </body> </html>