[vfio-users] dev_WARN for hotplugging to live VFIO group

[Shawn Anastasio]

Hello,

While developing the userspace VFIO components of libkvmchan[1], I've
run into a dev_WARN in VFIO when hotplugging devices on the same pci
host bridge as other VFIO-managed devices:

[  111.220260][ T6281] pci 0001:00:01.0: Device added to live group 1!
[  111.220390][ T6281] WARNING: CPU: 51 PID: 6281 at 
drivers/vfio/vfio.c:709 vfio_iommu_group_notifier+0x250/0x360 [vfio]

In spite of this warning, everything seems to be working fine. The
daemon that manages these PCI devices and owns the VFIO group fd
is able to connect to the new devices and use them just fine.

My question is: what exactly is the reason this warning was added?
Is there a subtle issue that could be triggered by hotplugging to
live groups, and if so, what needs to be done to resolve this?
I'd be happy to try to resolve any issues and submit patches if necessary.

For reference, this is all on a pseries/ppc64 guest on a POWER9 host,
with VFIO operating in the VFIO_SPAPR_TCE_IOMMU mode. In this config,
all PCI devices on a bus exist in the same IOMMU group (in this case #1).

Thanks in advance,
Shawn

[1] https://github.com/shawnanastasio/libkvmchan