[libvirt] [RFC] On present using dummy hostdev usb device

[Roman Kagan]

On Fri, Aug 30, 2019 at 10:09:06AM +0100, Daniel P. Berrangé wrote:
> On Fri, Aug 30, 2019 at 08:44:03AM +0000, Nikolay Shirokovskiy wrote:
> > Hi, all!
> > 
> >   We use an interesting approach when starting/migrating/etc domain with usb
> > hostdev with startupPolicy=optional. We add qemu usb-host device with
> > missing hostaddr/hostbus parameters (dummy device). I guess there are
> > 2 reasons why we do it. First without dummy device migration will fail as
> > described in [1]. Second is an interesting property of dummy device that
> > qemu starts to monitor for attaching of usb devices and binds the first
> > attached to node to the dummy device. So one can start a domain with
> > missing hostdev and attach it later or migrate a domain then detach
> > hostdev on source and attach it on destination. But as qemu binds the
> > first attached device this is not reliable, to say the least. And after
> > all this does not work if domain uses distinct mount namespace which
> > is default.
> 
> Even without mount namespaces, it should fail as QEMU is running  non-root
> and libvirt won't have granted access to any host USB devices in /dev, and
> also SELinux policy will forbid this.

Right, but the case with mount namespaces is particularly problematic:
if the device open fails due to missing device node, libusb removes the
device from its internal device list.  This results in the following
scenario:

- libvirt adds a dummy usb-host device to QEMU in place of a missing
  device

- QEMU (via libusb) installs a watch for udev add events

- the physical device is plugged into the host

- QEMU detects the addition of the device and, since the dummy device
  matches everything, tries to open it

- by this time libvirt may have not created a device node in QEMU's
  mount namespace, so the open fails due to missing device node, and
  libusb removes the device from its internal list

- libvirt removes the dummy usb-host device and adds the actual usb-host
  device

- QEMU fails to open it because it's no longer seen by libusb

IOW a usb-host device with missing=true can't (reliably, because
sometimes libvirt is quick enough to create the device node before QEMU
gives up opening it) turn into a working one without QEMU restart.

> >   So I question does it make sense to use dummy device at all? In case of
> > migration/resume from suspend/revert to snapshot we can either fix qemu to
> > ignore incoming missing hostdev data or add dummy device temporarily. The
> > latter solution is worse as it brings dummy device behaviour even for a short
> > period of time. However having a temporary dummy device is neccessary step
> > towards the time when all supported versions of qemu do the mentioned ignoring.
> > As to handling attaching of missing hostdev device to node it should be done in
> > libvirt which can do necessary mount namespace actions. (Actually I developing
> > such patches right now but some peculiarities of dummy device bring me here).
> 
> The problems around host USB device passthrough are conceptually similar
> to the problems of hots PCI device passthrough.
> 
> In both cases we cannot assume the device present on the source device
> exists on the target device in the same way.
> 
> In both cases, even if the device does exist on the target, we cannot
> serialize the state of the host device across the migration.

Right.

> For PCI devices we simply refuse to initiate the migration if any host
> PCI devices are attached. The mgmt app has to hot-unplug all devices
> before migration, and hot-plug new devices after migration if desired.
> 
> I'm inclined to suggest that same approach of hotunplug + hotplug either
> side of migration is the only viable option for host USB devices too.
> 
> As such any mgmt app could do this dance today without any changes in
> libvirt.

Are you trying to say that the mgmt app should just refrain from
creating usb-host devices with missing=true?

> If we turned host USB devices into a migration blocker though, that
> could be considered a significant change of behaviour for mgmt apps,
> even though this dummy USB device is effectively useless due to our
> security policies.

I'm afraid the issue is a bit more severe: the dummy device isn't just
useless, it stands in the way of the real device later on.

Thanks,
Roman.