[Date Prev][Date Next] [Thread Prev][Thread Next]
Re: [libvirt] [RFC PATCH] hostdev: add support for "managed='detach'"
- From: Laine Stump <laine laine org>
- To: libvir-list redhat com
- Cc: kraxel redhat com, mpolednik redhat com
- Subject: Re: [libvirt] [RFC PATCH] hostdev: add support for "managed='detach'"
- Date: Wed, 23 Mar 2016 14:06:13 -0400
On 03/22/2016 07:03 PM, John Ferlan wrote:
On 03/14/2016 03:41 PM, Laine Stump wrote:
Suggested by Alex Williamson.
If you plan to assign a GPU to a virtual machine, but that GPU happens
to be the host system console, you likely want it to start out using
the host driver (so that boot messages/etc will be displayed), then
later have the host driver replaced with vfio-pci for assignment to
the virtual machine.
However, in at least some cases (e.g. Intel i915) once the device has
been detached from the host driver and attached to vfio-pci, attempts
to reattach to the host driver only lead to "grief" (ask Alex for
details). This means that simply using "managed='yes'" in libvirt
And if you set "managed='no'" in libvirt then either you have to
manually run virsh nodedev-detach prior to the first start of the
guest, or you have to have a management application intelligent enough
to know that it should detach from the host driver, but never reattach
This patch makes it simple/automatic to deal with such a case - it
adds a third "managed" mode for assigned PCI devices, called
"detach". It will detach ("unbind" in driver parlance) the device from
the host driver prior to assigning it to the guest, but when the guest
is finished with the device, will leave it bound to vfio-pci. This
allows re-using the device for another guest, without requiring
initial out-of-band intervention to unbind the host driver.
I'm sending this with the "RFC" tag because I'm concerned it might be
considered "feature creep" by some (although I think it makes at least
as much sense as "managed='yes'") and also because, even once (if) it
is ACKed, I wouldn't want to push it until abologna is finished
hacking around with the driver bind/unbind code - he has enough grief
to deal with without me causing a bunch of merge conflicts :-)
Rather the burying this in one of the 3 other conversations that have
been taking place - I'll respond at the top level. I have been following
the conversation, but not at any great depth... Going to leave the
should we have "managed='detach'" conversation alone (at least for now).
Not that I want to necessarily dip my toes into these shark infested
waters; however, one thing keeps gnawing at my fingers from hanging onto
the don't get involved in this one ledge ;-)... That one thing is the
problem to me is less libvirt's ability to manage whether the devices
are or aren't detached from the host, but rather that lower layers (e.g.
kernel) aren't happy over the frequency of such requests.
That is definitely one of the reasons we need to do it less. My
conclusion from all this discussion has been
1) lower layers are woefully lacking in ways available to specify what
driver to use for a specific device at boot time, making it essentially
impossible, for example, to have devices that will only be used for vfio
assignment to a guest be bound to the vfio-pci driver straight from boot
time. Instead they must be bound to pci-stub at boot, then later
re-bound to vfio-pci by "something, who knows, shuddup and do it
yerself!". And even that only works if you want *all* devices with the
same vendor/device id to be treated in the exact same manner. (and
blacklisting a particular driver is just hopelessly useless.
2) (1) isn't necessarily because the people in the lower layers are
being unhelpful jerks about it, it's because the way that things are
organized in the kernel and its boot process make it really difficult to
any system isn't fun, but it's a lot easier to tell someone else to stop
doing it since it hurts when you do that. Tough to find a happy medium
between force user to detach rather than let libvirt manage and letting
libvirt be the culprit causing angst for the lower layers.
So, would it work to have some intermediary handle this thrashing by
creating some sort of "blob" that will accept responsibility for
reattaching the device to the host "at some point in time" as long as no
one else has requested to use it?
Why not add an attribute (e.g. delay='n') that is only valid when
managed='yes' to the device which means, rather than immediately
reattaching this to the host when the guest is destroy, libvirt will
delay the reattach by 'n' seconds. That way someone that knows they're
going to have a device used by multiple guests that could be thrashing
heavily in the detach -> reattach -> detach -> reattach -> etc loop
would be able to make use of an optimization of sorts that just places
the device back in the inactive list (as if it were detached manually),
then starts a thread that will reawaken when a timer fires to handle the
reattach. The thread would be destroyed in the event that something
codes along and uses (e.g. places back into the active list).
This sounds dangerous for two reasons: 1) someone may expect that
they'll be able to use the device on the host if it's not in use by a
guest, and this will be true only at undeterminable (from the outside)
times. 2) Rather than causing a failure immediately when the guest
terminates (or deletes the device), the failure will occur at some later
time, which could serve to obscure the source of the failure.
It's an interesting idea from the "this would be cool to code", but I
don't really see a practical use for it.
The caveats that come quickly to mind in using this is that devices that
were being managed could be left on the inactive list if the daemon dies
or is restarted, but I think it is detectable at restart so it may not
be totally bad. Also, failure to reattach is left in some thread which
has no one to message to other than libvirtd log files. Both would have
to be noted with any description of this.
Not all devices will want this delay logic and I think it's been pointed
out that there is a "known list" of them. In the long run it allows
some control by a user to decide how much rope they'd like to have to
Not sure if Gerd, Alex, and Martin are regular libvir-list readers, so I
did CC them just in case so that it's easier for them to respond if they
so desire since they were at part of the discussions in this thread.
libvir-list mailing list
libvir-list redhat com
[Date Prev][Date Next] [Thread Prev][Thread Next]