[libvirt] [RFC PATCH] support multifunction PCI device

Fri May 20 09:22:31 UTC 2011

On Fri, May 13, 2011 at 11:32:43AM +0900, KAMEZAWA Hiroyuki wrote:
> On Wed, 11 May 2011 08:55:56 +0100
> "Daniel P. Berrange" <berrange at redhat.com> wrote:
> 
> > On Wed, May 11, 2011 at 10:30:43AM +0800, Wen Congyang wrote:
> 
> > > 
> > > 3. After hot plug multi function PCI devices:
> > > # lspci
> > > 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
> > > 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
> > > 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
> > > 00:01.2 USB Controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
> > > 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
> > > 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
> > > 00:03.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 20)
> > > 00:04.0 RAM memory: Red Hat, Inc Virtio memory balloon
> > > 00:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> > > 00:06.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> > > 00:06.1 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> > > 00:06.2 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> > > 00:06.3 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> > > 00:06.4 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> > > 00:06.5 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> > > 00:06.6 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> > > 00:06.7 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> > > 
> > > 
> > > > 
> > > >>
> > > >> 2. support to hot unplug multi function PCI device
> > > >>    hot unpluging multi function PCI device meas that all the other
> > > >>    devices on the same slot will be hot unpluged. So we should do
> > > >>    some cleanup and remove the other devices too. If the other
> > > >>    device on the same slot does not support hot unpluged, or it is a
> > > >>    a controller and some other devices still use this controller,
> > > >>    I think we should refuse to hot unplug this mutlti function PCI
> > > >>    device.
> > > > 
> > > > IMHO these kind of restrictions will make life really unpleasant
> > > > for applications and are a reason we should *not* support the
> > > > multifunction code. Instead we should focus on one of the other
> > > > 3 options I mention above.
> > > 
> 
> Hmm, how about adding <unpluggable> attribute to <device> definitions ?
> IIUC, 
>   1. There are some unpluggable default devices.

This is a little complex. A PCI card can have a boolean unpluggable
attribute. A libvirt device can have a tri-state though, unpluggable,
not unpluggable, or unpluggable only if all these other functions
are unplugged at the same time. May be the third case is actually just
the same as the first case here and we should ignore it, and just have
a <unpluggable> wrt to the PCI device as a whole.

>   2. There are devices which never should be removed by mistake, as rootfs.

What devices are considered to be in list 2, can only be determined by the
guest OS, so I don't think we can represent that in the libvirt XML. It is
also a little bit more fuzzy that a boolean unpluggable. eg from the guest
OS POV, no device is unpluggable if it is in use. A block device becomes
unpluggable, once the guest filesystem is unmounted. A network card becomes
unpluggable, once the network interface is taken offline.

It could be desirable to expose this information to management apps, but
this won't be possible until QEMU gets some kind of guest agent that can
report this info from the guest OS. I think it needs to be reported
separately from whether the physical PCI card is unpluggable or not.

> When I've google'd "how to handle 100+ nics with KVM", a qemu community guy
> answers "please use multifunction devices" but I disappointed to know
> libvirt doesn't support it, now. 

Ok, I think it is reasonable to support multifunction devices in libvirt,
but only if the mgmt app explicitly configures them via the XML.

ie, the default behaviour where libvirt auto-assigns PCI addresses should
always be to assign slots. And when we get support for many PCI domains
or bridges, we'll assign slots from extra domains/bridges too by default.
The mgmt app could explicitly override this and configure multifunction
by providing an <address> element for devices in the XML, with the
function number set to non-zero.

> And, IIUC, if multifunction device is supporetd by libvirt, it can tie up default
> 'unpluggable' devices (as serial, IDE, etc) into a slot and we'll have 3? more
> empty slot at boot..

The IDE controller is a function on the PIIX device, as is the default USB
controller. The ISA bridge, behind which the serial/parallel ports live
is also a funtion on the PIIX device. So we are in fact already using a
multifunction device in this case, so there's no slots we can free up
wrt serial/IDE devices.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|