[libvirt] [RFC PATCH] support multifunction PCI device

[KAMEZAWA Hiroyuki]

On Fri, 20 May 2011 10:22:31 +0100
"Daniel P. Berrange" <berrange at redhat.com> wrote:

> On Fri, May 13, 2011 at 11:32:43AM +0900, KAMEZAWA Hiroyuki wrote:
> > On Wed, 11 May 2011 08:55:56 +0100
> > "Daniel P. Berrange" <berrange at redhat.com> wrote:
> > 
> > > On Wed, May 11, 2011 at 10:30:43AM +0800, Wen Congyang wrote:
> > 
> > > > 
> > > > 3. After hot plug multi function PCI devices:
> > > > # lspci
> > > > 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
> > > > 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
> > > > 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
> > > > 00:01.2 USB Controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
> > > > 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
> > > > 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
> > > > 00:03.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 20)
> > > > 00:04.0 RAM memory: Red Hat, Inc Virtio memory balloon
> > > > 00:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> > > > 00:06.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> > > > 00:06.1 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> > > > 00:06.2 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> > > > 00:06.3 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> > > > 00:06.4 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> > > > 00:06.5 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> > > > 00:06.6 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> > > > 00:06.7 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> > > > 
> > > > 
> > > > > 
> > > > >>
> > > > >> 2. support to hot unplug multi function PCI device
> > > > >>    hot unpluging multi function PCI device meas that all the other
> > > > >>    devices on the same slot will be hot unpluged. So we should do
> > > > >>    some cleanup and remove the other devices too. If the other
> > > > >>    device on the same slot does not support hot unpluged, or it is a
> > > > >>    a controller and some other devices still use this controller,
> > > > >>    I think we should refuse to hot unplug this mutlti function PCI
> > > > >>    device.
> > > > > 
> > > > > IMHO these kind of restrictions will make life really unpleasant
> > > > > for applications and are a reason we should *not* support the
> > > > > multifunction code. Instead we should focus on one of the other
> > > > > 3 options I mention above.
> > > > 
> > 
> > Hmm, how about adding <unpluggable> attribute to <device> definitions ?
> > IIUC, 
> >   1. There are some unpluggable default devices.
> 
> This is a little complex. A PCI card can have a boolean unpluggable
> attribute. A libvirt device can have a tri-state though, unpluggable,
> not unpluggable, or unpluggable only if all these other functions
> are unplugged at the same time. May be the third case is actually just
> the same as the first case here and we should ignore it, and just have
> a <unpluggable> wrt to the PCI device as a whole.
> 
> >   2. There are devices which never should be removed by mistake, as rootfs.
> 
> What devices are considered to be in list 2, can only be determined by the
> guest OS, so I don't think we can represent that in the libvirt XML. It is
> also a little bit more fuzzy that a boolean unpluggable. eg from the guest
> OS POV, no device is unpluggable if it is in use. A block device becomes
> unpluggable, once the guest filesystem is unmounted. A network card becomes
> unpluggable, once the network interface is taken offline.
> 
> It could be desirable to expose this information to management apps, but
> this won't be possible until QEMU gets some kind of guest agent that can
> report this info from the guest OS. I think it needs to be reported
> separately from whether the physical PCI card is unpluggable or not.
> 

Sure.

> > When I've google'd "how to handle 100+ nics with KVM", a qemu community guy
> > answers "please use multifunction devices" but I disappointed to know
> > libvirt doesn't support it, now. 
> 
> Ok, I think it is reasonable to support multifunction devices in libvirt,
> but only if the mgmt app explicitly configures them via the XML.
> 
> ie, the default behaviour where libvirt auto-assigns PCI addresses should
> always be to assign slots. And when we get support for many PCI domains
> or bridges, we'll assign slots from extra domains/bridges too by default.
> The mgmt app could explicitly override this and configure multifunction
> by providing an <address> element for devices in the XML, with the
> function number set to non-zero.
> 

Ok, thank you. Our team will discuss and implement it, try again.

Thanks,
-Kame