[libvirt] Proposal PCI/PCIe device placement on PAPR guests

[David Gibson]

On Thu, Jan 12, 2017 at 12:53:28PM -0500, Laine Stump wrote:
> On 01/12/2017 11:35 AM, Michael Roth wrote:
> > Quoting Laine Stump (2017-01-12 08:52:10)
> > > On 01/12/2017 05:31 AM, Andrea Bolognani wrote:
> > > > On Mon, 2017-01-09 at 10:46 +1100, David Gibson wrote:
> > > > > > >     * To allow for hotplugged devices, libvirt should also add a number
> > > > > > >       of additional, empty vPHBs (the PAPR spec allows for hotplug of
> > > > > > >       PHBs, but this is not yet implemented in qemu).
> > > > > > 
> > > > > > "A number" here will have to mean "one", same number of
> > > > > > empty PCIe Root Ports libvirt will add to a newly-defined
> > > > > > q35 guest.
> > > > > 
> > > > > Umm.. why?
> > > > 
> > > > Because some applications using libvirt would inevitably
> > > > start relying on the fact that such spare PHBs are
> > > > available, locking us into providing at least the same
> > > > number forever. In other words, increasing the amount at
> > > > a later time is always possible, but decreasing it isn't.
> > > > We did the same when we started automatically adding PCIe
> > > > Root Ports to q35 machines.
> > > > 
> > > > The rationale is that having a single spare hotpluggable
> > > > slot is extremely convenient for basic usage, eg. a simple
> > > > guest created by someone who's not necessarily very
> > > > familiar with virtualization; on the other hand, if you
> > > > are actually deploying in production you ought to conduct
> > > > proper capacity planning and figure out in advance how
> > > > many devices you're likely to need to hotplug throughout
> > > > the guest's life.
> > > 
> > > And of course the reason we don't want to add "too many" extra
> > > controllers by default is so that we don't end up with *all* guests
> > > burdened with extra hardware they don't need or want. The libguestfs
> > > appliance is one example of a libvirt consumer that definitely doesn't
> > > want extra baggage in its guests - guest startup time is very important
> > > to libguestfs, so any addition to the hardware list is looked upon with
> > > disappointment.
> > > 
> > > > 
> > > > Of course this all will be moot once we can hotplug PHBs :)
> > > 
> > > Will the guest OSes handle that properly? I remember being told that
> > 
> > I believe on pseries we *do* scan for devices on the PHB as part of
> > bringing the PHB online in the hotplug path. But I'm not sure that
> > matters (see below).
> > 
> > > Linux, for example, doesn't scan the new bus for devices when a new
> > > controller is added, making it pointless to hotplug a PCI controller (as
> > > usual, it could be that I'm remembering incorrectly...)
> > > 
> > 
> > Wouldn't that only be an issue if we hotplugged a PHB that already had
> > PCI devices on the bus?
> 
> 
> Yeah you're right, I'm probably remembering the wrong problem and wrong
> reason for the problem. I just remember there was *some* issue about
> hotplugging new PCI controllers. Possibly the internal representation of the
> bus hierarchy wasn't updated unless you forced a rescan of all the devices
> or something? My memory of it is vague, I just remember being told it wasn't
> just a case of the controller itself being initialized.
> 
> Alex or Marcel - since whatever it was I likely heard it from one of you (or
> imagined it in a dream), can you straighten me out?

Regardless, I'm pretty sure it won't be relevant for Power guests.
PHB hotplug has its own protocol in PAPR, and is used routinely for
Linux guests under PowerVM.

> 
> > That only seems possible if we had a way to
> > signal phb hotplug *after* we've hotplugged some PCI devices on the bus,
> > which means we'd need some interface to trigger hotplug  beyond the
> > standard 'device_add' calls, e.g.:
> > 
> >   device_add spapr-pci-host-bridge,hotplug-deferred=true,id=phb2,index=2
> >   device_add virtio-net-pci,bus=phb2.0,...,hotplug-deferred=true
> >   device_signal_hotplug phb2
> > 
> > That's actually akin to how it's normally done on pHyp (not only for PHB
> > hotplug, but for PCI hotplug in general, which is why this could be
> > reasonably expected to work on pseries guests), but it seems quite a bit
> > different from how we'd normally handle this on kvm, which I think would
> > be something more like:
> > 
> >   device_add spapr-pci-host-bridge,id=phb2,index=2
> >   <wait for hotplug completion event>
> >   device_add virtio-net-pci,bus=phb2.0,...
> > 
> > In which case it doesn't really matter if the guest scans the bus at
> > hotplug time or not. Is there some other scenario where this might
> > arise?
> > 
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20170113/69b77903/attachment-0001.sig>