[libvirt] new libvirt "pci" controller type and pcie/q35 (was Re: [PATCH 4/7] add pci-bridge controller type)

[Daniel P. Berrange]

On Mon, Apr 08, 2013 at 04:26:19PM -0600, Alex Williamson wrote:
> On Mon, 2013-04-08 at 12:37 -0400, Laine Stump wrote:
> > On 04/05/2013 03:26 PM, Alex Williamson wrote:
> > >> Sure. I just mean "pci bus 0" (hmm, but actually this does point out a
> > >> problem with my logic - the same namespace (well, "numbering space") is
> > >> used for both pcie and pci buses, so on a q35 system, bus=0 is already
> > >> taken by pcie.0; that means that the first pci bus would need to use a
> > >> different bus number anyway, so it wouldn't be so easy to switch an
> > >> existing domain from pc to q35 - every PCI device would need to have its
> > >> bus number modified. I suppose that's reasonable to expect, though.
> > > I would think you'd want to differentiate PCI from PCIe anyway.  PCI is
> > > a bus and you have 32 slots per bus to fill.  PCIe is a point-to-point
> > > link and you really only have slot 0 available.  Perhaps that puts them
> > > in different number spaces already.
> > 
> > Are you saying that it's okay to have a bus=0 for pci and a different
> > bus=0 for pcie?
> 
> In bus=<identifier> the identifier needs to be unique, but it's not a
> bus #, it's just an identifier.

This is mixing up QEMU syntax with libvirt syntax - we're not requiring
them to be the same. Libvirt operates solely in terms of bus numbers,
and will turn those into QEMU bus identifiers at the time it generates
the CLI args.

Indeed we explicitly do *not* want to follow QEMU's syntax in the libvirt
XML, since this is no good for non-QEMU hypervisors.

> > I was hoping that what is used in libvirt's config could mirror as
> > closely as possible the numbering that you see in the output of lspci on
> > the guest, but it sounds like that numbering is something done at the
> > whim of the guest, with no basis in (standard) reality, is that right?
> 
> Correct, the the BIOS determines the initial bus numbers and it can do
> it however it wants.  Most guests won't renumber buses, but they can if
> they want.  It's a lost cause to expect any correlation between the
> libvirt bus identifier and the actual bus number.

I don't think libvirt needs to require that its XML bus numbers match
the guest OS' view of bus numbers. If they do match, great, otherwise
so be it.

> > I think we're starting to get closer to the concrete problem that's
> > bothering me. As I understand it (and again - "what I understand" has
> > repeatedly been shown to be incorrect in this thread :-):
> > 
> > * Ihere are multiple different types of devices that provide a bus with
> > 1 or more "slots" that PCI devices (e.g., the virtio-net-pci device, the
> > e1000 network device, etc) can be plugged into.
> > 
> > * In the config for those devices, there is a required (auto-generated
> > if not explicitly provided) <address> element that indicates what
> > controller that device is plugged into e.g.:
> > 
> >     <interface type='direct'>
> >       ...
> >       <address type='pci' domain='0' bus='0' slot='3' function='0'/>
> >       ...
> >     </interface>
> 
> The above is not really how QEMU works.  QEMU PCI devices take an addr=
> parameter that specifies the "slot.function".  The bus= option is not
> numeric.  That's the identifier value.  So if you create a bus with:
> 
>  -device i82801b11-bridge,id=dmi-to-pci-bridge,addr=1e.0
> 
> Then to put a device on that bus you'd do:
> 
>  -device e1000,id=e1000-net-0,bus=dmi-to-pci-bridge,addr=0.0

As above, that's no problem. Libvirt is perfectly capable of translating
from <address type='pci' domain='0' bus='5' slot='3' function='0'/> to
to whatever bus=AAA and addr=BBB args that are appropriate for QEMU.

> 
> We don't have a way to generate new domains yet, but I imagine it would
> require a PCI host bridge device and be a parameter to that.  For
> instance:
> 
>  - device pci-host-bridge,domain=1,id=domain1-pcie.0
> 
> Which would create a new "pci-root".  You would then specify a device in
> the domain using the same bus= notation.  An e1000 on bus=pcie.0 is in
> domain0, but a nic on bus=domain1-pcie.0 is in domain1.  This is all
> just speculation though.
> 
> > * domain is always hardcoded to 0, and in the past bus was also always
> > hardcoded to 0 because until now there has only been a single place
> > where PCI devices could be connected - the builtin pci.0 bus, which is a
> > part of the basic "pc" (and some others) virtual machine and provides 32
> > slots.
> 
> Again, bus= is an identifier and I would guess that it will implicitly
> specify the domain when we get that far.  Libvirt specifying a numerical
> domain:bus:slot.fn and expecting the device to appear there to the guest
> is a flawed concept.

No, libvirt won't be using  addr=domain:bus:slot.fun. it will use
whatever  bus=AAA and addr=slot.fun  pairing is required for QEMU.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|