[libvirt] pci bus slot in libvirt

[Fong Vang]

On Tue, Jun 26, 2012 at 2:00 PM, Fong Vang <sudoyang at gmail.com> wrote:
> Very informational.  Thank you very much.
>
> On Tue, Jun 26, 2012 at 1:51 PM, Daniel P. Berrange <berrange at redhat.com> wrote:
>> On Tue, Jun 26, 2012 at 01:09:24PM -0700, Fong Vang wrote:
>>> What's the best way to configure multiple disks in libvirt?  How many
>>> slots can you have?  This is from the xml dump:
>>>
>>> In some of my configurations, I have this:
>>>
>>>     <disk type='block' device='disk'>
>>>       <driver name='qemu' type='raw' cache='none' io='native'/>
>>>       <source dev='/dev/sdb1'/>
>>>       <target dev='vdb' bus='virtio'/>
>>>       <alias name='virtio-disk1'/>
>>>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>>> function='0x0'/>
>>>     </disk>
>>
>> With the traditional VirtIO-Block device, the disk controller only
>> accepts a single drive, so each <disk> you add consumes 1 PCI slot.
>> You have a max of 31 PCI slots, of which at most 28 are usable for
>> (PIIX consumes 1, VGA device consumes 1 and Balloon driver consumes
>> 1). Those 28 must be split across everything else, so assuming no
>> NICs or other devices, you have create 28 virtio-block disks.

One last question on this.  Are there some slots that are reserved?
In my setup, there could only be multiple disks and NICs.  I'm trying
to figure out the best way to structure the different components with
respect to slot numbering.

>>
>>> while others are created as such:
>>>
>>>   <devices>
>>>     <emulator>/usr/bin/qemu-kvm</emulator>
>>>     <disk type='file' device='disk'>
>>>       <driver name='qemu' type='qcow2'/>
>>>       <source file='/data2/VMs/libvirt/images/win2k8-1.qcow2'/>
>>>       <target dev='hda' bus='ide'/>
>>>       <alias name='ide0-0-0'/>
>>>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>>     </disk>
>>
>> This is an IDE disk, where the IDE controller consumes the PCI slot,
>> and can support upto 4 drives (hda, hdb, hdc and hdd). So for 1 PCI
>> slot, you get 4 drives, which isn't great.
>>
>>> Which way is better?  The top one defines the bus and slot while the
>>> bottom defines the controller, bus, target, and unit.
>>
>> You won't typically choose between IDE and Virtio-Block based on the
>> number of disks. You'll simply always want virtio-block for performance
>> reasons. So there's no real choice of config here.
>>
>> With the latest libvirt, guest kernel and QEMU, there is a new device
>> type known as Virtio-SCSI. In this the Virtio-SCSI controller consumes
>> 1 PCI device and can handle many many many disks. This is the best
>> option for the future for people who need lots of disks.
>>
>>> BTW, how many slots can we define per bus?
>>
>> With virtio-block, currently 28.
>>
>> If you are willing todo without hotplug support, you can turn on
>> multi-function and increase the number of possible virtio-block
>> devices x7 ie  ~196
>>
>> In the future we'll also have support for PCI bridges, and multiple
>> PCI root complexes, which will increase max # of PCI slots even more.
>>
>> But virtio-SCSI really is the best way in the future.
>>
>>
>> Daniel
>> --
>> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
>> |: http://libvirt.org              -o-             http://virt-manager.org :|
>> |: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
>> |: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|