[libvirt] QEMU/KVM support for non zero PCI domain

[Laine Stump]

On 04/18/2013 03:13 PM, Charles Peri wrote:
> On Apr 17, 2013, at 12:29 PM, Laine Stump <laine at laine.org> wrote:
>
>> On 04/16/2013 11:23 AM, Charles Peri wrote:
>>
>>> Hi,
>>>  I am trying to run Virt-Manager, using KVM on a SGIUV100 system with RHEL 6.1. I am trying to enable PCI pass-thorugh for the Mellanox, but the PCI domain for the device is 0001. 
>>>> 4) Does libvirt actually get to the point of running qemu or kvm command?
>>>> 5) If the answer to (4) is "yes", can you send the qemu command that is
>>>> generated (it will be at the end of /var/log/libvirt/qemu/${guestname}.log)
>>>
>>> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name UV2-VM1-test1 -S -M rhel6.4.0 -enable-kvm -m 8192 -smp 4,sockets=4,cores=1,threads=1 -uuid e2428c26-f06a-b03c-c8d5-c528f2e85596 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/UV2-VM1-test1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/UV2-VM1-test1.img,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=23,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:51:5f:5f,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device pci-assign,host=02:00.4,id=hostdev0,configfd=24,bus=pci.0,addr=0x6 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>> Okay. The important part here is this:
>>
>>   -pci-assign,host=02:00.4,id=hostdev0,configfd=24,bus=pci.0,addr=0x6
>>
>>
>> This shows that libvirt is requesting to assign the device in domain 0,
>> bus 2, slot 0, function 4,
>>
>>>> 0001:01:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
>>>> 0001:02:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)
>>>>
>>>> When I try to add the device, I get an error stating that "Can't assign device inside non- zero PCI segment, as this KVM module doesn't support 
>> I don't see this error in the above log, nor how it could have been
>> generated. There is nothing in the pci-assign device about a non-zero
>> domain. So I'm not sure how you arrived at that message.
>>
>>
> Laine,
>     I was able to earlier install a later version of the libvirt on another system with RHEL 6.4, in which I got the error that "Can't assign device inside non-zero PCI segment, as this KVM module doesn't support" - I was not able to get back to that system, so could not get you the extant same configuration. But if I am able to reproduce it, I will send the config and message again.
>
>> At any rate, I've done some investigation and learned the following:
>>
>> 1) The qemu-0.12.x that is in RHEL6 only supports assigning devices in
>> domain 0.
>>
>> 2) "At some point" since then, the "host" option has been expanded to
>> allow including a domain id, e.g. "-device
>> pci-assign,host=0001:02:00.0,...", but that support was not backported
>> to RHEL.
>>
>> 3) Even if you were to have a version of qemu that supported assigning a
>> device in a non-zero domain, libvirt ignores the domain in the device
>> address, and only tells qemu about bus:slot.function.
>>
>>
>> So, libvirt should add support for assigning pci devices in non-0
>> domains (and it's reasonably simple to do so), but it will only work on
>> newer versions of qemu than what you have, so you would have to build
>> your own (unsupported) upstream versions of both qemu and libvirt.
>>
>>
>
> Do we have the support built into version 1.4.0 of qemu? 


Yes, I believe so. (purely based on what I was told by a qemu developer,
not personal experience).


> Also, do I have to make the changes to the libvirt source code or is the support already available in some version, which I could use?
> Let me know, which versions or which source code will be a get place to start, if I want to add the support myself.

There is no support for it currently; it's one of those things that
until now nobody has needed, so there was no push to do it.

It shouldn't be too hard to add - we just need a capability flag saying
that domains are supported in host-side PCI addresses, then add a %.4x
and a def->source.subsys.u.pci.addr.domain to a printf (if the
capability bit is set).

Any new development should be done first on the head of git master. You
can learn how to check that out from libvirt.org.