[libvirt] QEMU/KVM support for non zero PCI domain

[Charles Peri]

On Apr 17, 2013, at 12:29 PM, Laine Stump <laine at laine.org> wrote:

> On 04/16/2013 11:23 AM, Charles Peri wrote:
> 
>> 
>> Hi,
>>  I am trying to run Virt-Manager, using KVM on a SGIUV100 system with RHEL 6.1. I am trying to enable PCI pass-thorugh for the Mellanox, but the PCI domain for the device is 0001. 
> 
>> 
>>> 4) Does libvirt actually get to the point of running qemu or kvm command?
>> 
>>> 5) If the answer to (4) is "yes", can you send the qemu command that is
>>> generated (it will be at the end of /var/log/libvirt/qemu/${guestname}.log)
>> 
>> 
>> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name UV2-VM1-test1 -S -M rhel6.4.0 -enable-kvm -m 8192 -smp 4,sockets=4,cores=1,threads=1 -uuid e2428c26-f06a-b03c-c8d5-c528f2e85596 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/UV2-VM1-test1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/UV2-VM1-test1.img,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=23,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:51:5f:5f,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device pci-assign,host=02:00.4,id=hostdev0,configfd=24,bus=pci.0,addr=0x6 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> 
> Okay. The important part here is this:
> 
>   -pci-assign,host=02:00.4,id=hostdev0,configfd=24,bus=pci.0,addr=0x6
> 
> 
> This shows that libvirt is requesting to assign the device in domain 0,
> bus 2, slot 0, function 4,
> 
>>> 0001:01:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
>>> 0001:02:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)
>>> 
>>> When I try to add the device, I get an error stating that "Can't assign device inside non- zero PCI segment, as this KVM module doesn't support 
> 
> I don't see this error in the above log, nor how it could have been
> generated. There is nothing in the pci-assign device about a non-zero
> domain. So I'm not sure how you arrived at that message.
> 
> 

Laine,
    I was able to earlier install a later version of the libvirt on another system with RHEL 6.4, in which I got the error that "Can't assign device inside non-zero PCI segment, as this KVM module doesn't support" - I was not able to get back to that system, so could not get you the extant same configuration. But if I am able to reproduce it, I will send the config and message again.

> At any rate, I've done some investigation and learned the following:
> 
> 1) The qemu-0.12.x that is in RHEL6 only supports assigning devices in
> domain 0.
> 
> 2) "At some point" since then, the "host" option has been expanded to
> allow including a domain id, e.g. "-device
> pci-assign,host=0001:02:00.0,...", but that support was not backported
> to RHEL.
> 
> 3) Even if you were to have a version of qemu that supported assigning a
> device in a non-zero domain, libvirt ignores the domain in the device
> address, and only tells qemu about bus:slot.function.
> 
> 
> So, libvirt should add support for assigning pci devices in non-0
> domains (and it's reasonably simple to do so), but it will only work on
> newer versions of qemu than what you have, so you would have to build
> your own (unsupported) upstream versions of both qemu and libvirt.
> 
> 


Do we have the support built into version 1.4.0 of qemu? Also, do I have to make the changes to the libvirt source code or is the support already available in some version, which I could use?
Let me know, which versions or which source code will be a get place to start, if I want to add the support myself.

Thanks,
    Charles