[libvirt-users] PCI Passthrough of 2 identical devices
Thomas Jagoditsch
tja at tjasoft.com
Wed Mar 12 18:50:14 UTC 2014
hi laine,
thx for the fast answer.
i tried VFIO as it seems supported and got at least one step further.
starting the guest with one of the DVB cards is possible now, even when the other card is in the hosts pci config (not removed soft or hard).
adding the second card/device thru ...
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <driver name='vfio'/>
> <source>
> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
> </hostdev>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <driver name='vfio'/>
> <source>
> <address domain='0x0000' bus='0x03' slot='0x01' function='0x0'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
> </hostdev>
will render the guest unbootable:
--><-- /var/log/libvirt/qemu/tvBackend.log
2014-03-12 16:48:48.018+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name tvBackend -S -machine pc-i440fx-1.5,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid f647bb10-7f9a-f94c-33b9-3d99e8e753e0 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/tvBackend.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/kvm/tvBackend.img,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:10:26:ec,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:2 -vga std -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x6 -device vfio-pci,host=03:01.0,id=hostdev1,bus=pci.0,addr=0x7 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
W: kvm binary is deprecated, please use qemu-system-x86_64 instead
char device redirected to /dev/pts/0 (label charserial0)
qemu-system-x86_64: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x6: Warning, device 0000:03:00.0 does not support reset
qemu-system-x86_64: -device vfio-pci,host=03:01.0,id=hostdev1,bus=pci.0,addr=0x7: Warning, device 0000:03:01.0 does not support reset
qemu-system-x86_64: -device vfio-pci,host=03:01.0,id=hostdev1,bus=pci.0,addr=0x7: vfio: Error: Failed to setup INTx fd: Device or resource busy
qemu-system-x86_64: -device vfio-pci,host=03:01.0,id=hostdev1,bus=pci.0,addr=0x7: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=03:01.0,id=hostdev1,bus=pci.0,addr=0x7: Device 'vfio-pci' could not be initialized
2014-03-12 16:48:48.639+0000: shutting down
--><-- /var/log/syslog
Mar 12 18:07:45 father kernel: [34317.290958] genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:03:01.0)) vs. 00000080 (ehci_hcd:usb1)
--><--
as i have only 2 pci slots on the board and no other board with serious VT-d/IOMMU support i have no way to swap slots.
as the second card shared IRQ 16 with the ehci controller i soft-removed the ehci controller for testing purposes and voila the guest started with the 2 cards successfully inside, even mythtv works ... for now.
i hope to find a way to somehow rearrange the irqs as i sometimes need USB on the machine :D
wbr,tja...
----- Ursprüngliche Mail -----
Von: "Laine Stump" <laine at laine.org>
An: libvirt-users at redhat.com
CC: "Thomas Jagoditsch" <tja at tjasoft.com>
Gesendet: Mittwoch, 12. März 2014 16:13:15
Betreff: Re: [libvirt-users] PCI Passthrough of 2 identical devices
On 03/12/2014 06:29 AM, Thomas Jagoditsch wrote:
> hi,
>
> i have a small trouble with pci-passthrough.
>
> i have a working configuration passing a tv card into the guest, all is fine and dandy.
> as soon as i add my second identical card into the host i cant start the guest anymore, whether i add the second card to the guest or not. error message is identical in both cases.
>
> message of virt-manager|virsh|libvirtd.log:
>> libvirtError: internal error: Unable to reset PCI device 0000:03:00.0: internal error: Active 0000:03:01.0 devices on bus with 0000:03:00.0, not doing bus reset
> the host is a plain kvm server, no services or apps whatsoever accessing the two devices.
> the other guests (firewall and fileserver) do not use the cards either.
The problem is that for kvm device assignment to work properly, the
device needs to be reset by libvirt after detaching it from the host
driver and before passing it to kvm, and these devices you are trying to
reset support neither "function level reset" nor "power management
reset", so libvirt must fallback to resetting the entire bus where the
device is plugged in. But of course it can't do that if the bus contains
other devices that are in use by the host or by other guests.
If your host OS is new enough to support vfio device assignment, I would
suggest using that instead (if you update to the latest upstrem libvirt
it will automatically use VFIO if it is available, otherwise you can add
"<driver name='vfio'/>" to the device definition to assure that it will
either use vfio or fail. You will also probably need to run "modprobe
vfio" before starting the guest). The reason I suggest this is that VFIO
will automatically handle resetting the assigned devices whenever
necessary, compared to the old KVM device assignment, where libvirt must
always reset the device because it has no better information about
whether or not it is really necessary.
Another thing you can try is plugging one of the cards into a different
slot - if you can find a slot that is on a different bus, then libvirt
will be able to reset the bus containing the card that will be assigned
(since there won't be any other active devices on that same bus).
>
> if i (soft) remove the 2nd card via
>> echo -n 1 > /sys/bus/pci/devices/0000\:03\:01.0/remove
> i can start the guest with the 1st card assigned.
>
> thx for anyone looking into this.
libvirt attempts to reset the
>
> wbr,tja..
>
>
> PS:
> host:
>> root at father:~# lspci
>> 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
>> 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
>> 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
>> 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
>> 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 05)
>> 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
>> 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
>> 00:1c.3 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d5)
>> 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
>> 00:1f.0 ISA bridge: Intel Corporation H87 Express LPC Controller (rev 05)
>> 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
>> 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
>> 02:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03)
>> 03:00.0 Multimedia controller: Philips Semiconductors SAA7146 (rev 01)
>> 03:01.0 Multimedia controller: Philips Semiconductors SAA7146 (rev 01)
>> root at father:~# cat /etc/lsb-release
>> DISTRIB_ID=Ubuntu
>> DISTRIB_RELEASE=13.10
>> DISTRIB_CODENAME=saucy
>> DISTRIB_DESCRIPTION="Ubuntu 13.10"
>> root at father:~# uname -a
>> Linux father 3.11.0-18-generic #32-Ubuntu SMP Tue Feb 18 21:11:14 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>> root at father:~# libvirtd --version
>> libvirtd (libvirt) 1.1.1
> guest:
>> root at father:~# cat /etc/libvirt/qemu/tvBackend.xml
>> <!--
>> WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
>> OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
>> virsh edit tvBackend
>> or other application using the libvirt API.
>> -->
>>
>> <domain type='kvm'>
>> <name>tvBackend</name>
>> <uuid>f647bb10-7f9a-f94c-33b9-3d99e8e753e0</uuid>
>> <memory unit='KiB'>1048576</memory>
>> <currentMemory unit='KiB'>1048576</currentMemory>
>> <vcpu placement='static'>2</vcpu>
>> <os>
>> <type arch='x86_64' machine='pc-i440fx-1.5'>hvm</type>
>> <boot dev='hd'/>
>> </os>
>> <features>
>> <acpi/>
>> <apic/>
>> <pae/>
>> </features>
>> <clock offset='utc'/>
>> <on_poweroff>destroy</on_poweroff>
>> <on_reboot>restart</on_reboot>
>> <on_crash>restart</on_crash>
>> <devices>
>> <emulator>/usr/bin/kvm-spice</emulator>
>> <disk type='file' device='disk'>
>> <driver name='qemu' type='raw'/>
>> <source file='/kvm/tvBackend.img'/>
>> <target dev='vda' bus='virtio'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
>> </disk>
>> <disk type='block' device='cdrom'>
>> <driver name='qemu' type='raw'/>
>> <target dev='hdc' bus='ide'/>
>> <readonly/>
>> <address type='drive' controller='0' bus='1' target='0' unit='0'/>
>> </disk>
>> <controller type='usb' index='0'>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
>> </controller>
>> <controller type='pci' index='0' model='pci-root'/>
>> <controller type='ide' index='0'>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
>> </controller>
>> <interface type='bridge'>
>> <mac address='52:54:00:10:26:ec'/>
>> <source bridge='brlan'/>
>> <model type='virtio'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
>> </interface>
>> <serial type='pty'>
>> <target port='0'/>
>> </serial>
>> <console type='pty'>
>> <target type='serial' port='0'/>
>> </console>
>> <input type='mouse' bus='ps2'/>
>> <graphics type='vnc' port='-1' autoport='yes'/>
>> <video>
>> <model type='vga' vram='9216' heads='1'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
>> </video>
>> <hostdev mode='subsystem' type='pci' managed='yes'>
>> <source>
>> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
>> </source>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
>> </hostdev>
>> <memballoon model='virtio'>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
>> </memballoon>
>> </devices>
>> </domain>
--
thomas jagoditsch - tjaSoft
softWareEntwicklung - netzWerkManagement
More information about the libvirt-users
mailing list