[libvirt-users] PCI Passthrough of 2 identical devices
junkmail
junkmail at lone.ath.cx
Thu Mar 13 07:30:00 UTC 2014
This is not really an answer but you might find this forum posting
helpful.
https://bbs.archlinux.org/viewtopic.php?id=162768&p=1
Has people passing Video Cards to VMs with VFIO and talks about the
different things that need to be configured to get it to work. I did
see someone trying to pass two Video cards to a VM, although I do not
believe that have gotten it working yet.
Good luck.
12.03.2014 12:50 に Thomas Jagoditsch さんは書きました:
> hi laine,
>
> thx for the fast answer.
>
> i tried VFIO as it seems supported and got at least one step further.
> starting the guest with one of the DVB cards is possible now, even
> when the other card is in the hosts pci config (not removed soft or
> hard).
>
> adding the second card/device thru ...
> > <hostdev mode='subsystem' type='pci' managed='yes'>
> > <driver name='vfio'/>
> > <source>
> > <address domain='0x0000' bus='0x03' slot='0x00'
> function='0x0'/>
> > </source>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x0'/>
> > </hostdev>
> > <hostdev mode='subsystem' type='pci' managed='yes'>
> > <driver name='vfio'/>
> > <source>
> > <address domain='0x0000' bus='0x03' slot='0x01'
> function='0x0'/>
> > </source>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
> function='0x0'/>
> > </hostdev>
> will render the guest unbootable:
>
> --><-- /var/log/libvirt/qemu/tvBackend.log
> 2014-03-12 16:48:48.018+0000: starting up
> LC_ALL=C
> PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
> QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name tvBackend -S -machine
> pc-i440fx-1.5,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp
> 2,sockets=2,cores=1,threads=1 -uuid
> f647bb10-7f9a-f94c-33b9-3d99e8e753e0 -no-user-config -nodefaults
> -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/tvBackend.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/kvm/tvBackend.img,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:10:26:ec,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:2 -vga std -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x6 -device vfio-pci,host=03:01.0,id=hostdev1,bus=pci.0,addr=0x7 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> W: kvm binary is deprecated, please use qemu-system-x86_64 instead
> char device redirected to /dev/pts/0 (label charserial0)
> qemu-system-x86_64: -device
> vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x6: Warning, device
> 0000:03:00.0 does not support reset
> qemu-system-x86_64: -device
> vfio-pci,host=03:01.0,id=hostdev1,bus=pci.0,addr=0x7: Warning, device
> 0000:03:01.0 does not support reset
> qemu-system-x86_64: -device
> vfio-pci,host=03:01.0,id=hostdev1,bus=pci.0,addr=0x7: vfio: Error:
> Failed to setup INTx fd: Device or resource busy
> qemu-system-x86_64: -device
> vfio-pci,host=03:01.0,id=hostdev1,bus=pci.0,addr=0x7: Device
> initialization failed.
> qemu-system-x86_64: -device
> vfio-pci,host=03:01.0,id=hostdev1,bus=pci.0,addr=0x7: Device
> 'vfio-pci' could not be initialized
> 2014-03-12 16:48:48.639+0000: shutting down
>
> --><-- /var/log/syslog
> Mar 12 18:07:45 father kernel: [34317.290958] genirq: Flags mismatch
> irq 16. 00000000 (vfio-intx(0000:03:01.0)) vs. 00000080
> (ehci_hcd:usb1)
> --><--
>
> as i have only 2 pci slots on the board and no other board with
> serious VT-d/IOMMU support i have no way to swap slots.
> as the second card shared IRQ 16 with the ehci controller i
> soft-removed the ehci controller for testing purposes and voila the
> guest started with the 2 cards successfully inside, even mythtv works
> ... for now.
>
> i hope to find a way to somehow rearrange the irqs as i sometimes need
> USB on the machine :D
>
>
> wbr,tja...
>
>
> ----- Ursprüngliche Mail -----
> Von: "Laine Stump" <laine at laine.org>
> An: libvirt-users at redhat.com
> CC: "Thomas Jagoditsch" <tja at tjasoft.com>
> Gesendet: Mittwoch, 12. März 2014 16:13:15
> Betreff: Re: [libvirt-users] PCI Passthrough of 2 identical devices
>
> On 03/12/2014 06:29 AM, Thomas Jagoditsch wrote:
>> hi,
>>
>> i have a small trouble with pci-passthrough.
>>
>> i have a working configuration passing a tv card into the guest, all
>> is fine and dandy.
>> as soon as i add my second identical card into the host i cant start
>> the guest anymore, whether i add the second card to the guest or not.
>> error message is identical in both cases.
>>
>> message of virt-manager|virsh|libvirtd.log:
>>> libvirtError: internal error: Unable to reset PCI device
>>> 0000:03:00.0: internal error: Active 0000:03:01.0 devices on bus with
>>> 0000:03:00.0, not doing bus reset
>> the host is a plain kvm server, no services or apps whatsoever
>> accessing the two devices.
>> the other guests (firewall and fileserver) do not use the cards
>> either.
>
> The problem is that for kvm device assignment to work properly, the
> device needs to be reset by libvirt after detaching it from the host
> driver and before passing it to kvm, and these devices you are trying
> to
> reset support neither "function level reset" nor "power management
> reset", so libvirt must fallback to resetting the entire bus where the
> device is plugged in. But of course it can't do that if the bus
> contains
> other devices that are in use by the host or by other guests.
>
> If your host OS is new enough to support vfio device assignment, I
> would
> suggest using that instead (if you update to the latest upstrem libvirt
> it will automatically use VFIO if it is available, otherwise you can
> add
> "<driver name='vfio'/>" to the device definition to assure that it will
> either use vfio or fail. You will also probably need to run "modprobe
> vfio" before starting the guest). The reason I suggest this is that
> VFIO
> will automatically handle resetting the assigned devices whenever
> necessary, compared to the old KVM device assignment, where libvirt
> must
> always reset the device because it has no better information about
> whether or not it is really necessary.
>
> Another thing you can try is plugging one of the cards into a different
> slot - if you can find a slot that is on a different bus, then libvirt
> will be able to reset the bus containing the card that will be assigned
> (since there won't be any other active devices on that same bus).
>
>
>>
>> if i (soft) remove the 2nd card via
>>> echo -n 1 > /sys/bus/pci/devices/0000\:03\:01.0/remove
>> i can start the guest with the 1st card assigned.
>>
>> thx for anyone looking into this.
>
> libvirt attempts to reset the
>>
>> wbr,tja..
>>
>>
>> PS:
>> host:
>>> root at father:~# lspci
>>> 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM
>>> Controller (rev 06)
>>> 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200
>>> v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
>>> 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series
>>> Chipset Family USB xHCI (rev 05)
>>> 00:16.0 Communication controller: Intel Corporation 8 Series/C220
>>> Series Chipset Family MEI Controller #1 (rev 04)
>>> 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection
>>> I217-V (rev 05)
>>> 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series
>>> Chipset Family USB EHCI #2 (rev 05)
>>> 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
>>> Family PCI Express Root Port #1 (rev d5)
>>> 00:1c.3 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d5)
>>> 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series
>>> Chipset Family USB EHCI #1 (rev 05)
>>> 00:1f.0 ISA bridge: Intel Corporation H87 Express LPC Controller (rev
>>> 05)
>>> 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series
>>> Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
>>> 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family
>>> SMBus Controller (rev 05)
>>> 02:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI
>>> Bridge (rev 03)
>>> 03:00.0 Multimedia controller: Philips Semiconductors SAA7146 (rev
>>> 01)
>>> 03:01.0 Multimedia controller: Philips Semiconductors SAA7146 (rev
>>> 01)
>>> root at father:~# cat /etc/lsb-release
>>> DISTRIB_ID=Ubuntu
>>> DISTRIB_RELEASE=13.10
>>> DISTRIB_CODENAME=saucy
>>> DISTRIB_DESCRIPTION="Ubuntu 13.10"
>>> root at father:~# uname -a
>>> Linux father 3.11.0-18-generic #32-Ubuntu SMP Tue Feb 18 21:11:14 UTC
>>> 2014 x86_64 x86_64 x86_64 GNU/Linux
>>> root at father:~# libvirtd --version
>>> libvirtd (libvirt) 1.1.1
>> guest:
>>> root at father:~# cat /etc/libvirt/qemu/tvBackend.xml
>>> <!--
>>> WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO
>>> BE
>>> OVERWRITTEN AND LOST. Changes to this xml configuration should be
>>> made using:
>>> virsh edit tvBackend
>>> or other application using the libvirt API.
>>> -->
>>>
>>> <domain type='kvm'>
>>> <name>tvBackend</name>
>>> <uuid>f647bb10-7f9a-f94c-33b9-3d99e8e753e0</uuid>
>>> <memory unit='KiB'>1048576</memory>
>>> <currentMemory unit='KiB'>1048576</currentMemory>
>>> <vcpu placement='static'>2</vcpu>
>>> <os>
>>> <type arch='x86_64' machine='pc-i440fx-1.5'>hvm</type>
>>> <boot dev='hd'/>
>>> </os>
>>> <features>
>>> <acpi/>
>>> <apic/>
>>> <pae/>
>>> </features>
>>> <clock offset='utc'/>
>>> <on_poweroff>destroy</on_poweroff>
>>> <on_reboot>restart</on_reboot>
>>> <on_crash>restart</on_crash>
>>> <devices>
>>> <emulator>/usr/bin/kvm-spice</emulator>
>>> <disk type='file' device='disk'>
>>> <driver name='qemu' type='raw'/>
>>> <source file='/kvm/tvBackend.img'/>
>>> <target dev='vda' bus='virtio'/>
>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
>>> function='0x0'/>
>>> </disk>
>>> <disk type='block' device='cdrom'>
>>> <driver name='qemu' type='raw'/>
>>> <target dev='hdc' bus='ide'/>
>>> <readonly/>
>>> <address type='drive' controller='0' bus='1' target='0'
>>> unit='0'/>
>>> </disk>
>>> <controller type='usb' index='0'>
>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
>>> function='0x2'/>
>>> </controller>
>>> <controller type='pci' index='0' model='pci-root'/>
>>> <controller type='ide' index='0'>
>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
>>> function='0x1'/>
>>> </controller>
>>> <interface type='bridge'>
>>> <mac address='52:54:00:10:26:ec'/>
>>> <source bridge='brlan'/>
>>> <model type='virtio'/>
>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>>> function='0x0'/>
>>> </interface>
>>> <serial type='pty'>
>>> <target port='0'/>
>>> </serial>
>>> <console type='pty'>
>>> <target type='serial' port='0'/>
>>> </console>
>>> <input type='mouse' bus='ps2'/>
>>> <graphics type='vnc' port='-1' autoport='yes'/>
>>> <video>
>>> <model type='vga' vram='9216' heads='1'/>
>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
>>> function='0x0'/>
>>> </video>
>>> <hostdev mode='subsystem' type='pci' managed='yes'>
>>> <source>
>>> <address domain='0x0000' bus='0x03' slot='0x00'
>>> function='0x0'/>
>>> </source>
>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>>> function='0x0'/>
>>> </hostdev>
>>> <memballoon model='virtio'>
>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
>>> function='0x0'/>
>>> </memballoon>
>>> </devices>
>>> </domain>
More information about the libvirt-users
mailing list