[libvirt-users] intel quad gigabit nic and pci passthrough

[Ivan Kabaivanov]

Hi all

I have a very weird case of pci passthrough.

I have a machine with 7 network interfaces, all of them intel.  Four of them are on one quad giga ethernet device.  If I manually unbind the devices and allow qemu to use them, with intel IOMMU working, everything works like a charm.  Here's how I do it manually:

root at kybrat (x86_64) ~]$ lspci -nn | grep net
00:19.0 Ethernet controller [0200]: Intel Corporation 82579V Gigabit Network Connection [8086:1503] (rev 06)
02:00.0 Ethernet controller [0200]: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) [8086:107d] (rev 06)
05:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:10a4] (rev 06)
05:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:10a4] (rev 06)
06:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:10a4] (rev 06)
06:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:10a4] (rev 06)
0c:00.0 Ethernet controller [0200]: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) [8086:107d] (rev 06)


echo "8086 10a4" > /sys/bus/pci/drivers/pci-stub/new_id
echo "8086 107d" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:05:00.0" > /sys/bus/pci/devices/0000\:05\:00.0/driver/unbind
echo "0000:05:00.0" > /sys/bus/pci/drivers/pci-stub/bind
echo "0000:05:00.1" > /sys/bus/pci/devices/0000\:05\:00.1/driver/unbind
echo "0000:05:00.1" > /sys/bus/pci/drivers/pci-stub/bind
echo "0000:06:00.0" > /sys/bus/pci/devices/0000\:06\:00.0/driver/unbind
echo "0000:06:00.0" > /sys/bus/pci/drivers/pci-stub/bind
echo "0000:06:00.1" > /sys/bus/pci/devices/0000\:06\:00.1/driver/unbind
echo "0000:06:00.1" > /sys/bus/pci/drivers/pci-stub/bind
echo "0000:0c:00.0" > /sys/bus/pci/devices/0000\:0c\:00.0/driver/unbind
echo "0000:0c:00.0" > /sys/bus/pci/drivers/pci-stub/bind

qemu-system-i386 -enable-kvm \
                 -m 2048 \
                 -cpu host \
                 -machine pc-q35-1.6 \                                           
                 -drive file=/media/virtual/krym.qcow2,if=virtio \       
                 -net none \                                                     
                 -device pci-assign,host=06:00.1,rombar=0 \                      
                 -device pci-assign,host=05:00.0,rombar=0 \                      
                 -device pci-assign,host=06:00.0,rombar=0 \                      
                 -device pci-assign,host=05:00.1,rombar=0 \                      
                 -device pci-assign,host=0c:00.0,rombar=0 \                      
                 -cdrom /media/virtual/asg-9.107-33.1.iso \                      
                 -boot c \                                                       
                 -vnc :0 \                                             
                 -runas virtual \                                                
                 -monitor unix:/media/virtual/control/krym/monitor,server,nowait \                                                                                                               
                 -rtc base=localtime,clock=host,driftfix=none

This starts qemu and pci passthrough works nicely.

Now, if I try to do the same thing through libvirt, virt-install doesn't like the quad ethernet device pci passthrough and I get the error below:

virt-install --name=krym --ram=2048 --vcpus=1 --cpu=host --import --os-type=linux --disk path=/media/virtual/krym.qcow2,format=qcow2,bus=virtio --nonetworks --graphics vnc,port=5900 --virt-type=kvm --machine=q35 --accelerate --host-device=pci_0000_06_00_1 --host-device=pci_0000_05_00_0 --host-device=pci_0000_06_00_0 --host-device=pci_0000_05_00_1 --host-device=pci_0000_0c_00_0

Starting install...
ERROR    Requested operation is not valid: PCI device 0000:06:00.1 is not assignable
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect qemu:///system start krym
otherwise, please restart your installation.


If I try to use just one NIC (any NIC, as long as it's not the quad NIC), things do work:

virt-install --name=krym --ram=2048 --vcpus=1 --cpu=host --import --os-type=linux --disk path=/media/virtual/krym.qcow2,format=qcow2,bus=virtio --nonetworks --graphics vnc,port=5900 --virt-type=kvm --machine=q35 --accelerate --host-device=pci_0000_0c_00_0

Starting install...
Creating domain...                                                                               |    0 B  00:00:01     
Cannot open display: 
Run 'virt-viewer --help' to see a full list of available command line options
Domain creation completed. You can restart your domain by running:
  virsh --connect qemu:///system start krym



So I guess libvirt has some problem with the intel quad NIC.

I will appreciate any help.

Thanks,
IvanK.