[libvirt-users] SR-IOV: no traffic isolation between VFs with Broadcom 10Gbps cards

[Yoann Juet]

Hi all,

I'm testing on debian/unstable SR-IOV feature with Broadcom BCM57810 
cards and KVM hypervisor:

Compiled against library: libvirt 1.2.1
Using library: libvirt 1.2.1
Using API: QEMU 1.2.1
Running hypervisor: QEMU 1.7.0

bnx2x
-> firmware 7.8.17
-> driver from kernel 3.12.7

8 VFs are created on the first PF. For each VF, a specific mac address 
is set manually using "ip link set eth0 vf x mac xx:xx:xx:xx:xx" 
command. I run several KVM guests with PCI passthrough (same kernel, 
bnx2x driver and firmware as the host), performance is close to bare metal.

Well, that sounds good, until I start capturing the traffic inside each 
VM: host traffic is visible as well as traffic destined to other VM. 
It's like if internal card switching was inoperable. I made several 
tests with different kernels, different PCIe Passthrough method 
assignments for libvirt. All failed.

Has anyone successfully experiment SR-IOV with Broadcom cards on linux ?

-----

Some details:

01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet (rev 10)
01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet (rev 10)

01:09.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.2 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.3 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.4 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.5 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.6 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.7 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function


# virsh nodedev-dumpxml pci_0000_01_09_0
<device>
   <name>pci_0000_01_09_0</name>
   <path>/sys/devices/pci0000:00/0000:00:01.0/0000:01:09.0</path>
   <parent>pci_0000_00_01_0</parent>
   <driver>
     <name>vfio-pci</name>
   </driver>
   <capability type='pci'>
     <domain>0</domain>
     <bus>1</bus>
     <slot>9</slot>
     <function>0</function>
     <product id='0x16af'>NetXtreme II BCM57810 10 Gigabit Ethernet 
Virtual Function</product>
     <vendor id='0x14e4'>Broadcom Corporation</vendor>
     <capability type='phys_function'>
       <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
     </capability>
     <iommuGroup number='35'>
       <address domain='0x0000' bus='0x01' slot='0x09' function='0x0'/>
     </iommuGroup>
   </capability>
</device>


# virsh nodedev-dumpxml pci_0000_01_09_1
<device>
   <name>pci_0000_01_09_1</name>
   <path>/sys/devices/pci0000:00/0000:00:01.0/0000:01:09.1</path>
   <parent>pci_0000_00_01_0</parent>
   <driver>
     <name>vfio-pci</name>
   </driver>
   <capability type='pci'>
     <domain>0</domain>
     <bus>1</bus>
     <slot>9</slot>
     <function>1</function>
     <product id='0x16af'>NetXtreme II BCM57810 10 Gigabit Ethernet 
Virtual Function</product>
     <vendor id='0x14e4'>Broadcom Corporation</vendor>
     <capability type='phys_function'>
       <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
     </capability>
     <iommuGroup number='36'>
       <address domain='0x0000' bus='0x01' slot='0x09' function='0x1'/>
     </iommuGroup>
   </capability>
</device>


Guest A XML:
	...
     <hostdev mode='subsystem' type='pci' managed='yes'>
       <source>
         <address domain='0x0000' bus='0x01' slot='0x09' function='0x0'/>
       </source>
     </hostdev>
	...


Guest B XML:

	...
     <hostdev mode='subsystem' type='pci' managed='yes'>
       <source>
         <address domain='0x0000' bus='0x01' slot='0x09' function='0x1'/>
       </source>
     </hostdev>
	...


-- 
Université de Nantes - Direction des Systèmes d'Information

-------------- next part --------------
A non-text attachment was scrubbed...
Name: yoann_juet.vcf
Type: text/x-vcard
Size: 365 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20140204/940d4c91/attachment.vcf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3256 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20140204/940d4c91/attachment.p7s>