[vfio-users] sr-iov support on main boards

[Alex Williamson]

On Mon, 21 Aug 2017 23:09:55 +0200
Torbjorn Jansson <torbjorn.jansson at mbox200.swipnet.se> wrote:

> On 2017-08-21 19:56, Torbjorn Jansson wrote:
> > On 2017-08-19 07:37, Torbjorn Jansson wrote:  
> >> On 2017-08-19 02:34, Taiidan at gmx.com wrote:  
> >>> On 08/15/2017 02:12 AM, Torbjorn Jansson wrote:
> >>>  
> >>>> On 2017-08-15 02:44, Taiidan at gmx.com wrote:  
> >>>>> On 08/14/2017 12:28 AM, Torbjorn Jansson wrote:
> >>>>>  
> >>>>>> yes i noticed that part was missing, but the card i have is an I350-T2 
> >>>>>> and according to:
> >>>>>> https://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000005722.html 
> >>>>>>
> >>>>>>
> >>>>>> the I350-T2 is supposed to have sr-iov support.
> >>>>>> i guess it is possible they sent me the wrong type of card or something, 
> >>>>>> i'll double check what is listed on the card itself.  
> >>>>> Some OEM's (ex: dell) shut off the SR-IOV ability on their NIC's because 
> >>>>> reasons.
> >>>>> Please provide an lspci -n and lspci -v -v for that device so we can see 
> >>>>> what OEM it is.
> >>>>>
> >>>>> I would return it and have them send you a real one that isn't nerfed, 
> >>>>> also FYI most NIC's on ebay are counterfeit so be careful what you buy.
> >>>>> https://www.servethehome.com/investigating-fake-intel-i350-network-adapters/ 
> >>>>> (real intel NIC ASIC and SR-IOV but crappy secondary components)
> >>>>>  
> >>>>
> >>>> if i look at the back of the card there is a sticker with: I350T2V2
> >>>>
> >>>> here is the output:  
> >>> Weird.
> >>> I would say you almost definitely got a fake - either way send it back.
> >>>  
> >>
> >> yes very strange.
> >> thing is, i bought it from a well known web shop I've been using forever.
> >> i have opened a support ticket with them about this, hopefully i can return it.
> >>
> >> but then question is, how do i find a network card with sr-iov that works?
> >> clearly specs on web shops can't always be trusted.
> >>
> >> also strange is that this page:
> >> http://ark.intel.com/products/84804/Intel-Ethernet-Server-Adapter-I350-T2V2
> >> clearly lists sr-iov and this is the model i got.
> >>  
> > 
> > i did a bit more searching and checking.
> > found this pdf:
> > https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/ethernet-controller-i350-datasheet.pdf 
> > 
> > section 6.2.25 looks interesting specifically the IOV Enable bit.
> > 
> > i used ethtool to dump the eeprom to file and a hex editor to check the 
> > contents and this bit is set to a zero indicating IOV is not exposed.
> > 
> > but i'm definitely no expert on these things so i might be wrong too.
> > but i wonder if there is a way to flip this bit and see what happens.
> >   
> 
> i fixed it :)
> i figured out how to use ethtool to flip the bit, rebooted and:
> -------
> 03:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection 
> (rev 01)
>          Subsystem: Intel Corporation Ethernet Server Adapter I350-T2
>          Physical Slot: 6-1
>          Flags: bus master, fast devsel, latency 0, IRQ 61, NUMA node 0
>          Memory at fad00000 (32-bit, non-prefetchable) [size=1M]
>          Memory at fae04000 (32-bit, non-prefetchable) [size=16K]
>          Capabilities: [40] Power Management version 3
>          Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
>          Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
>          Capabilities: [a0] Express Endpoint, MSI 00
>          Capabilities: [100] Advanced Error Reporting
>          Capabilities: [140] Device Serial Number a0-36-9f-ff-ff-ed-43-b0
>          Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
>          Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
>          Capabilities: [1a0] Transaction Processing Hints
>          Capabilities: [1c0] Latency Tolerance Reporting
>          Capabilities: [1d0] Access Control Services
>          Kernel driver in use: igb
>          Kernel modules: igb
> -------
> as you can see above sr-iov is now listed in lspci, needed sriov entries is now 
> listed under /sys and i can enable the virtual cards.
> had to blacklist igbvf and libvirt kept complaining until i added the virtual 
> cards to vfio too.
> 
> so now i finally got it all working and have a vm with a virtual card.
> 
> but i think it is really stupid of the manufacture(s) (intel?) to disable 
> sr-iov in the eeprom and not provide any proper tool or documentation on how to 
> enable it.

Would you be so kind as to write-up instructions how you used ethtool
to resolve this so that others don't need to re-invent your solution?
Thanks,

Alex