[vfio-users] sr-iov support on main boards

[Alex Williamson]

On Sun, 27 Aug 2017 18:03:25 +0200
Torbjorn Jansson <torbjorn.jansson at mbox200.swipnet.se> wrote:

> On 2017-08-22 19:38, Alex Williamson wrote:
> > On Mon, 21 Aug 2017 23:09:55 +0200
> > Torbjorn Jansson <torbjorn.jansson at mbox200.swipnet.se> wrote:
> >   
> >> On 2017-08-21 19:56, Torbjorn Jansson wrote:  
> >>> On 2017-08-19 07:37, Torbjorn Jansson wrote:  
> >>>> On 2017-08-19 02:34, Taiidan at gmx.com wrote:  
> >>>>> On 08/15/2017 02:12 AM, Torbjorn Jansson wrote:
> >>>>>     
> >>>>>> On 2017-08-15 02:44, Taiidan at gmx.com wrote:  
> >>>>>>> On 08/14/2017 12:28 AM, Torbjorn Jansson wrote:
> >>>>>>>     
> >>>>>>>> yes i noticed that part was missing, but the card i have is an I350-T2
> >>>>>>>> and according to:
> >>>>>>>> https://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000005722.html
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> the I350-T2 is supposed to have sr-iov support.
> >>>>>>>> i guess it is possible they sent me the wrong type of card or something,
> >>>>>>>> i'll double check what is listed on the card itself.  
> >>>>>>> Some OEM's (ex: dell) shut off the SR-IOV ability on their NIC's because
> >>>>>>> reasons.
> >>>>>>> Please provide an lspci -n and lspci -v -v for that device so we can see
> >>>>>>> what OEM it is.
> >>>>>>>
> >>>>>>> I would return it and have them send you a real one that isn't nerfed,
> >>>>>>> also FYI most NIC's on ebay are counterfeit so be careful what you buy.
> >>>>>>> https://www.servethehome.com/investigating-fake-intel-i350-network-adapters/
> >>>>>>> (real intel NIC ASIC and SR-IOV but crappy secondary components)
> >>>>>>>     
> >>>>>>
> >>>>>> if i look at the back of the card there is a sticker with: I350T2V2
> >>>>>>
> >>>>>> here is the output:  
> >>>>> Weird.
> >>>>> I would say you almost definitely got a fake - either way send it back.
> >>>>>     
> >>>>
> >>>> yes very strange.
> >>>> thing is, i bought it from a well known web shop I've been using forever.
> >>>> i have opened a support ticket with them about this, hopefully i can return it.
> >>>>
> >>>> but then question is, how do i find a network card with sr-iov that works?
> >>>> clearly specs on web shops can't always be trusted.
> >>>>
> >>>> also strange is that this page:
> >>>> http://ark.intel.com/products/84804/Intel-Ethernet-Server-Adapter-I350-T2V2
> >>>> clearly lists sr-iov and this is the model i got.
> >>>>     
> >>>
> >>> i did a bit more searching and checking.
> >>> found this pdf:
> >>> https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/ethernet-controller-i350-datasheet.pdf
> >>>
> >>> section 6.2.25 looks interesting specifically the IOV Enable bit.
> >>>
> >>> i used ethtool to dump the eeprom to file and a hex editor to check the
> >>> contents and this bit is set to a zero indicating IOV is not exposed.
> >>>
> >>> but i'm definitely no expert on these things so i might be wrong too.
> >>> but i wonder if there is a way to flip this bit and see what happens.
> >>>      
> >>
> >> i fixed it :)
> >> i figured out how to use ethtool to flip the bit, rebooted and:
> >> -------
> >> 03:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection
> >> (rev 01)
> >>           Subsystem: Intel Corporation Ethernet Server Adapter I350-T2
> >>           Physical Slot: 6-1
> >>           Flags: bus master, fast devsel, latency 0, IRQ 61, NUMA node 0
> >>           Memory at fad00000 (32-bit, non-prefetchable) [size=1M]
> >>           Memory at fae04000 (32-bit, non-prefetchable) [size=16K]
> >>           Capabilities: [40] Power Management version 3
> >>           Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
> >>           Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
> >>           Capabilities: [a0] Express Endpoint, MSI 00
> >>           Capabilities: [100] Advanced Error Reporting
> >>           Capabilities: [140] Device Serial Number a0-36-9f-ff-ff-ed-43-b0
> >>           Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
> >>           Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
> >>           Capabilities: [1a0] Transaction Processing Hints
> >>           Capabilities: [1c0] Latency Tolerance Reporting
> >>           Capabilities: [1d0] Access Control Services
> >>           Kernel driver in use: igb
> >>           Kernel modules: igb
> >> -------
> >> as you can see above sr-iov is now listed in lspci, needed sriov entries is now
> >> listed under /sys and i can enable the virtual cards.
> >> had to blacklist igbvf and libvirt kept complaining until i added the virtual
> >> cards to vfio too.
> >>
> >> so now i finally got it all working and have a vm with a virtual card.
> >>
> >> but i think it is really stupid of the manufacture(s) (intel?) to disable
> >> sr-iov in the eeprom and not provide any proper tool or documentation on how to
> >> enable it.  
> > 
> > Would you be so kind as to write-up instructions how you used ethtool
> > to resolve this so that others don't need to re-invent your solution?
> > Thanks,
> >   
> 
> yes of course.
> 
> first of all, be careful when you poke around in the cards eeprom.
> below is what i did and it worked on my I350-T2V2 card.
> theoretically it might work on other I350 cards too but it is not something i 
> have tested.
> 
> before starting you need to know the device name of your ethernet device, it is 
> also useful to check output of lspci -v on your device and check that your card 
> is missing this line:
> Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
> 
> first make a backup of your eeprom:
> ethtool --eeprom-dump DEVNAME raw on >eth-eeprom.bin
> 
> i used this both as backup and to check the contents with a hex editor and to 
> make sure i understood the documentation correctly regarding endianess and offsets.
> 
> next we want to check what the iov enable bit is currently set to.
> according to documentation i found, this is bit zero of word 0x25 in eeprom.
> so this makes byte offset 0x4a (0x25*2 documentation uses 16 bit words so we 
> need to multiply by 2 to get byte offset)
> so:
> 
> ethtool --eeprom-dump DEVNAME offset 0x4a length 1
> 
> in my case i got 0xf6 back.
> so bit zero is set to a zero which means IOV capability is not exposed 
> (expected based on earlier lspci -v).
> there is no point in continuing unless it is a zero.
> 
> take above value you just read out and set bit zero to 1 and then use your 
> newly calculated value in below command to change it:
> 
> ethtool --change-eeprom DEVNAME magic 0x15218086 offset 0x4a length 1 value 
> 0xYOUR_VALUE_GOES_HERE
> 
> the magic value to ethtool is i think deliberately not documented to prevent 
> you from accidentally screwing up your eeprom.
> in this case it is the pci vendor and device id (lspci -n).
> 
> once this is done, reboot computer and check output of lspci -v and look for:
> Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
> if everything worked out it should now show up and you can go ahead with the 
> rest of sr-iov configuration.
> 
> again, be careful when you do this.
> 
> 
> 
> if you need to use this then i would recommend contacting intel support about 
> this because i think this is not the proper way of doing things.
> i consider this more of a "hack" or a workaround than a proper solution.
> i would rather have a proper tool or documentation or something from intel 
> specifying what to expect regarding IOV enable/disable on these cards.
> 
> i found no documentation or tool to enable this or i would not have done this 
> via ethtool.
> 
> it is not fun to waste weeks on troubleshooting sr-iov only to find out that it 
> was disabled in the cards eeprom.
> it also doesn't help that when you lookup specs for this card it clearly lists 
> that it supports sr-iov, even the place i bought it from and their support said 
> it supported sr-iov (incorrectly, kind of)
> 

Thanks for the write-up, this will be useful to have in the archives!
Thanks,

Alex