[vfio-users] sr-iov support on main boards

[Alex Williamson]

On Mon, 24 Jul 2017 09:35:40 +0200
Torbjorn Jansson <torbjorn.jansson at mbox200.swipnet.se> wrote:

> Hello.
> 
> i'm considering upgrading my linux box that i use for virtualization (i5-3470, 
> Z77 mb).
> at the moment i have a successfully working setup where i pass thru a gtx970 to 
> a windows vm.
> i also have an aging 4 port network card, see below:
> ------
> 04:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet 
> Controller (Copper) (rev 06)
> 04:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet 
> Controller (Copper) (rev 06)
> 05:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet 
> Controller (Copper) (rev 06)
> 05:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet 
> Controller (Copper) (rev 06)
> ------


Interestingly, we have quirks for most of the 82571 cards, but not the
quad port because the onboard switch doesn't support ACS so everything
would get grouped back together anyway.
 
> i was thinking of replacing this card since i at the moment can't assign this 
> in a good way, my only option now is to assign all 4 ports to same vm which 
> doesn't really help.

A dual port version of the same might give your two NICs in separate
groups depending on the ACS capabilities/quirks where it's installed
if you're looking for a cheap option.
 
> so, i was thinking of getting a x99 based mb, a xeon e5 v4 cpu and a I350-T2 
> network card that supports sr-iov.
> 
> question is, how do i know that the main board also have the necessary bits for 
> sr-iov support?
> as i understand it, the bios have to do something too to make it work.
> 
> 
> main boards i was considering was MSI X99A gaming 7 and also ASUS X99-E WS.
> the msi board have a nice block diagram that says how things is connected and 
> manual looks abit better
> 
> asus manual i don't really get, i didn't find any clear indication of how many 
> lanes you get on the slots and what is listed is a bit confusing.
> 
> my thinking was that maybe a workstation mb have higher probability of better 
> support for more advanced features like this.
> but this might be an incorrect assumption.
> 
> any advice would be welcomed.

An SR-IOV card needs to be installed into an ACS capable/quirked slot
or everything will be grouped together and it'd be pointless.  The PCH
slot on your Z77 (if it has one) would meet this requirement.  Beyond
that, ARI support can play a factor in the bus number layout and if it
exists can make something like an 82756/i350 work without BIOS support
given its scant SR-IOV resource requirements.  Without it, extra bus
numbers might need to be allocated which requires BIOS support, but
Linux can do as well with a pci=assign-buses boot options (pci=realloc
is also a useful options).  For a 1Gbps NIC, you're not really losing
anything by installing it into a PCH root port slot, so I'd give that a
shot on your current board unless you're just using this as an excuse to
upgrade the whole system.  Thanks,

Alex