[vfio-users] sr-iov support on main boards

[Zachary Boley]

Actually have a Xeon setup where ootb all iommu groups were sane and
separated, I just use the second NIC on my motherboard for the Windows vm
as virtio makes it a bottleneck

On Aug 13, 2017 8:19 AM, "Torbjorn Jansson" <
torbjorn.jansson at mbox200.swipnet.se> wrote:

> On 2017-08-08 00:40, Alex Williamson wrote:
>
>> On Mon, 24 Jul 2017 09:35:40 +0200
>> Torbjorn Jansson <torbjorn.jansson at mbox200.swipnet.se> wrote:
>>
>> Hello.
>>>
>>> i'm considering upgrading my linux box that i use for virtualization
>>> (i5-3470,
>>> Z77 mb).
>>> at the moment i have a successfully working setup where i pass thru a
>>> gtx970 to
>>> a windows vm.
>>> i also have an aging 4 port network card, see below:
>>> ------
>>> 04:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
>>> Controller (Copper) (rev 06)
>>> 04:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
>>> Controller (Copper) (rev 06)
>>> 05:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
>>> Controller (Copper) (rev 06)
>>> 05:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
>>> Controller (Copper) (rev 06)
>>> ------
>>>
>>
>>
>> Interestingly, we have quirks for most of the 82571 cards, but not the
>> quad port because the onboard switch doesn't support ACS so everything
>> would get grouped back together anyway.
>>
>>
>
> yes i have noticed they all get grouped together no matter what i do.
>
>
> i was thinking of replacing this card since i at the moment can't assign
>>> this
>>> in a good way, my only option now is to assign all 4 ports to same vm
>>> which
>>> doesn't really help.
>>>
>>
>> A dual port version of the same might give your two NICs in separate
>> groups depending on the ACS capabilities/quirks where it's installed
>> if you're looking for a cheap option.
>>
>>
>
> problem is, i will most likely need more than 2 ports.
>
>
> so, i was thinking of getting a x99 based mb, a xeon e5 v4 cpu and a
>>> I350-T2
>>> network card that supports sr-iov.
>>>
>>> question is, how do i know that the main board also have the necessary
>>> bits for
>>> sr-iov support?
>>> as i understand it, the bios have to do something too to make it work.
>>>
>>>
>>> main boards i was considering was MSI X99A gaming 7 and also ASUS X99-E
>>> WS.
>>> the msi board have a nice block diagram that says how things is
>>> connected and
>>> manual looks abit better
>>>
>>> asus manual i don't really get, i didn't find any clear indication of
>>> how many
>>> lanes you get on the slots and what is listed is a bit confusing.
>>>
>>> my thinking was that maybe a workstation mb have higher probability of
>>> better
>>> support for more advanced features like this.
>>> but this might be an incorrect assumption.
>>>
>>> any advice would be welcomed.
>>>
>>
>> An SR-IOV card needs to be installed into an ACS capable/quirked slot
>> or everything will be grouped together and it'd be pointless.  The PCH
>> slot on your Z77 (if it has one) would meet this requirement.  Beyond
>> that, ARI support can play a factor in the bus number layout and if it
>> exists can make something like an 82756/i350 work without BIOS support
>> given its scant SR-IOV resource requirements.  Without it, extra bus
>> numbers might need to be allocated which requires BIOS support, but
>> Linux can do as well with a pci=assign-buses boot options (pci=realloc
>> is also a useful options).  For a 1Gbps NIC, you're not really losing
>> anything by installing it into a PCH root port slot, so I'd give that a
>> shot on your current board unless you're just using this as an excuse to
>> upgrade the whole system.  Thanks,
>>
>>
> thanks for the advice.
> i decided to go ahead with my upgrade, the reason was not only the sr-iov
> part but mainly to get more cores and memory.
> i just want to make sure i can use the sr-iov part too while i'm upgrading.
>
> i decided to go for the "MSI X99A gaming 7" mainboard, but unfortunately i
> didn't have much luck in getting sr-iov working.
> booting with pci=realloc,assign-busses doesn't help.
> only difference i see in dmesg is things like:
> [    1.432766] pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 03
> [    1.432915] pci_bus 0000:04: busn_res: can not insert [bus 04-ff] under
> [bus 00-fe] (conflicts with (null) [bus 00-fe])
>
> so i suspect something is wrong.
> i have attached some logs.
>
> i also tried asking msi support about this and they were nice enough to
> create a custom bios where sr-iov is supposed to be enabled but
> unfortunately i still didn't have much luck.
>
> the mainboard in question is a desktop board and not a server board so i
> can kind of understand that it isn't working since sr-iov is not something
> your average desktop user is going to need or ask for.
>
> anyway, any input is appreciated.
>
> _______________________________________________
> vfio-users mailing list
> vfio-users at redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20170813/af0d5e7d/attachment.htm>