<div dir="auto">Actually have a Xeon setup where ootb all iommu groups were sane and separated, I just use the second NIC on my motherboard for the Windows vm as virtio makes it a bottleneck</div><div class="gmail_extra"> <div class="gmail_quote">On Aug 13, 2017 8:19 AM, "Torbjorn Jansson" <<a href="mailto:torbjorn.jansson@mbox200.swipnet.se">torbjorn.jansson@mbox200.swipnet.se</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 2017-08-08 00:40, Alex Williamson wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> On Mon, 24 Jul 2017 09:35:40 +0200 Torbjorn Jansson <<a href="mailto:torbjorn.jansson@mbox200.swipnet.se" target="_blank">torbjorn.jansson@mbox200.swipnet.se</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Hello. i'm considering upgrading my linux box that i use for virtualization (i5-3470, Z77 mb). at the moment i have a successfully working setup where i pass thru a gtx970 to a windows vm. i also have an aging 4 port network card, see below: ------ 04:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06) 04:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06) 05:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06) 05:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06) ------ </blockquote> Interestingly, we have quirks for most of the 82571 cards, but not the quad port because the onboard switch doesn't support ACS so everything would get grouped back together anyway. </blockquote> yes i have noticed they all get grouped together no matter what i do. <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> i was thinking of replacing this card since i at the moment can't assign this in a good way, my only option now is to assign all 4 ports to same vm which doesn't really help. </blockquote> A dual port version of the same might give your two NICs in separate groups depending on the ACS capabilities/quirks where it's installed if you're looking for a cheap option. </blockquote> problem is, i will most likely need more than 2 ports. <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> so, i was thinking of getting a x99 based mb, a xeon e5 v4 cpu and a I350-T2 network card that supports sr-iov. question is, how do i know that the main board also have the necessary bits for sr-iov support? as i understand it, the bios have to do something too to make it work. main boards i was considering was MSI X99A gaming 7 and also ASUS X99-E WS. the msi board have a nice block diagram that says how things is connected and manual looks abit better asus manual i don't really get, i didn't find any clear indication of how many lanes you get on the slots and what is listed is a bit confusing. my thinking was that maybe a workstation mb have higher probability of better support for more advanced features like this. but this might be an incorrect assumption. any advice would be welcomed. </blockquote> An SR-IOV card needs to be installed into an ACS capable/quirked slot or everything will be grouped together and it'd be pointless. The PCH slot on your Z77 (if it has one) would meet this requirement. Beyond that, ARI support can play a factor in the bus number layout and if it exists can make something like an 82756/i350 work without BIOS support given its scant SR-IOV resource requirements. Without it, extra bus numbers might need to be allocated which requires BIOS support, but Linux can do as well with a pci=assign-buses boot options (pci=realloc is also a useful options). For a 1Gbps NIC, you're not really losing anything by installing it into a PCH root port slot, so I'd give that a shot on your current board unless you're just using this as an excuse to upgrade the whole system. Thanks, </blockquote> thanks for the advice. i decided to go ahead with my upgrade, the reason was not only the sr-iov part but mainly to get more cores and memory. i just want to make sure i can use the sr-iov part too while i'm upgrading. i decided to go for the "MSI X99A gaming 7" mainboard, but unfortunately i didn't have much luck in getting sr-iov working. booting with pci=realloc,assign-busses doesn't help. only difference i see in dmesg is things like: [ 1.432766] pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 03 [ 1.432915] pci_bus 0000:04: busn_res: can not insert [bus 04-ff] under [bus 00-fe] (conflicts with (null) [bus 00-fe]) so i suspect something is wrong. i have attached some logs. i also tried asking msi support about this and they were nice enough to create a custom bios where sr-iov is supposed to be enabled but unfortunately i still didn't have much luck. the mainboard in question is a desktop board and not a server board so i can kind of understand that it isn't working since sr-iov is not something your average desktop user is going to need or ask for. anyway, any input is appreciated. _______________________________________________ vfio-users mailing list <a href="mailto:vfio-users@redhat.com">vfio-users@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/vfio-users" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/vfio-users</a> </blockquote></div></div>