[vfio-users] Deciding whether to stick with X399

[Brian Yglesias]

Quick update:

My understanding is that a processor firmware update is needed to fix this.  It rolled out on Ryzen months ago, but Threadripper is different, and has a dumber name.

I found a beta bios for my board, but it comes with the same microcode version as my current BIOS, which is 0x8001129 per cpuinfo.  This microcode is part of the CPU firmware (AGESA), and these are only sometimes updated with the UEFI.  The most recent version of the firmware (not to be confused with it's microcode or the bios version) is 1.0.0.3.  I read that some production BIOSes already come with 1.0.0.4, which worries me, because I have not heard that IOMMU has been fixed as accounts have not popped up as they did with Ryzen.

It could be a matter of sample size, or it could be that 1.0.0.4 is broken as well, and 1.0.0.5 will be, and so on, until eventually at long last...  the problem is just never fixed ;)

I've reached out to AMD, as I haven't been able to find a more recent official statement on the matter than a company REP on their reddit saying "I'll get back to you" in September, and then disappearing shortly thereafter.  I'll update if I get anything back.

Thanks for reading.


----- Original Message -----
From: "brian" <brian at atlanticdigitalsolutions.com>
To: "vfio-users" <vfio-users at redhat.com>
Sent: Thursday, December 7, 2017 2:35:02 AM
Subject: Re: [vfio-users] Deciding whether to stick with X399

The good news is that the motherboard is not broken.  (Ironically, it was a virtualization related setting (PCIE_ARI) that caused the system to not POST and made resetting the bios impossible, but only I had my m.2 stick in the port under the southbridge.)

The bad news is that the state of Threadripper for IOMMU is 95% percent broken.  The accounts wherein it works, upon closer inspection, all involve ESXi or VEGA cards.  I don't want to use the former, and I don't have the latter.

In addition, even when it works, the user still has to use the ACS patch, even though both the processor and the board support ACS.  When investing in a new platform, this in and of itself should be disqualifying, I would think, due to the potential for data loss.  Also, allow_unsafe_interrupts is necessary, though that probably is not as big of a deal.

I joined reddit just to ask what the status is for Threadripper.  If it sounds like a new AGESA will be out soon, and that it will fix the problem like on Ryzen, I may wait it out.

Otherwise, I don't suppose this is the best place to advertise a new high end AMD-based system. ;)

I'll update this thread if they tell me anything interesting on Reddit.

Thanks for reading.

----- Original Message -----
From: "brian" <brian at atlanticdigitalsolutions.com>
To: "vfio-users" <vfio-users at redhat.com>
Sent: Wednesday, December 6, 2017 2:16:15 PM
Subject: Re: [vfio-users] Deciding whether to stick with X399

Sorry for the double post.  I keep forgetting to make sure my email client doesn't use html to quote.  What follows is my original message:

======


Thanks for the response.

This guy had success with a similar board and ESXi.

https://www.reddit.com/r/Amd/comments/72ula0/tr1950x_gtx_1060_passthrough_with_esxi/


I also found an account of it working on the same board as mine (Zenith).  Note that my IOMMU groups were better than his, possibly due to a bios setting he's missed, im guessing:

https://forum.level1techs.com/t/threadripper-gpu-passthrough-working-with-vega/120594

This guy is using a similar board from Asrock and uses KVM:

https://forum.level1techs.com/t/threadripper-gpu-passthrough-working-with-vega/120594


I didn't get much of a chance to look at the groupings, but I did notice that all three GPUs got their own IOMMU groups, and I was able to bind vfio-pci.  In addition I noticed I had many more groups than on my X58s or than the second post referened above.  It was close to 50.   Perhaps that was down to the "unique IOMMU per IVR (or so) setting in the bios.

Asus tech suppoirt might think of something that I haven't, and could get a chance to test it tomorrow.  It's a great value, especially for this application, so I'm still inclined to keep it if I can.

-Brian

----- Original Message -----
From: "Taiidan" <Taiidan at gmx.com>
To: "brian" <brian at atlanticdigitalsolutions.com>, "vfio-users" <vfio-users at redhat.com>
Sent: Wednesday, December 6, 2017 3:55:59 AM
Subject: Re: [vfio-users] Deciding whether to stick with X399

FYI the X58 is very old and Intel didn't really make a proper IOMMU for 
that generation of CPU's - no Interrupt Remapping and most boards lack 
IOMMU-GFX support so you can't attach a video card. Don't let it color 
your experience and future endeavors.

I would RMA your X399 board, assuming it has proper IOMMU groups and 
enough slots you can easily do multi seat.

While I do not have a Ryzen (I didn't want a system with ME/PSP) I play 
games in a VM on my KGPE-D16 and AMD Opteron 6274 "Bulldozer" setup so 
it does work with AMD stuff - for the end user the intel/v/amd IOMMU 
technical details don't really matter they both work fine.

They've solved the Ryzen IOMMU bugs just *make sure* you get a board 
where the vendor releases BIOS updates, properly implements IOMMU (most 
don't, which is why I bought my coreboot board) and doesn't make five 
different revisions of the same board with slightly different features 
and issues.