[vfio-users] Lost link when pass through rtl8168 to guest

[Alex Williamson]

On Wed, 21 Sep 2016 14:04:20 +0800
Wei Xu <wexu at redhat.com> wrote:

> On 2016年09月21日 13:41, Wei Xu wrote:
>  > On 2016年09月21日 12:31, Alex Williamson wrote:  
>  >> On Wed, 21 Sep 2016 11:52:31 +0800
>  >> Wei Xu <wexu at redhat.com> wrote:
>  >>  
>  >>> On 2016年09月21日 02:59, Nick Sarnie wrote:  
>  >>>> Hi Wei,
>  >>>>
>  >>>> My system is a desktop, so it must just be a general Gigabyte BIOS   
> bug.
>  >>>> I submitted a help ticket about this issue and just gave a brief
>  >>>> explanation and then sent Alex's explanation. Hopefully it will be
>  >>>> escalated correctly.  
>  >>>
>  >>> Thanks for your feedback, i'm also using a Gigabyte board, i have
>  >>> checked out the firmware update history and updated my firmware to the
>  >>> latest one which was released at March, looks it's a long way to get a
>  >>> feedback for this issue from them.
>  >>>
>  >>> Alex,
>  >>> It's a hard time for us to do nothing but wait, the reason why i use my
>  >>> desktop is i got a com console on it, so it's quite convenient to
>  >>> debugging kernel via kgdb, and i want to keep my realtek nic for ssh
>  >>> access from my notebook, anyway to workaround it to just bypass the
>  >>> wireless nic only as a temporary experiment?
>  >>>
>  >>> I'm trying VirtIO DMAR patch with vIOMMU in the guest recently, which
>  >>> need pass through a pcie unit from host, and one more virtio nic   
> for the
>  >>> guest due to the feedbacks, maybe i can pass through a device in other
>  >>> groups instead of a nic?  
>  >>
>  >> Sure, but skylake platforms are notoriously bad for their lack of
>  >> device isolation, even things like USB controllers and audio devices
>  >> are now part of multifunction packages that do not expose isolation
>  >> through ACS.  If you can't resolve the IOMMU grouping otherwise, your
>  >> choices are as I told Nick in the other thread:
>  >>
>  >>    "Your choices are to run an unsupported (and unsupportable)
>  >>    configuration using the ACS override patch, get your hardware vendor
>  >>    to fix their platform, or upgrade to better hardware with better
>  >>    isolation characteristics."
>  >>
>  >> It's unfortunate that Intel provides VT-d on consumer platforms without
>  >> sufficient device isolation to really make it usable, but that's often
>  >> the state of things.  The workstation and server class platforms,
>  >> supporting Xeon E5 or High End Desktop Processors provide the necessary
>  >> isolation.  Thanks,  
>  >
>  > Yes, fortunately i get it solved finally, i tried adding the 'r8169'
>  > driver to the kernel group whitelist behind 'pci-stub' and recompile  &
>  > update the kernel firstly, and the VM boot up successfully, but a map
>  > page to iova error for realtek nic during DMA crashed the system later,
>  > looks it was caused by the group dependency, i remembered the vfio doc
>  > tells the group is the minimum isolation unit.

This approach is just a bad idea.

>  >
>  > Then i found there are 3 pci bridges on my board, 2 of them are with a
>  > group, another is a separate group, after plug the iwl wlan nic to this
>  > one, everything works well.  
> 
> Just noticed a topology change of my system, looks the PCI bridges is
> different as before after i changed the slot for my wlan nic, i used to
> think i plugged it to 00:1d.0 but it was connected to Sky Lake PCIe 
> controller, does this mean there are hidden PCI bridges for pci 
> enumeration in the system, is this allowable?
> 
> Before:
> 00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root 
> Port #5 (rev f1) (prog-if 00 [Normal decode])
> 00:1c.7 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root 
> Port #8 (rev f1) (prog-if 00 [Normal decode]) ------------ wlan nic
> 00:1d.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root 
> Port #9 (rev f1) (prog-if 00 [Normal decode])
> 
> Now:
> 00:01.0 PCI bridge: Intel Corporation Sky Lake PCIe Controller (x16) 
> (rev 07) (prog-if 00 [Normal decode]) ------------ wlan nic
> 00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root 
> Port #5 (rev f1) (prog-if 00 [Normal decode])
> 00:1d.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root 
> Port #9 (rev f1) (prog-if 00 [Normal decode])

There are generally two sources of PCIe root ports on Intel systems,
the processor itself and the PCH (Platform Controller Hub).  Look at a
block diagram for a modern system and you'll see this.  Typically for a
client processor (i3/i5/i7) there is no isolation between or
downstream of the individual processor root ports and isolation between
the individual PCH root ports is via quirks, because Intel didn't
include ACS or broke ACS.  You've found these processor root ports.
Why don't they show up in lspci when nothing is plugged into them?  Why
should they?  Chances are almost certain that your system does not
support PCI hotplug, so there's no requirement to expose empty
bridges.  I'm glad you've found a working setup, desktop class systems
often have poor isolation characteristics which make device assignment
difficult.  Thanks,

Alex