[vfio-users] vfio passthrough devices behind pcie switcher problem

[Alex Williamson]

On Thu, 9 Mar 2017 11:47:32 +0800
rhett rhett <rhett.kernel at gmail.com> wrote:

> somebody can help me ?

I asked for VM commandline or XML, you haven't provided it.  I asked
for lspci info, you haven't provided it.  Help us help you.

> 2017-03-08 14:34 GMT+08:00 rhett rhett <rhett.kernel at gmail.com>:
> 
> > here's some more error log from centos guest:
> >
> > Mar  7 05:38:07 localhost kernel: NVRM: loading NVIDIA UNIX x86_64 Kernel
> > Module  375.39  Tue Jan 31 20:47:00 PST 2017 (using threaded interrupts)
> > Mar  7 05:38:08 localhost kernel: nvidia-modeset: Loading NVIDIA Kernel
> > Mode Setting Driver for UNIX platforms  375.39  Tue Jan 31 19:41:48 PST 2017
> > Mar  7 05:39:27 localhost kernel: NVRM: RmInitAdapter failed!
> > (0x24:0x51:1060)
> > Mar  7 05:39:27 localhost kernel: NVRM: rm_init_adapter failed for device
> > bearing minor number 0
> > Mar  7 05:43:40 localhost kernel: NVRM: RmInitAdapter failed!
> > (0x24:0x51:1060)
> > Mar  7 05:43:40 localhost kernel: NVRM: rm_init_adapter failed for device
> > bearing minor number 0
> > Mar  8 05:07:47 localhost kernel: nvidia: module license 'NVIDIA' taints
> > kernel.
> > Mar  8 05:07:47 localhost kernel: NVRM: loading NVIDIA UNIX x86_64 Kernel
> > Module
> >
> > 2017-03-08 14:31 GMT+08:00 rhett rhett <rhett.kernel at gmail.com>:
> >  
> >> i have two guest , a windows 2008 server and a centos 7.2 . in windows,
> >> the device manager said the gpu can't start ,error code 10.
> >> in centos, when i run nvidia-smi,  it said no device found.
> >>
> >> no specil vm configurations,  whit the same config, i can use gpu
> >> successfully in my two gpu server. the biggest different is , that server
> >> is no pcie switcher.
> >>
> >> 2017-03-08 11:55 GMT+08:00 Alex Williamson <alex.williamson at redhat.com>:
> >>  
> >>> On Wed, 8 Mar 2017 11:26:17 +0800
> >>> rhett rhett <rhett.kernel at gmail.com> wrote:
> >>>  
> >>> > two gpus share the same irq , i found the reason. because the msi be
> >>> > disabled later , so irq 140 is being reused.
> >>> >
> >>> > but i don't know why somebady calls vfio_pci_ioctl to disable the msi.  
> >>>
> >>> vfio just does what the guest requests, but you're really providing
> >>> hardly any more information than when you asked off list.  My wild
> >>> guess, is that maybe you're running a Windows guest and not configuring
> >>> the VM for a vCPU type where Windows supports MSI.  For more
> >>> assistance, please provide basic information, like the QEMU command
> >>> line or VM XML, also the PCI information from the host (sudo lspci
> >>> -vvv), and of course any error codes in the guest or an actual
> >>> description of how the device doesn't work in the guest.  Thanks,
> >>>
> >>> Alex
> >>>
> >>>  
> >>> > 2017-03-08 10:55 GMT+08:00 rhett rhett <rhett.kernel at gmail.com>:
> >>> >  
> >>> > > i have a question about vfio , here is my description.
> >>> > >
> >>> > > i have 8 gpus in my server machine ,  but they are all behind a pcie
> >>> > > bridge.  when i make a vfio passthrough , i can't use the gpus in my  
> >>> guest  
> >>> > > os.
> >>> > > dmesg shows the following message
> >>> > >
> >>> > > [  662.208072] vfio-pci 0000:87:00.0: irq 140 for MSI/MSI-X
> >>> > > [  725.761623] vfio-pci 0000:04:00.0: irq 140 for MSI/MSI-X
> >>> > >
> >>> > > i started two vm , one use 87 and another use 04,  dmesg shows that  
> >>> they  
> >>> > > share the same irq 140 . is this normal ?
> >>> > >
> >>> > > i also saw the iommu groups, each gpu stays in a separate group， and  
> >>> with  
> >>> > > no other device in group. so this means ACS works correctly ?
> >>> > >
> >>> > > hope to get your helps !
> >>> > >  
> >>>
> >>>  
> >>  
> >