[vfio-users] Tracing PCI Device Behind PCI-Bridge

[Alex Williamson]

On Thu, 2015-10-22 at 09:30 -0400, John Stratoudakis wrote:
> Hi,
> 
> 
> I have spent a lot of time researching this and trying to figure out
> how to do this, so please forgive me if I did not find something
> already documented somewhere.
> 
> 
> So here is what I am trying to do:
> 
> 
> I have a PXI device (National Instruments PXI-7953R), where PXI is an
> extension of PCI.  This device is plugged in to a PXI-chassis
> (PXI-1033).  Then this PXI-chassis is connected to the host computer
> via a PCI card - PCI-8361.
> 
> 
> What I am trying to do is to trace the PCI-8361 device so that I can
> reverse engineer the device (proprietary) drivers given to me by
> National Instruments.

Seems like the wrong place to trace.  You'd only see the config space
accesses to access and setup the bridge.  In this case the bridge
doesn't even have any memory regions of its own, any accesses through
the bridge windows won't generate a trap, nor will you be able to see
config accesses to the device behind the bridge.

> When I plug in the device (PCI-8361) and I run lspci, I get the
> following output:

More importantly, does the PXI device show up in lspci, what's it look
like?  That's where you want to do the tracing.

> 0c:01.0 PCI bridge: Pericom Semiconductor PI7C9X111SL PCIe-to-PCI
> Reversible Bridge (rev 02) (prog-if 00 [Normal decode])
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
> Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 32, Cache Line Size: 64 bytes
> Bus: primary=0c, secondary=0d, subordinate=0f, sec-latency=0
> I/O behind bridge: 0000f000-00000fff
> Memory behind bridge: f8000000-f81fffff
> Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
> Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort+ <SERR- <PERR-
> BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> Capabilities: [a8] Subsystem: National Instruments Device 8361
> Kernel modules: shpchp
> 
> 
> So it looks like it is using the shpchp kernel driver.  When I assign
> this device to pci-stub during boot, and try to unbind it and bind it
> to vfio-pci, it fails.  The device shows up in lspci with no loaded
> driver.

Right, vfio-pci does not support bridges, they have a different config
space layout and there's never been any need to support them.  Generally
attempting to assign a bridge to a guest is the wrong thing to do.

> After I did some digging, I found out:
> * One cannot bind a PCI-bridge to vfio-pci.
> * One has to bind everything inside an iommu_group to vfio-pci

Not completely true.  All endpoints need to either be bound to vfio-pci
or pci-stub, obviously you can only directly access devices bound to
vfio-pci.  Devices with no driver are also acceptable, though won't be
accessible either.  We also whitelist pcieport, since often root ports
and downstream switch ports are in the same iommu group.  I also plan to
include a change in kernel v4.4 that opens that to whitelisting any
bridge drivers, which would help you out here
(https://lkml.org/lkml/2015/9/11/573).

> I also know that:
> * Using mmiotrace will not trace all Memory-Mapped writes/reads that
> are behind a PCI bridge
> 
> 
> My questions are:
> * How would someone reverse engineer PCI devices before IOMMU/VT-d
> hardware existed?

With a PCI bus analyzer ($$$$)

> * Can I pass this PCI-bridge the way it is (whatever that may mean) to
> a qemu host and trace it along with all reads/writes using the qemu
> tracing mechanisms?

No, you'll want to try to assign the PXI device to QEMU and trace that,
that's the only way you're going to see guest reads and writes to MMIO
space and config space of the device.  Hopefully the device is
sufficiently close to PCI for the driver to work.  Thanks,

Alex