<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Just retried this morning after waking my computer up from sleep
(and having previously loaded a VM from earlier) and this time,
the ROM dump did succeed.<br>
<br>
Diffing <a moz-do-not-send="true"
href="https://www.diffchecker.com/xPdXUeUg">the contents of
lspci from then and now</a>, the device does appear to have
changed power states, even though enable in sysfs still contains 0
and attempting to write to it still gives the same "Device or
resource busy" error. The kernel logs from the wake up process
also don't contain anything related to that device. iomem hasn't
changed from before. Alex and Auger both seem to have guessed
correctly, though the steps it took to get there aren't exactly
satisfying. I remember vfio-pci having a disable D3 parameter
which I'll probably be looking into later today, seeing how power
states seem to be involved in the problem.<br>
<br>
Thanks for both of your help, I'll keep posting if I find new
things.<br>
- Nicolas<br>
</p>
<br>
<div class="moz-cite-prefix">On 03/11/2019 01:46 AM, Nicolas
Roy-Renaud wrote:<br>
</div>
<blockquote type="cite"
cite="mid:ab056b2b-73ba-4a18-d052-5d1ddbe4dea7@ens.etsmtl.ca">Hey,
Alex, thanks for replying.
<br>
<br>
It seems like you're right on the Mem- part.
<br>
<br>
[user@OCCAM ~]$ lspci -s 07:00.0 -vvv
<br>
07:00.0 VGA compatible controller: NVIDIA Corporation GM204
[GeForce GTX 970] (rev a1) (prog-if 00 [VGA controller])
<br>
Subsystem: ASUSTeK Computer Inc. GM204 [GeForce GTX 970]
<br>
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR+ FastB2B- DisINTx-
<br>
<br>
For comparison, here's the result from last boot after firing up
and shutting down a VM using that device:
<br>
<br>
[user@OCCAM ~]$ lspci -s 07:00.0 -vvv
<br>
07:00.0 VGA compatible controller: NVIDIA Corporation GM204
[GeForce GTX 970] (rev a1) (prog-if 00 [VGA controller])
<br>
Subsystem: ASUSTeK Computer Inc. GM204 [GeForce GTX 970]
<br>
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR+ FastB2B- DisINTx-
<br>
<br>
Yet when I try enabling the device, I get this :
<br>
<br>
[root@OCCAM user]# echo 1 >
/sys/bus/pci/devices/0000:07:00.0/enable
<br>
bash: echo: write error: Device or resource busy
<br>
<br>
Other people have apparently had similar issues caused by other
drivers claiming the ressources, and /proc/iomem was showing efifb
bound to that device earlier, however neither disabling it in the
kernel command line or unbinding it once the system is live have
solved the issue (even though it no longer shows up in iomem).
Here's PCI section from iomem with video=efifb:off. All the
commands whose outputs you'll see here have been run under these
conditions:
<br>
<br>
00000000-00000000 : PCI Bus 0000:00
<br>
<br>
00000000-00000000 : PCI Bus 0000:07
<br>
<br>
00000000-00000000 : 0000:07:00.0
<br>
<br>
00000000-00000000 : 0000:07:00.0
<br>
<br>
00000000-00000000 : PCI Bus 0000:01
<br>
<br>
00000000-00000000 : 0000:01:00.0
<br>
<br>
00000000-00000000 : 0000:01:00.0
<br>
<br>
00000000-00000000 : PCI Bus 0000:07
<br>
<br>
00000000-00000000 : 0000:07:00.0
<br>
<br>
00000000-00000000 : 0000:07:00.0
<br>
<br>
00000000-00000000 : 0000:07:00.1
<br>
<br>
00000000-00000000 : PCI Bus 0000:01
<br>
<br>
00000000-00000000 : 0000:01:00.0
<br>
<br>
00000000-00000000 : nvidia
<br>
<br>
00000000-00000000 : 0000:01:00.1
<br>
<br>
00000000-00000000 : ICH HD audio
<br>
<br>
00000000-00000000 : PCI Bus 0000:04
<br>
<br>
00000000-00000000 : 0000:04:00.0
<br>
<br>
00000000-00000000 : 0000:04:00.0
<br>
<br>
00000000-00000000 : ahci
<br>
<br>
00000000-00000000 : PCI Bus 0000:03
<br>
<br>
00000000-00000000 : 0000:03:00.0
<br>
<br>
00000000-00000000 : 0000:03:00.0
<br>
<br>
00000000-00000000 : r8169
<br>
<br>
00000000-00000000 : 0000:00:14.0
<br>
<br>
00000000-00000000 : xhci-hcd
<br>
<br>
00000000-00000000 : 0000:00:1b.0
<br>
<br>
00000000-00000000 : ICH HD audio
<br>
<br>
00000000-00000000 : 0000:00:1f.3
<br>
<br>
00000000-00000000 : 0000:00:1f.2
<br>
<br>
00000000-00000000 : ahci
<br>
<br>
00000000-00000000 : 0000:00:1d.0
<br>
<br>
00000000-00000000 : ehci_hcd
<br>
<br>
00000000-00000000 : 0000:00:1a.0
<br>
<br>
00000000-00000000 : ehci_hcd
<br>
<br>
00000000-00000000 : 0000:00:16.0
<br>
<br>
00000000-00000000 : mei_me
<br>
<br>
00000000-00000000 : PCI MMCONFIG 0000 [bus 00-3f]
<br>
<br>
00000000-00000000 : Reserved
<br>
<br>
00000000-00000000 : pnp 00:06
<br>
<br>
00000000-00000000 : Reserved
<br>
<br>
00000000-00000000 : IOAPIC 0
<br>
<br>
Enable reverts back to 0 after the VM using the device shuts down,
and trying to write 1 in that file still yields "Device or
resource busy". Also, unlike what I said earlier from the last
time I looked into this, it seems doing this no longer allows you
to dump the ROM after the VM has shut down, with the same error
message as before.
<br>
<br>
- Nicolas
<br>
<br>
<br>
On 03/10/2019 10:09 PM, Alex Williamson wrote:
<br>
<blockquote type="cite">On Sun, 10 Mar 2019 18:06:37 -0400
<br>
Nicolas Roy-Renaud <a class="moz-txt-link-rfc2396E" href="mailto:nicolas.roy-renaud.1@ens.etsmtl.ca"><nicolas.roy-renaud.1@ens.etsmtl.ca></a>
wrote:
<br>
<br>
<blockquote type="cite">I've seen a lot of people before
reccomand VFIO newcomers to flash their
<br>
GPU if they couldn't get their passthrough working right
before, and
<br>
since I know how potentially risky and avoidable this sort of
procedure
<br>
is (since QEMU lets you just pass your own ROM to the VM to be
used
<br>
instead), and while attempting to go through the steps myself
<br>
<a class="moz-txt-link-rfc2396E" href="http://vfio.blogspot.com/2014/08/does-my-graphics-card-rom-support-efi.html"><http://vfio.blogspot.com/2014/08/does-my-graphics-card-rom-support-efi.html></a>
<br>
(even though I've had a working VFIO setup for years), got
something
<br>
unexpected.
<br>
<br>
Attempting to dump the ROM from my guest card _/freshly
/_/_after a
<br>
reboot_/ results in the following error message :
<br>
<br>
cat: '/sys/bus/pci/devices/0000:07:00.0/rom': Input/output
error
<br>
<br>
Accompanied by the following like in dmesg :
<br>
<br>
[ 1734.316429] vfio-pci 0000:07:00.0: Invalid PCI ROM header
signature: expecting 0xaa55, got 0xffff
<br>
</blockquote>
If lspci for the device reports:
<br>
<br>
Control: I/O- Mem- BusMaster- ...
<br>
<br>
(specifically Mem-), this could be the reason it's failing. The
PCI
<br>
ROM BAR is a memory region and memory decode needs to be enabled
on the
<br>
device in order to get access to it. Also if you have the
device
<br>
already bound to vfio-pci, the device might be in a D3 low power
state,
<br>
which could make that memory region unavailable. You can use
the
<br>
'enable' file in sysfs to fix both of these, so your sequence
would
<br>
look like this:
<br>
<br>
echo 1 > /sys/bus/pci/devices/0000:07:00.0/enable
<br>
echo 1 > /sys/bus/pci/devices/0000:07:00.0/rom
<br>
cat /sys/bus/pci/devices/0000:07:00.0/rom > gpu.rom
<br>
echo 0 > /sys/bus/pci/devices/0000:07:00.0/rom
<br>
echo 0 > /sys/bus/pci/devices/0000:07:00.0/enable
<br>
<br>
Thanks,
<br>
Alex
<br>
</blockquote>
<br>
_______________________________________________
<br>
vfio-users mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:vfio-users@redhat.com">vfio-users@redhat.com</a>
<br>
<a class="moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/vfio-users">https://www.redhat.com/mailman/listinfo/vfio-users</a>
<br>
</blockquote>
<br>
</body>
</html>