<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><br>
Thanks for the tips, swapping the cards is not an option as I use
a uatx board but also use all the other pcie ports which would be
blocked by the rx480 otherwise. Adding the vfio-pci module to
initramfs doesnt make any difference unfortunately.<br>
You cant set which gpu is the primary on an Asus board. It just
uses the one in the first pcie slot and you have no other choice,
thats why I said I made a mistake buying Asus ;)<br>
</p>
But everything works as intended in v4.14.9, it 'just' doesnt
anymore since v4.14.10<br>
<br>
@Alex, do you maybe have any suggestions? Unfortunately I am having
some difficulties understanding the changelog of kernel v4.14.10 and
their possible impact on my problem.<br>
<br>
Cheers, Pim<br>
<br>
<div class="moz-cite-prefix">On 11/01/18 21:43, Peter Maloney wrote:<br>
</div>
<blockquote type="cite"
cite="mid:2865c59d-bab6-eaa1-b874-d8aad3acfdc6@brockmann-consult.de">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div class="moz-cite-prefix">Ok, well they have unique ids, so you
can use the cmdline like you were. (And my script runs in
initramfs, and looks at the kernel cmdline, and doesn't care
what motherboard you have.) And I don't know if it would really
stop you because one is primary... I would think it would just
go blank or stop updating the screen instead (and likely not
work in a guest). I don't have much experience with trying to
send through the primary gpu... I think that's a bad idea...why
not just swap the cards?<br>
<br>
And I have a gigabyte board with a slight problem...it puts the
bios/efi on the 3rd gpu and that gpu will hang the host if I put
an r7 in there, but works with an old HD. How do you set which
one is primary?<br>
<br>
And regarding the loading amdgpu before vfio... you could make
sure your initramfs has vfio-pci in it, and that shouldn't
happen. Or you can test by blacklisting amdgpu (and rebuilding
initramfs0 just to see if vfio-pci binds then.<br>
<br>
On 01/11/18 11:47, P. Pronk wrote:<br>
</div>
<blockquote type="cite"
cite="mid:76af4607-b3c6-65ab-f9ce-9f3e38069d42@pronk.nl">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<p><br>
Unfortunately I have to use the kernel cmdline syntax as I
made a mistake of buying an Asus motherboard. The RX480 is my
primary/boot gpu and you cant change that in the Asus bios
like you can with eg Gigabyte.</p>
<p>Thinking about it, this is probably also the reason why vfio
doesnt bind anymore. Probably something changed in the kernel
because of which the primary/boot gpu cant be 'unloaded'
anymore?<br>
<br>
Comparing the dmesg outputs, it looks like the amd driver is
loaded before vfio in 4.14.13, eg dmesg still lists that vfio
is adding the 67df device but the amdgpu driver is already
loaded then. See excerpts of dmesg in 4.14.13 below:</p>
<p>[ 0.208096] pci 0000:02:00.0: vgaarb: VGA device added:
decodes=io+mem,owns=none,locks=none<br>
[ 0.208096] pci 0000:01:00.0: vgaarb: setting as boot VGA
device<br>
[ 0.208096] pci 0000:01:00.0: vgaarb: VGA device added:
decodes=io+mem,owns=io+mem,locks=none<br>
[ 0.208122] pci 0000:01:00.0: vgaarb: bridge control
possible<br>
[ 0.208152] pci 0000:02:00.0: vgaarb: bridge control
possible<br>
[ 0.208181] vgaarb: loaded<br>
...<br>
[ 0.276067] pci 0000:01:00.0: Video device with shadowed
ROM at [mem 0x000c0000-0x000dffff]<br>
...<br>
[ 4.584958] amdgpu 0000:01:00.0: Invalid PCI ROM header
signature: expecting 0xaa55, got 0xffff<br>
[ 4.585028] ATOM BIOS: 113-V34111-F1<br>
[ 4.585050] [drm] GPU post is not needed<br>
[ 4.585348] [drm] vm size is 64 GB, block size is 13-bit,
fragment size is 4-bit<br>
[ 4.585416] amdgpu 0000:01:00.0: VRAM: 8192M
0x000000F400000000 - 0x000000F5FFFFFFFF (8192M used)<br>
[ 4.585461] amdgpu 0000:01:00.0: GTT: 256M
0x0000000000000000 - 0x000000000FFFFFFF<br>
[ 4.585501] [drm] Detected VRAM RAM=8192M, BAR=256M<br>
[ 4.585525] [drm] RAM width 256bits GDDR5<br>
[ 4.585549] [drm] amdgpu: 8192M of VRAM memory ready<br>
[ 4.585572] [drm] amdgpu: 8192M of GTT memory ready.<br>
[ 4.585603] [drm] GART: num cpu pages 65536, num gpu pages
65536<br>
[ 4.585673] [drm] PCIE GART of 256M enabled (table at
0x000000F400040000).<br>
[ 4.585719] [drm] Supports vblank timestamp caching Rev 2
(21.10.2<br>
[ 4.585799] amdgpu 0000:01:00.0: amdgpu: using MSI.<br>
[ 4.585832] [drm] amdgpu: irq initialized.<br>
[ 4.595990] usb 3-10: new full-speed USB device number 10
using xhci_hcd<br>
[ 4.697313] amdgpu: [powerplay] amdgpu: powerplay sw
initialized<br>
[ 4.697560] [drm] AMDGPU Display Connectors<br>
[ 4.697581] [drm] Connector 0:<br>
...<br>
[ 6.013088] amdgpu 0000:01:00.0: fb1: amdgpudrmfb frame
buffer device<br>
[ 6.013235] amdgpu 0000:01:00.0: kfd not supported on this
ASIC<br>
[ 6.013255] [drm] Initialized amdgpu 3.19.0 20150101 for
0000:01:00.0 on minor 1<br>
[ 6.024444] scsi 0:0:0:0: Direct-Access Generic Ultra
HS-SD/MMC 1.82 PQ: 0 ANSI: 0<br>
[ 6.024627] sd 0:0:0:0: Attached scsi generic sg0 type 0<br>
[ 6.049487] sd 0:0:0:0: [sda] Attached SCSI removable disk<br>
[ 6.061022] VFIO - User Level meta-driver version: 0.3<br>
[ 6.064451] vfio_pci: add [1002:67df[ffff:ffff]] class
0x000000/00000000<br>
[ 6.088016] vfio_pci: add [1002:aaf0[ffff:ffff]] class
0x000000/00000000<br>
<br>
</p>
<p>As you requested, the output of lspci (in 4.14.13). The 67ed
is a RX460 which I use for the host, the RX480 is for the
guest.<br>
</p>
<p>01:00.0 VGA compatible controller [0300]: Advanced Micro
Devices, Inc. [AMD/ATI] Device [1002:67df] (rev c7)<br>
Subsystem: Micro-Star International Co., Ltd. [MSI]
Device [1462:3413]<br>
Kernel driver in use: amdgpu<br>
Kernel modules: amdgpu<br>
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc.
[AMD/ATI] Device [1002:aaf0]<br>
Subsystem: Micro-Star International Co., Ltd. [MSI]
Device [1462:aaf0]<br>
Kernel driver in use: vfio-pci<br>
Kernel modules: snd_hda_intel<br>
02:00.0 VGA compatible controller [0300]: Advanced Micro
Devices, Inc. [AMD/ATI] Device [1002:67ef] (rev cf)<br>
Subsystem: XFX Pine Group Inc. Device [1682:9460]<br>
Kernel driver in use: amdgpu<br>
Kernel modules: amdgpu<br>
02:00.1 Audio device [0403]: Advanced Micro Devices, Inc.
[AMD/ATI] Device [1002:aae0]<br>
Subsystem: XFX Pine Group Inc. Device [1682:aae0]<br>
Kernel driver in use: snd_hda_intel<br>
Kernel modules: snd_hda_intel<br>
<br>
<br>
</p>
<br>
<div class="moz-cite-prefix">On 11/01/18 10:57, Peter Maloney
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:f1859857-cc16-50a5-8794-20f0f9316594@brockmann-consult.de">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<div class="moz-cite-prefix">Let's see an `lspci -knn` for
each of those devices.<br>
<br>
I'm using 4.14.x and have 3 AMD gpus, one for the host, and
2 for VMs, and it works.<br>
<br>
But I don't use the pci-stub.ids or the vfio-pic.ids kernel
cmdline syntax... I bind by pci address instead of
vendor:device since they're non-unique (using my script here
<a class="moz-txt-link-freetext"
href="https://github.com/petermaloney/misc/blob/master/mkinitcpio-vfio-pci/hooks/vfio-pci"
moz-do-not-send="true">https://github.com/petermaloney/misc/blob/master/mkinitcpio-vfio-pci/hooks/vfio-pci</a>).
PCI address can change on firmware updates or moving cards
around, but stays the same otherwise in my experience.<br>
<br>
On 01/11/18 10:24, P. Pronk wrote:<br>
</div>
<blockquote type="cite"
cite="mid:2609b0da-d492-e7d6-0fd4-06fc2e9ec4b8@pronk.nl">
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
<p><font size="-1"><br>
Hi,<br>
<br>
</font>Is someone successfully using a kernel version of
4.14.10 or higher with an AMD graphics card? It seems my
RX480 vga controller (67df) wont use the vfio-pci driver
in 4.14.10+ anymore, even though the RX480 audio device
(aaf0) will. I have both
'pci-stub.ids=1002:67df,1002:aaf0' listed in my grub
cmdline as 'options vfio-pci ids=1002:67df,1002:aaf0' in
modprobe.d/vfio-pci.conf<br>
</p>
<p>Did something change since 4.14.10? Checking the kernel
changelogs doesnt show anything (immediately) related to
either vfio-pci or the amd gpu kernel driver.<br>
<br>
To be clear, in kernel version 4.14.9 everything still
works as expected for me.<br>
</p>
<p>Thanks, Pim<br>
</p>
<p><br>
</p>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
vfio-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:vfio-users@redhat.com" moz-do-not-send="true">vfio-users@redhat.com</a>
<a class="moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/vfio-users" moz-do-not-send="true">https://www.redhat.com/mailman/listinfo/vfio-users</a>
</pre>
</blockquote>
<p><br>
</p>
</blockquote>
<br>
</blockquote>
<p><br>
</p>
</blockquote>
<br>
</body>
</html>