[vfio-users] new video card. vm wont boot

[Roger Lawhorn]

A search of the modules for 5.4 vs 5.3 shows several vfio modules missing.
I guess there is some big change to 5.4 that I need to know about.

$ find /lib/modules/5.4.43-050443-generic/ | grep vfio
/lib/modules/5.4.43-050443-generic/kernel/drivers/vfio
/lib/modules/5.4.43-050443-generic/kernel/drivers/vfio/mdev
/lib/modules/5.4.43-050443-generic/kernel/drivers/vfio/mdev/mdev.ko

$ find /lib/modules/5.3.0-40-generic/ | grep vfio
/lib/modules/5.3.0-40-generic/kernel/drivers/vfio
/lib/modules/5.3.0-40-generic/kernel/drivers/vfio/mdev
/lib/modules/5.3.0-40-generic/kernel/drivers/vfio/mdev/mdev.ko
/lib/modules/5.3.0-40-generic/kernel/drivers/vfio/mdev/vfio_mdev.ko
/lib/modules/5.3.0-40-generic/kernel/drivers/vfio/vfio_virqfd.ko
/lib/modules/5.3.0-40-generic/kernel/drivers/vfio/vfio.ko
/lib/modules/5.3.0-40-generic/kernel/drivers/vfio/vfio_iommu_type1.ko
/lib/modules/5.3.0-40-generic/kernel/drivers/vfio/pci
/lib/modules/5.3.0-40-generic/kernel/drivers/vfio/pci/vfio-pci.ko

On 5/31/20 12:23 PM, Roger Lawhorn wrote:
> [ 0.886294] pci 0000:0d:00.0: Using iommu direct mapping
> [    0.886624] pci 0000:0d:00.1: Adding to iommu group 19
>
> Is there a reason that the video card is being used by direct mapping 
> instead of being added to a group like the audio?
> Dmesg shows vfio loaded on boot, but vfio-pci is still missing in action.
>
> [    0.948298] VFIO - User Level meta-driver version: 0.3
>
> still cant use modinfo on either.
> i have reached out to the maker of ukuu to get more info.
>
> On 5/31/20 9:14 AM, leesteken at pm.me wrote:
>> On Sunday, May 31, 2020 2:29 PM, Roger Lawhorn <rll at twc.com> wrote:
>>
>>> Thanks for responding.
>>> BTW: the fix for raid0 was to add raid0.default_layout=2 to the command
>>> line for grub for kernels higher than 5.3.3.
>>>
>>> I am using linux mint 19.3.
>>> The highest kernel available is 5.3
>>> I used the ukuu utility to install kernels 5.4,5.5, and 5.6.
>>> In any of those kernels if I do a modinfo vfio or modinfo vfio-pci I 
>>> get
>>> nothing.
>>> Modprobe obviously fails to load the modules. Nothing in lsmod.
>>> Video card boots without being owned by vfio-pci.
>> Other modules are available and are loaded, like ahci, kvm, or amdgpu?
>>
>>> In kernel 5.3 I get the full specs on those two modules with modinfo.
>>>
>>> Honestly, I'd say that ukuu pulled the kernels from ubuntu archives.
>>> If you have the instructions to pull a working kernel for ubuntu with
>>> vfio manually then let me know.
>> I found https://linuxhint.com/upgrade_kernel_linux_mint/. It looks 
>> like ukuu does install packages with modules. I have no experience 
>> with Mint nor ukuu. I don't know why vfio might be missing, sorry.
>>
>> They say Mint 20 with a 5.4 kernel will be releases in June. I guess 
>> you are just unlucky with the timing of your chosen distribution's 
>> release-cycle and its support for your new hardware.
>>
>>> On 5/31/20 3:20 AM, leesteken at pm.me wrote:
>>>> On Sunday, May 31, 2020 5:45 AM, Roger Lawhorn rll at twc.com wrote:
>>>>
>>>>> ok, I got kernel 5.4 booted.
>>>>> They changed how raid0 works and I am one of the few that use raid0.
>>>>> Anyway, vfio-pci is missing in kernel 5.4.
>>>>> Anyone know why?
>>>>> I use a Proxmox/Ubuntu 5.4 kernel and vfio is a module, so I had 
>>>>> to add vfio,
>>>>> vfio_iommu_type1, vfio_pci, vfio_virqfd to /etc/modules and 
>>>>> update-initramfs -u
>>>> Where did you get your kernel? Did you enable VFIO before building it?
>>>>
>>>>> On 5/29/20 8:41 PM, Roger Lawhorn wrote:
>>>>>
>>>>>> This appears to be the navi reset bug.
>>>>>> Supposedly kernel 5.4 fixes this.
>>>>>> However, I am not able to upgrade beyond kernel 5.3.
>>>>>> Kernel 5.4 gets a raid error when trying to read my encryption 
>>>>>> key and
>>>>>> I cannot type my passphrase in order to get booted.
>>>>>> Same with higher kernels.
>>>>>> Alas.
>>>>>> I can sit and watch qemu reset endlessly while trying to boot the 
>>>>>> vm.
>>>>>> On 5/28/20 1:02 PM, Roger Lawhorn wrote:
>>>>>>
>>>>>>> ok,
>>>>>>> I removed all cards but the new one and turned off vfio.
>>>>>>> Linux boots but finds no driver and uses fbdev instead:
>>>>>>> $ inxi -G
>>>>>>> Graphics:
>>>>>>>    Device-1: AMD Navi 14 [Radeon RX 5500/5500M / Pro 5500M] 
>>>>>>> driver: N/A
>>>>>>>    Display: x11 server: X.Org 1.19.6 driver: ati,fbdev
>>>>>>>    unloaded: modesetting,radeon,vesa resolution: 1024x768~76Hz
>>>>>>>    OpenGL: renderer: llvmpipe (LLVM 9.0 128 bits) v: 3.3 Mesa 
>>>>>>> 19.2.8
>>>>>>> So at least its a working card.
>>>>>>> It still wont work with either vm.
>>>>>>> I am wondering if the nvidia gtx 980 ti oc (card for linux side) 
>>>>>>> has
>>>>>>> a hardware issue with the newer card.
>>>>>>> Not compatible.
>>>>>>> On 5/28/20 11:58 AM, Roger Lawhorn wrote:
>>>>>>>
>>>>>>>> I upgraded from a radeon duo pro 8gb hbm to a radeon rx 5500.
>>>>>>>> Neither the win10 vm nor the win7 vm will boot with this card
>>>>>>>> installed.
>>>>>>>> The card is owned by vfio-pci on boot.
>>>>>>>> The script didnt need any changes as it shows up at the same
>>>>>>>> hardware address as the last card.
>>>>>>>> Very frustrated.
>>>>>>>> This card is so new it is listed as not supported till kernel 5.4.
>>>>>>>> I am on kernel 5.3 and cannot get 5.4 to work.
>>>>>>>> But that is for just trying to get linux to use it.
>>>>>>>> Windows should just boot in vga mode and then look for a windows
>>>>>>>> update.
>>>>>>>> I did try removing -vga none to force qemu to emulate a vga driver
>>>>>>>> long enough to boot.
>>>>>>>> No dice. qemu wont pull up the normal vga window on the linux 
>>>>>>>> side.
>>>>>>>> Something has got to give.
>>>>>>>> kind regards, Arjen
>>
>
> _______________________________________________
> vfio-users mailing list
> vfio-users at redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users