[vfio-users] problem with passthrough and unsafe interrupts

[Alex Williamson]

On Sat, Sep 12, 2015 at 12:59 PM, Janusz <januszmk6 at gmail.com> wrote:

> W dniu 12.09.2015 o 19:57, Alex Williamson pisze:
>
> On Sat, Sep 12, 2015 at 11:04 AM, Janusz <januszmk6 at gmail.com> wrote:
>
>> Hello,
>>
>> I have a question about allowing unsafe interrupts, what exactly is
>> happening when we allow it? why its unsafe?
>>
>
> On a standard PC, Message Signaled Interrupts (MSI) are triggered via a
> DMA write by the device to a special address range.  In the early revisions
> of VT-d, the IOMMU did not protect this range, which allowed devices to
> effectively spoof interrupts from other devices and potentially run attacks
> against the host using DMA writes to this interrupt block.  Later versions
> of VT-d introduced the interrupt remapping feature which, among other
> things, protects this range so that a device can only signal the interrupts
> programmed for it.
>
> Another thing that interrupt remapping does is provide a translation
> between IOAPIC and X2APIC interrupt domains on the system.  Pretty much all
> new Intel processors support X2APIC, which necessitates interrupt remapping
> support.  So, I'd be really surprised if your new skylake system doesn't
> have interrupt remapping.  You should really only need to enable the unsafe
> interrupts option if you try to assign the device, it fails, and dmesg
> tells you that you need it.  And only then if you trust your guests not to
> be malicious to the host.
>
>
> thanks for explanation
>
>
>
>> I am asking because I am not able to passthrough my gpu without allowing
>> unsafe interrupts, and have problems in running my virtual machine as it
>> sometimes rebooting its self before its able to run windows. I know that
>> there is also problem with using iGPU and not uefi bios, but its also
>> happening with OVMF and passing uefi rom for my gpu (except that with
>> OVMF sometimes I also get very high load). without passing VGA,
>> everyting works fine. Is it possible that its because those unsafe
>> interrupts?
>> My hardware: i7 6700k, MSI z170a M7, Sapphire R9 290
>>
>
> Congratulations, you're the first person to show up with a Skylake system,
> welcome to trailblazing a new Intel platform.  When you say you're not able
> to assign the GPU without the unsafe interrupts option, does that mean when
> you run the VM QEMU exits and dmesg reports:
>
> "No interrupt remapping support.  Use the module param
> "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform"
>
> That would be pretty epic if Intel dropped interrupt remapping support on
> Skylake.  Or perhaps you just mean that you have something closer to
> working with unsafe interrupts.  If the hardware supports interrupt
> remapping, then the unsafe option makes absolutely no difference.  It's
> only meant to allow an otherwise un-allowed scenario by making the user
> opt-in to the security risk.
>
>
> actually I didn't have CONFIG_IRQ_REMAP in kernel config set... now, when
> I set vfio_iommu_type1.allow_unsafe_interrupts=0, its working
>
>
> those are my options for non-uefi:
>> -enable-kvm -m 10000 -cpu host -smp 8,cores=4,threads=2,sockets=1
>> -device
>> ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1
>> -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on
>> -device vfio-pci,host=01:00.1
>>
>
> Looks like you're adding an ioh3420 for absolutely no reason.  Same with
> the multifunction option.
>
>
> didn't know about that, I took it from some howto on wiki.debian.org,
> disabled it now
>
>
>
>> and for uefi:
>>
>> -drive if=pflash,format=raw,readonly,file=/home/janusz/uefi/OVMF_CODE.fd
>> -drive if=pflash,format=raw,file=/home/janusz/uefi/OVMF_VARS.fd
>> -enable-kvm -m 10000 -cpu host -smp 8,cores=4,threads=2,sockets=1
>> -device
>>
>> vfio-pci,host=01:00.0,romfile=/home/janusz/uefi/uefi-vga.bin,multifunction=on
>> -device vfio-pci,host=01:00.1
>>
>
> Useless multifunction option again.  Why does this one specify a ROM
> file?  Does the on-card ROM not support UEFI?
>
>
> Yes, my GPU doesn't support UEFI, but I found UEFI rom in web for this GPU
>
>
>
>
>> + options with assigning /dev/sdb and two usb devices
>>
>> I am using ovmf build recently from master, kernel 4.2.0 and qemu-2.4.50
>>
>>
>> My iommu groups:
>>
>> /sys/kernel/iommu_groups/0/devices/0000:00:00.0
>>
>
>
>> /sys/kernel/iommu_groups/1/devices/0000:00:01.0
>> /sys/kernel/iommu_groups/1/devices/0000:01:00.0
>> /sys/kernel/iommu_groups/1/devices/0000:01:00.1
>>
>
> So this ought to be the group we care about.
>
>
>> /sys/kernel/iommu_groups/2/devices/0000:00:02.0
>> /sys/kernel/iommu_groups/3/devices/0000:00:08.0
>> /sys/kernel/iommu_groups/4/devices/0000:00:14.0
>> /sys/kernel/iommu_groups/4/devices/0000:00:14.2
>> /sys/kernel/iommu_groups/5/devices/0000:00:15.0
>> /sys/kernel/iommu_groups/5/devices/0000:00:15.1
>> /sys/kernel/iommu_groups/6/devices/0000:00:16.0
>> /sys/kernel/iommu_groups/7/devices/0000:00:17.0
>>
>
>
>> /sys/kernel/iommu_groups/8/devices/0000:00:1c.0
>> /sys/kernel/iommu_groups/8/devices/0000:00:1c.2
>> /sys/kernel/iommu_groups/8/devices/0000:00:1c.7
>> /sys/kernel/iommu_groups/8/devices/0000:02:00.0
>> /sys/kernel/iommu_groups/8/devices/0000:03:00.0
>> /sys/kernel/iommu_groups/8/devices/0000:04:00.0
>>
>
> Lovely, all the PCH root ports are grouped together, people looking for
> hardware please take note.
>
>
>> /sys/kernel/iommu_groups/9/devices/0000:00:1e.0
>> /sys/kernel/iommu_groups/10/devices/0000:00:1f.0
>> /sys/kernel/iommu_groups/10/devices/0000:00:1f.2
>> /sys/kernel/iommu_groups/10/devices/0000:00:1f.3
>> /sys/kernel/iommu_groups/10/devices/0000:00:1f.4
>>
>> and lspci:
>>
>> 00:00.0 Host bridge: Intel Corporation Sky Lake Host Bridge/DRAM
>> Registers (rev 07)
>>
>
>
>> 00:01.0 PCI bridge: Intel Corporation Sky Lake PCIe Controller (x16)
>> (rev 07)
>>
>
>
>> 00:02.0 VGA compatible controller: Intel Corporation Sky Lake Integrated
>> Graphics (rev 06)
>>
>
>
>> 00:08.0 System peripheral: Intel Corporation Sky Lake Gaussian Mixture
>> Model
>>
>
>
>> 00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI
>> Controller (rev 31)
>> 00:14.2 Signal processing controller: Intel Corporation Sunrise Point-H
>> Thermal subsystem (rev 31)
>>
>
> Don't you like how they've put the USB3 controller on a multifunction
> device with some random thermal management device, without isolation of
> course.  Wonder what chaos assigning that to a VM would cause.
>
>
>> 00:15.0 Signal processing controller: Intel Corporation Sunrise Point-H
>> LPSS I2C Controller #0 (rev 31)
>> 00:15.1 Signal processing controller: Intel Corporation Sunrise Point-H
>> LPSS I2C Controller #1 (rev 31)
>>
>
>
>> 00:16.0 Communication controller: Intel Corporation Sunrise Point-H CSME
>> HECI #1 (rev 31)
>>
>
>
>> 00:17.0 SATA controller: Intel Corporation Device a102 (rev 31)
>>
>
>
>> 00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root
>> Port #1 (rev f1)
>> 00:1c.2 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root
>> Port #3 (rev f1)
>> 00:1c.7 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root
>> Port #8 (rev f1)
>> 00:1e.0 Signal processing controller: Intel Corporation Sunrise Point-H
>> LPSS UART #0 (rev 31)
>>
>
>
>> 00:1f.0 ISA bridge: Intel Corporation Sunrise Point-H LPC Controller
>> (rev 31)
>> 00:1f.2 Memory controller: Intel Corporation Sunrise Point-H PMC (rev 31)
>> 00:1f.3 Audio device: Intel Corporation Sunrise Point-H HD Audio (rev 31)
>> 00:1f.4 SMBus: Intel Corporation Sunrise Point-H SMBus (rev 31)
>>
>
> Oh look, the audio controller that used to be a nice separate device is
> now also buried into a multifunction device without isolation, no more
> assigning that to a VM.
>
>
>> 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc.
>> [AMD/ATI] Hawaii PRO [Radeon R9 290]
>
> 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aac8
>>
>
>
>> 02:00.0 USB controller: ASMedia Technology Inc. Device 1242
>> 03:00.0 Ethernet controller: Qualcomm Atheros Device e0a1 (rev 10)
>> 04:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721
>> Gigabit Ethernet PCI Express (rev 21)
>>
>>
>> If its not problem of unsafe interrupts, does anyone know why this can
>> happen?
>>
>
> I don't see any reason to implicate unsafe interrupts.  It would be nice
> to understand exactly what happens without specifying unsafe interrupts.
> It is possible to turn off interrupt remapping support with kernel config
> options and boot options, so please make sure you have CONFIG_IRQ_REMAP=y
> in your kernel and aren't disabling it on the commandline.
>
>
> So, now with unsafe interrupts turned off it works, but any idea why for
> the most of time virtaul machine is restarting itself? when I am trying
> OVMF, its restarting sometimes when windows is starting to load (both
> windows installation process and installed OS, windows 8 and windows 10),
> sometimes at tainocore logo, and sometimes this huge cpu usage... Also I
> noticed that in boot menu of ovmf its detecting only 2GHz CPU and 6GB ram,
> but on ram testing at start - its detecting proper number.
> Without OVMF I have no idea in what step its restarting as monitor gets
> signal only when windows loads, and after windows loads there is no problem
> with using VM, but i know its restarting as there is problem with using
> iGPU with vga passthrough - colors on one of my monitors are corrupted and
> when I restore it with changing mode by xrandr, it breaks again until
> windows boots. Also I hear some weird noise in sound when VM is starting
> and this noise is also indicating that VM is restarting (I have
> pulseaudio). The iGPU problem is also the reason why I want to use OVMF so
> I don't have to fix colors or patch kernel and disable some of iGPU
> functions, but with OVMF restarting problem happens more frequently.
>

So it sounds like you're attempting to do VGA mode assignment with i915 as
the host graphics without patching your kernel for the i915 VGA arbitration
issue.  That's just not going to work.  All of the VGA region accesses for
your assigned device are being intercepted by i915, causing your color
issues and who knows what other damage to the integrity of your system.
VGA mode assignment with Intel integrated primary host graphics requires
both a kernel patch and a commandline option to enable it, you're playing
with your own fate otherwise.

On the OVMF side, you have no log messages from QEMU/libvirt indicating a
VM reboot and nothing in dmesg when a VM reboot occurs?  Is there a BSOD
from the guest or just a silent reboot?  Is there some reason you're
running the development branch of QEMU rather than 2.4.0 proper?  Do you
have the AMD Catalyst driver installed in the guest?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20150912/04c6b61f/attachment.htm>