[vfio-users] Boot using second GPU?

Sun Aug 7 14:49:18 UTC 2016

I an running a gigabyte z170 board and cannot set primary graphic card. Primary is what is in PCIe 1.

On August 7, 2016 7:14:05 AM PDT, Rokas Kupstys <rokups at zoho.com> wrote:
>Thanks for reply. Since i am making a new build i am looking for proper
>motherboard. One i sided with is from asus, but it seems asus
>motherboards do not support switching primary GPU. I might go with
>gigabyte if there is no way to solve this. I am not sure yet though.
>Switching primary gpu in bios is ultimate solution. Directions i
>provided in earlier mail are workaround for people who do not have such
>capability. It is a tricky choice.. Go with beloved asus and suffer
>minor inconvenience or go with gigabyte.. Did you have any problems
>with
>gigabyte motherboard(s)?
>
>
>Rokas Kupstys
>
>On 2016.08.05 14:44, Hristo Iliev wrote:
>> Am 05.08.2016 10:22, schrieb Rokas Kupstys:
>>
>>> Okay this is unexpected luck. After more tinkering i got it to work!
>>> Here is my setup:
>>>
>>> * AMD FX-8350 CPU + Sabertooth 990FX R2 motherboard
>>> * 0000:01:00.0 - gpu in first slot
>>> * 0000:06:00.0 - gpu in third slot
>>> * UEFI on host and guest.
>>> * Archlinux
>>>
>>> In order to make host use non-boot GPU:
>>>
>>> 1. Add Kernel boot parameter "video=efifb:off". This makes kernel
>not
>>> use first gpu and boot messages appear on second gpu.
>>>
>>> 2. Bind first gpu (0000:01:00.0) to vfio-pci driver. I did this by
>>> adding line
>>>
>>>> options vfio-pci         ids=1002:677B,1002:AA98
>>> to /etc/modprobe.d/kvm.conf. They are obtained from "lspci -n" which
>>> in my case show:
>>>
>>>> 01:00.0 0300: 1002:677B
>>>> 01:00.1 0403: 1002:AA98
>>> 3. Configure xorg to use second gpu (0000:06:00.0). I added file
>>> /etc/X11/xorg.conf.d/secondary-gpu.conf with contents:
>>>
>>>> Section "Device"
>>>> Identifier     "Device0"
>>>> Driver         "radeon"
>>>> VendorName     "AMD Corporation"
>>>> BoardName      "AMD Secondary"
>>>> BusID          "PCI:6:0:0"
>>>> EndSection
>>> And thats it! Now when machine boots it shows POST messages and
>>> bootloader on first gpu, but as soon as boot option is selected
>>> display goes blank and kernel boot messages show on second gpu.
>After
>>> boot you can assign first gpu to VM as usual and it works.
>>> HELP REQUEST: could someone with intel hardware (ideally x99
>chipset)
>>> test this method? I am planning a build and if this works i could
>>> settle with 28 lane cpu and save couple hundred dollars. Intel's 40
>>> lane cpus are way overpriced.. And with 28 lane cpus only first slot
>>> can run at x16 speed while other slots downgrade to x8 or less.
>>> Anyhow i would love to hear if this works on intel hardware.
>>>
>>
>> Hi,
>>
>> I have a Gigabyte GA-X99-UD4 motherboard and i7-5820K. There are two
>GPUs
>> in it - a GTX 970 for pass-through on 03:00.0 and a GT 730 as primary
>GPU
>> on 06:00.0. The PCIE slot of the GT is selected as primary video
>output
>> in the UEFI settings. All text and graphics output goes to it - the
>> output
>> of the GTX card remains off the entire time until the VM is booted.
>The X
>> server does see both cards but since the nvidia module is prevented
>from
>> binding to the GTX, X cannot use it and starts on the GT. No fiddling
>> with
>> the console driver parameters necessary.
>>
>> Distribution:
>>    Arch Linux, 4.6.4-1-ARCH
>>
>> Kernel parameters:
>>    ... pci-stub.ids=10de:13c2,10de:0fbb,8086:8d20
>nvidia-drm.modeset=1
>> ...
>>
>> /etc/modprobe.d/vfio.conf:
>>    options vfio-pci ids=10de:13c2,10de:0fbb,8086:8d20
>>
>> /etc/mkinitcpio.conf:
>>    ...
>>    MODULES="vfio vfio_iommu_type1 vfio_pci vfio_virqfd vfat
>aes_x86_64
>> crc32c_intel nvidia nvidia_modeset nvidia_uvm nvidia_drm"
>>    ...
>>
>> /etc/X11/xorg.conf.d/20-nvidia.conf:
>>    Section "Device"
>>     Identifier                "Device0"
>>     Driver                    "nvidia"
>>     VendorName                "NVIDIA Corporation"
>>     Option "ConnectToAcpid"   "0"
>>    EndSection
>>
>> The only problem with my setup is that the GTX is in PCIE_2, which
>works
>> as x8 with i7-5820K installed. I cannot fit the card in PCIE_1
>because of
>> the oversized CPU cooler. This doesn't actually bug me at all as
>multiple
>> tests (for example, [1]) have shown negligible difference in gaming
>FPS
>> between PCI-e 3.0 x8 and x16. The GT card is in PCIE_4.
>>
>> Kind regards,
>> Hristo
>>
>> [1]
>>
>http://www.gamersnexus.net/guides/2488-pci-e-3-x8-vs-x16-performance-impact-on-gpus
>>
>>> Rokas Kupstys
>>>
>>> On 2016.08.05 10:34, Rokas Kupstys wrote:
>>>
>>> I think i got half-way there.. My primary gpu is at 0000:01:00.0 and
>>> secondary on 0000:06:00.0. I used following xorg config:
>>>
>>> Section "Device"
>>> Identifier     "Device0"
>>> Driver         "radeon"
>>> VendorName     "AMD Corporation"
>>> BoardName      "AMD Secondary"
>>> BusID          "PCI:6:0:0"
>>> EndSection
>>>
>>> After booting 0000:06:00.0 was still bound to vfio-pci (im yet to
>sort
>>> it out why as i removed modprobe configs and kernel parameters) and
>i
>>> ran following script to bind gpu to correct driver:
>>>
>>> #!/bin/bash
>>>
>>> unbind() {
>>> dev=$1
>>> if [ -e /sys/bus/pci/devices/${dev}/driver ]; then
>>> echo "${dev}" > /sys/bus/pci/devices/${dev}/driver/unbind
>>> while [ -e /sys/bus/pci/devices/${dev}/driver ]; do
>>> sleep 0.1
>>> done
>>> fi
>>> }
>>>
>>> bind() {
>>> dev=$1
>>> driver=$2
>>> vendor=$(cat /sys/bus/pci/devices/${dev}/vendor)
>>> device=$(cat /sys/bus/pci/devices/${dev}/device)
>>> echo "${vendor} ${device}" > /sys/bus/pci/drivers/${driver}/new_id
>>> echo "$dev" > /sys/bus/pci/drivers/${driver}/bind
>>> }
>>>
>>> unbind "0000:06:00.0"
>>> bind "0000:06:00.0" "radeon"
>>> #unbind "0000:01:00.0"
>>>
>>> After restarting sddm.service (display manager) i could switch to
>>> secondary gpu and log in to desktop. All worked. Problem is i can
>not
>>> unbind 0000:01:00.0 so i could pass-through it. Attempt to unbind
>driver
>>> resulted in display freezing. Even secondary gpu froze.
>>>
>>> Rokas Kupstys
>>>
>>> On 2016.08.05 04:55, Nicolas Roy-Renaud wrote:
>>>
>>> That's something you should fix in the BIOS. The boot GPU is special
>>> because the motherboard has to use it to display things such as POST
>>> messages and such, so it's already "tainted" by the time the kernel
>>> gets a hold of it. I had to put my guest GPU on my motherboard's
>>> second PCI slot because of that (can't change the boot GPU in the
>BIOS
>>> settings), which is pretty unconveinient because it blocks access to
>>> most of my sata ports.
>>>
>>> If there's a way to cleanly pass the boot GPU to a VM, I don't know
>>> about it. I'd be interested to know too, however.
>>>
>>> - Nicolas
>>>
>>> On 2016-08-04 13:59, Rokas Kupstys wrote:
>>>
>>> Hey is it possible to make kernel use GPU other than one that is in
>>> first slot? If so - how?
>>>
>>> I have multiple PCIe slots but only first can run at max speed so i
>>> would like to use it for VGA passthrough. However if i put powerful
>GPU
>>> into the first slot - linux boots using that GPU. I would like to
>make
>>> kernel use GPU in slot 3. So result should be bios and bootloader
>>> running on gpu in slot #1, but kernel should use gpu in slot #3. I
>tried
>>> binding first gpu to vfio-pci driver hoping kernel would use next
>>> available gpu. That did not work, i could see one line with systemd
>>> version in low-res console (normally its high-res). I also tryed
>>> fbcon=map:1234 (not exactly being sure what im doing) but that
>yielded
>>> black screen. Not sure what else i could try.
>>>
>>
>> _______________________________________________
>> vfio-users mailing list
>> vfio-users at redhat.com
>> https://www.redhat.com/mailman/listinfo/vfio-users
>>
>
>
>_______________________________________________
>vfio-users mailing list
>vfio-users at redhat.com
>https://www.redhat.com/mailman/listinfo/vfio-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160807/e766d673/attachment.htm>