[vfio-users] permission denied on /dev/vfio/1 gpu passthrough kvm
Manuel Ullmann
labre at posteo.de
Fri Jan 5 21:13:02 UTC 2018
Hi Tiago,
thanks for trying it. The key should be resolving the issues shown in the serial console. Qemu should output the pseudo terminal devices, which it is attaching to. (/dev/pts/*) You have to use screen (Gentoo package and command line tool) to attach to them. Check the ubuntu wiki link, I sent, for an example. When you find the errors, check the archives for them. I think Alex has explained all kinds of errors a dozen times, so you might be able to resolve them with the existing hints. If you don‘t find any like yours, post them.
It usually comes down to a missing Uefi Bios (in GPU). If you don’t find any errors, you probably have to tell the EFI console to boot. Google for it. I vaguely remember it being something like bootx64
Best regards,
Manuel
>Hello Manuel,
>
>Thank you very much for your help! You've definitely put me on the
>right track.
>Using the following qemu command I'm able to launch the vm without
>errors. I will now work around getting the remainder of the hardware
>working!
>
>LC_ALL=C /usr/bin/qemu-system-x86_64 \
> -machine
>pc-i440fx-2.10,accel=kvm,usb=off,vmport=off,dump-guest-core=off,kernel_irqchip=on
>\
>-cpu
>host,migratable=no,+invtsc,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff
>\
> -m 4096 \
> -realtime mlock=off \
> -smp 3,sockets=1,cores=3,threads=1 \
> -object iothread,id=iothread1 \
> -display none \
> -no-user-config \
> -nodefaults \
> -rtc base=localtime,clock=vm \
> -global kvm-pit.lost_tick_policy=delay \
> -no-hpet \
> -no-reboot \
> -serial pty \
> -serial pty \
> -monitor stdio \
> -nographic \
> -global PIIX4_PM.disable_s3=1 \
> -global PIIX4_PM.disable_s4=1 \
> -boot menu=off,strict=on \
> -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 \
> -device
>ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5
>\
> -device
>ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 \
> -device
>ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 \
> -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 \
>-drive
>file=/usr/share/edk2-ovmf/OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on
>\
>-drive
>file=/var/lib/libvirt/qemu/nvram/win10_VARS.fd,if=pflash,format=raw,unit=1
>\
> -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x6 \
> -device virtio-net-pci,netdev=hostnet0,addr=0xc \
> -netdev tap,ifname=tap0,script=no,downscript=no,id=hostnet0 \
> -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x2 \
> -device
>scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
>\
>-drive
>file=/home/tse/kvm_vm/test1-clone.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0
>\
> -device vfio-pci,host=01:00.1,id=hostdev1,bus=pci.0,addr=0x9 \
> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa \
> -device usb-audio,id=sound0 \
> -msg timestamp=on 2>&1
>
>On Thu, Jan 4, 2018 at 11:08 PM, Manuel Ullmann <labre at posteo.de>
>wrote:
>> Hm,
>>
>> I happen to use a qemu command line currently. I recall, that
>guest/host
>> sound sharing was close to impossible otherwise (using Alsa).
>>
>> I incorporated your drives and vfio devices, but since I don’t have
>> discrete graphics, I have no idea how to get console output with
>> OVMF. Found something here and added the serial ports and stdio
>monitor
>> as suggested. https://wiki.ubuntu.com/UEFI/OVMF
>> You can attach to them with screen.
>>
>> If you don’t get any OVMF errors anymore, but still no output, you
>could
>> try to chmod the vfio device to 0666 temporarily, although this
>should
>> not be necessary, since you don’t get a permission denied error
>anymore.
>>
>> I use the following to initialize my tap device. You probably don’t
>use
>> shorewall, so I commented the restart. Also you should replace the
>> device placeholders and those from ip and gateway. Use ip link and ip
>> addr to get yours.
>>
>> #!/bin/sh
>> date >> /var/log/virsh.log
>> sysctl net.ipv4.ip_forward=1 >> /var/log/virsh.log
>> ip link add name br0 type bridge >>/var/log/virsh.log 2>&1
>> ip link set dev br0 up >>/var/log/virsh.log 2>&1
>> ip link set dev <your_dev> up >>/var/log/virsh.log 2>&1
>> ip link set dev <your_dev> master br0 >>/var/log/virsh.log 2>&1
>> ip tuntap add dev tap0 mode tap user qemu group qemu
>>>/var/log/virsh.log 2>&1
>> ip link set dev tap0 up >>/var/log/virsh.log 2>&1
>> ip link set dev tap0 master br0 >>/var/log/virsh.log 2>&1
>> dhclient -4 br0 >>/var/log/virsh.log 2>&1
>> ip route change <your_ip>/24 via <gateway_ip> >>/var/log/virsh.log
>2>&1
>> #rc-service shorewall restart >>/var/log/virsh.log 2>&1
>> #rc-service shorewall6 restart >>/var/log/virsh.log 2>&1
>> iptables --table nat --append POSTROUTING --out-interface br0 -j
>MASQUERADE >>/var/log/virsh.log 2>&1
>> iptables --insert FORWARD --in-interface tap0 -j ACCEPT
>>>/var/log/virsh.log 2>&1
>> echo "finished net setup" >>/var/log/virsh.log
>>
>> That VM won’t have input devices momentarily. Once qemu indicates a
>> successful start and you get output, kill the VM and replace the
>> commented -device usb-host entries with appropriate ids for your
>> keyboard and mouse device. You get these with lsusb. Bus from lsusb
>> output is hostbus and Device is hostaddr. Should the VM get stuck,
>you
>> can reattach the keyboard to regain control on the host.
>>
>> You were correct. The openrc runlevel is optional. If you would have
>one
>> and can write openrc-run scripts, you could put the preparation and
>> cleaning up stuff into one, so that you later just have to do
>something
>> like init qemuvm and init default to switch into a VM friendly mode
>> (less stuff running) and back.
>>
>https://wiki.gentoo.org/wiki/Handbook:AMD64/Working/Initscripts#Writing_initscripts
>>
>> Let me know, what you get.
>>
>> LC_ALL=C /usr/bin/qemu-system-x86_64 \
>> -machine
>pc-i440fx-2.7,accel=kvm,usb=off,vmport=off,dump-guest-core=off,kernel_irqchip=on
>\
>> -cpu
>host,migratable=no,+invtsc,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff
>\
>> -m 5000 \
>> -realtime mlock=off \
>> -smp 3,sockets=1,cores=3,threads=1 \
>> -object iothread,id=iothread1 \
>> -display none \
>> -no-user-config \
>> -nodefaults \
>> -rtc base=localtime,clock=vm \
>> -global kvm-pit.lost_tick_policy=delay \
>> -no-hpet \
>> -no-reboot \
>> -serial pty \
>> -serial pty \
>> -monitor stdio \
>> -nographic \
>> -global PIIX4_PM.disable_s3=1 \
>> -global PIIX4_PM.disable_s4=1 \
>> -boot menu=off,strict=on \
>> -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 \
>> -device
>ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
>\
>> -device
>ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 \
>> -device
>ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 \
>> -device ahci,id=sata0,bus=pci.0,addr=0x8 \
>> -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5
>\
>> -drive if=none,media=cdrom,id=drive-sata0-0-1,readonly=on \
>> -device ide-cd,bus=sata0.1,drive=drive-sata0-0-1,id=sata0-0-1 \
>> -device ide-cd,bus=sata0.4,drive=drive-sata0-0-4,id=sata0-0-4 \
>> -drive
>file=/usr/share/edk2-ovmf/OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on
>\
>> -drive
>file=/var/lib/libvirt/qemu/nvram/win10_VARS.fd,if=pflash,format=raw,unit=1
>\
>> -device
>virtio-blk-pci,iothread=iothread1,scsi=off,bus=pci.0,addr=0xa,drive=drive-virtio-disk1,id=virtio-disk1
>\
>> -device
>virtio-blk-pci,iothread=iothread1,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk4,id=virtio-disk4
>\
>> -device virtio-net-pci,netdev=hostnet0,addr=0xc \
>> -netdev
>tap,ifname=tap0,script=no,downscript=no,vhost=on,id=hostnet0 \
>> -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x2 \
>> -device vfio-pci,host=01:00.1,id=hostdev1,bus=pci.0,addr=0x9 \
>> # -device
>usb-host,hostbus=3,hostaddr=7,id=hostdev1,bus=usb.0,port=1 \
>> # -device
>usb-host,hostbus=3,hostaddr=6,id=hostdev3,bus=usb.0,port=2 \
>> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 \
>> -device usb-audio,id=sound0 \
>> -msg timestamp=on 2>&1
>>
>>> Additionally, I experimented removing references to hostnet0, and
>>> while I get no error messages, I also fail to get any image on the
>>> display port in the monitor.
>>>
>>> /ts
>>>
>>> On Thu, Jan 4, 2018 at 7:28 PM, Tiago Seco <tiago.seco at gmail.com>
>wrote:
>>>> Hello Manuel
>>>>
>>>> With regards to permissions, I will try getting qemu to run with a
>>>> lower privilege user, but since my setup is not working right now,
>I
>>>> want to impose as little restrictions as possible. Thank you for
>the
>>>> input, as well as the tip regarding hugepages.
>>>>
>>>> As far as your suggestion concerning openrc I must confess I do not
>>>> fully follow what you're suggesting, but you made it sound like it
>was
>>>> an improvement rather than a requirement?
>>>>
>>>> Finally, for virsh start win10-clone
>>>> /var/log/libvirt/qemu/win10-clone.log reads:
>>>>
>>>> 2018-01-04 20:10:41.397+0000: starting up libvirt version: 3.8.0,
>qemu
>>>> version: 2.10.1, hostname: rathalos
>>>> LC_ALL=C
>>>>
>PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/x86_64-pc-linux-gnu/gcc-bin/6.4.0:/usr/lib/llvm/4/bin:/opt/bin
>>>> HOME=/root USER=root QEMU_AUDIO_DRV=none
>/usr/bin/qemu-system-x86_64
>>>> -name guest=win10-clone,debug-threads=on -S -object
>>>>
>secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-8-win10-clone/master-key.aes
>>>> -machine
>pc-i440fx-2.10,accel=kvm,usb=off,vmport=off,dump-guest-core=off
>>>> -cpu
>>>>
>Skylake-Client,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,xsaves=on,pdpe1gb=on,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff
>>>> -drive
>file=/usr/share/edk2-ovmf/OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on
>>>> -drive
>file=/var/lib/libvirt/qemu/nvram/win10_VARS.fd,if=pflash,format=raw,unit=1
>>>> -m 4096 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1
>-uuid
>>>> a673c9c8-a1f5-4c3d-97a7-16096bcf1b01 -display none -no-user-config
>>>> -nodefaults -chardev
>>>>
>socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-8-win10-clone/monitor.sock,server,nowait
>>>> -mon chardev=charmonitor,id=monitor,mode=control -rtc
>>>> base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=delay
>>>> -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
>>>> PIIX4_PM.disable_s4=1 -boot strict=on -device
>>>> ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device
>>>>
>ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5
>>>> -device
>ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1
>>>> -device
>ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2
>>>> -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x6 -device
>>>> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive
>>>>
>file=/home/tse/kvm_vm/test1-clone.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0
>>>> -device
>scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
>>>> -drive if=none,id=drive-ide0-0-1,readonly=on -device
>>>> ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -drive
>>>> if=none,id=drive-ide0-1-0,readonly=on -device
>>>> ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev
>>>> tap,fd=21,id=hostnet0 -device
>>>>
>rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:45:6f:82,bus=pci.0,addr=0x3
>>>> -chardev pty,id=charserial0 -device
>>>> isa-serial,chardev=charserial0,id=serial0 -device
>>>> AC97,id=sound0,bus=pci.0,addr=0x4 -chardev
>>>> spicevmc,id=charredir0,name=usbredir -device
>>>> usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev
>>>> spicevmc,id=charredir1,name=usbredir -device
>>>> usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device
>>>> vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x2 -device
>>>> vfio-pci,host=01:00.1,id=hostdev1,bus=pci.0,addr=0x9 -device
>>>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
>>>> 2018-01-04T20:10:41.406580Z qemu-system-x86_64: -chardev
>>>> pty,id=charserial0: char device redirected to /dev/pts/3 (label
>>>> charserial0)
>>>> 2018-01-04T20:10:41.432099Z qemu-system-x86_64: -device
>>>> vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x2: vfio error:
>>>> 0000:01:00.0: failed to open /dev/vfio/1: Permission denied
>>>> 2018-01-04 20:10:41.442+0000: shutting down, reason=failed
>>>>
>>>> Interestingly enough, if I try to run that same command (as root) I
>get:
>>>>
>>>> 2018-01-04T20:14:51.989313Z qemu-system-x86_64: -object
>>>>
>secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-8-win10-clone/master-key.aes:
>>>> Unable to read
>/var/lib/libvirt/qemu/domain-8-win10-clone/master-key.aes:
>>>> Failed to open file
>>>> '/var/lib/libvirt/qemu/domain-8-win10-clone/master-key.aes': No
>such
>>>> file or directory
>>>>
>>>> Removing '-object
>>>>
>secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-8-win10-clone/master-key.aes'
>>>> I get:
>>>>
>>>> 2018-01-04T20:15:57.204616Z qemu-system-x86_64: -chardev
>>>>
>socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-8-win10-clone/monitor.sock,server,nowait:
>>>> Failed to bind socket to
>>>> /var/lib/libvirt/qemu/domain-8-win10-clone/monitor.sock: No such
>file
>>>> or directory
>>>>
>>>> I'm unsure wether or not this file should exist, if I create it,
>and
>>>> run the same command I get:
>>>>
>>>> 2018-01-04T20:17:07.490052Z qemu-system-x86_64: -chardev
>>>> pty,id=charserial0: char device redirected to /dev/pts/3 (label
>>>> charserial0)
>>>> 2018-01-04T20:17:07.490500Z qemu-system-x86_64: -netdev
>>>> tap,fd=21,id=hostnet0: TUNGETIFF ioctl() failed: Bad file
>descriptor
>>>> TUNSETOFFLOAD ioctl() failed: Bad file descriptor
>>>>
>>>> (the logs show nothing)
>>>>
>>>> If I do not create the file, and run the command without any
>>>> references to the monitor (so without '-chardev
>>>>
>socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-8-win10-clone/monitor.sock,server,nowait
>>>> -mon chardev=charmonitor,id=monitor,mode=control'), I get:
>>>>
>>>> 2018-01-04T20:22:04.201180Z qemu-system-x86_64: -chardev
>>>> pty,id=charserial0: char device redirected to /dev/pts/3 (label
>>>> charserial0)
>>>> 2018-01-04T20:22:04.201597Z qemu-system-x86_64: -netdev
>>>> tap,fd=21,id=hostnet0: TUNGETIFF ioctl() failed: Bad file
>descriptor
>>>> TUNSETOFFLOAD ioctl() failed: Bad file descriptor
>>>>
>>>> (Which leaves me wondering wether or not the -chardev and -mon
>>>> switches are really necessary).
>>>>
>>>> I have very little experience with qemu, and could not spot
>anything
>>>> blatantly wrong in that command line.
>>>>
>>>> Once again, thank you very much for your help.
>>>>
>>>> /ts
>>>>
>>>> On Thu, Jan 4, 2018 at 5:32 PM, Manuel Ullmann <labre at posteo.de>
>wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I had initially posted this on the gentoo forums, but since the
>thread
>>>>>> is getting little traction, perhaps here is a better place to ask
>the
>>>>>> question:
>>>>>>
>>>>>> I'm trying to achieve PCI-express GPU passthrough to a Window 10
>VM,
>>>>>> however, when trying to start the virtual machine (as root) with
>the
>>>>>> hardware attached, this is what I get:
>>>>>>
>>>>>> virsh # start win10-clone
>>>>>> error: Failed to start domain win10-clone
>>>>>> error: internal error: process exited while connecting to
>monitor:
>>>>>> 2018-01-03T16:58:47.761436Z qemu-system-x86_64: -chardev
>>>>>> pty,id=charserial0: char device redirected to /dev/pts/4 (label
>>>>>> charserial0)
>>>>>> 2018-01-03T16:58:47.786124Z qemu-system-x86_64: -device
>>>>>> vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x2: vfio error:
>>>>>> 0000:01:00.0: failed to open /dev/vfio/1: Permission denied
>>>>>>
>>>>>> The file however, exists:
>>>>>>
>>>>>> # file /dev/vfio/1
>>>>>> /dev/vfio/1: character special (250/0)
>>>>>> # ls -l /dev/vfio/1
>>>>>> crw------- 1 root root 250, 0 Jan 3 16:51 /dev/vfio/1
>>>>>>
>>>>>>
>>>>>> I followed the steps detailed in
>>>>>> https://wiki.installgentoo.com/index.php/PCI_passthrough for host
>>>>>> preparation, and in
>>>>>>
>https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Setting_up_the_guest_OS
>>>>>> for setting up the guest.
>>>>>>
>>>>>> Before running virsh, I:
>>>>>>
>>>>>> /etc/init.d/libvirtd stop
>>>>>> /etc/init.d/libvirt-guests stop
>>>>>> ./vfio-bind 0000:01:00.0 0000:01:00.1
>>>>>> /etc/init.d/libvirtd start
>>>>>> /etc/init.d/libvirt-guests start
>>>>>>
>>>>>>
>>>>>> I've tried playing around with /etc/libvirt/qemu.conf, namely
>with the
>>>>>> "group", "user", and "dynamic_ownership" settings - even though
>I'm
>>>>>> running everything as root - and with "cgroup_device_acl" (by
>adding
>>>>>> "/dev/vfio/1" to the list).
>>>>>> I cannot figure out what the problem in my setup is or what to
>try
>>>>>> next. Is there anyone that can shine some light on this?
>>>>>>
>>>>>>
>>>>>> System information, scripts used, and qemu USE flags can be found
>>>>>> here: https://pastebin.com/XuGtDmSm
>>>>>>
>>>>>> Thank you!
>>>>> Hello Tiago,
>>>>>
>>>>> I can vaguely recall having the same problem initially. Can you
>verify,
>>>>> that qemu really runs as root? If you strip the libvirt stuff from
>the
>>>>> qemu command line indicated in libvirtd logs and run it as root,
>does
>>>>> that work?
>>>>>
>>>>> Although I never would question the freedom, that Gentoo gives to
>its
>>>>> users, I still would suggest setting user and group to qemu
>though. root
>>>>> is quite insecure and we have a longstanding Intel CPU Bug
>disclosed in
>>>>> a few weeks tops, leaking page table content to userspace, which
>affects
>>>>> VM sandboxing gravely. Add kvm and audio groups to qemu user. If
>you use
>>>>> hugepages, those also have to belong to qemu.
>>>>>
>>>>> vfio nodes (or at least /dev/vfio/vfio) have not much for udev to
>match
>>>>> against, so it could be easier to modify permission in your
>startup
>>>>> scripts. I have this conveniently tackled by using a second
>bootentry
>>>>> with another default boot runlevel, which my /etc/local.d/*.start
>>>>> scripts will recognize and prepare everything for the VM (with
>openrc of
>>>>> course). This makes less sense with discrete graphics passthrough
>>>>> though, so I still would create a dedicated runlevel, but add a
>>>>> openrc-run script, which prepares it, adding a rc_need entry in
>>>>> conf.d/libvirtd.
>>>>>
>>>>> Best regards,
>>>>> Manuel
More information about the vfio-users
mailing list